Title: Markov Persuasion Processes: Learning to Persuade From Scratch
Venue: NeurIPS 2025
Accepted: yes
Keywords: Bayesian Persuasion

MPP
Abstract: In Bayesian persuasion, an informed sender strategically discloses information to a receiver so as to persuade them to undertake desirable actions. Recently, Markov persuasion processes (MPPs) have been introduced to capture sequential scenarios where a sender faces a stream of myopic receivers in a Markovian environment. The MPPs studied so far in the literature suffer from issues that prevent them from being fully operational in practice, e.g., they assume that the sender knows receivers' rewards. We fix such issues by addressing MPPs where the sender has no knowledge about the environment. We design a learning algorithm for the sender, working with partial feedback.
We prove that its regret with respect to an optimal information-disclosure policy grows sublinearly in the number of episodes, as it is the case for the loss in persuasiveness cumulated while learning. Moreover, we provide lower bounds for our setting matching the guarantees of our algorithm.

Markov Persuasion Processes: Learning to Persuade From Scratch

Abstract

In Bayesian persuasion , an informed sender strategically discloses information to a receiver so as to persuade them to undertake desirable actions. Recently, Markov persuasion processes (MPPs) have been introduced to capture sequential scenarios where a sender faces a stream of myopic receivers in a Markovian environment. The MPPs studied so far in the literature suffer from issues that prevent them from being fully operational in practice, e.g. , they assume that the sender knows receivers' rewards . We fix such issues by addressing MPPs where the sender has no knowledge about the environment. We design a learning algorithm for the sender, working with partial feedback. We prove that its regret with respect to an optimal information-disclosure policy grows sublinearly in the number of episodes, as it is the case for the loss in persuasiveness cumulated while learning. Moreover, we provide lower bounds for our setting matching the guarantees of our algorithm.

1 Introduction

Bayesian persuasion [Kamenica and Gentzkow, 2011] studies how an informed sender should strategically disclose information to influence the behavior of a self-interested receiver. Bayesian persuasion has received a growing attention over the last years, since it captures several fundamental problems arising in real-world applications, such as, e.g. , online advertising [Bro Miltersen and Sheffet, 2012, Emek et al., 2014, Badanidiyuru et al., 2018, Bacchiocchi et al., 2022], voting [Cheng et al., 2015, Alonso and Cˆ amara, 2016, Castiglioni et al., 2020a, Castiglioni and Gatti, 2021], traffic routing [Vasserman et al., 2015, Bhaskar et al., 2016, Castiglioni et al., 2021a], recommendation systems [Mansour et al., 2016], e-commerce [Castiglioni et al., 2022], security [Rabinovich et al., 2015, Xu et al., 2016], marketing [Babichenko and Barman, 2017, Candogan, 2019], clinical trials [Kolotilin, 2015], and financial regulation [Goldstein and Leitner, 2018].

Most of the works on Bayesian persuasion focus on one-shot interactions, where information disclosure is performed 'one shot'. Despite real-world problems are usually sequential , there are only few exceptions that consider multi-step information disclosure [Wu et al., 2022, Gan et al., 2022,

Francesco Emanuele Stradi ∗

Alberto Marchesi

2023, Bernasconi et al., 2022, 2023b, Iyer et al., 2023, Lin et al., 2024]. Specifically, Wu et al. [2022] initiated the study of Markov persuasion processes (MPPs), which model scenarios where a sender sequentially faces a stream of myopic receivers in an unknown Markovian environment. In each state of the environment, the sender privately observes some information-encoded in an outcome stochastically determined according to a prior distribution-and faces a new receiver, who is then called to take an action. The outcome and the receiver's action jointly determine the agents' rewards and the next state. In an MPP, the goal of the sender goal is to disclose information about the outcome at each state, so as to persuade the receivers to take actions maximizing sender's long-term rewards.

The MPP formalism finds application in several real-world settings, such as e-commerce and recommendation systems [Wu et al., 2022]. For example, an MPP can model the problem faced by an online streaming platform recommending movies to its users. The platform has an informational advantage over users ( e.g. , it has access to views statistics), and it exploits available information to induce users to watch suggested movies, so as to maximize views. The MPPs studied by Wu et al. [2022] rely on some limiting assumptions that prevent them from being fully operational in practice. For instance, they make the assumption that the sender has perfect knowledge of receiver's rewards . In the online streaming platform example, this assumption requires that the platform knows everything about users' (private) preferences over movies, which is unrealistic in practice.

1.1 Original Contributions

We relax the assumptions of Wu et al. [2022], by addressing MPPs where the sender does not know anything about the environment . We consider settings in which they have no knowledge about transitions, prior distributions over outcomes, sender's stochastic rewards, and receivers' ones. Ideally, the goal is to design learning algorithms that are persuasive and attain regret sublinear in the number of episodes of learning T . The regret is the difference between sender's rewards cumulated over the episodes and what would have been obtained by always using an optimal information-disclosure policy. Persuasiveness is about ensuring receivers are incentivized to take desired actions. Learning in MPPs without knowledge of receivers' rewards begets considerable additional challenges compared to the case of Wu et al. [2022]. Indeed, the latter design a sublinear-regret algorithm that is persuasive at every episode with high probability, while we show that this is not attainable in our setting. Intuitively, this is due to the fact that, since the sender does not know receivers' rewards, some episodes must be used to learn how to be 'approximately' persuasive. As a consequence, in this work we look for algorithms that attain sublinear regret while ensuring that the cumulative violation of persuasiveness grows sublinearly in T . This is the most natural requirement in all the cases where persuasiveness cannot be achieved at every episode, and it has already been addressed in related settings (see, e.g. , [Bernasconi et al., 2022, Cacciamani et al., 2023, Gan et al., 2023]).

As a warm-up, we start studying a full feedback case where, after each episode, the sender observes the reward associated with every possible action in all the state-outcome pairs encountered during the episode. We propose an algorithm, called Optimistic Persuasive Policy Search ( OPPS ), which uses information-disclosure policies computed by being optimistic with respect to both sender's expected rewards and persuasiveness requirements. We show that, under full feedback, OPPS attains ˜ O ( √ T ) regret and violation. Then, we switch to the partial feedback case, where the sender only observes the rewards for the state-outcome-action triplets actually visited during the episode. We extend the OPPS algorithm to this setting, by adding a preliminary exploration phase having the goal of gathering as much feedback as possible about persuasiveness. After that, the algorithm switches to an optimistic approach over information-disclosure policies that are 'approximately' persuasive. We prove that OPPS with partial feedback attains ˜ O ( T α ) regret and ˜ O ( T 1 -α / 2 ) violation, where α ∈ [ 1 / 2 , 1] is a parameter controlling the amount of exploration. Finally, we provide a lower bound showing that the trade-off between regret and violation achieved by means of OPPS is tight.

1.2 Related Works

We refer to Appendix A for additional related works.

The work most related to ours is [Wu et al., 2022], studying MPPs where the sender knows everything about receivers' rewards, with the only elements unknown to them being their rewards, transition probabilities, and prior distributions. Moreover, Wu et al. [2022] also assume that the receivers know everything about the environment, so as to select a best-response action, and that all rewards are

deterministic. In contrast, we consider MPPs in which sender and receivers have no knowledge of the environment, including their rewards, which we assume to be stochastic. Other related works are [Gan et al., 2022], studying Bayesian persuasion problems where a sender sequentially interacts with a myopic receiver in a multi-state environment, and [Bernasconi et al., 2023b], addressing MPPs with a farsighted receiver. These two works considerably depart from ours, as they both assume that the sender knows everything about the environment, including transitions, priors, and rewards. Thus, they are not concerned with learning problems. Finally, [Bernasconi et al., 2022] studies settings where a sender faces a farsighted receiver in a sequential environment with a tree structure, addressing the case in which the only elements unknown to the sender are the prior distributions over outcomes, while rewards are deterministic and known. The tree structure considerably eases learning, as it intuitively allows to factor the uncertainty about transitions in the rewards at the leaves of the tree.

Our work is also related to learning in one-shot Bayesian persuasion played repeatedly [Castiglioni et al., 2020b, 2021b, Zu et al., 2021, Bernasconi et al., 2023a], and works on online Markov decision processes (MDPs) [Auer et al., 2008, Even-Dar et al., 2009, Neu et al., 2010, Rosenberg and Mansour, 2019, Jin et al., 2020], in particular those on constrained MDPs [Wei et al., 2018, Zheng and Ratliff, 2020, Efroni et al., 2020, Qiu et al., 2020, Stradi et al., 2024].

2 Preliminaries

2.1 Bayesian Persuasion

The classical Bayesian persuasion framework introduced by Kamenica and Gentzkow [2011] models a one-shot interaction between a sender and a receiver . The latter has to take an action a from a finite set A , while the former privately observes an outcome ω sampled from a finite set Ω according to a prior distribution µ ∈ ∆(Ω) , which is known to both the sender and the receiver. 2 The rewards of both agents depend on the receiver's action and the realized outcome, as defined by the functions r S , r R : Ω × A → [0 , 1] , where r R ( ω, a ) and r S ( ω, a ) denote the rewards of the sender and the receiver, respectively, when the outcome is ω ∈ Ω and action a ∈ A is played. The sender can strategically disclose information about the outcome to the receiver, by publicly committing to a signaling scheme ϕ , which is a randomized mapping from outcomes to signals being sent to the receiver. Formally, ϕ : Ω → ∆( S ) , where S denotes a suitable finite set of signals. For ease of notation, we let ϕ ( ·| ω ) ∈ ∆( S ) be the probability distribution over signals employed by the sender when the realized outcome is ω ∈ Ω , with ϕ ( s | ω ) being the probability of sending signal s ∈ S .

The sender-receiver interaction goes as follows: (i) the sender publicly commits to a signaling scheme ϕ ; (ii) the sender observes the realized outcome ω ∼ µ and draws a signal s ∼ ϕ ( ·| ω ) ; and (iii) the receiver observes the signal s and plays an action. Specifically, after observing s under a signaling scheme ϕ , the receiver infers a posterior distribution over outcomes and plays a best-response action b ϕ ( s ) ∈ A according to such distribution. Formally, b ϕ ( s ) ∈ arg max a ∈ A ∑ ω ∈ Ω µ ( ω ) ϕ ( s | ω ) r R ( ω, a ) , where the expression being maximized encodes the (unnormalized) expected reward of the receiver. As it is customary in the literature (see, e.g. , [Dughmi and Xu, 2016]), we assume that the receiver breaks ties in favor of the sender, by selecting a best response maximizing sender's expected reward when multiple best responses are available.

The goal of the sender is to commit to a signaling scheme ϕ that maximizes their expected reward, which is computed as follows: ∑ ω ∈ Ω µ ( ω ) ∑ s ∈S ϕ ( s | ω ) r S ( ω, b ϕ ( s )) .

2.2 Markov Persuasion Processes

An MPP [Wu et al., 2022] generalizes one-shot Bayesian persuasion to settings where the sender faces a stream of receivers in an MDP, with each receiver myopically taking an action maximizing immediate reward. An ( episodic ) MPP is a tuple M := ( X,A, Ω , µ, P, { r S,t } T t =1 , { r R,t } T t =1 ) :

T is the number of episodes. 3

X , A , and Ω are finite sets of states, actions, and outcomes, respectively.

µ : X → ∆(Ω) is a prior function defining a probability distribution over outcomes at each state. We let µ ( ω | x ) be the probability of sampling outcome ω ∈ Ω in state x ∈ X .

P : X × Ω × A → ∆( X ) is a transition function. We let P ( x ′ | x, ω, a ) be the probability of going from x ∈ X to x ′ ∈ X by taking action a ∈ A , when the outcome in state x is ω ∈ Ω .

{ r S,t } T t =1 is a sequence specifying a sender's reward function r S,t : X × Ω × A → [0 , 1] at each episode t . Given x ∈ X , ω ∈ Ω , and a ∈ A , each r S,t ( x, ω, a ) for t ∈ [ T ] is sampled independently from a distribution ν S ( x, ω, a ) ∈ ∆([0 , 1]) with mean r S ( x, ω, a ) .

{ r R,t } T t =1 is a sequence defining a receivers' reward function r R,t : X × Ω × A → [0 , 1] at each episode t . Given x ∈ X , ω ∈ Ω , and a ∈ A , each r R,t ( x, ω, a ) for t ∈ [ T ] is sampled independently from a distribution ν R ( x, ω, a ) ∈ ∆([0 , 1]) with mean r R ( x, ω, a ) .

We focus w.l.o.g. on loop-free episodic MPPs, as customary in online learning in MDPs (see, e.g. , [Rosenberg and Mansour, 2019]). In a loop-free MPP, states are partitioned into L +1 layers X 0 , . . . , X L such that X 0 := { x 0 } and X L := { x L } , with x 0 being the initial state starting the episode and x L being the final one, in which the episode ends. Moreover, by letting K := [0 . . . L -1] for ease of notation, P ( x ′ | x, ω, a ) > 0 only when x ′ ∈ X k +1 and x ∈ X k for some k ∈ K . 4

At each episode of an episodic MPP, the sender commits to a signaling policy ϕ : X × Ω → ∆( S ) , which defines a probability distribution over a finite set S of signals for the receivers for every state x ∈ X and outcome ω ∈ Ω . For ease of notation, we denote by ϕ ( ·| x, ω ) ∈ ∆( S ) such probability distributions, with ϕ ( s | x, ω ) being the probability of sending a signal s ∈ S in state x when the realized outcome is ω . Similarly to one-shot Bayesian persuasion, a myopic receiver acting at state x ∈ X and receiving signal s ∈ S infers a posterior distribution over outcomes and plays a best-response action. We denote by b ϕ ( s, x ) ∈ A the best response played by such a receiver under the signaling policy ϕ (assuming ties are broken in favor of the sender).

As customary in Bayesian persuasion (see, e.g. , [Arieli and Babichenko, 2019]), a revelation-principlestyle argument allows to focus w.l.o.g. on signaling policies that are direct and persuasive. Formally, a signaling policy is direct if the set of signals coincides with the set of actions, namely S = A . Intuitively, signals should be interpreted as action recommendations for the receivers. Moreover, a direct signaling policy is said to be persuasive if it incentivizes the receivers to follow recommendations. Formally, ϕ : X × Ω → ∆( A ) is persuasive if for every state x ∈ X and recommendation a ∈ A ,

Intuitively, the inequality above states that a receiver acting at state x is better off following sender's recommendation to play action a , since by doing so they get an (unnormalized) expected reward greater than or equal to what they would obtain by playing a best-response action b ϕ ( a, x ) .

Algorithm 1 shows the interaction between sender and receivers at t ∈ [ T ] . Sender and receivers do not know anything about the transition function P , the prior function µ , and the rewards r S,t ( x, ω, a ) , r R,t ( x, ω, a ) (including their distributions). At the end of each episode, the sender gets to know the triplets ( x k , ω k , a k ) -for all k ∈ K -that are visited during the episode, and an additional feedback about rewards. In this work, we consider two types of feedback. The first one-called full feedback-encompasses all agents' rewards for the pairs ( x k , ω k ) visited during the episode, i.e. , the rewards for all the triplets ( x k , ω k , a ) for a ∈ A . The second

type-called partial feedback-only consists in agents' rewards for the visited triplets ( x k , ω k , a k ) . 5

Notice that Algorithm 1 assumes that receivers always play recommended actions. This is standard in settings where the sender has not enough information to be persuasive at every episode, and

it motivates why learning algorithms are designed to guarantee that the cumulative violation of persuasiveness grows sublinearly in T , or, equivalently, that the per-round violation of persuasiveness goes to zero as T grows [Bernasconi et al., 2022, Cacciamani et al., 2023, Gan et al., 2023]. Indeed, this ensures that it is in the receivers' best interest to stick to recommendations.

3 The Learning Problem

In this section, we formally introduce the learning problem tackled in the rest of the paper. First, in Section 3.1, we extend the notion of occupancy measure to MPPs. In Section 3.2, we formally introduce learning objectives. Finally, in Section 3.3, we provide some preliminary elements needed by our algorithms, developed in Sections 4 and 5. The proofs of all our results are in Appendixes D and E.

3.1 Occupancy Measures in MPPs

Next, we extend the well-known notion of occupancy measure of an MDP [Rosenberg and Mansour, 2019] to MPPs. Given a transition function P , a signaling policy ϕ , and a prior function µ , the occupancy measure induced by P , ϕ , and µ is a vector q P,ϕ,µ ∈ [0 , 1] | X × Ω × A × X | whose entries are specified as follows. For every x ∈ X k , ω ∈ Ω , a ∈ A , and x ′ ∈ X k +1 with k ∈ K , it holds:

which is the probability that the next state is x ′ after the receiver plays action a in state x when the realized outcome is ω , under transition function P , signaling policy ϕ , and prior µ . Moreover, we also let q P,ϕ,µ ( x, ω, a ) := ∑ x ′ ∈ X k +1 q P,ϕ,µ ( x, ω, a, x ′ ) , q P,ϕ,µ ( x, ω ) := ∑ a ∈ A q P,ϕ,µ ( x, ω, a ) , and q P,ϕ,µ ( x ) := ∑ ω ∈ Ω q P,ϕ,µ ( x, ω ) , for the ease of notation.

The following lemma characterizes the set of valid occupancy measures and it is a generalization to the MPP setting of a similar lemma by Rosenberg and Mansour [2019].

Lemma 1. A vector q ∈ [0 , 1] | X × Ω × A × X | is a valid occupancy measure of an MPP if and only if

where P is the transition function of the MPP and µ its prior function, while P q and µ q are the transition and prior functions, respectively, induced by q (see definitions below).

As it is the case in standard MDPs, a valid occupancy measure q ∈ [0 , 1] | X × Ω × A × X | induces a transition function P q and a signaling policy ϕ q . Moreover, in an MPP, a valid occupancy measure also induces a prior function µ q . These are defined as follows:

Thus, using valid occupancy measures is equivalent to using signaling policies. In the following, we denote by Q ⊆ [0 , 1] | X × Ω × A × X | the set of all the valid occupancy measures of an MPP.

3.2 Learning Objectives

Our goal is to design learning algorithms for the sender in an episodic MPP. We would like algorithms that prescribe sequences of signaling policies ϕ t that maximize sender's cumulative reward over the T episodes, while at the same time guaranteeing that the violation of persuasiveness constraints is bounded. Notice that, differently from Wu et al. [2022], we do not aim at designing learning algorithms whose policies ϕ t are persuasive at every episode t with high probability, since this is unattainable in our setting in which the sender does not know anything about the environment (see Theorem 6). Thus, in this paper we pursue a different objective, formally described in the following.

Baseline First, we introduce the baseline used to evaluate sender's performances. This is defined as the value of the optimization problem faced by the sender in the offline version of the MPP. Such a problem is concerned with expectations of the stochastic quantities in the episodic MPP. By exploiting occupancy measures, the problem can be formulated as the following linear program:

̸

Intuitively, Problem (1) computes an occupancy measure (or, equivalently, signaling policy) maximizing sender's expected reward subject to persuasiveness constraints. By letting r S ∈ [0 , 1] | X × Ω × A | be the vector whose entries are the mean values r S ( x, ω, a ) of sender's rewards, our baseline is defined as OPT := r ⊤ S q ∗ , where q ∗ ∈ Q denotes an optimal solution to Problem (1). In the following, we denote by ϕ ∗ an optimal signaling policy, defined as ϕ ∗ := ϕ q ∗ .

Metrics We evaluate the performances of learning algorithms by means of two distinct metrics. The first one is the (cumulative) regret R T , which accounts for the difference between the cumulative sender's expected reward obtained by always playing ϕ ∗ and that achieved by using the signaling policies ϕ t prescribed by the algorithm. Formally,

where we let q t := q P,ϕ t ,µ be the occupancy measure induced by ϕ t . The second metric used to evaluate learning algorithms is the (cumulative) violation V T , which is formally defined as follows:

Intuitively, V T encodes the overall expected loss in persuasiveness over the T episodes. Our goal is to develop learning algorithms that prescribe signaling policies ϕ t guaranteeing that both R T and V T grow sublinearly in T , namely R T = o ( T ) and V T = o ( T ) .

3.3 Estimators and Confidence Bounds

Before delving in algorithm design, we introduce estimators and confidence bounds for the stochastic quantities involved in an MPP, namely, transitions, priors, sender's rewards, and receivers' ones. In the following, we let N t ( x, ω, a, x ′ ) ∈ N be the number of episodes up to t ∈ [ T ] (this excluded) in which the tuple ( x, ω, a, x ′ ) is visited. Formally, N t ( x, ω, a, x ′ ) := ∑ τ ∈ [ t -1] ✶ τ { x, ω, a, x ′ } , where the indicator function is 1 if and only if the tuple is visited at τ . Similarly, we define the counters N t ( x, ω, a, ) , N t ( x, ω ) , and N t ( x ) in terms of their respective indicators ✶ τ { x, ω, a } , ✶ τ { x, ω } , and τ { x } , which are equal to 1 if and only if ( x, ω, a ) , ( x, ω ) , and x , respectively, are visited at τ .

Appendix B formally defines the confidence bounds employed by our algorithms. For the transition function P , at each t ∈ [ T ] , for every x ∈ X , ω ∈ Ω , and a ∈ A , we provide a confidence bound ϵ t ( x, ω, a ) for the probability distribution over next states associated with the triplet ( x, ω, a ) , where the distance between distributions is expressed in ∥ · ∥ 1 -norm (see Lemma 4). Similarly, we provide a confidence bound ζ t ( x ) in terms of ∥ · ∥ 1 -norm for the prior µ ( x ) at each state x ∈ X (see Lemma 5). Moreover, for every x ∈ X , ω ∈ Ω , and a ∈ A , we provide confidence bounds ξ S,t ( x, ω, a ) and ξ R,t ( x, ω, a ) for sender's and receivers' rewards, respectively, associated with the triplet ( x, ω, a )

✶ Next, we define the estimators employed by our algorithms. At the beginning of each episode t ∈ [ T ] , the estimated probability of going from x ∈ X to x ′ ∈ X by playing a ∈ A , when the outcome realized in state x is ω ∈ Ω , is equal to P t ( x ′ | x, ω, a ) := N t ( x,ω,a,x ′ ) max { 1 ,N t ( x,ω,a ) } . Moreover, for every x ∈ X and ω ∈ Ω , the estimated probability of sampling outcome ω from the prior probability distribution at state x is defined as µ t ( ω | x ) := N t ( x,ω ) max { 1 ,N t ( x ) } . Finally, for every state x ∈ X , outcome ω ∈ Ω , and action a ∈ A , the estimated sender's and receivers' rewards are defined as r S,t ( x, ω, a ) := ∑ τ ∈ [ t -1] r S,τ ( x,ω,a ) ✶ τ { x,ω,a } max { 1 ,N t ( x,ω,a ) } , and r R,t ( x, ω, a ) := ∑ τ ∈ [ t -1] r R,τ ( x,ω,a ) ✶ τ { x,ω,a } max { 1 ,N t ( x,ω,a ) } .

(see Lemmas 6 and 7 for the full feedback case, while Lemmas 8 and 9 for the partial feedback one). In conclusion, for ease of presentation, for a confidence parameter δ ∈ (0 , 1) , we refer to the event in which all the confidence bounds hold-called clean event -as E ( δ ) . By combining all the lemmas in Appendix B, E ( δ ) holds with probability at least 1 -4 δ (by applying a union bound).

4 The Full Feedback Case

We first address settings with full feedback, as a warm-up towards the analysis of partial feedback.

4.1 The OPPS Algorithm With Full Feedback

We propose an algorithm called Optimisitc Persuasive Policy Search ( OPPS ). At each episode, the algorithm solves a variation of the offline optimization problem (Problem (1)), called Opt-Opt , obtained by substituting mean values with upper/lower confidence bounds. Specifically, Opt-Opt is optimistic with respect to both sender's rewards and persuasiveness constraints satisfaction. For reasons of space, we defer Opt-Opt to Problem (2) in Appendix C. Crucially, by using occupancy measures, Opt-Opt can be formulated as an LP, and, thus, solved efficiently. Notice that, since confidence bounds for P and µ are expressed in terms of ||·|| 1 -norm, in order to formulate Opt-Opt as an LP we need some additional variables and linear constraints, as described in detail in Appendix C.

Algorithm 2 provides the pseudocode of OPPS with full feedback. At each t ∈ [ T ] , the algorithm first updates all the estimators and confidence bounds according to the feedback received in previous episodes (Line 3). Then, it commits to the signaling policy ϕ t induced by an optimal solution ̂ q t to Opt-Opt , computed in Line 4. Notice that, the occupancy measure q t resulting from committing to ϕ t (and used in the definitions of R T and V T ) is in general different from ̂ q t , as the former is defined in

Algorithm 2 Optimistic Persuasive Policy Search (full)

Require: X , A , T , confidence parameter δ ∈ (0 , 1)

1: Initialize all estimators to 0 and all bounds to + ∞

2: for t = 1 , . . . , T do

3: Update all estimators P t , µ t , r S,t , r R,t and bounds ϵ t , ζ t , ξ S,t , ξ R,t given new observations

4: ̂ q t ← Solve Opt-Opt (Problem (2))

5: ϕ t ← ϕ ̂ q t

6: Run Algorithm 1 by committing to ϕ t

7: Observe full feedback from Algorithm 1

8: end for

terms of the true (and unknown) transition and prior functions, namely P and µ .

4.2 Algorithm Analysis With Full Feedback

Next, we prove the guarantees of OPPS with full feedback. The first crucial component is the following lemma, which shows that Opt-Opt admits a feasible solution at every episode with high probability.

Lemma 2. Given δ ∈ (0 , 1) , under event E ( δ ) , Opt-Opt admits a feasible solution at every t ∈ [ T ] .

Intuitively, Lemma 2 is proved by showing (a) that Problem 1 always admits a feasible solution, which is the occupancy measure q ⋄ induced by the signaling policy that fully reveals outcomes to the receiver, and (b) that q ⋄ is a feasible solution to Opt-Opt at every episode, under E ( δ ) . Point (b) holds thanks to the fact that Opt-Opt optimistically accounts for persuasiveness constraints satisfaction, by using suitable upper and lower confidence bounds.

The second crucial component of our analysis is a relation between the occupancy measures ̂ q t computed by the OPPS algorithm and the occupancy measures q t that actually result from committing to ϕ t under the true transitions and priors. This is formally stated by the following lemma.

Lemma 3. Given any δ ∈ (0 , 1) , under the clean event E ( δ ) , with probability at least 1 -2 δ , it holds that ∑ t ∈ [ T ] ∥ q t -̂ q t ∥ 1 ≤ O ( L 2 | X | √ T | A || Ω | ln ( T | X || Ω || A | / δ ) ) .

Intuitively, Lemma 3 is proved by an inductive argument that relates the uncertainty associated with both the transition and the prior functions to the ∥ · ∥ 1 -norm difference between q t and ̂ q t cumulated over the episodes. Lemmas 2 and 3 pave the way to our two main theorems for the full feedback setting. The first theorem bounds the regret R T achieved by OPPS , while the second one bounds its cumulative violation V T . Formally:

Theorem 1. Given any δ ∈ (0 , 1) , with probability at least 1 -7 δ , Algorithm 2 attains regret R T ≤ ˜ O ( L 2 | X | √ T | A || Ω | ln ( 1 / δ ) ) .

Theorem 2. Given δ ∈ (0 , 1) , with probability at least 1 -7 δ , Algorithm 2 attains violation V T ≤ ˜ O ( L 2 | X | √ T | A || Ω | ln ( 1 / δ ) ) .

In conclusion, in the full feedback case, OPPS attains R T and V T growing as ˜ O ( √ T ) . Intuitively, this is made effective by the fact that all the estimators concentrate at a 1 / √ T rate. As shown next, achieving such regret and violation bounds is not possible anymore under partial feedback.

4.3 Lower Bound

We conclude the section by showing that the regret and violation bounds attained by the OPPS algorithm with full feedback are tight. Formally:

Theorem 3. Let δ ∈ (0 , 1 / 4 ) . For every algorithm that guarantees R T ≤ O ( √ T ) with probability at least 1 -δ , there exists an absolute constant ψ > 0 and a problem instance in which the algorithm must have V T ≥ Ω( √ T ) with probability larger than ψ.

5 The Partial Feedback Case

In this section, we switch the attention to partial feedback. The crucial aspect that makes the case of partial feedback more challenging than the one of full feedback is that, after committing to a signaling policy ϕ t , the sender does not observe sufficient feedback about the persuasiveness of ϕ t . This makes achieving sublinear violation in the partial feedback case much harder than in the full feedback case. In order to overcome such a challenge, some episodes of learning must be devoted to the estimation of the quantities involved in persuasiveness constraints. This is necessary to build a suitable approximation of such constraints to be exploited in the remaining episodes, in which an optimistic approach similar to that employed with full feedback

Algorithm 3 Optimistic Persuasive Policy Search (partial)

must be adopted to control the regret. As a result, there is a trade-off between regret and violation that is determined by the amount of exploration performed. In the rest of this section, we design an algorithm that is able to optimally control such a trade-off.

5.1 The OPPS Algorithm With Partial Feedback

We extend the OPPS algorithm introduced in Section 4 to deal with the partial feedback case. The idea behind the new algorithm is to split episodes into two phases. The first one is an exploration phase with the goal of building a sufficiently-good approximation of persuasiveness constraints, so as to achieve sublinear violation. Such a phase lasts for the first N | X || Ω || A | episodes, where we let N := ⌈ T α ⌉ with α ∈ [0 , 1] being a parameter controlling the length of the two phases, given as input to the algorithm. The second phase is instead devoted to achieving sublinear regret, and it follows the same steps of OPPS with full feedback (Algorithm 2). The first phase works by considering each ( x, ω, a ) ∈ X × Ω × A for N episodes. When ( x, ω, a ) is considered at episode t ∈ [ T ] , the algorithm commits to a signaling scheme induced by an occupancy measure ̂ q t that maximizes the probability ∑ x ′ ∈ X q ( x, ω, a, x ′ ) of visiting such a triplet, while at the same time satisfying all the constraints of the Opt-Opt problem. Crucially, such a procedure does not guarantee that every triplet

is visited N times. Indeed, there might be triplets ( x, ω, a ) that are visited with very low probability. This can be the case when either transitions and priors place very low probability on ( x, ω ) or action a is associated with very low receivers' rewards, and, thus, it must be recommended with very low probability to satisfy the optimistic persuasiveness constraints defined in Opt-Opt .

Algorithm 3 provides the pseudocode of OPPS with partial feedback. Notice that the variables C ( x, ω, a ) (initialized in Line 3 and updated in Line 9) are counters used to keep track of how many times each triplet ( x, ω, a ) is considered during the first phase, namely when t ≤ N | X || Ω || A | . Moreover, the algorithm ensures that every triplet is considered exactly N times during the first phase, by selecting them accordingly as in Line 7. Let us also observe that Algorithm 3 updates all the estimators and bounds (by using partial feedback) and selects the signaling policy ϕ t as done by Algorithm 2. The main difference with respect to Algorithm 2 is that ̂ q t used to define ϕ t is computed in a different way during the first (exploration) phase (see Line 8).

5.2 Algorithm Analysis With Partial Feedback

In the following, we prove the guarantees attained by OPPS with partial feedback. We start by stating the following result on the regret attained by the algorithm.

Theorem 4. Given any δ ∈ (0 , 1) , with probability at least 1 -7 δ , Algorithm 3 attains regret R T ≤ ˜ O ( NL | X || Ω || A | + L 2 | X | √ T | A || Ω | ln ( 1 / δ ) ) .

In order to prove Theorem 4, we split the analysis into two cases: one targets exploration episodes in the first phase of the algorithm, while the other is concerned with the subsequent (exploitation) phase. In the first N episodes in which the OPPS algorithm explores without being driven by the Opt-Opt objective, the algorithm incurs in linear regret. Instead, in the second phase, OPPS employs an optimistic approach, since the algorithm is driven by the objective of the Opt-Opt problem. Thus, in the second phase, the algorithm attains regret sublinear in T . The two cases combined give the regret bound provided in Theorem 4. Next, we state the result on the violations attained by OPPS .

Theorem 5. Given any δ ∈ (0 , 1) , with probability at least 1 -9 δ , Algorithm 3 attains cumulative violation V T ≤ ˜ O ( ρ ( | A | T √ N + | A | √ N + L 2 √ T )) , where ρ := ( | X || Ω || A | ) 3 / 2 √ ln ( 1 / δ ) .

Proving Theorem 5 requires a non-trivial analysis. The result follows by showing that uniformly exploring over feasible solutions to the Opt-Opt problem leads to a violation bound of the order of O ( √ N ) during the exploration phase. Intuitively, this follows by upper bounding the occupancy measure in each triplet ( x, ω, a ) with an occupancy of a previous (exploration) episode, relative to the best response of the follower in state x upon receiving action recommendation a . Theorems 4 and 5 establish the trade-off between regret and violation achieved by the OPPS algorithm. Indeed, by recalling the definition of N (see Line 1 in Algorithm 3), it is easy to see that the algorithm attains regret R T ≤ ˜ O ( T α ) and violation V T ≤ ˜ O ( T 1 -α / 2 ) , where α ∈ [ 1 / 2 , 1] is the parameter controlling the trade-off, given as input to the algorithm.

5.3 Lower Bound

We conclude the section and the paper by showing that the regret and violation bounds attained by the OPPS algorithm (see Theorems 4 and 5) are tight for any choice of α ∈ [ 1 / 2 , 1] . We do so by devising a lower bound matching these bounds (Theorem 6). Its main idea is to consider two instances of

1

/

2

2

/

3

1

1

2

2

3

3

4

1

Order of

T

in

R

T

Order of

T

in

V

T

Lower Bound

Algorithm 3

Bern. et al. [2022]

episodic MPP involving a receiver with two actions a 1 , a 2 such that only a 1 provides positive reward to the sender. In one instance, receiver's rewards by playing a 1 are higher than those obtained by taking a 2 , while in the second instance the opposite holds. As a result, recommending action a 1 results in low regret in the first instance and high violation in the second one, while recommending action a 2 results in low violation in the second instance and high regret in the first one. This leads to the trade-off formally stated by the following theorem.

Theorem 6. Given α ∈ [1 / 2 , 1] , there is no learning algorithm achieving both R T = o ( T α ) and V T = o ( T 1 -α / 2 ) with probability greater or equal to a fixed constant ψ > 0 .

Theorem 6 shows that the bounds in Theorems 4 and 5 are tight for any α ∈ [ 1 / 2 , 1] . See Figure 1 for a graphical depiction. Theorem 6 also proves that it is impossible to achieve sublinear regret while being persuasive at every episode with high probability, when the sender has no information about the receivers. Notice that, in our MPP setting with partial feedback, we deal with a trade-off between regret and violation that is similar to the one faced by Bernasconi et al. [2022] in related settings. Differently from them, we are able to achieve an optimal trade-off for any α ∈ [ 1 / 2 , 1] . Indeed, Bernasconi et al. [2022] only obtain optimality for α ∈ [ 1 / 2 , 2 / 3 ] , leaving as an open problem matching the lower bound for the other values of the parameter α (see Figure 1). Crucially, we are able to achieve trade-off optimality by using a clever exploration method. Indeed, when considering a triplet ( x, ω, a ) in the first phase, the OPPS algorithm does not simply commit to a signaling policy that maximizes the probability of visiting such a triplet, but it rather does so while also optimistically accounting for persuasiveness constraints. This allows to reduce the violation cumulated during the first phase, thus achieving trade-off optimality.

Acknowledgments

This paper is supported by the Italian MIUR PRIN 2022 Project 'Targeted Learning Dynamics: Computing Efficient and Fair Equilibria through No-Regret Algorithms', by the FAIR (Future Artificial Intelligence Research) project, funded by the NextGenerationEU program within the PNRRPE-AI scheme (M4C2, Investment 1.3, Line on Artificial Intelligence), and by the EU Horizon project ELIAS (European Lighthouse of AI for Sustainability, No. 101120237).

References

Ricardo Alonso and Odilon Cˆ amara. Persuading voters. American Economic Review , 2016.

Itai Arieli and Yakov Babichenko. Private bayesian persuasion. Journal of Economic Theory , 182: 185-217, 2019.

Peter Auer, Thomas Jaksch, and Ronald Ortner. Near-optimal regret bounds for reinforcement learning. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou, editors, Advances in Neural Information Processing Systems , volume 21. Curran Associates, Inc., 2008.

Yakov Babichenko and Siddharth Barman. Algorithmic aspects of private Bayesian persuasion. In ITCS , 2017.

Francesco Bacchiocchi, Matteo Castiglioni, Alberto Marchesi, Giulia Romano, and Nicola Gatti. Public signaling in bayesian ad auctions. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022 , pages 39-45. ijcai.org, 2022.

Ashwinkumar Badanidiyuru, Kshipra Bhawalkar, and Haifeng Xu. Targeting and signaling in ad auctions. In Proceedings of the Twenty-Ninth Annual ACM-SIAM Symposium on Discrete Algorithms , pages 2545-2563, 2018.

Martino Bernasconi, Matteo Castiglioni, Alberto Marchesi, Nicola Gatti, and Francesco Trov` o. Sequential information design: Learning to persuade in the dark. Advances in Neural Information Processing Systems , 35:15917-15928, 2022.

Martino Bernasconi, Matteo Castiglioni, Andrea Celli, Alberto Marchesi, Francesco Trov` o, and Nicola Gatti. Optimal rates and efficient algorithms for online Bayesian persuasion. In Proceedings

of the 40th International Conference on Machine Learning , volume 202, pages 2164-2183. PMLR, 2023a.

Martino Bernasconi, Matteo Castiglioni, Alberto Marchesi, and Mirco Mutti. Persuading farsighted receivers in MDPs: the power of honesty. In Advances in Neural Information Processing Systems , volume 36, pages 1-13, 2023b.

Umang Bhaskar, Yu Cheng, Young Kun Ko, and Chaitanya Swamy. Hardness results for signaling in bayesian zero-sum and network routing games. In EC , 2016.

Peter Bro Miltersen and Or Sheffet. Send mixed signals: earn more, work less. In EC , 2012.

Federico Cacciamani, Matteo Castiglioni, and Nicola Gatti. Online mechanism design for information acquisition. In International Conference on Machine Learning , pages 3299-3326. PMLR, 2023.

Ozan Candogan. Persuasion in networks: Public signals and k-cores. In EC , 2019.

Matteo Castiglioni and Nicola Gatti. Persuading voters in district-based elections. In AAAI , 2021.

Matteo Castiglioni, Andrea Celli, and Nicola Gatti. Persuading voters: It's easy to whisper, it's hard to speak loud. In AAAI , 2020a.

Matteo Castiglioni, Andrea Celli, Alberto Marchesi, and Nicola Gatti. Online Bayesian persuasion. In NeurIPS , pages 16188-16198, 2020b.

Matteo Castiglioni, Andrea Celli, Alberto Marchesi, and Nicola Gatti. Signaling in Bayesian network congestion games: the subtle power of symmetry. In AAAI , 2021a.

Matteo Castiglioni, Alberto Marchesi, Andrea Celli, and Nicola Gatti. Multi-receiver online Bayesian persuasion. In ICML , pages 1314-1323, 2021b.

Matteo Castiglioni, Giulia Romano, Alberto Marchesi, and Nicola Gatti. Signaling in posted price auctions. Proceedings of the AAAI Conference on Artificial Intelligence , 36(5):4941-4948, Jun. 2022.

Nicolo Cesa-Bianchi and G´ abor Lugosi. Prediction, learning, and games . Cambridge university press, 2006.

Yu Cheng, Ho Yee Cheung, Shaddin Dughmi, Ehsan Emamjomeh-Zadeh, Li Han, and Shang-Hua Teng. Mixture selection, mechanism design, and signaling. In 56th Annual Symposium on Foundations of Computer Science , pages 1426-1445, 2015.

Shaddin Dughmi and Haifeng Xu. Algorithmic bayesian persuasion. In Proceedings of the fortyeighth annual ACM symposium on Theory of Computing , pages 412-425, 2016.

Yonathan Efroni, Shie Mannor, and Matteo Pirotta. Exploration-exploitation in constrained mdps, 2020.

Yuval Emek, Michal Feldman, Iftah Gamzu, Renato PaesLeme, and Moshe Tennenholtz. Signaling schemes for revenue maximization. ACM Transactions on Economics and Computation , 2(2):1-19, 2014.

Eyal Even-Dar, Sham M Kakade, and Yishay Mansour. Online markov decision processes. Mathematics of Operations Research , 34(3):726-736, 2009.

Jiarui Gan, Rupak Majumdar, Goran Radanovic, and Adish Singla. Bayesian persuasion in sequential decision-making. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 5025-5033, 2022.

Jiarui Gan, Rupak Majumdar, Debmalya Mandal, and Goran Radanovic. Sequential principal-agent problems with communication: Efficient computation and learning, 2023.

Itay Goldstein and Yaron Leitner. Stress tests and information disclosure. Journal of Economic Theory , 177:34-69, 2018.

Krishnamurthy Iyer, Haifeng Xu, and You Zu. Markov persuasion processes with endogenous agent beliefs. arXiv preprint arXiv:2307.03181 , 2023.

Chi Jin, Tiancheng Jin, Haipeng Luo, Suvrit Sra, and Tiancheng Yu. Learning adversarial Markov decision processes with bandit feedback and unknown transition. In Hal Daum´ e III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 4860-4869. PMLR, 13-18 Jul 2020.

Kamenica and M. Gentzkow. Bayesian persuasion. AM ECON REV , 101(6):2590-2615, 2011.

Anton Kolotilin. Experimental design to persuade. Games and Economic Behavior , 90:215-226, 2015.

Yue Lin, Wenhao Li, Hongyuan Zha, and Baoxiang Wang. Information design in multi-agent reinforcement learning. Advances in Neural Information Processing Systems , 36, 2024.

Yishay Mansour, Aleksandrs Slivkins, Vasilis Syrgkanis, and Zhiwei Steven Wu. Bayesian exploration: Incentivizing exploration in Bayesian games. In EC , 2016.

Gergely Neu, Andras Antos, Andr´ as Gy¨ orgy, and Csaba Szepesv´ ari. Online markov decision processes under bandit feedback. Advances in Neural Information Processing Systems , 23, 2010.

Francesco Orabona. A modern introduction to online learning. CoRR , abs/1912.13213, 2019.

Shuang Qiu, Xiaohan Wei, Zhuoran Yang, Jieping Ye, and Zhaoran Wang. Upper confidence primal-dual reinforcement learning for cmdp with adversarial loss. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 15277-15287. Curran Associates, Inc., 2020.

Zinovi Rabinovich, Albert Xin Jiang, Manish Jain, and Haifeng Xu. Information disclosure as a means to security. In AAMAS , 2015.

Aviv Rosenberg and Yishay Mansour. Online convex optimization in adversarial Markov decision processes. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 5478-5486. PMLR, 09-15 Jun 2019.

Francesco Emanuele Stradi, Jacopo Germano, Gianmarco Genalti, Matteo Castiglioni, Alberto Marchesi, and Nicola Gatti. Online learning in CMDPs: Handling stochastic and adversarial constraints. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning , volume 235 of Proceedings of Machine Learning Research , pages 4669246721. PMLR, 21-27 Jul 2024.

Shoshana Vasserman, Michal Feldman, and Avinatan Hassidim. Implementing the wisdom of waze. In Twenty-Fourth International Joint Conference on Artificial Intelligence , pages 660-666, 2015.

Xiaohan Wei, Hao Yu, and Michael J. Neely. Online learning in weakly coupled markov decision processes: A convergence time study. Proc. ACM Meas. Anal. Comput. Syst. , 2(1), apr 2018. doi: 10.1145/3179415.

Jibang Wu, Zixuan Zhang, Zhe Feng, Zhaoran Wang, Zhuoran Yang, Michael I. Jordan, and Haifeng Xu. Sequential information design: Markov persuasion process and its efficient reinforcement learning. In Proceedings of the 23rd ACM Conference on Economics and Computation , EC '22, page 471-472, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450391504. doi: 10.1145/3490486.3538313.

Haifeng Xu, Rupert Freeman, Vincent Conitzer, Shaddin Dughmi, and Milind Tambe. Signaling in Bayesian Stackelberg games. In AAMAS , 2016.

Liyuan Zheng and Lillian Ratliff. Constrained upper confidence reinforcement learning. In Alexandre M. Bayen, Ali Jadbabaie, George Pappas, Pablo A. Parrilo, Benjamin Recht, Claire Tomlin, and Melanie Zeilinger, editors, Proceedings of the 2nd Conference on Learning for Dynamics and Control , volume 120 of Proceedings of Machine Learning Research , pages 620-629. PMLR, 10-11 Jun 2020.

You Zu, Krishnamurthy Iyer, and Haifeng Xu. Learning to persuade on the fly: Robustness against ignorance. In EC , pages 927-928, 2021.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction state all the main contributions made by the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: All the assumptions are clearly stated in Section 2.1 and Section 2.2.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All the theoretical results clearly state their assumptions, while all their proofs are provided in the Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The paper conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no societal impact of the work performed, since the work is mainly theoretical.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: The paper does not involve LLMs as any important, original, or non-standard components.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Appendix

The appendix is organized as follows:

In Appendix A we report the related works concerning the online learning in Markov decision processes and online Bayesian persuasion literatures.

In Appendix B we describe the estimators and the confidence bounds related to the stochastic quantities of the Markov persuasive processes.

In Appendix C we report the per-round optimization problem performed by the algorithms we present.

In Appendix D we report the omitted proofs related to the full-feedback setting.

In Appendix E we report the omitted proofs related to the partial-feedback setting.

A Related works

Sequential Bayesian persuasion The work that is most related to ours is [Wu et al., 2022], which introduces MPPs. Specifically, Wu et al. [2022] study settings where the sender knows everything about receivers' rewards, with the only elements unknown to them being their rewards, transition probabilities, and prior distributions over outcomes. Moreover, they also assume that the receivers know everything they need about the environment, so as to select a best-response action, and that all rewards are deterministic. In contrast, we consider MPP settings in which sender and receivers have no knowledge of the environment, including their rewards, which we assume to be stochastic. Moreover, Wu et al. [2022] obtain a regret bound of the order of ˜ O ( √ T/D ) , where D is a parameter related to receivers' rewards. Notice that such a dependence is particularly unpleasant, as D may be exponentially large in instances in which there are some receivers' actions that are best responses only for a 'small' space of information-disclosure policies. Other works related to ours are [Gan et al., 2022], which studies a Bayesian persuasion problem where a sender sequentially interacts with a myopic receiver in a multi-state environment, and [Bernasconi et al., 2023b], which addresses MPPs with a farsighted receiver. These two works considerably depart from ours, as they both assume that the sender knows everything about the environment, including transitions, priors, and rewards. Thus, they are not concerned with learning problems, but with the problem of computing optimal information-disclosure policies. Finally, [Bernasconi et al., 2022] studies settings where a sender faces a farsighted receiver in a sequential environment with a tree structure, addressing the case in which the only elements unknown to the sender are the prior distributions over outcomes, while rewards are deterministic and known. The tree structure considerably eases the learning task, as it allows to express sender's expected rewards linearly in variables defining information-disclosure policies. Intuitively, this allows to factor the uncertainty about transitions in the rewards at the leaves of the tree.

Online Bayesian persuasion It is also worth citing some works that study learning problems in which a one-shot Bayesian persuasion setting is played repeatedly [Castiglioni et al., 2020b, 2021b, Zu et al., 2021, Bernasconi et al., 2023a]. These works considerably depart from ours, since they do not consider any kind of sequential structure in the sender-receiver interaction at each episode.

Online learning in constrained MDPs Our paper is also related to the problem of designing no-regret algorithms in online constrained Markov decision processes. The literature on online learning in Markov decision processes is extensive (see, e.g. , Auer et al. [2008], Even-Dar et al. [2009], Neu et al. [2010] for fundamental works on the topic). In such settings, two types of feedback are usually investigated. The full-information feedback setting [Rosenberg and Mansour, 2019], in which the entire reward function is observed after the learner's choice and the partial feedback setting [Jin et al., 2020], where the learner only observes the reward gained during the episode. Over the last decade, there has been significant attention to the field of online Markov decision processes in presence of constraints. The majority of previous works on this topic have focused on settings where constraints are stochastically sampled from a fixed distribution (see, e.g. , Zheng and Ratliff [2020]). Wei et al. [2018] deal with adversarial reward and stochastic constraints, assuming known transition probabilities and full information feedback. Efroni et al. [2020] propose two approaches to address the exploration-exploitation dilemma in episodic constrained MDPs. These approaches guarantee

sublinear regret and constraint violation when transition probabilities, rewards, and constraints are unknown and stochastic, and the feedback is partial. Qiu et al. [2020] provide a primal-dual approach based on optimism in the face of uncertainty . This work shows the effectiveness of such an approach when dealing with episodic constrained MDPs with adversarial rewards and stochastic constraints, achieving both sublinear regret and constraint violation with full-information feedback. Finally, Stradi et al. [2024] propose a best-of-both-worlds algorithm in constrained Markov decision processes with full information feedback. While the previous works are related to ours, the aforementioned techniques cannot be easily generalized to our setting as they are not designed to properly handle the presence of outcomes and IC constraints.

B Confidence bounds

In this section, we further describe the estimators and confidence bounds for the stochastic quantities involved in an episodic MPP, namely, transitions, priors, sender's rewards, and receivers' ones.

B.1 Transition probabilities

First, we introduce confidence bounds for transition probabilities P ( x ′ | x, ω, a ) , by generalizing those introduced by Rosenberg and Mansour [2019] for MDPs to MPPs. In the following, we let N t ( x, ω, a ) , respectively N t ( x, ω, a, x ′ ) , be the counter specifying the number of episodes up to episode t ∈ [ T ] (excluded) in which the triplet ( x, ω, a ) , respectively the tuple ( x, ω, a, x ′ ) , is visited. Then, the estimated probability of going from x ∈ X to x ′ ∈ X by playing action a ∈ A , when the outcome realized in state x is ω ∈ Ω , is defined as follows:

For any δ ∈ (0 , 1) , the confidence set at episode t ∈ [ T ] for the transition function P is P t := ⋂ ( x,ω,a ) ∈ X × Ω × A P x,ω,a t , where P x,ω,a t is a set of transition functions defined as:

where ̂ P ( ·| x, ω, a ) and P t ( ·| x, ω, a ) are vectors whose entries are the values ̂ P ( x ′ | x, ω, a ) and P t ( x ′ | x, ω, a ) , respectively, while ϵ t ( x, ω, a ) is a confidence bound defined as:

The following lemma formally proves that P t is a suitable confidence set for the transition function of an MPP.

Lemma 4. Given any δ ∈ (0 , 1) , with probability at least 1 -δ , the following condition holds for every x ∈ X , ω ∈ Ω , a ∈ A , and t ∈ [ T ] jointly:

Lemma 4 can be easily proven by applying the same analysis as presented in [Auer et al., 2008] and employing a union bound over all x , ω , a , and t ..

B.2 Prior distributions

Next, we introduce confidence bounds for prior distributions. For every state x ∈ X , we define µ t ( ·| x ) ∈ ∆(Ω) as the estimator of the prior distribution at x built by using observations up to episode t ∈ [ T ] (this excluded). Formally, the entries of vector µ t ( ·| x ) are such that, for every ω ∈ Ω :

where N t ( x ) is the number of visits to state x up to episode t (excluded), while ✶ τ { x, ω } is an indicator function equal to 1 if and only if the pair ( x, ω ) is visited at episode τ . The following lemma provides confidence bounds for priors.

Lemma 5. Given any δ ∈ (0 , 1) , with probability at least 1 -δ , the following holds for all x ∈ X and t ∈ [ T ] jointly:

Lemma 5 follows by applying Bernstein's inequality and a union bound over all states and episodes.

B.3 Sender's and receivers' rewards

Finally, we introduce estimators for rewards. In the following, present the results related to sender's rewards and receiver's rewards under full and partial feedback. For every x ∈ X , ω ∈ Ω , and a ∈ A , the estimated sender's and receivers' rewards built with observations up to episode t ∈ [ T ] (this excluded) are defined as follows:

The following lemma provides confidence bounds for sender's rewards, when full feedback is available.

where ✶ τ { x, ω, a } is an indicator function equal to 1 if and only if the triplet ( x, ω, a ) is visited during episode τ .

Lemma 6. Given any δ ∈ (0 , 1) , with probability at least 1 -δ , the following condition holds for every x ∈ X , ω ∈ Ω , a ∈ A , and t ∈ [ T ] jointly:

Lemma 6 follows by applying Hoeffding's inequality and a union bound over all x , ω and t .

The following lemma provides confidence bounds for receiver's rewards, when full feedback is available.

Lemma 7. Given any δ ∈ (0 , 1) , with probability at least 1 -δ , the following condition holds for every x ∈ X , ω ∈ Ω , a ∈ A , and t ∈ [ T ] jointly:

Lemma 7 follows by applying Hoeffding's inequality and a union bound over all x , ω and t .

The following lemma provides confidence bounds for sender's rewards, when only partial feedback is available.

Lemma 8. Given any δ ∈ (0 , 1) , with probability at least 1 -δ , the following condition holds for every x ∈ X , ω ∈ Ω , a ∈ A , and t ∈ [ T ] jointly:

Lemma 8 follows by applying Hoeffding's inequality and a union bound over all x , ω , a , and t .

Finally, the following lemma provides confidence bounds for receiver's rewards, when only partial feedback is available.

Lemma 9. Given any δ ∈ (0 , 1) , with probability at least 1 -δ , the following condition holds for every x ∈ X , ω ∈ Ω , a ∈ A , and t ∈ [ T ] jointly:

Lemma 9 follows by applying Hoeffding's inequality and a union bound over all x , ω , a , and t .

C Optimistic optimization problem

In the following section we describe the linear program solved by Algorithm 2 and Algorithm 3, namely Opt-Opt . Intuitively, Opt-Opt is the optimistic version of Program (1), since the objective is guided by the optimism and the confidence bounds of the estimated parameters are chosen to make constraints easier to be satisfied. Notice that the confidence bounds on the transitions and the prior are applied to the ∥·∥ 1 differences between the empirical and the real mean of the distributions. Thus, in order to insert the aforementioned confidence bounds in a LP-formulation, the related constraints must be linearized by means of additional optimization variables.

The linear program solved by Algorithm 2 and Algorithm 3 is the following.

where Objective (2a) maximizes the upper confidence bound of the sender reward, Constraint (2b) ensures that the occupancy measure sums to 1 for every layer, Constraint (2c) is the flow constraint, Constraint (2d) is related to the confidence interval on the transition functions, Constraint (2e) is still related to the confidence bounds on the transition function, Constraint (2f) allows to write linearly the constraints related to the transition functions even if the interval holds for the ∥ · ∥ 1 , Constraint (2g) is related to the confidence interval on the outcomes, Constraint (2h) is still related to the confidence bounds on the outcomes, Constraint (2i) allows to write linearly the constraints related to the outcomes even if the interval holds for the ∥ · ∥ 1 , Constraint (2j) is the optimistic constraint for the Incentive Compatibility (IC) property and, finally, Constraint (2k) ensures that the occupancy are greater than zero.

Lemma 2. Given δ ∈ (0 , 1) , under event E ( δ ) , Opt-Opt admits a feasible solution at every t ∈ [ T ] .

Proof. First we notice that under the clean event E ( δ ) the true transition function P and the prior µ are included in the their confidence interval; thus, they are available in the constrained space defined by Opt-Opt . Then, we focus on the incentive compatibility constraints. Referring as q ⋄ to an incentive compatible occupancy measure, under E ( δ ) , we have that:

for any k ∈ [ L -1] , ( x, a ) ∈ X k × A, ∀ ¯ a ∈ A . As a result, if q ⋄ is incentive compatible, it belongs to the optimistic decision space, which concludes the proof.

D Full feedback

In this section we report the omitted proof related to Algorithm 2. Notice that the bound on the transition function estimations still hold when the feedback is partial.

D.1 Transition functions

We start by showing that the estimated occupancy measures which encompass the information related to the outcomes and the transitions concentrate with respect to the true occupancy measures.

Lemma 3. Given any δ ∈ (0 , 1) , under the clean event E ( δ ) , with probability at least 1 -2 δ , it holds that ∑ t ∈ [ T ] ∥ q t -̂ q t ∥ 1 ≤ O ( L 2 | X | √ T | A || Ω | ln ( T | X || Ω || A | / δ ) ) .

Proof. We start noticing that, for any ( x, ω, a ) ∈ X × Ω × A , we have:

Thus, summing over t ∈ [ T ] and ( x, ω, a ) ∈ X × Ω × A we obtain:

Next, we focus on the first part of the equation, noticing that:

Bound on ∣ ∣ q P t ,ϕ t ,µ t ( x, ω, a ) -q P t ,ϕ t ,µ ( x, ω, a ) ∣ ∣ We bound this term by induction. At the first layer we have:

observing that X 0 = { x 0 } . Now we show that, if the result holds for x k -1 , it holds for x k , as follows,

∑ x k ∈ X k ∑ ω k ∈ Ω ∑ a k ∈ A ∣ ∣ q P t ,ϕ t ,µ t ( x k , ω k , a k ) -q P t ,ϕ t ,µ ( x k , ω k , a k ) ∣ ∣ = ∑ x k -1 ∈ X k -1 ∑ ω k -1 ∈ Ω ∑ a k -1 ∈ A ∑ x k ∈ X k ∑ ω k ∈ Ω ∑ a k ∈ A | q P t ,ϕ t ,µ t ( x k -1 , ω k -1 , a k -1 ) · · P t ( x k | x k -1 , ω k -1 , a k -1 ) µ t ( x k , ω k )+ -q P t ,ϕ t ,µ ( x k -1 , ω k -1 , a k -1 ) P t ( x k | x k -1 , ω k -1 , a k -1 ) µ ( x k , ω k ) | ϕ t ( a k | x k , ω k ) = ∑ x k -1 ∈ X k -1 ∑ ω k -1 ∈ Ω ∑ a k -1 ∈ A ∑ x k ∈ X k ∑ ω k ∈ Ω | q P t ,ϕ t ,µ t ( x k -1 , ω k -1 , a k -1 ) · · P t ( x k | x k -1 , ω k -1 , a k -1 ) µ t ( x k , ω k )+ -q P t ,ϕ t ,µ ( x k -1 , ω k -1 , a k -1 ) P t ( x k | x k -1 , ω k -1 , a k -1 ) µ ( x k , ω k ) | ≤ ∑ x k -1 ∈ X k -1 ∑ ω k -1 ∈ Ω ∑ a k -1 ∈ A ∑ x k ∈ X k ∑ ω k ∈ Ω | q P t ,ϕ t ,µ t ( x k -1 , ω k -1 , a k -1 ) · · P t ( x k | x k -1 , ω k -1 , a k -1 ) µ t ( x k , ω k )+ -q P t ,ϕ t ,µ ( x k -1 , ω k -1 , a k -1 ) P t ( x k | x k -1 , ω k -1 , a k -1 ) µ t ( x k , ω k ) | + + ∑ x k -1 ∈ X k -1 ∑ ω k -1 ∈ Ω ∑ a k -1 ∈ A ∑ x k ∈ X k ∑ ω k ∈ Ω | q P t ,ϕ t ,µ ( x k -1 , ω k -1 , a k -1 ) · · P t ( x k | x k -1 , ω k -1 , a k -1 ) µ t ( x k , ω k )+ -q P t ,ϕ t ,µ ( x k -1 , ω k -1 , a k -1 ) P t ( x k | x k -1 , ω k -1 , a k -1 ) µ ( x k , ω k ) | ≤ ∑ x k -1 ∈ X k -1 k ∑ ω k -1 ∈ Ω ∑ a k -1 ∈ A ∣ ∣ q P t ,ϕ t ,µ t ( x k -1 , ω k -1 , a k -1 ) -q P t ,ϕ t ,µ ( x k -1 , ω k -1 , a k -1 ) ∣ ∣ + ∑ x k ∈ X k q P t ,ϕ t ,µ ( x k ) ∑ ω k ∈ Ω | µ t ( x k , ω k ) -µ ( x k , ω k ) | . Thus, by induction hypothesis, it follows, ∑ x k ∈ X k ∑ ω k ∈ Ω ∑ a k ∈ A ∣ ∣ q P t ,ϕ t ,µ t ( x k , ω k , a k ) -q P t ,ϕ t ,µ ( x k , ω k , a k ) ∣ ∣

Bound on ∣ ∣ q P t ,ϕ t ,µ ( x, ω, a ) -q P,ϕ t ,µ ( x, ω, a ) ∣ ∣ To bound this term, we proceed again by induction. Thus, we notice that:

∑ x 1 ∈ X 1 ∑ ω 1 ∈ Ω ∑ a 1 ∈ A | q P t ,ϕ t ,µ ( x 1 , ω 1 , a 1 ) -q P,ϕ t ,µ ( x 1 , ω 1 , a 1 ) | = ∑ ω 0 ∈ Ω ∑ a 0 ∈ A ∑ x 1 ∈ X 1 ∑ ω 1 ∈ Ω ∑ a 1 ∈ A | µ ( x 0 , ω 0 ) ϕ t ( a 0 | x 0 , ω 0 ) P t ( x 1 | x 0 , ω 0 , a 0 ) µ ( x 1 , ω 1 ) ϕ t ( a 1 | x 1 , ω 1 ) -µ ( x 0 , ω 0 ) ϕ t ( a 0 | x 0 , ω 0 ) P ( x 1 | x 0 , ω 0 , a 0 ) µ ( x 1 , ω 1 ) ϕ t ( a 1 | x 1 , ω 1 ) | = ∑ ω 0 ∈ Ω ∑ a 0 ∈ A µ ( x 0 , ω 0 ) ϕ t ( a 0 | x 0 , ω 0 ) ∑ x 1 ∈ X 1 | P t ( x 1 | x 0 , ω 0 , a 0 ) -P ( x 1 | x 0 , ω 0 , a 0 ) |· · ∑ ω 1 ∈ Ω ∑ a 1 ∈ A µ ( x 1 , ω 1 ) ϕ t ( a 1 | x 1 , ω 1 ) | ≤ ∑ ω 0 ∈ Ω ∑ a 0 ∈ A q P,ϕ t ,µ ( x 0 , ω 0 , a 0 ) ∥ P t ( ·| x 0 , ω 0 , a 0 ) -P ( ·| x 0 , ω 0 , a 0 ) ∥ 1 . Now, we proceed with the induction step, ∑ x k ∈ X k ∑ ω k ∈ Ω ∑ a k ∈ A | q P t ,ϕ t ,µ ( x k , ω k , a k ) -q P,ϕ t ,µ ( x k , ω k , a k ) | = ∑ x k -1 ∈ X k -1 ∑ ω k -1 ∈ Ω ∑ a k -1 ∈ A ∑ x k ∈ X k ∑ ω k ∈ Ω ∑ a k ∈ A | q P t ,ϕ t ,µ ( x k -1 , ω k -1 , a k -1 ) · · P t ( x k | x k -1 , ω k -1 , a k -1 ) µ ( x k , ω k ) ϕ t ( a k | x k , ω k )+ -q P,ϕ t ,µ ( x k -1 , ω k -1 , a k -1 ) P ( x k | x k -1 , ω k -1 , a k -1 ) µ ( x k , ω k ) ϕ t ( a k | x k , ω k ) | = ∑ x k -1 ∈ X k -1 ∑ ω k -1 ∈ Ω ∑ a k -1 ∈ A ∑ x k ∈ X k | q P t ,ϕ t ,µ ( x k -1 , ω k -1 , a k -1 ) P t ( x k | x k -1 , ω k -1 , a k -1 ) -q P,ϕ t ,µ ( x k -1 , ω k -1 , a k -1 ) P ( x k | x k -1 , ω k -1 , a k -1 ) | ≤ ∑ x k -1 ∈ X k -1 ∑ ω k -1 ∈ Ω ∑ a k -1 ∈ A ∑ x k ∈ X k | q P t ,ϕ t ,µ ( x k -1 , ω k -1 , a k -1 ) P t ( x k | x k -1 , ω k -1 , a k -1 ) -q P,ϕ t ,µ ( x k -1 , ω k -1 , a k -1 ) P t ( x k | x k -1 , ω k -1 , a k -1 ) | + ∑ x k -1 ∈ X k -1 ∑ ω k -1 ∈ Ω ∑ a k -1 ∈ A ∑ x k ∈ X k | q P,ϕ t ,µ ( x k -1 , ω k -1 , a k -1 ) P t ( x k | x k -1 , ω k -1 , a k -1 ) -q P,ϕ t ,µ ( x k -1 , ω k -1 , a k -1 ) P ( x k | x k -1 , ω k -1 , a k -1 ) | ≤ ∑ x k -1 ∈ X k -1 ∑ ω k -1 ∈ Ω ∑ a k -1 ∈ A | q P t ,ϕ t ,µ ( x k -1 , ω k -1 , a k -1 ) -q P,ϕ t ,µ ( x k -1 , ω k -1 , a k -1 ) | + ∑ x k -1 ∈ X k -1 ∑ ω k -1 ∈ Ω ∑ a k -1 ∈ A q P,ϕ t ,µ ( x k -1 , ω k -1 , a k -1 ) ·

Thus by induction hypothesis we obtain,

Returning to the quantity of interest we have:

We proceed bounding the first term in Inequality (3). Fixing a layer k ∈ [0 , . . . , L -1] , employing Azuma-Hoeffding inequality and noticing that ∥ P t ( ·| x k , ω k , a k ) -P ( ·| x k , ω k , a k ) ∥ 1 ≤ 2 , we have, with probability 1 -2 δ :

Finally summing over L , we have, with probability at least 1 -2 δ (which derives from a union bound between Azuma-Hoeffding inequality and the bound on the transitions):

To bound the remaining term in Inequality (3), we proceed as follows,

The second term is bounded by the previous analysis paying an additional L factor, while, to bound the first terms we apply the Azuma-Hoeffding inequality and proceed as follows:

with probability at least 1 -2 δ , given the union bound over the Azuma-Hoeffding and the bound on the outcomes. Finally, with a union bound between the bound on the transitions and the outcomes (which are both encompassed by the clean event) and the Azuma-Hoeffding inequalities, with probability at least 1 -4 δ , we have:

which concludes the proof.

D.2 Regret

In the following section we show that Algorithm 2 attains ˜ O ( √ T ) regret. This is done showing that the confidence intervals over transitions, outcomes and sender reward concentrate at a rate of ˜ O (1 / √ T ) .

Theorem 1. Given any δ ∈ (0 , 1) , with probability at least 1 -7 δ , Algorithm 2 attains regret R T ≤ ˜ O ( L 2 | X | √ T | A || Ω | ln ( 1 / δ ) ) .

Proof. We notice that the regret can be decomposed as follows:

The second term is bounded by H¨ older inequality and applying Lemma 3. To bound the first term we notice that, under the clean event, and by definition of the linear program solved by Algorithm 2, it holds:

Thus, we have,

The second term is bounded by H¨ older inequality and applying Lemma 3, which holds under the clean event, with probability at least 1 -2 δ . To bound the first term we employ Lemma 10 which holds under the clean event, with probability at least 1 -δ , and a union bound, which concludes the proof.

D.3 Violations

In the following section we show that Algorithm 2 attains ˜ O ( √ T ) violations. This is possible since, in the full-feedback setting, the incentive compatibility constraints collapse to standard linear constraints.

Theorem 2. Given δ ∈ (0 , 1) , with probability at least 1 -7 δ , Algorithm 2 attains violation V T ≤ ˜ O ( L 2 | X | √ T | A || Ω | ln ( 1 / δ ) ) .

Proof. In the proof, we compactly denote the receivers' best response in a given state-action pair ( x, a ) ∈ X × A at time t ∈ [ T ] as b t ( a, x ) := b ϕ ̂ q t ( a, x ) . Furthermore, by employing the definition of the linear program and summing over ( x, ω, a ) , for any episode t , under the clean event, it holds:

which, in turn, implies that:

Thus, noticing that, in the full-feedback setting, we have ξ R,t ( x, ω, a ) = ξ R,t ( x, ω, b t ( a, x )) , we obtain:

where ξ R,t ( x, ω ) = √ ln(3 T | X || Ω | /δ ) max { 1 ,N t ( x,ω ) } .

Now we combine the previous equations to bound the first term of the last inequality as follows:

where Inequality (5) holds by H¨ older inequality and Lemma 3, which holds under the clean event, with probability at least 1 -2 δ , Inequality (6) follows by Azuma-Hoeffding and Inequality (7) by Cauchy-Schwarz inequality and observing that 1 + ∑ t ∈ [ T ] 1 √ t ≤ 3 √ T .

Finally, returning to the quantity of interest, we bound the cumulative violations as follows:

where the last steps hold by H¨ older inequality, Lemma 3 and the previous bound on the estimated occupancy measure. The final result holds with probability at least 1 -7 δ employing a union bound over the clean event, which holds with probability at least 1 -4 δ , the Azuma-Hoeffding inequality used above, which holds with probability at least 1 -δ and Lemma 3, which, under the clean event, holds with probability at least 1 -2 δ .

D.4 Lower Bound

Theorem 3. Let δ ∈ (0 , 1 / 4 ) . For every algorithm that guarantees R T ≤ O ( √ T ) with probability at least 1 -δ , there exists an absolute constant ψ > 0 and a problem instance in which the algorithm must have V T ≥ Ω( √ T ) with probability larger than ψ.

Proof. We consider two instances with a single possible outcome and a single state. In the following, we omit the dependence on the sender and receiver utility from these parameters. We assume that the sender's utility in the first instance is a deterministic function given by r 1 S ( a 1 ) = 1 and r 1 S ( a 2 ) = 0 , while the receiver's utility is given by r 2 R ( a 1 ) ∼ Be (1 / 2 + ϵ ) and r 2 R ( a 2 ) ∼ Be (1 / 2) . Meanwhile, the sender's utility in the second instance is r 2 S ( a 1 ) = 1 and r 1 S ( a 2 ) = 0 , while the follower's utility is equal to r 2 R ( a 1 ) ∼ Be (1 / 2 + ϵ ) and r 2 R ( a 2 ) ∼ Be (1 / 2 + 2 ϵ ) , for some ϵ ∈ (0 , 1 / 2) . Thus, the sender's regret in the first instance is given by:

since the optimal signaling scheme is the one that always recommends action a 1 ∈ A in the first instance. In the following, we define P 1 ( respectively, P 2 ) as the probability measure induced by recommending, at each round, signaling schemes according to some algorithm in the first (respectively, second) instance. Then, we bound the probability that the regret in the first instance is larger than a constant C ∈ N as follows:

for some η ∈ (0 , 3 / 4) . Furthermore, by Pinsker's inequality and Equation (8) the following holds.

where we denote with D KL ( · , · ) the Kullback-Leibler divergence between two probability measure. By means of the well known divergence decomposition, we have:

where in the latter inequality we used the well known property ensuring that D KL ( Be ( p ) , Be ( q )) ≤ ( p -q ) 2 q (1 -q ) . Furthermore, by means of the latter inequality and Equation (10) we have:

We now consider the receiver's violations in the second instance which can be computed as follows:

Then, by means of Equation (9) we get:

Finally, by setting ϵ = 1 16 √ T we get:

Then, the latter result implies that:

which concludes the proof.

E Partial feedback

E.1 Regret

Lemma 10. Under the event E ( δ ) , with probability at least 1 -δ , it holds:

Proof. We bound the quantity of interest as follows:

where Inequality (11) holds by the Azuma-Hoeffding inequality with probability 1 -δ , Inequality (12) follows by observing that 1 + ∑ t ∈ [ T ] 1 √ t ≤ 3 √ T , Inequality (13) follows from the Cauchy-Schwarz inequality, and Inequality (14) holds, noticing that ∑ x ∈ X,ω ∈ Ω ,a ∈ A N T ( x, ω, a ) ≤ LT . With the same analysis, we can prove that the same upper bound holds for ∑ t ∈ [ T ] ξ ⊤ R,t q t , concluding the proof.

Theorem 4. Given any δ ∈ (0 , 1) , with probability at least 1 -7 δ , Algorithm 3 attains regret R T ≤ ˜ O ( NL | X || Ω || A | + L 2 | X | √ T | A || Ω | ln ( 1 / δ ) ) .

Proof. As a first step, we decompose the sender's regret as follows:

We observe that the last inequality holds under the event E ( δ ) , with a probability of at least 1 -2 δ , and it is derived by applying the H¨ older inequality and employing Lemma 3. To bound the first term in Equation (15), we notice that under E ( δ ) , we have:

for each t > N | X || Ω || A | because of the optimality of ̂ q t . Thus, rearranging the latter chain of inequalities we have:

In the first inequality above, we employ the fact that the support of each reward function belongs to [0 , 1] , while in the second inequality, we make use of Lemma 3, the H¨ older inequality, and Lemma 10,

which hold with a probability of at least 1 -3 δ . Substituting the latter inequality into Equation (15), we obtain:

Finally, we observe that the previous upper bound holds with probability at least 1 -7 δ . This follows by employing a union bound and observing that E ( δ ) holds with a probability at least 1 -4 δ , which concludes the proof.

E.2 Violations

In the following we denote the receivers' best response in a given action a ∈ A and state x ∈ X as b t ( a, x ) := b ϕ ̂ q t ( a, x ) .

Lemma 11. Under the event E ( δ ) the following holds:

with probability at least 1 -3 δ .

Proof. As a first step, we observe that by employing the definition of ξ R,t and noticing that ̂ q t is a feasible solution to LP (2) for each t ∈ [ T ] under the event E ( δ ) , we have:

Then, rearranging the above inequality we get:

Furthermore, we can decompose the receivers' violations as follows:

where the first and third inequalities hold by Lemma 3, the second inequality is a consequence of Inequality (16), and the third inequality follows by means of Lemma 10, which holds with a probability of at least 1 -δ . Therefore, employing a union bound over the events of Lemma 3 and Lemma 10, the previous result holds with probability at least 1 -3 δ , under the clean event.

Theorem 5. Given any δ ∈ (0 , 1) , with probability at least 1 -9 δ , Algorithm 3 attains cumulative violation V T ≤ ˜ O ( ρ ( | A | T √ N + | A | √ N + L 2 √ T )) , where ρ := ( | X || Ω || A | ) 3 / 2 √ ln ( 1 / δ ) .

Proof. As a preliminary observation, we notice that Algorithm 3 is divided into N epochs of length ℓ = | X || Ω || A | , where in each epoch, Algorithm 3 maximizes the probability of visiting each triplet ( x, ω, a ) . In the following, we define t j ( x, ω, a ) ∈ [ T ] as the round in which Algorithm 3 maximizes the occupancy of the triplet ( x, ω, a ) in the epoch j ∈ [ N -1] . Formally:

Furthermore, for each occupancy measure q t with t ∈ [ T ] , the following holds:

for each j ∈ [ N -1] where q ∈ Q is an occupancy measure that satisfies the IC constraints of the offline optimization problem (see Program (1)). The first equality above follows by observing that there always exists an occupancy that satisfies the IC constraints that recommends action b t ( a, x ) ∈ A instead of a ∈ A in the state x ∈ X with the same probability of q t . The inequality, on the other hand, follows by observing that the space of occupancy measures satisfying the IC constraint of the offline optimization problem (1) is always a subset of the feasibility set of Program (2).

Furthermore, by Lemma 11 we have that:

We focus on bounding the second term in the inequality above in the first Nℓ rounds of Algorithm 3. Thus, with probability at least 1 -δ we have:

where we let N j ( x, ω, a ) = ∑ i ≤ j ✶ t i ( x,ω,a ) ( x, ω, a ) for the the sake of simplicity. Furthermore, we notice that Inequality (18) follows observing that ξ r,t ( x, ω, a ) ≤ 1 for each ( x, ω, a ) ∈ X × Ω × A and t ∈ [ T ] , and because the occupancy defines a probability distribution over each layer k ∈ [0 , . . . , L ] . Inequality (19) holds thanks to Inequality (17). Inequality (20) follows because each indicator function takes value of at most one. Inequality (21) follows by observing that the number of times that the triplet ( x, ω, a ′ ) is visited overall is always greater or equal to the the number of times such a triplet has been visited during the rounds in which Algorithm 3 maximizes the exploration of that triplet. Inequality (22) holds with probability at least 1 -δ and follows from the AzumaHoeffding inequality, and Inequality (24) holds, noticing that ∑ x ∈ X,ω ∈ Ω ,a ∈ A N T ( x, ω, a ) ≤ LNℓ and employing the Cauchy-Schwarz inequality.

We focus on bounding the cumulative violations suffered in the remaining T -Nℓ rounds of Algorithm 3. With probability at least 1 -δ the following holds:

Inequality (25) holds thanks to Inequality (17) and observing that the indicator function takes value of at most one. Inequality (27) holds, noticing that ∑ x ∈ X,ω ∈ Ω ,a ∈ A N T ( x, ω, a ) ≤ LNℓ and employing the Cauchy-Schwarz inequality. Inequality (26) holds with probability at least 1 -δ and follows by employing the Azuma-Hoeffding and observing the following:

which can be written as follows:

Finally, thanks to Lemma 11 and employing Inequality (24) and Inequality (27) we get:

With ρ := ( | X || Ω || A | ) 3 / 2 √ ln ( 1 / δ ) , such a result holds with a probability of at least 1 -9 δ , employing a union bound and observing that E ( δ ) holds with a probability of at least 1 -4 δ , Lemma 11 holds with a probability of at least 1 -3 δ , and both Inequality (24) and Inequality (27) hold with a probability of at least 1 -δ .

E.3 Lower bound

Theorem 6. Given α ∈ [1 / 2 , 1] , there is no learning algorithm achieving both R T = o ( T α ) and V T = o ( T 1 -α / 2 ) with probability greater or equal to a fixed constant ψ > 0 .

Proof. We consider two instances with a single possible outcome and a single state. In the following, we omit the dependence on the sender and receiver utility from these parameters. We assume that the

sender's utility in the first instance is a deterministic function given by r 1 S ( a 1 ) = 1 and r 1 S ( a 2 ) = 0 , while the receiver's utility is given by r 2 R ( a 1 ) ∼ Be (1 / 2 + ϵ ) and r 2 R ( a 2 ) ∼ Be (1 / 2) . Meanwhile, the sender's utility in the second instance is r 2 S ( a 1 ) = 1 and r 1 S ( a 2 ) = 0 , while the follower's utility is equal to r 2 R ( a 1 ) ∼ Be (1 / 2 + ϵ ) and r 2 R ( a 2 ) ∼ Be (1 / 2 + 2 ϵ ) , for some ϵ ∈ (0 , 1 / 2) . Thus, the sender's regret in the first instance is given by:

since the optimal signaling scheme is the one that always recommends action a 1 ∈ A in the first instance. In the following, we define P 1 ( respectively, P 2 ) as the probability measure induced by recommending, at each round, signaling schemes according to some algorithm in the first (respectively, second) instance. Then, we bound the probability that the regret in the first instance is larger than a constant C ∈ N as follows:

for some η ∈ (0 , 1) . Furthermore, by Pinsker's inequality and Equation (29) the following holds.

where we denote with D KL ( · , · ) the Kullback-Leibler divergence between two probability measure. By means of the well known divergence decomposition, we have:

where in the latter inequality we used the well known property ensuring that D KL ( Be ( p ) , Be ( q )) ≤ ( p -q ) 2 q (1 -q ) . Then, by reverse Markov inequality and Equation (29) we get:

Furthermore, by means of the latter inequality and Equation (31) we have:

We now consider the receiver's violations in the second instance which can be computed as follows:

Then, by means of Equation (30) we get:

since α ∈ [1 / 2 , 1] . Then, the latter result implies that:

which concludes the proof.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles Markov persuasion processes (MPP) where a sender interacts with a sequence of myopic receivers, where the sender must learn how to disclose information optimally without access to the environment’s dynamics or the preferences of he receivers. It proposes a learning algorithm for the sender for the case of partial feedback where the regret (w.r.t. an policy optimal of information disclosure) is sublinear w.r.t. the episodes and also the violations of the persuasiveness constraints remain sublinear. The paper also shows that the sublinear bounds it obtains in the analysis are tight.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is well written and the relevance of its results clearly explained -- overall, the paper looks rather impressive when it comes to its results. Due to its technical nature, though, most of the paper's results are confined to the appendices,, and not much is done in the actual paper to summarize how the results of its many theorems have been obtained. The paper is also completely theoretical in nature, and lacks any empirical validation. As far as I can see, it's the first learning framework for MPPs where the sender does not know the transition dynamics, the priors, or the utility functions; as such, I think a bit of extra effort should be made in explaining in some detail what applications may benefit from this framework (some applications are mentioned, but only in a very abstract way). Also, if the motivations of the paper are not purely theoretical, some form of empirical validation (which is lacking) should be included.\n\nQUESTIONS:\n- Can the authors comment on practical applications of their setup and proposed algorithm?\n\n- Can the authors comment on the practical efficiency of their algorithm? Some, even if of small-size, computaitonal results would better emphasize the relevance of the work.\n\n- Can the authors comments on the numerics of their algorithm? I am asking about the many linear programming problems that the proposed algorithm solves. Are they well-conditioned or can there be cases (e.g, with small state spaces, symmetric transitions, or almost identical receivers) where the persuasiveness and occupancy constraints become very similar, leading to rank-deficient basis matrices?\n\nLIMITATIONS:\nYes (there was nothing to report).", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the no-regret learning problem in Markov persuasion processes (MPPs), relaxing the assumption of previous work that the agent knows the receiver's utility function. It introduces the Optimistic Persuasive Policy Search (OPPS) algorithm, capable of learning the receiver's utility function from scratch, and provides rigorous regret bounds for the proposed algorithm under both full and partial feedback settings. The paper also provide a lower bound analysis on the optimality of the regret achieved by OPPS.\n\nSTRENGTHS AND WEAKNESSES:\nStrength \n\n- The authors consider the RL problem in MPPs, which is an interesting new problem from prior work. The analysis of this paper relies on the techniques of concentration measure, which offers a novel perspective to the studies of MPPs.\n\n- The overall writing of this paper is clear, and it is clear to me how their results can be derived from the technical lemmas and proof sketches outlined in the main body.\n\nWeakness\n\n- I have reviewed earlier versions of this paper at several different venues before. The outcome of this paper has been at the borderline for acceptance and I have not noticed any big change since then. I support the authors' decision for resubmission but I notice there's almost no additional effort made to improve the paper.\n\n- I am not sure if the limitations of this paper have been clearly acknowledged by the authors in Section 2.1 and 2.2.\n\n- The paper measures the learning performance with two quantities, total regret and total violation of persuasiveness. In contrast, [Wu et al. 2022] ensures persuasiveness in high probability and only considers the total regret. I can see the technical reasons behind this choice, but is there any practical motivation for this evaluation metric? Why would the receiver follow the persuasion if the recommendation is not completely incentive-compatible? Are receivers no-regret learners as well so they can tolerate \\sqrt{T} loss? Is it possible to convert any result of \\sqrt{T} in total regret and total violation of persuasiveness into an equivalent regret bound where the persuasiveness is guaranteed with high probability by introducing certain robustification/pessimism design to the signaling scheme? Without these explanations, I am not quite convinced that this learning result in MPPs is meaningful enough.\n\nQUESTIONS:\nCan the authors explicitly point out the improvement of their paper since the last version?\n\nOverall, I believe my technical questions about this paper have been answered again and again, and I have become overall satisfied with their answers, given that many problems cannot be meaningfully addressed under the current framework. I think the authors could have used some of their answers to acknowledge the limitations of this work.\n\nLIMITATIONS:\nNo, though I see that there is no potential negative societal impact from this theory work, I think the authors could have better acknowledge the limitation of their modeling assumptions or technical results.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper considers the problem of solving Markov Bayesian persuasion problems when the sender knows nothing about the game and proposes a learning algorithm that finds the solution with sublinear regret. Moreover, the authors show that the proposed bound is tight.

STRENGTHS AND WEAKNESSES:
The problem studied in the paper is interesting and practical. No assumption is made about the sender's knowledge about the game. This significantly improve the applicability of the Markov Bayesian persuasion model. The paper is well-written and easy to follow. The proposed method is similar to the bandit algorithms where optimistic actions are taken. I like the idea of the exploration phase in the partial feedback case. Such a phase provides a good enough initial guess but affects the overall regret. The regret can be tuned by controlling the length of the phase. The tightness result is strong, though not surprising. 

Overall, I think this is a solid theory paper. The results could also open up new research opportunities for similar problems.

QUESTIONS:
Can the framework be extended to the case where the learning steps are limited, or where each step incurs a cost?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper gives an interesting extension of the so called MPP (Markov Persuasion Process) by introducing the learning component. The paper considers the setting of MPP where both senders and receivers have no knowledge of the (implicit) \"markovian\" environment's transitions and rewards. The paper addresses non-trivial extensions to MPP with a novel solution (specially relating the occupancy measure to the MPP formulation).\n\nSTRENGTHS AND WEAKNESSES:\nS:\n\n1) Extremely well written with a clear model description, especially the first three sections.\n\n2) While the setting is still not practical in the current format, the MPP formulation definitely deserves a good consideration and analysis. Involving learning, this paper could give a reasonable first one.\n\n3) Occupancy measure and signaling policy equivalence is very insightful and may be useful independently.\nW:\n\n1)I learned about MPP while reading this submission. I do not follow what do the states of the MDP represent for any practical situation. Can the authors explain that? Further, what are some practical examples where MPP might be useful?\n\n2)Why is the assumption that a new receiver observes a_k at each time step important?\n\n3)In their model, the sender/principal commits to a signaling policy throughout the play. This seems to be very restrictive. Very reasonably, I would assume the signal policy is allowed to be adopted as new information is revealed.\n\n4)A minor comment: In RL terminology, partial feedback (as referred to by the authors) is called bandit feedback in most other papers. While the full feedback (as referred to by the authors) is still weaker (as rightly pointed out in the footnote). My suggestion would be to to call the full feedback as simply partial feedback\n\n5)In Lines 150-155, the authors argue for a direct and persuasive signal policy for a one-shot Bayesian persuasion framework. The argument is not clear. Further, the actual algorithm 2 does not play a persuasive policy. It is not clear what the structure of the sender's signals is with algorithm 2. What is the significance of playing a persuasive policy?\n\n6)The MDP structure of MPP induces a POMDP - Partially observable MDP. The authors should include this characterization as well.\n\n7)A lot of information is pushed to the appendix about Problem 2, as solved in Algorithm 2. A summary of the optimization program will be helpful. e.g., what does the objective look like, the main constraints? I am concerned about the computational traceability of the same. If it is tractable, what is the runtime? Further \"optimistic exploration\" is hidden in the objective of the optimization. Given the details of the main paper is hard to understand how the uncertainty quantification is being done by the use of optimism. Unfortunately, this is one of the biggest weaknesses of the paper.\n\nQUESTIONS:\nsee weaknesses\n\nLIMITATIONS:\nNeed discussions on: the practical utility of the MPP model; computational traceability of the proposed algorithms", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-06"}