SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yige Hong $^{1*}$  Qiaomin Xie $^{2}$  Yudong Chen $^{2}$  Weina Wang $^{1}$ \
 $^{1}$ Carnegie Mellon University  $^{2}$ University of Wisconsin-Madison  $\{\text{yigeh, weinaw}\} @ \text{cs.cmu.edu}$ $\{\text{qiaomin.xie,yudong.chen}\} @ \text{wisc.edu}$
We study the infinite-horizon restless bandit problem with the average reward criterion, in both discrete-time and continuous-time settings. A fundamental goal is to efficiently compute policies that achieve a diminishing optimality gap as the number of arms,  $N$ , grows large. Existing results on asymptotic optimality all rely on the uniform global attractor property (UGAP), a complex and challenging-to-verify assumption. In this paper, we propose a general, simulation-based framework, Follow-the-Virtual-Advice, that converts any single-armed policy into a policy for the original  $N$ -armed problem. This is done by simulating the single-armed policy on each arm and carefully steering the real state towards the simulated state. Our framework can be instantiated to produce a policy with an  $O(1 / \sqrt{N})$  optimality gap. In the discrete-time setting, our result holds under a simpler synchronization assumption, which covers some problem instances that violate UGAP. More notably, in the continuous-time setting, we do not require any additional assumptions beyond the standard unichain condition. In both settings, our work is the first asymptotic optimality result that does not require UGAP.
The restless bandit (RB) problem is a dynamic decision-making problem that involves a number of Markov decision processes (MDPs) coupled by a constraint. Each MDP, referred to as an arm, has a binary action space, {passive, active}. At every decision epoch, the decision maker is constrained to select a fixed number of arms to activate, with the goal of maximizing the expected reward accrued. The RB problem finds applications across a spectrum of domains, including wireless communication [ALT19], congestion control [AADJ13], queueing models [ABG09], crawling web content [AB19], machine maintenance [GMA05], clinical trials [VBW15], to name a few.
In this paper, we focus on infinite-horizon RBs with the average-reward criterion. Since the exact optimal policy is PSPACE-hard to compute [PT99], it is of great theoretical and practical interest to focus on policies that approximately achieve the optimal value and compute such policies in an efficient manner. The optimality gap of a policy is defined as the difference between its average reward per arm and that of an optimal policy. In a typical asymptotic regime where the number of arms,  $N$ , grows large, we say that a policy is asymptotically optimal if its optimality gap is  $o(1)$  as  $N \to \infty$ .
Prior work and the uniform global attractor property assumption. Prior work has studied the celebrated Whittle index policy [Whi88] and LP-Priority policies [Ver16] and established sufficient conditions for their asymptotic optimality [WW90, Ver16, GGY20, GGY22]. One key assumption underpinning all prior work is the uniform global attractor property (UGAP)—also known as globally asymptotic stability—which pertains to the mean-field/fluid limit of the restless bandit system in the
Paper	Policy	Optimality Gap	Conditions*
Discrete-time setting	[GGY20]	Whittle Index	O(exp(-cN))	UGAP & Non-singular
[GGY22]	LP-Priority	O(exp(-cN))	UGAP & Non-degenerate
This paper	FTVA(π*)	O(1/√N)	SA
Continuous-time setting	[WW90]	Whittle Index	o(1)	UGAP
[Ver16]	LP-Priority	o(1)	UGAP
[GGY20]	Whittle Index	O(exp(-cN))	UGAP & Non-singular
[GGY22]	LP-Priority	O(exp(-cN))	UGAP & Non-degenerate
This paper	FTVA-CT(π*)	O(1/√N)	-
Table 1: Optimality gap results and conditions. *All papers require the standard unichain assumption.
asymptotic limit  $N\to \infty$ . UGAP stipulates that the system's state distribution in the mean-field limit converges to the optimal state distribution attaining the maximum reward, from any initial distribution. It has been well recognized that UGAP is a highly technical assumption and challenging to verify: the primary way to test UGAP is numerical simulations; see [GGY20] for a detailed discussion.
More recent work studies the rate at which the optimality gap converges to zero. The work [GGY20] and [GGY22] prove a striking  $O(\exp(-cN))$  optimality gap for the Whittle index policy and LP-Priority policies, respectively, where  $c$  is a constant. In addition to UGAP, these results require a non-singularity or non-degenerate condition. We are not aware of any rate of convergence result without assuming UGAP or non-singularity/degeneracy. See Table 1 for a summary.
Therefore, prior work on average-reward restlest bandit leaves two fundamental questions open:
1. Is it possible to achieve asymptotic optimality without UGAP?
2. Is is possible to establish a non-trivial convergence rate for the optimality gap in the absence of the non-singular/non-degenerate assumption (and UGAP)?
Our contributions. We consider both the discrete-time and continuous-time settings of the average-reward restless bandit problem. We propose a general, simulation-based framework, Follow-the-Virtual-Advice (FTVA) and its continuous-time variant FTVA-CT, which convert any single-armed policy into a policy for the original  $N$ -armed problem, with a vanishing performance loss. Our framework can be instantiated to produce a policy with an  $O(1/\sqrt{N})$  optimality gap, under the conditions summarized in Table 1, which we elaborate on later.
Under our framework, computing an asymptotically optimal policy is efficient since it reduces to deriving an optimal single-armed policy, whose complexity is independent of  $N$ . The resultant policy can be implemented with a linear-in- $N$  computational cost and some of its subroutines can be implemented in a distributed fashion over the arms (see Appendix B for more details).
Our results can also be extended to RBs with heterogeneous arms. See Appendix H for details.
We now elaborate on the conditions in Table 1. In the discrete-time setting, our result holds under a condition called the Synchronization Assumption (SA), in addition to the standard unichain assumption required by all prior work. The SA condition, which is imposed on the MDP associated with a single arm, admits several intuitive sufficient conditions. While it is unclear whether SA subsumes UGAP, we show that there exist problem instances that violate UGAP but satisfy SA. Figure 1 shows one such instance (constructed by [GGY20], described in Appendix G for ease of reference), in which the Whittle Index and LP-Priority policies coincide and have a nondiminishing optimality gap, whereas our policy, named as  $\mathrm{FTVA}(\bar{\pi}^*)$ , is asymptotically optimal. In addition, our  $O(1/\sqrt{N})$  bound on the optimality gap does not require a non-singularity/non-degeneracy assumption.
Figure 1: An discrete-time RB problem that satisfies SA but not UGAP.
More notably, in the continuous-time setting, we completely eliminate the UGAP assumption. We show that our policy FTVA-CT $(\bar{\pi}^{*})$  achieves an  $O(1 / \sqrt{N})$  optimality gap under only the standard unichain assumption (which is required by all prior work).
In both settings, our results are the first asymptotic optimality results that do not require UGAP. We point out that UGAP is considered necessary for LP-Priority policies [Ver16, GGY22]. Our policy,  $\mathrm{FTVA}(\bar{\pi}^{*})$ , is not a priority policy. As such, we hope our results open up new directions for developing new classes of RB algorithms that achieve asymptotic optimality without relying on UGAP.
Intuitions. Our algorithm and many existing approaches solve an LP relaxation of the original problem. The solution of the LP induces a policy and gives an "ideal" distribution for the system state. Existing approaches directly apply the LP policy to the current system state, and then perform a simple rounding of the resulting actions so as to satisfy the budget constraint. When the current system state is far from the ideal distribution, the actual actions after rounding may deviate substantially from the LP solution and thus, in the absence of UGAP, would fail to drive the system to the optimum.
Our FTVA framework, in contrast, prioritizes constraint satisfaction. We apply the LP solution to the state of a simulated system, which is constructed carefully so that the resulting actions satisfy the constraint after a minimal amount of rounding. Consequently, starting from any initial state, our policy steers the system towards the ideal distribution and hence approximates the optimal value.
This method is inspired by the approach in [HXW23], which also involves a simulated system, but for a stochastic bin-packing problem.
Other related work. Another condition extensively discussed in the RB literature is the indexability condition [Whi88], which is necessary for the Whittle index policy to be well-defined, but not required by LP-Priority policies [Ver16]. However, indexability alone does not guarantee the asymptotic optimality of Whittle index.
So far we have discussed prior work on infinite-horizon RBs with average reward. For the related setting of finite-horizon total reward RBs, a line of recent work has established an  $O(1 / \sqrt{N})$  optimality gap [HF17, ZCJW19, BS20, GNJT23], and an  $O(\exp (-cN))$  gap assuming non-degeneracy [ZF21, GGY22]. For the infinite-horizon discounted reward setting, [ZF22, GNJT23] propose policies with  $O(1 / \sqrt{N})$  optimality gap without assuming indexability and UGAP. While these results are not directly comparable to ours, it is of future interest to see if our simulation-based framework can be applied to their settings. For a more detailed discussion of prior work, see Appendix A.
Paper Organization. While our continuous-time result is stronger, the discrete-time setting is more accessible. Therefore, we first discuss the discrete-time setting, which includes the problem statement in Section 2, our proposed framework, Follow-the-Virtual-Advice, in Section 3, and our results on the optimality gap in Section 4. Results for the continuous-time setting are presented in Section 5. We conclude the paper in Section 6. Proofs and additional discussion are given in the appendices.
Consider the infinite-horizon, discrete-time restless bandit problem with  $N$  arms indexed by  $[N] \triangleq \{1, 2, \dots, N\}$ . Each arm is associated with an MDP described by the tuple  $(\mathbb{S}, \mathbb{A}, P, r)$ . Here  $\mathbb{S}$  is the state space, assumed to be finite;  $\mathbb{A} = \{0, 1\}$  is the action set, and we say the arm is activated or pulled when action 1 is applied;  $P: \mathbb{S} \times \mathbb{A} \times \mathbb{S} \to [0, 1]$  is the transition kernel, where  $P(s, a, s')$  is the probability of transitioning from state  $s$  to state  $s'$  upon taking action  $a$ ;  $r = \{r(s, a)\}_{s \in \mathbb{S}, a \in \mathbb{A}}$  is the reward function, where  $r(s, a)$  is the reward for taking action  $a$  in state  $s$ . Throughout the paper, we assume that the transition kernel  $P$  is unichain [Put05]; that is, under any Markov policy for this single-armed MDP  $(\mathbb{S}, \mathbb{A}, P, r)$ , the induced Markov chain has a single recurrent class. The unichain assumption is standard in most existing work on restless bandits [WW90, GGY20, GGY22, Ver16]. We will discuss relaxing the unichain assumption in Appendix D.
In the above setting, we are subject to a budget constraint that exactly  $\alpha N$  arms must be activated in each time step, where  $\alpha \in (0,1)$  is a given constant and  $\alpha N$  is assumed to be an integer for simplicity. This  $N$ -armed RB problem can be represented by the tuple  $(N,\mathbb{S}^N,\mathbb{A}^N,P,r,\alpha N)$ .
A policy  $\pi$  for the  $N$ -armed problem specifies the action for each of the  $N$  arms based on the history of states and actions. Under policy  $\pi$ , let  $S_{i}^{\pi}(t) \in \mathbb{S}$  denote the state of the  $i$ th arm at time  $t$ , and we
call  $S^{\pi}(t) \triangleq (S_{i}^{\pi}(t))_{i \in [N]} \in \mathbb{S}^{N}$  the system state. Similarly, let  $A_{i}^{\pi}(t) \in \mathbb{A}$  denote the action applied to the  $i$ th arm at time  $t$ , and let  $A^{\pi}(t) \triangleq (A_{i}^{\pi}(t))_{i \in [N]} \in \mathbb{A}^{N}$  denote the joint action vector.
The controller's goal is to find a policy that maximizes the long-run average of the total expected reward from all  $N$  arms, subject to the budget constraint, assuming full knowledge of the model:
$$
\underset {\text {p o l i c y} \pi} {\operatorname {m a x i m i z e}} V _ {N} ^ {\pi} \triangleq \lim  _ {T \rightarrow \infty} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \frac {1}{N} \sum_ {i = 1} ^ {N} \mathbb {E} \left[ r \left(S _ {i} ^ {\pi} (t), A _ {i} ^ {\pi} (t)\right)\right] \tag {1}
$$
$$
\text {s u b j e c t} \quad \sum_ {i = 1} ^ {N} A _ {i} ^ {\pi} (t) = \alpha N, \quad \forall t \geq 0. \tag {2}
$$
Under the unichain assumption, the value  $V_{N}^{\pi}$  of any policy  $\pi$  is independent of the initial state. Let  $V_{N}^{*} \triangleq \sup_{\pi} V_{N}^{\pi}$  denote the optimal value. The optimality gap of  $\pi$  is defined as  $V_{N}^{*} - V_{N}^{\pi}$ . We say a policy  $\pi$  is asymptotically optimal if its optimality gap converges to 0 as  $N \to \infty$ .
Classical theory guarantees that for a finite-state Markov decision process like an RB, there exists an optimal policy that is Markovian and stationary [Put05]. Nevertheless, the policies we propose are not Markovian policies; rather, they have internal states. Under such a policy  $\pi$ , the system state  $S^{\pi}(t)$  together with the internal state form a Markov chain, and the action  $A^{\pi}(t)$  depends on both the system and internal states. We design a policy such that this Markov chain has a stationary distribution. Let  $S^{\pi}(\infty)$  and  $A^{\pi}(\infty)$  denote the random elements that follow the stationary distributions of  $S^{\pi}(t)$  and  $A^{\pi}(t)$ , respectively. Then the average reward of  $\pi$  is equal to  $V_{N}^{\pi} = \frac{1}{N}\sum_{i=1}^{N}\mathbb{E}[r(S_{i}^{\pi}(\infty), A_{i}^{\pi}(\infty))]$ . In later sections, when the context is clear, we drop the superscript  $\pi$  from  $S_{i}^{\pi}$  and  $A_{i}^{\pi}$ .
3 Follow-the-Virtual-Advice: A simulation-based framework
In this section, we present our framework, Follow-the-Virtual-Advice (FTVA). We first describe a single-armed problem, which involves an "average arm" from the original  $N$ -armed problem. We then use the optimal single-armed policy to construct the proposed policy  $\mathrm{FTVA}(\bar{\pi}^*)$ .
3.1 Single-armed problem
The single-armed problem involves the MDP  $(\mathbb{S},\mathbb{A},P,r)$  associated with a single arm (say arm 1 without loss of generality), where the budget is satisfied on average. Formally, consider the problem:
$$
\underset {\text {p o l i c y} \bar {\pi}} {\operatorname {m a x i m i z e}} V _ {1} ^ {\bar {\pi}} \triangleq \lim  _ {T \rightarrow \infty} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ r \left(S _ {1} ^ {\bar {\pi}} (t), A _ {1} ^ {\bar {\pi}} (t)\right)\right] \tag {3}
$$
$$
\text {s u b j e c t} \quad \lim  _ {T \rightarrow \infty} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ A _ {1} ^ {\bar {\pi}} (t) \right] = \alpha . \tag {4}
$$
The constraint (4) stipulates that the average rate of applying the active action must equal  $\alpha$ . Various equivalent forms of this single-armed problem have been considered in prior work [WW90, GGY20, GGY22, Ver16].
By virtue of the unichain assumption, the single-armed problem can be equivalently rewritten as the following linear program, where each decision variable  $y(s,a)$  represents the steady-state probability that the arm is in state  $s$  taking action  $a$ :
$$
\underset {\{y (s, a) \} _ {s \in \mathbb {S}, a \in \mathbb {A}}} {\text {m a x i m i z e}} \sum_ {s \in \mathbb {S}, a \in \mathbb {A}} r (s, a) y (s, a) \tag {LP}
$$
$$
\text {s u b j e c t} \quad \sum_ {s \in \mathbb {S}} y (s, 1) = \alpha \tag {5}
$$
$$
\sum_ {s ^ {\prime} \in \mathbb {S}, a \in \mathbb {A}} y \left(s ^ {\prime}, a\right) P \left(s ^ {\prime}, a, s\right) = \sum_ {a \in \mathbb {A}} y (s, a), \forall s \in \mathbb {S} \tag {6}
$$
$$
\sum_ {s \in \mathbb {S}, a \in \mathbb {A}} y (s, a) = 1, \quad y (s, a) \geq 0, \forall s \in \mathbb {S}, a \in \mathbb {A}. \tag {7}
$$
Algorithm 1 Follow-the-Virtual-Advice (FTVA): A simulation-based framework
Input:  $N$ -armed problem  $(N,\mathbb{S}^N,\mathbb{A}^N,P,r,\alpha N)$ , initial states  $S(0)$ , single-armed policy  $\bar{\pi}$   
Initialize: Virtual states  $\widehat{S}(0)$  are  $N$  i.i.d. samples following the stationary distribution of  $\bar{\pi}$   
1: for  $t = 0,1,2,\ldots$  do  
2: Independently sample  $\widehat{A}_i(t) \gets \bar{\pi}(\cdot|\widehat{S}_i(t))$  for each arm  $i \in [N]$   
3: if  $\sum_{i=1}^{N} \widehat{A}_i(t) \geq \alpha N$  then  
4:  $\mathcal{A} \gets$  a set of  $\alpha N$  arms chosen from  $\{i: \widehat{A}_i(t) = 1\}$  (any tie-breaking)  
5: else  
6:  $\mathcal{B} \gets$  a set of  $\alpha N - \sum_{i=1}^{N} \widehat{A}_i(t)$  arms chosen from  $\{i: \widehat{A}_i(t) = 0\}$  (any tie-breaking)  
7:  $\mathcal{A} \gets \{i: \widehat{A}_i(t) = 1\} \cup \mathcal{B}$   
8: Apply  $A_i(t) = 1$  and observe  $S_i(t + 1)$  for each arm  $i \in \mathcal{A}$   
9: Apply  $A_i(t) = 0$  and observe  $S_i(t + 1)$  for each arm  $i \notin \mathcal{A}$   
10: for  $i = 1,2,3,\ldots$  do  
11: if  $\widehat{S}_i(t) = S_i(t)$  and  $\widehat{A}_i(t) = A_i(t)$  then  
12:  $\widehat{S}_i(t + 1) \gets S_i(t + 1)$   
13: else  
14: Independently sample  $\widehat{S}_i(t + 1)$  from the distribution  $P(\widehat{S}_i(t),\widehat{A}_i(t),\cdot)$
Here (5) corresponds to the relaxed budget constraint, (6) is the flow balance equation, and (6)-(7) guarantee that  $y(s,a)$ 's are valid steady-state probabilities.
By standard results for average reward MDPs [Put05], an optimal solution  $\{y^{*}(s,a)\}_{s\in \mathbb{S},a\in \mathbb{A}}$  to (LP) induces an optimal policy  $\bar{\pi}^*$  for the single-armed problem via the following formula:
$$
\bar {\pi} ^ {*} (a | s) = \left\{ \begin{array}{l l} y ^ {*} (s, a) / \left(y ^ {*} (s, 0) + y ^ {*} (s, 1)\right), & \text {i f} y ^ {*} (s, 0) + y ^ {*} (s, 1) > 0, \\ 1 / 2, & \text {i f} y ^ {*} (s, 0) + y ^ {*} (s, 1) = 0. \end{array} \right. \quad \text {f o r} s \in \mathbb {S}, a \in \mathbb {A}. \tag {8}
$$
Let  $V_{1}^{\mathrm{rel}} = V_{1}^{\bar{\pi}^{*}}$  be the optimal value of (LP) and the single-armed problem.
(LP) can be viewed as a relaxation of the  $N$ -armed problem. To see this, take any  $N$ -armed policy  $\pi$  and set  $y(s,a)$  to be the fraction of arms in state  $s$  taking action  $a$  in steady state under  $\pi$ , i.e.,  $y(s,a) = \mathbb{E}\left[\frac{1}{N}\sum_{i=1}^{N}\mathbb{1}_{\{S_i^\pi(\infty) = s,A_i^\pi(\infty) = a\}}\right]$ . Whenever  $\pi$  satisfies the budget constraint (2),  $\{y(s,a)\}$  satisfies the relaxed constraint (5) and the consistency constraints (6)-(7). Therefore, the optimal value of (LP) is an upper bound of the optimal value of the  $N$ -armed problem:  $V_1^{\mathrm{rel}} \geq V_N^*$ .
3.2 Constructing the  $N$ -armed policy
We now present Follow-the-Virtual-Advice, a simulation-based framework for the  $N$ -armed problem. FTVA takes as input any single-armed policy  $\bar{\pi}$  that satisfies the relaxed budget constraint (4) and converts it into a  $N$ -armed policy, denoted by  $\mathrm{FTVA}(\bar{\pi})$ . Of particular interest is when  $\bar{\pi}$  is an optimal single-armed policy, which leads to our result on the optimality gap. Below we introduce the general framework of FTVA without imposing any restriction on the input policy  $\bar{\pi}$ .
The proposed policy  $\mathrm{FTVA}(\bar{\pi})$  has two main components:
- Virtual single-armed processes. Each arm  $i$  simulates a virtual single-armed process, whose state is denoted as  $\widehat{S}_i(t)$ , with action  $\widehat{A}_i(t)$  chosen according to  $\bar{\pi}$ . To make the distinction conspicuous, we sometimes refer to the state  $S_i(t)$  and action  $A_i(t)$  in the original  $N$ -armed problem as the real state/action. The virtual processes associated with different arms are independent.
- Follow the virtual actions. At each time step  $t$ , we choose the real actions  $A_{i}(t)$ 's to best match the virtual actions  $\widehat{A}_{i}(t)$ 's, to the extent allowed by the budget constraint  $\sum_{i=1}^{N} A_{i}(t) = \alpha N$ .
FTVA is presented in detail in Algorithm 1. Note that we use an appropriate coupling in Algorithm 1 to ensure that the virtual processes  $(\widehat{S}_i(t),\widehat{A}_i(t))$ ’s are independent and each follows the Markov chain induced by the single-armed policy  $\bar{\pi}$ . FTVA is designed to steer the real states to be close to the virtual states, thereby ensuring a small conversion loss  $V_{1}^{\bar{\pi}} - V_{N}^{\mathrm{FTVA}(\bar{\pi})}$ . Here recall that  $V_{1}^{\bar{\pi}}$  is the
Figure 2: Time evolution of the fraction of arms in each state under LP-Priority (upper), or after switching to  $\mathrm{FTVA}(\bar{\pi}^{*})$  (lower) since time slot 250. The x-axis represents the time slot, which ranges from 250 to 289; the y-axis represents the states; the color represents the fraction of arms in each state at each time slot. The colors and magnitudes of the arrows represent the average directions and rates at which the arms move away from each state.
average reward achieved by the input policy  $\bar{\pi}$  in the single-armed problem, and that  $V_{N}^{\mathrm{FTVA}(\bar{\pi})}$  is the average reward per arm achieved by policy FTVA  $(\bar{\pi})$  in the  $N$ -armed problem.
3.3 Discussion on FTVA and the role of virtual processes
In this subsection, we provide insights into the mechanism of FTVA and explain the crucial role of the virtual processes. In particular, we contrast FTVA with the alternative approach of directly using the real states to choose actions, e.g., by applying the single-armed policy  $\bar{\pi}^*$  to each arm's real state. We note that existing policies are essentially real-state-based, so the insights here can also explain why UGAP is necessary for existing policies to have asymptotic optimality.
We first observe that the above two approaches are equivalent in the absence of the budget constraint. In particular, even if the initial virtual state and real state of an arm  $i$  are different, they will synchronize (i.e., become identical) in finite time by chance under mild assumptions (see Section 4.1). After this event, if there were no constraint, each arm  $i$  will consistently follow the virtual actions, i.e.,  $A_{i}(t) = \widehat{A}_{i}(t) = \bar{\pi}\big(\widehat{S}_{i}(t)\big)$  for all  $t$ , and the virtual states will remain identical to real states.
In the presence of the budget constraint, the arms may not remain synchronized, so the virtual processes become crucial: they guarantee that the virtual actions  $\widehat{A}_i(t) = \bar{\pi}\big(\widehat{S}_i(t)\big)$  nearly satisfy budget constraint, which allows the real system to approximately follow the virtual actions to remain synchronized. To see this, note that regardless of the current real states, the  $N$  virtual states  $\widehat{S}_1,\ldots ,\widehat{S}_N$  independently follow the single-armed policy  $\bar{\pi}^*$ , so, in the long run, each  $(\widehat{S}_i(\infty),\widehat{A}_i(\infty))$  is distributed per  $y^{*}(\cdot ,\cdot)$ , the optimal solution to (LP). For large  $N$ , the sum  $\sum_{i = 1}^{N}\widehat{A}_{i}(\infty)$  concentrates around its expectation  $N\sum_{s\in \mathbb{S}}y^{*}(s,1) = \alpha N$  and thus tightly satisfy the budget constraint. In contrast, the actions generated by applying  $\bar{\pi}$  to the real states are likely to significantly violate the constraint, especially when the empirical distribution of the current real states deviates from  $y^{*}(\cdot ,\cdot)$ .