SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[Put05] Martin L Puterman. Markov decision processes: Discrete stochastic dynamic programming. John Wiley & Sons, 2005.
[VBW15] Sofia S. Villar, Jack Bowden, and James Wason. Multi-armed Bandit Models for the Optimal Design of Clinical Trials: Benefits and Challenges. Statistical Science, 30(2):199 - 215, 2015.
[Ver16] I. M. Verloop. Asymptotically optimal priority policies for indexable and nonindexable restless bandits. Ann. Appl. Probab., 26(4):1947-1995, 2016.
[Whi88] Peter Whittle. Restless bandits: activity allocation in a changing world. J. Appl. Probab., 25:287 - 298, 1988.
[WW90] Richard R. Weber and Gideon Weiss. On an index policy for restless bandits. J. Appl. Probab., 27(3):637-648, 1990.
[ZCJW19] Gabriel Zayas-Caban, Stefanus Jasin, and Guihua Wang. An asymptotically optimal heuristic for general nonstationary finite-horizon restless multi-armed, multi-action bandits. Advances in Applied Probability, 51:745–772, 2019.
[ZF21] Xiangyu Zhang and Peter I. Frazier. Restless bandits with many arms: Beating the central limit theorem. arXiv:2107.11911 [math.OC], 2021.
[ZF22] Xiangyu Zhang and Peter I. Frazier. Near-optimality for infinite-horizon restless bandits with many arms. arXiv:2203.15853 [cs.LG], 2022.
A More related work
In this section, we provide additional discussion on related work.
Relationship between the infinite-horizon average-reward setting and the finite-horizon total-reward/infinite-horizon discounted-reward setting. While infinite-horizon average-reward RBs, finite-horizon total-reward RBs, and infinite-horizon discounted-reward RBs are three different settings, one may think that the approaches in the other two settings apply to the infinite-horizon average-reward setting by taking a large enough horizon in the finite-horizon setting or letting the discount factor to go to 1 in the infinite-horizon discounted-reward setting. Here we briefly discuss the fundamental difference between the infinite-horizon average-reward setting and the other two settings for RB problems. For simplicity, we will refer to the other two settings as the finite-horizon/discounted-reward setting.
From the algorithm design perspective, the finite-horizon/discounted-reward setting focuses on optimizing the transient performance, while the infinite-horizon average reward setting focuses on optimizing the steady-state performance. This distinction has a big impact on the algorithm design and the complexity. To see this, we note that both the finite-horizon/discounted-reward setting [HF17, ZCJW19, BS20, GNJT23, ZF21, ZF22, GGY22] and the infinite-horizon average-reward setting [Whi88, WW90, Ver16, GGY20, GGY22] utilize certain forms of linear programming (LP) relaxations to design algorithms. However, the LPs utilized differ substantially. In the finite-horizon/discounted-reward setting, the LP optimizes the future trajectory for a predetermined number of time steps. In contrast, in the infinite-horizon average-reward setting, the LP solves for the optimal state-action frequency in steady state, which can be seen as a fixed point rather than a trajectory. As a result, the number of variables and constraints of the LP for the finite-horizon/discounted-reward setting scales with the number of time steps, which is not the case for the infinite-horizon average-reward setting. On the other hand, because the LP relaxation for the infinite-horizon average-reward setting does not take into account the transient behavior, it provides less information, making the algorithm design more tricky. Consequently, the policies in prior work require the strong assumption of UGAP to achieve asymptotic optimality.
From the analysis perspective, the infinite-horizon average-reward setting requires a more careful analysis of the long-term effect of actions. To see this, note that the optimality gap bounds in the finite-horizon/discounted-reward setting have at least a quadratic dependency on the (effective) horizon [see, e.g. ZF22, GGY22]. Consequently, when translated to the bounds on the optimality gap of average reward per time slot, those bounds diverge to infinity as the time horizon goes to infinity.
Relationship between the RB problem and other bandit problems. While the exact optimality of the RB problem is in general intractable, there is a special case that has been solved optimally. Specifically, consider the case when an arm stays in the same state when it is not pulled, and only one arm is pulled at a time. The optimal policy for this special case is the celebrated Gittins index policy [GJ74, Git79]. A more recent reference on this topic is [GGW11].
The RB problem falls within the broader class of bandit problems, for which different formulations exist including stochastic bandits, adversarial bandits, and Bayesian bandits. The common theme in these formulations is to find a reward-maximizing strategy of pulling arms in the presence of uncertainty in the arms' rewards; see the book [LS20] for a comprehensive overview. Among these formulations, closely related to RBs is the Bayesian bandit problem, where Bayesian posteriors are used to model knowledge of unknown reward distributions. The Bayesian posterior can be seen as a state with known transition probabilities, hence the Bayesian bandit problem can be analyzed by applying tools from RBs. Examples demonstrating this connection can be found in [BS20].
B Discussion on the computational costs of FTVA
FTVA defined in Algorithm 1 can be computed and implemented efficiently.
First, computing FTVA requires solving (LP) once. Given that (LP) does not depend on  $N$ , the computational cost here is a constant.
Next, we examine the computational cost of implementing FTVA.
- Each arm simulates a virtual process. Because the computation required for each simulation does not scale with  $N$ , the overall computational cost here is linear in  $N$ .
- In each time step, FTVA selects arms from a subset of arms to determine the real actions. The computational cost for this selection process scales linearly with  $N$ .
Combining these components, we can see that the computational cost of implementing FTVA is linear in  $N$ .
Additionally, we note that the simulation of the virtual processes is independent across the arms, so they can be implemented in a fully distributed manner. However, when taking real actions, the policy needs to know the virtual actions from all arms, so this step cannot be implemented distributedly.
C Sufficient conditions for Synchronization Assumption (SA)
Recall that for the discrete-time setting, our bound on the conversion loss in Theorem 1 holds when the input single-armed policy  $\bar{\pi}$  satisfies the Synchronization Assumption (SA). SA stipulates that the two arms in the leader-and-follower system (cf. Section 4.1) under the policy  $\bar{\pi}$  will reach the same state in finite expected time. All other existing work requires the UGAP assumption, which pertains to the global behavior of a non-linear dynamic system and has no known sufficient conditions that are easily checkable. In comparison, SA is a reachability assumption imposed on a finite state Markov chain and thus substantially simpler.
To provide a more intuitive understanding of SA, in this section we present several sufficient conditions for SA to hold. These conditions involve the recurrent classes in the Markov chains induced by certain single-armed policies as well as the existence of self-loops or cycles in the corresponding transition diagrams. Such conditions can be verified, often in a straightforward manner, by inspecting the transition probabilities. As self-loops and cycles are common in many classes of MDPs, these conditions showcase that SA is a relatively mild assumption. We emphasize that the sufficient conditions presented in this section are not exhaustive. They serve as illustrative examples of when SA holds and offer insights on the nature of synchronization.
The discussion in this section pertains to the discrete-time setting. We reiterate that our results for the continuous-time setting do not require SA.
Before presenting the sufficient conditions, we first prove a few preliminary facts. The first one is a basic property of unichain MDPs.
Proposition 1. Consider two arbitrary Markovian policies  $\bar{\pi}$  and  $\bar{\pi}'$  for the unichain MDP  $(\mathbb{S},\mathbb{A},P,r)$ . Let the recurrent classes of the two policies  $\bar{\pi}$  and  $\bar{\pi}'$  be  $S$  and  $S'$ , respectively. Then  $S \cap S' \neq \emptyset$ .
Proof. We prove this proposition by contradiction. Suppose  $\mathcal{S} \cap \mathcal{S}' = \emptyset$ . We define a new policy  $\bar{\pi}''$  as
$$
\bar {\pi} ^ {\prime \prime} (a | s) = \left\{ \begin{array}{l l} \bar {\pi} (a | s), & \text {i f} s \in \mathcal {S}, \\ \bar {\pi} ^ {\prime} (a | s), & \text {o t h e r w i s e}. \end{array} \right. \quad \text {f o r} s \in \mathbb {S}, a \in \mathbb {A}.
$$
Then  $S$  and  $S'$  are two distinct recurrent classes under  $\bar{\pi}''$ , because by definition of  $\bar{\pi}''$ , an arm with the initial state in  $S$  remains in  $S$ , so it cannot reach  $S'$ ; similarly an arm cannot reach  $S$  from  $S'$ . The existence of two recurrent classes under  $\bar{\pi}''$  contradicts the unichain condition. Therefore, we must have  $S \cap S' \neq \emptyset$ .
The next proposition provides a convenient way for verifying SA and is used for establishing other sufficient conditions.
Proposition 2. Consider the leader-and-follower system under the policy  $\bar{\pi}$ . If there exists some  $t < \infty$  such that for all initial states and actions  $(s, a, \widehat{s}, \widehat{a}) \in \mathbb{S} \times \mathbb{A} \times \mathbb{S} \times \mathbb{A}$ ,
$$
\mathbb {P} \left(\tau^ {\text {s y n c}} (s, a, \widehat {s}, \widehat {a}) <   t\right) > 0,
$$
then Synchronization Assumption holds for the policy  $\bar{\pi}$ .
Proof. We claim that for any  $(s, a, \widehat{s}, \widehat{a}) \in \mathbb{S} \times \mathbb{A} \times \mathbb{S} \times \mathbb{A}$ ,
$$
\mathbb {P} \left(\tau^ {\text {s y n c}} (s, a, \widehat {s}, \widehat {a}) \geq k t\right) \leq \left(\max  _ {s ^ {\prime}, a ^ {\prime}, \widehat {s} ^ {\prime}, \widehat {a} ^ {\prime}} \mathbb {P} \left(\tau^ {\text {s y n c}} \left(s ^ {\prime}, a ^ {\prime}, \widehat {s} ^ {\prime}, \widehat {a} ^ {\prime}\right) \geq t\right)\right) ^ {k}, \tag {22}
$$
where the  $\max_{s', a', \hat{s}', \hat{a}'}$  is taken over  $\mathbb{S} \times \mathbb{A} \times \mathbb{S} \times \mathbb{A}$ . We prove this claim by induction on  $k$ . The base case of  $k = 1$  is given. Suppose we have proved (22) for a certain  $k$ , then
$$
\begin{array}{l} \mathbb {P} \left(\tau^ {\text {s y n c}} (s, a, \widehat {s}, \widehat {a}) \geq (k + 1) t\right) \\ = \mathbb {P} \left(\tau^ {\text {s y n c}} (s, a, \widehat {s}, \widehat {a}) \geq k t\right) \cdot \mathbb {P} \left(\tau^ {\text {s y n c}} (s, a, \widehat {s}, \widehat {a}) \geq (k + 1) t \mid \tau^ {\text {s y n c}} (s, a, \widehat {s}, \widehat {a}) \geq k t\right) \\ = \mathbb {P} \left(\tau^ {\text {s y n c}} (s, a, \widehat {s}, \widehat {a}) \geq k t\right) \cdot \mathbb {P} \left(\tau^ {\text {s y n c}} (S (k t), A (k t), \widehat {S} (k t), \widehat {A} (k t)) \geq t \mid S (k t) \neq \widehat {S} (k t)\right) \\ \leq \mathbb {P} \left(\tau^ {\text {s y n c}} (s, a, \widehat {s}, \widehat {a}) \geq k t\right) \cdot \max  _ {s ^ {\prime}, a ^ {\prime}, \widehat {s} ^ {\prime}, \widehat {a} ^ {\prime}} \mathbb {P} \left(\tau^ {\text {s y n c}} \left(s ^ {\prime}, a ^ {\prime}, \widehat {s} ^ {\prime}, \widehat {a} ^ {\prime}\right) \geq t\right) \\ \leq \Big (\max  _ {s ^ {\prime}, a ^ {\prime}, \widehat {s} ^ {\prime}, \widehat {a} ^ {\prime}} \mathbb {P} \left(\tau^ {\text {s y n c}} \left(s ^ {\prime}, a ^ {\prime}, \widehat {s} ^ {\prime}, \widehat {a} ^ {\prime}\right) \geq t\right) \Big) ^ {k + 1}. \\ \end{array}
$$
where in the second equality we have used the Markov property of the system, and in the last inequality we apply the induction hypothesis. We have proved (22).
The bound on the expectation  $\mathbb{E}\left[\tau^{\mathrm{sync}}(s,a,\widehat{s},\widehat{a})\right]$  follows by summing the tail bound (22) over  $k$  ..
$$
\begin{array}{l} \mathbb {E} \left[ \tau^ {\text {s y n c}} (s, a, \widehat {s}, \widehat {a}) \right] \leq \sum_ {k = 0} ^ {\infty} t \mathbb {P} \left(\tau^ {\text {s y n c}} (s, a, \widehat {s}, \widehat {a}) \geq k t\right) \\ \leq \frac {t}{1 - \max  _ {s ^ {\prime} , a ^ {\prime} , \widehat {s} ^ {\prime} , \widehat {a} ^ {\prime}} \mathbb {P} \left(\tau^ {\text {s y n c}} \left(s ^ {\prime} , a ^ {\prime} , \widehat {s} ^ {\prime} , \widehat {a} ^ {\prime}\right) \geq t\right)} \\ = \frac {t}{\operatorname* {m i n} _ {s ^ {\prime} , a ^ {\prime} , \widehat {s} ^ {\prime} , \widehat {a} ^ {\prime}} \mathbb {P} \left(\tau^ {\text {s y n c}} \left(s ^ {\prime} , a ^ {\prime} , \widehat {s} ^ {\prime} , \widehat {a} ^ {\prime}\right) <   t\right)} <   \infty . \\ \end{array}
$$
We have established SA and thereby finished the proof.
Remark 1. Proposition 2 implies that SA only requires that the Markov chain of the leader-and-follower system,  $(S(t),\widehat{S} (t))$ , can reach the subset of states  $\{(s,\widehat{s})\in \mathbb{S}^2\colon s = \widehat{s}\}$  from any initial state. As a result, SA can be efficiently verified using path-finding graph algorithms on the state transition diagram of the Markov chain.
C.2 Sufficient conditions based on self-loops
The unichain condition and Proposition 1 guarantee that there are some states that the leader arm and the follower arm will both visit for a positive fraction of times. If we can further show that the two arms have a positive probability to visit one of those states at the same time, then Proposition 2 can be applied to verify SA. A natural sufficient condition is the existence of self-loops, which guarantees that with a positive probability, the arm reaching those states first will wait for the other arm to come to the same state.
Proposition 3. Consider the single-armed problem with the unichain MDP  $(\mathbb{S},\mathbb{A},P,r)$  and budget  $\alpha \in (0,1)$ . Let  $\bar{\pi}$  be a single-armed policy with recurrent class  $\mathcal{S}$ . If  $P(s,0,s) > 0$  and  $P(s,1,s) > 0$  for all  $s\in \mathcal{S}$ , then Synchronization Assumption holds for the policy  $\bar{\pi}$ .
Proposition 3 assumes that all states in the recurrent class of the policy  $\bar{\pi}$  have self-loops. We can actually require fewer self-loops if we can characterize the recurrent states of the follower arm. In particular, let  $\bar{\pi}_1$  denote the all-one policy, the policy that applies action 1 in all states. Observe that if the leader arm applies action 1 for a sufficiently long time, the follower arm will also apply action 1 and effectively follows  $\bar{\pi}_1$ . The recurrent class of the policy  $\bar{\pi}_1$  must intersect the recurrent class of  $\bar{\pi}$ , so it suffices to have self-loops in the intersection of these two recurrent classes. The idea is formalized in Proposition 4.
2Although the follower arm does not follow a Markovian policy since it copies actions from the leader arm, by standard results on average reward MDP, the set of states in which the follower arm spends a positive fraction of time is the same as the recurrent class of some Markovian policy (cf. Chapter 8.9 of [Put05]). This class of states must intersect with  $S$  by Proposition 1 and the unichain condition.
Figure 4: The illustration of the positive probability sample paths that lead to synchronization in Proposition 4 (left) and Proposition 5 (right). In each figure, the dotted arrows correspond to the sample path of the leader arm's state, while the solid arrows correspond to the sample path of the follower arm's state. The numbers near the arrows denote the temporal order of the transition events.
Proposition 4. Consider the single-armed problem with the unichain MDP  $(\mathbb{S},\mathbb{A},P,r)$  and budget  $\alpha \in (0,1)$ . Let  $\bar{\pi}$  be a single-armed policy with recurrent class  $S$ . Let the recurrent class of the all-one policy  $\bar{\pi}_1$  be denoted as  $S_1$ . If the following conditions hold:
1. There exists  $s^a \in S$  such that  $\bar{\pi}(1|s^a) > 0$  and  $P(s^a, 1, s^a) > 0$ ;
2. There exists  $s^b \in S \cap S_1$  such that  $P(s^b, 0, s^b) > 0$  and  $P(s^b, 1, s^b) > 0$ ,
Proposition 5 below gives another sufficient condition, which only requires one self-loop. Note that this condition does not subsume the one in Proposition 4.
Proposition 5. Consider the single-armed problem with the unichain MDP  $(\mathbb{S},\mathbb{A},P,r)$  and budget  $\alpha \in (0,1)$ . Let  $\bar{\pi}$  be a single-armed policy with recurrent class  $S$ . Let the recurrent class of the all-one policy  $\bar{\pi}_1$  be denoted as  $S_1$ . If there exists  $s^* \in S \cap S_1$  such that  $\bar{\pi}(1|s^*) > 0$  and  $P(s^*,1,s^*) > 0$ , then Synchronization Assumption holds for the policy  $\bar{\pi}$ .
We remark that Propositions 4 and 5 have analogous versions that are stated in terms of the all-zero policy  $\bar{\pi}_0$ , the policy that applies action 0 in all states. We omit the details.
Now we prove the above propositions. Note that Proposition 3 is strictly weaker than Proposition 4: because the single-armed policy  $\bar{\pi}$  satisfies the budget constraint with  $\alpha \in (0,1)$ , there must be a state  $s^a \in S$  such that  $\bar{\pi}(1|s^a) > 0$ . Therefore, we only need to prove Propositions 4 and 5.
Proof of Proposition 4. Given any initial states and actions  $(S(0), A(0), \widehat{S}(0), \widehat{A}(0)) = (s, a, \widehat{s}, \widehat{a}) \in \mathbb{S} \times \mathbb{A} \times \mathbb{S} \times \mathbb{A}$ , we construct a sequence of positive probability events that leads to  $S(t) = \widehat{S}(t)$ :
1. The leader arm reaches the state  $s^a \in S$  after  $t_1$  time slots;
2. The leader arm stays in state  $s^a$  and keeps applying action 1 for  $t_2$  times slots; meanwhile, the follower arm also applies action 1 and reaches the state  $s^b \in S \cap S_1$ ;
3. The leader arm reaches the state  $s^b$  in another  $t_3$  time slots; meanwhile, the follower arm stays in the state  $s^b$ , so the two arms synchronize.
The transitions of the two arms during the above sequence of events are illustrated in Figure 4. We argue that there exists suitably large  $t_1, t_2, t_3$  such that the above three events happen with a positive probability. The first event can happen for a suitably large  $t_1$  because the leader arm follows the policy  $\bar{\pi}$ , and  $s^a$  is in the recurrent class of  $\bar{\pi}$ . In the second event, the leader arm can stay in state  $s^a$  and keep applying action 1 because  $\bar{\pi}(1|s^a) > 0$  and  $P(s^a, 1, s^a) > 0$ . The follower arm can reach  $s^b$  after a suitably large  $t_2$  time slots because it keeps applying action 1 and  $s^b$  is in the recurrent class of the all-one policy  $\bar{\pi}_1$ . In the third event, the leader arm can reach  $s^b$  after a suitably large  $t_3$  time slots because  $s^b$  is also in the recurrent class of  $\bar{\pi}$ . Meanwhile, the follower arm can stay in  $s^b$  because  $P(s^b, 0, s^b) > 0$  and  $P(s^b, 1, s^b) > 0$ . Therefore, we have proved that for any  $(s, a, \widehat{s}, \widehat{a})$ ,
$$
\mathbb {P} \left(\tau^ {\text {s y n c}} (s, a, \widehat {s}, \widehat {a}) \leq t _ {1} + t _ {2} + t _ {3}\right) > 0.
$$
By Proposition 2, we establish SA.
Proof of Proposition 5. Given any initial states and actions  $(S(0), A(0), \widehat{S}(0), \widehat{A}(0)) = (s, a, \widehat{s}, \widehat{a}) \in \mathbb{S} \times \mathbb{A} \times \mathbb{S} \times \mathbb{A}$ , we construct a sequence of positive probability events that leads to  $S(t) = \widehat{S}(t)$ :
1. The leader arm reaches the state  $s^*$  after  $t_1$  time slots;
2. The leader arm stays in the state  $s^*$  and keeps applying action 1 for  $t_2$  time slots; meanwhile, the follower arm also applies action 1 and reaches the state  $s^*$ , so the two arms synchronize.
The transitions of the two arms during the above sequence of events are illustrated in Figure 4. We argue that there exists suitable  $t_1$  and  $t_2$  such that the above two events happen with a positive probability. The first event can happen for a suitably large  $t_1$  because  $s^*$  is in the recurrent class of  $\bar{\pi}$ . In the second event, the leader arm can stay in the state  $s^*$  and keeps applying action 1 because  $\bar{\pi}(1|s^*) > 0$  and  $P(s^*, 1, s^*) > 0$ . Meanwhile, the follower arm can reach  $s^*$  after a suitably large  $t_2$  time slots because  $s^*$  is also in the recurrent class of the all-one policy  $\bar{\pi}_1$ . Therefore, we have proved that for any  $(s, a, \hat{s}, \hat{a})$ ,
$$
\mathbb {P} \left(\tau^ {\text {s y n c}} (s, a, \widehat {s}, \widehat {a}) \leq t _ {1} + t _ {2}\right) > 0.
$$
C.3 Sufficient conditions based on cycles
The above sufficient conditions are based on self-loops, which can be viewed as cycles of minimal length. These conditions can be generalized to longer cycles, formally defined below.
Definition 1 (Cycle). Consider the MDP  $(\mathbb{S},\mathbb{A},P,r)$ . We call an ordered set of states  $C = (s_0,s_1,\ldots ,s_L)$  a cycle under the Markovian policy  $\bar{\pi}$  if there is a positive probability of transitioning from  $s_j$  to  $s_{j + 1}$  for all  $j$  under the policy  $\bar{\pi}$  (we identify  $s_{L + 1}$  with  $s_0$ ). We call the ordered set  $C$  a cycle under any policy if there is a positive probability of transitioning from  $s_j$  to  $s_{j + 1}$  for all  $j$  under any policy. In both cases, we call  $L$  the length of the cycle  $C$ .
We give two sufficient conditions based on cycles. These conditions are relaxations of the conditions in Proposition 4 and 5.
Proposition 6. Consider the single-armed problem with the unichain MDP  $(\mathbb{S},\mathbb{A},P,r)$  and budget  $\alpha \in (0,1)$ . Let  $\bar{\pi}$  be a single-armed policy with recurrent class  $S$ . Let the recurrent class of the all-one policy  $\bar{\pi}_1$  be denoted as  $S_1$ . If the following conditions hold:
1. There exists a cycle  $C^a$  under the policy  $\bar{\pi}$  in  $S$ , and  $\bar{\pi}(1|s) > 0$  for all  $s \in C^a$ ;
2. There exists cycle  $C^b$  under any policy in  $\mathcal{S} \cap \mathcal{S}_1$ ;
3. The lengths of the two cycles  $C^a$  and  $C^b$  are relatively prime.
Proposition 7. Consider the single-armed problem with the unichain MDP  $(\mathbb{S},\mathbb{A},P,r)$  and budget  $\alpha \in (0,1)$ . Let  $\bar{\pi}$  be a single-armed policy with recurrent class  $S$ . Let the recurrent class of the all-one policy  $\bar{\pi}_1$  be denoted as  $S_1$ . If the following conditions hold:
1. There exists a cycle  $C^*$  under the policy  $\bar{\pi}$  in  $S \cap S_1$ , and  $\bar{\pi}(1|s) > 0$  for all  $s \in C^a$ ;
2. The policy  $\bar{\pi}_1$  is aperiodic,
Similarly to Propositions 4 and 5, Propositions 6 and 7 have analogous versions that are stated in terms of the all-zero policy  $\bar{\pi}_0$ . We omit the details.
Now we prove Propositions 6 and 7.
Figure 5: The illustration of the positive probability sample paths that lead to synchronization in Proposition 6 (left) and Proposition 7 (right). In each figure, the dotted arrows correspond to the sample path of the leader arm's state, while the solid arrows correspond to the sample path of the follower arm's state. The numbers near the arrows denote the temporal order of the transition events.
Proof of Proposition 6. Given any initial states and actions  $(S(0), A(0), \widehat{S}(0), \widehat{A}(0)) = (s, a, \widehat{s}, \widehat{a}) \in \mathbb{S} \times \mathbb{A} \times \mathbb{S} \times \mathbb{A}$ , we construct a sequence of positive probability events that leads to  $S(t) = \widehat{S}(t)$ :