SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We select 17 multi-class time-series classification datasets from the UCR time-series repository [11] (see Appendix D for details of the datasets and selection process). We use 10-random  $70\% /30\%$  (train/test) data splits to train all models, holding out a  $30\%$  of training data as a validation set. Figure 3(b) reports the average stopping costs of the policies learned by the various methods on the test set. RNN-OSPG outperforms backward-induction and FQI methods on 15 of the 17 datasets.
We introduce a policy gradient algorithm well suited to learning RNN-based optimal stopping policies effective in non-Markovian settings. Our optimal stopping policy gradient (OSPG) algorithm leverages a new Bayesian net formulation of non-Markovian state-action-reward trajectories specific to optimal stopping. This formulation leads to an offline policy gradient algorithm where Bayes net inference eliminates inefficient Monte-Carlo policy rollouts. OSPG outperforms competing methods in non-Markovian settings where existing methods must either deal with the curse of dimensionality (from extended state spaces) or the curse of non-Markovianity (from recursive Markovian approximation). A limitation of OSPG is that since it uses a temporal loss defined over complete trajectories, it may not be suitable for scenarios with only a few extremely long trajectories or when trajectories are partially observed.
We acknowledge Jerry Liu, M. Anthony Lewis, Matt Ellis, Scott Hallworth, and Jennifer Hill at HP Inc. for supporting and sponsoring this work. Finally, we thank all the reviewers for their insightful comments that have helped improve the paper.
[1] Youssef Achenchabe, Alexis Bondu, Antoine" Cornuéjols, and Asma Dachraoui. Early classification of time series. In Machine Learning, volume 110, pages 1481-1504. Springer International Publishing, 2021.
[2] Sebastian Becker, Patrick Cheridito, and Arnulf Jentzen. Deep optimal stopping. J. Mach. Learn. Res., 20:74:1-74:25, 2019.
[3] R. Bellman and Rand Corporation. Dynamic Programming. Rand Corporation Research Study. Princeton University Press, 1959.
[4] D. Bertsekas and J.N. Tsitsiklis. Neuro-Dynamic Programming. Athena Scientific, 1996.
[5] Bruno Bouchard and Warin Xavier. Monte-Carlo Valuation of American Options: Facts and New Algorithms to Improve Existing Methods. Numerical Methods in Finance, pages 215-255, 2012.
[6] Siddharth Chandak, Pratik Shah, Vivek S Borkar, and Parth Dodhia. Reinforcement learning in non-markovian environments. CoRR, abs/2211.01595, 2023.
[7] Yangang Chen and Justin W. L. Wan. Deep neural network framework based on backward stochastic differential equations for pricing and hedging american options in high dimensions. Quantitative Finance, 21(1):45-67, 2021.
[8] Y.S. Chow, H. Robbins, and D. Siegmund. Great expectations: the theory of optimal stopping. Houghton Mifflin, 1971.
[9] Asma Dachraoui, Alexis Bondu, and Antoine Cornuéjols. Early classification of time series as a non myopic sequential decision making problem. In Machine Learning and Knowledge Discovery in Databases, pages 433-447. Springer International Publishing, 2015.
[10] Niranjan Damera Venkata and Chiranjib Bhattacharyya. When to intervene: Learning optimal intervention policies for critical events. In Advances in Neural Information Processing Systems, volume 35, pages 30114-30126, 2022.
[11] Hoang Anh Dau, Eamonn Keogh, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, Yanping, Bing Hu, Nurjahan Begum, Anthony Bagnall, Abdullah Mueen, Gustavo Batista, and Hexagon-ML. The UCR time series classification archive, October 2018. https://www.cs.ucr.edu/~eamonn/time_series_data_2018/.
[12] Peter Dayan and Geoffrey E. Hinton. Using expectation-maximization for reinforcement learning. In Neural Computation, volume 9, page 271-278, 1997.
[13] Don Dennis, Chirag Pabbaraju, Harsha Vardhan Simhadri, and Prateek Jain. Multiple instance learning for efficient sequential data classification on resource-constrained devices. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montréal, Canada., pages 10976-10987, 2018.
[14] T.S. Ferguson. Mathematical Statistics: A Decision Theoretic Approach. Probability and mathematical statistics. Academic Press, 1967.
[15] Tom Fischer. Stopping times are hitting times: a natural representation. Statistics & Probability Letters, 2011.
[16] Calypso Herrera, Florian Krach, Pierre Ruyssen, and Josef Teichmann. Optimal stopping via randomized neural networks. CoRR, abs/2104.13669, 2021.
[17] Daniel Jiang and Warren Powell. An approximate dynamic programming algorithm for monotone value functions. Operations Research, 63, 01 2014.
[40] John Tsitsiklis and Benjamin Van Roy. Regression Methods for Pricing Complex American-Style Options. IEEE Transactions on Neural Networks, 12(4):694–703, 2001.
[41] Steven D. Whitehead and Long-Ji Lin. Reinforcement learning of non-markov decision processes. Artificial Intelligence, 73(1):271-306, 1995.
[42] Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3-4):229-256, may 1992.
[43] Huizhen Yu and Dimitri P. Bertsekas. Q-Learning Algorithms for Optimal Stopping Based on Least Squares. European Control Conference, 2007.
Deep Recurrent Optimal Stopping: Supplementary Material
We provide an outline of the supplement that complements our paper.
Appendix A: provides proofs of all Theorems in the paper.
Appendix B: considers formulations of stopping problems specified in terms of costs instead of rewards and shows how to transform these into a consistent reward formulation.
Appendix C: provides discussion and pseudo-code to implement the temporal loss used to learn fitted Q-iteration policies for DNN-FQI and RNN-FQI
Appendix D: includes further details and discussion of the experiments, including model hyperparameter settings, numerical results with confidence intervals, details of compute environment, model sizes, train and inference times, etc. In Appendix E.3, we also include a new experiment that compares OSPG to state-of-the-art PDE benchmarks in the pricing of American options, which is a continuous-time optimal stopping problem.
Appendix E: includes a treatment of baseline subtraction in the context of optimal stopping policy gradients (OSPG).
A Appendix A: Proofs
Proof. The Lemma follows from the Bayes net trajectory model of Figure 1a, and the special structure of OS action trajectories:  $A_{\tau} = 1$  and  $A_{n} = 0, \forall n < \tau$ . For any OS state-action trajectory, we have:
$$
\mathbb {P} \left(\mathbf {A} _ {\tau}, \mathbf {S} _ {H}\right) = \underbrace {\mathbb {P} \left(\mathbf {S} _ {0}\right) \prod_ {j = 1} ^ {H} \mathbb {P} \left(\mathbf {S} _ {j} \mid \mathbf {S} _ {j - 1}\right)} _ {\mathbb {P} (\mathbf {S} _ {H})} \prod_ {n = 0} ^ {\tau} \mathbb {P} \left(A _ {n} \mid \mathbf {S} _ {n}\right) = \mathbb {P} \left(\mathbf {S} _ {H}\right) \prod_ {n = 0} ^ {\tau} \mathbb {P} \left(A _ {n} \mid \mathbf {S} _ {n}\right) \tag {15}
$$
Therefore, conditioning on the state trajectory, we have  $\mathbb{P}(\mathbf{A}_{\tau}|\mathbf{S}_H) = \prod_{n = 0}^{\tau}\mathbb{P}(A_n|\mathbf{S}_n)$ . Note that by the structure of finite-horizon OS trajectories  $\mathbf{A}_{\tau}$  is a sequence of continue actions terminated by a stop action at  $\tau$ . Thus there is a bijective mapping between stopping times and complete action trajectories given by:
$$
\kappa : \{0, 1, 2, \dots , H \} \mapsto \{1, 0 1, 0 0 1, \dots \underbrace {0 0 \cdots 0} _ {H - 1 0 s} 1 \} \tag {16}
$$
So  $\mathbb{P}(\mathbf{A}_{\tau}|\mathbf{S}_H) = \mathbb{P}(\kappa (\tau)|\mathbf{S}_H) = \mathbb{P}(\tau |\mathbf{S}_H)$ . Consider a trajectory stopping at  $\tau = j$  and recall that our stochastic stopping policy is defined as  $\phi_j(\mathbf{S}_j):\mathbb{P}(A_j = 1|\mathbf{S}_j)$ .
Thus, if  $j = 0$ :
$$
\mathbb {P} (\tau = 0 | \mathbf {S} _ {H}) = \mathbb {P} (\mathbf {A} _ {0} | \mathbf {S} _ {H}) = \mathbb {P} (A _ {0} = 1 | \mathbf {S} _ {0}) = \phi_ {0} (\mathbf {S} _ {0}) \tag {17}
$$
If  $0 <   j <   H$  ..
$$
\begin{array}{l} \mathbb {P} (\tau = j | \mathbf {S} _ {H}) = \mathbb {P} (A _ {0} = 0, \dots , A _ {j - 1} = 0, A _ {j} = 1 | \mathbf {S} _ {H}) \\ = \mathbb {P} (A _ {j} = 1 | \mathbf {S} _ {H}) \prod_ {n = 0} ^ {j - 1} \mathbb {P} (A _ {n} = 0 | \mathbf {S} _ {H}) \\ = \phi_ {j} \left(\mathbf {S} _ {j}\right) \prod_ {n = 0} ^ {j - 1} \left(1 - \phi_ {n} \left(\mathbf {S} _ {n}\right)\right) \tag {18} \\ \end{array}
$$
Finally, if  $j = H$ :
$$
\begin{array}{l} \mathbb {P} (\tau = H | \mathbf {S} _ {H}) = \mathbb {P} (A _ {0} = 0, \dots , A _ {H - 1} = 0, A _ {H} = 1 | \mathbf {S} _ {H}) \\ = \mathbb {P} (A _ {H} = 1 | \mathbf {S} _ {H}) \prod_ {n = 0} ^ {H - 1} \mathbb {P} (A _ {n} = 0 | \mathbf {S} _ {H}) \\ = \phi_ {H} (\mathbf {S} _ {H}) \prod_ {n = 0} ^ {H - 1} \left(1 - \phi_ {n} (\mathbf {S} _ {n})\right) \\ = \prod_ {n = 0} ^ {H - 1} \left(1 - \phi_ {n} \left(\mathbf {S} _ {n}\right)\right) \tag {19} \\ \end{array}
$$
where we have used the fact that in the finite horizon setting  $\phi_H(\mathbf{S}_H) \coloneqq 1$  by definition.
We may verify  $\sum_{j=0}^{H} \mathbb{P}(\tau = j|\mathbf{S}_H) = 1$  since:
$$
\begin{array}{l} 1 - \sum_ {j = 0} ^ {H - 1} \mathbb {P} (\tau = j | \mathbf {S} _ {H}) = 1 - \phi_ {0} (\mathbf {S} _ {0}) - \sum_ {j = 0} ^ {H - 1} \phi_ {j} (\mathbf {S} _ {j}) \prod_ {n = 0} ^ {j - 1} (1 - \phi_ {n} (\mathbf {S} _ {n})) \\ = \left(1 - \phi_ {0} (\mathbf {S} _ {0})\right) \left[ 1 - \phi_ {1} (\mathbf {S} _ {1}) - \sum_ {j = 1} ^ {H - 1} \phi_ {j} (\mathbf {S} _ {j}) \prod_ {n = 1} ^ {j - 1} (1 - \phi_ {n} (\mathbf {S} _ {n})) \right] \\ = \left(1 - \phi_ {0} (\mathbf {S} _ {0})\right) \left(1 - \phi_ {1} (\mathbf {S} _ {1})\right) \times \\ \left[ 1 - \phi_ {2} (\mathbf {S} _ {2}) - \sum_ {j = 2} ^ {H - 1} \phi_ {j} (\mathbf {S} _ {j}) \prod_ {n = 2} ^ {j - 1} (1 - \phi_ {n} (\mathbf {S} _ {n})) \right] \\ \begin{array}{c} \vdots \\ \vdots \\ \vdots \end{array} \\ = \prod_ {n = 0} ^ {H - 1} \left(1 - \phi_ {n} (\mathbf {S} _ {n})\right) = \mathbb {P} (\tau = H | \mathbf {S} _ {H}) \\ \end{array}
$$
Further, since  $\tau$  is a stopping time random variable we have:
$$
\mathbb {P} (\tau = j | \mathbf {S} _ {H}) = \mathbb {P} (\tau = j | \mathbf {S} _ {j}) := \psi_ {j} (\mathbf {S} _ {j}) \tag {20}
$$
This is because stopping time random variables have the property that  $\mathbb{1}(\tau = j)$  is a function of  $\mathbf{S}_j$ . So we can determine if  $\tau = j$  or not by only considering  $\mathbf{S}_j$  making the event  $\{\tau = j\}$  conditionally independent of  $S_{j + k},\forall k > 0$  given  $\mathbf{S}_j[35]$ . This completes the proof.
A.2 Proof of Theorem 3.1
Proof. We apply Jensen's inequality to the objective  $J_{\mathrm{WML}}(\pmb{\theta})$  to obtain a lower bound:
$$
J _ {\mathrm {W M L}} (\boldsymbol {\theta}) = \sum_ {i = 1} ^ {N} \tilde {w} _ {i} \log \sum_ {j = 0} ^ {H} \psi_ {j} ^ {\boldsymbol {\theta}} \left(\mathbf {s} _ {i j}\right) \tilde {r} _ {i j} \geq \sum_ {i = 1} ^ {N} \sum_ {j = 0} ^ {H} \tilde {w} _ {i} q _ {i j} \log \left[ \frac {\psi_ {j} ^ {\boldsymbol {\theta}} \left(\mathbf {s} _ {i j}\right) \tilde {r} _ {i j}}{q _ {i j}} \right] := J (\mathbf {Q}, \boldsymbol {\theta}) \tag {21}
$$
where  $\mathbf{Q} = [q_{ij}]$  is any row-stochastic matrix satisfying  $q_{ij} \geq 0, \forall i,j$  and  $\sum_{j=0}^{H} q_{ij} = 1$ . Starting with a given  $\pmb{\theta}^{(0)}$  the lower bound is maximized (Jensen's inequality becomes an equality) when  $\mathbf{Q} = \mathbf{Q}^{(0)} := [q_{ij}^{(0)}]$  such that:
$$
q _ {i j} ^ {(0)} = \frac {\psi_ {j} ^ {\boldsymbol {\theta} ^ {(0)}} (\mathbf {s} _ {i j}) \tilde {r} _ {i j}}{\sum_ {j = 0} ^ {H} \psi_ {j} ^ {\boldsymbol {\theta} ^ {(0)}} (\mathbf {s} _ {i j}) \tilde {r} _ {i j}} \tag {22}
$$
This is the E-step in Theorem 3.1. Note that the E-step does not change the objective, so  $J_{\mathrm{WML}}(\pmb{\theta}^{(0)}) = J(\mathbf{Q}^{(0)}, \pmb{\theta})$ . By ignoring terms that do not depend on  $\pmb{\theta}$ , maximizing  $J(\mathbf{Q}^{(0)}, \pmb{\theta})$  w.r.t.  $\pmb{\theta}$  can be seen as equivalent to maximizing
$$
J _ {M} (\boldsymbol {\theta}) = \sum_ {i = 1} ^ {N} \sum_ {j = 0} ^ {H} \tilde {w} _ {i} q _ {i j} \log \psi_ {j} ^ {\boldsymbol {\theta}} \left(\mathbf {s} _ {i j}\right) \tag {23}
$$
This is the M-step. By the lower bound established in equation (21) and since the M-step maximizes  $J(\mathbf{Q}^{(0)},\pmb {\theta})$  to obtain  $\pmb{\theta}^{(1)}$ , we have  $J_{\mathrm{WML}}(\pmb{\theta}^{(1)})\geq J(\mathbf{Q}^{(0)},\pmb{\theta}^{(1)})\geq J(\mathbf{Q}^{(0)},\pmb{\theta}^{(0)}) = J_{\mathrm{WML}}(\pmb{\theta}^{(0)})$ . Therefore a round of E-M results in either an increase or no change in the objective. Since the WML objective is upper-bounded by  $N\log H$ , the monotone increasing sequence  $J_{\mathrm{WML}}(\pmb{\theta}^{(k)})$  converges to a local maximum of the objective.
A.3 Proof of Corollary 3.1.1
Proof. It suffices to show that the W-step also increases the objective  $J_{\mathrm{WML}}(\tilde{\mathbf{w}}, \boldsymbol{\theta})$ .
We maximize  $J_{\mathrm{WML}}(\tilde{\mathbf{w}}, \boldsymbol{\theta})$  w.r.t.  $\tilde{\mathbf{w}}$ , subject to the constraint  $\sum_{i=1}^{N} \tilde{w}_i = 1$ , resulting the following Lagrangian.
$$
\mathcal {L} (\tilde {\mathbf {w}}, \lambda) = \sum_ {i = 1} ^ {N} \tilde {w} _ {i} \log \sum_ {j = 0} ^ {H} \psi_ {j} ^ {\boldsymbol {\theta}} \left(\mathbf {s} _ {i j}\right) \tilde {r} _ {i j} - \sum_ {i = 1} ^ {N} \tilde {w} _ {i} \log \frac {\tilde {w} _ {i}}{\tilde {r} _ {i}} + \lambda \left(1 - \sum_ {i = 1} ^ {N} \tilde {w} _ {i}\right) \tag {24}
$$
Taking partial derivative w.r.t.  $\tilde{w}_i$  and  $\lambda$  and setting to zero, we have:
$$
\frac {\partial \mathcal {L} (\tilde {\mathbf {w}} , \lambda)}{\partial \tilde {w} _ {i}} = \log \sum_ {j = 0} ^ {H} \psi_ {j} ^ {\boldsymbol {\theta}} \left(\mathbf {s} _ {i j}\right) \tilde {r} _ {i j} + \log \tilde {r} _ {i} - (1 + \log \tilde {w} _ {i}) - \lambda = 0 \tag {25}
$$
$$
\frac {\partial \mathcal {L} (\tilde {\mathbf {w}} , \lambda)}{\partial \lambda} = \left(1 - \sum_ {i = 1} ^ {N} \tilde {w} _ {i}\right) = 0 \tag {26}
$$
Noting that  $\tilde{r}_i\coloneqq \frac{\sum_{j = 0}^Hr_{ij}}{\sum_{i = 1}^N\sum_{j = 0}^Hr_{ij}}$  and  $\tilde{r}_{ij}\coloneqq \frac{r_{ij}}{\sum_{j = 0}^{H}r_{ij}}$  and simplifying, we have:
$$
1 + \lambda = \log \left[ \frac {\sum_ {j = 0} ^ {H} \psi_ {j} ^ {\boldsymbol {\theta}} (\mathbf {s} _ {i j}) r _ {i j}}{r \tilde {w} _ {i}} \right] \tag {27}
$$
where  $r = \sum_{i=1}^{N} \sum_{j=0}^{H} r_{ij}$ . Exponentiation of both sides and cross-multiplication results in:
$$
\left[ r \exp (1 + \lambda) \right] \tilde {w} _ {i} = \sum_ {j = 0} ^ {H} \psi_ {j} ^ {\boldsymbol {\theta}} \left(\mathbf {s} _ {i j}\right) r _ {i j} \tag {28}
$$
Summing over  $i$ , we have:
$$
r \exp (1 + \lambda) = \sum_ {i = 1} ^ {N} \sum_ {j = 0} ^ {H} \psi_ {j} ^ {\boldsymbol {\theta}} (\mathbf {s} _ {i j}) r _ {i j} \tag {29}
$$
Substituting back in equation (28), we finally have:
$$
\tilde {w} _ {i} ^ {*} = \frac {\sum_ {j = 0} ^ {H} \psi_ {j} ^ {\boldsymbol {\theta}} \left(\mathbf {s} _ {i j}\right) r _ {i j}}{\sum_ {i = 1} ^ {N} \sum_ {j = 0} ^ {H} \psi_ {j} ^ {\boldsymbol {\theta}} \left(\mathbf {s} _ {i j}\right) r _ {i j}} \tag {30}
$$
Thus the maximizing  $\tilde{\mathbf{w}}^*$  is exactly the W-step. Therefore, the form of the W-step ensures  $J_{\mathrm{WML}}(\tilde{\mathbf{w}}^{(k)},\pmb{\theta}^{(k)})\geq J_{\mathrm{WML}}(\tilde{\mathbf{w}}^{(k - 1)},\pmb{\theta}^{(k)})\geq J_{\mathrm{WML}}(\tilde{\mathbf{w}}^{(k - 1)},\pmb{\theta}^{(k - 1)})$ . Thus we have a monotone increasing and bounded sequence  $J_{\mathrm{WML}}(\tilde{\mathbf{w}}^{(k)},\pmb{\theta}^{(k)})$  converging to a local maximum of  $J_{\mathrm{WML}}(\tilde{\mathbf{w}},\pmb {\theta})$ .
A.4 Proof of Theorem 4.1
Proof. We use the log-derivative trick [42].  $\nabla_{\pmb{\theta}}\psi_{j}^{\pmb{\theta}}(\mathbf{s}_{j}) = \psi_{j}^{\pmb{\theta}}(\mathbf{s}_{j})\nabla_{\pmb{\theta}}\log \psi_{j}^{\pmb{\theta}}(\mathbf{s}_{j})$  . So:
$$
\nabla_ {\boldsymbol {\theta}} J _ {O S} (\boldsymbol {\theta}) = \mathbb {E} _ {\mathbf {s} _ {H} \sim \mathbb {P} (\mathbf {s} _ {H})} \left[ \sum_ {j = 0} ^ {H} r _ {j} \nabla_ {\boldsymbol {\theta}} \psi_ {j} ^ {\boldsymbol {\theta}} (\mathbf {s} _ {j}) \right] = \mathbb {E} _ {\mathbf {s} _ {H} \sim \mathbb {P} (\mathbf {s} _ {H})} \left[ \sum_ {j = 0} ^ {H} r _ {j} \psi_ {j} ^ {\boldsymbol {\theta}} (\mathbf {s} _ {j}) \nabla_ {\boldsymbol {\theta}} \log \psi_ {j} ^ {\boldsymbol {\theta}} (\mathbf {s} _ {j}) \right] (3 1)
$$
A.5 Proof of Proposition 4.1
Proof. First, for a given  $\theta^{(k)}$ , substituting for  $q_{ij}^{(k)}$  from Theorem 3.1 and  $\tilde{w}_i^{(k)}$  from the W-step yields:
$$
\begin{array}{l} \tilde {w} _ {i} ^ {(k)} q _ {i j} ^ {(k)} = \left[ \frac {\sum_ {n = 0} ^ {H} \psi_ {n} ^ {\boldsymbol {\theta} ^ {(k)}} (\mathbf {s} _ {i n}) r _ {i n}}{\sum_ {m = 1} ^ {N} \sum_ {n = 0} ^ {H} \psi_ {n} ^ {\boldsymbol {\theta} ^ {(k)}} (\mathbf {s} _ {m n}) r _ {m n}} \right] \left[ \frac {\psi_ {j} ^ {\boldsymbol {\theta} ^ {(k)}} (\mathbf {s} _ {i j}) \tilde {r} _ {i j}}{\sum_ {n = 0} ^ {H} \psi_ {n} ^ {\boldsymbol {\theta} ^ {(k)}} (\mathbf {s} _ {i n}) \tilde {r} _ {i n}} \right] (32) \\ = \left(\sum_ {n = 0} ^ {H} r _ {i n}\right) \psi_ {j} ^ {\boldsymbol {\theta} ^ {(k)}} \left(\mathbf {s} _ {i j}\right) \tilde {r} _ {i j} \left[ \frac {1}{\sum_ {m = 1} ^ {N} \sum_ {n = 0} ^ {H} \psi_ {n} ^ {\boldsymbol {\theta} ^ {(k)}} \left(\mathbf {s} _ {m n}\right) r _ {m n}} \right] (33) \\ = \underbrace {\left[ \psi_ {j} ^ {\boldsymbol {\theta} ^ {(k)}} \left(\mathbf {s} _ {i j}\right) r _ {i j} \right]} _ {v _ {i j} ^ {(k)}} \underbrace {\left[ \frac {1}{\sum_ {m = 1} ^ {N} \sum_ {n = 0} ^ {H} \psi_ {n} ^ {\boldsymbol {\theta} ^ {(k)}} \left(\mathbf {s} _ {m n}\right) r _ {m n}} \right]} _ {z ^ {(k)}} (34) \\ \end{array}
$$
Therefore, we may write the M-step objective as:
$$
J _ {M} ^ {(k)} (\boldsymbol {\theta}) = z ^ {(k)} \sum_ {i = 1} ^ {N} \sum_ {j = 0} ^ {H} v _ {i j} ^ {(k)} \log \psi_ {j} ^ {\boldsymbol {\theta}} \left(\mathbf {s} _ {i j}\right) \tag {35}
$$
$$
\propto \frac {1}{N} \sum_ {i = 1} ^ {N} \sum_ {j = 0} ^ {H} v _ {i j} ^ {(k)} \log \psi_ {j} ^ {\boldsymbol {\theta}} (\mathbf {s} _ {i j}) = \bar {J} _ {M} ^ {(k)} (\boldsymbol {\theta}) \tag {36}
$$
Dropping the iteration index  $(k)$ , by defining constants  $v_{ij} \coloneqq \psi_j^\theta(\mathbf{s}_{ij})r_{ij}$  calculated with the most recent value of  $\pmb{\theta}$ , the M-step objective is equivalent to the following objective:
$$
\bar {J} _ {M} (\boldsymbol {\theta}) = \frac {1}{N} \sum_ {i = 1} ^ {N} \sum_ {j = 0} ^ {H} v _ {i j} \log \psi_ {j} ^ {\boldsymbol {\theta}} \left(\mathbf {s} _ {i j}\right) \tag {37}
$$
where  $v_{ij} \coloneqq \psi_j^\theta(\mathbf{s}_{ij}) r_{ij}$  is calculated with the most recent value of  $\pmb{\theta}$  and held constant. Taking the gradient w.r.t  $\pmb{\theta}$  we have
$$
\nabla_ {\boldsymbol {\theta}} \bar {J} _ {M} (\boldsymbol {\theta}) = \frac {1}{N} \sum_ {i = 1} ^ {N} \sum_ {j = 0} ^ {H} v _ {i j} \nabla_ {\boldsymbol {\theta}} \log \psi_ {j} ^ {\boldsymbol {\theta}} \left(\mathbf {s} _ {i j}\right) \tag {38}
$$
Now, substituting for  $v_{ij}$ , we have:
$$
\nabla_ {\boldsymbol {\theta}} \bar {J} _ {M} (\boldsymbol {\theta}) = \frac {1}{N} \sum_ {i = 1} ^ {N} \sum_ {j = 0} ^ {H} r _ {i j} \psi_ {j} ^ {\boldsymbol {\theta}} \left(\mathbf {s} _ {i j}\right) \nabla_ {\boldsymbol {\theta}} \log \psi_ {j} ^ {\boldsymbol {\theta}} \left(\mathbf {s} _ {i j}\right) \tag {39}
$$
This can be expressed as an expectation over sample trajectories as:
$$
\nabla_ {\pmb {\theta}} \bar {J} _ {M} (\pmb {\theta}) = \mathbb {E} _ {\mathbf {s} _ {H} \sim \mathbb {P} (\mathbf {s} _ {H})} \left[ \sum_ {j = 0} ^ {H} r _ {j} \psi_ {j} ^ {\pmb {\theta}} (\mathbf {s} _ {j}) \nabla_ {\pmb {\theta}} \log \psi_ {j} ^ {\pmb {\theta}} (\mathbf {s} _ {j}) \right] = \nabla_ {\pmb {\theta}} J _ {O S} (\pmb {\theta})
$$
Thus the gradient of the transformed M-step objective is identical to the optimal stopping policy gradient (OSPG). Therefore, if we perform the E-step, W-step and a single gradient update, the sequence of policy parameters  $\theta_{n}$  will exactly correspond to the updated OSPG policy parameters. We may therefore appeal to literature on incremental partial M-step E-M algorithms [27] and gradient descent [21] to conclude that for small enough step-size, that increases  $J_{M}(\pmb{\theta})$ , the policy updates converge to a local maximum of both  $J_{WML}(\tilde{\mathbf{w}},\pmb{\theta})$  and  $J_{OS}(\pmb{\theta})$ .
A.6 Proof of Corollary 4.1.1
Proof. From the proof of Proposition 4.1, the M-step objective is equivalent to the following objective:
$$
\bar {J} _ {M} (\boldsymbol {\theta}) = \frac {1}{N} \sum_ {i = 1} ^ {N} \sum_ {j = 0} ^ {H} v _ {i j} \log \psi_ {j} ^ {\boldsymbol {\theta}} \left(\mathbf {s} _ {i j}\right) \tag {40}
$$
Now, substituting for  $\psi_j^\theta (\mathbf{s}_{ij})$  in terms of the stopping policy using the trajectory reparameterization lemma (Lemma 3.1):
$$
\begin{array}{l} J _ {M} (\boldsymbol {\theta}) \propto \bar {J} _ {M} (\boldsymbol {\theta}) = \frac {1}{N} \sum_ {i = 1} ^ {N} v _ {i 0} \log \phi_ {0} ^ {\boldsymbol {\theta}} (\mathbf {s} _ {i 0}) + \sum_ {j = 1} ^ {H - 1} v _ {i j} \left[ \log \phi_ {j} ^ {\boldsymbol {\theta}} (\mathbf {s} _ {i j}) + \sum_ {n = 0} ^ {j - 1} \log (1 - \phi_ {n} ^ {\boldsymbol {\theta}} (\mathbf {s} _ {i n})) \right] \\ + v _ {i H} \sum_ {n = 0} ^ {H - 1} \log \left(1 - \phi_ {n} ^ {\boldsymbol {\theta}} (\mathbf {s} _ {i n})\right) \\ = \frac {1}{N} \sum_ {i = 1} ^ {N} \left[ \sum_ {j = 0} ^ {H - 1} v _ {i j} \log \phi_ {j} ^ {\boldsymbol {\theta}} (\mathbf {s} _ {i j}) \right] \\ + v _ {i 1} \log (1 - \phi_ {0} ^ {\boldsymbol {\theta}} (\mathbf {s} _ {i 0})) \\ + v _ {i 2} \left[ \log \left(1 - \phi_ {0} ^ {\boldsymbol {\theta}} (\mathbf {s} _ {i 0})\right) + \log \left(1 - \phi_ {1} ^ {\boldsymbol {\theta}} (\mathbf {s} _ {i 1})\right) \right] \\ + v _ {i 3} \left[ \log \left(1 - \phi_ {0} ^ {\boldsymbol {\theta}} (\mathbf {s} _ {i 0})\right) + \log \left(1 - \phi_ {1} ^ {\boldsymbol {\theta}} (\mathbf {s} _ {i 1})\right) + \log \left(1 - \phi_ {2} ^ {\boldsymbol {\theta}} (\mathbf {s} _ {i 2})\right) \right] \dots \\ + v _ {i H} \sum_ {n = 0} ^ {H - 1} \log \left(1 - \phi_ {n} ^ {\boldsymbol {\theta}} (\mathbf {s} _ {i n})\right) \\ = \frac {1}{N} \sum_ {i = 1} ^ {N} \left[ \sum_ {j = 0} ^ {H - 1} v _ {i j} \log \phi_ {j} ^ {\boldsymbol {\theta}} (\mathbf {s} _ {i j}) \right] + \left[ \sum_ {j = 0} ^ {H - 1} \log (1 - \phi_ {j} ^ {\boldsymbol {\theta}} (\mathbf {s} _ {i j})) \left\{\sum_ {n = j + 1} ^ {H} v _ {i n} \right\} \right] \\ = \frac {1}{N} \sum_ {i = 1} ^ {N} \sum_ {j = 0} ^ {H - 1} v _ {i j} \log \phi_ {j} ^ {\boldsymbol {\theta}} (\mathbf {s} _ {i j}) + \log (1 - \phi_ {j} ^ {\boldsymbol {\theta}} (\mathbf {s} _ {i j})) \left[ \sum_ {n = j + 1} ^ {H} v _ {i n} \right] \\ \end{array}
$$
Since by Proposition 4.1, we have  $\nabla_{\pmb{\theta}}J_{OS}(\pmb{\theta}) = \nabla_{\pmb{\theta}}\bar{J}_M(\pmb{\theta})$ . Setting  $k_{ij} \coloneqq \left[\sum_{n=j+1}^{H} v_{in}\right]$  in the above expression for  $\bar{J}_M(\pmb{\theta})$  and taking the gradient, we have:
$$
\nabla_ {\boldsymbol {\theta}} J _ {O S} (\boldsymbol {\theta}) = \frac {1}{N} \sum_ {i = 1} ^ {N} \sum_ {j = 0} ^ {H} \left[ \frac {v _ {i j} \left(1 - \phi_ {j} ^ {\boldsymbol {\theta}} \left(\mathbf {s} _ {i j}\right)\right) - k _ {i j} \phi_ {j} ^ {\boldsymbol {\theta}} \left(\mathbf {s} _ {i j}\right)}{\phi_ {j} ^ {\boldsymbol {\theta}} \left(\mathbf {s} _ {i j}\right) \left(1 - \phi_ {j} ^ {\boldsymbol {\theta}} \left(\mathbf {s} _ {i j}\right)\right)} \right] \nabla_ {\boldsymbol {\theta}} \phi_ {j} ^ {\boldsymbol {\theta}} \left(\mathbf {s} _ {i j}\right) \tag {41}
$$
This completes the proof of Corollary 4.1.1.
B Dealing with costs instead of rewards