SlowGuess/NeurIPS_2023_OCR huggingface dataset:
<sup>3</sup>We use the XOR operator since exactly one of the  $\{Z_j\}$  can be equal to one. The reward for a trajectory can only be claimed by a stop action at a single time-step.
However, maximizing this likelihood over samples while allowing optimal stopping decisions on any trajectory ignores the relative magnitude of rewards between different trajectories. Thus, we seek to weight the likelihood of each trajectory. Therefore, considering a dataset  $\mathcal{D}$  of environment trajectories and using index  $i$  for the  $i^{th}$  trajectory in the sample, we have:
$$
l _ {\mathrm {w m l}} (\boldsymbol {\theta}) = \prod_ {i = 1} ^ {N} l (\boldsymbol {\theta} \mid \mathbf {s} _ {i, H}, \mathbf {r} _ {i, H}) ^ {\tilde {w} _ {i}} \quad \text {w i t h} \quad \sum_ {i = 1} ^ {N} \tilde {w} _ {i} = 1 \tag {8}
$$
where  $\tilde{w}_i$  is the relative weight for the  $i^{th}$  sample. Taking logarithms, we have the following objective to maximize:
$$
J _ {\mathrm {W M L}} (\boldsymbol {\theta}) = \sum_ {i = 1} ^ {N} \tilde {w} _ {i} \log \sum_ {j = 0} ^ {H} \psi_ {j} ^ {\boldsymbol {\theta}} \left(\mathbf {s} _ {i j}\right) \tilde {r} _ {i j} \tag {9}
$$
We may use the following Expectation-Maximization (E-M) procedure to maximize this objective:
Theorem 3.1 (E-M procedure for optimal stopping policy). The following iterative procedure converges to a local maximum of  $J_{WML}(\pmb{\theta})$  with monotonic increase in the objective with iteration index  $(k)$ , starting with an initial value  $\pmb{\theta} = \pmb{\theta}^{(0)}$ :
$$
q _ {i j} ^ {(k)} = \frac {\psi_ {j} ^ {\boldsymbol {\theta} ^ {(k)}} (\mathbf {s} _ {i j}) \tilde {r} _ {i j}}{\sum_ {j = 0} ^ {H} \psi_ {j} ^ {\boldsymbol {\theta} ^ {(k)}} (\mathbf {s} _ {i j}) \tilde {r} _ {i j}} \quad 0 \leq j \leq H, 1 \leq i \leq N \tag {E-step}
$$
$$
J _ {M} ^ {(k)} (\boldsymbol {\theta}) = \sum_ {i = 1} ^ {N} \sum_ {j = 0} ^ {H} \tilde {w} _ {i} ^ {(k)} q _ {i j} ^ {(k)} \log \psi_ {j} ^ {\boldsymbol {\theta}} \left(\mathbf {s} _ {i j}\right), \boldsymbol {\theta} ^ {(k + 1)} = \underset {\boldsymbol {\theta}} {\arg \max } J _ {M} ^ {(k)} (\boldsymbol {\theta}) \tag {M-step}
$$
where  $\tilde{w}_i^{(k)} = \tilde{w}_i, \forall k$ . The proof follows from an application of Jensen's inequality and is given in Appendix A. Dayan and Hinton pioneered using E-M methods for RL policy search [12].
Specification of  $\{\tilde{r}_{ij}\}$  and  $\tilde{\mathbf{w}}$ :  $\tilde{r}_{ij}$  is a specification of the relative intra-trajectory probability of obtaining a reward when stopping at time-step  $j$  of the  $i^{th}$  trajectory. A natural choice (and one that we adopt in this paper) is to set  $\tilde{r}_{ij} := \frac{r_{ij}}{\sum_{j=0}^{H} r_{ij}}$ , where  $r_{ij}$  is a realization of  $R_j = g_j(\mathbf{S}_j)$  corresponding to the  $i^{th}$  trajectory. This formulation encourages stopping at time-steps of the trajectory, proportional to achievable reward. Alternatively, formulations such as  $\tilde{r}_{ij} := \frac{r_{ij}}{\max_k r_{ik}}$  or  $\tilde{r}_{ij} := \frac{\exp(r_{ij})}{\sum_{k=0}^{H} \exp(r_{ik})}$  could be used to express different stopping preferences within a trajectory. The weights  $\tilde{\mathbf{w}}$ , on the other hand, express relative inter-trajectory importance. One plausible choice would be to set  $\tilde{w}_i = \tilde{r}_i = \frac{\sum_{j=0}^{H} r_{ij}}{\sum_{i=1}^{N} \sum_{j=0}^{H} r_{ij}}$ , the relative reward available for extraction from a trajectory. Similarly, one could weight trajectories by the maximum reward achievable:  $\tilde{w}_i = \tilde{r}_i = \frac{\max_j r_{ij}}{\sum_i \max_j r_{ij}}$ . We may also consider reweighting schemes that vary weights over rounds of the E-M algorithm. In the RL literature, Reward-weighted regression [30] takes such an approach. Reward-weighted regression typically conducts rollouts of the current policy and weights each resulting sample trajectory with the relative reward achieved by the current policy on that trajectory. However, the trajectory reward is unknown since we do not perform policy rollouts. Instead, we weight each trajectory offline by its relative expected reward under the current policy. Specifically, we compute weights at each round according to a W-step:
$$
\tilde {w} _ {i} ^ {(k)} = \frac {\sum_ {j = 0} ^ {H} \psi_ {j} ^ {\boldsymbol {\theta} ^ {(k)}} (\mathbf {s} _ {i j}) r _ {i j}}{\sum_ {i = 1} ^ {N} \sum_ {j = 0} ^ {H} \psi_ {j} ^ {\boldsymbol {\theta} ^ {(k)}} (\mathbf {s} _ {i j}) r _ {i j}} \quad 1 \leq i \leq N \tag {W-step}
$$
This weighting is not arbitrary as it leads to a surprising equivalence with a policy gradient method, established in the next section. Further, we have the following Corollary (proof in Appendix A):
Corollary 3.1.1 (Reweighted E-M). Let  $\tilde{r}_i = \frac{\sum_{j=0}^H r_{ij}}{\sum_{i=1}^N \sum_{j=0}^H r_{ij}}$ . Using the reweighting scheme of the W-step in the  $E-M$  procedure of Theorem 3.1 achieves a local maximum of:
$$
J _ {W M L} (\tilde {\mathbf {w}}, \boldsymbol {\theta}) = \sum_ {i = 1} ^ {N} \tilde {w} _ {i} \log \sum_ {j = 0} ^ {H} \psi_ {j} ^ {\boldsymbol {\theta}} \left(\mathbf {s} _ {i j}\right) \tilde {r} _ {i j} - \sum_ {i = 1} ^ {N} \tilde {w} _ {i} \log \frac {\tilde {w} _ {i}}{\tilde {r} _ {i}} \tag {10}
$$
with monotonic increase in the objective, starting with an initial value  $\pmb{\theta} = \pmb{\theta}^{(0)}$ . The second term is the KL-divergence between distributions  $\tilde{\mathbf{w}}$  and  $\tilde{\mathbf{r}}$ .
4 Optimal stopping policy gradients (OSPG)
In this section we show that an incremental version of the E-M algorithm described in the previous section, with a single gradient step replacing a full M-step, is equivalent to a policy gradient method that maximizes expected reward. We call this the optimal stopping policy gradient (OSPG) method.
Consider a direct maximization of the optimal stopping objective  $J_{OS}(\pmb{\theta}) = \mathbb{E}[R_{\tau}]$  from definition 2.4 where the expectation is over state-action trajectories. We can use our Bayes net model of state-action trajectories without reward augmentation shown in Figure 1a to obtain:
$$
\mathbb {P} \left(\mathbf {S} _ {H}, \mathbf {A} _ {\tau} \mid \boldsymbol {\theta}\right) = \mathbb {P} \left(\mathbf {S} _ {0}\right) \prod_ {j = 1} ^ {H} \mathbb {P} \left(\mathbf {S} _ {j} \mid \mathbf {S} _ {j - 1}\right) \prod_ {n = 0} ^ {\tau} \mathbb {P} \left(A _ {n} \mid \mathbf {s} _ {n}, \boldsymbol {\theta}\right) = \mathbb {P} \left(\mathbf {S} _ {H}\right) \mathbb {P} (\tau | \mathbf {S} _ {H}, \boldsymbol {\theta}) \tag {11}
$$
where we have used the trajectory reparameterization lemma to reparameterize in terms of stopping times. Therefore, we may write:
$$
J _ {O S} (\boldsymbol {\theta}) = \mathbb {E} _ {\mathbf {s} _ {H} \sim \mathbb {P} (\mathbf {s} _ {H})} \left[ \mathbb {E} _ {\tau \sim \mathbb {P} (\tau | \mathbf {s} _ {H}, \boldsymbol {\theta})} \left[ R _ {\tau} \right] \right] = \mathbb {E} _ {\mathbf {s} _ {H} \sim \mathbb {P} (\mathbf {s} _ {H})} \left[ \sum_ {j = 0} ^ {H} \psi_ {j} ^ {\boldsymbol {\theta}} (\mathbf {s} _ {j}) r _ {j} \right] \tag {12}
$$
We leverage methods used to establish the policy gradient theorem [42, 38] to derive a convenient form for the gradient of the optimal stopping objective w.r.t policy parameters  $\theta$ .
Theorem 4.1 (Optimal stopping policy gradient theorem). For any differentiable stopping policy  $\phi^{\theta}$ , The gradient of the objective  $J_{OS}(\pmb {\theta}) = \mathbb{E}[R_{\tau}]$  w.r.t. policy parameters  $\pmb{\theta}$ , where  $\{R_j\}$  is a risk process and  $\tau$  is a stopping-time random variable, is given by:
$$
\nabla_ {\pmb {\theta}} J _ {O S} (\pmb {\theta}) = \mathbb {E} _ {\mathbf {s} _ {H} \sim \mathbb {P} (\mathbf {s} _ {H})} \left[ \sum_ {j = 0} ^ {H} r _ {j} \psi_ {j} ^ {\pmb {\theta}} (\mathbf {s} _ {j}) \nabla_ {\pmb {\theta}} \log \psi_ {j} ^ {\pmb {\theta}} (\mathbf {s} _ {j}) \right]
$$
where the  $\psi_j^\theta (\mathbf{s}_j)$  are functions of  $\phi^{\theta}$  obtained by Lemma 3.1.
The proof is an application of the log-derivative trick [42] and is given in Appendix A. Note that optimal stopping policy gradients are estimated using offline state trajectories only. Unlike the general RL case, we never need to do policy rollouts to sample trajectories. Instead, we explicitly calculate the probability distribution over all possible stopping actions (or stopping times) and use this to average over all possible stop-continue decisions. Also, an update to the policy parameters can be done without fresh data collection since we can correctly (due to explicit Bayes net modeling) model the impact of the updated policy on the loss using the already sampled environment trajectories. Appendix B describes specialization to settings with costs instead of rewards.
We now establish a surprising equivalence between the reweighted E-M algorithm of Corollary 3.1.1 and the policy gradient method derived in this section when an incremental E-M solution approach [27] is used. Although the objective functions  $J_{WML}(\tilde{\mathbf{w}},\boldsymbol {\theta})$  and  $J_{OS}(\pmb {\theta})$  are different in general, the same procedure maximizes both.
Proposition 4.1 (Incremental E-M as a policy gradient method). If a partial M-step is used in the reweighted E-M approach of Corollary 3.1.1, comprising of a single gradient-step, then the resulting incremental E-M procedure is equivalent to the optimal stopping policy gradient method of Theorem 4.1, each converging to a local maximum of both  $J_{WML}(\tilde{\mathbf{w}}, \boldsymbol{\theta})$  and  $J_{OS}(\boldsymbol{\theta})$ .
The proof is given in Appendix A. Thus, we may view the incremental E-M approach that optimizes a weighted maximum likelihood objective as a generalization of the policy gradient method that seeks to optimize the expected reward objective. The two are equivalent for a particular weighting scheme and Bayes net specification (as in Corollary 3.1.1). Further, the OSPG gradient has the following insightful form, obtained by applying Lemma 3.1 to the M-step objective of Theorem 3.1.
Corollary 4.1.1 (Value form of the OSPG gradient). Let  $v_{ij} \coloneqq \psi_j^\theta(\mathbf{s}_{ij}) r_{ij}$ ,  $k_{ij} \coloneqq \left[ \sum_{n=j+1}^{H} v_{in} \right]$ .
We may rewrite the policy gradient of Theorem 4.1 as:
$$
\nabla_ {\boldsymbol {\theta}} J _ {O S} (\boldsymbol {\theta}) = \frac {1}{N} \sum_ {i = 1} ^ {N} \sum_ {j = 0} ^ {H} \left[ \frac {v _ {i j} \left(1 - \phi_ {j} ^ {\boldsymbol {\theta}} \left(\mathbf {s} _ {i j}\right)\right) - k _ {i j} \phi_ {j} ^ {\boldsymbol {\theta}} \left(\mathbf {s} _ {i j}\right)}{\phi_ {j} ^ {\boldsymbol {\theta}} \left(\mathbf {s} _ {i j}\right) \left(1 - \phi_ {j} ^ {\boldsymbol {\theta}} \left(\mathbf {s} _ {i j}\right)\right)} \right] \nabla_ {\boldsymbol {\theta}} \phi_ {j} ^ {\boldsymbol {\theta}} \left(\mathbf {s} _ {i j}\right) \tag {13}
$$
The proof is given in Appendix A.  $v_{ij}$  is an empirical estimate of immediate reward, while  $k_{ij}$  is an empirical estimate of continuation value under the current policy. The term in the bracket is the derivative of the objective w.r.t. the policy output, evaluated at  $\mathbf{s}_{ij}$ . This is positive, and hence calls for increasing stopping probability, if  $\frac{v_{ij}}{k_{ij}} > \frac{\phi_j^\theta(\mathbf{s}_{ij})}{1 - \phi_j^\theta(\mathbf{s}_{ij})}$ , i.e. when the ratio of immediate to continuation value exceeds the current odds of stopping. Of course, the policy is updated considering expected immediate reward and continuation values at all time-steps. The updated policy produces new estimates of immediate reward and continuation value from which a new OSPG gradient is computed, and process proceeds until convergence to an optimal policy. Thus, this process implicitly uses value functions at all time-steps to update the policy which implicitly further improves value functions. Since there is no recursion across time-steps OSPG is able to mitigate the curse of non-Markovianity.
Temporal loss functions for learning RNN stopping policies: We may directly adopt the M-step objective of Theorem 3.1 as a loss function for learning an RNN-based policy with mini-batch stochastic gradient-descent (SGD), effective in non-Markovian settings. Algorithm 1 shows the pseudocode for computing our OSPG loss. This loss is ideally suited for learning stopping policies with RNNs since it takes as input stopping probabilities at every time-step  $\phi_j^\theta (\mathbf{s}_{ij})$  that are each a function of process observation up to that point. An RNN performs this mapping naturally since it maps input history to a hidden state at each time-step with recurrent computation. So, for example, return_sequences=True, coupled with a TimeDistributed Dense layer in Keras, returns a complete trajectory of RNN predictions for input into the loss. Note also that this is a temporal loss that considers the entire process trajectory and does not decompose across time-steps.
Algorithm 1 Pseudocode for mini-batch computation of our temporal OSPG loss
Input:  $\mathbf{R} := [r_{ij}]$ ,  $\Phi := [\phi_j^\theta (\mathbf{s}_{ij})]$ $\{1\leq i\leq N_b,0\leq j\leq H,N_b$  is batch size}   
 $\Psi_{:,0} = \Phi_{:,0}$ $\Psi_{:,H} = \exp \left(\sum_{j = 0}^{H - 1}\log (1 - \Phi_{:,j})\right)$  {numerically stable computation of  $\psi_j^\theta (\mathbf{s}_{ij})\}$ $\Psi_{:,1:H - 1} = \exp \left(\log (1 - \Phi_{:,H - 2})\mathbf{U} + \log \Phi_{:,1:H - 1}\right)$  {U is unit upper triangular, CUMSUM}   
 $\mathbf{V} =$  stop-gradient(R  $\odot$ $\Psi$  ) {held constant, so no gradient.  $\odot$  denotes Schur product}   
Output:  $J = -\frac{1}{N_b}\mathbf{1}^T[\mathbf{V}\odot \log \Psi]\mathbf{1}$
We compare our OSPG algorithm using deep neural networks (DNN-OSPG) and recurrent neural networks (RNN-OSPG) against the following model-free discrete-time optimal stopping approaches.
Backward Induction: Deep Optimal Stopping (DOS) [2] including an extension of state space with process history (DOS-ES) [2] and Randomized Recurrent Least Square Monte Carlo (RRLSM) [16].
Fitted Q-iteration: While fitted Q-Iteration (FQI) approaches in the optimal stopping literature rely on linear approximation with chosen basis functions [39, 40] break down in high-dimensional settings, they may be readily extended to leverage deep neural network (DNN-FQI) and recurrent neural network (RNN-FQI) function approximators. Appendix C provides details of our corresponding DNN-FQI and RNN-FQI baselines.
We conduct experiments with both Markovian and non-Markovian datasets. Markovian data include the pricing of high-dimensional Bermudan max-call options and American geometric-call options (relegated to Appendix D due to space). Non-Markovian data includes stopping a fractional Brownian motion and the early stopping of a sequential multi-class classifier. Except for the American option pricing experiment, all DNN function approximators use two hidden layers of 20 hidden units each, while all RNN function approximators leverage an RNN/GRU with a single hidden layer of 20 units. Backward induction methods (DOS, RRLSM) require independent networks at each time-step, while other approaches share network parameters across time-steps. Details of hyper-parameter settings, model size, and compute times are provided in Appendix D.
Exercise of a Bermudan max-call option: In this Markovian setting, we seek optimal strategies to maximize the returns of a Bermudan max-call option, a scenario described in [2]. Bermudan options may be exercised only at fixed times, unlike American options which may be exercised at any time prior to option expiration. The payoff of these options depends on the maximum of  $d$  underlying assets with multi-dimensional Black-Scholes dynamics and the strike price  $K$ . The dynamics and
Table 1: Bermudan max-call option pricing: Results
d	p0	average return (standard deviation)
DOS	DNN-FQI	DNN-OSPG	RNN-FQI	RNN-OSPG	RRLSM
20	90	37.08 (0.09)	36.98 (0.29)	37.20 (0.08)	36.98 (0.32)	37.21 (0.07)	35.99 (0.05)
20	100	50.86 (0.08)	50.92 (0.05)	51.02 (0.08)	50.76 (0.22)	50.99 (0.09)	49.68 (0.06)
20	110	64.73 (0.11)	64.75 (0.22)	64.91 (0.10)	64.67 (0.17)	64.84 (0.10)	63.42 (0.06)
50	90	53.30 (0.08)	53.38 (0.12)	53.46 (0.08)	53.43 (0.12)	53.36 (0.11)	51.87 (0.09)
50	100	68.87 (0.12)	68.91 (0.17)	69.06 (0.11)	68.98 (0.11)	68.93 (0.07)	67.37 (0.09)
50	110	84.50 (0.13)	84.45 (0.28)	84.65 (0.11)	84.62 (0.19)	84.52 (0.16)	82.85 (0.10)
100	90	65.83 (0.10)	65.94 (0.13)	66.02 (0.14)	65.78 (0.15)	65.80 (0.16)	64.56 (0.08)
100	100	82.79 (0.16)	82.88 (0.17)	82.96 (0.13)	82.82 (0.14)	82.74 (0.17)	81.41 (0.08)
100	110	99.73 (0.16)	99.86 (0.12)	99.93 (0.18)	99.74 (0.20)	99.70 (0.14)	98.28 (0.11)
200	90	78.23 (0.15)	78.40 (0.16)	78.43 (0.12)	78.11 (0.16)	78.20 (0.09)	77.28 (0.07)
200	100	96.57 (0.16)	96.68 (0.13)	96.79 (0.08)	96.21 (0.28)	96.44 (0.13)	95.53 (0.08)
200	110	114.87 (0.10)	115.03 (0.25)	115.10 (0.07)	114.55 (0.27)	114.74 (0.23)	113.78 (0.08)
payoff (reward) are given by:
$$
S _ {t} ^ {m} = s _ {0} ^ {m} \exp \left(\left[ r - \delta_ {m} - \sigma_ {m} ^ {2} / 2 \right] t + \sigma W _ {t} ^ {m}, \quad R _ {t} = e ^ {r t} \left(\max  _ {1 \leq m \leq d} S _ {t} ^ {m} - K\right) ^ {+}\right) \tag {14}
$$
for  $m = 1,2,\dots d$ .  $r \in \mathbb{R}$  is the risk-free rate,  $s_0^m \in (0,\infty)$  represents the initial price,  $\delta_m \in [0,\infty)$  is the dividend yield,  $\sigma_m \in (0,\infty)$  the volatility and  $W$  is a  $d$ -dimensional Brownian motion, with zero instantaneous correlation between its  $d$  components. The reward for exercise at time  $t$  is given by  $R_t$ . We discretize  $t$  into  $H + 1$  possible exercise opportunities, using times  $t_j = jT / H$  for  $j = 0,1,\dots H$ .  $T$  is the option expiration duration in years. This procedure translates the problem into our discrete-time finite horizon optimal stopping setting:  $\sup_{0 \leq \tau \leq H} \mathbb{E}[R_\tau]$ .
Mirroring Becker et. al. [2] we set,  $K = 100$ ,  $r = 0.05$ ,  $\sigma_{m} = 0.2$ ,  $\delta_{m} = 0.1$ ,  $T = 3$ ,  $H = 9$ . We generate a dataset of 40,000 sample trajectories of the underlying assets, generating different datasets for various values of  $d$  and  $s_0 = s_0^m$ ,  $\forall m$ . We train models on ten random  $50\%$  train-test splits, holding  $20\%$  of training data as a validation dataset. Means and standard deviations of the achieved exercise values by the various methods are reported in Table 1. DNN-OSPG consistently performs best, while the other Markovian DNN methods, DOS and DNN-FQI, perform well in this setting. Among the non-Markovian RNN-based methods, RLLSM fares the worst due to increased noise from prior time-steps. RNN-OSPG and RNN-FQI are still relatively competent since they can learn to ignore information from prior time-steps.
Stopping a non-Markovian fractional Brownian motion: We mimic the scenario considered in [2]. A fractional Brownian motion [25],  $(W_{t}^{h})_{t\geq 0}$  is a centered-Gaussian process with co-variance structure determined by its Hurst parameter  $h\in (0,1]$  as  $\mathbb{E}[W_t^h W_s^h ] = 0.5(t^{2h} + s^{2h} - |t - s|^{2h})$ . The increments of  $W^{h}$  are positively correlated for  $h\in (0.5,1]$  and negatively correlated for  $h\in (0,0.5)$  and uncorrelated (reducing to a standard Brownian motion) for  $h = 0.5$ . We discretize  $t$  in the range  $[0,1]$  into time-steps by  $t_j = j / H$  for  $j = 0,1,\dots H$ , and seek to solve the optimal stopping problem:  $\sup_{0\leq \tau \leq H}\mathbb{E}[W_\tau^h ]$ . Thus, the reward function is the identity mapping in this case. For each distinct setting of the Hurst parameter  $h$ , we generate 40,000 trajectories and train models on ten random  $50\%$  train-test splits, holding  $20\%$  of training data as a validation dataset.
Figure 2 plots average returns achieved by competing methods for various values of  $h$ . RNN-OSPG dominates competing methods in this non-Markovian setting across all values of  $h$ . Also, note the strong outperformance of DNN-OSPG that fits an optimal policy directly vs other Markovian methods (DOS and DNN-FQI) that propagate approximation errors due to recursive estimation of value functions. DOS-ES seeks to adapt to non-Markovian settings by extending its state space with process history. This extension, in turn, explodes the parameter space and training times (see Appendix D, Table 6) while reducing sample efficiency. RRLSM solves this issue by using an RNN but, in turn, trades model flexibility (since the RNN layer is not-trainable). The performance of DOS-ES and RRLSM is quite close in this scenario. Note that even though an RNN is used, the performance of RNN-FQI is significantly worse than RNN-OSPG for positive values of  $h$ .
Early-stopping a sequential multi-class classifier: This is a non-Markovian setting where we are interested in early stopping a sequential multi-class time-series classifier. The longer one delays
Figure 2: Fractional Brownian motion
Figure 3: Stopping a multi-class classifier
classification, the easier the task becomes since every new time-step reveals information about the class of the series. When deciding on the class of a given series, there is a tradeoff between the number of observations made and classification accuracy. Early stopping of a classifier is an optimal stopping problem with costs (instead of rewards):  $\inf_{\tau} \mathbb{E}[C_{\tau}]$  using a cost process  $\{C_j\}$  with  $C_j = [1 - \max_k \mathbb{P}(Y = k | \mathbf{S}_j)] + \alpha \frac{j}{H}$ . Here,  $Y$  represents the actual class of the time series, with  $\mathbf{S}_j$  representing the process history of the time series until time  $j$ . The first term represents the probability of a classification error at time  $j$  while the second term imposes a cost of delaying the classification decision, controlled by  $\alpha \in \mathbb{R}^+$ . In practice, the true probabilities  $\mathbb{P}(Y = k | \mathbf{S}_j)$  are unknown and must be approximated by training a classifier to get a cost model. This approach is related but not equivalent to early classification literature [26, 13, 9, 26, 1] where the goal is to learn both classifier and stopping decision jointly. Since our intent in this paper is to solve the optimal stopping problem, not optimize the cost/reward models, we train a simple baseline RNN classifier that learns parameterized approximations  $\mathbb{P}(Y = k | \mathbf{S}_j, \zeta) \approx \mathbb{P}(Y = k | \mathbf{S}_j) \forall j, k$ , with parameters  $\zeta$ . A limitation of this approach is that we are constrained by the quality of the trained classifier. Our classifier is an RNN with a single 20-unit hidden layer connected to a final time-distributed dense layer with softmax activation which produces a probability distribution over classes at each time-step. We train this model to minimize negative log-likelihood (NLL) to produce a calibrated classifier.
There are two possibilities for the cost model. Either we accept the probabilities estimated by the classifier as the true probabilities yielding:  $C_j^{est} = [1 - \max_k \mathbb{P}(Y = k | \mathbf{S}_j, \zeta)] + \alpha \frac{j}{H}$ , or use empirical miss-classification error, yielding:  $C_j^{emp} = \mathbb{I}(Y \neq \arg \max_k \mathbb{P}(Y = k | \mathbf{S}_j, \zeta)) + \alpha \frac{j}{H}$ . The latter is a more robust choice and is the evaluation metric used since the stopping decisions will not be blind to classifier calibration errors. However, this setting exposes a significant limitation of value-function methods (backward induction and FQI) since they require the cost model to be available at inference time. Thus, they can only use  $C_j^{est}$  and not  $C_j^{emp}$ , since during inference, empirical errors of the classifier are not revealed until a decision to stop has been made.