SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Niranjan Damera Venkata
Digital and Transformation Organization
HP Inc., Chennai, India
niranjan.damera.venkata@hp.com
Chiranjib Bhattacharyya
Dept. of CSA and RBCCPS
Indian Institute of Science, Bangalore, India
chiru@iisc.ac.in
Deep neural networks (DNNs) have recently emerged as a powerful paradigm for solving Markovian optimal stopping problems. However, a ready extension of DNN-based methods to non-Markovian settings requires significant state and parameter space expansion, manifesting the curse of dimensionality. Further, efficient state-space transformations permitting Markovian approximations, such as those afforded by recurrent neural networks (RNNs), are either structurally infeasible or are confounded by the curse of non-Markovianity. Considering these issues, we introduce, for the first time, an optimal stopping policy gradient algorithm (OSPG) that can leverage RNNs effectively in non-Markovian settings by implicitly optimizing value functions without recursion, mitigating the curse of non-Markovianity. The OSPG algorithm is derived from an inference procedure on a novel Bayesian network representation of discrete-time non-Markovian optimal stopping trajectories and, as a consequence, yields an offline policy gradient algorithm that eliminates expensive Monte Carlo policy rollouts.
In the classic optimal stopping setting, an agent monitors the trajectory of a stochastic process, with the opportunity to either continue observing or stop and claim a reward (or suffer a cost), which is generally a function of process history. Once the agent decides to stop, no further actions are possible, and no further reward can be claimed (or cost suffered). The agent's goal is to produce stopping decisions based on the process observations to maximize the expected reward (or minimize the expected cost). For example, whether to exercise a stock option or not at any time is an optimal stopping problem where the goal is to maximize expected exercise value, as is the decision of when to replace a degrading machine. Optimal stopping methods have a wide array of applications in finance [5, 2], operations research [17, 24, 10], disorder/change-point detection [36, 32], early classification of time-series [13, 1] among others.
In this paper, we are concerned with the discrete-time, finite-horizon, model-free setting, where the process evolves over discrete time-steps, and a decision to stop must occur by a given finite horizon. Further, the dynamics of process evolution are unknown. This setting is in contrast with classic approaches [8, 28, 35, 14, 36] which assume that the dynamics of process evolution are known and characterized with simplified and usually Markovian stochastic models. Note that continuous-time optimal stopping problems, including the pricing of American options, may often be reduced to the discrete-time setting by suitable discretization [18]. Solving optimal stopping problems in non-Markovian settings is challenging due to the following curses:
- The curse of dimensionality[3, 4, 33]: A non-Markovian process may be made Markovian by extending state space with process history. However, such an extension significantly increases the problem's dimensionality, complicating function approximation. Estimating high-dimensional value functions directly rules out conventional approaches that approximate value functions with a linear combination of basis functions [40, 23, 39, 40, 43].
- The curse of non-Markovianity [41, 6]: One may mitigate the curse of dimensionality by approximating a non-Markovian process by a Markovian process with a transformed state-space. For example, RNNs can learn to summarize relevant process history into a Markovian hidden-state process efficiently. However, state-of-the-art Markovian optimal stopping methods [19, 20, 2, 16] rely on the recursive computation of value functions (see Section 2) which propagates approximation bias in non-Markovian settings, a phenomenon known as the curse of non-Markovianity [6].
Existing deep neural network approaches: Deep neural network approaches to optimal stopping [19, 20, 2, 16] have emerged as a powerful way of handling high-dimensional Markovian optimal stopping problems. Existing model-free methods compute value function approximations relying on a recursive temporal relationship between value functions at successive time-steps in Markovian settings embodied by the Wald-Bellman equation [36] or closely related equivalent (see Section 2).
Backward Induction methods [19, 20, 2, 16] start from the last time-step and recurse backward sequentially through time. Non-Markovian settings force Backward Induction methods to augment their state space with process history [2] significantly exacerbating the curse of dimensionality since elegant approaches such as RNNs, which share parameters across time-steps are not an option for backward induction methods. If the value function of time-step  $j + 1$  is already optimized, it cannot change when we fit the value function at time-step  $j$ . Further, the inability to share parameters across time-steps means parameter space grows linearly with the time-steps processed, thereby reducing sample efficiency. A notable exception is the method of Herrera et al. [16], which uses an RNN. However, the RNN weights are random and not learnable to permit backward induction to proceed. Only the final hidden-layer weights are fit, and these are not shareable across time-steps.
Fitted  $Q$ -Iteration (FQI) methods [34, 16] start with bootstrapped value functions at all time-steps and recursively update these for temporal consistency with the Wald-Bellman equation. While they can use RNN-based function approximators, they suffer from the curse of non-Markovianity.
We note that a body of work uses deep neural networks to solve continuous-time optimal stopping problems by formulating them as free-boundary Partial Differential Equations (PDE). These include methods such as the deep Galerkin method [37] and the backward stochastic differential equation method [7]. We consider such methods model-based since they start with a specific PDE to be solved that assumes prior knowledge of process evolution.
Outline of contributions: The above context points to (1) using a suitable characterization of state space (as afforded by RNNs) to mitigate the curse of dimensionality and (2) direct policy estimation methods such as policy gradients to mitigate the curse of non-Markovianity. Our approach brings together, for the first time, probabilistic graphical models, policy gradient methods [42], and RNNs to design effective stopping policies for non-Markovian settings. policy gradient methods are notably missing from the optimal stopping literature. We make the following contributions:
- We present a reward augmented trajectory model (RATM) (see Section 3), a novel Bayes net parameterization of non-Markovian state-action-reward trajectories encountered in optimal stopping, where conditional probability distributions (CPDs) encode stochastic policy actions and reward possibilities (Section 3). In particular, this formulation allows the stochastic policy to share parameters across time-steps, allowing RNN-based CPD approximations, which mitigate the curse of dimensionality issues discussed earlier. Also, the problem of finding an optimal stopping policy reduces to a direct policy optimization procedure based on the E-M algorithm (Theorem 3.1, and Corollary 3.1.1) that leverages inference over the Bayes net. This mitigates the curse of non-Markovianity associated with recursive value function estimation. Further, modeling optimal stopping with graphical models opens new avenues. As noted by Levine [22], in the context of reinforcement learning, "The extensibility and compositionality of graphical models can likely be leveraged to produce more sophisticated reinforcement learning methods, and the framework of probabilistic inference can offer a powerful toolkit for deriving effective and convergent learning algorithms for the corresponding models"; for instance, augmentation with additional latent variables may be explored to model process disorders such as change-points.
- Inference on the RATM yields a policy gradient algorithm when we take an incremental view of the E-M algorithm (see Section 4, Theorem 4.1 and Proposition 4.1). We call this algorithm the optimal stopping policy gradient (OSPG) method, the first policy gradient algorithm, to the best of our knowledge, for computing optimal stopping policies suitable for non-Markovian settings. A key benefit of OSPG is that unlike classic policy gradients [42], it is an offline algorithm and does
not require Monte-Carlo policy rollouts. Crucially, we show (Corollary 4.1.1) that our procedure is learning a policy that implicitly optimizes value functions at every time-step without requiring recursive value function computation, thereby mitigating the curse of non-Markovianity.
- We introduce a new loss function based on OSPG that is well-suited for learning RNN-based stopping policies with mini-batch stochastic gradient descent (Algorithm 1). This makes OSPG (unlike E-M) practical for large datasets while leveraging powerful off-the-shelf optimizers such as Adam. The OSPG loss considers an entire process trajectory and does not decompose across timesteps, allowing policy decisions across a trajectory to be evaluated jointly rather than independently.
2 Discrete-time finite-horizon optimal stopping
We represent the stochastic process monitored by  $d$ -dimensional random fully observable environment state variables  $\{S_j\}$ . Process history until and including time-step  $j$  is represented by the  $d \times j$  random matrix  $\mathbf{S}_j \coloneqq [S_0, S_1, \dots, S_j]$ . Associated with state history is a reward process  $\{R_j\}$  with  $R_j = g_j(\mathbf{S}_j)$ , where  $g_j: \mathbb{R}^{d \times j} \mapsto \mathbb{R}^+$  is a reward function that maps observation history to a reward for stopping at time-step  $j$ . Uppercase letters denote random variables, while lowercase letters represent their realizations. Bold font is used to indicate histories. Thus  $\mathbf{s}_j$  is an observed partial trajectory from  $\mathbf{S}_j$ . Also, in case we need to refer to a specific trajectory, we use an extra index  $i$ , so for example,  $r_{ij}$  refers to the observed reward for stopping at time-step  $j$  when monitoring the  $i^{th}$  process trajectory.
Definition 2.1 (stopping policy). A stopping policy is defined as a sequence of functions  $\varphi = (\varphi_{1},\varphi_{2},\dots \varphi_{j},\dots)$ , where  $\varphi_j:\mathbb{R}^{d\times j}\mapsto \{0,1\}$  maps the history of observations at a time-step  $j$  to a decision of whether to stop  $(\varphi_{j}(\mathbf{s}_{j}) = 1)$  or not  $(\varphi_{j}(\mathbf{s}_{j}) = 0)$ .
Definition 2.2 (stochastic stopping policy). A stochastic stopping policy is defined as a sequence of functions  $\phi = (\phi_1, \phi_2, \dots, \phi_j, \dots)$ , where  $\phi_j: \mathbb{R}^{d \times j} \mapsto [0,1]$  maps the history of observations at a time-step  $j$  to a probability of stopping at time  $j$ .
Note that a stochastic policy may be converted to a deterministic stopping policy to make stopping decisions by augmenting the state space with i.i.d. virtual observations from a random variable  $U$  uniformly distributed in  $[0,1]$ . Thus, the equivalent deterministic policy is  $\varphi_j(\mathbf{s}_j) \coloneqq \mathbb{I}(\phi_j(\mathbf{s}_j) \geq u_j)$ .
This paper considers the finite horizon setting where  $j \leq H \in \mathbb{N}$ , so a decision to stop must be made at or before a finite horizon  $H$ . Thus we define  $\varphi_H(\mathbf{S}_H) \coloneqq 1$ . The decision to stop is based on a stochastic process, making the stopping time a random variable.
Definition 2.3 (policy stopping time [31]). Stopping occurs when the policy first triggers a stop action. This occurs at random time  $\tau \coloneqq \min \{0\leq j\leq H:\varphi_j(\mathbf{S}_j) = 1\}$ , called the stopping time<sup>1</sup>.
With this background, we can formally define the optimal stopping problem:
Definition 2.4 (optimal stopping problem [36]). Solve for the optimal stopping time  $\tau^{*}$  (if one exists) for which  $\mathbb{E}[R_{\tau^*}] = \sup_{\tau} \mathbb{E}[R_{\tau}]$  where  $R_{j} = g_{j}(\mathbf{S}_{j})$  is a reward process,  $\tau$  is a stopping time random variable, and  $\{\mathbf{S}_j\}$  is a stochastic process.
Following Shiryaev and Peskir [36, 28], in the finite-horizon, discrete-time setting, the existence of a finite optimal stopping-time is guaranteed when  $\mathbb{E}\left[\sup_{j\leq H}|R_j|\right] < \infty$ . The optimal stopping time can then be determined by the method of backward induction which recursively constructs the Snell-envelope<sup>2</sup> comprising random variables  $\{U_j\}$  given by:
$$
U _ {H} := R _ {H}, U _ {j} = \max  \left\{R _ {j}, \mathbb {E} \left[ U _ {j + 1} \mid \mathbf {S} _ {j} \right] \right\}, j = H - 1, \dots 0 \quad (\text {S n e l l - e n v e l o p e})
$$
The optimal stopping time is given by  $\tau^{*} = \min \{0\leq j\leq H:U_{j} = R_{j}\}$ . From a practical perspective, the expectations in the recursion are hard to estimate since they depend on process history, manifesting the curse of dimensionality. The assumption of Markovian state evolution, i.e.  $\mathbb{P}(S_{j + 1}|\mathbf{S}_j) = \mathbb{P}(S_{j + 1}|S_j)$ , and a reward process expressable as a function  $G_{j}$  of the current state,
This is equivalent (see [15]) to the classic definition [36] of a stopping time w.r.t a filtration  $\{\mathcal{F}_j\}$  as a RV such that  $\{\tau = j\} \in \mathcal{F}_j\forall j$ . Since  $\{\mathcal{F}_j\}$  is the minimal filtration generated by  $\{\mathbf{S}_j\}$  in our setting, we can determine if  $\tau = j$  or not by considering  $\mathbf{s}_j$ . We use lowercase to represent  $\tau$  for consistency with literature.
2The Snell-envelope is also the smallest super-martingale that dominates/envelopes the reward process [36].
(a) state-action trajectory model
(b) reward augmented Bayesian network model
Figure 1: Bayesian networks representing (a) state-history  $(\mathbf{S}_j)$  and action  $(A_{j})$  trajectories and (b) corresponding reward augmented trajectory model for optimal stopping.
i.e.  $R_{j} = G_{j}(S_{j})$ , directly leads to the much more tractable Wald-Bellman equation [28, 35, 36]:
$$
V _ {H} (s) := G _ {H} (s), V _ {j} (s) = \max  \left\{G _ {j} (s), T _ {j} V _ {j + 1} (s) \right\}, j <   H \quad (\text {W a l d - B e l l m a n e q u a t i o n})
$$
The value functions  $V_{j}(s)$  give the maximum expected reward obtainable from time  $j$  having observed state  $s$ . Thus  $V_{j}(s) = \sup_{\tau \geq j} \mathbb{E}_{s}[R_{\tau}] \forall j$ . The operator  $T_{j}$  is defined by  $T_{j}F(s) \coloneqq \mathbb{E}_{j,s}[F(S_{j + 1})]$  for any function  $F$  of the state. The Snell-envelope reduces to  $U_{j} = V_{j}(S_{j})$  in this case. We define  $K_{j}(s) \coloneqq T_{j}V_{j + 1}(s)$  as the continuation value-function, which is the maximum expected reward one can obtain by choosing to continue at step  $j$  having observed state  $s$ . The optimal stopping policy is given by:
$$
\varphi_ {H} ^ {*} (\mathbf {S} _ {j}) := 1, \varphi_ {j} ^ {*} (\mathbf {S} _ {j}) = \mathbb {I} \left(G _ {j} \left(S _ {j}\right) \geq K _ {j} \left(S _ {j}\right)\right), j = 0, 1, \dots H - 1 \tag {1}
$$
The value functions are, therefore, related to the optimal policy as follows:
$$
V _ {j} \left(S _ {j}\right) = \varphi_ {j} ^ {*} \left(S _ {j}\right) G _ {j} \left(S _ {j}\right) + \left(1 - \varphi_ {j} ^ {*} \left(S _ {j}\right)\right) K _ {j} \left(S _ {j}\right) \tag {2}
$$
Thus, if we can estimate the continuation value functions  $K_{j}$ , we can trigger optimal stopping decisions using equation (1). Note that this requires knowledge of reward function  $G_{j}$  at inference time. Backward induction approaches typically, first estimate  $K_{H - 1}$  to approximate  $K_{H - 1}(S_{H - 1}) \approx G_{H}(S_{H})$  and then use either the Wald-Bellman equation [40, 19] or equations (1) and (2) [23, 20, 16] to produce approximation targets  $V_{H - 1}(S_{H - 1})$  which are used as targets to fit  $K_{H - 2}(S_{H - 2}) \approx V_{H - 1}(S_{H - 1})$ , and the process continues backwards recursively. As a variation on this theme, Becker et al. [2] first set  $K_{H - 1}(S_{H - 1}) = G_{H}(S_{H})$  and then estimate  $\varphi_{j}^{*}(s)$  in equation (2) by maximizing the RHS of equation (2) with  $j = H - 1$  and  $\varphi_{H - 1}$  replacing  $\varphi_{H - 1}^{*}$ . Then equation (2) is used to compute  $V_{H - 1}(S_{H - 1})$  which is in turn used to set  $K_{H - 2}(S_{H - 2}) = V_{H - 1}(S_{H - 1})$  and the process proceeds recursively. Fitted Q-iteration methods [39, 40, 43] use bootstrapped continuation value functions  $K_{j}$  to produce  $V_{j}(S_{j})$  at all time-steps using the Wald-Bellman equation. Then, updated continuation value-functions  $K_{j}$  are fit simultaneously across all time-steps to approximate  $V_{j + 1}(S_{j + 1})$ , and the process continues recursively until convergence. If Markovian assumptions do not hold, states  $S_{j}$  need to be replaced with  $\mathbf{S}_{j}$  exploding the state space and significantly complicating function approximation (the curse of dimensionality). The recursive procedures discussed above propagate bias across time-steps when using Markovian approximations, since bias in value estimation at one time-step infects the preceding time-step (the curse of non-Markovianity).
3 A Bayesian network view of optimal stopping
This section presents a new Bayesian network view of optimal stopping. While we draw inspiration from a corresponding view of general reinforcement learning (RL) [22], the resulting modeling choices needed to capture the structure of optimal stopping make the approaches distinct.
State-action trajectory model: Figure 1(a) shows a Bayes net model of state-action trajectories in optimal stopping. An action  $A_{j}$  at any time-step  $j$  depends on the state history  $\mathbf{S}_j$  up to that time-step, permitting general non-Markovian stopping policies. Further, actions are restricted to 0 (continue) or 1 (stop), with the stop action terminating the action trajectory. Therefore, a key feature of optimal stopping problems is that, unlike general reinforcement learning, actions do not impact the probability distribution of the following states over the control horizon, conditioned on state history. We do not model the stopping time  $\tau$  explicitly. Instead, it is a consequence of stochastic policy actions over
the horizon. The action trajectory stops at a random stopping time  $\tau = \min \{0\leq j\leq H:A_j = 1\}$ . Therefore, we have the special structure  $A_{\tau} = 1$  and  $A_{j} = 0,\forall j < \tau$ . Note that Figure 1(a) represents a family of stopping trajectories  $\mathbf{A}_{\tau}$ , each member stopping at a specific value of  $\tau$ . We denote a specific stopping trajectory by  $\mathbf{A}_j$ . Consider stochastic policies, given by the functions  $\phi_j(\mathbf{S}_j)\coloneqq \mathbb{P}(A_j = 1|\mathbf{S}_j)$ . We define  $\psi_j(\mathbf{S}_j)\coloneqq \mathbb{P}(\tau = j|\mathbf{S}_j)$  to represent the stopping-time distribution. We have the following key relationship between action trajectories and the stopping-time random variable encapsulated in the following lemma, adapted from [14], proved in Appendix A.
Lemma 3.1 (trajectory reparameterization lemma).
$$
\mathbb {P} (\tau = j | \mathbf {S} _ {H}) = \psi_ {j} (\mathbf {S} _ {j}) = \mathbb {P} (\mathbf {A} _ {j} | \mathbf {S} _ {j}) = \left\{ \begin{array}{l l} \phi_ {0} (\mathbf {S} _ {0}) & i f j = 0 \\ \phi_ {j} (\mathbf {S} _ {j}) \prod_ {n = 0} ^ {j - 1} (1 - \phi_ {n} (\mathbf {S} _ {n})), & i f 0 <   j <   H \\ \prod_ {n = 0} ^ {H - 1} (1 - \phi_ {n} (\mathbf {S} _ {n})), & i f j = H \end{array} \right.
$$
Reward augmented trajectory model: Figure 1b shows an augmentation of the basic state action trajectory model to bring the notion of stopping rewards into the model. The variables  $R_{j}$  represent the reward process, which are functions of state history: recall  $R_{j} = g_{j}(\mathbf{S}_{j})$ . We introduce  $Z_{\tau}$  as a family of binary random variables representing if a reward is obtained  $(Z_{\tau} = 1)$  or not  $(Z_{\tau} = 0)$  when we stop at various values of  $\tau$ . Thus stopping at  $j$  no longer guarantees reward  $R_{j}$ . The idea is to use random variables  $Z_{\tau}$  to transform absolute rewards which are not readily interpreted as probabilities into probabilities (in this case, probabilities of obtaining a reward at any time-step) that may be represented in the Bayes net. We parameterize the conditional probability distribution of  $Z_{\tau}$  with Bernoulli distributions defined as follows:
$$
\mathbb {P} \left(Z _ {\tau} = 1 \mid \mathbf {R} _ {H}, \mathbf {A} _ {\tau}\right) = \tilde {R} _ {\tau} := \tilde {r} (\tau , \mathbf {R} _ {H}) \quad \mathbb {P} \left(Z _ {\tau} = 0 \mid \mathbf {R} _ {H}, \mathbf {A} _ {\tau}\right) = 1 - \tilde {R} _ {\tau} \tag {3}
$$
where  $\tilde{r}:\{0,1,\dots H\} \times (\mathbb{R}^{+})^{H}\mapsto [0,1]$  is a function that transforms real-valued stopping rewards into probabilities that encode the notion of the relative possibility of obtaining rewards over a trajectory by stopping at a particular time-step of the trajectory. We may write the joint distribution of a trajectory stopping at time  $\tau$  as:
$$
\begin{array}{l} \mathbb {P} (\mathbf {A} _ {\tau}, Z _ {\tau}, \mathbf {R} _ {H}, \mathbf {S} _ {H}) = \underbrace {\mathbb {P} (\mathbf {S} _ {0}) \mathbb {P} (\mathbf {R} _ {0} | \mathbf {S} _ {0}) \prod_ {j = 1} ^ {H} \mathbb {P} (\mathbf {S} _ {j} | \mathbf {S} _ {j - 1}) \mathbb {P} (\mathbf {R} _ {j} | \mathbf {S} _ {j})} _ {\mathbb {P} (\mathbf {S} _ {H}) \mathbb {P} (\mathbf {R} _ {H} | \mathbf {S} _ {H})} \underbrace {\prod_ {n = 0} ^ {\tau} \mathbb {P} (A _ {n} | \mathbf {S} _ {n})} _ {\mathbb {P} (\mathbf {A} _ {\tau} | \mathbf {S} _ {\tau})} \mathbb {P} (Z _ {\tau} | \mathbf {A} _ {\tau}, \mathbf {R} _ {H}) \\ = \mathbb {P} \left(\mathbf {S} _ {H}\right) \mathbb {P} \left(\mathbf {R} _ {H} \mid \mathbf {S} _ {H}\right) \mathbb {P} (\tau \mid \mathbf {S} _ {H}) \mathbb {P} \left(Z _ {\tau} \mid \mathbf {A} _ {\tau}, \mathbf {R} _ {H}\right) \tag {4} \\ \end{array}
$$
We have used the reparameterization lemma 3.1 to replace the probability distribution over actions with the corresponding induced probability distribution over stopping times. Conditioning on  $\mathbf{R}_H$  and  $\mathbf{S}_H$  we have:
$$
\mathbb {P} \left(\mathbf {A} _ {\tau}, Z _ {\tau} \mid \mathbf {R} _ {H}, \mathbf {S} _ {H}\right) = \mathbb {P} (\tau \mid \mathbf {S} _ {H}) \mathbb {P} \left(Z _ {\tau} \mid \mathbf {A} _ {\tau}, \mathbf {R} _ {H}\right) \tag {5}
$$
Optimal policy estimation: We are particularly interested in the conditional probability of getting a reward when we stop. We can obtain this by using Bayes net inference to obtain equation (5) and then marginalizing over the family of stopping trajectories  $\mathbf{A}_{\tau}$ . Since there is a one-to-one correspondence between  $\mathbf{A}_{\tau}$  and stopping time  $\tau$ , we just need to sum over all possible stopping times. Thus, if we define  $Z \coloneqq Z_0 \oplus Z_1 \oplus \dots \oplus Z_H$ , where  $\oplus$  is the XOR operator<sup>3</sup>, we have, from equation (5):
$$
\mathbb {P} (Z = 1 | \mathbf {R} _ {H}, \mathbf {S} _ {H}) = \sum_ {j = 0} ^ {H} \mathbb {P} (\tau = j | \mathbf {S} _ {H}) \mathbb {P} (Z _ {j} = 1 | \mathbf {A} _ {j}, \mathbf {R} _ {H}) = \sum_ {j = 0} ^ {H} \psi_ {j} (\mathbf {S} _ {j}) \tilde {R} _ {j} \tag {6}
$$
Ideally, we would want our policy to maximize this conditional probability. To this end, we consider parameterized policy functions  $\phi_j^\theta$  and induced stopping time distributions  $\psi_j^\theta$  and define the conditional likelihood function over a sample trajectory as:
$$
l (\boldsymbol {\theta} \mid \mathbf {s} _ {H}, \mathbf {r} _ {H}) = \mathbb {P} (Z = 1 \mid \mathbf {s} _ {H}, r _ {H}; \boldsymbol {\theta}) = \sum_ {j = 0} ^ {H} \psi_ {j} ^ {\boldsymbol {\theta}} (\mathbf {s} _ {j}) \tilde {r} _ {j} \tag {7}
$$