Title: Deep Recurrent Optimal Stopping
Venue: NeurIPS 2023
Accepted: yes
Keywords: optimal stopping

recurrent neural networks

probabilistic graphical models

policy gradient methods
Abstract: Deep neural networks (DNNs) have recently emerged as a powerful paradigm for solving Markovian optimal stopping problems. However, a ready extension of DNN-based methods to non-Markovian settings requires significant state and parameter space expansion, manifesting the curse of dimensionality. Further, efficient state-space transformations permitting Markovian approximations, such as those afforded by recurrent neural networks (RNNs), are either structurally infeasible or are confounded by the curse of non-Markovianity. Considering these issues, we introduce, for the first time, an optimal stopping policy gradient algorithm (OSPG) that can leverage RNNs effectively in non-Markovian settings by implicitly optimizing value functions without recursion, mitigating the curse of non-Markovianity. The OSPG algorithm is derived from an inference procedure on a novel Bayesian network representation of discrete-time non-Markovian optimal stopping trajectories and, as a consequence, yields an offline policy gradient algorithm that eliminates expensive Monte Carlo policy rollouts.

Deep Recurrent Optimal Stopping

Abstract

Deep neural networks (DNNs) have recently emerged as a powerful paradigm for solving Markovian optimal stopping problems. However, a ready extension of DNN-based methods to non-Markovian settings requires significant state and parameter space expansion, manifesting the curse of dimensionality. Further, efficient state-space transformations permitting Markovian approximations, such as those afforded by recurrent neural networks (RNNs), are either structurally infeasible or are confounded by the curse of non-Markovianity. Considering these issues, we introduce, for the first time, an optimal stopping policy gradient algorithm (OSPG) that can leverage RNNs effectively in non-Markovian settings by implicitly optimizing value functions without recursion, mitigating the curse of non-Markovianity. The OSPG algorithm is derived from an inference procedure on a novel Bayesian network representation of discrete-time non-Markovian optimal stopping trajectories and, as a consequence, yields an offline policy gradient algorithm that eliminates expensive Monte Carlo policy rollouts.

1 Introduction

In the classic optimal stopping setting, an agent monitors the trajectory of a stochastic process, with the opportunity to either continue observing or stop and claim a reward (or suffer a cost), which is generally a function of process history. Once the agent decides to stop, no further actions are possible, and no further reward can be claimed (or cost suffered). The agent's goal is to produce stopping decisions based on the process observations to maximize the expected reward (or minimize the expected cost). For example, whether to exercise a stock option or not at any time is an optimal stopping problem where the goal is to maximize expected exercise value, as is the decision of when to replace a degrading machine. Optimal stopping methods have a wide array of applications in finance [5, 2], operations research [17, 24, 10], disorder/change-point detection [36, 32], early classification of time-series [13, 1] among others.

In this paper, we are concerned with the discrete-time, finite-horizon, model-free setting, where the process evolves over discrete time-steps, and a decision to stop must occur by a given finite horizon. Further, the dynamics of process evolution are unknown. This setting is in contrast with classic approaches [8, 28, 35, 14, 36] which assume that the dynamics of process evolution are known and characterized with simplified and usually Markovian stochastic models. Note that continuoustime optimal stopping problems, including the pricing of American options, may often be reduced to the discrete-time setting by suitable discretization [18]. Solving optimal stopping problems in non-Markovian settings is challenging due to the following curses:

The curse of dimensionality [3, 4, 33]: A non-Markovian process may be made Markovian by extending state space with process history. However, such an extension significantly increases the problem's dimensionality, complicating function approximation. Estimating high-dimensional value functions directly rules out conventional approaches that approximate value functions with a linear combination of basis functions [40, 23, 39, 40, 43].

The curse of non-Markovianity [41, 6]: One may mitigate the curse of dimensionality by approximating a non-Markovian process by a Markovian process with a transformed state-space. For example, RNNs can learn to summarize relevant process history into a Markovian hidden-state process efficiently. However, state-of-the-art Markovian optimal stopping methods [19, 20, 2, 16] rely on the recursive computation of value functions (see Section 2) which propagates approximation bias in non-Markovian settings, a phenomenon known as the curse of non-Markovianity [6].

Existing deep neural network approaches: Deep neural network approaches to optimal stopping [19, 20, 2, 16] have emerged as a powerful way of handling high-dimensional Markovian optimal stopping problems. Existing model-free methods compute value function approximations relying on a recursive temporal relationship between value functions at successive time-steps in Markovian settings embodied by the Wald-Bellman equation [36] or closely related equivalent (see Section 2).

Backward Induction methods [19, 20, 2, 16] start from the last time-step and recurse backward sequentially through time. Non-Markovian settings force Backward Induction methods to augment their state space with process history [2] significantly exacerbating the curse of dimensionality since elegant approaches such as RNNs, which share parameters across time-steps are not an option for backward induction methods. If the value function of time-step j +1 is already optimized, it cannot change when we fit the value function at time-step j . Further, the inability to share parameters across time-steps means parameter space grows linearly with the time-steps processed, thereby reducing sample efficiency. A notable exception is the method of Herrera et al. [16], which uses an RNN. However, the RNN weights are random and not learnable to permit backward induction to proceed. Only the final hidden-layer weights are fit, and these are not shareable across time-steps.

Fitted Q-Iteration (FQI) methods [34, 16] start with bootstrapped value functions at all time-steps and recursively update these for temporal consistency with the Wald-Bellman equation. While they can use RNN-based function approximators, they suffer from the curse of non-Markovianity.

We note that a body of work uses deep neural networks to solve continuous-time optimal stopping problems by formulating them as free-boundary Partial Differential Equations (PDE). These include methods such as the deep Galerkin method [37] and the backward stochastic differential equation method [7]. We consider such methods model-based since they start with a specific PDE to be solved that assumes prior knowledge of process evolution.

Outline of contributions: The above context points to (1) using a suitable characterization of state space (as afforded by RNNs) to mitigate the curse of dimensionality and (2) direct policy estimation methods such as policy gradients to mitigate the curse of non-Markovianity. Our approach brings together, for the first time, probabilistic graphical models, policy gradient methods [42], and RNNs to design effective stopping policies for non-Markovian settings. policy gradient methods are notably missing from the optimal stopping literature. We make the following contributions:

We present a reward augmented trajectory model (RATM) (see Section 3), a novel Bayes net parameterization of non-Markovian state-action-reward trajectories encountered in optimal stopping, where conditional probability distributions (CPDs) encode stochastic policy actions and reward possibilities (Section 3). In particular, this formulation allows the stochastic policy to share parameters across time-steps, allowing RNN-based CPD approximations, which mitigate the curse of dimensionality issues discussed earlier. Also, the problem of finding an optimal stopping policy reduces to a direct policy optimization procedure based on the E-M algorithm (Theorem 3.1, and Corollary 3.1.1) that leverages inference over the Bayes net. This mitigates the curse of non-Markovianity associated with recursive value function estimation. Further, modeling optimal stopping with graphical models opens new avenues. As noted by Levine [22], in the context of reinforcement learning, "The extensibility and compositionality of graphical models can likely be leveraged to produce more sophisticated reinforcement learning methods, and the framework of probabilistic inference can offer a powerful toolkit for deriving effective and convergent learning algorithms for the corresponding models"; for instance, augmentation with additional latent variables may be explored to model process disorders such as change-points.

Inference on the RATM yields a policy gradient algorithm when we take an incremental view of the E-M algorithm (see Section 4, Theorem 4.1 and Proposition 4.1). We call this algorithm the optimal stopping policy gradient (OSPG) method, the first policy gradient algorithm, to the best of our knowledge, for computing optimal stopping policies suitable for non-Markovian settings. A key benefit of OSPG is that unlike classic policy gradients [42], it is an offline algorithm and does

not require Monte-Carlo policy rollouts. Crucially, we show (Corollary 4.1.1) that our procedure is learning a policy that implicitly optimizes value functions at every time-step without requiring recursive value function computation, thereby mitigating the curse of non-Markovianity.

We introduce a new loss function based on OSPG that is well-suited for learning RNN-based stopping policies with mini-batch stochastic gradient descent (Algorithm 1). This makes OSPG (unlike E-M) practical for large datasets while leveraging powerful off-the-shelf optimizers such as Adam. The OSPG loss considers an entire process trajectory and does not decompose across timesteps, allowing policy decisions across a trajectory to be evaluated jointly rather than independently.

2 Discrete-time finite-horizon optimal stopping

Werepresent the stochastic process monitored by d -dimensional random fully observable environment state variables { S j } . Process history until and including time-step j is represented by the d × j random matrix S j := [ S 0 , S 1 , · · · S j ] . Associated with state history is a reward process { R j } with R j = g j ( S j ) , where g j : R d × j ↦→ R + is a reward function that maps observation history to a reward for stopping at time-step j . Uppercase letters denote random variables, while lowercase letters represent their realizations. Bold font is used to indicate histories. Thus s j is an observed partial trajectory from S j . Also, in case we need to refer to a specific trajectory, we use an extra index i , so for example, r ij refers to the observed reward for stopping at time-step j when monitoring the i th process trajectory.

Definition 2.1 (stopping policy) . A stopping policy is defined as a sequence of functions ϕ = ( ϕ 1 , ϕ 2 , · · · ϕ j , · · · ) , where ϕ j : R d × j ↦→{ 0 , 1 } maps the history of observations at a time-step j to a decision of whether to stop ( ϕ j ( s j ) = 1 ) or not ( ϕ j ( s j ) = 0 ).

Definition 2.2 (stochastic stopping policy) . A stochastic stopping policy is defined as a sequence of functions φ = ( φ 1 , φ 2 , · · · φ j , · · · ) , where φ j : R d × j ↦→ [0 , 1] maps the history of observations at a time-step j to a probability of stopping at time j .

Note that a stochastic policy may be converted to a deterministic stopping policy to make stopping decisions by augmenting the state space with i.i.d. virtual observations from a random variable U uniformly distributed in [0 , 1] Thus, the equivalent deterministic policy is ϕ j ( s j ) := I ( φ j ( s j ) ≥ u j ) .

This paper considers the finite horizon setting where j ≤ H ∈ N , so a decision to stop must be made at or before a finite horizon H . Thus we define ϕ H ( S H ) := 1 . The decision to stop is based on a stochastic process, making the stopping time a random variable.

Definition 2.3 (policy stopping time [31]) . Stopping occurs when the policy first triggers a stop action. This occurs at random time τ := min { 0 ≤ j ≤ H : ϕ j ( S j ) = 1 } , called the stopping time 1 .

With this background, we can formally define the optimal stopping problem:

Definition 2.4 (optimal stopping problem [36]) . Solve for the optimal stopping time τ ∗ (if one exists) for which E [ R τ ∗ ] = sup τ E [ R τ ] where R j = g j ( S j ) is a reward process, τ is a stopping time random variable, and { S j } is a stochastic process.

Following Shiryaev and Peskir [36, 28], in the finite-horizon, discrete-time setting, the existence of a finite optimal stopping-time is guaranteed when E [ sup j ≤ H | R j | ] < ∞ . The optimal stopping time can then be determined by the method of backward induction which recursively constructs the Snell-envelope 2 comprising random variables { U j } given by:

The optimal stopping time is given by τ ∗ = min { 0 ≤ j ≤ H : U j = R j } . From a practical perspective, the expectations in the recursion are hard to estimate since they depend on process history, manifesting the curse of dimensionality. The assumption of Markovian state evolution, i.e. P ( S j +1 | S j ) = P ( S j +1 | S j ) , and a reward process expressable as a function G j of the current state,

A

0

A

1

A

τ

S

0

S

1

S

τ

S

H

(a) state-action trajectory model

Z

τ

R

0

R

1

R

τ

R

H

A

0

A

1

A

τ

S

0

S

1

S

τ

S

H

(b) reward augmented Bayesian network model

i.e. R j = G j ( S j ) , directly leads to the much more tractable Wald-Bellman equation [28, 35, 36]:

The value functions V j ( s ) give the maximum expected reward obtainable from time j having observed state s . Thus V j ( s ) = sup τ ≥ j E s [ R τ ] ∀ j . The operator T j is defined by T j F ( s ) := E j,s [ F ( S j +1 )] for any function F of the state. The Snell-envelope reduces to U j = V j ( S j ) in this case. We define K j ( s ) := T j V j +1 ( s ) as the continuation value-function , which is the maximum expected reward one can obtain by choosing to continue at step j having observed state s . The optimal stopping policy is given by:

The value functions are, therefore, related to the optimal policy as follows:

Thus, if we can estimate the continuation value functions K j , we can trigger optimal stopping decisions using equation (1). Note that this requires knowledge of reward function G j at inference time. Backward induction approaches typically, first estimate K H -1 to approximate K H -1 ( S H -1 ) ≈ G H ( S H ) and then use either the Wald-Bellman equation [40, 19] or equations (1) and (2) [23, 20, 16] to produce approximation targets V H -1 ( S H -1 ) which are used as targets to fit K H -2 ( S H -2 ) ≈ V H -1 ( S H -1 ) , and the process continues backwards recursively. As a variation on this theme, Becker et al. [2] first set K H -1 ( S H -1 ) = G H ( S H ) and then estimate ϕ ∗ j ( s ) in equation (2) by maximizing the RHS of equation (2) with j = H -1 and ϕ H -1 replacing ϕ ∗ H -1 . Then equation (2) is used to compute V H -1 ( S H -1 ) which is in turn used to set K H -2 ( S H -2 ) = V H -1 ( S H -1 ) and the process proceeds recursively. Fitted Q-iteration methods [39, 40, 43] use bootstrapped continuation value functions K j to produce V j ( S j ) at all time-steps using the Wald-Bellman equation. Then, updated continuation value-functions K j are fit simultaneously across all time-steps to approximate V j +1 ( S j +1 ) , and the process continues recursively until convergence. If Markovian assumptions do not hold, states S j need to be replaced with S j exploding the state space and significantly complicating function approximation (the curse of dimensionality). The recursive procedures discussed above propagate bias across time-steps when using Markovian approximations, since bias in value estimation at one time-step infects the preceding time-step (the curse of non-Markovianity).

3 A Bayesian network view of optimal stopping

This section presents a new Bayesian network view of optimal stopping. While we draw inspiration from a corresponding view of general reinforcement learning (RL) [22], the resulting modeling choices needed to capture the structure of optimal stopping make the approaches distinct.

State-action trajectory model: Figure 1(a) shows a Bayes net model of state-action trajectories in optimal stopping. An action A j at any time-step j depends on the state history S j up to that time-step, permitting general non-Markovian stopping policies. Further, actions are restricted to 0 (continue) or 1 (stop), with the stop action terminating the action trajectory. Therefore, a key feature of optimal stopping problems is that, unlike general reinforcement learning, actions do not impact the probability distribution of the following states over the control horizon, conditioned on state history. We do not model the stopping time τ explicitly. Instead, it is a consequence of stochastic policy actions over

the horizon. The action trajectory stops at a random stopping time τ = min { 0 ≤ j ≤ H : A j = 1 } . Therefore, we have the special structure A τ = 1 and A j = 0 , ∀ j < τ . Note that Figure 1(a) represents a family of stopping trajectories A τ , each member stopping at a specific value of τ . We denote a specific stopping trajectory by A j . Consider stochastic policies, given by the functions φ j ( S j ) := P ( A j = 1 | S j ) . We define ψ j ( S j ) := P ( τ = j | S j ) to represent the stopping-time distribution. We have the following key relationship between action trajectories and the stopping-time random variable encapsulated in the following lemma, adapted from [14], proved in Appendix A.

Lemma 3.1 (trajectory reparameterization lemma) .

Reward augmented trajectory model: Figure 1b shows an augmentation of the basic state action trajectory model to bring the notion of stopping rewards into the model. The variables R j represent the reward process, which are functions of state history: recall R j = g j ( S j ) . We introduce Z τ as a family of binary random variables representing if a reward is obtained ( Z τ = 1 ) or not ( Z τ = 0 ) when we stop at various values of τ . Thus stopping at j no longer guarantees reward R j . The idea is to use random variables Z τ to transform absolute rewards which are not readily interpreted as probabilities into probabilities (in this case, probabilities of obtaining a reward at any time-step) that may be represented in the Bayes net. We parameterize the conditional probability distribution of Z τ with Bernoulli distributions defined as follows:

where ˜ r : { 0 , 1 , · · · H } × ( R + ) H ↦→ [0 , 1] is a function that transforms real-valued stopping rewards into probabilities that encode the notion of the relative possibility of obtaining rewards over a trajectory by stopping at a particular time-step of the trajectory. We may write the joint distribution of a trajectory stopping at time τ as:

We have used the reparameterization lemma 3.1 to replace the probability distribution over actions with the corresponding induced probability distribution over stopping times. Conditioning on R H and S H we have:

Optimal policy estimation: We are particularly interested in the conditional probability of getting a reward when we stop. We can obtain this by using Bayes net inference to obtain equation (5) and then marginalizing over the family of stopping trajectories A τ . Since there is a one-to-one correspondence between A τ and stopping time τ , we just need to sum over all possible stopping times. Thus, if we define Z := Z 0 ⊕ Z 1 ⊕··· ⊕ Z H , where ⊕ is the XOR operator 3 , we have, from equation (5):

Ideally, we would want our policy to maximize this conditional probability. To this end, we consider parameterized policy functions φ θ j and induced stopping time distributions ψ θ j and define the conditional likelihood function over a sample trajectory as:

However, maximizing this likelihood over samples while allowing optimal stopping decisions on any trajectory ignores the relative magnitude of rewards between different trajectories. Thus, we seek to weight the likelihood of each trajectory. Therefore, considering a dataset D of environment trajectories and using index i for the i th trajectory in the sample, we have:

where ˜ w i is the relative weight for the i th sample. Taking logarithms, we have the following objective to maximize:

We may use the following Expectation-Maximization (E-M) procedure to maximize this objective: Theorem 3.1 (E-M procedure for optimal stopping policy) . The following iterative procedure converges to a local maximum of J WML ( θ ) with monotonic increase in the objective with iteration index ( k ) , starting with an initial value θ = θ (0) :

where ˜ w ( k ) i = ˜ w i , ∀ k . The proof follows from an application of Jensen's inequality and is given in Appendix A. Dayan and Hinton pioneered using E-M methods for RL policy search [12].

Specification of { ˜ r ij } and ˜ w : ˜ r ij is a specification of the relative intra-trajectory probability of obtaining a reward when stopping at time-step j of the i th trajectory. A natural choice (and one that we adopt in this paper) is to set ˜ r ij := r ij ∑ H j =0 r ij , where r ij is a realization of R j = g j ( S j ) corresponding to the i th trajectory. This formulation encourages stopping at time-steps of the trajectory, proportional to achievable reward. Alternatively, formulations such as ˜ r ij := r ij max k r ik or ˜ r ij := exp( r ij ) ∑ H k =0 exp( r ik ) could be used to express different stopping preferences within a trajectory. The weights ˜ w , on the other hand, express relative inter-trajectory importance. One plausible choice would be to set

one could weight trajectories by the maximum reward achievable: ˜ w i = ˜ r i = max j r ij ∑ i max j r ij . We may also consider reweighting schemes that vary weights over rounds of the E-M algorithm. In the RL literature, Reward-weighted regression [30] takes such an approach. Reward-weighted regression typically conducts rollouts of the current policy and weights each resulting sample trajectory with the relative reward achieved by the current policy on that trajectory. However, the trajectory reward is unknown since we do not perform policy rollouts. Instead, we weight each trajectory offline by its relative expected reward under the current policy. Specifically, we compute weights at each round according to a W-step:

This weighting is not arbitrary as it leads to a surprising equivalence with a policy gradient method, established in the next section. Further, we have the following Corollary (proof in Appendix A):

Corollary 3.1.1 (Reweighted E-M) . Let ˜ r i = ∑ H j =0 r ij ∑ N i =1 ∑ H j =0 r ij . Using the reweighting scheme of the W-step in the E-M procedure of Theorem 3.1 achieves a local maximum of:

with monotonic increase in the objective, starting with an initial value θ = θ (0) . The second term is the KL-divergence between distributions ˜ w and ˜ r .

4 Optimal stopping policy gradients (OSPG)

In this section we show that an incremental version of the E-M algorithm described in the previous section, with a single gradient step replacing a full M-step, is equivalent to a policy gradient method that maximizes expected reward. We call this the optimal stopping policy gradient (OSPG) method.

Consider a direct maximization of the optimal stopping objective J OS ( θ ) = E [ R τ ] from definition 2.4 where the expectation is over state-action trajectories. We can use our Bayes net model of state-action trajectories without reward augmentation shown in Figure 1a to obtain:

where we have used the trajectory reparameterization lemma to reparameterize in terms of stopping times. Therefore, we may write:

We leverage methods used to establish the policy gradient theorem [42, 38] to derive a convenient form for the gradient of the optimal stopping objective w.r.t policy parameters θ .

Theorem 4.1 (Optimal stopping policy gradient theorem) . For any differentiable stopping policy φ θ , The gradient of the objective J OS ( θ ) = E [ R τ ] w.r.t. policy parameters θ , where { R j } is a risk process and τ is a stopping-time random variable, is given by:

where the ψ θ j ( s j ) are functions of φ θ obtained by Lemma 3.1.

The proof is an application of the log-derivative trick [42] and is given in Appendix A. Note that optimal stopping policy gradients are estimated using offline state trajectories only. Unlike the general RL case, we never need to do policy rollouts to sample trajectories. Instead, we explicitly calculate the probability distribution over all possible stopping actions (or stopping times) and use this to average over all possible stop-continue decisions. Also, an update to the policy parameters can be done without fresh data collection since we can correctly (due to explicit Bayes net modeling) model the impact of the updated policy on the loss using the already sampled environment trajectories. Appendix B describes specialization to settings with costs instead of rewards.

We now establish a surprising equivalence between the reweighted E-M algorithm of Corollary 3.1.1 and the policy gradient method derived in this section when an incremental E-M solution approach [27] is used. Although the objective functions J WML ( ˜ w , θ ) and J OS ( θ ) are different in general, the same procedure maximizes both.

Proposition 4.1 (Incremental E-M as a policy gradient method) . If a partial M-step is used in the reweighted E-M approach of Corollary 3.1.1, comprising of a single gradient-step, then the resulting incremental E-M procedure is equivalent to the optimal stopping policy gradient method of Theorem 4.1, each converging to a local maximum of both J WML ( ˜ w , θ ) and J OS ( θ ) .

The proof is given in Appendix A. Thus, we may view the incremental E-M approach that optimizes a weighted maximum likelihood objective as a generalization of the policy gradient method that seeks to optimize the expected reward objective. The two are equivalent for a particular weighting scheme and Bayes net specification (as in Corollary 3.1.1). Further, the OSPG gradient has the following insightful form, obtained by applying Lemma 3.1 to the M-step objective of Theorem 3.1.

Corollary 4.1.1 (Value form of the OSPG gradient) . Let v ij := ψ θ j ( s ij ) r ij , k ij := [ ∑ H n = j +1 v in ] . We may rewrite the policy gradient of Theorem 4.1 as:

The proof is given in Appendix A. v ij is an empirical estimate of immediate reward, while k ij is an empirical estimate of continuation value under the current policy. The term in the bracket is the the derivative of the objective w.r.t. the policy output, evaluated at s ij . This is positive, and hence calls for increasing stopping probability, if v ij k ij > φ θ j ( s ij ) 1 -φ θ j ( s ij ) , i.e. when the ratio of immediate to continuation

value exceeds the current odds of stopping. Of course, the policy is updated considering expected immediate reward and continuation values at all time-steps. The updated policy produces new estimates of immediate reward and continuation value from which a new OSPG gradient is computed, and process proceeds until convergence to an optimal policy. Thus, this process implicitly uses value functions at all time-steps to update the policy which implicitly further improves value functions. Since there is no recursion across time-steps OSPG is able to mitigate the curse of non-Markovianity.

Temporal loss functions for learning RNN stopping policies: We may directly adopt the M-step objective of Theorem 3.1 as a loss function for learning an RNN-based policy with mini-batch stochastic gradient-descent (SGD), effective in non-Markovian settings. Algorithm 1 shows the pseudocode for computing our OSPG loss. This loss is ideally suited for learning stopping policies with RNNs since it takes as input stopping probabilities at every time-step φ θ j ( s ij ) that are each a function of process observation up to that point. An RNN performs this mapping naturally since it maps input history to a hidden state at each time-step with recurrent computation. So, for example, return_sequences=True, coupled with a TimeDistributed Dense layer in Keras, returns a complete trajectory of RNN predictions for input into the loss. Note also that this is a temporal loss that considers the entire process trajectory and does not decompose across time-steps.

Algorithm 1 Pseudocode for mini-batch computation of our temporal OSPG loss

Input: R := [ r ij ] , Φ := [ φ θ j ( s ij ) ] { 1 ≤ i ≤ N b , 0 ≤ j ≤ H , N b is batch size} Ψ : , 0 = Φ : , 0 , Ψ : ,H = exp( ∑ H -1 j =0 log(1 -Φ : ,j )) {numerically stable computation of ψ θ j ( s ij ) } Ψ : , 1: H -1 = exp(log(1 -Φ : ,H -2 ) U +logΦ : , 1: H -1 ) { U is unit upper triangular, CUMSUM} V = stop-gradient ( R glyph[circledot] Ψ) {held constant, so no gradient. glyph[circledot] denotes Schur product} Output: J = -1 N b 1 T [ V glyph[circledot] log Ψ] 1

5 Experiments

We compare our OSPG algorithm using deep neural networks (DNN-OSPG) and recurrent neural networks (RNN-OSPG) against the following model-free discrete-time optimal stopping approaches.

Backward Induction : Deep Optimal Stopping (DOS) [2] including an extension of state space with process history (DOS-ES) [2] and Randomized Recurrent Least Square Monte Carlo (RRLSM) [16].

Fitted Q-iteration : While fitted Q-Iteration (FQI) approaches in the optimal stopping literature rely on linear approximation with chosen basis functions [39, 40] break down in high-dimensional settings, they may be readily extended to leverage deep neural network (DNN-FQI) and recurrent neural network (RNN-FQI) function approximators. Appendix C provides details of our corresponding DNN-FQI and RNN-FQI baselines.

We conduct experiments with both Markovian and non-Markovian datasets. Markovian data include the pricing of high-dimensional Bermudan max-call options and American geometric-call options (relegated to Appendix D due to space). Non-Markovian data includes stopping a fractional Brownian motion and the early stopping of a sequential multi-class classifier. Except for the American option pricing experiment, all DNN function approximators use two hidden layers of 20 hidden units each, while all RNN function approximators leverage an RNN/GRU with a single hidden layer of 20 units. Backward induction methods (DOS, RRLSM) require independent networks at each time-step, while other approaches share network parameters across time-steps. Details of hyper-parameter settings, model size, and compute times are provided in Appendix D.

Exercise of a Bermudan max-call option: In this Markovian setting, we seek optimal strategies to maximize the returns of a Bermudan max-call option, a scenario described in [2]. Bermudan options may be exercised only at fixed times, unlike American options which may be exercised at any time prior to option expiration. The payoff of these options depends on the maximum of d underlying assets with multi-dimensional Black-Scholes dynamics and the strike price K . The dynamics and

payoff (reward) are given by:

for m = 1 , 2 , · · · d . r ∈ R is the risk-free rate, s m 0 ∈ (0 , ∞ ) represents the initial price, δ m ∈ [0 , ∞ ) is the dividend yield, σ m ∈ (0 , ∞ ) the volatility and W is a d -dimensional Brownian motion, with zero instantaneous correlation between its d components. The reward for exercise at time t is given by R t . We discretize t into H + 1 possible exercise opportunities, using times t j = jT/H for j = 0 , 1 , · · · H . T is the option expiration duration in years. This procedure translates the problem into our discrete-time finite horizon optimal stopping setting: sup 0 ≤ τ ≤ H E [ R τ ] .

Mirroring Becker et. al. [2] we set, K = 100 , r = 0 . 05 , σ m = 0 . 2 , δ m = 0 . 1 , T = 3 , H = 9 . We generate a dataset of 40,000 sample trajectories of the underlying assets, generating different datasets for various values of d and s 0 = s m 0 , ∀ m . We train models on ten random 50% train-test splits, holding 20% of training data as a validation dataset. Means and standard deviations of the achieved exercise values by the various methods are reported in Table 1. DNN-OSPG consistently performs best, while the other Markovian DNN methods, DOS and DNN-FQI, perform well in this setting. Among the non-Markovian RNN-based methods, RLLSM fares the worst due to increased noise from prior time-steps. RNN-OSPG and RNN-FQI are still relatively competent since they can learn to ignore information from prior time-steps.

Stopping a non-Markovian fractional Brownian motion: We mimic the scenario considered in [2]. A fractional Brownian motion [25], ( W h t ) t ≥ 0 is a centered-Gaussian process with co-variance structure determined by its Hurst parameter h ∈ (0 , 1] as E [ W h t W h s ] = 0 . 5( t 2 h + s 2 h -| t -s | 2 h ) . The increments of W h are positively correlated for h ∈ (0 . 5 , 1] and negatively correlated for h ∈ (0 , 0 . 5) and uncorrelated (reducing to a standard Brownian motion) for h = 0 . 5 . We discretize t in the range [0 , 1] into time-steps by t j = j/H for j = 0 , 1 , · · · H , and seek to solve the optimal stopping problem: sup 0 ≤ τ ≤ H E [ W h τ ] . Thus, the reward function is the identity mapping in this case. For each distinct setting of the Hurst parameter h , we generate 40,000 trajectories and train models on ten random 50% train-test splits, holding 20% of training data as a validation dataset.

Figure 2 plots average returns achieved by competing methods for various values of h . RNN-OSPG dominates competing methods in this non-Markovian setting across all values of h . Also, note the strong outperformance of DNN-OSPG that fits an optimal policy directly vs other Markovian methods (DOS and DNN-FQI) that propagate approximation errors due to recursive estimation of value functions. DOS-ES seeks to adapt to non-Markovian settings by extending its state space with process history. This extension, in turn, explodes the parameter space and training times (see Appendix D, Table 6) while reducing sample efficiency. RRLSM solves this issue by using an RNN but, in turn, trades model flexibility (since the RNN layer is not-trainable). The performance of DOS-ES and RRLSM is quite close in this scenario. Note that even though an RNN is used, the performance of RNN-FQI is significantly worse than RNN-OSPG for positive values of h .

Early-stopping a sequential multi-class classifier: This is a non-Markovian setting where we are interested in early stopping a sequential multi-class time-series classifier. The longer one delays

0.2

0.4

0.6

0.8

Hurst parameter

h

0.0

0.2

0.4

0.6

0.8

1.0

1.2

mean reward at stopping time

DOS

DNN-FQI

DNN-OSPG

DOS-ES

RRLSM

RNN-FQI

RNN-OSPG

CBF

ChlorineConcentration

Crop

ElectricDevices

FaceAll

FacesUCR

FiftyWords

InsectWingbeatSound

MedicalImages

MelbournePedestrian

MixedShapesRegularTrain

NonInvasiveFetalECGThorax2

StarLightCurves

Symbols

UWaveGestureLibraryX

WordSynonyms

0.2

0.4

0.6

0.8

avg cost at stopping time

RNN-FQI

RRLSM

RNN-OSPG

classification, the easier the task becomes since every new time-step reveals information about the class of the series. When deciding on the class of a given series, there is a tradeoff between the number of observations made and classification accuracy. Early stopping of a classifier is an optimal stopping problem with costs (instead of rewards): inf τ E [ C τ ] using a cost process { C j } with C j = [1 -max k P ( Y = k | S j )] + α j H . Here, Y represents the actual class of the time series, with S j representing the process history of the time series until time j . The first term represents the probability of a classification error at time j while the second term imposes a cost of delaying the classification decision, controlled by α ∈ R + . In practice, the true probabilities P ( Y = k | S j ) are unknown and must be approximated by training a classifier to get a cost model. This approach is related but not equivalent to early classification literature [26, 13, 9, 26, 1] where the goal is to learn both classifier and stopping decision jointly. Since our intent in this paper is to solve the optimal stopping problem, not optimize the cost/reward models, we train a simple baseline RNN classifier that learns parameterized approximations P ( Y = k | S j , ζ ) ≈ P ( Y = k | S j ) ∀ j, k , with parameters ζ . A limitation of this approach is that we are constrained by the quality of the trained classifier. Our classifier is an RNN with a single 20-unit hidden layer connected to a final time-distributed dense layer with softmax activation which produces a probability distribution over classes at each time-step. We train this model to minimize negative log-likelihood (NLL) to produce a calibrated classifier.

glyph[negationslash]

There are two possibilities for the cost model. Either we accept the probabilities estimated by the classifier as the true probabilities yielding: C est j = [1 -max k P ( Y = k | S j , ζ )] + α j H , or use empirical miss-classification error, yielding: C emp j = I ( Y = arg max k P ( Y = k | S j , ζ )) + α j H . The latter is a more robust choice and is the evaluation metric used since the stopping decisions will not be blind to classifier calibration errors. However, this setting exposes a significant limitation of value-function methods (backward induction and FQI) since they require the cost model to be available at inference time. Thus, they can only use C est j and not C emp j , since during inference, empirical errors of the classifier are not revealed until a decision to stop has been made.

We select 17 multi-class time-series classification datasets from the UCR time-series repository [11] (see Appendix D for details of the datasets and selection process). We use 10-random 70% / 30% (train/test) data splits to train all models, holding out a 30% of training data as a validation set. Figure 3(b) reports the average stopping costs of the policies learned by the various methods on the test set. RNN-OSPG outperforms backward-induction and FQI methods on 15 of the 17 datasets.

6 Conclusions

We introduce a policy gradient algorithm well suited to learning RNN-based optimal stopping policies effective in non-Markovian settings. Our optimal stopping policy gradient (OSPG) algorithm leverages a new Bayesian net formulation of non-Markovian state-action-reward trajectories specific to optimal stopping. This formulation leads to an offline policy gradient algorithm where Bayes net inference eliminates inefficient Monte-Carlo policy rollouts. OSPG outperforms competing methods in non-Markovian settings where existing methods must either deal with the curse of dimensionality (from extended state spaces) or the curse of non-Markovianity (from recursive Markovian approximation). A limitation of OSPG is that since it uses a temporal loss defined over complete trajectories, it may not be suitable for scenarios with only a few extremely long trajectories or when trajectories are partially observed.

Acknowledgments

We acknowledge Jerry Liu, M. Anthony Lewis, Matt Ellis, Scott Hallworth, and Jennifer Hill at HP Inc. for supporting and sponsoring this work. Finally, we thank all the reviewers for their insightful comments that have helped improve the paper.

References

Youssef Achenchabe, Alexis Bondu, Antoine" Cornuéjols, and Asma Dachraoui. Early classification of time series. In Machine Learning , volume 110, pages 1481-1504. Springer International Publishing, 2021.

Sebastian Becker, Patrick Cheridito, and Arnulf Jentzen. Deep optimal stopping. J. Mach. Learn. Res. , 20:74:1-74:25, 2019.

R. Bellman and Rand Corporation. Dynamic Programming . Rand Corporation Research Study. Princeton University Press, 1959.

D. Bertsekas and J.N. Tsitsiklis. Neuro-Dynamic Programming . Athena Scientific, 1996.

Bruno Bouchard and Warin Xavier. Monte-Carlo Valuation of American Options: Facts and New Algorithms to Improve Existing Methods. Numerical Methods in Finance , pages 215-255, 2012.

Siddharth Chandak, Pratik Shah, Vivek S Borkar, and Parth Dodhia. Reinforcement learning in non-markovian environments. CoRR , abs/2211.01595, 2023.

Yangang Chen and Justin W. L. Wan. Deep neural network framework based on backward stochastic differential equations for pricing and hedging american options in high dimensions. Quantitative Finance , 21(1):45-67, 2021.

Y.S. Chow, H. Robbins, and D. Siegmund. Great expectations: the theory of optimal stopping . Houghton Mifflin, 1971.

Asma Dachraoui, Alexis Bondu, and Antoine Cornuéjols. Early classification of time series as a non myopic sequential decision making problem. In Machine Learning and Knowledge Discovery in Databases , pages 433-447. Springer International Publishing, 2015.

Niranjan Damera Venkata and Chiranjib Bhattacharyya. When to intervene: Learning optimal intervention policies for critical events. In Advances in Neural Information Processing Systems , volume 35, pages 30114-30126, 2022.

Hoang Anh Dau, Eamonn Keogh, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, Yanping, Bing Hu, Nurjahan Begum, Anthony Bagnall, Abdullah Mueen, Gustavo Batista, and Hexagon-ML. The UCR time series classification archive, October 2018. https://www.cs.ucr.edu/~eamonn/time_series_ data_2018/ .

Peter Dayan and Geoffrey E. Hinton. Using expectation-maximization for reinforcement learning. In Neural Computation , volume 9, page 271-278, 1997.

Don Dennis, Chirag Pabbaraju, Harsha Vardhan Simhadri, and Prateek Jain. Multiple instance learning for efficient sequential data classification on resource-constrained devices. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montréal, Canada. , pages 10976-10987, 2018.

T.S. Ferguson. Mathematical Statistics: A Decision Theoretic Approach . Probability and mathematical statistics. Academic Press, 1967.

Tom Fischer. Stopping times are hitting times: a natural representation. Statistics & Probability Letters , 2011.

Calypso Herrera, Florian Krach, Pierre Ruyssen, and Josef Teichmann. Optimal stopping via randomized neural networks. CoRR , abs/2104.13669, 2021.

Daniel Jiang and Warren Powell. An approximate dynamic programming algorithm for monotone value functions. Operations Research , 63, 01 2014.

P.E. Kloeden and E. Platen. Numerical Solution of Stochastic Differential Equations . Applications of mathematics : stochastic modelling and applied probability. Springer, 1992.

Michael Kohler, Adam Krzy˙ zak, and N Zaklina Todorovic. Pricing of High-Dimensional American Options by Neural Networks. Wiley-Blackwell: Mathematical Finance , 2010.

Bernard Lapeyre and Jérôme Lelong. Neural Network Regression for Bermudan Option Pricing. Monte Carlo Methods and Applications , 2021.

Jason D. Lee, Max Simchowitz, Michael I. Jordan, and Benjamin Recht. Gradient descent only converges to minimizers. In Vitaly Feldman, Alexander Rakhlin, and Ohad Shamir, editors, 29th Annual Conference on Learning Theory , volume 49 of Proceedings of Machine Learning Research , pages 1246-1257, Columbia University, New York, New York, USA, 23-26 Jun 2016. PMLR.

Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. CoRR , abs/1805.00909, 2018.

Francis A. Longstaff and Eduardo S. Schwartz. Valuing American Options by Simulation: A Simple Least-Squares Approach. Review of Financial Studies , 2001.

V. Makis and Xiaoyue Jiang. Optimal replacement under partial observations. Math. Oper. Res. , 28:382-394, 05 2003.

Benoit B. Mandelbrot and John W. Van Ness. Fractional brownian motions, fractional noises and applications. SIAM Review , 10(4):422-437, 1968.

C. Martinez, G. Perrin, E. Ramasso, and M. Rombaut. A deep reinforcement learning approach for early classification of time series. In 2018 26th European Signal Processing Conference (EUSIPCO) , pages 2030-2034, 2018.

Radford M. Neal and Geoffrey E. Hinton. A View of the EM Algorithm that Justifies Incremental, Sparse, and other Variants , pages 355-368. Springer Netherlands, Dordrecht, 1998.

Goran Peskir and Albert Shiryaev. Optimal stopping and free-boundary problems . Lectures in Mathematics ETH Zürich. Springer, Dordrecht, 2006.

Jan Peters and Stefan Schaal. Policy gradient methods for robotics. In 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems , 2006.

Jan Peters and Stefan Schaal. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th International Conference on Machine Learning , ICML '07, page 745-750, 2007.

H.V. Poor. An Introduction to Signal Detection and Estimation . Springer Texts in Electrical Engineering. Springer New York, 2013.

H.V. Poor and O. Hadjiliadis. Quickest Detection . Cambridge University Press, 2008.

W. B. Powell. Approximate Dynamic Programming: Solving the Curses of Dimensionality . Wiley Series in Probability and Statistics. Wiley, 2011.

Martin Riedmiller". Neural fitted Q iteration - first experiences with a data efficient neural reinforcement learning method. In Machine Learning: ECML 2005 , pages 317-328. Springer Berlin Heidelberg, 2005.

A. N. Shiryaev. Optimal Stopping Rules . Stochastic Modelling and Applied Probability. Springer Berlin Heidelberg, 2007.

A.N. Shiryaev. Stochastic Disorder Problems . Probability theory and stochastic modelling. Springer International Publishing, 2019.

Justin Sirignano and Konstantinos Spiliopoulos. Dgm: A deep learning algorithm for solving partial differential equations. Journal of Computational Physics , 375:1339-1364, 2018.

Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems , volume 12, 1999.

John Tsitsiklis and Benjamin Van Roy. Optimal Stopping of Markov Processes: Hilbert Space Theory, Approximation Algorithms, and an Application to Pricing High-Dimensional Financial Derivatives. IEEE Transactions on Automatic Control , 44:1840-1851, 1997.

John Tsitsiklis and Benjamin Van Roy. Regression Methods for Pricing Complex AmericanStyle Options. IEEE Transactions on Neural Networks , 12(4):694-703, 2001.

Steven D. Whitehead and Long-Ji Lin. Reinforcement learning of non-markov decision processes. Artificial Intelligence , 73(1):271-306, 1995.

Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning , 8(3-4):229-256, may 1992.

Huizhen Yu and Dimitri P. Bertsekas. Q-Learning Algorithms for Optimal Stopping Based on Least Squares. European Control Conference , 2007.

Deep Recurrent Optimal Stopping: Supplementary Material

We provide an outline of the supplement that complements our paper.

Appendix A : provides proofs of all Theorems in the paper.

Appendix B : considers formulations of stopping problems specified in terms of costs instead of rewards and shows how to transform these into a consistent reward formulation.

Appendix C : provides discussion and pseudo-code to implement the temporal loss used to learn fitted Q-iteration policies for DNN-FQI and RNN-FQI

Appendix D : includes further details and discussion of the experiments, including model hyperparameter settings, numerical results with confidence intervals, details of compute environment, model sizes, train and inference times, etc. In Appendix E.3, we also include a new experiment that compares OSPG to state-of-the-art PDE benchmarks in the pricing of American options, which is a continuous-time optimal stopping problem.

Appendix E : includes a treatment of baseline subtraction in the context of optimal stopping policy gradients (OSPG).

A Appendix A: Proofs

A.1 Proof of Lemma 3.1

Proof. The Lemma follows from the Bayes net trajectory model of Figure 1a, and the special structure of OS action trajectories: A τ = 1 and A n = 0 , ∀ n < τ . For any OS state-action trajectory, we have:

Therefore, conditioning on the state trajectory, we have P ( A τ | S H ) = ∏ τ n =0 P ( A n | S n ) . Note that by the structure of finite-horizon OS trajectories A τ is a sequence of continue actions terminated by a stop action at τ . Thus there is a bijective mapping between stopping times and complete action trajectories given by:

So P ( A τ | S H ) = P ( κ ( τ ) | S H ) = P ( τ | S H ) . Consider a trajectory stopping at τ = j and recall that our stochastic stopping policy is defined as φ j ( S j ) : P ( A j = 1 | S j ) .

Thus, if j = 0 :

If 0 < j < H :

Finally, if j = H :

where we have used the fact that in the finite horizon setting φ H ( S H ) := 1 by definition.

We may verify ∑ H j =0 P ( τ = j | S H ) = 1 since:

Further, since τ is a stopping time random variable we have:

This is because stopping time random variables have the property that 1 ( τ = j ) is a function of S j . So we can determine if τ = j or not by only considering S j making the event { τ = j } conditionally independent of S j + k , ∀ k > 0 given S j [35]. This completes the proof.

A.2 Proof of Theorem 3.1

Proof. We apply Jensen's inequality to the objective J WML ( θ ) to obtain a lower bound:

where Q = [ q ij ] is any row-stochastic matrix satisfying q ij ≥ 0 , ∀ i, j and ∑ H j =0 q ij = 1 . Starting with a given θ (0) the lower bound is maximized (Jensen's inequality becomes an equality) when Q = Q (0) := [ q (0) ij ] such that:

This is the E-step in Theorem 3.1. Note that the E-step does not change the objective, so J WML ( θ (0) ) = J ( Q (0) , θ ) . By ignoring terms that do not depend on θ , maximizing J ( Q (0) , θ ) w.r.t. θ can be seen as equivalent to maximizing

This is the M-step. By the lower bound established in equation (21) and since the M-step maximizes J ( Q (0) , θ ) to obtain θ (1) , we have J WML ( θ (1) ) ≥ J ( Q (0) , θ (1) ) ≥ J ( Q (0) , θ (0) ) = J WML ( θ (0) ) . Therefore a round of E-M results in either an increase or no change in the objective. Since the WML objective is upper-bounded by N log H , the monotone increasing sequence J WML ( θ ( k ) ) converges to a local maximum of the objective.

A.3 Proof of Corollary 3.1.1

Proof. It suffices to show that the W-step also increases the objective J WML ( ˜ w , θ ) .

We maximize J WML ( ˜ w , θ ) w.r.t. ˜ w , subject to the constraint ∑ N i =1 ˜ w i = 1 , resulting the following Lagrangian.

Taking partial derivative w.r.t. ˜ w i and λ and setting to zero, we have:

Noting that ˜ r i := ∑ H j =0 r ij ∑ N i =1 ∑ H j =0 r ij and ˜ r ij := r ij ∑ H j =0 r ij and simplifying, we have:

where r = ∑ N i =1 ∑ H j =0 r ij . Exponentiation of both sides and cross-multiplication results in:

Substituting back in equation (28), we finally have:

Thus the maximizing ˜ w ∗ is exactly the W-step. Therefore, the form of the W-step ensures J WML ( ˜ w ( k ) , θ ( k ) ) ≥ J WML ( ˜ w ( k -1) , θ ( k ) ) ≥ J WML ( ˜ w ( k -1) , θ ( k -1) ) . Thus we have a monotone increasing and bounded sequence J WML ( ˜ w ( k ) , θ ( k ) ) converging to a local maximum of J WML ( ˜ w , θ ) .

A.4 Proof of Theorem 4.1

Proof. We use the log-derivative trick [42]. ∇ θ ψ θ j ( s j ) = ψ θ j ( s j ) ∇ θ log ψ θ j ( s j ) . So :

Summing over i , we have:

A.5 Proof of Proposition 4.1

Proof. First, for a given θ ( k ) , substituting for q ( k ) ij from Theorem 3.1 and ˜ w ( k ) i from the W-step yields:

Therefore, we may write the M-step objective as :

Dropping the iteration index ( k ), by defining constants v ij := ψ θ j ( s ij ) r ij calculated with the most recent value of θ , the M-step objective is equivalent to the following objective:

where v ij := ψ θ j ( s ij ) r ij is calculated with the most recent value of θ and held constant. Taking the gradient w.r.t θ we have

Now, substituting for v ij , we have:

This can be expressed as an expectation over sample trajectories as:

Thus the gradient of the transformed M-step objective is identical to the optimal stopping policy gradient (OSPG). Therefore, if we perform the E-step, W-step and a single gradient update, the sequence of policy parameters θ n will exactly correspond to the updated OSPG policy parameters. We may therefore appeal to literature on incremental partial M-step E-M algorithms [27] and gradient descent [21] to conclude that for small enough step-size, that increases J M ( θ ) , the policy updates converge to a local maximum of both J WML ( ˜ w , θ ) and J OS ( θ ) .

A.6 Proof of Corollary 4.1.1

Proof. From the proof of Proposition 4.1, the M-step objective is equivalent to the following objective:

Now, substituting for ψ θ j ( s ij ) in terms of the stopping policy using the trajectory reparameterization lemma (Lemma 3.1):

Since by Proposition 4.1, we have ∇ θ J OS ( θ ) = ∇ θ ¯ J M ( θ ) . Setting k ij := [ ∑ H n = j +1 v in ] in the above expression for ¯ J M ( θ ) and taking the gradient, we have:

This completes the proof of Corollary 4.1.1

B Dealing with costs instead of rewards

Although one could have used the negative of cost as a reward, this is inconsistent with our Bayesian net model since we require positive rewards to define the reward augmented trajectory model. The following result addresses this issue by transforming a problem with costs to one with a suitable reward specification.

Proposition B.1. Given costs c ij ≥ 0 the following two problems are equivalent:

Proof. Starting with the cost minimization problem, we may write:

where c i := ∑ H j =0 c ij and ˜ c ij = c ij c i . Since ∑ H j =0 ψ θ j ( s ij ) = 1 , we may subtract the constant 1 N ∑ N i =1 c i ∑ H j =0 ψ θ j ( s ij ) from the objective yielding:

where we have expanded c i and ˜ c ij . This completes the proof.

C Neural Fitted Q-iteration approaches: DNN-FQI and RNN-FQI

Fitted Q-iteration methods [39, 40] use the Wald-Bellman equation (WBE) as follows: First, given parameterized function approximations K θ j ( S j ) , a single Wald-Bellman step is bootstrapped, for a batch of trajectories yielding ˆ V j ( s ij ) = max { r ij , K θ j ( s ij ) } , ∀ j < H, 1 ≤ i ≤ N . Next the parameters of the continuation function are fit: θ ∗ = arg min ∑ i ∑ j ( K θ j ( s ij ) -ˆ V j +1 ( s ij +1 )) 2 , and the process is iterated. To provide competent DNN/RNN baselines for fitted Q-iteration (FQI) methods that are missing in the literature, we introduce a temporal FQI loss (Algorithm 2).

Algorithm 2 Pseudo-code for mini-batch computation of the FQI loss

Input: R := [ r ij ] , K := [ K θ j ( s ij ) ] { 1 ≤ i ≤ N b , 0 ≤ j ≤ H , N b is batch size} V : , 0: H -1 = stop-gradient (max { R : , 0: H -1 , K : , 0: H -1 } ) {bootstrap WBE to get value targets} V : ,H = R : ,H {final target is reward for last step}

J = MSE( V : , 1: H , K 0 , 0: H -1 ) {next step value is target for current continuation function}

D Further details of the experiments

All experiments were performed on a shared server configured with 2 × Intel Xeon Silver 12core, 2.10 GHz CPUs with 256GB RAM and equipped with 6 NVIDIA 2080Ti GPUs. However, experiments were run on a single GPU at a time and no computation was distributed across GPUs.

D.1 Model hyper-parameter settings

Table 2 shows general hyper-parameter settings used for all experiments. We apply Batch normalization to the input and outputs of layer activation functions at all hidden layers. Due to the dense correlation structure between assets at each time-step of the American option pricing experiment, we choose the hidden units to be greater than the input dimension d .

As with RL policy gradients, we may subtract a baseline value[29] to reduce variance. The OSPG algorithm uses baseline b that does not depend on time-index j , sufficient to guarantee an unbiased OSPG estimator (see Appendix E). In our experiments, we use:

D.2 Pricing Bermudan options

Table 3 shows model sizes (in trainable parameters), training, and inference times (per time-step). Model sizes grow with input dimension except for RRLSM, which uses an RNN with random, non-trainable weights to extract features from the input. The parameter size of DOS is about an order of magnitude higher than DNN-FQI and DNN-OSPG since parameters in backward induction methods like DOS grow linearly with the number of time-steps (parameters are not shareable across time-steps). Training and inference times are also high since individual models must be fit and inferred at each time-step.

D.3 Pricing American options

The scope of the paper is solving discrete-time, finite-horizon, model-free optimal stopping problems. Bermudan options that have discrete exercise opportunities are one example application. American options, which are more popular, are based on continuous-time asset price evolution and have a continuum of possible exercise times. One way to convert this to our discrete-time setting is to solve related Bermudan options. These options limit exercise opportunities to a fine discrete time grid.

State-of-the-art algorithms for pricing American options are based on Partial differential equation (PDE) methods. These methods are model-based since they start with a PDE (such as the multidimensional Black-Scholes Model) defining process evolution. For example, PDE methods often assume Markovian Black-Scholes Dynamics, and the PDEs to be solved require the Black-Scholes model parameters, such as covariance of the Brownian motion, volatility, risk-free interest rate, and dividend yield. In contrast, model-free methods, such as FQI and OSPG algorithms, do not use prior information on the evolution dynamics of the underlying stochastic process.

Nevertheless, we compare our model-free OSPG method against state-of-the-art PDE methods such as the Deep Galerkin Method (DGM) [37] and Backward Stochastic Differential Equations method (BSDE)[7] by suitable discretization of the original continuous time-problem. Note that the PDE methods also require discretization of the original PDE (ex, using the Euler-Maruyama scheme) or random sampling (as used in DGM) but do not end up directly solving a Bermudan option.

We consider multi-dimensional continuous-time American geometric-average call options with BlackScholes dynamics considered in [37] and [7]. The payoff of these options depends on the price of d underlying assets with multi-dimensional Black-Scholes dynamics and the strike price K . The dynamics are Markovian, with payoff (reward) given by:

for m = 1 , 2 , · · · d . r ∈ R is the risk-free rate, s m 0 ∈ (0 , ∞ ) represents the initial price, δ m ∈ [0 , ∞ ) is the dividend yield, σ m ∈ (0 , ∞ ) the volatility and W is a d -dimensional Brownian motion, with instantaneous correlation between its d components given by E [ W i t W j t ] = ρ ij t . The reward for exercise at time t is given by R t . We discretize t into H +1 = 100 possible exercise opportunities, using times t j = jT/H for j = 0 , 1 , · · · H . T is the option expiration duration in years. This yields the stopping problem: sup 0 ≤ τ ≤ H E [ R τ ] .

glyph[negationslash]

The specific option we consider is characterized by the following parameters: K = 100 , r = 0 . 0 , σ m = 0 . 25 , ρ ij = 0 . 75 ∀ i = j , δ m = 0 . 02 , T = 2 . The exact price of this option, p ∗ , can be determined semi-analytically for comparison [7]. We generate 10,000 batches (with a batch size of 128) for training and compute option prices on a 3,000-batch test sample. We compare vs. published results from state-of-the-art model-based PDE baselines, including the Deep Galerkin Method (PDE-DGM) [37], the Backward Stochastic Differential Equation (PDE-BSDE) method [7]. We also include published results [7] from the industry standard Longstaff-Schwartz (LS) option

pricing algorithm [23], which uses linear approximation with multi-dimensional basis functions, suitable for low dimensional settings.

Table 4 summarizes the experiment's results. Published results [7] for LS, DGM, and BSDE are included. NA denotes results not reported in the literature, while OOM indicates out of memory . Our model-free DNN-OSPG algorithm compares favorably with state-of-the-art model-based PDE-based option pricing methods that have prior knowledge of process evolution. Specifically, unlike LS, whose accuracy degrades with the number of correlated assets, and DNN-FQI, whose accuracy degrades up to 6.5%, DNN-OSPG retains excellent performance (< 0.55% error) closely approaching the model-based PDE methods (< 0.35% error). Of course, a key advantage of DNN-OSPG is that it can be used to price options for which no known PDE is available to describe process evolution.

D.4 Stopping a fractional Brownian motion

Table 5 provides numerical values corresponding to Figure 2. RNN-OSPG dominates competing methods across all values the Hurst parameter in this non-Markovian setting.

D.5 Early stopping a sequential multi-class classifier

We start with the 34 datasets, and corresponding values of α used in the [1] and remove binary classification datasets. This leaves 17 datasets. However, many of these have very few series or have a large number of classes relative to series size, which might make them unsuitable for training RNN classifiers. Nevertheless, we report results on all 17 datasets. Table 7 reports the experiment's mean and standard deviation (over ten random splits) results over 17 UCR time-series datasets. Figure 2 provides a graphical visualization of the same results. N denotes the number of trajectories, H denotes the length of the series, and K denotes the number of classes. α trades-off classification accuracy and earliness. RNN-OSPG achieves the best performance on 15 of the 17 datasets.

E Baseline subtraction for variance reduction

As with RL policy gradients, we may subtract a baseline value[29] to reduce variance. However, unlike the general RL case, due to the stopping-time-based formulation of the OSPG, the OSPG baseline should not be time-varying.

Proposition E.1 (baseline subtraction for OSPG) . The optimal stopping policy gradient (OSPG) of Theorem 4.1 is invariant to the subtraction of a constant (w.r.t. trajectory) baseline from every reward in the trajectory. Thus:

Proof. It suffices to show E s H ∼ P ( s H ) [ ∑ H j =0 bψ θ j ( s j ) ∇ θ log ψ θ j ( s j ) ] = 0 . We proceed as follows:

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper purports to develop a framework of optimal stopping that generalizes the previous approaches by incorporating non-Markovian settings and using a Bayesian network formulation.\n\nSTRENGTHS:\nThe only strength of this paper in this reviewer's opinion is the fact that it tackles an important problem.\n\nWEAKNESSES:\nThe paper is written in a manner that makes it very difficult to discern its contents. The presentation and the proofs are ridiculously long-winded even though they could they be easily written in much more succinct manner. Although the reformulation of the optimal stopping problem as a Bayesian is interesting at first sight, I don't see how it is useful unfortunately, especially given the fact that I have concerns about they are even maximizing. See the next section for more concrete examples of the problems this paper has.\n\nQUESTIONS:\n1. It isn’t clear why every finite stopping time (as defined in the literature as a measurable function $\\tau \\colon \\Omega \\to \\mathbb{N}$ such that $\\\\{\\tau \\le n\\\\} \\in \\mathcal{F} _ n$ for all $n \\in \\mathbb N$; here and elsewhere in the review $(\\mathcal{F})_{n \\in \\mathbb N}$ is the appropriate filtration which makes the relevant stochastic processes adapted) can be written in the form of Definition 2.3. I don’t believe it is even true — for intuitively the same reason why there are stopping times that are not hitting times. This makes me doubt that the paper is solving the desired optimal stopping problem and is limited to a rather special case.\n2. What does the notation $\\arg \\sup$ in Definition 2.4 mean? It isn’t immediately obvious that a optimal stopping time $\\tau^*$ will exist and therefore a small note showing this would be helpful. Perhaps a mention of the classical result that a finite optimal stopping time exists if and only if $\\tau_0 := \\inf \\\\{n \\in \\mathbb N : U_n = R_n\\\\} < \\infty$ a.s., where $U_n := \\text{ess sup} _ {\\tau \\in \\mathcal{T} _ n} \\mathbb{E}[R_\\tau \\mid \\mathcal{F}_n]$ and $\\mathcal{T}_n$ being the collection of all finite stopping times $\\tau$ satisfying $\\tau \\ge n$, would make this immediate. (Note: it can be shown that $U_n = V_n$ as defined in the paper.)\n3. As mentioned above $V_j$ should be\n    \n    $$\n    V_j(\\mathbf S_j) = \\text{ess sup}_{\\tau \\in \\mathcal{T}_j} \\mathbb{E}[R_\\tau \\mid \\mathcal{F}_j],\n    $$\n    \n    which is a *random variable* (see https://almostsuremath.com/2019/01/06/essential-suprema/ for the definition of essential supremum)*.* This isn’t what equation (1) says. Even the Wald-Bellman equation as written in the paper says that $V_j(\\mathbf S_j)$ is a random variable.\n\nLIMITATIONS:\nNot applicable.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

comments: SUMMARY:
Optimal stopping is the problem of choosing a time to take a given action based on sequentially observed random variables in order to maximize an expected payoff. Previous works used Deep Neural Networks to find the optimal stopping time (e.g. Backward Induction method), however, as the authors mentioned, these approaches have several limitations in non-Markovian settings. The paper presents the Optimal Stopping Problem as an inference problem on a Bayesian Network and they use RNNs to learn the model-free optimal stopping strategies.

STRENGTHS:
- Authors introduce a reasonable way to solve the optimal stopping problem and provide the corresponding theoretical justifications.
- The approach is well motivated (lines 43-60).
- The experimental results demonstrate that their method outperforms the baseline.
- The authors compared the training and inference times for DROS and the baseline.
- The paper is well written and organized.
- Experiments were done on real-world benchmarks.

WEAKNESSES:
- There is a whole literature on how to use deep learning to solve partial differential equations (PDEs) in general, and more specifically option pricing problems. Many papers solve PDEs using deep learning in the context of optimal stopping time.
- There are several papers on optimal stopping problems, I encourage the authors to include more baselines in their comparisons.

QUESTIONS:
- American options are the most popular in practice and is a very good case study to solve.
- Have you tested your algorithm on different American options? If not, why not?

LIMITATIONS:
- From the tables 2 and 3 (Appendix C), the training time of DROS is bigger compared to the baselines.
- The proposed method can suffer in terms of time complexity when we test it to solve problems in high dimensions.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an RNN-based approach for optimal stopping which is based on a Bayesian inference view of optimal stopping. The proposed model can be trained with direct optimization via policy gradients, or with expectation-maximization (EM). These two appraoches are shown to be equivalent. This new RNN-based approach is shown to outperform state of the art methods on a couple of commonly used datasets.\n\nSTRENGTHS:\nThe limitations of the existing deep neural network approaches are discussed and contributions of this work are clearly stated. The background on optimal stopping is also introduced in detail.\n\nWEAKNESSES:\n- **Writing and organization of the paper have much room for improvement.** \n\n**Overuse of abbreviations makes the paper difficult to read.** Although abbreviations like DNN and EM are common and inevitable, I personally would suggest against abbreviating weighted maximum likelihood as WML and policy gradiants as PG in the text. To make things worse, one of the main approaches is named DROS-OSPG, with an unnecessary repeat of \"optimal stopping\" which I find to be cumbersome and confusing.\n\n**Equations with conflicting real and dummy indices.** In equation 2 and theorem 3.1, the index $j$ is both used as a real index and a dummy index for summation. \n\n**There's too much technical detail in the main paper.** Equivalence of EM and policy gradient is interesting but probably belong better in the supplements. However I do understand that the significance of this fact might have eluded me since I am not an expert in this field. The Keras-specific implementation detail on lines 246-247 can also be saved for the supplements.\n\n**No conclusion or discussion paragraph at the end.** As a consequence, the paper does not find room to discuss its limitations, which is a requirement for NeurIPS papers.\n\n- **Ablations are missing.** Since I am not an expert in this specific field, I do not know if the improvements over the state of the art are significant enough. From the prospective of a research paper, I feel like some design decisions need to be justified by running ablations. For example, how important are the weights in the weighted maximum likelihood objective? Abalation studies like this would justify the various claims put forth throughout the paper.\n---\nOnce again I must say that it is very likely that I do not understand the significance of the paper due to lack of familiarity with the field. My main concerns with the paper lie in its presentation. I do not think the paper at its current state is doing a good job of illustrating the key ideas of the new method and of convincing me that the contributions are novel and significant.\n\nQUESTIONS:\n- In the first equation in equation 2, why is the numerator $R_\\tau$?\n- What is the justification for the paramterization of $Y_j$ in equation 2? Is this an arbitrary design decision?\n- Why is the XOR used for $Y$ on line 167?\n- In what sense is the proposed method a Bayesian interpretation of optimal stopping? Does the Bayesian interpretation still hold once the weights are introduced into the objective?\n\nLIMITATIONS:\nThe authors did not adequately address the limitations of the work since there is no conclusion or discussion section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}