When to Intervene: Learning Optimal Intervention Policies for Critical Events
Authors: NIRANJAN DAMERA VENKATA
Chiranjib Bhattacharyya
Venue: NeurIPS 2022 Accept (2022)
Keywords: time-series
neural survival analysis
optimal stopping

Abstract: Providing a timely intervention before the onset of a critical event, such as a system failure, is of importance in many industrial settings. Before the onset of the critical event, systems typically exhibit behavioral changes which often manifest as stochastic co-variate observations which may be leveraged to trigger intervention. In this paper, for the first time, we formulate the problem of finding an optimally timed intervention (OTI) policy as minimizing the expected residual time to event, subject to a constraint on the probability of missing the event. Existing machine learning approaches to intervention on critical events focus on predicting event occurrence within a pre-defined window (a classification problem) or predicting time-to-event (a regression problem). Interventions are then triggered by setting model thresholds. These are heuristic-driven, lacking guarantees regarding optimality. To model the evolution of system behavior, we introduce the concept of a hazard rate process. We show that the OTI problem is equivalent to an optimal stopping problem on the associated hazard rate process. This key link has not been explored in literature. Under Markovian assumptions on the hazard rate process, we show that an OTI policy at any time can be analytically determined from the conditional hazard rate function at that time. Further, we show that our theory includes, as a special case, the important class of neural hazard rate processes generated by recurrent neural networks (RNNs). To model such processes, we propose a dynamic deep recurrent survival analysis (DDRSA) architecture, introducing an RNN encoder into the static DRSA setting. Finally, we demonstrate RNN-based OTI policies with experiments and show that they outperform popular intervention methods

When to Intervene: Learning Optimal Intervention Policies for Critical Events
Niranjan Damera Venkata
Chiranjib Bhattacharyya
Digital and Transformation Organization HP Inc., Chennai, India niranjan jan. damera. venkata@hp.c com
Dept. of CSA and RBCCPS
Indian Institute of Science, Bangalore, India
Aaruuiisc.ac.o
Abstract
Providing a timely intervention before the onset of critical event, such as system failure, is of importance in many industrial settings. Before the onset of the critical event, systems typically exhibit behavioral changes which often manifest as stochas- tic co-variate observations which may be leveraged to trigger intervention. In this paper, for the first time, we formulate the problem of finding an optimally timed intervention (OTI) policy as minimizing the expected residual time to event, subject to a constraint on the probability of missing the event. Existing machine learning approaches intervention critical events focus predicting event occurrence within pre-defined window (a classification problem) or predicting time-to-event (a regression problem). Interventions then triggered by setting model thresholds. These are heuristic-driven, lacking guarantees regarding optimality. To model the evolution of system behavior, introduce the concept of hazard rate process. We show that the OTI problem is equivalent to an optimal stopping problem on the associated hazard rate process. This key link has not been explored literature. Under Markovian assumptions on the hazard rate process, we show that an OTI policy any time can be analytically determined from the conditional hazard rate function at that time. Further, we show that our theory includes, as special case, the important class of neural hazard rate processes generated by recurrent neural networks (RNNs). To model such processes. we propose a dynamic deep recurrent survival analysis (DDRSA) architecture, introducing an RNN encoder into the static DRSA setting. Finally, we demonstrate RNN -ased OTI policies with experiments and show that they outperform popular intervention methods.
Introduction
We consider a problem that involves the monitoring of observations from discrete-time stochastic process with the goal of intervening to prevent the costly occurrence of a critical event in the future. critical event is missed when intervention is not triggered before the critical event occurs. Intervening too soon, before the critical event occurs, also incurs cost. In general, delaying an intervention in order to be more confident of the intervention decision leads to increased probability of missing the event. The general goal is the design algorithms that can be used trigger a timely intervention. The objective is intervene as late possible, by minimizing residual time-to-event subject to constraint on the probability of missing the event. We term this problem the Optimally Timed Intervention problem (OTI Problem). Note that we are concerned only with the timing of an intervention, not with its effectiveness mitigating the occurrence of the critical event.
Niranjaa Damera Venkata was with the Artificial Intelligence and Emerging Compute Lab (AIECL) at HP Labs during part of this work.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

The OTI problem has significant practical significance in a number of fields. Critical events could be, for example, the failure of machine or key component, customer churn, or a fall that could result in death. Contemplated interventions in these cases could be machine or part replacement, sending a marketing offer, or move to managed care, respectively. Intervening on such events ahead of time, in an optimal manner is of great significance to businesses and society, due to the potential of mitigating the impact of the occurrences of these events. However, the literature formally studying this problem has remained sparse.
1.1 Outline of contributions
The OTI problem is hard since evolution towards a critical event is complex with typically scarce occurrences of critical events. We make the following contributions to the study of the OTI problem:
A process under observation may exhibit degradation/disorder as it evolves towards a critical event. We capture this feature by introducing the concept of a hazard rate process, which is a stochastic process modeling the evolution of conditional hazard rates as new observations are taken. key contribution of our work is to formalize the OTI problem in equation (OTI problem) and show that the OTI problem is equivalent to an optimal stopping problem on the underlying hazard rate process (Proposition 3.T). This allows us to tap into optimal stopping theory [37 9 to unlock the structure of OTI policies and also leverage computational methods of estimating/learing hazard rate functions. These are key to our development of a general machine learning approach to the OTI problem.
Under Markovian assumptions on the hazard rate process, we show that the optimal interven- tion policy decision rule at each time step can be analytically determined by the conditional hazard rate function at that time step (Theorem[4.) [.1]). Further we show that the optimal policy reduces to variable threshold on the expected nime-toeevent(ooollla 4.1.1), showing that static threshold policies are sub-optimal in general.
We show that our theory applies to an important class of hazard rate processes generated by recurrent neural networks (Section 5). We term these processes neural hazard rate processes. We may therefore apply survival analysis methods to learn such neural hazard rate processes and analytically compute optimal intervention policies for critical events. We adapt the state of the art deep dynamic survival analysis (DRSA) work (29) to our setting. Unlike DRSA, our dynamic DRSA (DDRSA) setting requires modeling dynamic hazard rate functions. We do this by adapting the DRSA architecture to our dynamic covariate evolution setting using encoder-decoder sequence-to-sequees (seq2seq) architecture.
2 Optimally Timed Interventions (OTI) on critical events
2.1 Notation
We represent the monitored stochastic process by a discrete, temporal sequence of random variables with the random vector X, representing a d-dimensional real-valued observation at time step j. Let the d j random matrix X, = [X,, X2, X,) represent the observation history until time step j. The critical event is assumed to occur intervention at time step j is considered early if the intervention occurs at a time-step j < L. We consider interventions at j > L late. by definition. We assume, without loss of generality, that interventions are instantaneous. If a fixed lead time of ó time-steps is required for an intervention, may simply consider the random variable L' L ō as the event occurrence time-step and seek timely intervention before L'. For any process realization, at time-step j. we assume that we know with certainty, if the critical event has already occurred at prior time-step or not. We define Ty == 1(L < x = 1(L < j(,,). Ty serves to indicate if the critical event has already occurred (TT = 1) or not (Ny = 0) prior to time step j. given X,-
random time-step L S Immas F Lmaz E N. An
Given a dataset of process trajectories along with corresponding critical event occurrences, we seek intervention strategies for timely intervention. We consider data driven sequential decision rules (intervention policies) that are defined as follows:
"This upper bound assumption does not pose practical limitations due to our survival analysis approach. We may treat unobserved critical events as censored. See Section 5.1 f for details

---

Definition 2.1 (Intervention policies), An intervention policy is a sequence of functions 4 = (f)1 92,--. "..""...""........."...""""...." ) where 99 : R + {0. 1) maps the history of observations at time step j to a decision of whether to intervene (fy (23) = 1) or not (4)(()) 0).
Pj....)
We define Plimas 1 (1) =- to ensure that we have at least one decision to intervene before Lmaz- Thus, for technical completeness, we assume observations can be taken until Lmaz 1. In cases where the occurrence of the critical event makes it impossible to collect further observations in practice, we can can simply fill such observations with dummy values (ex: zeros) without any loss of generality since these observations are not predictive of the critical event, and therefore do not feature in computing optimal intervention decisions (see discussion of Proposition 3.1.
The intervention policies we consider in this paper are terminal, so only single intervention is contemplated on each process realization. Once the first decision to intervene is triggered on a given realization of the monitored process, the policy effectively stops, i.e. an intervention action is assumed to be initiated and all future policy triggers for that process realization are ignored. Note however, that we can allow for policy reactivation after recovering from the critical event. The observations starting at such policy reactivations are treated as independent realizations of the process. So if we recover from the occurrence of first critical event and re-activate the policy to monitor for second second critical event, these cases will be considered as independent trials.
Definition 2.2 (policy stopping time). Intervention on a monitored process happens at a random time step == min < j < Limax (2g) = 13. We call this time the stopping time
timeFof Of the policy.
2.2 Optimally timed intervention (OTI) on critical events
We now formally state the optimally timed intervention (OTI) problem for the first time. We seek to find intervention policy (or equivalenty, the stopping time) that minimizes expected residual time-to-event, defined as B[(L- +] subject to the constraint that the probability of missing the critical event is at most a E (0, 1]:
r¹ = arg E(L where: M. =10 < Limax :PLL T) S Saf rEM.
(I)
The expected residual time-to-event is a variant of mean residual life (MRL). quantity that is well studied in literature with application multiple domains [4. Note that there exists a multiplier Ca, such that the OTI problem may be expressed in the following equivalent, unconstrained form:
ing B[(L-r)+ C O<r<_meez
C, S r) for some C. > 0
(OTI problem)
The objective encodes the natural trade-off between expected residual time-to-event, and the proba- bility of missing the critical event: as one attempts to intervene closer to the event, the probability of missing the event goes up. C. can directly be interpreted as the risk (cost) of missing the critical event. Changing C. allows one to vary the risk trade off between the miss-rate and expected time-to-event. Note that the theory allows observations after an intervention in the problem formulation. So an intervention policy can consider these covariates and fire multiple times, however, the OTI objective function in equation (OTI problem) is designed consider only the first policy trigger due to the fact that the stopping time 7 is set based on the first policy trigger (see definition 2.2).
A related objective also appears in optimal stopping formulations for Bayesian quickest change point detection problem 35 famously called the Shirayaev problem (27,, where the objective is to monitor a stochastic process for the quickest possible detection of change point stochastic disorder. Here the trade-off's is between stopping early (a false alarm), before a change point manifests, versus incurring the expected delay in detecting a change point, after it has occurred. The goal is to minimize expected detection time delay from the true change point subject to a constraint on the probability of false alarm.
a stopping time in probability theory is random variable such that ((r = j) is function of XG. So we determine if j or not by only considering X; [27].
3

---

2.3 Related work
Existing widely used approaches to timely intervention on critical events focus on predicting event occurrence with either classification or regression problem formulation. Interventions are triggered by tuning model thresholds, a process which lacks any guarantee on optimality:
Window based critical event prediction: Intervention is a binary decision. At each time-step a decision must be made to intervene or not. In window based intervention (WBI) this is often modeled directly as a binary classification problem where we look for event occurrence within a specified future window. A classifier may then be learnt to predict the probability of event occurrence in the window. An optimal threshold on this probability is then tuned to choose an optimal intervention trigger threshold. WBI policies are standard practice in predictive maintenance 31
38
Tome-to-Evee Event prediction (Survival analysis): Classical survival analysis deals with predicting the time to an event (ex: death, machine failure) given vector of observations (co- variates) obtained from a subject in the presence of incomplete (censored) observations [2T] The goal is to estimate functions such as the hazard rate (rate of death function of time) survival function (probability of survival past a given time) and understand the impact of co-variates on these functions. Classical parametric models TO
39] have been extended in recent work using machine learning approaches including approaches that extend survival analysis to model the time-varying B3 B. These have deep learning counterparts 22 12 that use recurrent neural networks to model the survival function. Recent work has applied modern deep learning approaches to longitudinal data which are sequential observations taken over time for each subject. [8] I6]. Corresponding work also exists in the area of machine failure prediction 24 These methods are complementary to the primary goal of this paper where we focus on obtaining optimal intervention policies. In practice, intervention policies based on survival analysis are typically just thresholds on the predicted time-to-event.
44
17 5 28
effects of covariates
[8]
3 OTI as an optimal stopping problem
Definition 3.1 (Optimal Stopping Problem). A problem of the form: +" arg inf, EE (R,
where (-) is risk function, - is stopping time, and (X,) } is stochastic process.
a subclass of stochastic control problems with only two possible actions, stop and continue. The goal is to determine when to stop, based on the observations, so that overall expected cost/risk over possible stopping times, is minimized. Optimal stopping techniques have been applied to several problems such as sequential hypothesis testing (37
Optimal stopping problems 37
9
(the sequential probability ratio test (SPRT)), early change point detection 27
35
34
30
(the
Shirayaev problem), early classification 23 and reliability theory 6 [2). However, the approaches
in such cases are limited to simple probabilistic models of machine degradation with single or few co-variates. General model-free reinforcement learning approaches to the optimal stopping problem are generally feasible in situations where we have lot of data or in a model-based setting where the underlying randomness can efficiently be simulated and the model can be used to draw samples [8. This is not an option in most critical event prediction settings when the critical event data is often scarce and no simple model exists for the evolution of the co-variates.
In this section we formulate the OTI problem of the OTT problem in this section, establishes, for the first time, a link between the OTI problem, survival analysis (allowing the modeling of complex co-variate evolution with impact critical-event occurrence) and optimal stopping theory. We model the dependence of the critical event time on the monitored covariate observations by hazard rate functions conditioned on observation history. To this end, we will use the following definitions of conditional hazard rate function, (-). and conditional survival function, Sy(-):
optimal stopping problem. Our formulation
(k) = = j Note that we have omitted the implied conditioning on the covariates in our notation for hy (k)), purely for convenience. We now define the concept of hazard rate process that encodes the relationship between the evolution of co-variates and onset of the critical event.
= P(L P(
(2)
Definition 3.2 (Hazard Rate Process). Consider the random vector constructed from the conditional hazard rate function (.).) given by H, = (0), hy(1) ... hy - 1))"" We call the sequence of tuples {(K,, H,}} a hazard rate process.

---

3.1 OTI as an optimal stopping problem on hazard rate process
Proposition 3.1 (OTI on hazard rate process). Given hazard rate process H,),, the optimal critical event intervention problem of equation (OTI problem reduces to the following optimal stopping problem on the corresponding hazard rate processes:
arg SCTT<Iimas
E
[(1 NI R (H,) + C,
== arg inf B[R(T,, H.)]
(3)
Oct
where R (Hr) is the intervention risk of stopping at time T, given that the critical event has not already occurred prior to T, given by:
Lmaa
R H, = Σ (1-
+ (a),(0))
(4)
m==
Proof. We only give proof sketch here. The proof of the theorem is given in Appendix A. Starting with terms in the OTI problem: for residual time to event we may write (L 7) = Ex>0 1 r+k), and then the law of iterated expectations, [[[[[_][[]==
r] ). We see that this expression is [B[(1 -- T.) E S, = (1 mイ)
Es20 E o
h, (m))]. Similarly, we have (L S [[ nT ( + m. Therefore the OTI objective may be expressed as E [R H,) = E [11 ाT, R2 (H,) + Ca]
Note that the formulation equations 5 and exposes the dependence of the OTI objective on the co-variates, since (.) depends the covariates. This was implicit in the original formulation in the previous section. A key observation is that if the critical event has already occurred (when , = 1), the objective reduces to constant (Ca), independent of the co-vertiv variates. and policy actions have influence on the objective. Hence, we may effectively restrict our attention to the case n
4 Analytic solution to the OTI problem
In order solve the OTI optimal stopping problem make the assumption that the hazard rate processes we deal with satisfy the Markov property:
Definition 4.1 (Markov hazard rate process). Consider the hazard rate process ((A, H2)}. We call a Markov hazard rate process, we have:
P(X, ++1 12,, (X, Vj
(5)
As a consequence, we can write H,- = f, (X)) where f,(.) is a function encoding the dependence of the hazard rates on the co-variates at time-step j. With this assumption in place, we have:
Theorem 4.1 (OTI policy for a Markov hazard rate process). Given Markov hazard rate process {(X, } and associated risk process (R# (H,)). the optimal intervention policy that solves the problem posed in Proposition 3.7 is policy 4p" that calls for intervention at stopping time
T" = -min(0~ < <j < Lima (X,) = 1} where:
,3(x,) = 11 ( (H S VHH))), ifj < Iinax - 1 and π
(6)
1
else
with continuation value functions V,(H, given by:
Lemax
k-1
mm
min 0cl<LLmax
Σ
II -1- ((]-)((m))-00 Co,, C.. Σ - (1=));(k)) hy(m)
(7)
Proof. We only give proof sketch here. The complete proof of the theorem is given in Appendix A. The key is to notice that under the Markov hazard rate process assumption, and the form of of equations (3) and (4). that risk R. = R(my, H,) a function X,, so we have an optimal stopping problem that has so-called Markov Representation (27]. The optimal policy has can be expressed
1p (X,) 1(R, < (2g))) with V)),, min E|R X2). So we need only compute
5

---

E[ X3), which is the expected future risk over the possible occurrences of the critical event for policy that stops + When π 0, we have R2 RR (H)). Further, reasoning using the hazard rates (K)),, we calculate E [ analytically giving rise to the final form for Vj (xt) which we express as (H))), When J = 1, R2 = minss [ X = Co.. so it is optimal to trigger intervention, setting (X,) 1. Finally, we have 40 "mmma-- (X) 1, by definition, giving rise to the final form of the optimal policy.
Thus, the optimal policy may be constructed explicitly with knowledge of only the conditional hazard rate functions at that time. The continuation value function (-()) gives the minimum expected cost of continuing take observations, having obtained a conditional hazard rate function (H2) at time j. It is clear that the the optimal policy is time-varying threshold on the risk process in general.
The following Corollary allows further important simplification of the optimal policy:
Corollary 4.1.1 (OTI policy as a time varying threshold on time-to-event), Given x, = 0, let T be the expected residual time-to-event at time j. The optimal OTI policy of Theorem 4.7 may be reduced to the following form:
()(2)) [T; S (H))),
if < Lmar - and T = else
(8)
1,
Ling 1-1 k+1
m-1
where, V2(H)) =
min o<t<Lmos-1
II (1 - hy(m)) + Ca Σ
II - nk))))
hy(mm)
m=0
m=1
A=
Lina
Imaz
with, T)
Σ
(1--
Σ k=0
Sg(
(10)
m=0
Proof. The full proof is given in Appendix A. We note that the immediate stopping risk for time can be written as: R2 T, + (O), Further, the second term, (0), may be canceled from both sides of the inequality defining the optimal stopping policy, resulting the simplified rule. N
This Corollary clearly shows that static threshold TTE policies are sub-optimal in general, since
V (H,) may vary with time index j. (H,, contains two terms. The first term, which is the risk of stopping early, before the critical event, intuitively gets smaller as j progresses toward the critical event. The second term is the risk of missing the critical event that depends on the cost of missing the event. It is high where the onset of the critical event is anticipated.
5 Neural hazard rate processes
The Markov assumption not significant restriction since any co-variate process may be made Markovian, by including all relevant information from the past into the current state (at the cost of increasing the dimension of the problem). While one may adapt any survival analysis technique (in theory) that estimates conditional hazard rate functions and apply our theory, recurrent neural networks (RNNs) are a good fit for being able to model complex patterns multi-variate time-series data and for summarizing observation history.
The class of hazard rate processes (Zz, H2)} that can be modeled by recurrent neural networks with hidden-states Z, and conditional hazard rates given by H, =((Z,) for some function f(-) of the hidden states), satisfy our Markov hazard rate process assumptions. Indeed, from the RNN dynamics, the probability of the next hidden state is conditionally independent of the past hidden states, given the current hidden state. Specifically, we have: 11,,+1|Z,) we do not require the co-variate evolution to be Markovian. The hazard rate functions can depend on the past history of co- variates in general. RNNs compute feature transformations of past covariate history (prior to j) and seek to embed all required information about the past into Zj. so a hazard rate function that is computed from Zj, can be quite complex and include (to the extent it present in Z,) information from past covariates.
(Zzz (Z,). Vj. Note, however, that

---

During training, each training sample, randomly selected from one of the time-series at time step includes a look-back window of co-variates Xj-K X, that are fed into the encoder input. The encoder's final hidden state replicated and provided to the decoder as input to each time step. We hy hy(Lman 1)
h2
(1)
O
0
D
D
D
Q
Q f1
Qihmar
RNN
RNN
RNN X
Z
RNN
RNN
RNN
1
X 2-2
j-1
Zj
Z
Zj
Figure 1: Architecture diagram for Dynamic Deep Recurrent Survival Analysis (DDRSA).
5.1 Dynamic Deep Recurrent Survival Analysis (DDRSA)
Since the OTI theory developed the previous section directly applies to neural neural hazard rate processes, we may fit neural hazard rate process model to data and then apply the optimal OTI policy intervene. Thus, our approach requires estimates of conditional hazard rates, that evolve over time (hazard rate processes).
Ren al. [29] developed Deep Recurrent Survival Analysis (DRSA). deep recurrent neural architecture that uses and RNN to map a static covariate vector to a single conditonal hazard rate function (so it is not hazard rate process that evolves time). Unlike DRSA, our Dynamic DRSA (DDRSA) setting requires modeling dynamic survival and hazard rate functions. We do this by adapting the DRSA architecture
dynamic covariate evolution setting. While there other attempts at modeling the dynamic evolution of survival functions [88 24 they either make distributional assumptions 244 or have difficulty with long sequences due large softmax layer (18).
Figure shows our DDRSA architecture which is a classic encoder-decoder architecture. DDRSA uses encoder RNN that maps variate history X3 vector is then passed into a decoder DRSA-RNE architecture, producing decoder hidden states Q₁k and corresponding conditional hazard rates h, (k) a (D(Q ((kz)). where a(.) represents the sigmoid activation function and (-)) represents a Dense layer. DDRSA can also be seen as a generative model for neural hazard rate process.
encoder hidden state vector Zj. This
leverage the DRSA loss of Ren et. al. [29] which is composed of the following loss components for each sample:
Lmuxx
Lomax (1 (1- hy(k))) : la
1
II k=
(k) (1))) Lu
1
II k=
II k=
(1- (1-)((K) (II)
1g is the likelihood of the critical event occurrence at at time j I, assuming that the event is not censored. l, is the likelihood of event occurrence (a.k.a. event rate) at any future time in the decoder output, given that the sample is not censored. le gives the likelihood event does not occur at future time over the decoder time-steps. The final loss for sample is
censored sample, when
L, > logg, (1 - [11 c) log 4, + clog lc]
(12)
where À a trade off fraction and is the censoring indicator, with c = for a censored sample and used to learn the survival
= 0 for uncensored. Losses averaged over mini-batch of samples model. The sequential DRSA loss allows effective supervision at each time-step allowing it to be applied to model long survival functions without resorting restrictive parametric distributions.
During inference, we estimate, any time-index j. hy (0), (,) hyy(Tmar - 1) as decoder outputs by passing values of co-variates X3-K X, into the DDRSA architecture encoder. Then,
7

---

we estimate the expected residual time-to-event using equation (10) and the threshold function using equation 9 Intervention is triggered time j if the estimated expected residual time-to-event falls below the threshold function.
Note that the model architecture is such that we can generate long sequence of hazard rates. During training time, we restrict the decoder time-steps to a fixed value (M < Lmaa) and consider events appearing outside this implied range as censored. However, during inference, the RNN is unrolled over the full Lmar time-steps. To allow for such unrolling, we eschew explicit decoder time-step inputs, instead foreing the decoder to learn to increment time-index. Providing the decoder time-step indices explicitly as inputs to the decoder as is done in the original DRSA-RNN A-RN [29] architecture, not desirable, because, when we unroll the RNN to Lmaz steps, we would need to give the decoder inputs it has not seen during training. Such flexible handling of long sequences is a key practical advantage of the DDRSA architecture and our survival formulation.
Experiments
The goal of this section is twofold:
1. To complement the theory developed in the earlier sections with examples validate and build intuition into survival and hazard rate processes how the optimal policy works to produce interventions. (Section 6.1
2. To demonstrate that the theory does indeed translate into an effective practical methodology embodied by the DDRSA-RNN that can model hazard rate processes and generate inter- ventions using the OTI policy. We call this combined approach the OTI-DDRSA-RNE We show that OTI-DDRSA-RNY outperforms static threshold policies that are either window based (WBI-RNN) or time-to-event based (TTE-RNN)(Section) 6.2).
Experiments performed on two real-world datasets to validate goals. Experiments third synthetic dataset are included Appendix C.
This dataset from NASA provides train and test data to predict Turbofan engine failures. The training set has simulated multivariate time-series observations from 21 sensors fault at some point resulting in eventual failure. The test set consists of 218 partial engine time-series with remaining useful life (in engine cycles) indicated in each case. We could not use the test data as provided since it is missing part of every time series close to failure.
Turbofan Engine Failure Data 26 32
fleet of 218 engines. Each engine operates normally the start and develops
Azure Predictive Maintenance Guide Data 25]: This is dataset from a guide provided by Mi- crosoft as case study in failure prediction. The data comes from multiple sources including time-series of voltage, rotation, pressure and vibration measurements collected from 100 machines in real time averaged over every hour, error logs, machine information (type, age etc.) and 720 failure records. We adopt the same pre-processing, feature generation as given in the guide. Also, we down-san down-sample the data to take one sample every 15 hours.
For both datasets, we train on 70% of randomly selected co-variate time-series sequences and hold out 30% of the sequences for testing. From the training set a further 30% of the sequences are set aside as validation data to tune model parameters and policy thresholds. This process is repeated to produce 10 random train-validation transforms training inputs to lie in the range [-1, 1]. Model selection details and parameter settings are discussed Appendix
splits. Observations are normalized with min-max transformation that
6.1 Discussion of OTI interventions on real data
We fit the DDRSA-RNE on the covariate time-series leading upto failure and predict on a single test set sequence. This produces, at each time step j of the sequence, an estimated conditional hazard rate vector H, (representing the conditional hazard rates into the future), which gives us probabilistic estimate of the onset of the critical event at each time step, taking into account co-variate history upto that time step. Figure 2 shows the evolution of estimated conditional hazard rate functions. We see that the onset of the critical event is clearly reflected in the progression of the estimated vectors H,.
8

---

120
expected
011 threshold when C=8
0.4
100
threshold when C-64
OTI threshold when C-128
80
rate
risk
60
0.2
F-L-00
---100
-1.50
0.0
--1-1
0
50
100
150
200
250
300
350
50
100
150
200
time steps from
time steps
(a) Progression of conditional hazard rates
(b) OTI policy in action
Figure 2: (a) Example of typical evolution estimated conditional hazard rates estimated by DDRSA as approaches the critical event. (b)The OTI policy in action. The OTI policy calls for intervention when estimated time-to-event falls below a varying policy threshold. As the cost of missing the critical event increases, interventions triggered earlier.
OTI
On
17.5
TTF
TTE
60
average to ovent
15.0
average time to event
12.5
10.0
40
30
20
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
0.5
event miss rate
event miss rate
(a) NASA turbofan data: policy trade-off plot
(b) Azure PM guide data: policy trade-off plot
Figure Performance comparison of OTI-DDRSA-RNN WBI-RNN and TTE-RNN policies. Trade-off plots showing the median (solid circle) and inter-quartile range (vertical bars) for each C. {8, 10..... 256) computed over 10 randomized folds.
Next, we demonstrate the working of the OTI-policy. Figure plots the estimated time-to-event at each time step, estimated from H. using equation (10). We also plot the adaptive threshold V/ (H,). calculated using equation (9) of Corollary 4.1.1 Attevvntio is triggered when our time-to-event estimate falls on or below the adaptive threshold. Consider what happens as the cost missing the critical event is varied. First, note that since time-to-event is independent of C., only the threshold changes. Increasing the cost of missing the critical event increases the threshold causing earlier intervention as would expect.
6.2 Performance Evaluation
In this section will evaluate the performance of the proposed OTI-DDRSA-RNY vs. two static threshold policy baselines: A window based intervention RNN (WBI-RNN) and a time-to-even, based intervention RNN (TTE-RNN). All RNNs share the same encoder architecture which is an LSTM with 128 step look-back and hidden state dimension of 16 units.
WBI-RNN architecture: The WBI-RNN has the same encoder as the DDRSA-RNN with the final encoder output Z, Figure passed to a dense (fully connected) layer to produce a single output probability per training sample. We train the model with binary cross-entropy loss between the

---

probability output and the binary state of event in a future window W of time steps. A threshold is used on this probability to trigger interventions once the probability greater than the threshold. The window size and threshold are selected by cross-validation (see Appendix B).
TTE-RNN architecture: The TTE-RNN uses the DDRSA architecture shown in Figure | and is learnt with the DRSA loss as discussed above. However, interventions are triggered with a static threshold that is set via cross-validation similar the WBI-RNN.
All methods were trained, validated and tested using randomized vriinvaalidation-tes splits. Model parameter initialization seeds also randomized across trials. Results reported are medians over the 10 test folds. WBI and TTE thresholds are tuned (individually, for 125 different settings of on each validation set using empirical intervention policy risk:
Ca {8, 10,.................. .00.-...00. ..6))))))))))
Remp (10)
1 N
Σ (LI-T))(44 N TH) +C
Σ 1(L ST) N
1
(13)
average time-to-ovent
event miss-rate
where (L = L Li, 75) are realized outcomes for the ith series.
Computational setup and inference time: All experiments were run on a Tensorbook Laptop with 32GB RAM and having a single NVIDIA GeForce GTX 1070 with Max-Q GPU. Training single dataset for 10 folds and 125 cost points took around hours. DDRSA inference time per time-step time was around 12 milliseconds. The corresponding OTI policy evaluation time per time-step was around 0,5 milliseconds.
Figure 3 summarizes the evaluation results in the form of policy trade-off curves. Each point represents plot of the median average time-to-even, vs the median event miss-rate, calculated for a particular setting of C. (the corresponding inter-quartile ranges are also shown vertical bars). Regarding the trade-off plots, lower frontier is preferred, since implies a lower average time-to-event for a given miss-rate and lower miss-rate for given average time-to-event. We that the OTI-DDRSA-RRE outperforms the static threshold policies on both datasets.
The quality of the OTI policy depends the accuracy of the underlying survival analysis. Note that our approach requires fitting a survival model for (k, at each time step over several values of k. It is possible that some of these are better estimated than others. Also, in contrast, a threshold algorithm can be tuned for a specific Ca on a validation set while there is no C. specific tuning for the OTI policy. Also, there is no direct way of setting Ca guarantee particular a. We are forced to try a range values for C. and pick the values with most acceptable trade-offs.
Limitations:
Conclusions
We consider the task of monitoring a system for the occurrence of critical event, by observing co-variates over time. The goal is to intervene in a timely manner. We formalize this for the first time the Optimally Timed Intervention (OTI problem) which seeks a policy that triggers interventions such that for given probability of missing the event, the expected residual time-to-event is minimized. We show that the OTI problem may be posed as optimal stopping problem which may be solved analytically under Markovian assumptions. A significant new insight is that the optimal policy any time is an analytically computable threshold on expected time-to event that depends only on the conditional hazard rate function at that time. Finally, based on the theory we develop practical RNN architecture (the OTI-DDRSA A-RNN) produce optimal interventions. Future directions include an analysis of the impact of inaccuracies in estimating the hazard rate process on the resulting intervention policy and the ability to focus the estimation of the hazard rate process to achieve optimal performance for a specified event miss-rate.
hazard rate process,
8 Acknowledgments
This work was initiated under a research agreement between HP Inc. and the Indian Institute of Science (IISc). We acknowledge Jerry Liu, M. Anthony Lewis, Matt Ellis, Scott Hallworth and Jennifer Hill at HP Inc. for their support and sponsorship of this work. Finally, we thank all the reviewers for their insightful comments that have helped improve the paper.
10

---

References
[1] Mean Residual Life Concepts and Applications in Reliability Analysis, pages 109-138. Springer New York, New York, NY, 2006.
[2] Azure ai guide for predictive maintenance solutions. 2018. https://docs. http:://docs.mirroofftt.o httpaaaccciiiiaaaiiineeaddadttte wwtpe//eweeppooees/oooooaaaaaaee prca predictive- maintenance.
[3] Odd Aalen, Ørnulf Borgan, and Hakon Gjessing. Survival and Event History Analysis: A Process Point of View. Springer, NY, 2008.
[4] A.B. Aries and A.N. Shiryaev. Optimal Stopping Rules. Stochastic Modelling and Applied Probability. Springer Berlin Heidelberg, 2007.
[5] Anand Avati, Kenneth Jung, Stephanie Harman, Lance Downing, Andrew Y. Ng. and Nigam H. Shah. Improving palliative care with deep learning. BMC Med. Inf. & Decision Making, 18(S-4):55-66 64, 2018.
[6] T. Aven and U. Jensen. Stochastic Models in Reliability, Stochastic Modelling and Applied Probability. Springer New York, 2013.
[7] Jay Bartroff, Tze Leung Lai, and Mei-Chiung Shih. Sequential experimentation clinical trials: Design and analysis. 2012.
[8] Sebastian Becker, Patrick Cheridito, and Arnulf Jentzen. Deep optimal stopping. J. Mach.
Learn. Res., 20.74:1-74:25,, 2019.
[9] Y.S. Chow, H. Robbins, and D. Siegmund. Great expectations: the theory of optimal stopping. Houghton Mifflin, 1971.
[10] D. R. Cox. Regression models and life-tables. Journal of the Royal Statistical Society. Series (Methodological), 34(2):187-220, 1972.
[11] T.S. Ferguson. Muthematical Statistics: Decision Theoretic Approach. Probability and mathematical statistics. Academic Press, 1967.
[12] Eleonora Giunchiglia, Anton Nemchenko, and Mihaela van der Schaar. Rnn-surv: A deep recurrent model for survival analysis. In Artificial Neural Networks and Machine Learning ICANN 2018, pages 23-32. Springer International Publishing, 2018.
[13] loana Giurgiu, Jácint Szabó, Dorothea Wiesmann, and John Bird. Predicting DRAM reliability in the field with machine learning. In Proceedings of the 18th ACM/FIP//SENIX Middleware Conference: Industrial Track, Las Vegas, NV, USA, December 11 15, 2017, pages 15-21, 2017. [14] Frank Guess and Frank Proschan. Mean residual life: Theory and applications. In Quality Control and Reliability, volume 7 of Handbook of Statistics, pages 215-224. Elsevier, 1988.
[15] Daniel Jarrett, Jinsung Yoon, and Mihaela van der Schaar. Match-net: Dynamic prediction in survival analysis using convolutional neural networks. 2018.
[16] Daniel Jarrett, Jinsung Yoon, and Mihaela van der Schaar. Dynamic prediction in clinical survival analysis using temporal convolutional networks. IEFE Journal of Biomedical and Health Informatics, 24:424-436, 2019.
[17] Jared Katzman, Uri Shaham, Alexander Cloninger, Jonathan Bates, Tingting Jiang, and Yuval Kluger. Deep survival: A deep cox proportional hazards network. CoRR, 0bs/1606.009) 2016.
[18) Changhee Lee, Jinsung Yoon, and Mihaela der Schaar. Dynamic-deephit: A deep learning approach for dynamic survival analysis with competing risks based on longitudinal data. IEEE Transactions Biomedical Engineering, 67:122-133, 2019.
[19] Changhee Lee, William Zame, Jinsung Yoon, and Mihaela van der Schaar. Deephit: A deep learning approach to survival analysis with competing risks. 2018.
[20] You-Luen Lee, Da-Cheng Juan, Xuan-An Tseng, Yu-Ting Chen, and Shih-Chieh Chang. De- prophet: Predicting catastrophic machine failures in datacenters. In Machine Learning and Knowledge Discovery in Databases European Conference, ECML PKDD 2017, Skopje, Macedonia, September 18-22, 2017, Proceedings, Part III, pages 64-76, 2017.
[21] Mackenzie, John D. Kalbfleisch, and Ross Prentice. The Statistical Analysis of Failure Time Data 1980.
11

---

[22] V. Makis and Xiaoyue Jiang. Optimal replacement under partial observations. Math. Oper. Res., 28:382-394, 05 2003.
[23] C. Martinez, G. Perrin, E. Ramasso, and M. Rombaut. A deep reinforcement learning approach for early classification of time series. In 2018 26th European Signal Processing Conference (EUSIPCO), pages 2030-2034, 2018.
[24] Egil Martinsson. Wite-rnn: Weibull time to event recurrent neural network. Master's 's the- sis, Department of Computer Science and Engineering (Chalmers), Chalmers University of Technology, 2017. 103.
[25]
Microsoft. Dataset from azure modeling guide for predictive maintainence. https: httes:///itie //tp//btttubbb com/microsoft/ AtMoottshh/Mooooohphpoop 2017.
[26] NASA. Phm08 challenge data set. httpt::t////tt //ti arc. www.:aaarc.aaaaaggoo. wwt.r///tttcccaaaaaarrrpppeeeeee
progiostic-data--repiiooo
2008.
[27]
H. V. Poor and O. Hadjiliadis. Quickest Detection. Cambridge University Press, 2008.
[28]
Rajesh Ranganath, Adler Perotte, Noémie Elhadad, and David Blei. Deep survival analysis. In Finale Velez, Jim Fackler, David Kale, Byron Wallace, and Jenna Wiens, editors, Proceedings of the 1st Machine Learning for Healthcare Conference, volume 56 of Proceedings of Machine Learning Research, pages 101-114, Children's 's Hospital LA, Los Angeles, CA, USA, 18-19 Aug 2016. PMLR.
[29] Kan Ren, Jiarui Qin, Lei Zheng, Zhengyu Yang. Weinan Zhang. Lin Qiu, and Yong Yu. Deep recurrent survival analysis. volume 33, pages 4798-4805, 07 2019.
[30] S. W. Roberts. A comparison of some control chart procedures. Technometries, 8(3) 1966.
[31] Felix Salfner, Maren Lenk, and Miroslaw Malek, A survey of online failure prediction methods. ACM Comput. Surv., 10:110 2010.
[32] A. Saxena, K. Goebel, D. Simon, and N. Eklund. Damage propagation modeling for aircraft engine run-to-failure simulation. In 2008 International Conference on Prognostics and Health Management, pages 1-9, 2008
[33] Thomas H. Scheike and Torben Martinussen. Dynamic Regression models for survival data. Springer, NY, 2006.
[34] A. Shiryaev. On optimum methods in quickest detection problems. Theory of Probability & Its Applications, 8(1):22-46, 1963.
[35] A.N. Shiryaev. Stochastic Disorder Problems. Probability theory and stochastic modelling. Springer International Publishing, 2019.
[36] Stephan Spiegel, Fabian Mueller, Dorothea Weismann, and John Bird. Cost-sensitive learning for predictive maintenance. CORR, (bss1809.10979 2018.
[37] A. Wald. Sequential Analysis. Wiley Series in Probability and Mathematical Statistics: Proba- bility and Mathematical Statistics. J. Wiley & sons, Incorporated, 1947.
[38] J. Wang, C. Li, S. Han, S. Sarkar, and X. Zhou. Predictive maintenance based on event-log analysis: A case study. IBM Journal of Research and Development, 61))1:11, 2017.
[39] I.. J. Wei. The accelerated failure time model: A useful alternative the regression model
in survival analysis. Statistics in Medicine, 1(14-15):1871-1888
1992.
[40] Chun-Nat Yu, Russell Greiner, Hsiu-Chin Lin, and Vickie Baracos. Learning patient-specine cancer survival distributions as sequence of dependent regressors. In Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 201 I. Proceedings of a meeting held 12 12-14 December 2011, Granada, Spain., pages 1845 1853, 2011.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Section 6.2
(c) Did you discuss any potential negative societal impacts of your work? [N/A] No obvious negative societal impact
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? (Yes) See Sections
2.1, 2.2, and statements of Proposition 3.1 and Theorem 4.1
(b) Did you include complete proofs of all theoretical results? [Yes] Proof sketches in main paper and detailed proofs in the Appendix A, included in supplemental material.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or URL)? [No] The code is proprietary. Links to the data are given [25,26.32].
(b) Did you specify all the training details (e.g.. data splits, were chosen)? [Yes] Section 6.2 of the main paper and Appendix of supplemental material
how they
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] Section 6.2
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] Section 6.2
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Cited in [25,26.32].
(b) Did you mention the license of the assets? [No] The links to data provided in the paper have the license details
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A] Only free license resources used
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A] We do not use such data.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "WL0wY-40m0S", "reviewer": "Reviewer_mWmk", "summary": "This paper studies the problem of deciding when to intervene and “stop” a stochastic process before it transitions to a critical event. A new problem setting is defined called optimally timed intervention (OTI), where the goal is to minimize the expected time between intervention and the event L under the constraint that the probability of missing is small. An optimal policy is derived through the introduction of survival and hazard processes. The theory is then used to motivate an RNN approach to predict these processes. The method is evaluated on real-world data and appears to marginally outperform a baseline and significantly outperform a naive strategy.", "questions": "- Is it true that the original problem can be equivalently formulated as a standard optimal stopping problem (before even considering the survival and hazard processes)? In that case, what prevents one from directly using existing tools from optimal stopping and how would they compare with the optimal policy derived in this paper through the survival/hazard processes?\n- Can the authors clarify the novelty of the survival and hazard processes? These seem to be presented as new ideas, but also the paper by Ren et al also seems to use the hazard process.\n", "limitations": "Limitations were addressed.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- This is not within my area so I cannot comment on the significance compared to related work, but the paper seems to do a good job differentiating itself.\n- The OTI seems to be an interesting and well-motivated problem to study (again I can not really comment on novelty)\n- The theory about computation of the optimal policies given the known survival/hazard rates is also fairly comprehensive.\n- The experiments are promising, suggesting that the theory and insights developed in the paper have strong practical implications.\n\nWeaknesses\n- It doesn’t seem that it is ever described formally what the relationship is between the covariates of the process and the event time L. All that is known in the problem definition is that L is some random variable and a stopping time. Nevertheless in the experiments this is implicitly posed as a supervised learning problem where the covariates of the stochastic process are somehow predictive of L. While this makes intuitive sense and seems to be sufficient for the experiments, I think it would be helpful to make this more rigorous so that we can reason about familiar characteristics of statistical learning problems like realizability, noise, etc. In general the shift from the OTI theory to the experiments is a bit abrupt and I think it would have been helpful to explain in math more clearly how that jump is made. Some of these details are in the appendix already, but it would be helpful to bolster the formalization and have them in the main paper.\n- It doesn’t seem that one can “ask” for a specific probability of failure (missing L) and then pay the price of this ask in the expected time to intervene, like one would expect to see in many similar learning/decision problems. Instead this is controlled implicitly by $C_{\\alpha}$ which is a hyperparameter that must be set.\n- I think that the experiments could be a lot more compelling if there were a plot showing the trade-off between the probability of failure and the expected time to intervene so that one could see visually what shape this trade-off curve takes (by setting different values of $C_{\\alpha}$). Perhaps this is not feasible with the real-world data. However, even synthetic experiments could suffice to exhibit this.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "OclXuDTKBwN", "reviewer": "Reviewer_vH1X", "summary": "The paper formulates and studies the problem of optimally timed intervention (OPI) for the first time. More specifically, the problem is formulated as minimizing the expected time-to-event while keeping the probability of missing the critical event below a certain threshold. The authors show that the OTI problem is equivalent to an optimal stopping problem on the underlying survival process. Other theoretical characterizations are also performed showing that the optimal decision rule at each time step can be analytically determined by the conditional survival function and that the optimal policy is non-stationary in general (so static threshold policies are often sub-optimal). The authors also propose a dynamic deep recurrent survival analysis (DDRSA) architecture. Experimental results on two real-world datasets show that the proposed method provides a better tradeoff between the average time to event and event miss rate compared to some baselines.", "questions": "- Is the proposed OTI policy applicable only to the RNN model, or can it be more flexible? If it is more flexible, I thought it would be interesting to see if the proposed framework can work with a range of models to estimate the hazard rate function rather than focusing only on RNN. \n\n- the OTI policy is expressed with the true hazard rate function for now. Would it be possible to theoretically characterize the accuracy of the function estimation and the performance of the resulting policy? In other words, how does the error of the function estimation propagate to the performance of the intervention policy?\n", "limitations": "I was unsure why it is necessary to focus only on the RNN model, even though the high-level OTI policy framework seems to be more flexible in my understanding. Then, it would be more comprehensive to show this flexibility in the experiment rather than focusing only on a single model. ", "rating": 5, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- formulate the OTI problem and link it to both survival analysis and optimal stopping theory for the first time, deriving the analytical expression of the optimal intervention \n- propose a method based on recurrent neural networks to lead an OTI policy\n- promising empirical results of the proposed method compared to some minimum baselines\n- the paper seems well written and the contributions were easy to follow\n\n\nWeaknesses\n- it seems that the experiment focuses only on RNN rather than trying to show the flexibility of the proposed framework. I could not see any particular reason why it is better to focus on RNN even though the proposed framework in section 3 does not necessarily rely on a particular model  (at least in my current understanding)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5bSvhKYnjmM", "reviewer": "Reviewer_KFJC", "summary": "This paper studies the problem of triggering stopping interventions in dynamical systems before a critical event occurs, named the optimally timed intervention problem (OTI). The authors formulate this problem as that of minimizing the expected time to event (remaining time lost when the intervention is triggered before the critical event), while also minimizing the expected cost of an event happening (fixed cost when no intervention is triggered before the critical event). They show that two popular approaches, window-based intervention (WBI) and time-to-event (TTE), are sub-optimal for OTI, and they propose a new approach similar to TTE but with dynamic thresholds, which they show is optimal. They conduct experiments on two datasets which also support the superiority of their approach.", "questions": "I would like the authors to clarify the following points:\n - how does your derivation of the optimal intervention $\\tau^\\star$ translate into an optimal policy ?\n - what is a stationary process ? A stationary policy ?\n - given that any Markov process can be made homogeneous by including the index $j$ into each observation $X_j$, what is the point of considering non-homogeneous processes ?\n\nDetailed comments:\n\np1 l21: networks(RNNs) -> typo\n\np2 l38-44: This whole description is a bit vague. \"The critical event occurs\" -> So there can only be one such event ? \"The sequence of observations terminates after the occurrence of the critical event\" -> what if there is no critical event ? Is there always a critical event ? If proper interventions happen at the right time, can't the system be kept alive for ever ? It seems what you describe does not allow that. \"An intervention is considered late [...] if no intervention occurs\" -> you say a late intervention is when there is no intervention... This sentence is contradictory by itself. I'd suggest to rework and clarify your setup description.\n\np2 l47: $\\mathcal{X}_j \\to \\\\{0,1\\\\}$ -> this notation is inconsistent. Earlier you define $\\mathcal{X}_j$ as a single point in $\\mathbb{R}^{j \\times d}$. Now you use the same notation to depict the input space of a function.\n\np2 l53: again, the math here is quite loose and ambiguous. The index $j$ is defined twice in the left-hand-side and right-hand-side of the equation. On the left $j$ is given and constant, on the right $j$ is a free variable in a minimization problem. Are $\\mathcal{X}_j$ on the left and on the right the same objects ? If not, how can $\\tau$ be a function of $\\mathcal{X}_j$ ? Also, what about the case where there is no intervention ? If the process reaches a terminal state $X_L$ without any intervention, then which value does the policy stopping time take ? Finally, the random variable $\\tau$ does not depend only on a history at time $j$, it depends on the entire realization of the stochastic process and the policy $\\phi$. Expressing $\\tau$ as a function of $\\mathcal{X}_j$ is really misleading.\n\np2 l58-60: This definition of OTI seem arbitrary. A motivation is missing, as to why such an objective makes sense in practice, or a reference to the literature if this objective is commonly accepted.\n\np2 eq1: Again, this equation is ill-defined. First, what is $L_\\textit{max}$ ? What is $(L-\\tau)^+$ ? These are not defined. Second, why isn't this  expression conditioned on the history $\\mathcal{X}_j$ ? As it is written, equation (1) suggests that you are looking for a universal optimal time to intervene, $\\tau^\\star$, regardless of the current realization of the process. Surely this is sub-optimal, and at any time step $j$ the decision to intervene or not should be conditioned on the history $\\mathcal{X}_j$, which can reduce the uncertainty about $L$.\n\np2 l67:  can be directly be -> typo\n\np3 l120: \"system evolution towards a critical event is a non-stationary in general\" -> I hardly get the meaning of that statement. Do you mean that your training or testing data are not i.i.d. realizations of random variables ($X_j$, $L$) ? Do you mean that each random variable $X_j$ follows a different distribution ? Is stationarity equivalent to homogeneousity ? A proper definition of stationarity is missing here.\n\np4 eq2: According to your setup description in Section 1.1, the existence of a history $\\mathcal{X}_j$ implies that the process is not terminated yet, and hence $L>j$ is always true (or $L \\geq j$, that is ambiguous from the definition of $L$). Thus I do not understand why the survival function is conditioned on $L>j$ in addition to $\\mathcal{X}_j$ (or why the hazard rate is conditioned on $K \\geq j$), as it does not bring any additional information.\n\np4 l160: From eq (2) I even have $S_j(0)=1$.\n\np4 l164: Again, $L_\\textit{max}$ has not been defined yet.\n\np4 prop2.1: It seems that the notions of survival process and hazard rate process can be used interchangeably to define your OTI problem. I'd suggest to choose and stick to one of the two notions, this would simplify the manuscript a lot.\n\np5 eq5: I think the definition is missing $\\forall j$. Otherwise this statement does not make much sense.\n\np5 l186: Is homogeneousity your definition of stationarity ? Furthermore, it seems that any Markovian survival process can be made homogeneous trivially, by including the index $j$ into each observed random variable $X_j$. What is the point of considering non-homogeneous Markov processes then ?\n\np5 l189: risk process risk process -> typo\n\np5 eq6: It is unclear here how this random variable $\\tau^\\star$ is related to a decision rule. At each time step $j$, a decision rule has to decide whether to stop or not, given only information from the past ($\\mathcal{X}_j$). In equation (6), the optimal stopping time depends on the entire realization of the process (every $j$ has to be considered, in the right-hand-side of the equation). A clarification is missing, as to how equation (6) translates into a proper decision rule.\n\np5 eq7: $\\prod_{m=0}^k (1-h_j(l+k))$ -> the $m$ index does not appear inside the factors here. This product does not make sense.\n\np5 l193: the optimal policy is stationary -> what does it mean for a policy to be stationary ?\n\np6 l239: $\\mathbf{H}_0)$ -> typo\n\np6 l241: \"produce conditional hazard rate\" -> there is a limitation here to how many hazard rates your model can output, which directly depends on the model architecture and in particular the width of the final dense layer. This limitation is not discussed.\n\np6 l241: \"a novel sequential loss\" -> a description of this novel loss function is missing.\n\np8 fig2: dynamicaly -> typo\n\np8 l287: If in the end you model everything with hazard rates, which can be equivalently represented as survival functions, then I really suggest that you use only one of the two concepts in the paper. I do not see the additional value of expressing all your results using both concepts each time. This really hinders the readability of the paper.\n\np8 l292: If the OTI policy can be equivalently expressed as an adaptive threshold on the estimated time-to-event, then why bothering introducing hazard rates and survival functions in the first place ? This make me feel like the presented theoretical results are stated in an overly complicated fashion.\n\np9 fig3: These figures are missing information about the variance, such as boxplots or standard deviation curves.\n\np9 l320: Allmodels -> typo", "limitations": "Limitations are discussed very briefly. I see several limitations that are missing in the discussion. How should the cost C be chosen in practice ? What is the affect of the size of the predicted hazard rate vector ? What is the computational complexity of the proposed approach at test time ?\n\n### rebuttal update\n\nI have been through the author's response to my comments. I appreciate that the authors have now substantially simplified the paper, notably by moving the analysis in terms of survival process (a somewhat duplicate of the analysis in terms of hazard rate process) to the appendix. I also note that the authors have fixed several mathematical typos and errors from the original manuscript that I had pointed to.\n\nHowever, I have also been through the updated manuscript, and I still find the mathematical notations and definitions to lack somewhat loose and ambiguous. I could not check in detail the entire revised paper due to the amount of changes and the limited time, but I have concerns already after reading the first few pages:\n\n - p2 l41: but bounded time-step $L < L\\_{max}$ in the series. -> typo ? but at a bounded time step ?\n\n - p2 l43: before the the policy stops -> typo. Moreover, I still find this whole definition rather vague. What does \"the policy stops\" mean ? How different is that from saying \"before an intervention\" ? Does the policy stop / intervention necessarily triggers the end of the time series ? After an intervention at time step $\\\\tau$, is there still a final observation $X\\_{\\\\tau + 1}$ ? If $X\\_{L+1}=0$ indicates the end of the time series, then is this specific value forbidden, in the domain of the random variables $X\\_j$ ? Despite the changes made, this whole definition still appears rather sloppy to me, which is problematic for a theoretical paper.\n\n - p2 l44: we can take $\\\\tau=L+1$ -> At this point $\\\\tau$ has not been introduced yet. Also, why not considering that the policy stops immediately, with $\\\\tau=L$ ? This would seem to make more sense, given that you consider interventions at $\\\\tau=L$ too late by definition.\n\n - p2 l50: Definition 1.1 -> First a suggestion. Because $j$ is somehow embedded into $\\\\mathcal{X}\\_j$ (it is the length of the history), why not simply defining your intervention policy as $\\\\varphi(\\\\mathcal{X}\\_j)$, instead of a sequence of $j$-indexed functions ? This would simplify a lot an otherwise heavy notation. Second, I again find the math a bit sloppy here. Earlier you defined $\\\\mathcal{X}\\_j$ as a random variable, but here in $\\\\{\\\\mathcal{X}\\_j\\\\} \\\\to \\\\{0,1\\\\}$ you seem to use it as the input space of a mapping. I understand the intention, but again this is a rather loose notation. A random variable and its support are not the same thing.\n\n - p2 l53: this means that intervention triggers after the first trigger are ignored -> typo ?\n\nAgain, I am not an expert in the field of OTI, but at the very least I should be able to read the problem setup unambiguously. To me the overall technical quality of the paper remains below the NeurIPS threshold.\n\nTaking all of that into account I am willing to increase my rating to 4, but my recommendation remains rejection.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Originality\n\nThe proposed approach seems original, although I am not very familiar with the OTI problem.\n\n### Quality\n\nWhile the paper appears quite technical, I found the technical quality rather poor. The mathematical definitions used in the paper are loose and ambiguous at times, which affects readability and also brings doubt about the rigor of the presented results. Some parts of the reasoning are missing, such as how the optimal intervention $\\tau^\\star$ (Theorem 3.1, Corollaries 3.1.1 and 3.1.2) translates into an actual policy, or what the authors mean by a stationary process or a stationary policy. The experimental setup seems ok, but the experimetns are lacking a bit in size (only 2 datasets, only 5 and 3 values respectively for the cost $C$ for each dataset), and in presentation (no error bars or standard deviation).\n\n### Clarity\n\nI found the paper hard to read, due to several ambiguities in the notations and definitions. I also found the presentation of the proposed approach unnecessarily heavy and redundant. The authors derive their method twice from two equivalent concepts: the hazard rate and the survival function. I believe the presentation could be greatly simplified. Finally, the paper is missing a conclusions section.\n\n### Significance\n\nThe presented analysis and approach might be significant to the OTI community. Still, the technical quality and presentation of the results is too poor to correctly assess significance in my opinion.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-v3w12nlfBg", "reviewer": "Reviewer_sQwG", "summary": "This paper studies the task of finding optimal intervention time for failures. The problem can be viewed abstractly as a sequence of stochastically generated features. A failure happens at some point for this task, and the goal is to intervene optimally. This is done by a policy that either allows the process to run or decides to intervene at a given time. Optimal intervention is framed as deciding to intervene such that one does not intervene too early before the actual failure event, while also ensuring that the probability of late intervention (i.e., intervention after the failure) is below a threshold. \n\nTwo processes are then defined: a survival process and a hazard process. The survival process models the probability that failure will happen after a certain time from the current time. The hazard process $h_j(k)$ models the probability that failure will happen at a given time $j+k$ in the future when the current time is $j$. The optimal intervention time problem is then framed as finding the optimal stopping problem on the survival/hazard process. Under assumption that features evolve using Markovian dynamics, the optimal stopping problem can be solved. \n\nExperiments are presented on two datasets and compare with two baselines. The model used for prediction is a  Recurrent Neural Network that converts a history of features $\\{X_1, \\cdots, X_j\\}$ into a vector representation. This vector representation is used by a decoder to generate a sequence of vectors $\\{Z_{jk}\\}_{k}$ that is then directly converted to the hazard rates $h_j(k)$. Experiments show that broadly the new approach outperforms the baselines on both dataset.\n\n ", "questions": "1. How is the DRSA loss computed? Is my understanding correct that the model is trained using DRS, then kept fixed, and used to solve equations in theorem 3 to find the optimal stopping time? As mentioned earlier, these experimental details would be helpful.\n\n2. How will be $C_\\alpha$ set in practice. This hyperparameter is less intuitive than $\\alpha$ which might be easy to set based on domain knowledge.\n\n3. Is there any particular reason why $(L-\\tau)^+$ loss is used instead of square loss or any other loss?\n\n4. Why is TTE seem to be doing better than OTI for large values of $C$ in Figure 3a?\n\nMinor presentation:\n\n1. \"Corrolary\" spelling error on Line 294. It should be \"corollary\" \n\n2. Space in \"Allmodels\" on Line 320\n\n3. In Figure 1, the sub-captions for the two figures seem to be wrong. ", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength:**\n1. Problem is relevant to machine learning community from both theoretical and real-world impact\n2. Solution is novel and well-motivated\n3. Paper is mostly well-written\n\n**Weakness**\n1. The paper can use more details on the experiments. E.g., how the RNN model is used to make predictions will be useful and how is the RNN model trained, and are the equations Theorem 3.1 solved exactly or approximately? Details of computational aspects can also be useful since if the approach takes a long time to predict then it may be not useful for real-world applications where intervention must be done in real-time. \n\n2. Discussion of Theorem 3 would be useful. Perhaps the other theorem can be removed to make space to discuss theorem 3 in more detail. E.g., a proof sketch can be provided, or a discussion of the theoretical results.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept