Beyond Average Value Function in Precision Medicine: Maximum Probability-Driven Reinforcement Learning for  Survival Analysis
Authors: Jianqi Feng
Chengchun Shi
Zhenke Wu
Xiaodong Yan
Wei Zhao
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement Learning; Survival Analysis; Recurrent Events
Primary area: general_machine_learning
TL;DR: A novel RL objective maximizing probabilities for recurrent event data is proposed, achieving lower variance and faster convergence compared to traditional methods.

Abstract: Constructing multistage optimal decisions for alternating recurrent event data is critically important in medical and healthcare research. Current reinforcement learning (RL) algorithms have only been applied to time-to-event data, with the objective of maximizing expected survival time. However, alternating recurrent event data has a different structure, which motivates us to model the probability and frequency of event occurrences rather than a single terminal outcome. 
In this paper, we introduce an RL framework specifically designed for alternating recurrent event data. Our goal is to maximize the probability that the duration between consecutive events exceeds a clinically meaningful threshold. To achieve this, we identify a lower bound of this probability, which transforms the problem into maximizing a cumulative sum of log probabilities, thus enabling direct application of standard RL algorithms. We establish the theoretical properties of the resulting optimal policy and demonstrate through numerical experiments that our proposed algorithm yields a larger probability of that the time between events exceeds a critical threshold compared with existing state-of-the-art algorithms.

Beyond Average Value Function in Precision Medicine: Maximum Probability-Driven Reinforcement Learning for Survival Analysis
Jianqi Feng' Shandong University 202412070
Chengehun Shi LSE
Zhenke Wu
University of Michigan
C. shi78lss ac.uk uk
zhenkewue umich edu
Xiaodong Yan Xi'an Jiaotong University
Wei Zhao
Shandong University
yanxiaodongexxtu..om edu.
wzhao928sdu edu.cn
Abstract
Constructing multistage optimal decisions for alternating recurrent event data is critically important in medical and healthcare research. Current reinforcement learning (RL) algorithms have only been applied time-to-event data, with the objective of maximizing expected survival time. However, alternating recurrent data has a different structure, which motivates frequency of event occurrences rather than a single terminal outcome. In this paper, introduce RL framework specifically designed for alternating recurrent event data. Our goal is to maximize the probability that the duration between consecutive events exceeds a clinically meaningful threshold. To achieve this, identify a lower bound of this probability, which transforms the problem into maximizing a cumulative sum of log probabilities, thus enabling direct application of standard RL algorithms. We establish the theoretical properties of the resulting optimal policy and demonstrate through numerical experiments that our proposed algorithm yields a larger probability of that the time between events exceeds a critical threshold compared with existing sfate-of-the-ae algorithms.
model the probability and
Introduction
Motivation. Precision medicine is a paradigm that aims to tailor treatments to individual patient characteristics. Recent studies have increasingly developed reinforcement learning (RL) algorithms in this context (see, e.g., 28 33 48 52 49 53 58
4 7 56) 57). This paper focuses on RL for survival analysis. The existing literature has primarily studied time-to-event data, with the objective of maximizing expected survival time (see Section I. for details). However, for patients with chronic diseases, data often involves alternating recurrent events, and the clinical goal shifts to maximizing the probability that the duration between events exceeds critical threshold. In response, we propose an RL framework designed to maximize this probability. Below, we provide an example to illustrate the structure of alternating recurrent event data.
35
36
31 25
50 26,
Our work is motivated by the ongoing Intern Health Study (IHS) which recruited first-year medical interns at US institutions to study factors that may impact their mental health and general well-being (29). The medical internship an initial step towards trained and practicing physicians. They
authors contributed equally and are listed alphabetical order by surname.
1 CCoresponding Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

often face difficult decisions, challenging shift work schedules, lack of time for exercise, and sleep disruptions, resulting in higher rates of depression (34). Wearable and smartphone devices provide daily ecological momentary assessments about physical activities, sleep, and mood scores over multiple months since the start of the internship. Digital health interventions sent via prompts on the study app serve as low-touch and easily accessible approach to delivering data insights, tips, and other personalized content for mitigating mental health issues especially when access to therapists may not always be timely [9 43). It has been an ongoing cohort that facilitates the study of genetic and lifestyle risk factors for mental and behavioral health outcomes. this paper, we focus on the management of recurrent www- episodes, which requires shift toward dynamic, personalized interventions. For example, we may define a low-mood episode by a significant drop (e.g., 20%) from an individual's ' baseline mood before internship. Moving beyond prediction, RL offers a powerful computational framework to learn optimal, adaptive intervention policies. The objective is to train an agent to select sequences of actions that either preemptively steer individual away from an impending low-mood state or, should an episode occur, expedite their return to baseline, thereby minimizing the episode's 's duration and severity.
Challenges. Applying RL to alternating recurrent event data raises two challenges:
Alternating recurrent event data is substantially more complex than the time-to-event data, the latter being the focus of the current literature (14) 47). Unlike time-to-event data, alternating recurrent events can switch back and forth multiple times during the follow-up period, which substantially increases the modeling complexity.
Most existing RL algorithms consider maximizing the expectation. However, in survival analysis, the primary interest often lies in the probability that the duration between recurrent events exceeds a specified threshold, a perspective that has received limited attention in the existing literature.
Contributions. To address these challenges, we propose a novel crntnnuuu--oobbbbittt-tyee algorithm tailored to alternating recurrent event data. Our primary methodological contribution is the derivation of a lower bound for the Baxximum-probabille original objective into sum of cumulative log probabilities. This enables the application of standard RI algorithms designed to maximize cumulative reward in the context of recurrent event data. We further provide theoretical guarantees and conduct extensive numerical experiments demonstrate the effectiveness of the proposed algorithm.
RL
objective, which reformulates the
1.1 Related Work
Our paper is closely related to two strands of research in survival analysis: one focusing on the use of RL for survival analysis, and the other handling recurrent event data.
RL for survival analysis. RL has recently received considerable attention in the survival analysis literature with censored outcomes (I1). Specifically, Goldberg & Kosorok (14) and Liu et al. (23) construct Q-function suitable for time-to-event censored data by inverse probability weighting. To address the issue of unequal numbers of individuals at each stage, they construct an auxiliary problem and provide an unbiased analysis the optimal policy. Zhao et (54) offers a doubly robust estimation method based on this work. Liu et al. (22) uses the estimated Q-function values to fill in censored values, but no theoretical analysis is provided. Lee et al. (17) uses deep learning to learn the distribution of survival time for analysis. However, these methods are applied to time-to-event data and focus on maximizing expected value of survival time.
Recurrent event data. Given the recurrent nature of chronic diseases, the more recent trend of the existing literature has moved beyond the time-to-even data handle recurrent event data. Among those available, Lee et al. (18) and Wang et al. (44) propose to use counting process for modeling recurrent events and estimate the gap time between recurrent events. Xia et al. (45) and Loe et al. (24) propose to segment recurrent events within follow-up windows and estimate the probability of the duration of recurrent events considering all stages. However, these papers do not study policy optimization.

---

2 Data Structure and Optimization Objectives
In this section, we introduce the data structure of alternating recurrent events and our considered optimization objective. As mentioned earlier, unlike most existing studies in the literature, we focus on maximizing the probability that the duration time exceeds pre-specified threshold.
Data structure. We consider multistage decision-making process based on a censored recurrent event structure. Let Gk (k = ..) be the duration of the k-th follow-up stage. For each stage, let X E X and A₁ E AA respectively be the covariates collected and the treatment received at the beginning of the k-th stage, where X' C. R" is the set of states and A is compactness set of available treatment options. The observed values covariates X₁ and treatment A are denoted as ay and (t,, respectively. Suppose that each subject's follow-up period begins at time O. We consider subjects who can experience two types of alternating events that recur in multiple follow-up stages (45). We denote L; the i-th occurrence time from high mood low mood (first-type event) and H, as the i-th occurrence time from low mood to high mood (second-type event). Suppose that and Lo Ho O.
S Li H1 L2 H2 <
In this work, we are interested in the following recurrent event duration T₂ for stage k, which represents the duration from the occurrence of a second-type event to the next first-type event. The formula for Th is given by
Jk
k-1 - G.,0 + max EG. H 00 c....
min
(L++1 11:) + max
-
==1
where J/ arg (1x S Ex= G.) denotes the time of the last second-type recurrent event 0 for b a. Since Iy and H, are random
occurring stage k. We assume that Jo = 0 and Σ variables, we assume here reward sample space R such that Tg R. Let C E R represent the censoring time measured from the beginning of the study until a censoring event occurs, such as the end of the study the loss of follow-up, then the observed duration time stage k is given by Y& (Tk C Ek=l Ti} and the corresponding censoring indicator stage k is denoted by A = 1 T.S.). C). We assume that the censoring time is independent of covariates, treatments, and duration times for simplicity. Note that the censoring event can oceur at any stage. Once the censoring event occurs at stage k, the treatments, covariates, and duration times after stage k are not observed. Let the number of stages of treatment received by subject be K N+, where K = inf A 0}. With notations above, the observed individual trajectories for one subject can be represented (subject index omitted): (X1, A1, A1, Yi,..., X R AR. Ag. YR)-
In Figure D we illustrate an individual trajectory from the motivating Intern Health Study. In this study, treatment policy needs be developed to determine whether send weekly text messages to medical interns to maintain high mood levels for a longer duration. As illustrated, the shifts between high mood and low mood occur repeatedly. In this example, G₁ is consistently (days) for any and T3 of primary interest represents the duration of the high-mood period within each week. For this specific individual, a censoring event occurs in stage resulting in K 3 and Y/3 being observed instead of the true T3.
X
Qg
Stage 1
4
Stage 2
42 1
Stage
4
0
High Mood
Ti
T2
Ta
H
Ls
L
"H
Low Mood
Y1
Y2
Y
G,
G2
G3
Ty: Actual duration of stage
Ye: Observed duration stage
Ag: Censoring indicator at stage k
Figure 1: An example for the individual trajectory in the Intern Health Study.
3

---

Optimization objective. In traditional RL. the strategy is often formulated with the objective of the value function EE Po Pr E k Tx (,), where Po is the transition kernel that maps state space X and action space A to distribution over state space X, π is policy that maps state space A' to probability distribution over the action space A, and Pr the distribution on R under condition of X' and A. Moreover, in health studies, we place more emphasis on the probability that the duration time is greater than a threshold at each stage. Therefore, we propose the following objective:
Pd (2+) = PP.. (Te a.Gr..... Tg > G X; =x))
where ag [0. 1]. for k = -..... K. In the context of the Intern Health Study illustrated in Figure I setting ag = 4/7 for all k < K implies that we aim for interns to maintain high mood for at least four days per week throughout the follow-up period. However, if we expect interns to have better mood in the first few weeks, 0g 's can be increased slightly during those weeks. See Appendix A.3 for further discussion of or selection.
To handle (T) we first impose two assumptions (Assumption 2.1 2.12.22 These assumptions allow us to decompose ( into a product of stage-specific survival functions, resulting in simplified objective function (see Lemma 2.3).
Assumption 2.1 (Markov assumption). For any v 2 0 and k > t, assume that
P(T > =|[T,) (x.)= (A.)=_ ( ul x Ax).
Assumption 2.2 (Conditional independence assumption). For any k: > L, we assume that (XR, As) and {Te}s=] are conditional independent given (x)) Lemma 2.3. When Assumptions 2.1 and 2.2 hold, we have
R II k=t
pro (T₃ a.G......... TR > agggaa G X2) Po.
ST. Ga Ak)
X,
(2)
where ST. (-1X;; Ax) is the conditional survival function of the duration time Te given Xx and Ak-
The proof of Lemma 2.3 is provided in Appendix B.1 However, directly optimizing (T(x)) is challenging because evaluating the objective requires computing the product of probabilities from
k == /. to K, which can become extremely small when K is large, making the resulting optimization unstable. To address this, we observe that, by Jensen's 's inequality,
Vi" (xx) == Po.sT
Σ
log ST:
X. = xz S log E Po.ar
II ST X; X
log P (xt), (3)
where ST, is shorthand for ST, (ar,Ga| A&) and we denote the lower bound of log PT (æx) by Vi (æ:). (3) motivates us to maximize this lower bound of log PT, which equals a sum of expected cumulative log probabilities and can thus be optimized using existing RL frameworks. A natural question arises: does this shift in the objective alter the optimal policy? Theorem [.40rem2.4]]llow answer this question, showing that the two objectives induce the same optimal policy when Po is deterministic transition kernel; that is, for every æ, a, there exists a unique x' such that Po [x, a) =
Theorem 2.4. Suppose that for any It, the supremum of V (xx)) is attainable, and that Po is deterministic transition kernel. Then, II" = II", where II" = [Vo' (æx) max, Vx())) and
11 = [ (x,) = max, (=(x)))
Remark 2.5. Theorem 2.4 applies both discrete and continuous action spaces. In the continuous case, a deterministic policy can be represented as a degenerate distribution (a x:) = 8(a ((x)), where 5(-) is the Dirac delta function and f is the deterministic mapping from states actions.
The proof of Theorem 2.4 is provided in Appendix B.3
The value function in 3 remains very difficult to optimize, particularly in the long horizon setting with large K, due the absence of discount factor. To address this, we further derive a lower bound for the value function in using the following discounted value function and introduce its associated Q-function:
K
(*())) = -o.O.
log Sr. 0e (=(((,,,,)) =E Po.
Σ A=t
log Sr. Fee 0r
(4)
k=
for some the discount factor 7 < 1. Our optimization procedure for this discounted value function will be detailed in Section B

---

3 Estimation Procedure
In this section, we provide a detailed description of the proposed policy optimization procedure. When censoring occurs, i.e., when A₂ 0, we treat the process as terminating in the RL framework (a "Done" state), and then return the reward to optimize policy. Consequently, the data collected at each stage are subject to censoring. Unlike general RL, the reward here needs to be estimated. So we first estimate the reward, then plug in the estimated reward to optimize the policy.
We first present the estimation methods and asymptotic properties of the survival function estimator Sn. obtained using either the Cox proportional hazards model (10) or the Aalen's 's additive hazards model (. These models are chosen for their widespread use and relative simplicity, although more complex alternatives exist (e.g., transformation models, varying-coefficien models).
Definition. Let D {(X,,,, A. Y, ASss), denote the data collected at stage k, rep- resenting N₂ independent observations. Denote the counting process at stage as Unx = I(Y2, 2 l). Since ST, (1) xx, A&) = exp(-Ar, (4 Xx, At)), where An, (t) Xx. Ax) is the cumulative hazard func- tion, estimating Sr, is equivalent estimating Ar, (1) X R, Ak). We define the covariates vector Zy (x, AA, A ) and Zi (X,,, (x Xx,x) for subjects. Subsequently, provide the estimation of Ar, (4 Xx, K, Ax) in the following two models.
T(Yi, S 1, A,
1), and the corresponding at-risk process as Va (1)
Cox model. We assume stage-specific Cox proportional hazards model as Ar, (1|x,, A&) = Ao. (f) exp ( 2x} k = I....: : K, where (t)'s are the stage-specific unknown cumulative baseline hazard functions and vn are the stage-specific unknown (2p + 1)-dimension unknown parameters. Estimates of r) and (1) can be obtained by maximizing the partial likelihood (6 2). Specifically,
N₂ Ne ViS (Vr,)) en Zzk
e" Zik
((,,
n = argmax,, II
and
i=1
(t) = J/" ΣE ViL (s) eh Zikk
EN dU)((s
(5)
Consequently, estimators for Ar, (t| k: A&) are given by
NT, (1|X,, A&) = λo exp ( Zz} k = 1,...,,,,
(6)
Additive hazard model. We assume a stage-specific Aalen's additive hazards model as Ar, (t
Zz) Ao,x + BT Zet, k L...., K. where (t)'s are the cumulative baseline hazard
functions, and Bx's 's are the vectors of regression coefficients. Following the arguments in (11). the estimator for Br is given by
Σ No
1-=1Jo S0 (s)(s) Zjz [s)] ds
N₂ Zyk (s) dU ds; where Zn(ss - (ix(s) Zyk
Br
(7)
E 1o)
E/ Vis(s)
with a = aa (t, (t) is then estimated by
No (n,, [ Σ (UU)(()) V3x(s)BT Zzeds) / Vs)()).
Consequently, the estimator for AT, (L) Zx) is given by Ån. (4) Z (Be, 1) + +33 Zel.
We summarize the consistency and the asymptotic normality property of ÅT, (t Zz) in Theorem The detailed proof is provided in Appendix B.7
Theorem 3.1. For each k < K, suppose that there exists a constant My such that G2 S MR-
Then, under assumptions
B.5 for the Cox model, or assumptions B.6 B.9 for the additive hazards model, for any E [0, GR.. as No + 0o, we have
B.2
(i) supre(0.CM) |An (t) Zz) An. (t Zx)l 2 0;
5

---

(ii) År. (t Zx) An (t 2x) Op(N
(iii) VN {An, (t] Zạ) - An. (t Zk)} 4 M(,, oilt: Zx)), where the variance function (1; Zz) is given in A.6Ifor the Cox model and (A7)) for the additive hazards model.
Theorem characterizes the uniform convergence, convergence rate, and asymptotic normality of the estimator Âr, (t 2x.), thus demonstrating validity. Accordingly, the use of Sr, (t) [x, A&) =
exp xt, } as an estimator for Sr, Xx Ag. also reasonable. Therefore, we replace Sr, in the Q-function in (4) with Sn, resulting in feasible Q-function as follows:
(01 a.)
a=)= = -EE Σ Ent-'an
( Ax) X₂ = Zz, A at
(8)
Through the lower bound of log PT, transform the problem of maximizing the probability that the duration between consecutive events exceeds a meaningful threshold into maximizing a discounted sum of estimated cumulative hazards. This enables the direct application of standard RI. algorithms to determine the optimal policy.
Specifically, we optimize Q" in using the soft-update Deep Q-Network (DQN) algorithm (20) for discrete action spaces and the Deep Deterministic Policy Gradient (DDPG) algorithm (20) for continuous action spaces. The detailed algorithms for obtaining the optimal policy are provided in Appendix A.6 to space limitations.
4 Simulation Studies
In this section, we conduct numerical studies to assess the finite-sample performance of the proposed method and demonstrate its advantages. Specifically, our aim is address the following research questions (RQs). RQI: How does the proposed decision-making strategy compare with existing approaches terms of improving the total recurrent-event duration? RQ2: Does our method exhibit stable and consistent performance across various distributions of Ti We first outline the simulation setups and subsequently address the aforementioned questions through extensive numerical experiments.
4.1 Simulation Setups
Setting 1. Linear transition and discrete actions. For the initial stage (k: 1), we generate X from a standard normal distribution (0,, I2). For each stage, the available actions are chosen from A 1). X1,4+1 evolves according the following iterative formula,
(2Ar, 1)/4 (1 - 2A, /44 X . Ei for k = 1, K
0
X1,++1
0
where 6i iid N' (02, 12)4). Similar simulation setups can be found in Shi al. (37 and Chen et al.
②.. We let 0L = exp ((2,, 1) where T
Setting 2. nonlinear transition and continuous actions. For the initial stage (k- 1), we generate X,1 from a standard normal distribution ((sso, Is)) For each stage, the available actions are chosen from A = [-111] x evolves according to the following iterative formula,
X 3. // sin ) + cos(X + Ei,k, for k = 1, 1.2.....K -
where Ei 2id NO /5/ /4). We let 0,, = exp((-A,, B). where 3 E 50 is vector whose first three elements are generated from the standard normal distribution, and whose remaining elements are O.
In Setting and Setting 2, we consider total stages K 20. A a is selected by maximizing proposed Q-function in (8) and is chosen from A with a decreasing greedy probability. We set G = 7, and the duration Ti generated according to T.,. = min { Bi,, Gk), where Ba ~ Camma(1/0,, x, We let C2 follow the uniform distribution U(0, 7 x 10), independent of

---

Tak: X.A- A- We set the pre-specified threshold a = 1/14. To simulate the real-world uncertainty. we consider transition kemnels with stochastic perturbations. We optimize the objective function defined in ( using the soft-update DQN algorithm for Setting and the DDPG algorithm for Setting The implementation code is available at htthtt:://giiee
com wtt////ffegginannggggggoooooo
4.2 Methods and Metrics
Methods. Our investigation of the aforementioned RQs based on the comparison of the following several methods:
Cox: proposed method that uses the Cox model (10) to estimate the survival function.
Aah: our proposed method that uses the additive hazards model ①1 to estimate the survival function.
Baseline: the optimal treatment policy proposed by Liu et al. (33 Specifically, their optimal treatment policy is determined by optimizing the total recurrent-event duration Σ Tk via sinite-stage dynamic treatment regimes. The corresponding objective function, derived from Liu et al. 23). is provided in Appendix [B8]]
Metrics. We conduct M 50 epochs with sample size of N, each progressing from stage 1 to stage K = 20, with different methods updating their policies as the experiments progressed. The experiments are repeated for S = 10 different seeds. For each seed ,,..., S and epoch m 1, ..... ., we record T()..) and (fa.m) at each stage We then predict the values of Ti, and 0. epoch m derived from simulations using the Cox, Aah, and Baseline methods, denoted as ((() Σ ((s.m)) /S and ((m) = = pls.m)) /S, respectively. In this paper, we employ the following metrics to evaluate the effectiveness and robustness of the proposed method:
R Average Recurrent-Event Duration (ARED): ARED (m) DE
Ek=1
iik /N.
Average Estimated Variance (AEV): AEV(m) = Em ER
R 8lmm)
/N.
k=1
1,k
Average ARED over Last 40 Epochs (AARED): AARED
ARED (m) /40.
Average AEV over Last 40 Epochs (AAEV): AAEV ->10
AEV /40.
4.3 Comparisons on the recurrent-event duration (RQ1)
Figure illustrates the changes in ARED and log ARED with increasing epochs in Setting and Setting 2 for varying sample sizes N. The results suggest that our proposed method exhibits faster convergence than the baseline model.
N 100
N - 100
20.5
20
2.9
19.5
2.8
1000
N 1000
ARED
20
2.9
19.5
2.8
10000
10000
20
2.9
19.5
10
30
20
50
Epochs
Epochs
Aah Model
Cox Model
Baseline Model
Aah Model
Cox Model
Baseline Model
(a) Setting
Setting 2
Figure 2: The average recurrent-event duration over epochs with 10 seeds.
7

---

Table 1: Mean (Standard Deviation) of AARED under different settings, models and sample sizes N
Setting
N
Aah
Cox
Baseline
100
19.96 (0.006)
19.93 (0.007)
19.80 (0.021)
Setting
1000
19.94 (<0.001)
19.94 (0.001)
19.83 (0.008)
10000
20.00 (<0.001)
19.98 (<0.001)
19.87 (0.006)
100
18.88 (0.125)
19.19 (0.086)
17.17 (0.074)
Setting 2
1000
19.52 (0.047)
19.45 (0.057)
17.06 (0.074)
10000
19.57 (0.055)
19.54 (0.072)
17.16 (0.080)
100
100
40
10
20
5
1000
N 1000
40
10
AEV
20
log
N 10000
N 10000
40
20
10
20
30
40
50
10
20
30
40
50
Epochs
E ppochs
Aah Model
Model
Baseline Model
Aah Model
Cox Model
Baseline Model
(a) Setting
(b) Setting 2
Figure 3: The average estimated variance over epochs with 10 seeds.
Table Presents a comparison of the AARED values for different models with varying sample sizes N. From Table decreases. At the same time, our model consistently achieves a higher AARED compared to the baseline model. Under Setting the maximum improvement is 0.8% (N 100), while under Setting 2, the maximum improvement reaches 14.4% (N = 1000). This demonstrates that our method achieves faster convergence and yields higher recurrent-event durations.
observe that as the sample size increases, the variance of AARED
4.4 Stability of the proposed strategy (RQ2)
Figure illustrates the relationship between the AEV and log AEV and the number of epochs for Setting and Setting 2, with results shown for various sample sizes N. It can be observed that our method leads to faster decrease and stabilization of the variance.
Table 2 presents a comparison of the AAEV values for different models with varying sample sizes N. From Table 2 we can observe that as the sample size increases, the variance of AEV decreases. Moreover, our model consistently achieves a smaller variance compared to the baseline model, with maximum reduction of 49% (N 100) Setting and 99.89% (N 10000) in Setting 2.
Table 2: Mean (Standard Deviation) of AAEV under different settings, models and sample sizes N
Setting
N
Aah
Cox
Baseline
100
10.53 (0.001)
11.11 (0.016)
20.70 (27.33)
Setting
1000
10.54 (<0.001)
11.95 (0.005)
20.00 (19.47)
10000
10.54 (<0.001)
12.00 (40.001)
20.18 (20.85)
100
2533 (1 x 10*)
224 (2 X 10$)
2251 (3 x 10")
Setting
1000
202 x 105)
441 (3 10%)
41189 (2 x 1020)
10000
49 (2 x 103)
181 (1 10a)
43657 (3 x 10"0)

---

Aah Model
Concordance Index
Cox Model
0.65
Logioemaa Model
Beseline
0.50
0.45
0.40
15
20
Study Week
Figure The values of the concordance index fitted for each stage under different models.
Figure and Table 12 together demonstrate that the proposed strategy not only maintains effec- tiveness also exhibits high robustness. This indicates that the recurrent-event duration achieved by our method is not only higher than that of the baseline method but also shows significantly smaller variance fluctuations. Additional simulation results are provided in Appendix A. A.
and A.4
5 Real Data Analysis
In this section, we apply the proposed method to analyze decision decision-making strategies based on the 2018 Intern Health Study (IHS). IHS was microrandomized trial (MRT) designed to evaluate the impact of app-based push notifications on physical and mental health outcomes. Over 26-week period, participants were re-randomized weekly to receive activity suggestions.
Data description. The data contains weekly records for each participant, whether a message was sent, the duration of daily sleep. the count of steps and the mood scores. We define the daily average duration of sleep and the step count for the i-th participant in the k-th week as the state variable Xi the binary indicator of message sending as the decision action A, and the cumulative mood score as the reward T. Let stage duration G = 70, representing the maximum possible mood score per week. Since mood scores not recorded daily within week, instances with incomplete mood data are treated as censored, denoted by A = O. After data manipulation, the final dataset includes 1,176 participants followed for up to 26 weeks, with censoring rate of 87.8%.
Analysis. Because the action space discrete, we use an offline soft-update DQN method to determine the optimal policy with K 26. We analyze the fitting performance of different survival models, i.c., the Aah model, the Cox model and the Log-Normal accelerated failure time (AFT) model (as shown in Appendix A.5), to the dataset. Furthermore, given the optimal policy т. we calculate the average probability that the high mood score exceeds a threshold the stage duration GG by
Σ5 (T..)
K II
where PaT (x;,,)))) = E"
ST. (ax Ai,)) X wi,t k=t
1176
Results. Figure presents the concordance index (C-index) for each model at each stage. C-index values are between and I; values greater than 0.5 indicate better than random discrimination. As shown in Figure 4 the Log-Normal AFT model demonstrates better discriminative performance than the Aah and Cox models.
We denote the average probability derived from the Aah model, the Cox model and the baseline model pad p=* and pa respectively. We then present the log probability difference (log p*
log p*") for the Aah model and (log p*" - log p=") for the Cox model across varying ag settings. As shown in Figure [ our model outperforms the baseline model across all og values, achieving higher survival probabilities. The optimal performance occurs at ag = 1, where the probability improved by up 35% Additional analyses using different metrics provided in Appendix A.5
To conclude, our proposed model formulates policies by maximizing the probability that the high- mood duration in each stage exceeds threshold, achieving higher probabilities of high mood duration

---

Log Probabiltty Difference
Aah Model
Cox Model
0.2
Baseline Model
0.1
1/14
2/14
3/14
4/14
5/14
6/14
7/14
8/14
9/14
10/14
11/14
12/14
13/14
14/14
d
Figure 5: Comparison of log survival probability difference across policies with different og.
compared to traditional RL strategies that maximize expected rewards. Additionally, the proposed method dynamically adapts the policies individual preferences in different values of ar.
Conclusion
This paper presents novel RI. framework for survival analysis tailored to alternating recurrent event data, addressing the limitations traditional methods designed for time-to-event data and expectation maximization. We propose probability event durations exceed given threshold, and reformulate the task as a standard RL problem by optimizing lower bound of this objective. We provide theoretical guarantees for the equivalence of the optimal policy, as well as for the consistency and asymptotic normality of the estimated cumulative hazard functions. Simulations and real-date experiments demonstrate faster convergence, lower variance, and higher event wenttdurationn probabilities of our proposed method compared to existing traditional RL-based methods.
objective that maximizes the probability that recurrent-
Acknowledgments and Disclosure of Funding
This work was partly supported by the National Natural Science Foundation of China (12401 366, 12301348, 1237 1292), the Natural Science Foundation of Shandong Province (2R2023QA086), and the Shandong University Future Plan for Young Scholars. The authors are also grateful for the contributions the researchers, administrators, and participants involved in the Intern Health Study (NCT03972293).
10

---

References
[1] Aalen, O. O. linear regression model for the analysis of life times. Statistics in Medicine, 8(8):907-925. 1989.
Andersen, P. K. and Gill, D. Cox's regression model for counting processes: large sample study. The Annals of Statistics, 10(4): 100-1 120, 1982.
[]] Bellman, Dynamic programming. Science, 153(3731):34-37, 1966.
Bian, Z., Moodie, E. E.,, Shortreed, S. M., and Bhatnagar, S. (2023). Variable selection in regression-based estimation of dynamic treatment regimes. Biometrics, 79(2), 988-999.
Bradtke, and Duff, M. Reinforeement learning methods for contimuous-time markov dccision problems. Advances in neural information processing systems, 7, 1994.
[6] Breslow, E. Contrihution to discussion of paper by cox. Journal of the Royal Statistical Society Series B: Statistical Methodology, 14:216-217, 1972.
[7] Cai, H., Shi, C., Song. R., and Lu, W. (2023). Jump interval-learning for individualized decision making with continuous treatments. Journal of Machine Learning Research, 24(140), 1-92.
[8] Chen, E. Y., Song, R., and Jordan, M. I. Reinforcement learning with heterogeneous data: estimation and inference. arXiv preprint arXiv:2202.0000 2022.
[9] Cho, H., Holloway, S. T., and Couper, J. D. Multi-stage optimal dynamic treatment regimes for survival outcomes with dependent censoring. Biometrika, 110:395-422 420, 2023.
[10] Cox, D. R. Regression models and life-tables. Journal of the Royal Statistical Society Seriey B: Statistical Methodology, 34(2)187-202, 12 1972.
[11] Cui, Y., Kosorok, M. R., Sverdrup, E., Wager, S., and Zhu, R. Estimating heterogeneous treatment effects with right-censored data via causal survival forests. Journal the Royal Statistical Society Series B: Statistical Methodology, 85:179-211, 2023.
[12] Dabney, W., Rowland, M., Bellemare, M., and Munos, R. Distributional reinforcement learning with the quantile regression. In Proceedings the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[13] Fleming, R., Lin, D. Y., and Wei, Hazards model. Biometrika, 81 81(1):73-8) 81, 1994.
Confidence bands survival curves under the proportional:
[14] Goldberg, Y. and Kosorok, M. R. Q-lcarning with censored data. The Annals of Statistics, 40:529-560, 2012.
[15] Hargrave, M.. Spaeth, A., and Grosenick, L. Epicare: a reinforcement learning benchmark for dynamic treatment regimes. Advances in neural information processing systems, 37:130536- 130566,,1388,,-81111111111118118 2024.
[16] Jiang, R., Lu, W., Song, R., and Davidian, M. On estimation of optimal treatment regimes for maximizing L-yaar survival probability. Journal of the Royal Statistical Society Series B: Statistical Methodology. 79 14):1165-1185, 2017.
[17] Lee, C., Yoon, J., and Van Der Schaar, M. Dynamic-deephit: A deep learning approach for dynamic survival amalysis with competing risks based longitudinal data. IEEE Transactions Biomedical Engineering, 67(1):122-133, 2019.
[18] Lee, C. H., Huang, C.-Y., Xu, G., Luo, X. Semiparametric regression analysis for alternating recurrent event data. Statistics in Medicine, 37(6)9996 10088996-0008, 2018.
[19] Li, M. Shi, C., Wu, Z. and Fryzlewicz, Testtin stationarity and change point detection reinforcement lcarning. The Annals of Storisties, 53(3): 1230-1256, 2025.
[20] Lillicrap. T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y. Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning. arXiv preprint 1509.0150000999 2015.
[21] Lin, D. Y. and Ying, Z. Semiparametric analysis of the additive risk model. Biometrika, (1(1):61-71, 1994.
[22] Liu, Y., Logan, B., Liu, N., Xu, Z., Tang. J., and Wang. Y. Deep reinforcement learning for dynamic treatment regimes on medical registry data. IEEE International Conference on Healthcare Informatics (ICHI), pp. 380 385. 2017.

---

[23] Liu. Z., Zhan, Z.. Liu, J., Yi, D.. Lin, C., and Yang. Y. On estimation of optimal dynamic treatment regimes with multiple treatments for survival data-with application to colorectal cancer study. arXiv preprint arXiv:2310.05049) 2023.
[24] Loe. A., Murra, S., and Wu, Z. Random forest for dynamic risk prediction of recurrent events: pseudo-observation approach. Biostatistics, 26(1):xxa0077 2025.
[25] Luckett, D. J., Laber, E. B., Kahkoska, A. R., Maahs, D. M., Mayer-Davis, E., and Kosorok, M. R. Estimating dynamic treatment regimes in mobile health using v-learning. Journal of the American Statistical Association, 1 115(530):692-7000 706, 2020.
[26] Mo, W. Qi Z., and Liu, (2021). Learning optimal distributionally robust individualized treatment rules. Journal of the American Statistical Association, / /6(534), 559-674.
[27] Munk-J, P., Mortensen, P. B., et al. Incidence and other aspects of the epidemiology of schizophrenia denmark, 1971-87. The British Journal of Psychiatry, 161(4)4889-495 1992.
[28] Murphy, A. (2003). Optimal dynamic treatment regimes, Journal of the Royal Statistical Society: Series B: Statistical Methodology, 65(2). 331-355.
[29] NeCamp, T., Sen, S., Frank, E.,, Walton, M., lonides, E., Fang. Y., Tewari, A., and Wu, Z. Assessing real-time modcration for developing adaptive mobile health interventions for medical interns: Micro- randomized trial. Journal of Medical Internet Research, 22(3)==15033, 2020.
[30] Papadimitriou, C. H. and Tsitsiklis, N. The complexity markov decision processes. Mathematics of operations research, 1 12(3):441-45)) 1987.
[31] Qi, Z., Liu, D., Fu, H., and Liu, Y. (2020). Multi-armed angle-based direct learning for estimating optimal individualized treatment rules with various outcomes. Journal of the American Statistical Association, 175(530), 678-691.
[32] Ren, B. and Barnett, I. Combining mixed effects hidden markov models with latent alternating recurrent event processes to model diurnal active-rest cycles. Biometriex, 79(4):3402-341) 2023.
[33] Robins, J. M. (2004). Optimal structural nested models for optimal sequential decisions. In Proceedings of the Second Seante Symposium in Biostatistics: Analysis of Correlated Data (pp. 189-326), New York, NY: Springer New York.
[34] Sen, S., Kranzler, H. R., Krystal, J. H., Speller, H., Chan, G., Gelernter, J. and Guille, C.A prospective cohort study investigating factors associated with depression during medical internship.Archin Archives General Psychiatry, 67(6): 557 565, 2010.
[35] Shi, C., Fan, A., Song, R., and Lu, W. (2018). High-dimensional A-learnng for optimal dynamic treatment regimes. Annals of Statisties, 46(3), 925-950.
[36] Shi, C., Song, R., Lu, W., and Fu, B. (2018). Maximin projection learning for optimal treatment decision with heterogeneous individualized treatment effects. Journal of the Royal Statistical Society: Series B: Statistical Methodology, 80(4), 681-702.
[37] Shi, C., Zhang, S., Lu, W., and Song, R. Statistical inference of the value function for reinforcement learning in infinite-horizon settings. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84 (3):765-793, 2022.
[38] Shin, G., Jarrahi, M. H., Fei, Y. Karami, A., Gafinowitz, N., Byun, A., and Lu, X. Wearable activity trackers, accuracy, adoption, acceptance and health impact: A systematic literature review. Journal of Biomedical Informatics, 93:103153, 2019.
[39] Shinohara, R. T., Sun, Y., and Wang. M.-C Alternating event processes during lifetimes: population dynamics and statistical inference. Lifetime Data Analysis, 20:110-18 105, 2018
[40] Simoneau, G., Moodie, E. E., Nijjar, J. S., Platt, R. W., Investigators, S. E. R. A. L. C., et al. Estimating optimal dynamic treatment regimes with survival outcomes. Journal of the American Statistical Association, 115(531):1531 1539, 2020.
[41] Sutton, R. S. and Barto, A. G. Reinforvement learning: An introduction. Robotica, 17(2):229-2355 1999.
[42] Wahed, A. S. and Thall, P. F. Evaluating joint effects of induction salvage treatment regimes on overall survival in acute leukaemia. Journal of the Royal Statistical Society Series B: Statistical Methodology, 62: 67-83, 2013.
12

---

[43] Wang. J. Fang, Y. Frank, E., Walton, M. A. Burmeister, M. Tewari, A., Dempsey. W. NeCamp. T.. Sen, S. and Wu, Z. Effectivvenss of gamified team competition mllealth intervention for medical interns: cluster micro-randomized trial.npj Digital Medicine, 6(1): 4, 2023.
[44] Wang. L.. He, K., and Schaubel. D. E. Penalized survival models for the analysis of altemating recurrent event data. Biometrics, 76(2):48- 459, 2020.
[45] Xia, M., Murray, S., and Tayob, N. Regression analysis of recurrent-event-free time from multiple uo windows. Statisties Medicine, 39(1):1 10.1-11,222 2020.
[46] Xu, Y., Muller, P., Wahed, A. S., and Thall, P. Bayesian nonparametric estimation for dynamic treatment regimes with sequential transition times. Journal of the American Statistical Association. 111:911-9500 2016.
[47] Xue, F., Zhang, Y., Zhou, Fu, H., and Qu, A. Multicategory angle-based learning for estimating optimal dynamic treatment regimes with censored data. Journal of the American Statistical Association, 117(539); 1438-144 11881451,220 2022.
[48] Zhang, B., Tsiatis, A. A., Laber, E. B., and Davidian, M. (2012). A robust method for estimating optimal treatment regimes. Biometrics, 68(4), 1010-1018.
[49] Zhang, B.. Tsiatis, A. A., Laber, E. B., and Davidian, M. (2013). Robust estimation of optimal dynamic treatment regimes for sequential treatment decisions. Biometrika, 100(3), 1010-1093.
[50] Zhang, J., and Bareinboim, E. (2019). Near-optimal reinforcement learning in dynamic treatment regimes. Advances in Neural Information Processing Systems (NeurIPS), 32.
[51] Zhao, Y., Zeng, D., Socinski, M. A., and Kosorok, M. R. Reinforcement learning strategies for clinical trials in nonsmall cell lung cancer. Biometrics, 67(4):1422-1433 2011.
[52] Zhao, Y., Zeng, D.,, Rush, A. J., and Kosorok, M. R. (2012). Estimating individualized treatment rules using outcome weighted learning. Journal of the American Statistical Association, 107(499), 1106-1118
[53] Zhao, Y. Q., Zeng, D., Laber, E. B., and Kosorok, M. R. (2015). New statistical learning methods for estimating optimal dynamic treatment regimes. Journal of the American Statistical Association, 110(510), 983-598.
[54] Zhao, Y.Q... Zeng. D., Laber, E. B., Song. R.. Yuan, M., and Kosorok, M. R. Doubly robust learning for estimating individualized treatment with censored data. Biometrika, 102(1):151-168, 2015.
[55] Zhou, Y., Wang, L., Song, R., and Zhao, T. caanfformation-invaiiinn learning of optimal individualized decision rules with time-to-event outcomes. Journal of the American Statistical Association, 118:2632- 2644. 2023.
[56] Zhou, Y., Qi, Z., Shi, C., and Li, L. (2023, April). Optimizing pessimism in dynamic treatment regimes: Bayesian learning approach. Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS) (pp. 6704 6721). PMLR.
[57] Zhou, W., Zhu, R., and Qu, (2024). Estimating optimal infinite-horizon dynamic treatment regimes via PT-learning. Journal of the American Statisticul Association, 119(545), 625-638. 10i:10.11800 00000016214992022.5..0
[58] Zhu, R., Zhao, Y. Q. Chen, G. Ma. S., and Zhao, H. (2017). Greedy outcome weighted tree learning of optimal personalized treatment rules. Biometrics, 73(2), 391-400.
13

---

A Additional Data Analysis
This section provides supplementary data and analysis for the simulation study in Section 4 We conducted M = 50 epochs with a sample size of N, each progressing from stage to stage K = 20, with different methods updating their policies were repeated for = 10 different seeds. For each seed = 1,..., S and epoch m = 1,..., ,....,,.. M, recorded T!...) and g.e-m) each stage k. We then predicted the values of Ti, and 0z,: epoch derived from simulations using the Cox, Aah, and Baseline methods, denoted as T: Σ T//s /S and (/m)) 1,k - (/5-m)) /S. respectively.
the experiments progressed. The experiments
The following metrics are used to evaluate the effectiveness and robustness of the proposed method:
Average Recurrent-Eveee Duration (ARED): ARED (m)
R 1( /N.
Average Estimated Variance (AEV): AEV(m) = Ei Σ k= (lml) /N.
Average ARED over Last 40 Epochs (AARED): AARED Σ Em=1l ARED (m) /40.
Average AEV over Last 40 Epochs (AAEV): AAEV = Em=n1 AEV /40.
Relative Difference of Estimated Variance (RDEV):
For the Cox method: RDEV(m) =E> (C(m)
N 1= f.b
Σi=1 OB(m))
Σ N
@P(m)
1=1
1.b
For the Aah method: RDEV(m) ~E Di=l (M(m)
E 0ik
@B(m)))
Σ 1=
B(()
A.1 Compare with RDEV and f
Here, we present the RDEV values for each stage in Figure 6
1000
RDEV
46
22 10000
0
68
8 0
0
8
80 8
0.0
50
11 Stages
18
10
10 Stage
Modill
Due Moold
Braddin Mbokd
baf Mocal
Ove Mochl
Booeiin Mookd
(a) Setting
(b) Setting 2
Figure Relative difference estimated variance at each stage under different sample sizes.
Furthermore, examine the boxplots of 2 under varying sample sizes during the final 40 epochs in Figure 7 The results reveal two key observations:
The variance of (m) decreases progressively with increasing sample size.
This reduction in variance indicates accelerated convergence rates larger sample sizes.
A.2 Computation Cost
Here we compare the average time cost per epoch for one seed under Setting Table 3 The experiments were conducted GTX 1650 GPU.
14

---

Stage 6
Singe 8
Stage 12
Slage 16
Stage
Stace
Siage
Singe 16
1.45
1.05
Edthare Valance
Esomattd Varenccs
+
0.95
0.35
Westo
Hel8s0
Wal800
Natoogo
N-tto Belatg Rotnon
Netto Hetesg
No100
Nal88
Natongo
#-100
Belsto
Holton
Neli
Retag
Not0g Nalli
(a) Setting with Aah model
(b) Setting 2 with Aah model
Stage
Stage
Stage
Edamated Vorance
Exdmatd Estmaee Veriance
0.05
0.95
Nefoo
Helitog
Not00
9300 Helloog
Hel Buswoo Rate
Hofop tet Helibo
0.85
N-4000
8-2000
Nellbbp
Hed00
Hol
Hotto #=000
N-j00 8-200
Hal
Setting with Cox model
(d) Setting with Cox model
Figure 7: Boxplots of estimated variance 0((m) for each model under different settings across the last
40 epochs sample size varies.
Table Computation time (in seconds) of different models under varying sample sizes N
Model
N
100
N
1000
N 10000
Baseline
0.22s
0.77s
25.50s
Cox
0.89s
4.08s
35.66s
Aah
0.56s
3.10s
42.30s
A.3
Selection of Or
The selection of ag is tailored the specific research question being addressed. For example, in (16)s 's study on HIV data, they chose aę = 0.5, focusing on the optimal strategy for maximizing the probability of survival exceeding the median. In our study, we selected or 1/14 to maximize the probability of an intern's positive mood exceeding one day per bi-weekly period. We generally advise using an og 0.5 less, mainly because 0g values near rely heavily on tail probability estimation, which can be unreliable. These results consistent with the numerical results we obtained from our experiment results. Here we present the AARED and AAEV under different ag values in Setting in and for reference.
A.4 Compare with Epicare
In this section, we demonstrate the performance of our model on dynamic treatment regimes RL benchmark, namely the Epicare environment (15). Since the Epicare environment does not include a censoring setting, but the original work defines a maximum number inquiries after which the process becomes unobservable, we adopt this as our definition of censoring. We define a reinforcement learning environment (X, A, P, R) simulate the process of chronic disease treatment. Next, we describe the components of this environment in detail.
15

---

Table 4: Mean (Standard Deviation) of AARED under different ag values and models
O
Aah
Cox
Baseline
1/14
19.9330 (0.0074)
19.9421 (0.0091)
1/7
19,9460 (0.0089)
19,9228 (0.0130)
3/14
19.1728 (0.0304)
19.8896 (0.0157)
2/7
18.9851 (0.0459)
19.8285 (0.0177)
5/14
18.9972 (0.0438)
19.8193 (0.0207)
3/7
18.9831 (0.0467)
19.7919 (0.0192)
1/2
19.0010 (0.0435)
19.7900 (0.0218)
19.83 (7.9e-3)
4/7
19.0134 (0.0454)
19.7571 (0.0295)
9/14
19.0015 (0.0387)
19.7232 (0.0315)
5/7
19.0007 (0.0433)
19.6999 (0.0291)
11/14
19.0027 (0.0453)
19.6703 (0.0515)
6/7
19.0238 (0.0455)
19.6379 (0.0497)
13/14
19.0107 (0.0434)
19.6413 (0.0373)
19.0591 (0.0465)
19.6885 (0.0334)
Table 5: Mean (Standard Deviation) of AAEV under different ag values and models
Aah
Cox
Baseline
1/14
10.5408 (0.0027)
11.9509 (2.3916)
1/7
10,8309 (0.0818)
13.9811 (4.4648)
3/14
64.5515 (37.9364)
17.7416 (12.5118)
2/7
78.1629 (14.4570)
20.6589 (16.5979)
5/14
78.5551 (13.8795)
22.2620 (26.8484)
3/7
78.5710 (14.2459)
23.5382 (27.2738)
1/2
78.6225 (13.8094)
24.4125 (35.8999)
20.00 (19.47)
4/7
78.6280 (14.3543)
25.0995 (50.6035)
9/14
78.6196 (14.6698)
27.7541 (67.8201)
5/7
78.4769 (17.6261)
29.4384 170.1726)
11/14
78.2809 (19.1925)
30.3147 (91.0969)
6/7
78.0577 (19.7367)
32,2957 (104.3780)
13/14
77.9280
(19.0170)
32.6802 (79.8917)
1
71.9542 (83.7705)
29.4135 (76.7759)
Disease state: Let the index set of disease states be I (l,...,nd) na), and let denote the absorbing remission state. At each stage t, the underlying state
de (0) UI, nd 16,
where de = 0 represents recovery and is absorbing. For each disease index t E I, symptoms follow a multivariate Gaussian A (Jus, Ex), with μi randomly generated to induce diversity:
IR. "syu Σn R
yym Payu
Asym
Parameters are
μ4 U(0, Thgg, Σ1 = A diag(a (0
A
where ke (11 and Aş " x Men a random orthogonal matrix, ensuring Σg is symmetric positive definite.
Symptom state: The observable state stage is normalized symptom vector
[0, 10m0 for d, = k E I dts N(uu: Ex), az = (
where a(.) ) is the sigmoid, guaranteeing æ, E (0, 1] Togi, If de = 0, the episode has terminated and no further observations are drawn.
Action Space: The action space is finite set of treatments:
ag A = {0, 1, Aueat 1}. Rhrcat 16.
16

---

Each action corresponds to treatment intervention selected by the agent.
Transition matrix P": Disease-to-disease evolution (excluding remission) is governed by random row-stochastic matrix
= [Pulatez E R where Pat = P(d4+1 = 1 de = k) and
Σ
Pk = 1,
IET
constructed as
((0.01,022) 0,
with probability mag L k +
PRI
PEK 11
Σ
PRI
otherwise,
1
This yields a sparse, approximately symmetric graph over Z, reflecting realistic limited comorbidity pathways. Remission is handled separately, for k I and action a,
P(dd+1 de k,
px(),, where pa(a) (4088,
Equivalently, define the augmented action-dependent kernel P on {0) UT: (a)(a) (a(a), (a((a) (1 (a)) Psa for l EI,
(a) = 1.
Stationary distribution: The initial disease is sampled from the stationary distribution p (Pi,"" :Pna) of P over I:
P
Epx=1 = do m.
KEI
(Note that remission d, 0 is excluded here since it is an absorbing terminal state.)
Reward function:
The stage-wise reward balances cost, remission benefit, and adverse-event risk:
R₂ (eos((as) Premission (dc+1) Radverso (x++1),
with
Roos(aa) -e(ar). d(a)~~ ~ 21(11)).
+rrem: de+1 = 0,
Reemission
Trem = 64,
00
otherwise,
-adv:
max( > 0.999,
Ralverse
Tadv 64.
0, otherwise,
Termination: We assume that the process terminates the patient enters the remission state, i.e., when de = In addition, we impose a maximum follow-up horizon K maax = 8. termination has not occurred by t Kmaxx the trajectory is regarded right-censored, meaning that the observation is incomplete. Formally, we define the event time as K ming/ de 0}. and introduce the event indicator A, (tCR) [(1SKmar): which equals if the event (remission) occurs before censoring, and () otherwise.
We defined (æ,, de, RG, Ae) for 1,.... Kmax where R₃ is analogous to Y, in the main text, both representing the reward. An offline dataset with sample size N = 1000 was generated. Using 10 different random seeds for network initialization, trained multiple models and evaluated variety of policies. For each policy, we report the cumulative reward as well as the final censoring rate. The following four approaches are compared:
Cox: our proposed method utilizing the survival function estimated from the Cox propor- tional hazards model (10).
Aah: our proposed method utilizing the survival function estimated from Aalen's additive
hazards model (I).
Km: the optimal treatment policy proposed by Liu et al. (23) which maximizes the expected value via Kaplan-Meier estimation. The corresponding objective function, derived from the method of Liu et al. (2023), is provided in Appendix B.8
17

---

RL: classical reinforcement learning methods that maximize reward without accounting for A, consistent with the objective proposed in (15).
We set the number of training epochs to 50 and the number of evaluation episodes to 200. Since we require Y all Ht values were normalized across individuals at each stage prior to estimation and training. We computed the mean and variance of the cumulative reward (E Σi=1 Rst/N) and censoring rate (1 Σx= ANk../N) across all 10 random seeds, including results from our proposed methods under different values of Org. These results, reported in Table 6 and Table 7 indicate that
our methods consistently achieve higher rewards and lower censoring rates, thereby demonstrating superior performance in the Epicare environment.
Table 6: Mean (Standard Deviation) of Cumulative Reward Under different model and ag
Ok
Aah
Cox
Km
RL
0.1
(1.0323(4.919)
8.9952(3.5328)
0.2
11.0323(4.9149)
6.5327(4.3448)
0.3
10.0921(3.4005)
7.2183(4.1138)
0.4
10.0921(3.4005)
7.5531(4.6677)
0.5
11.0028(4.3290)
8.5168(3.0017)
-15.8804(111859)
6.2507(3.3373)
0.6
11.1656(3.8367)
7.3527(4.5401)
0.7
11.1656(3.8367)
5.4090(2.7325)
0.8
10.7549(4.7615)
7.4012(3.6093)
0.9
10.7549(4.7615)
8.2964(4.4176)
1.0
12.711993..0012)
0.6448(6.0716)
Table Mean (Standard Deviation) of Censoring Rate Under different model and oe
a&
Aah
Cox
Km
RL
0.1
0.5050(0.0575)
0.5405(0.0452)
0.2
0.50500..0555)
0.5665(0.0525)
0.3
0.5125(0.0053))
0.5605(0.0494)
0.4
0.5125(0.0453)
0.5540(0.0602)
0.5
0.5055(0.0535)
0.5450(0.0398)
0.9610(0.0139) 0.6410(0.043))
0.6
0.5060(0.0444
0.5585(0.0539)
0.7
0.5060(0.0444)
0.5785(0.0369)
0.8
0.5120(0.0586)
0.5555(0.0440)
0.9
0.5120(0.0586)
0.5455(0.0469)
1.0
0.7975(0.0354)
0.6365(0.0603)
A.5
Real Data Analysis Model
We then presented the relative probability (p** - -p=")/p=" for the Aah model and (p=
px*) / for the Cox model across varying o& settings in Figure As shown in Figure our model outperformed the baseline across all values, achieving higher survival probabilities. At the same time, we observe that the Cox model also provides a satisfactory fit. Below, we present the results of estimating pa in (I) under the Cox model, then show the log probability differences in Figure 9 and the relative probabilities in Figure 10
Log Normal AFT model assumes that the natural logarithm of the survival time Tş follows a normal distribution: log(Ti) (( (2+), a²)) where: The location parameter (=z) is linear combination + Q1 + a222 + ... + a, 2,, Here, ao, 01..... are the regression coefficients to be estimated, reflecting the impact of the covariates z on the mean of the logarithm of the survival time. The scale parameter is fixed constant (independent of covariates), describing the standard deviation of log(7) and controlling the degree dispersion of the distribution. The model estimates parameters by maximizing the likelihodo function.
of the covariates z = [z1. ......, znl: p(z)
18

---

Aah Model
Relative Probability
0.3
Cox Model
Baseline Model
0.2
0.1
1/14
2/14
3/14
4/14
5/14
6/14
7/14
8/14
9/14
10/14
11/14
12/14
13/14
14/14
Figure 8: Comparison of relative survival probability improvements across policies with different ox under Log Normal AFT model estimation.
Log Probabilit Difference
1.0
0.8
Cox Model
Baseline Model
0.4
0.2
1/14
2/14
3/14
4/14
5/14
6/14
7/14
8/14
9/14
10/14
11/14
12/14
13/14
14/14
Figure 9: Comparison of log survival probability differences across policies with different og under Cox model estimation.
1.5
Aah Model
Relative Probabiltyy
Cox Model
Baseline Model
1.0
0.0
1/14
2/14
3/14
4/14
5/14
6/14
7/14
8/14
9/14
10/14
11/14
12/14
13/14
14/14
Figure 10: Comparison of relative survival probability across policies with different og under Cox model estimation.
19

---

A.6 Algorithm for Recurrent Event Data
We use the Deep Deterministic Policy Gradient (DDPG) algorithm 20) to optimize the objective function defined in (8) with continuous action space and use soft-update Deep Q-Network (DQN) to update the discrete action space. initial network Q and target network Q'. Here we update the value of the Q-function using the optimal Bellman operator and compute the loss (temporal difference error) to adjust the weights of the network Q. Subsequently, we perform a soft update to the weights of the target network Q'.
model free, off policy algorithm that employs an
For discrete action space. We define greedy optimal policy π as any policy that selects the action that produces the highest Q value. Specifically, for any E X, x x-(- x) satisfies I (a x) if Q (x, a) * max,'eA QTT (x, We denote the Bellman optimality operator as T. To maximize the Q-function, following the strategy of Bellman's 's equation (3), an estimate of Q can be obtained via dynamic programming using the operator T as follows:
TQ(X,, A&) = EEAr, (o Xx, Ax) + max Q(K++1, ARt1).
(A.1)
A4tIEA
is shown in Appendix B.4 that updating the Q-function using the Bellman operator T can converge unique Q-function.
For continuous action space. According to DDPG (20), we define the Policy Function: x(0): S - A, parameterized function (e.g., neural network) that maps states to actions. For fixed policy μ(0), the Q-function follows the Bellman evaluation equation: Qu(+)(X: A.) = B ( A1) pected return (() = [[Q X, A.).. Then, parameter updates are performed using the chain rule in DDPG.
To optimize μu(0), maximize the ex-
One of the key features of DDPG and soft-update DQN is the use of soft updates for the target networks. Rather than directly copying the weights from the current network to the next network, they gradually updates the parameters. This approach enhances learning stability by smoothing out parameter updates, thus preventing large oscillations parameter values. The proposed estimation procedure can be implemented as outlined in Algorithm I For the neural network in simulation, we employ RL agent with the following hyperparameters: a batch size 32, a replay buffer of 6400, learning rate of 0.001, soft update of 0.01, and discount factor y of 0.99.
A.7 Supplementary Experiment
Under identical assumptions for Setting 1 in Section we vary K compare the AAEV and AARED metrics across different models, using a sample size of N = 1000 for this analysis. The performance of AAEV and AARED is presented in Table g and Table g respectively. Because the censoring rates under different K are also different, we present the changes in censoring rates
(1- (1-E i=1 Ih Ar,N/N, under different Models in Table 10 The results demonstrate that our model consistently outperforms the baseline model across different values of K while maintaining comparable censoring rates.
Table 8: Mean (Standard Deviation) of AAEV under different K when N = 1000 Setting
K
Model
10
20
30
40
50
Aah
8.45(1.14)
13.14(0.68)
18.45(0.59)
23.99(0.59)
29.57(0.48)
Cox
9.94 (2.62)
15.14 (2.69)
21.13 (2.97)
27.29 (3.52)
33.65 (4.91)
Baseline
16.46 (6.
23.29 (6.53)
29.97 (6.63)
37.20 (7.35)
43.81 (7.54)
B Definitions and Proofs of Theoretical Results
B.1 Proof of Lemma 2.3
We first propose the following Lemma B.I (the proof is provided in Section B.2
20

---

Algorithm DDPG for Recurrent Event Data (Continuous and Discrete action space)
Randomly initialize action value network Q with weights 09,
Initialize target network Q' with weights 0º + 82
For continuous actions:
Initialize actor network j with weights 0r
Initialize target actor network y' with weights g² +- 0". Initialize replay buffer R, set soft update weight т, exploration noise e, and Og.- for episode = to M do
For discrete actions:
Initialize a random process as explore probability for discrete actions.
For continuous actions:
Initialize a noise process N for continuous action exploration.
Receive initial observation state )
for = 1 to K do
For continuous actions:
With probability c, select a random action A, from the continuous action space perturbed by noise N.
Otherwise, select Ai, (Ki,,; 0*) +W.
For discrete actions:
With probability select a random discrete action As,
Otherwise, select Ar, = arg max Q( A; RQ)
Execute action A, emulator and observe Yi, Ai R and image X,,++1-
Store transition (Xi,,, Ai,e, Yi,,, Ai.,, ac, Xi,++) in R.
Sample random minibatch of (Xi,t, Ai, Yi,t, Ai,r; 0t Xi,+1) from R. M Estimate Ar., ( (- A1,)) with (X,,,; Yr,,, ) using Cox or Aah model.
Calculate Rg,, - (a
A, A Q' (' A1;
Update critic network:
Update 0º by minimizing the loss: L = N. Σ (R., Q(X Ace; A4e; 09)) Update actor network (only for continuous actions):
Update ( to maximize Q(Xi,,, pl (x,,, 0"); 09).
Update target networks:
0 - 709 + (1 7)09'
For continuous actions: - Ot + -0" + (1 T)
end for end for
Table 9; Mean (Standard Deviation) of AARED under different K when N 1000 Setting
K
Model
10
30
40
50
Aah
9.92(0.08)
19.89(0.10)
29.93(0.11)
39.91(0.12)
49.90(0.14)
Cox
9.91 (0.09)
19.89 (0.11)
29.90 (0.12)
39.90 (0.15)
49.86 (0.18)
Baseline
9.81 (0.14)
19.78 (0.16)
29.78 (0.18)
39.77 (0.22)
49.74 (0.21)
Table 10: Mean (Standard Deviation) of Censoring Rates under K when N = 1000 Setting
K
Model
10
20
30
40
50
Aah
15.1% (0.004)
29.8% (0.005)
42.4% (0.004)
56.8% (0.005)
71.0% (0.006)
Cox
15.0% (0.004)
29.9% (0.007)
42.7% (0.006)
56.8% (0.006)
71.1% % (0.006)
Baseline
15.1% (0.005)
29.6% (0.007)
42.5% (0.005)
56.6% (0.007)
70.6% (0.008)
21

---

Lemma B.1. When Assumptions 2.1 and 2.2 hold, we have
R II
( > ag a.G......... TR > ag G AR)
An),
==
where Sn (:Ix;; Ax) is the conditional survival function of the duration time T6 given Xx and Ak-
Next, we prove Lemma 2.3 below.
Proof. We define Bg = {w[TT(W) au C Og for I,..., K, where Og is the measurable
sample space of stage k. Then we have:
p.u. Bi.... , X,) = E.b [B,,... ,B) X₂]
E
[ [L[B.... .Bg)
X Ae, ... XR, Xi] [Bg)
Eooi
(Bes... BR Ax,Ar," Ars.... X R AR) X1]
E..,
P[ Xk; As) X,
II Ammtt
Sr, (aR.Gk Ab) X,
where the second last equality follows from Lemma
B.1
B.2
Proof of Lemma B.1
Proof. Define Bg = (TwTTTTau > Gx C Qr k =l K, where 24 is the measurable sample
space of stage k: Let O2 (X h; Ak), then have
P PBB Bgl ,.,
=P <P( BI...., O 2R)
BR...,,...........
=P( OR) P B
BR-1 O1, 0,,...,0k ,0k)
P(B
BK-1 0 1O.....,OR-1) (P(R B OI..,,
=P(BR 1OR)
P[OO
-P(BR (R) BR-1
[P(BRIOR) (R OR P (B....., BR O 0x
K
II P(Ba(Ok)
R
II
Sr, (on XR, G An).
B.3 Proof of Theorem 2.4
Po be the state at stage k under policy Po- For any deterministic policy Po. by definition, Proof. Let a
there exists unique af such that
Po
xx .xx) 1 for 2 t.
where x; = I, is the initial state. Define I as the set of all possible trajectories, each consisting of sequence of states, actions, and rewards from stage to K. Specifically.
r = (z..
Po at, Te. .. at11 T......,,
Po ag. TR JaLEA.orrssk
22

---

where Tx- == Ta ( ax) is the reward received at stage l given the state-action pair (a ox).
Then, for any Yt I we define the reward along the path 7/t
k
R(w)
II k=t
ST. Ga x .2)-
Then we can obtain
(²)x))) =
Σ
(*())) R(m)),
(xt)= E (*())) log R(7)).
mer
NEr
where
R-1
R Pb)
P" x(a= | x
II k=
Pa 2 ax)
xb41)
II
T(ak
k=
Note that Σ Exer,o II (aa
Pa) = 1, therefore we have kkk
k
(T(x))) Σ
II
π T
x R(())
-R()) & max R(%) and merT.0
mer. Po k=
(o Σ
1 20) log R(x)
max log R(1)). n.er,"
7.Er.o
Let us define
max R(y)) }. and r rh log R(*")= = max log R(m)) ne
D Th
R((f)=
The monotonicity of log(z) (z) directly implies that D I which then denote I7. Therefore we have
pd (T)))) < R(()))
(o())) s log R())), for yt 17.
Define the policy set
II ( a - -11-
K
I
MEL kmt
For (*xx), we denote R = max
Po R(m.). Then, for & 17, we have R(yx < R*. Thus,
7r
(= = Σ (T)) R(x)) + Σ ("())) R.
TTT:
MET
follows that π E IT O Σner; PTT = + Pa (*x))) = R" i.e., Pd (*)))) = R(7() = II. Following similar proof for P*((x)) R()?),
demonstrate that
(x,)) = log R(1;) holds and only if π E II*. Therefore, we conclude that II" =I'
For the continuous action space, the policy can be expressed as T" (a æ) 8(a - p(xx). where 5(-) the Dirac delta function and μ : X
(2.4also holds for the continuous action space. In the following, we provide an alternative and more direct proof, as shown
deterministic mapping. Theorem 22.oormm2.44asso
B3.
B.3.1 Proof of Theorem for continuous action space
Proof. For deterministic transition kernel Po and deterministic policy μs, starting from any initial state F,, the next state is uniquely determined by
Po
Po 6+1
Po 2
(() = 1, for t,
23

---

so that there exists unique trajectory
* = f ulxxf") ), Tr, offi ((x)f1) Tet1...... t R ula P0), ).T.).
Along this trajectory, define the reward as
K R()1") II Sn. (ax | off u((a
kmn
Then the corresponding objective functions simplify to
P" R(()), V(((2))) = log R(
Define the set of optimal trajectories
r
R())) max R(())) ?
where the maximization is taken over all possible deterministic policies μ. equivalently over all possible trajectories. Since the logarithm is strictly increasing, maximizing R(()") is equivalent to maximizing log R(1(". Thus,
TH E IIP
R( = max R(1) = log R(*(") = max log R(m) & T² E II.
74
Therefore,
(ax
xx
5(ax u(xxf)) att c
which proves the equivalence of the optimal policy sets for the continuous action space.
B.4 Proof of the coverage of T
First we proof statement: have max, f(a) max, g(a) < f(a") g(a"), and if max, g(a) = g(a'). we have max, f(a) max, g(a) 2 f(a') - g(a'). Then get
Proof.
max fa) max g(a) s max (If(a') g(a')), g(a")l) max g(a)).
Next, we proof the contracciiv of T:
TQQlllls = max
Po(ylæ,a) a)
r(x,a) + max Qaxx)((,,),,) r(x, a) max Q' Q'(y,b) BEA
BEA
yEX
max
Σ NEN
Po(ylor,)) 2r, a)
max Q(y,)) - max (2(y,b)
m.a
BEA
max y
Σ
Po(y x a)
max Q(y.
max ('y BEA
m.a
BEA
yEX
max
Σ
Po D.
max (Q Q'(
VEX
max Y
Σ
Poly 11 Q'1
a.a
VEX
= x - Q'I
It is shown that the Bellman operator T is y-contractive with respect to the supremum norn over X' x A. Specifically, for any two action-valle functions and Q' defined on x A, it holds that ITQQ - -TQ'll < 7IQ - Q' || This contraction property forms the foundation of the well-known value iteration algorithm (41). which constructs a sequence of action-value functions (Qx k20 by iteratively applying T, where Q TQ for all & 2 1, starting from an arbitrary initial function
Qo It follows that ||Q, - Q" Il I 7*t I|Qu Q" ll indicating that the sequence (Qx )220
converges to the optimal value function Q at lincar rate.
24

---

B.5 Assumption for Cox model
We first define that
NR
5 17. 8)
s (n, s) Nx
Σ
2,x Znk
Zx(n,s)
S 77.
(0)
j=1
$ 11) (7.8)
((2) (1,
ex (n. 8)
(n(
6x (7, 8) 82
S
n7 8)
5 (1,
Let the cumulative baseline hazard function be (s, and let 77s be the true parameter at stage k. Assumption B.2. (Finite interval).
Ao,x((G)) - (0)l <0.
Assumption B.3. (Asymptotic stability). There exists neighborhood B of mh and deterministic (1)
bounded limit functions 8 8 -Bx [0, Gx] - IR TR,,, TR such that for j = 0, 1, 2,
(0)
(2)
sup sup || (n, 8) 2 s)l *E[o.Ga] nEB
Assumption B.4. (Lindeberg). time Lime-invariant covariates Zik.
| are uniformly bounded, and there exists A such that, with
N- sup sup ISiEN, sE[U,Ga]
2, - 2 0 uniformly for v) E B.
Assumption B.5. (Regularity). Let Eb, Uk be defined above. Require:
1. For every & E [0, Gp).
(1)
(n.
a (0) (n.
sk
(2) $
(n,
ôrÿ 8 (17 s),
0²
(0)
On
(5) and 6g (t, 8) are continuous on B and bounded on B x [0, Ga| for j = 0, 1,2.
2. 8/ (17, 8) is bounded away from zero on B x [0, Gk].
(0)
3. The matrix
Σx = 1 uk (,k, 8) 5/ (0)
(Ns s)dAp,
is positive definite.
B.6 Assumption for Aah model
Assumption B.6 (Bounded covariates). The covariate vectors are uniformly bounded: || Zisk|| S Cz.
Assumption B.7 (Risk-set positivity). There exists a constant CR such that
Vs((s) 2 CR.N for all s S G
in probability.
Assumption B.8 (Baseline hazard and linear predictor bounds). The baseline hazard function satisfies dn No.x locally integrable with
(Gx) $ CA < co, and sup (su,(()) + || Za < Cx. 5SGr
Assumption B.9 (Asymptotic positive definiteness). We define
(x ΣNr, Via (x) EN (s)
N₂ Σ
A
Vin(s) 21(ss))2211
N
The matrix Ag converges in probability to positive definite matrix As,so whose eigenvalues are bounded away from 0 and oc.
25

---

B.7 Proof of Theorem 3.1
B.7.1 Proof of Theorem 3.1 under Cox model
Proof. (i) Consisteney. From Theorem 3.2 in Andersen & Gill (2. holds that in & 77 and supigG, (Aa.(t)) - At, & O. Under the Cox model, we have
!'" exp(n Z Zx)dAo,(u)) (n Zs) = S. expl Z 3s
Ar. (t) Zk)
For any t S Gk. write
Ar. (t) Zk) Ar. Zx)
/'" Texp(1i fk Zk} expl nh [ dAa.a(u) ( exp(ti Z&} d(Ao, (w)
((,
=:2ax (1)
For (t), by the mean value theorem and boundedness of Zk.
exp(1) i ] - exp(1) nz Zx} I exp(C)C2ll - nell S 0,
for some finite constant C. In addition, under assumption B.4 At,x(t) is of bounded variation. That is, [Ao,s [0, Gx]] = A (Gx) < oc. Hence
sup [[]x(t)]= < TV (Aoui [0, G&]l| exp Zx) exp(n Zx)l & 0.
SSGx
For Rex (t), since 7i D mh and | Z S Cz, the continuous mapping theorem implies (xp [[[z] & (pp(lllllC)) Hence for any > there exists M such that P (exp( Zx} > M) for sufficiently large No. Thus have
sup [R2x(t)] s M sup (t), - Ao, L0 I5G2
1SG
follows that
sup Ån, (t Zs) (t Z+) S rsC
sup [Rx(t)] + sup [R2x(t) & 0.
tsG,
1SG,
(ii) Convergence rate. Applying the first-order Taylor expansion to Ån Zt) with respect to n). we obtain
W1 Zx) VN {NT, (1 Zx) Ara (4 [ Au(1. Zx) + H (1. ( -Ba0m)))BB(000
()),), (A.2)
where
N
e Zx
Agl1, Zx) N
dM)
( b
(1),8)
i=1
No
Bx(t) = NE
( (Za(s) - -Z(ns, (sM,
iml
H1(1, Zv)
Th Zu { Zx(mi, 8)} ddNo.(());
Gk (2) s)
(1) (Tk. (1k. 692
I+(m6) fo
(0)) (n s)dAa,
S
5 (7k,))
(0) (Tk. s)
and Mtx(t) (t) So Vis(s) exp T3 Zu dd (s) a local martingale. By Assumption B.3
and B.5 ii),
sup
|| (n) 8) 2 (TE,
(r)
2 0,
(0) ( 8) 2 co > 0.
26

---

hence there exist constants Co and co. such that //0 (Tk. 8) 2 co/2 and s (m:s S Co for
r =0 0, 1, and all s Σ t. Assumption B.4 implies that Z I Cz, and thus eh Zu I Ce for some
Ce co. Assumption B.2 implies that Ao, (t) I CA
(a) Bounded variance of A (t, Zz). Since }Mix are orthogonal square integrable martingales with d(Ma))() Vu (s)e Zu (s). we have
No
L.(
e Zx
Var(Au (t, Zx)) = No i=1 2 s)
Visls)eni Zu dAo,
Becasue 1 No
E. ViA (s)e Z. = ((0) (m,, 8), hence
Var(Ax(1, Z+)) = S:" 2 Ze
dA), S 20 < 00.
(A.3)
S (mc.s) 8)
(b) Bounded variance of (t) and scaling of (MR) (t).). By definition,
Gn
(t) N 1/2
1 ZR(ME, 8)} dM)
Since } are orthogonal square integrable martingales with (s) = Viu(s)e Zi dAo,x(s), we have
So Gx
Var(Bx(t)) (
us (n s)S{ (7k 8) dAo (m(,).
(A.4)
Let Ix (Tt) [/N) By Assumption B.5 mii, (77, 2 Σn 0. Therefore, for sufficiently large Nç. there exists a constant C₁ < 3o such that
C₁ N₂²
[mx( 11
N₂
Therefore,
C1
Var (T=((In)))- = (II(TT)) S (x())))-11
N*
which is uniformly bounded.
(c) Boundedness of H& (t, Zx) and variance of He(t, Zx) Ix= (mc) (1). Since || / S Cz and
| (Mk,s) $)Il S 2Cu/co /co by Assumptions B.3 B.5 there exists CH < co, such that
Ca = Ch. / e Zu ( Zz= Zn(,,, (s). s CC Cz + 2C
(t, Z+)l
ea
Then we conclude
C7,C1
Var(Hă (1, (T))) BB(()) < (1Ha(1, (ar -B)(
N₂
which is also uniformly bounded.
(d) Boundedness of Cov((A (1, Zz), Be())). We have
No.
e Ze
Cov(AA(t, 2,), (+) = No i=1
(Zu(u)- Zu (nk. u) (s)e Zi dAu, = =0.
S (ne, u)
(00)
Combining (d), there exists Cw 0o such that,
sup Var(W& Zx)) = sup Var(A (t, Z +HE (t, ZT)Z&(H)) = 2C2Cs, + C2C = Ow. 15G
tSG
CD
N₂
Thus, by Chebyshev inequality, P(sup, (t, Zx)l > M) s Cw/M2 for all M > 0, ie..
sup, WWe(t, Zx)l (1),), then we have Ån, (t Z Ar, (t Zx) O,(N,²²) $).
27

---

(iii) Asymptotic normality. Following from the martingale central limit theorem, it can be obtained that the process W (t, Zx) converges weakly to a zero mean Gaussian process on [0, G&]. Next we and Ig is a symmetric matrix, we have
calculate the variance of the process. Since A.2
A.3
A.4
Var(We(1, 2x) Cov 2x), Au 2x))
(1, Za)Za (TR) Hma)) Zs)
!'" kk (T))))
2 Zu
(s) + M (t. (mx) (t, Zb).
(A.5)
Thus (t,t, Zx) converges weakly to zero mean Gaussian process with covariance given by (A.5). Replace all population quantities by their sample counterparts, a consistent estimator of (A.5) is
Var(Wa(t, Zx)) /
2 Z
d)), + Ht (t, Zz (1, Zz),
(A.6)
T (7x.8)
where
0 (TR,8) 8)
1
Σ
Nv
Via())eni Zu, Hn = 1 E 2. (Zx Zz(r)} ddoo,(t))
Nk
i=1
( s)
(Gr (s s)
S 5t (iti (Ti:.)) 82 (( (fi s))
[x)s) = ((0) (ft s)
In = So
((0) (f)
S (fi:: 8)
B.7.2 Proof of the Aalen Additive Hazards Model Estimator
Proof. (i) Consistency. According to Lin & Ying 21, we have
3u 2. Bas
sup (t) At,x(t) 2 0. G5Gb
Under assumption B.6
sup An. (t) Zx) AT, [+] ZR)l S sup [t),((t) - Ao,x(t) + Bxll 2 O. ASG
1SGx
(ii) Convergence rate. Let us define (s), = (s.ds According to assumptions B.6 B.9 we have (i) (s) 2 CR.Nk, (ii) || Z & Cz, (iii) (s) + + Za I Cx. (iv) cala 3 Ak. with finite strictly positive constants cn. Cz, Cx, CA. According to Lin & Ying (D., we have
N₂
Gn
Vi - Bx) A
(Zi 2x(#)} dMac(s),
(o) - (o, 1 N
l''
Σ
J Rs(s)
(Ma - Cx(t) (3: -B))
NO
where (s) = Ust(s)) S6" (u, + Br Zik du is local martingale, and
(x(s)= = ΣNV, (iA(s) Zik
N2
A = N₂
Vi Via(s) - -x(ss) 2sds
Ii (s)
Ct(() 1' (x(ssss, Ra(s)) Σ (s).
Nv
j=1
Hence, for the cumulative hazard at covariate Zi.
Wt Zx) VN (t Zx) An (t Z+)) A(2(() + A (t, Zn).
28

---

with
M
N2 Gh VN+
A
(1)
(b), A (1,
duls; Ze)dMis(0))
VNk
(s((s)
where
(s;1; ==) = (1Z, - Cx(1))] (Zik Zz(8)}
Since ds) = (A, 3. Zix} ds S C2 ds,
N₂
Cat C2Gs [R(s} d(Maa)(s) - C
(4 = N i=1
S
According to assumptions B.6 and
B.9 we have || || S +/4 and [x(( 5 Cz. Then
2 411 Z $ Gx + G Co.
(s; Zx)) - 2 + [[Cx(t))) 14
CA
hence
No NO
Σ 1 (s; ((Mu)(s) S C Cx.x
(or(A Zt)
By the covariance formula for martingale integrals and Cauchy-Sehwarz Schwarz inequality,
(o (t), A/ (t, Z+)) s (1)) Vvar(A (t, (t)) S V
C2Gk
VC.C,G C
There exists Cw < oo (independent of N₂. such that
sup Var(Wa Zx)) S 1sG,
(t)) 15Gx
sup
{ (Vr (+)) + e (t) Zx)) + 2| Cov(A (1). ( (t. Z+)1]}
Cw.
By Chebyshev's inequality,
Cw M
sup (L, Zn)l M) S esG
|TT Zz) -A,, (1 Zx) ON 1/2). GG.
sup
(ii) Asymptotic normality. Following from the martingale central limit theorem, it can be obtained that the process (t. Zx) converges weakly to a zero-mean Gaussian process on [0, Gx]. Next we calculate the variance of the process.
(a) Variance of A (t, ZR). Let (t, Zx) ttx - Cx(()), we have
N
Var(A (t) Zx)) =
du (s)
NO
N
(Oult, 2x) AT A - 2n(s)))² d(M)))(s)
N
1
N2
Gn (Zik =c(s)) d(Ma)(s)) 1 Zn).
(t, Zx) A
N
Define the matrix valued process
N₂
/
BL(1)= = N 1=1 Ju
[Z 22 2
Then according to Lemma B.10 we have
Var(A (2) (t, Zx)) Ox Zo) B Or Zz).
29

---

(b) Covariance of A (1) and (1). Again by orthogonality across i and the martingale isometry,
N₂
1
(1), 1/ (t, Zx))
(s L, Zx) (s)
Nk
(x)
i=1
N₁
1
(k Z T 12 [*)3 (s)
N 1=1
(s(s)
JU
EN (Zik [s)3 (sMa)(s)
= Oz,t, Zx)
A Ns. 3
Rs(
Let
/0
Li=1
NK
Zik Zs(s) (dU
D2(0) =
1
(s(s)
By Lemma B.10 we have
Co (t (1, Oxl1, Zx) Dz())
Thus.
Vtr(Wa(t, Zx))
=Var(A()))) (1)) (ar(A (1, 200m (()) Zx))
Gi-1 (U)
+ O2(1, 2 T (e() Z,) + 20,(1 Zx) AT
(s)
A consistent estimator of V Var(W& (f, Zx)) is obtained by
Var Zx)} 1-"
N. dU)(s)
+ Ōx(1, Zx) T Be()), 2.) 2.)+2x((() 2&) T AT²Dx((),
Nx)a(s)²
(A.7)
where
No Σ (o), Zx(s)- J==
No [sn(s Ox(() = S: (n(s)dss O₂(1, Zx) 12, -ZZ1-
Rs(()
EA Vis(s)
j=1
1 N2
N₂
Gk
A
(ix(s) Zix- En(s) 02 ds,
B = N₁.
Σ
ZZix- - [x(ss 02 dd
No
i=1
(x)t)= 1 ".Ne Zikk Z (s) (s)
Rs(s)
B.8
The objective function for Liu et al.'s (2023) method
Next we will show that the objective function in RL from
Liu et al. s (023)) is
oc
YR k
Q" az) F.o
TT
xe, at
Po.n
X A
(A.8)
So L
ck
Yi)
where So is the Kaplan-Meits estimator.
Proof. Note that
E
A
T
P
02
ΣT:
T:
Sc
Σ T:
i=1
30

---

and thus
E
k
kk Ok: Tk
=1.
Sc
Zimi Ti)
Then we have
R
K
4
k
TT
It. at
=E Eb. Pb.
T
Tbi aks Tk
2k as
Sc
ΣE=) T:
km
k=
R
TeAx
18
Pb
FR, ak, TB
Tee
Sc
Σ Ti)
k=1
YAAR
pb
k-t
Øt, at
Sc
E
B.9 Large-sample replacement of d(M) by empirical dU
Lemma B.10. Fix a stage k and subjects 1.....N. Nr. Let Ua(t) be counting processes with predictable intensities (a(() and at-risk indicators Via (t). Assume:
(AI) (Doob-Meyer) (t) = Ax(t) + Ma(() with (t) = So Va(s)Auxss) ds; hence d{Ma))()) = dAx((s) = (s)
(A2) (Orthognnditynnddeeedennee across i) The martingales {Mik Miklin are pairwise orthogonal fe.g. subjects independent).
(A3) (Bounded predictable weights) For each i, Hss(s) is predictable, sup.sa, H4s 1 Cи < sc.
(A4) (Integrabilityy supi, Ast(s) S Cx cc.
Define, for E [0, Gk].
RN. (1) = NR
1 No
(
1 No
Σ [
Ha(() d(
Σ N
1 Nº
(s())
N
Then:
RN (1) = op(1 for each fixed i
sup t5G,
RNn (1) op(]),
In particular, for any bounded predictable Hik.
No Σ ( Hss(s) dUn(s))
No
1
H())))((M)(())) + o1))
No.
Proof. By (A1), Us = Aik + Mik and d(Maa) = dAik. hence
1
Σ [' (s)(s) (s).
RN (t)
N
Using orthogonality in (A2) and the isometry for martingale integrals,
No.
L Hs) d(Ma)(s)
No
Gx
C7CAGR
[12] = No
E
Via(s)Au((sdd
N₂.
31

---

Thus RN, (1) E 0 for each fixed l. For the uniform version, Doob's L inequality yields
CAG 0,
E
sup
RN, (t)
4E (G.))
LSG
No
hence supisGa [RN, (t) = Op()
Remark B.11. If we define (s) = (s)ds, then in the Cox PH model, (x(s) (s)e Z.a so (sM4x)(s) Vau(s) Zek oAo, (s);; In the Aalen additive model, Ax(ss (s) + B Zax so d(.ss)((s) Va (Ao,, 3 Zaa Jds. Lemma B.10 justifies replacing these predictable quadratic variations by the corresponding empirical averages of dUsa in large samples when evaluating variaaces/covaarianes of martingale integral terms.
32

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly state the goal of maximizing recur-
rent event probabilities using RL. the development of MDP framework, and theoreti- cal/experimental contributions, aligning with the content in Sections 3-7.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [No]
Justification: The paper focuses on methodology and experiments but does not explicitly address limitations such as computational complexity or the impact of assumptions (e.g.. Markov property) on real-world applicability.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
33

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Theoretical results (e.g., Theorem 3.2, Lemma 4.1) are supported by assump- tions (Markov property, independence of estimation errors) with detailed proofs in the appendices (e.g. A.7, A.10).
Guidelines:
The answer NA means that the paper does not include theoretical results,
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Simulation setups (Section 5.1), metrics (Section 5.2). and algorithms (Ap- pendix A.5) are described in detail, enabling result reproduction.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
34

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
The code is available htttpsr//gitubb.cm///fffeennnn/e NIPS2025- RL for-Survivaa
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/ cc/
pub Sublicc/e guides CodeSubma wwwosssissssioosooo olicy) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https? //nips. cc/ http://ulllccggggeoo / http://///giiieee/eoeetttttttte Sttp://ooeeSSmmmmma s onPoliong for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Hyperparameters (e.g., discount factor 7, soft-update DQN/DDPG), sample sizes, and stage configurations are detailed in Sections 5.1 and Appendix A.5.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Error bars (standard deviation) and statistical tests are reported in Tables 1-2 and Figures 2-3, with clear definitions of metrics like ARED and AEV.
Guidelines:
The answer NA means that the paper does not include experiments.
35

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eπror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: The paper does not specify hardware (e.g., CPU/GPU type, memory) or execution time for experiments, focusing only on algorithmic details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httig:/ttees//erroooocc////////t
Answer: [Yes]
Justification: The research involves ethical analysis of medical data with proper anonymiza- tion (e.g., Intern Health Study), adhering standards for secondary data use.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
36

---

The paper focuses methodological advancements in precision medicine and does not address broader societal impacts like privacy or clinical bias.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No high-risk models/datasets are released; the work focuses on theoreti- cal/experimental methods without deployment-specinn safeguards.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Datasets (Intern Health Study) and models (Cox/Aalen) are cited (c.g.. NeCamp et al., 2020), though specific licenses for real-worke data are not mentioned.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
37

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paapersw syperswithcoee com / data sets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No new datasets/models are released; the work relies on existing data and standard RL frameworks.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The study uses existing medical records (non-interventional) and simulations, not direct human subject participation.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
supplemental material is fine, but if the main contribu-
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
38

---

Justification: No direct human subjects research is conducted; ethical handling of existing data is assumed, but IRB specifies are detailed.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: LLMs are not used the core methodology; the work relies on traditional RL and survival analysis techniques.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (attps: httpss//eeerissscccoofoeereeeeee
LLM) for what should or should not be described.
39


Reviews:
[{"review_id": "19zUEcr4Jy", "reviewer": "Reviewer_9MY2", "summary": "The authors propose a nice solution to an important problem: finding an optimal strategy for maximizing the probability of the total recurrent-event duration exceeding a user given threshold (and thus going beyond just event times to accommodate recurrent events prevalent in healthcare). They cleverly formulate an auxiliary problem and theoretically demonstrate that the auxiliary problem (based on modified data) is consistent with the original optimization problem. Then inspired by MDPs, they transform the proposed objective into a Q-function and use a Bellman operator to derive an optimal policy. They show that this approach can reduce variance compared to only maximizing the total duration time, as is standard practice.", "questions": "**Questions**\n\n- Why not release code to improve reproducibility?\n- How would the authors recommend selecting \\alpha in practice?\n- What happens to variance benefits on data sets with lower/higher censoring?\n- Why not include a more detailed benchmark (e.g. Hargrave et al. NeurIPS 2024) or more baselines that directly model return distributions and could optimize tail probabilities?\n- What is the computational cost of training per epoch?", "limitations": "I was unable to find a Limitations section in the paper... the authors should add one.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- Good motivation for an important and understudied problem of strong clinical relevance. \n- The auxiliary problem trick is elegant and, I believe, novel. \n- Strong simulations showing fast convergence and improved/decreased variance. \n- Paper is well written with clear figures.\n\n**Major Weaknesses**\n\n- Lacking in empirical validation: only one real-world dataset on intern moods is used, which is not typical of medical settings. Why not include an EHR dataset (e.g. MIMIC-IV) or validate on a more detailed medical RL benchmark (e.g. Hargrave, Spaeth, and Grosenick NeurIPS 2024)?\n- Are there really no examples of RL methods optimizing either probabilities rather than expectations or on non time-to-event data? The statements in the paper surrounding this seem overly absolute.\n- Baselines could be stronger: comparison appears to only be against a Q-learning variant maximizing total time. \n- Threshold selection seems important, but how to do so is unclear to me. Lack of sensitivity studies or discussion on how  to learn \\alpha.\n- The paper is lacking ablation studies and discussion of memory/runtime/wall clock metrics.", "was_revised": true, "final_justification": "We thank the authors for their thoughtful responses, and provided the carry out the additional improvements they suggest, find they have addressed the concerns raised. We therefore raise our score to a 5.", "rebuttal": "**Response to Reviewer Comments**\n\nWe sincerely appreciate your valuable feedback. Below are our responses and revisions addressing the questions you raised.\n\n---\n\n### 1. Code Availability\n\n**Reviewer Comment**:  \n\"Why not release code to improve reproducibility?\"\n\n**Response**:  \nTo ensure reproducibility, our detailed source code and its explanation are available in the submission materials.\n\n**Revision**:  \nWe will add the following statement at the end of line 255 in Section 5 (Simulation Studies):  \n*\"Our code is available in the submission materials.\"*\n\n---\n\n### 2. Selection of $\\alpha$ in Practice\n**Reviewer Comment**:  \n\"How would the authors recommend selecting $\\alpha$ in practice?\"\n\n**Response**:  \nThe selection of $\\alpha$ depends on the research question. For instance, Jiang, et al. (2017) used $\\alpha = 0.5$ in their HIV study to maximize the probability of survival exceeding the median. In our research, $\\alpha = 1/14$ was chosen to maximize the probability of an intern's positive mood exceeding one day per bi-weekly period. We generally recommend $\\alpha$ of 0.5 or less, mainly because: (1) $\\alpha$ values near 1 rely too much on tail probability estimation, which is unreliable; (2) Additional simulation experiments show our algorithm performs better when $\\alpha$ is 0.5 or lower, addressing related concerns.\n\n**Revision**:  \nWe will add a new appendix section before A.2 titled:  \n*\"**A.2 $\\alpha$ Selection***\n*Mean (Standard Deviation) of AARED under different $\\alpha_k$*\n\n| $\\alpha_k$ | Aah Model            | Cox Model            | Baseline model       |\n|------------------|----------------------|----------------------|----------------------|\n| 1/14             | 19.9330 (0.0074)     | **19.9421 (0.0091)** | 19.83 (7.9e-3)       |\n| 1/7              | **19.9460 (0.0089)** | 19.9228 (0.0130)     | 19.83 (7.9e-3)       |\n| 3/14             | 19.1728 (0.0304)     | **19.8896 (0.0157)** | 19.83 (7.9e-3)       |\n| 2/7              | 18.9851 (0.0459)     | 19.8285 (0.0177)     | **19.83 (7.9e-3)**   |\n| 5/14             | 18.9972 (0.0438)     | 19.8193 (0.0207)     | **19.83 (7.9e-3)**   |\n| 3/7              | 18.9831 (0.0467)     | 19.7919 (0.0192)     | **19.83 (7.9e-3)**   |\n| 1/2              | 19.0010 (0.0435)     | 19.7900 (0.0218)     | **19.83 (7.9e-3)**   |\n| 4/7              | 19.0134 (0.0454)     | 19.7571 (0.0295)     | **19.83 (7.9e-3)**   |\n| 9/14             | 19.0015 (0.0387)     | 19.7232 (0.0315)     | **19.83 (7.9e-3)**   |\n| 5/7              | 19.0007 (0.0433)     | 19.6999 (0.0291)     | **19.83 (7.9e-3)**   |\n| 11/14            | 19.0027 (0.0453)     | 19.6703 (0.0515)     | **19.83 (7.9e-3)**   |\n| 6/7              | 19.0238 (0.0455)     | 19.6379 (0.0497)     | **19.83 (7.9e-3)**   |\n| 13/14            | 19.0107 (0.0434)     | 19.6413 (0.0373)     | **19.83 (7.9e-3)**   |\n| 1                | 19.0591 (0.0465)     | 19.6885 (0.0334)     | **19.83 (7.9e-3)**   |\n\n*Mean (Standard Deviation) of AAEV under different $\\alpha_k$*\n\n| $\\alpha_k$ | Aah Model  | Cox Model            | Baseline model |\n|------------------|----------------------|----------------------|----------------------|\n| 1/14            | **10.5408 (0.0027)** | 11.9509 (2.3916)     | 20.00 (19.47)        |\n| 1/7              | **10.8309 (0.0818)** | 13.9811 (4.4648)     | 20.00 (19.47)        |\n| 3/14             | 64.5515 (37.9364)    | **17.7416 (12.5118)** | 20.00 (19.47)        |\n| 2/7              | 78.1629 (14.4570)    | 20.6589 (16.5979)    | **20.00 (19.47)**    |\n| 5/14             | 78.5551 (13.8795)    | 22.2620 (26.8484)    | **20.00 (19.47)**    |\n| 3/7              | 78.5710 (14.2459)    | 23.5382 (27.2738)    | **20.00 (19.47)**    |\n| 1/2              | 78.6225 (13.8094)    | 24.4125 (35.8999)    | **20.00 (19.47)**    |\n| 4/7              | 78.6280 (14.3543)    | 25.0995 (50.6035)    | **20.00 (19.47)**    |\n| 9/14             | 78.6196 (14.6698)    | 27.7541 (67.8201)    | **20.00 (19.47)**    |\n| 5/7              | 78.4769 (17.6261)    | 29.4384 (70.1726)    | **20.00 (19.47)**    |\n| 11/14            | 78.2809 (19.1925)    | 30.3147 (91.0969)    | **20.00 (19.47)**    |\n| 6/7              | 78.0577 (19.7367)    | 32.2957 (104.3780)   | **20.00 (19.47)**    |\n| 13/14            | 77.9280 (19.0170)    | 32.6802 (79.8917)    | **20.00 (19.47)**    |\n| 1                | 71.9542 (83.7705)    | 29.4135 (76.7759)    | **20.00 (19.47)**    |\n\n*To clarify our recommendation for alpha selection, we will include the following remark on page 4, line 153: ”The selection of $\\alpha$ is tailored to the specific research question being addressed. For example, in Jiang et al. (2017)‘s study on HIV data, they chose $\\alpha = 0.5$, focusing on the optimal strategy for maximizing the probability of survival exceeding the median. In our study, we selected $\\alpha = 1/14$ to maximize the probability of an intern’s positive mood exceeding one day per bi-weekly period. We generally advise using an $\\alpha$ of 0.5 or less, mainly because $\\alpha$ values near 1 rely heavily on tail probability estimation, which can be unreliable. These results are consistent with the numerical results we obtained from our experiment results (see Appendix A.2 for details).“*\n\n\n*Furthermore, to better demonstrate the sensitivity of our method to different $\\alpha$ values, we conducted a new simulation experiment. The results will be presented in a new section, A.2, in the Appendix. The specific results are shown in the tables below, representing the results for Average Recurrent-event Duration of last 40 epoches (AARED) and Average Estimated Variance of last 40 epoches (AAEV), respectively. The results indicate that our method exhibits a more pronounced advantage when alpha is no greater than 0.5. This provides numerical simulation-based support for our recommendation regarding $\\alpha$.\"*\n\n---\n\n### 3. Variance Benefits Under Different Censoring Rates\n**Reviewer Comment**:  \n\"What happens to variance benefits on data sets with lower/higher censoring?\"\n\n**Response**:  \nThank you for the valuable suggestion. For improved clarity, we will add a new table to the Appendix A.3 to show the censoring rates at different stages. As can be seen from this table, the censoring rates vary across stages in the simulation experiments (ranging from approximately 15% to approximately 70%). This table, together with Table 3 and 4  in the original manuscript, demonstrates that the magnitude of the censoring rate has a limited impact on our method, and that our method can achieve good variance benefits under different censoring rates.\n\n**Revision**:  \nWe will add the following table in line 310:  \n*\"Censoring rates differ across stages. Below are the mean (variance) of censoring rates over 50 simulations:*  \n\n| Model    | Week 10       | Week 20       | Week 30       | Week 40       | Week 50       |\n|----------|---------------|---------------|---------------|---------------|---------------|\n| Aah      | 15.11%(0.004) | 29.80%(0.005) | 42.44%(0.004) | 56.78%(0.005) | 70.96%(0.006) |\n| Cox      | 14.95%(0.004) | 29.87%(0.007) | 42.70%(0.006) | 56.84%(0.006) | 71.09%(0.006) |\n| Baseline | 15.09%(0.005) | 29.64%(0.007) | 42.50%(0.005) | 56.56%(0.007) | 70.62%(0.008) |  \n\n*Tables 3 and 4 in Appendix A.3(origanol pdf) confirm that our method maintains strong performance under varying censoring rates.\"*\n\n---\n\n### 4. Benchmark Comparison\n**Reviewer Comment**:  \n\"Why not include a more detailed benchmark (e.g. Hargrave et al. NeurIPS 2024) or baselines optimizing tail probabilities?\"\n\n**Response**:  \nThank you for your suggestion. While Hargrave et al. (2024) present a reinforcement learning framework for longitudinal data, their work is limited to the complete observation setting, unlike our research which addresses censored data. Their approach becomes a special case of our framework when full observability is assumed. This is why we did not use it as a benchmark, as our previous simulations focused on methods capable of handling censoring. However, in response to your suggestion, we will conduct a new simulation study to directly compare our method to that of Hargrave et al. (2024).\n\n**Revision**:  \nWe will add the following to Related Work (line 81):  \n*\"Hargrave et al. (2024) proposed a RL framework for longitudinal medical settings but did not address censored data.\"*  \n\nAdditionally, we will add a new appendix section before A.4 titled:  \n*\"A.4 Comparison with EpiCare (including supplementary experimental results which includes the detailed simulation results that compare our method to that of Hargrave et al. (2024)).\"*\n\n---\n\n### 5. Computational Cost\n**Reviewer Comment**:  \n\"What is the computational cost of training per epoch?\"\n\n**Response**:  \nWe appreciate your question. Experiments were conducted on an NVIDIA GTX 1650. The results is shown in revision.\n\n**Revision**:  \nIn Appendix A.5, the following Table 5 includes the \ncomputation time of our proposed method and the \nbaseline method.\n*\"A.5 Computational Time*\n\n*Table 5: Training time per epoch (seconds)*\n\n| Model    | N=100  | N=1000 | N=10000 |  \n|----------|--------|--------|---------|  \n| Baseline | 0.22s  | 0.77s  | 25.50s  |  \n| Cox      | 0.89s  | 4.08s  | 35.66s  |  \n| Aah      | 0.56s  | 3.10s  | 42.30s  |\"\n\n---\n\n### Acknowledgments\nWe are particularly grateful for your insightful feedback, which has significantly contributed to the enhancement of our work. We have thoroughly addressed the issues you raised and have revised the manuscript accordingly.\n\n**Reference**  \nJiang, R., Lu, W., Song, R., Hudgens, M. G., & Naprvavnik, S. (2017). Doubly Robust Estimation of Optimal Treatment Regimes for Survival Data—With Application to an HIV/AIDS Study. *The Annals of Applied Statistics*, *11*(4), 1763-1786.\n\nHargrave, M., Spaeth, A., & Grosenick, L. (2024). EpiCare: A Reinforcement Learning Benchmark for Dynamic Treatment Regimes. Advances in neural information processing systems, 37, 130536–130568."}, {"review_id": "gP0RXFNj4X", "reviewer": "Reviewer_VWkH", "summary": "The work approaches the recurrent event data from a different angle: the probability of event occurrences. The authors tried to reformulate the problem as an equivalent MDP. Numerical studies are conducted to validate the effectiveness of the approach.", "questions": "See weakness.", "limitations": "NA", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 3, "strengths_and_weaknesses": "- Strength\n\nThe authors proposed a more practical view of recurrent event scenarios and describe the whole picture in a well-rounded way. Empirical studies were conducted to validate the effectiveness of the proposed approach.\n\n- Weakness\n\n1. The review is not convinced by the key theoretical theorem 2. How does adding an extra log in the objective still give the same optimal solution? The review is not convinced by the proof in Appendix 9, it seems to state that a greedy action simply solve the MDP. Plus, the reviewer believe that the result in Figure 2 also proves that adding an extra log will give different optimal solutions. The authors are encouraged to make this part more clear as the equivalence claims are build upon it. \n\n2. As the reviewer understand, the offline evaluation part is based on the simulation using the Cox. Will such evaluation cause biased results towards the proposed method based on Cox model?", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Reviewer Comments**\n\nWe sincerely appreciate your valuable feedback. Below are our responses and revisions addressing the questions you raised.\n\n---\n### 1. Theoretical Justification of Theorem 2\n**Reviewer Comment**: The review is not convinced by the key theoretical theorem 2. How does adding an extra log in the objective still give the same optimal solution? The review is not convinced by the proof in Appendix 9, it seems to state that a greedy action simply solve the MDP. Plus, the reviewer believe that the result in Figure 2 also proves that adding an extra log will give different optimal solutions. The authors are encouraged to make this part more clear as the equivalence claims are build upon it.\n\n**Response**:  \nWe greatly appreciate your comments. Following your suggestion,  we carefully examined the theoretical section and identified a missing condition in Theorem 3.2: the deterministic property of $P_0$. We emphasize that, because our work focuses on offline RL, this condition is naturally met, and therefore the correction to Theorem 3.2 does not affect the overall properties of our proposed framework.  In the revised manuscript, we will update Theorem 3.2 and provide its complete proof as below. \n\n**Revision**:  \nWe will revise the theorem in line 169 of the paper to:\n\n*\"**Theorem 3.2.** If for any $ \\boldsymbol{x}_t $, the supremum of $ V^{\\pi}(\\boldsymbol{x}_t) $ can be attained, and when $ P_0(\\boldsymbol{x}'|\\boldsymbol{x},a) $ is a deterministic transition kernel, with $\\Pi^v = ${$\\pi^v | V_0^{\\pi^v}(\\boldsymbol{x}_t) = \\text{max}(\\pi) V_0^{\\pi}(\\boldsymbol{x}_t) $} and $\\Pi^p =$ {$\\pi^p | \\text{P}^{\\pi^p}(\\boldsymbol{x}_t) = \\text{max}(\\pi)\\text{P}^{\\pi}(\\boldsymbol{x}_t)$}, then $ \\Pi^v = \\Pi^p $.*\n\n*（$ P_0(\\boldsymbol{x}'|\\boldsymbol{x},a) $ is a deterministic transition kernel if and only if $ \\forall \\boldsymbol{x},a $, there exists a unique $ \\boldsymbol{x}' $ such that $ P_0(\\boldsymbol{x}'|\\boldsymbol{x},a)=1 $ ）\"*\n\nFurthermore, we will revise the proof of Theorem 3.2 in lines 548 to 556 in Appendix to:\n\n\"**Proof:**\nFor any deterministic policy $ P_0 $, by definition, there exists a unique $ x_{t+1}^{P_0} $ such that:\n$$\nP_0(x_{t+1}^{P_0} \\mid x_t, a_t) = 1\n$$\nWe denote $ x_t^{P_0} = x_t $ and define\n$$\nR(\\gamma_t) = \\prod_{k=t}^K S_{T_k}(\\alpha_k G_k \\mid x_k^{P_0}, a_k)\n$$\nwhere $ \\gamma_t $ is the path:\n$$\n\\gamma_t = (x_t^{P_0}, a_t,T_t, x_{t+1}^{P_0}, a_{t+1},T_{t+1} \\dots, x_{\\bar{K}}^{P_0}, a_{\\bar{K}},T_{\\bar{K}})\n$$\nWe define the set of paths as $ \\Gamma_t^{P_0} $, then we can obtain:\n$$\nP^\\pi(x_t) = \\sum_{\\gamma_t \\in \\Gamma_t^{P_0}} \\mathcal{P}(\\gamma_t) \\cdot R(\\gamma_t), \\quad V_0^\\pi(x_t) = \\sum_{\\gamma_t \\in \\Gamma_t^{P_0}} \\mathcal{P}(\\gamma_t) \\cdot \\log R(\\gamma_t)\n$$\nwhere:\n$$\n\\mathcal{P}(\\gamma_t) = \\pi(a_t \\mid x_t^{P_0}) \\prod_{k=t}^{\\bar{K}-1} P_0(x_{k+1}^{P_0} \\mid x_k^{P_0}, a_k) \\pi(a_{k+1} \\mid x_{k+1}^{P_0})\\\\\n= \\prod_{k=t}^{\\bar{K}} \\pi(a_k \\mid x_k^{P_0})\n$$\nThen we can know that\n$$\nP^\\pi(x_t) = \\sum_{\\gamma_t \\in \\Gamma_t^{P_0}} \\prod_{k=t}^K \\pi(a_k \\mid x_k^{P_0}) \\cdot R(\\gamma_t) \\leq \\text{max} ({\\gamma_t \\in \\Gamma_t^{P_0}}) R(\\gamma_t)\n$$\n$$\nV_0^\\pi(x_t) = \\sum_{\\gamma_t \\in \\Gamma_t^{P_0}} \\prod_{k=t}^K \\pi(a_k \\mid x_k^{P_0}) \\cdot \\log R(\\gamma_t) \\leq \\text{max}({\\gamma_t \\in \\Gamma_t^{P_0}}) \\log R(\\gamma_t)\n$$\nwhere $ \\sum_{\\gamma_t \\in \\Gamma_t^{P_0}} \\prod_{k=t}^K \\pi(a_k \\mid x_k^{P_0}) = 1 $. \nIf we define:\n$\\Gamma_t^{p,\\*} = $ {$\\gamma_t^\\* \\mid R(\\gamma_t^\\*) = \\text{max}({\\gamma_t \\in \\Gamma_t^{P_0}}) R(\\gamma_t) $}\nand\n$\\Gamma_t^{v,\\*} = $ {$\\gamma_t^o \\mid \\log R(\\gamma_t^\\*) = \\text{max}({\\gamma_t \\in \\Gamma_t^{P_0}}) \\log R(\\gamma_t) $}\nHere, since $ a \\geq b \\iff \\log a \\geq \\log b \\ (\\text{where } a,b > 0) $, we have $ \\Gamma_t^{v,\\*} = \\Gamma_t^{p,\\*} $, which we denote as $ \\Gamma_t^{\\*} $. Then:\n$$\nP^\\pi(x_t) \\leq R(\\gamma_t^\\*), \\quad V_0^\\pi(x_t) \\leq \\log R(\\gamma_t^\\*), \\quad \\gamma_t^\\* \\in \\Gamma_t^\\*.\n$$\nDefine the policy set:\n$\\Pi^\\* = ${$\\pi \\mid \\sum_{\\gamma_t \\in \\Gamma_t^\\*} \\prod_{k=t}^{\\bar{K}} \\pi(a_k \\mid x_k^{P_0}) = 1$}\nThen we can know that the equality in the inequalities holds if and only if $ \\pi \\in \\Pi^\\* $ (see the following proof), so it can be concluded that $ \\Pi^v = \\Pi^p = \\Pi^\\* $\n1. $ P^\\pi(x_t) = R(\\gamma_t^\\*) \\Rightarrow \\pi \\in \\Pi^\\* $:\nFor $ P^\\pi(x_t) $, we denote $ R^\\* = \\text{max}({\\gamma_t \\in \\Gamma_t^{P_0}}) R(\\gamma_t) $. Then, for $ \\gamma_t \\notin \\Gamma_t^\\* $, we have $ R(\\gamma_t) < R^* $. Thus:\n$$\nP^\\pi(x_t) = \\sum_{\\gamma_t \\notin \\Gamma_t^\\*} \\mathcal{P}(\\gamma_t) \\cdot R(\\gamma_t) + \\sum_{\\gamma_t \\in \\Gamma_t^\\*} \\mathcal{P}(\\gamma_t) \\cdot R^\\* = R^\\*\n$$\nSo, $ \\sum_{\\gamma_t \\in \\Gamma_t^*} \\mathcal{P}(\\gamma_t) = 1 $, that is, $ \\pi \\in \\Pi^\\* $.\n2. $ P^\\pi(x_t) = R(\\gamma_t^\\*) \\Leftarrow \\pi \\in \\Pi^\\* $:\nProve by contradiction: Assume there exists $ \\pi \\notin \\Pi^\\* $, then $ \\sum_{\\gamma_t \\in \\Gamma_t^\\*} \\prod_{k=t}^{\\bar{K}} \\pi(a_k \\mid x_k^{P_0}) < 1 $. We denote $ R^\\* = \\text{max}({\\gamma_t \\in \\Gamma_t^{P_0}}) R(\\gamma_t) $. At this time:\n$$\nP^\\pi(x_t) = \\sum_{\\gamma_t \\in \\Gamma_t^\\*} \\mathcal{P}(\\gamma_t) \\cdot R^\\* + \\sum_{\\gamma_t \\notin \\Gamma_t^\\*} \\mathcal{P}(\\gamma_t) \\cdot R(\\gamma_t).\n$$\nSince in the second term, $ R(\\gamma_t) < R^\\* $ and $ \\mathcal{P}(\\gamma_t) > 0 $, we have:\n$$\nP^\\pi(x_t) < \\sum_{\\gamma_t \\in \\Gamma_t^\\*} \\mathcal{P}(\\gamma_t) \\cdot R^\\* + \\sum_{\\gamma_t \\notin \\Gamma_t^\\*} \\mathcal{P}(\\gamma_t) \\cdot R^\\* = R^\\* \\cdot \\sum_{\\gamma_t} \\mathcal{P}(\\gamma_t) = R^\\*,\n$$\nwhich contradicts \"$ P^\\pi(x_t) = R(\\gamma_t^\\*) $.\"\n\n---\n### 2. Potential Evaluation Bias Using Cox Model\n**Reviewer Comment**: As the reviewer understand, the offline evaluation part is based on the simulation using the Cox. Will such evaluation cause biased results towards the proposed method based on Cox model?\n\n**Response**:  \nThank you for your suggestion. Following your advice, we explored using a log-normal AFT model for data generation. This alternative approach mitigates the risk of estimation bias associated with the Cox model, as it offers a distinct modeling framework. The results presented in the revision demonstrate that the findings remain consistent with those obtained using the Cox model, even with this new data generation process. \n\n**Revision**:  \nWe will change in line 332 with *\"Below, we use the Log Normal AFT Model to estimate $\\pi$ in (1). It can be seen that the fitting effect of this model is better than that of Cox and Aah. Under this method, the calculated results of $ \\frac{\\log \\hat{\\text{P}}^{\\pi^{A \\text{ or } C}} - \\log \\hat{\\text{P}}^{\\pi^B}}{\\log \\hat{\\text{P}}^{\\pi^B}} $ are as follows:*\"\n\n| $\\alpha$ | Aah         | Cox         |\n|----------|-------------|-------------|\n| 0.07     | 2.814×10⁻⁴  | 2.233×10⁻⁴  |\n| 0.14     | 3.256×10⁻⁴  | 2.578×10⁻⁴  |\n| 0.21     | 3.526×10⁻⁴  | 2.791×10⁻⁴  |\n| 0.29     | 3.721×10⁻⁴  | 2.946×10⁻⁴  |\n| 0.36     | 3.875×10⁻⁴  | 3.068×10⁻⁴  |\n| 0.43     | 4.001×10⁻⁴  | 3.169×10⁻⁴  |\n| 0.50     | 4.108×10⁻⁴  | 3.255×10⁻⁴  |\n| 0.57     | 4.200×10⁻⁴  | 3.330×10⁻⁴  |\n| 0.64     | 4.282×10⁻⁴  | 3.397×10⁻⁴  |\n| 0.71     | 4.355×10⁻⁴  | 3.457×10⁻⁴  |\n| 0.79     | 4.421×10⁻⁴  | 3.511×10⁻⁴  |\n| 0.86     | 4.481×10⁻⁴  | 3.560×10⁻⁴  |\n| 0.93     | 4.537×10⁻⁴  | 3.606×10⁻⁴  |\n| 1.00     | 4.588×10⁻⁴  | 3.648×10⁻⁴  |\n\nthen redraw the figure using the data in the table , and add the following content in the appendix: \n\n*\" Log Normal AFT Model assumes that the natural logarithm of the survival time $ T_k $ follows a normal distribution: $ \\log(T_k) \\sim \\mathcal{N}(\\mu(x_k), \\sigma^2) $, where: The location parameter $\\mu(x)$ is a linear combination of the covariates $ x = [x_1, x_2, \\dots, x_n] $: $ \\mu(x) = a_0 + a_1 x_1 + a_2 x_2 + \\dots + a_n x_n $\nHere, $ a_0, a_1, \\dots, a_n $ are the regression coefficients to be estimated, reflecting the impact of the covariates $ x $ on the mean of the logarithm of the survival time. The scale parameter $ \\sigma $ is a fixed constant (independent of covariates), describing the standard deviation of $ \\log(T) $ and controlling the degree of dispersion of the distribution. The model estimates parameters by maximizing the log-likelihood function. The concordance index under the Model is as follows:*\n\n| Week | Concordance Index |\n|------|-------------------|\n| 1    | 0.551             |\n| 2    | 0.567             |\n| 3    | 0.540             |\n| 4    | 0.600             |\n| 5    | 0.533             |\n| 6    | 0.501             |\n| 7    | 0.583             |\n| 8    | 0.650             |\n| 9    | 0.523             |\n| 10   | 0.571             |\n| 11   | 0.554             |\n| 12   | 0.593             |\n| 13   | 0.521             |\n| 14   | 0.510             |\n| 15   | 0.608             |\n| 16   | 0.541             |\n| 17   | 0.578             |\n| 18   | 0.518             |\n| 19   | 0.536             |\n| 20   | 0.587             |\n| 21   | 0.543             |\n| 22   | 0.570             |\n| 23   | 0.583             |\n| 24   | 0.540             |\n| 25   | 0.718             |\n| 26   | 0.548             |\"\n\n--- \n### Acknowledgments\nThis revision addresses all reviewer comments with clear explanations, empirical support, and targeted edits to the manuscript. Let us know if further clarifications are needed."}, {"review_id": "HBf2peSRMn", "reviewer": "Reviewer_2BRd", "summary": "This paper optimizes multistage decision-making for alternating recurrent event data in survival analysis. By transforming recurrent event and censored data into a probability objective, the authors apply traditional deep reinforcement learning (DRL) algorithms to maximize returns. Experiments show that the DRL method, combined with various survival functions, achieves faster convergence and higher probability of recurrent-event durations in both simulated and real data.", "questions": "1) It seems that some of the theoretical analysis assumes a discrete action space, such as Theorem 3.2 and the assumption in lines 213-214, which is hard to be met in a continuous space. Do these assumptions still apply in a continuous action space, as used in DDPG?\n\n2) Despite the rigorous theoretical analysis, the code implementation is relatively simple. Does the core contribution lie in treating the logarithm of the survival probability given a specific threshold as the immediate reward?\n\n3) Could you explain the changes in the relative log probability shown in Figure 4? Specifically, why does it increase from 2/14 to 8/14 and then decrease? Does the absolute log probability follow the same trend?", "limitations": "YES", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1) This paper addresses the multistage decision-making problem by maximizing the probability that the duration exceeds a given threshold, offering a more practical approach than traditional RL objectives in survival analysis.\n\n2) The paper is well-structured and clearly motivated.\n\n3) Experimental results demonstrate clear improvements in both convergence speed and the probability of longer recurrent-event durations.\n\n4) The supplementary codebase helps in understanding the method.\n\n\n**Weaknesses**\n\n1) This work adapts the probability-based objective to the traditional RL framework with minimal modification, which may limit its impact in areas where RL is already widely used.\n\n2) Some of the theoretical analysis is difficult to understand. Please refer to the questions for further clarification. I would like to increase my score once these questions are properly addressed.", "was_revised": true, "final_justification": "Thanks to the authors for their reply. My concerns regarding the theoretical analysis have been well addressed in the rebuttal. I appreciate the contribution of incorporating recurrent data into the Q-function and would like to raise my score.", "rebuttal": "**Response to Reviewer Comments**\n\nWe sincerely appreciate your valuable feedback. Below are our responses and revisions addressing the questions you raised.\n\n---\n### 1. Continuous Action Space\n\n**Reviewer Comment**:It seems that some of the theoretical analysis assumes a discrete action space, such as Theorem 3.2 and the assumption in lines 213-214, which is hard to be met in a continuous space. Do these assumptions still apply in a continuous action space, as used in DDPG?\n\n**Response**: Thank you for pointing this out. Although the action space in the paper was formulated as discrete, the objective itself is applicable to both discrete and continuous action spaces. Following your advice, we have revised all relevant content in the paper (shown in the revision) to discuss discrete and continuous action spaces separately.\n\n**Revision**: \n- We will change *\"**Theorem 3.2**\"* to *\"**Theorem 3.2 (Discrete action space)**\"*, and then add a new theorem after line 170:*\"\n**Theorem 3.3 (Continuous action space)**  \nLet the state transition kernel $ P_0(x'|x,a) $ be deterministic (i.e., for any $ x,a $, there exists a unique $ x' $ such that $ P_0(x'|x,a)=1 $). Let the policy $ \\pi $ be represented by a deterministic function $ \\mu $ (i.e., $ a_k = \\mu(x_k) $), and for any initial state $ x_t $, the supremum of the value function $ V^{\\mu}(x_t) $ is attainable. Define the policy sets: $ \\Pi^v = ${$ \\mu^v \\mid V_0^{\\mu^v}(x_t) = \\text{max}({\\mu}) V_0^{\\mu}(x_t) $} , $ \\Pi^p = ${$ \\mu^p \\mid P^{\\mu^p}(x_t) = \\text{max}({\\mu}) P^{\\mu}(x_t) $} , then $ \\Pi^v = \\Pi^p $.\"*\\\n**Proof:** Given a deterministic transition kernel $ P_0 $ and a deterministic policy $ \\mu $, starting from any state $ x_t $, there exists a unique trajectory $ \\gamma_t^\\mu = (x_t, \\mu(x_t), T_t, x_{t+1}^\\mu, \\mu(x_{t+1}^\\mu), T_{t+1}, \\dots, x_{\\bar{K}}^\\mu, \\mu(x_{\\bar{K}}^\\mu), T_{\\bar{K}}) $, where $ x_{k+1}^\\mu $ is uniquely determined by the transition equation. Based on the above unique trajectory, the reward of trajectory $ \\gamma_t^\\mu $ can be expressed as $ R(\\gamma_t^\\mu) = \\prod_{k=t}^{\\bar{K}} S_{T_k}(\\alpha_k G_k \\mid x_k^\\mu, \\mu(x_k^\\mu)) $. Correspondingly, the objective functions can be simplified to $ P^\\mu(x_t) = R(\\gamma_t^\\mu) $ and $ V_0^\\mu(x_t) = \\log R(\\gamma_t^\\mu) $.\nDefine the set of optimal trajectories $ \\Gamma_t^\\* = $ { $ \\gamma_t^\\* \\mid R(\\gamma_t^\\*) = \\text{max}({\\gamma_t \\in \\Gamma_t}) R(\\gamma_t) $}, where $ \\Gamma_t $ is the set of all possible trajectories. Since $ \\log(\\cdot) $ is a strictly monotonically increasing function, $ \\text{max}(\\mu) R(\\gamma_t^\\mu)$ is equivalent to $ \\text{max}(\\mu) \\log R(\\gamma_t^\\mu) $. It follows that $ \\mu \\in \\Pi^p \\iff R(\\gamma_t^\\mu) = \\text{max}({\\gamma_t \\in \\Gamma_t}) R(\\gamma_t) \\iff \\log R(\\gamma_t^\\mu) = \\text{max}({\\gamma_t \\in \\Gamma_t}) \\log R(\\gamma_t) \\iff \\mu \\in \\Pi^v $, i.e., $ \\Pi^v = \\Pi^p = ${$ \\mu \\mid \\gamma_t^\\mu \\in \\Gamma_t^\\* $} $, which proves the equivalence of optimal policies.\n\n- Line 211 will be changed to *\"4.2 Optimization in discrete action space\"*, followed by adding *\"4.3 Optimization in continuous action space\nAccording to DDPG (Lillicrap et al., 2015), we define the Policy Function: $ \\mu(\\theta): \\mathcal{S} \\to \\mathcal{A} $, a parameterized function (e.g., neural network) that maps states to actions. For a fixed policy $ \\mu(\\theta) $, the Q-function follows the Bellman evaluation equation:  \n$Q^{\\mu(\\theta)}(\\tilde{X}_t, \\tilde{A}_t) = \\mathbb{E}^{P_0} [(\\tilde{X}_t, \\tilde{A}_t) + \\gamma Q^{\\mu(\\theta)} (\\tilde{X} _{t+1}, \\mu(\\theta)(\\tilde{X} _{t+1})) ]$\nTo optimize $ \\mu(\\theta) $, we maximize the expected return $ J(\\theta) = \\mathbb{E}[Q^{\\mu(\\theta)}(\\tilde{X}_t, \\tilde{A}_t)] $. Then, parameter updates are performed using the chain rule in DDPG.\n“*\n\n---\n### 2. Core contribution\n\n**Reviewer Comment**: Thank you for the question. Despite the rigorous theoretical analysis, the code implementation is relatively simple. Does the core contribution lie in treating the logarithm of the survival probability given a specific threshold as the immediate reward?\n\n**Response**: Our core contribution lies in constructing a novel framework that integrates offline RL with recurrent event data. Furthermore, we propose a new recurrent data-driven Q-function and prove that the proposed practical algorithm is consistent with classical Q-learning, rather than using the logarithm of the survival probability given a specific threshold as the immediate reward.\n\n**Revision**: For improved clarity regarding our core contribution, we have included a sentence on line 67:*\"Our core contribution is developing a novel framework that integrates offline RL with recurrent data, proposing a new recurrent data-driven Q-function, and proving our algorithm aligns with classical Q-learning.\"*\n\n---\n### 3. Figure 4 Performence\n\n**Reviewer Comment**: Could you explain the changes in the relative log probability shown in Figure 4? Specifically, why does it increase from 2/14 to 8/14 and then decrease? Does the absolute log probability follow the same trend?\n\n**Response**:Thank you for your question. In Figure 4, we previously presented the relative difference in log probabilities (calculated as $\\frac{\\log \\hat{\\text{P}}^{\\pi^A} - \\log \\hat{\\text{P}}^{\\pi^B}}{\\log \\hat{\\text{P}}^{\\pi^B}}$ for the Aah model and $\\frac{\\log \\hat{\\text{P}}^{\\pi^C} - \\log \\hat{\\text{P}}^{\\pi^B}}{\\log \\hat{\\text{P}}^{\\pi^B}}$ for the Cox model). To better address your question, we have now presented the absolute difference in log probabilities in the revision below. What is shown here is the relative magnitude, which does not have a clear pattern. The absolute probability values do not change in this way. As can be seen below, in fact, as $ \\alpha $ increases, the absolute difference ($\\log \\hat{\\text{P}}^{\\pi^A} - \\log \\hat{\\text{P}}^{\\pi^B}$ for the Aah model and $\\log \\hat{\\text{P}}^{\\pi^C} - \\log \\hat{\\text{P}}^{\\pi^B}$ for the Cox model) becomes larger and larger, indicating that our performance is getting better.\n\n| $\\alpha$ | Cox       | Aah       |\n|----------|-----------|-----------|\n| 0.07     | 3.54e-04  | 2.50e-04  |\n| 0.14     | 2.46e-04  | 1.08e-04  |\n| 0.21     | 1.24e-03  | 6.16e-04  |\n| 0.29     | 4.37e-03  | 2.71e-03  |\n| 0.36     | 1.03e-02  | 6.73e-03  |\n| 0.43     | 3.84e-02  | 2.81e-02  |\n| 0.50     | 1.16e-01  | 8.50e-02  |\n| 0.57     | 3.07e-01  | 2.24e-01  |\n| 0.64     | 4.36e-01  | 3.04e-01  |\n| 0.71     | 7.29e-01  | 4.97e-01  |\n| 0.79     | 7.83e-01  | 5.24e-01  |\n| 0.86     | 9.54e-01  | 6.52e-01  |\n| 0.93     | 8.43e-01  | 5.93e-01  |\n| 1.00     | 9.13e-01  | 6.65e-01  |\n\n\n**Revision**: We will revise lines 327-329 in the paper to *\"We then presented the log probability differences $\\log \\hat{\\text{P}}^{\\pi^A} - \\log \\hat{\\text{P}}^{\\pi^B}$ for the Aah model and $\\log \\hat{\\text{P}}^{\\pi^C} - \\log \\hat{\\text{P}}^{\\pi^B}$ across varying $\\alpha_k$ settings.\"* and then replace the figure with the data shown in the table.\n\n---\n\n### References  \nLillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.\n\n--- \n### Acknowledgments\nAll comments from the reviewers have been thoroughly addressed in this revision, with explicit explanations, empirical support, and precise modifications made to the manuscript. Do not hesitate to inform us if additional clarifications are needed."}, {"review_id": "69TiAz5T8G", "reviewer": "Reviewer_fiaA", "summary": "The paper considers RL to maximize the probability of durations of being in the good state being greater than some threshold. This objective decomposes and they take logs to maximize the sum of log survival probabilities.", "questions": "Questions: \n- How should the threshold be chosen? If it's not clear, then isn't this more or less the same as optimizing duration? \n- Why not max the durations of being in the good state directly? Since this problem maximizes the sum of log Survival functions. Why not just maximize the expected durations per stage? \n- However, in survival analysis, we are more interested in focusing on the probability that the recurrent-event duration is greater than a certain threshold.\n\nCan you please explain more why this is relevant? I see that this might come up a lot in survival analysis but it's not apparent why this is a decision-theoretically relevant quantity to the decision-maker.", "limitations": "n/a", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "strengths: \n\nsignificance: \n\n- the paper presents a specific challenge of alternating recurrent events and optimizes the probability that the recurrent-event duration within a time period is greater than a certain threshold (such as greater than 4/7 time of the time being in a good state). Because this is a joint \n- the paper nicely leverages standard survival analysis tools and censoring, which are common practical challenges in important settings. \n\n\n\nweaknesses: \n\n- significance: the choice of reward function seems somewhat driven by tractability. It doesn't seem appropriate from a first-principles point of view. What objectives that a decision-maker might care about correspond to the probability of duration increasing beyond a threshold? This seems to incentivize policies that \"satisfice\" (i.e. just achieve the threshold + one standard deviation of conditional duration times.) The authors compare computationally to maximizing the duration fraction via the work of Liu et al 2023 via finite-stage dynamic treatment regimes. But it's not clear if this is a weak baseline, i.e. if instead Liu et al 2023 considers the \"right\" objective with a \"wrong\" discretization via finite-stage DTR. \n\n- relevance: the set-up of alternate recurrent events seems somewhat narrow. It also seems that this can be handled with discrete-time MDPs as in this paper because the time of taking actions is fixed. So it seems somewhat overindexed to the intern RCT data, i.e. measurement and action of an underlying CTMC (that generates the underlying durations) at a fixed rate. This seems a bit of an approximation to the underlying goal, which is the duration fraction on the underlying CTMC. It would be nice to justify the correspondence here a bit. \n\nThe authors propose a joint probability objective: $P(T_t > \\alpha_t G_t, ..., T_K > \\alpha_K G_K \\mid X_t)$ which decomposes nicely into a product of survival probabilities and the authors optimize the log of this. However, it's not clear why they optimize the joint probability objective rather than just the long-run average occupancy time in the \"good\" state, i.e. $E_\\pi[I[s = good]]$\n\nClarity: \n\n- figure 1 is confusing and is not self-explanatory; the given explanation is not sufficient. \n- typos: \nWe first proposed a framework in RL that aims to maximize the probability, and applied it to recurrent events data. Meanwhile, we innovatively provided the division definition of recurrent events data", "was_revised": true, "final_justification": "The paper makes advances on tackling survival-type objectives with recurrent events. It has some limitations still, namely the arbitrariness around selecting alpha, and some of the discussions in the rebuttal phase would have improved the paper if it were submitted with that level of detail and context.", "rebuttal": "**Response to Reviewer Comments**\n\nWe sincerely appreciate your valuable feedback. Below are our responses and revisions addressing the questions you raised.\n\n---\n\n### 1. Figure 1 Clarification  \n**Reviewer Comment**:  \n\"Figure 1 is confusing and is not self-explanatory; the given explanation is not sufficient.\"  \n**Response**:  \nThank you for your suggestion. We will incorporate the descriptions from lines 115–138 into the figure to improve clarity.  \n**Revision**:  \n- $C \\in \\mathbb{R}^+$: Censoring time.\n- $Y_k$: Observed duration at stage $k$.\n- $\\Delta_k$: Censoring indicator at stage $k$.\n- $\\bar{K} \\in \\mathbb{N}^+$: Number of treatment stages a subject received.\n- $T_k$: Actual duration of stage $k$.\n\n---\n\n### 2. Threshold Selection  \n**Reviewer Comment**:  \n\"How should the threshold be chosen? If it's not clear, then isn't this more or less the same as optimizing duration?\"  \n**Response**:  \nThank you for the question. The selection of $\\alpha$ is tailored to the specific research question being addressed.  For example, in Jiang, et al. (2017)'s study on HIV data, they chose $\\alpha = 0.5$, focusing on the optimal strategy for maximizing the probability of survival exceeding the median. In our study, we selected $\\alpha = 1/14$ to maximize the probability of an intern's positive mood exceeding one day per bi-weekly period. We generally advise using an $\\alpha$ of 0.5 or less, mainly because: (1) $\\alpha$ values near 1 rely heavily on tail probability estimation, which can be unreliable; (2) to address your concern, we have conducted an additional  simulation experiment to demonstrate the sensitivity of our algorithm to different $\\alpha$ values. The results reveal that our method performs better when $\\alpha$ is 0.5 or lower.\n\n**Revision**:  \nWe will add a new appendix section before A.2 titled:  \n*\"**A.2 $\\alpha$ Selection***\n\n**Table 1: Mean (Standard Deviation) of AARED under different $\\alpha_k$ (rounded to two decimal places)**\n\n| $\\alpha_k$ | Aah Model       | Cox Model       | Baseline model  |\n|------------|-----------------|-----------------|-----------------|\n| 1/14       | 19.93 (0.01)    | **19.94 (0.01)**| 19.83 (0.01)    |\n| 1/7        | **19.95 (0.01)**| 19.92 (0.01)    | 19.83 (0.01)    |\n| 3/14       | 19.17 (0.03)    | **19.89 (0.02)**| 19.83 (0.01)    |\n| 2/7        | 18.99 (0.05)    | 19.83 (0.02)    | **19.83 (0.01)**|\n| 3/7        | 18.98 (0.05)    | 19.79 (0.02)    | **19.83 (0.01)**|\n| 1/2        | 19.00 (0.04)    | 19.79 (0.02)    | **19.83 (0.01)**|\n| 4/7        | 19.01 (0.05)    | 19.76 (0.03)    | **19.83 (0.01)**|\n| 5/7        | 19.00 (0.04)    | 19.70 (0.03)    | **19.83 (0.01)**|\n| 6/7        | 19.02 (0.05)    | 19.64 (0.05)    | **19.83 (0.01)**|\n| 1          | 19.06 (0.05)    | 19.69 (0.03)    | **19.83 (0.01)**|\n\n\n**Table 2: AAEV values (Mean ± Standard Deviation) under different $\\alpha_k$ (rounded to two decimal places)**\n\n| $\\alpha_k$ | Aah Model        | Cox Model        | Baseline model   |\n|------------|------------------|------------------|------------------|\n| 1/14       | **10.54 (0.00)** | 11.95 (2.39)     | 20.00 (19.47)    |\n| 1/7        | **10.83 (0.08)** | 13.98 (4.46)     | 20.00 (19.47)    |\n| 3/14       | 64.55 (37.94)    | **17.74 (12.51)**| 20.00 (19.47)    |\n| 2/7        | 78.16 (14.46)    | 20.66 (16.60)    | **20.00 (19.47)**|\n| 3/7        | 78.57 (14.25)    | 23.54 (27.27)    | **20.00 (19.47)**|\n| 1/2        | 78.62 (13.81)    | 24.41 (35.90)    | **20.00 (19.47)**|\n| 4/7        | 78.63 (14.35)    | 25.10 (50.60)    | **20.00 (19.47)**|\n| 5/7        | 78.48 (17.63)    | 29.44 (70.17)    | **20.00 (19.47)**|\n| 6/7        | 78.06 (19.74)    | 32.30 (104.38)   | **20.00 (19.47)**|\n| 1          | 71.95 (83.77)    | 29.41 (76.78)    | **20.00 (19.47)**|\n\nTo clarify our recommendation for alpha selection, we will include the following remark on page 4, line 153: *”The selection of $\\alpha$ is tailored to the specific research question being addressed. For example, in Jiang et al. (2017)‘s study on HIV data, they chose $\\alpha = 0.5$, focusing on the optimal strategy for maximizing the probability of survival exceeding the median. In our study, we selected $\\alpha = 1/14$ to maximize the probability of an intern’s positive mood exceeding one day per bi-weekly period. We generally advise using an $\\alpha$ of 0.5 or less, mainly because $\\alpha$ values near 1 rely heavily on tail probability estimation, which can be unreliable. These results are consistent with the numerical results we obtained from our experiment results (see Appendix A.2 for details).“*\n\n---\n\n### 3. Maximizing Duration vs. Survival Probability  \n**Reviewer Comment**:  \n\"Why not maximize the durations of being in the good state directly? Since this problem maximizes the sum of log Survival functions. Why not just maximize the expected durations per stage?\"  \n\n**Response**:  \n\n1. Challenges with Censored Data: Estimating durations in the “good state” becomes significantly more challenging when dealing with censored data, as complete information about the duration is not always available.\n2. Limited Reward Identifiability: In discrete settings, the binary nature of the rewards (1 or 0) lacks the discriminative power needed to effectively differentiate between various actions and their impact on stage duration.\n3. Focus on Cumulative Benefit: Our primary objective is to maximize the cumulative benefit over the entire process, rather than focusing solely on stage-specific rewards. This requires a framework that considers the long-term impact of actions.\n4. Robustness of the Method: We chose to use the survival probabilities as our objective precisely because of its robustness. Existing methods (e.g., Liu et al. (2023)) maximize expected durations using Kaplan-Meier estimates, but expectations are sensitive to outliers. Extreme survival times can skew averages, leading to a misrepresentation of actual patient benefits. \n\n**Revision**:  \nWe will add after line 51:  \n*\"The challenges in relevant research include the difficulty of estimating durations in the \"good state\" due to censored data, the limited discriminative power of binary rewards in discrete settings for differentiating action impacts, and the need to focus on cumulative benefits. Additionally, maximizing expectations is sensitive to outliers—extreme survival times can inflate averages, misrepresenting actual patient benefits (Bai et al., 2013; Jiang et al., 2017a, 2017b). Thus, we explore maximizing survival probabilities, which better reflect clinical outcomes.\"*  \n\n---\n\n### 4. Relevance of Threshold-Based Probability in Survival Analysis  \n**Reviewer Comment**:  \n\"In survival analysis, we focus on the probability that the recurrent-event duration exceeds a threshold. Can you explain why this is decision-theoretically relevant? It’s unclear why this quantity matters to decision-makers.\"  \n\n**Response**:  \nThank you for your question. The importance of this quantity can be explained from both theoretical and practical perspectives. Theoretically, compared to traditional methods that focus on the mean survival time, this quantity is more robust and less sensitive to outliers, providing a more stable representation of the patients outcome. Furthermore, by studying the survival curve, our method can characterize the impact of treatment at different time points. From a practical application perspective, the study of this quantity is relevant to many medical problems and has been investigated by some existing literature. We provide some specific examples below.\n\n1. **Bai et al. (2013)**: In the ASCERT study, CABG and PCI showed crossing survival curves—PCI had lower short-term risk, but CABG outperformed after 4 years. Here, 4-year survival probability precisely captured long-term differences, whereas average survival time obscured this.  \n2. **Jiang et al. (2017a)**: In ACTG 175, young patients on zidovudine + zalcitabine had significantly higher 400-day survival probabilities. Maximizing $ t $-year survival probability aligned better with short-term clinical needs than averaging.  \n3. **Jiang et al. (2017b)**: For HIV patients, median survival time (50% probability) was more representative of typical outcomes than restricted mean survival time, as it is less influenced by extreme values.  \n\n**Revision**:  \nWe will add after line 35:  \n\"In survival analysis, expectations (e.g., mean survival time) are sensitive to outliers and may obscure time-specific treatment effects. Threshold-based probabilities (e.g., median survival) better align with clinical needs. For example:  \n- Bai (2013) found CABG’s 4-year survival probability outperformed PCI’s, despite crossing survival curves.  \n- Jiang (2017a) showed 400-day survival probability was critical for evaluating short-term HIV treatment efficacy.  \n- Jiang (2017b) demonstrated median survival time’s robustness for typical patient outcomes in HIV studies.  \nThese studies validate that probability-based metrics overcome the limitations of expectations.\"  \n\n---\n\n### References  \n1. Bai, X., Tsiatis, A. A., and O’Brien, S. M. (2013). \"Doubly-Robust Estimators of Treatment-Specific Survival Distributions in Observational Studies With Stratified Sampling.\" *Biometrics*, 69, 830–839.  \n2. Jiang, R., Lu, W., Song, R., and Davidian, M. (2017a). \"On Estimation of Optimal Treatment Regimes for Maximizing t-Year Survival Probability.\" *Journal of the Royal Statistical Society, Series B*, 79, 1165–1185.  \n3. Jiang, R., Lu, W., Song, R., Hudgens, M. G., and Naprvavnik, S. (2017b). \"Doubly Robust Estimation of Optimal Treatment Regimes for Survival Data—With Application to an HIV/AIDS Study.\" *The Annals of Applied Statistics*, 11, 1763–1786.  \n\n--- \n### Acknowledgments\nWe have addressed all of the reviewers' comments in this revision, offering clear explanations, empirical support, and specific revisions to the manuscript. Please advise if further clarifications are needed."}]

Author rebuttal:

Meta-review: The paper proposes an RL framework for recurrent event survival analysis that maximizes survival probabilities above thresholds rather than expectations. It reformulates the problem via an auxiliary MDP, proves equivalence under log transformation, and demonstrates variance reduction and faster convergence.

**Strengths:**

- Novel, clinically motivated RL objective.

- Clear theoretical framing and proofs of equivalence/unbiasedness.

- Strong simulations with convergence/variance benefits.

**Weaknesses:**

- Limited real-world validation (only one dataset; no EHR benchmarks).

- Theoretical justification of log transformation remains unconvincing to some.

- Threshold selection feels ad hoc despite clinical motivation.

- Debate on whether probabilities beyond thresholds are more decision-relevant than expected durations.

--

Two reviewers raised their scores after rebuttal, noting improved theory, benchmarks, and sensitivity analyses. One reviewer remained unconvinced about the log transformation and equivalence, and another still questioned threshold justification and relevance. Overall, the rebuttal resolved several issues (but not all).
Decision: Accept (poster)