Time After Time: Deep-Q Effect Estimation for Interventions on When and What to do
Authors: Yoav Wald
Mark Goldstein
Yonathan Efroni
Wouter A.C. van Amsterdam
Rajesh Ranganath
Venue: ICLR 2025 Poster (2025)
Keywords: effect estimation
treatment times
irregular times
sequential decision making
Primary area: causal reasoning
TL;DR: A deep-Q effect estimation method for intervention on times and types of treatments

Abstract: Problems in fields such as healthcare, robotics, and finance requires reasoning about the value both of what decision or action to take and when to take it. The prevailing hope is that artificial intelligence will support such decisions by estimating the causal effect of policies such as how to treat patients or how to allocate resources over time. However, existing methods for estimating the effect of a policy struggle with \emph{irregular time}.  They either discretize time, or disregard the effect of timing policies.

We present a new deep-Q algorithm that estimates the effect of both when and what to do called Earliest Disagreement Q-Evaluation (EDQ). EDQ makes use of recursion for the Q-function that is compatible with flexible sequence models, such as transformers. EDQ provides accurate estimates under standard assumptions. We validate the approach through experiments on survival time and tumor growth tasks.

Published as a conference paper at ICLR 2025
TIME AFTER TIME: DEEP-Q EFFECT ESTIMATION FOR INTERVENTIONS ON WHEN AND WHAT TO DO
Yoav Wald yoav. sald@nyu..eu
Mark Goldstein'
Yonathan Efroni?
Wouter A.C. van Amsterdam²
Rajesh Ranganath"
ABSTRACT
Problems in fields such as healthcare, robotics, and finance requires reasoning about the value both of what decision or action to take and when to take it. The prevailing hope is that artificial intelligence will support such decisions by estimating the causal effect of policies such how to treat patients or how allocate resources over time. However, existing methods for estimating the effect of policy struggle with irregular time. They either discretize time, or disregard the effect of timing policies. We present new deep-Q algorithm that estimates the effect of both when and what to do called Earliest Disagreement Q-Evaluutiio (EDQ). EDQ makes use of recursion for the Q-function that is compatible with flexible sequence models, such as transformers. EDQ provides accurate estimates under standard assumptions. We validate the approach through experiments on survival time and tumor growth tasks.
INTRODUCTION
Sequential decision-maked making is common in healthcare, finance, and beyond (Chen et al., 2021a; Upadhyay et al., 2018). In hospitals, medical professionals administer treatments at different times based on the evolving observations of patient's condition; in financial markets, traders execute orders based on sequential information flows. Algorithmic decision support systems can optimize these processes by evaluating different policies with respect their expected outcomes. Estimating the difference in expected outcomes between various policies is a causal effect estimation question (Chen al., 202 la; Joshi et al., 2025; van Amsterdam al., 2024). This question involves several future treatment decisions taken at varying time points, hence it is a sequential decision eecision-making making problem.
Formally, this problem falls within the framework of off-policy evaluation (Fu et al., 2021; Uehara et al., 2022). A defining feature is that timings of observations and treatments are irregularly spaced, represented by stochastic point process with intensity whereas the type of the treatments at those times are specified as the marks of this process, governed by a distribution π. Unlike traditional formulations of off- off-policy evaluation that focus on action types, here the times must be accounted for, as they have large effect on the outcome. This formulation is relevant to many decision-making scenarios, for example, in transplants, where it is often desirable to delay treatment as much as possible to lower the risk of complications. However, delaying too much can result in deterioration in the patient's condition. Here, the type of treatment is fixed and what matters the timing-
Estimating the effect of intervention treatment timing is crucial part of evaluating sequential policies. With irregular times, frameworks for sequential decision decision-making making that discretize time can be problematic, discretization can be inaccurate or inefficient and requires choosing appropriate time scales. Further, length scales for decision-makinn Consider for example a patient with heart failure; such a patient may be stable for months years with occasional treatment adjustments made during visits to the cardiologist. However, at some point, they may experience acute decompensation (Felker et al., 201 1), which requires rapid treatment and
within a single trajectory can vary dramatically.
I New York University
2 Meta
3 University Medical Center Utrecht

---

Published as a conference paper at ICLR 2025
Q(Hi,e: 0) update
Counterfactual 1
observation z
2s
treatment a
Counterfactual 2
Xobe
21
Hip
Figure 1: summary of EDQ. Conditional expectations of the outcome, [p]Y] are estimated by Q,(H,; 0). Given observed trajectory H4 sampled under the training policy Xobe (bottom trajectory), we fit values (+ 0) by regressing them on a label g == Qi+o determined by a "counterfactual" trajectory H sampled from the target policy X. is the earliest disagreement time between the observed and counterfactual trajectories. It is either the time of the next observed treatment (counterfactual 1) or that of the treatment under the target policy (counterfactual 2).
intensive monitoring during hospitalization. Existing methods for continuous- time causal inference do scale gracefully, since they solve complex estimation problems, such as integrating importance weights across time (Røysland, 201 1). Those methods that scale to high-capacity models and large datasets not handle dynamic policies (i.e., policies that take past states into account) and are implemented with differential equation solvers (Seedat et al., 2022), restricting architectural choices.
In this work, we give two methods for off-policy evaluation with irregularly sampled data. Our contributions are as follows:
We define off-policy evaluation with decision poinl processes and develop Earliest Disagreement Q-Evaluation (EDQ), model-free solution to the problem. While other methods are intractable in high dimensions are limited to static treatments, EDQ eliminates these restrictions. EDQ is based on direct regressions and dynamic programming, which makes easily applicable to flexible architectures including sequence models such transformers.
In Theorem we prove that EDQ is an empirical estimator of the correct policy value. The estimator produces accurate causal effects under assumptions on causal validity based on Røysland 1); Røysland et al. (2022).
We validate the efficacy of EDQ through an experimental demonstration on time-to-failure prediction and tumor growth simulation tasks. For these tasks, we implement transformer-base solution. The results show EDQ's advantage relative to basclines that rely on discretization.
We define the estimation problem, develop solution, discuss related work and validate empirically.
2 OFF-POLICY EVALUATION WITH DECISION POINT PROCESSES
Consider decision process defined by marked point process P (Andersen et al., 2012; Snyder and Miller, 2012) over observations (which take values in X'), treatments (in A respectively) and real outcomes. We are interested in estimating overall quantity Y E IR that is a function of a sequence Y1, ..... of observed rewards. For convenience, we let Y = D Yy. where k is an index for observed outcomes along the trajectory.' 1 Though, the methods extend to other outcome functions like discounted future outcomes. We assume the number of rewards in the segment (0. TJ is finite.
We define Y% (w) 0 for an where k is larger than the number of outcomes the trajectory.
2

---

Published as a conference paper at ICLR 2025
Marked point processes. A marked point process is distribution over event times, along with distributions over marks, or details of the events at each time (i.e. treatment times and which treatment was given). We consider multivariate counting processes (t) = (N (t). (t)....) on the time interval [0, 7). For a univariate process, e.g. No, (t) is the number of events of type until time t. A trajectory of event times and their marks is a set H {(to,eo), (1,,0)) e1),..., (Ln, e We denote events up to time by He- = {(ta, ex) E H :::60 & S t and ((..++5)) for events in the interval (t, + 6]. We use 7. refer to events of type b on the trajectory and H for events of all types other than b.
Finally, we assume intensity functions H() = E E[dN( (t) Hc] for processes exist, Ntt)) is almost surely finite for any [0, and that the process can depend on its own history. That is, the filtration is the o-algebra generated by random variables (t) and their marks (Aalen et al., 2008).
2.1 PROBLEM DEFINITION
We follow notation from the RL literature (Upadhyay et al., 2018). We begin with the data generating process and then summarize our goal of inferring causal effects. This involves off off-policy evaluation under a distribution P. while observing samples from Pabs.
Definition 1. A marked decision point process P is a marked point process with observed components Ne for {a, y, }} that have corresponding intensity functions X5, and mark spaces X, R, A, and a multivariate unobserved process with intensity X" By default, we omit unobserved events from the trajectories H. hence = {(to: eg). (t1, (t,,e)..... (ty: en)3 where es E (XUAUR) The intensity function and mark distribution X (t))) T(A, (H4) are called the policy. The mark distributions for X, Y are denoted by Px (x, (x,) Py (H.).
process
Off-policy evaluation. We are given dataset of m trajectories, where trajectory 'H has n2 ob- These are sampled from an observed decision process Pobs with policy (Aober Tobs). Treatment times are samples from a counting process with intensity Xobe and treatments those times are sampled from Tohs- We reason about outcomes when (Ao Robs) is replaced with a target policy (At, R), with other processes in Pobs fixed. The resulting decision process is denoted P. and our goal is estimate the expected future outcome Ep [Y [Y] for all t E (") and H4 in the natural filtration associated with the point process.
servations: ((tx,,,e.,)))a je]m). (ej
When treat. To simplify notation, we omit the marks o (AA,)H H.) and focus on intensities X (t))H). That is, we explore interventions on when to treat (medicine weekly or monthly?) instead of how to treat (which medication?). Technically, "when" is the more challenging and underexplored part of the problem, and solutions can be easily extended to incorporate interventions on и using existing methods (Chakraborty and Murphy, 2014; Li et al., 2021). In the ASCVD example, this corresponds to reasoning about questions like: "Consider prescribing statins to patients with characteristics He, whose LDL cholesterol is below 180 mg/dl up to time t. What would be the expected change in 10-year ASCVD risk if going forward, we prescribe daily dose of statins for patients whose LDL cholesterol goes above 180 mg/dL., instead of the existing policy followed in the population?".
2.2 ROADMAP TO IDENTIFIABILITY VIA LOCAL INDEPENDENCES
The goal of this section is to elucidate the con- ditions under which the algorithm we present in section 3 estimates valid causal effects. We briefly summarize the essential conditions and supplement this summary in appendix D. Our assumptions to ensure identifiability of causal estimands follow Didelez (2008); Røysland (2011); Røysland et al. (2022), who study graphical models for point processes. In this setting, where the goal to intervene on N and estimate Ep [H.] under P rather than Pobss in the presence of unobserved processes U. Røysland (2012) define and analyze the
U1
N
N
U2
No
Figure 2: The assumed local independence graph for decision point processes, where our estimand is identifiable from observed data NN,N N2). following notions:
3

---

Published as a conference paper at ICLR 2025
A graphical condition called causal validity ensures that changing the treatment intensity from Xobs to the interventional X", while changing no other intensities, changes the joint distribution from Pots to P. A graph may not be causally valid when it contains unobserved variables U.
Local independence (Aalen, 1987; Schweder, 1970) adapts sequential exchangeability (Hernan and Robins, 2023; Robins. 1986) to continuous time. It is an asymmetric form of independence where one process' intensity may depend on the history of another, but not vice versa.
A certain set of local independences together are referred to as eliminability (a generalization of the backdoor criterion), which implies casual validity, even with unobserved variables U.
Consider the graph in Figure 2, where an edge means that the history of the source node affects the future of the target node (Didelez, 2008). It is possible show that it satisfies eliminability (see appendix D). To understand this condition, we start with the basic local independence requirements. Definition 2. For multivariate process ((t)= = (N*(t), (t)......) on variables V we say that N is locally independent of No given N or No A N (t (t A graphical local independence model (P, (P.G) class of processes P V and directed graph G (V, E), such that (b- - a) E F. + No + Nal holds for all P E P.
This condition means that the intensity of process only makes use of certain information from other processes. Røysland et al. (2022) package together the set of local independences that imply causal validity under the name eliminability, defined below. They use graphical criterion that is akin to using d-separation in graphical models, while we state the conditions in terms of the implied functional independencies of intensities. We expand on this in appendix D.
Definition 3. Let U be the set of unobserved variables. Suppose U can be written as a sequence (UI...., Uk) such that for each k either
(N", (U>)) is locally independent of Us given NU,,N,,,N N" (UK,, or
No is locally independent of Uk given (N*, .N,,N' Ne,,NNUUS U..).
Then the graph said to satisfy Eliminability. Here, for sets A, C and variable b we say A is locally independent of given A AUC C if each a E A is locally independent of b given A AUG, C {a}.
We summarize that causal validity holds under the local independences that satisfy eliminability. We assume the graph in Figure 2, which satisfies these assumptions. We also assume mutual independence of increments of all processes at a one time rule out instantaneous effects. We refer to validity and this independence together ignorability, in accordance with existing terminology confounding. Assumption 1. Ignorability (in continuous time) is satisfied when:
1. the graph satisfies causal validity.
2. the increments of features, treatments, and outcome are mutually independent given the history, i.e.. (IdN*(t),,, X.) IL (t) 1L (dN*(t), Y.))
In addition to ignorability, we require a second, standard assumption, overlap, for the conditional expectations we estimate to be well-defined. Recall that the interventional distribution P is defned by replacing the treatment distribution in Pobs. i.e., replacing No.. with Xo.
Assumption 2. Overlap is said to hold between the observational and interventional distributions, Poss and P, if P is absolutely continuous with respect to Pohss denoted by P < Pobs-
Ignorability and overlap are the core assumptions that allow identification in our setting. Under these assumptions, we can now present algorithms for estimating causal effects in continuous time.
MODEL FREE OFF-POLICY EVALUATION FOR DECISION POINT PROCESSES
To estimate [p[Y)] for times and Hr that overlap with Pobs, we express the expecta- tion recursively as a function of expectations | for some > 0 and trajectory H. Then, assuming that expectations at times larger than t have been learned correctly, this recursive expression allows us to propagate information for conditioning on earlier histories.
4

---

Published as a conference paper at ICLR 2025
Let us describe this solution in more detail, as applied in the context of Q- Q-evaluation.
Algorithm 1 Fitted Q-Evaluation (discrete time)
1: Input: Trajectories {Hi 1=1+
Fitted Q evaluation (FQE) in discrete time. Q-evaluation relies on the tower prop- enty of conditional expectations, given be- low in eq. (I). In discrete time decision processes, where we consider d = the property suggests a dynamic programming solution that we lay out in algorithm 1 (Le et al., 2019; Watkins and Dayan, 1992). Here, Hi includes all treatments and ob- servations up to and including time i, and since they occur simultaneously, there are exactly of each. H4+1 is defined in the same manner, except that includes d,+1 sampled from the target policy πT
2: Policy π _ 2 x A + A
3: Model class F where f: /:U x A R
4: Initialize Q randomly
5: Set T. +- max (f, E Hil E |ml]
6: for N rounds do
Vi,t E [Ti] 7: Draw āt+1 ~ T(- T(11H1,,)xx4t)
8:
Hi,1+1
( (t+ 1,ū++1))
9:
Set yi.t = -iit + ((H,t))) Vi,lE [Ti]
10:
Q f arg min JEP
Σ
(I(Hs,.) ic.))
11: end for
12: Return Q
Ep H = ER P(PP)) Ep [
An attractive property of this algorithm is that is model-free. That is, to form the label ÿ we only need to sample d from our target policy æ, while model of P(xx+1|H4) is not necessary.
With accurate optimization over a sufficiently expressive hypothesis class and arbitrarily large datasets, algorithm I returns correct estimates. This is because if we fix Qc+1 (H(++) and assume that it accurately estimates Ep Ep.(E.244 Y, [.) then the minimizer of the regression is the conditional expectation, equal to Ep 122z Ha) according to eq. (1). The model-free solution is enabled by the equality + (Ht)) = Poba
which validates the use of xi,++1 to form Øi. In practice, we take gradient steps on randomly drawn times and training samples instead of walking backward from T to 1. Crucially. for 5 > 1, e.g. 5 = 2, we have P(x (Pa) * Pobs (x++2) (He). Hence, an algorithm using eq. (1) must either be model-based, or resort to solutions such as importance weights that suffer high variance (Hallak et al., 2016; Precup et al., 2000), or restrict the problem, e.g., by discounting rewards (Harutyunyan et al., 2016; Munos et al., 2016; Precup et al., 2000).
Challenges in application to continuous time. Moving to continuous time, the tower property turns into a differential equation, and solving it requires tools that go beyond common FQE (c.g.. Jia and Zhou (2023)). Regressing to outcome that is arbitrarily close to the observation at time t is ill-defined. While we may work under a fine discretization of time, this approach is wasteful, as a single update in the minimization for estimating Q2 takes into account the development of the process in the interval 1,,++ 5). and for small values of this will usually yield a very small change to the estimate. Hence, intuitively, when updating Qc. we would like to use estimates of Q++8 for a large d. As explained above, this is seemingly difficult to achieve in a model-free fashion. However, for point processes, since the number of decisions over [0, T] is countable, seems plausible that a simple and efficient dynamic programming solution can be devised. In what follows, this is what we present.
3.1 EDQ: FITTED Q-EVALUATION FOR DECISION POINT PROCESSES VIA EARLIEST DISAGREEMENT TIMES
We wish to reason about what the outcome would have been for an observed trajectory sampled under policy Ag had we treated it with policy Xa from time onward. Intuitively, it seems plausible that we can use similar approach to FQE, but instead of going one time unit forward, we can move to the first time + 5 where the two policies differ their treatments. That is, we will sample alternative treatment trajectories, asking what the target policy X would have done at each time, given the observed history up until that time and find the earliest disagreement. The resulting algorithm is rather simple, and summarize it graphically in fig. 1. The attractive property of this approach is that the "lookahead" time is adaptive. It will likely be short when applied in parts of trajectories where many treatments are applied, and longer when treatments are scarce.
To formalize the method, we present some additional notation, prove an appropriate variation of the tower property, and explain how it is operationalized by the implementation in algorithm 2.
5

---

Published as a conference paper at ICLR 2025
Definition 4. For process Pots and policy (X, define an augmented process N (N , ,N* where intensities are independent of the history of No (1) the intensity of No is X (t == (t Hi Hee ). i.e. X where history of treatments is given by H 2 and (2) X (t H4) = Xbe H, ;1, for e E {aobe o. y). For a trajectory H. ~ P and time t t [0,T), define oz (1) min(u L
1, (u.-) E Holaore), where 51z(1) = T I when the set is empty.
The augmented process maintains an additional treatment trajectory. H as an alternative to H the one observed under Xobs- From definition 4, we observe that the intensities of the augmented process N do not depend on ' history. It follows that the marginal over {z, 2, is Pohss while H has similar role to the alternative treatment â++1 in algorithm 1. This notation is helpful for denoting sampled alternative trajectories over time intervals. Finally, (t) denotes the earliest disagreement between observed and target treatments after time t.? Our model-frre evaluation method is based on the following result, which expresses our estimand an expectation over trajectories H. Theorem Let Pabs be marked decision point processes, P the process obtained by replacing the policy with (X2, x), and P the augmented process obtained from P, Pabs definition 4. Further, let t [0, T and Hi measurable w.r.t P. Under Assumption 2, we have that
Ep [Y]
E ER P(11H)
Ep Y
H1+ He U Gota (1048
Takeaways from Theorem 1 and derivation of EDQ. Equation (2) suggests method to calculate expectations Ep(Y(H+) similar to how FQE follows from eq. (1). The practical version of the resulting method, EDQ, is given in algorithm 2. To arrive at the method from eq. (2), let examine the regressions solved in algorithm 2. Taking random variable Ys, which is the sum of all outcomes after some time s, eq. (2) can be rewritten as follows (see Appendix for a detailed derivation).
Ep [Y HH]
Σ
yk = En
Σ
yk + Ep Y (04 He-s He- = He U H Mabe
(1,+5
(L..
(E (Ett,,8&
Q(He+&mtt)
Q(H6))
Algorithm 2 uses gradient descent to fit Q functions that, at optimality, satisfy a self-consistency condition that appears in the equation above, ((H)) B |2 yk + Q(H(+611(0)) To this end, it solves regression problems where Q(H(,,) is fitted to label ýs- This label is defined in line 7 and coincides with the term in the expectation E [-] above. To conclude that the algorithm estimates
the effect of interest, two simple arguments suffice: (1) uniqueness argument showing that the selffconsistency equation holds only when (HH,) matches Ey H4.), and (2) that despite ignoring the unobserved process X", the estimation yields the desired causal effect due to the ignorability assumption. We summarize the conclusion below and provide detailed steps in appendix D.
Corollary 1. Under assumption I, a Q-function satisfying eq. (3) yields the causal effect of the intervention that replaces (Nots Tobs) with (A",T). ..)
An analogue of eq. (2) holds for discrete-time decision processes, where EDQ bears some resemblance to the eligibility traces approach of Precup et al. (2000). We provide this result in appendix B.3 for completeness but focus here on the point process case, as this our main motivation and where the earliest disagreement approach is most fruitful.
4 RELATED WORK
Our coverage of related work is divided into an overview of works that solve adjacent tasks to ours, before transitioning into detailed discussion in section 4.1 about techniques more closely aligned with our goal of large scale causal inference in sequential decision making.
Causal inference with sequential decisions. Causal effect estimation for sequential treatments is usually studied in discrete time under the sequential exchangeability assumption (Hernan and
Note that this is slight abuse of notation, since Hl not random variable.
'The minimum treatment time is also the earliest disagreement, since under some regularity conditions, the processes N and ao have probability of jumping simultaneously.

---

Published as a conference paper at ICLR 2025
Algorithm 2 Earliest Disagreement Fitted Q-Evaluation
1: Input: Trajectories (-
2: Policy (-(-)H4), (-1+4)
3: Initialize 0 randomly
4: for N rounds do
5: Draw Unif((o, and i Unif((m])
6: Draw H ~ P(-(H(,)) and set (1,1466g(00 H, U H Mobss
(1++6((1)
7:
D. Σ NE::; yk Q(H,, 0)
(t)
8:
0 ++ 0 V (Q(Hi,)))) 9.)
9: end for
10: Return ((:
Robins, 2023; Robins, 1986). Addressing unobserved confounders of interest (Namkoong et al., 2020; Tennenholtz et al., 2020), but this is beyond our scope here. As described in section 2.2, the framework of Røysland et al. (2022) draws a parallel to sequential exchangeability for continuous time, which adopt here. For estimation in these continuous-tine problems, several methods have been explored (Lin et al., 2004; Lok, 2008; Røysland, 2011; 1; Rytgaard et al., 2022; Zhang et al., 2011). Most do not scale large and high-dimensional datasets. For instance, Røysland (2011) 1) requires estimating an integral of propensity weights over time; Rytgaard et al. (2022) propose targeted estimator that fits each process in Pobs but it is limited interventions at fixed times (i.e. no interventions on treatment schedules) and it is unclear how to implement it with expressive models. Nic al. (2021) study the effect of timing t when applying a fixed policy from time onward. However, they do not discuss interventions on all treatment timings, as we do here.
Reinforcement learning (RL) techniques. As discussed in section 3, EDQ is related to FQE and Q-learning (Le et al., 2019; Murphy, 2005; Watkins and Dayan, 1992), and to n-step methods from RL (De Asis et al., 2018; 8; Munos et al., 2016; Precup et al., 2000). In the causality literature, Q-learning often appears in the context of dynamic treatment regimes (Chakraborty and Moodie, 2013; Chakraborty and Murphy, 2014), where discrete-timg time policies are learned off- off-policy. All these methods not applicable to irregularly sampled times. Works in RL that consider irregularly sampled times, either do not intervene on treatment times, or operate in the on-policy setting (Qu al., 2023; Upadhyay al., 2018 8), incorporate continuous- decision transformers (Chen et al., 2021b). The latter facilitates the recommendation of sets of actions arrive at desired outcome (Zhang al., 2023) rather than evaluating policy of interest. Recent work suggests that goal-conditioned imitation learning methods such as decision transformers may fail to estimate the causal effect of actions (Malenica and Murphy, 2023) in some scenarios where there are no unobserved confounders, whereas Q-learning methods produce correct estimates.
positional embeddings into
We next discuss scalable causal estimation methods for sequential treatments. We delineate assump- tions, implementation choices, and subsequent properties of solutions from recent work.
4.1 LARGE SCALE ESTIMATION APPROACHES FOR SEQUENTIAL. TREATMENTS
Notable early work on machine learning for estimating counterfactual quantities related to treatment timelines. Schulam and Saria (2017), used Gaussian Processes for estimation. Limitations such as scalability and incorporating various features prompted the development of deep learning approaches.
Large scale models. One family of solutions (Bica et al., 2020; Lim, 2018; Melnychuk al., 2022) took an important step forward by using RNNs and transformers for estimation. All of these methods build on the idea of learning balancing representations (Johansson et al., 2016). Roughly, these are representations under which the treatment is randomly assigned. This facilitates the training of high-capacity effect estimators on large datasets, but these works are restricted discrete times.
Dynamic policies. The above methods can only estimate effects of static treatments, meaning the treatment plan cannot dynamically depend on future observations. For instance, consider a policy that prescribes a daily dose of statins, and if the patient develops side effects in the future, switches
"We mark TE-CDE with as non-scalable, since the algorithm relies differential equation solvers which limits the scalability of methods can use.
7

---

Published as a conference paper at ICLR 2025
Table Qualitative comparison estimation methods under sequential treatments, compatible with large scale ML. Our method EDQ is marked in bold, while others can be found in (Bica et al., 2020;
Chakraborty and Moodie, 2013; et al., 2019; Li et al., 2021; Lim, 2018; Melnychuk al., 2022;
Schulam and Saria, 2017; Seedat et al., 2022). The acronym DP stands for Dynamic Programming.
Properties
Estimation Method
Irregular Times
Large Scale
Dynamic Policy
Prop. Weights
Model
Balancing
DP
Based
Rep.
CGP
X
X
CRN, CT
X
/
R-MSN
X
V
TE-CDE
J
x
:
G-Net
X
V
FQE
X
J
EDQ
V
to another medication. This policy depends possible future states, which the above methods do not accommodate. G-Net (Li et al., 2021; Xiong et al., 2024) takes a model-based approach, where models are fit for both (Tba(A()))HL dynamic policy is estimated when Tobe is replaced with the desired policy and conditional expectations of Y are estimated with Monte-Carlo simulations.
(()), and ( Y (())HH, Then at inference time,
Irregular times. None of the above solutions address irregular observation and action times, key focus of this paper. Some steps in this direction have been taken by Seedat et al. (2022); Vanderschueren et al. (2023). Past work on irregular times includes Seedat et al. (2022), who combine balanced representations with a neural controlled differential equation architecture suited for irregular sampling, while Vanderschueren et al. (2023) use reweighting to adjust for sampling times that are informative of the outcome. However, these methods mitigate sampling-inducee bias rather than estimate outcomes under interventions treatment times. We further discuss them in appendix C.
Table summarizes the above techniques, along with FQE and EDQ. Notably, EDQ possesses the desirable qualities mentioned here and handles interventions X", which other solutions do not.
5 IMPLEMENTATION AND EXPERIMENTS
To implement EDQ for experimentation we use the GPT-2 architecture. Each token is a concatenation of embeddings of time t, value z; and event type e; E {A, X.Y. AT). The event types A, ., cor- respond to actions, features, and outcomes, while AT is introduced for convenience to represent cases where time passes but no events occur. Both absolute times l, and time gaps AT use continuous time positional embeddings: the k-th dimension sin (#C-*/datme) for even and cos (k--(k11)//mn for odd k. Here 10 and deime the embedding dimension. We also keep a target network and update it with soft-Q f updates, as is common in Deep Q Q-Networks, e.g., Van Hasselt et al. (2016).
5.1 BASELINES
We are unaware of baselines for effect estimation on treatment timing with high-dimensional or long-sequence data. Thus, we implement two baselines that let glean important aspects of EDQ.
ERM/ MC is an Empirical Risk Minimizer (ERM) trained to predict observed outcomes, which, also known Monte-Carlo (MC) prediction in RL and policy evaluation. We the same GPT-2 architecture and data representation as EDQ, but instead running algorithm 2, we train fo (He) by minimizing prediction loss observed data. Given each training trajectory H, with outcome label yı. we solve ming Li,t,,,,.)HH (C( (C(fe(HH;))) yi). where E(.,-) is the squared loss. This method estimates outcomes under Aobe, therefore we expect it to perform as well as or better than off- off-policy evaluation methods, including EDQ, when À dobs, and suffer drop otherwise.
FQE is implemented following section 3, but with discretized time and Q-updates using one timestep forward. At each iteration, we draw training example i E [m] and time [0, T). define Ú, Vt+1 + Q++1 (7++1; 0)), and take a gradient step on loss ((Q,,H, 9.). Similarly, we define discrete-time approximations of our policies of interest, described later. The positional embeddings
8

---

Published as a conference paper at ICLR 2025
Vital Treatment
ERM/ MC
FQR
EDQ
Aua 0.1 0.02 1.04
Aoha 0.1
0.20 = 0.007
0.20 0.005
Aulox 0.5
0.38 + 0.011
0.20 + 0.006
2-
Aim 0.5
Aota 0.5
0.11 1 0.004
0.197 0.013
0.10 0.003
Aohs 0.1
0.28 + 0.004
0.31 0.01
0.11 + 0.006
30
Figure 3: Left. An example of a trajectory from our simulation. The blue curve denotes the value of the vital 2; and red dots mark treatment times. Right. Normalized RMSE under the different simulation settings. The mean is taken over all points in the history of patients in the test data. Rows colored blue have Anbs = Aint and we expect all methods to perform well since train and test distributions match. Red rows are those where the effect of an intervention needs to be estimated.
correspond to discrete times, and the representations of actions, features and outcomes at each time are concatenated. Other than this, we use the same architecture and hyperparameters of EDQ. This baseline examines the effects of time discretization on estimation and optimization.
Computational complexity: The per-iteration runtime of EDQ is similar to FQE, which is common tool for large-scale offline RL problems, e.g. Paine et al. (2020); Voloshin et al. (2021). EDQ and FQE differ in computation times due to sampling methods from the target policy used to draw the treatments used in the Q-update. We discuss this in appendix A.
5.2 SIMULATIONS ON TIME TO FAILURE AND CANCER TUMOR GROWTHI PREDICTION
To validate our method, we construct two settings. The first is to predict the effect of treatment timing policies on patients' time-to-event. The second uses cancer tumor growth simulator from Geng et al. (2017) to form a policy evaluation problem on applications of chemotherapy and radiotherapy.
Simulators. We use two simulators. (i) Time-to-failure: In this setting, each data point simulates the vital of a patient z; E R4 measured regularly at a frequency of one time unit, and treatments a, E R. assigned irregularly in time according an observational policy. Without treatment, the vital drops linearly dx,/ dt = - (a + {t) where a > 0 and & ~ N(0, σ) is a noise term drawn at each time unit. Upon receiving treatment, the vital rises by an amount inversely proportional to the number of treatments, 1 < k < m, applied up until that time, where m is the maximal number of treatments that a patient can receive. That is, the efficacy of treatment reduces with repeated applications. We also inject small noise terms into the treatment dosage that a patient receives, which further affect the vital and add randomness to the problem. Section 5.2 shows an example simulated patient trajectory. (ii) Tumor growth: We use the experimental setting from Bica et al. (2020), which other works use to study irregular sampling (Seedat et al., 2022: Vanderschueren et al., 2023). As this is commonly used simulator, we defer its details to appendix A and focus on the type of irregular sampling and policies we use. The simulator works in discrete time & E [7']. and irregular sampling is induced by the features being unobserved at certain times. Namely, the covariate T2 E R. which represents tumor volume is observed with probability ((#)-::1-1/ dmaxx - 1.5), where 2t-d:t- is the average tumor volume over the last d timesteps, and dmax is the maximum considered volume.
Outcomes and policies. For (i) dime-to-failure our outcome of interest is failure time y E R.,, where patient dies if the vital drops to value of 0. We focus effect estimation for interventions on a rate parameter X that controls the timing of treatment. At each time t where the observed vital crosses threshold, i.e. 24 exponential distribution ~ exp(X*) (X*) and treatment is applied at t + 6. The threshold r and the dosage of treatment given are also part of the policy æ, yet to focus on the effects of timing do not intervene on them in this experiment. At each experiment we observe m patients treated under a policy with \" = Aobs and aim to reason about the expected failure times under the interventional Ajat- In (ii) tumor-growth, the goal is to predict ZT, where T 20, under policy that at time i assigns treatment a, from four possible options: no-treatment, radiotherapy, chemotherapy, or combined
for some predetermined r E R.,, random time is drawn from an
Note that the vital changes outside measurement times, hence death time does not generally coincide with vital measurement times.

---

Published as a conference paper at ICLR 2025
ERM/ /MC
FQE
EDQ
ERM/ MC
FQB
EDQ
Aun 0.2
Autoe
0.17 0.00
0.18
0.178
(5,2hu (6, 0.76)
Acta
0.228 1: 0.01
0.20 0.03
0.178 1: 0.01
(h (6.5
0.034 0.001
0.048
0.001
0.037
0.001
12.0 (10,0.5)
0.07 = 0.004
0.080 + 0.013
0.052 = 0.006
Ami
Acta
0.04
0.013
4.000
Auise 0.2
0:32 + 0.02
0.31 0.01
0.22 + 0.007
Figure 4: Left. Normalized RMSE on the tumor-growth simulation. All methods are affected by distribution shift. EDQ is the most robust out of the methods considered. Right. Normalized RMSE for time-to-failure simulation on short trajectories.
therapy. Therefore, the estimation here is both on "when" and "what" to do. Policies are determined by two parameters (7, 3) and assign each type of treatment with probability a (7 - -)+ + -- (ast) ) Here, Wlast the last observed volume and 3 is an intercept controlling how often treatments are applied, while controls the dependence of treatment assignment tumor volume. Finally fast is the time of the last treatment, and the term - tlast induces lag between consecutive treatments.
Experiments. We perform two sets of experiments for time-to-failure In the first set, trajectory lengths range in 10 100, and there are 5 treaments. In the second set (results in Figure 4, right), we change the parameters of the problem by taking high slope and capping at one treatment. This creates short trajectories of length between 3 and 10. To evaluate the performance of the estimator, we sample trajectories (H4, Us) PAi.. under the target policy and treat every (Hi,,, ys) as labeled data point. We then evaluate normalized RMSE between fe (H...,,) and the true labels yr. For tumor growth, we evaluate a policy that increases the likelihood of treatments (i.e increases 33 and reduces y, the correlation to the observed volume. Error is also calculated with normalized RMSE.
The tables in fig. and fig. 4 present results. They show that for time-to-faillre, EDQ solves the estimation problem both when Aobs Autt (blue rows, no intervention performed), and when Apbs F Aint (red rows). This is evident by comparing ERM under the setting where Xahs Aint where ERM should be nearly optimal. ERM takes significant performance drop when λohs * Xint. as expected. For FQE, while in the first set of experiments (fig. 3), discretization should not result in significant information loss, it does create more difficult optimization problem. This is because the updates to (H+)) need to propagate backwards and most updates get noisy gradient signals by fitting to the Q value of a trajectory sampled one time step ahead, Q(H(+11) This challenge for FQE is most evident in Figure 3, where Amnt 0.5 and FQE incurs a significant loss both when Aobs Aint and when λahs * Xint The results in Figure 4 (right) demonstrate the potential effects of information loss due to discretization. Here, since trajectories are short, the optimization problem of losses propagating is likely less pronounced. However, we see that there is still a significant drop when Aint = and Aobe = 0.2 (approximating a high rate of treatment when data was sampled under low rates). Taken together, these two experiments demonstrate two possible drawbacks of discretization. For tumor-growth, EDQ still outperforms the alternatives but suffers a decrease in performance due to distribution shift between observational and interventional distributions.
Results.
6 LIMITATIONS AND FUTURE WORK
We have developed a method for off-policy evaluation with irregular treatment and observation times, which facilitates interventions on treatment intensities. We connected the setting identifiability results from causal inference to highlight the conditions under which the estimates are meaningful, and proved the estimator's correctness. EDQ is a "direct" method based on regression and, as demonstrated in experiments, is readily applicable to high-capacity sequence modeling architectures. To the best of our knowledge, is the first available solution to this estimation problem that applied with such architectures. Several limitations motivate exciting future research. Empirically, we plan to apply the method to large real-world datasets and additional simulators (Namkoong et al., 2020; Oberst and Sontag, 2019). The method does not handle censoring, which is required for reliable application in survival analysis and real trial data. Further developements include policy optimization in the setting we studied here and deriving bounds errors due to unobserved confounding.
'this up to numerical optimization issues, as we see FQB can outperform it in certain cases
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We thank Stefan Groha for helpful discussion in early stages of the project. This work was supported by National Science Foundation Award 1922658, NIH/NHLBI Award ROIHL148248, NSF CAREER Award 2145542, NSF Award 2404476, ONR N00014-23-1-2666 IITP with grant from ROK-MSIT in connection with the Global AI Frontier Lab International Collaborative Research, Apple, and Optum.
REFERENCES
Aalen, O., Borgan, O., and Gjessing. H. (2008). Survival and event history analysis: a process point of view. Springer Science & Business Media.
Aalen, O. O. (1987). Dynamic modelling and causality. Scandinavian Actuarial Journal, (3-7 4):177-190.
Andersen, P. K., Borgan, O., Gill, R. D., and Keiding, N. (2012). Statistical models based on counting processes. Springer Science & Business Media.
Bica, I., Alaa, A. M.. Jordon, J., and van der Schaar, M. (2020), Estimating counterfactual treatment outcomes over time through adversarially balanced representations. In International Conference on Learning Representations.
Chakraborty, B. and Moodie, E. E. (2013). Statistical methods for dynamic treatment regimes. Springer-Verlag doi, 10(97--1)44-1
Chakraborty, B. and Murphy, S. A. (2014). Dynamic treatment regimes. Annual review of statistics and its application, 1:447 464.
Chen, I. Y., Joshi, S., Ghassemi, M., and Ranganath, R. (2021a). Probabilistic machine learning for healthcare. Annual review of biomedical data science, 4(1):393-411
Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. (2021b). Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:1 14:15084-15097.
De Asis, K., Hernandez- Hernandez-Garcia, J., Holland, G., and Sutton, R. (2018). Multi-step reinforcement
learning: A unifying algorithm. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32.
Didelez, V. (2008). Graphical models for marked point processes based on local independence. Journal of the Royal Statistical Society Series B: Statistical Methodology, 70(1):245-264,
Felker, G. M., Lee, K. L., Bull, D. A., Redfield, M. M., Stevenson, I.. W., Goldsmith, S. R., LeWinter, M. M., Deswal, A., Rouleau, J. L., Ofili, E. O., et al. (2011). Diuretic strategies in patients with acute decompensated heart failure. New England Journal of Medicine, 364(9):797-805.
Fu, J., Norouzi, M., Nachum, O., Tucker, G., Wang. Z., Novikov, A., Yang. M., Zhang, M. R., Chen, Y., Kumar, A., et al. (2021). Benchmarks for deep off policy evaluation. arXiv preprins arXiv:2103.16596
Geng, C., Paganetti, H., and Grassberger, C. (2017). Prediction of treatment response for combined chemo-and radiation therapy for non-small cell lung cancer patients using a bio-mathematical model. Scientific reports, 7(1):13542.
Hallak, A., Tamar, A., Munos, R., and Mannor, S. (2016). Generalized emphatic temporal difference learning: Bias-variance analysis. Proceedings of the AAAI Conference on Artificial Intelligence, volume 30.
Harutyunyan, A., Bellemare, M. G., Stepleton, T., and Munos, R. (2016). Q.0 with off-policy correc- tions. In International Conference on Algorithmic Learning Theory, pages 305-320. Springer.
Hernan, M. and Robins, J. (2023). Causal Inference: What If. Chapman & Hall/CRC Monographs on Statistics & Applied Probab. CRC Press.
11

---

Published as a conference paper at ICLR 2025
Jia, Y. and Zhou, X. Y. (2023). q-learning in continuous time. Journal of Machine Learning Research, 24(161):1-61.
Johansson, F., Shalit, U., and Sontag, D. (2016). Learning representations for counterfactual inference. In International conference on muchine learning, pages 3020-3029. PMLR.
Joshi, S., Urteaga, I., van Amsterdam, W. A., Hripesak, G., Elias, P., Recht, B., Elhadad, N., Fackler, J., Sendak, M. P., Wiens, J., et al. (2025). Ai as an intervention: improving clinical outcomes relies on causal approach to ai development and validation. Journal of the American Medical Informatics Association, page ocae301.
Le, H., Voloshin, C., and Yue, Y. (2019). Batch policy learning under constraints, In International Conference on Machine Learning, pages 3703-3712. PMLR.
Lewis, W. and Shedler, G. S. (1979). Simulation of nonhomogeneous poisson processes by thinning. Naval research logistics quarterly, 26(3):403-413.
Li, R., Hu, S., Lu, M., Utsumi, Y., Chakraborty, Sow, D. M., Madan, P., Li, J., Ghalwash, M., Shahn, Z., and Lehman, L.-w. (2021). G-net: recurrent network approach to g- g-computation for counterfactual prediction under a dynamic treatment regime. In Roy, S., Pfohl, S., Rocheteau, E., Tadesse, G. A., Oala, L., Falck, F., Zhou, Y., Shen, L., Zamzmi, G., Mugambi, P., Zirikly, A., McDermott, M. B. A., and Alsentzer, E., editors, Proceedings of Machine Learning for Health, volume 158 of Proceedings of Machine Learning Research, pages 282-299. PMLR.
Lim, (2018). Forecasting treatment responses over time using recurrent marginal structural networks. Advances in neural information processing systems, 31.
Lin, H., Scharfstein, D. O., and Rosenheck, R. A. (2004). Analysis of longitudinal data with irregular, outcome-dependent follow-up. Journal of the Royal Statistical Society Series B: Statistical Methodology, 66(3):791-813.
Lok, J. J. (2008). Statistical modeling of causal effects in continuous time, The Annals of Statistics, pages 1464-1507.
Malenica, and Murphy, S. (2023). Causality in goal conditioned rl: Return to no future? NeurlPS 2023 Workshop on Coal-Conditioned Reinforcement Learning.
McDermott, M., Nestor, B., Argaw, P., and Kohane, (2023). Event stream gpt: A data pre- neeprocessing and modeling library for generative, pre-trained transformers over continuous-ting sequences of complex events. arXiv preprint arXiv:2306.1 1547.
Melnychuk, V., Frauen, D., and Feuerriegel, S. (2022). Causal transformer for estimating coun- terfactual outcomes. In International Conference Machine Learning, pages PMLR.
Munos, R., Stepleton, T., Harutyunyan, A., and Bellemare, M. (2016). Safe and efficient off-policy reinforcement learning. Advances in neural information processing systems, 29.
Murphy, S. A. (2005). A generalization error for q-learning. Journal of Machine Learning Research.
Nagpal, C., Jeanselme, V., and Dubrawski, A. (2021). Deep parametric time-to-event regression with lime-varying covariates. Survival Prediction-Algoriihin Challenges and Applications, pages PMLR.
Namkoong, H., Keramati, R., Yadlowsky, S., and Brunskill, F. (2020). Off-policy policy evaluation for sequential decisions under unobserved confounding. Advances in Neural Information Processing Systems, 33:18819-18831
Nie. X., Brunskill, E., and Wager, S. (2021). Learning when-to-treat policies. Journal of the American Statistical Association, 1166533):322-409.
Oberst, M. and Sontag, D. (2019). Counterfactual off oolicy evaluation with gumbel-man max structural causal models. In International Conference on Machine Learning, pages 4881-4890. PMLR.
12

---

Published as a conference paper at ICLR 2025
Ogata, Y. (1981). On lewis' simulation method for point processes. IEEE transactions information theory, 27(1):
Paine, L., Paduraru, C., Michi, A., Gulcehre, C., Zolna, K., Novikov, A., Wang, Z., and de Fre- itas, N. (2020). Hyperparameter selection for offline reinforcement learning. arXiv preprint arXiv:2007.09055.
Precup, D., Sutton, R. S., and Singh, S. P. (2000). Eligibility traces for off-policy policy evaluation. In Proceedings of the Seventeenth International Conference on Machine Learning, pages 759-766.
Qu, C.. Tan, X., Xue, S., Shi, X., Zhang. J., and Mei, H. (2023). Bellman meets hawkes: Model-based reinforcement learning via temporal point processes. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pages 9543-9551.
Robins, J. (1986). A new approach to causal inference in mortality studies with a sustained exposure period-ape application to control of the healthy worker survivor effect. Mathematical modelling. 709-12:1393311522
Røysland, K. (2011). A martingale approach to continuous- time marginal structural models. Bernoulli, 17(3):895 -915.
Røysland, K. (2012). Counterfactual analyses with graphical models based on local independence. The Annals of Statistics, 40(4):2162 2194.
Røysland, K., Ryalen, P., Nygaard, M., and Didelez, V. (2022). Graphical criteria for the identification of marginal causal effects in continuous-time survival and event-history analyses. arXiv preprint arXiv:2202.023. 11
Rytgaard, H. C., Gerds, T. A., and van der Laan, M. J. (2022). Continuous time targeted minimum loss-based estimation of intervention-speciff mean outcomes. The Annals Statistics, 50(5):2469- 2491.
Schulam, P. and Saria, S. (2017). Reliable decision support using counterfactual models. Advances in neural information processing systems, 30.
Schweder, T. (1970). Composable markov processes. Journal of applied probability, 7(2):400-410.
Seedat, N., Imrie, F., Bellot, A., Qian, Z.,, and van der Schaar, M. (2022). Contimuous-time mod- eling of counterfactual outcomes using neural controlled differential equations. arXiv preprini 07Xiv:2206.08311.
Snyder, D. L. and Miller, M. I. (2012). Random point processes in time and space. Springer Science & Business Media.
Tennenholtz, G., Shalit, U., and Mannor, S. (2020). Off-policy evaluation in partially observable environments. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 10276-10283.
Uehara, M., Shi, C., and Kallus, N. (2022). A review of off-policy evaluation in reinforcement learning. arXiv preprint arXiv:2212.06:3
Upadhyay, U., De. A., and Gomez Rodriguez, M. (2018). Deep reinforcement lcarning of marked temporal point processes, Advances in neural information processing systems, 31.
Amsterdam, W. A., de Jong, P. A., Verhoeff, J., Leiner, T., and Ranganath, R. (2024). From algorithms to action: improving patient care requires causality. BMC Medical Informatics and Decision Making, 24(1):111.
Van Hasselt, H., Guez, A., and Silver, D. (2016). Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30.
Vanderschueren, T., Curth, A., Verbeke, W., and van der Schaar, M. (2023). Accounting for informative sampling when learning to forecast treatment outcomes over time. arXiv preprint wrXiv:2306.04255.
13

---

Published as a conference paper at ICLR 2025
Voloshin, C., Le, H. M., Jiang, N., and Yue, Y. (2021). Empirical study of off-policy policy evaluation for reinforcement learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1).
Watkins, C. J. and Dayan, (1992). Q-learning. Machine learning, 8:279-292.
Xiong, H., Wu, F,, Deng, L.,, Su, M., and Lehman, L.-w. H. (2024). G-transformer: Counter- factual outcome prediction under dynamic and time-varying treatment regimes. arXiv preprint arXiv:2406.055 0554..
Zhang, M., Joffe, M. M., and Small, D. S. (2011). Causal inference for continuous- processes when covariates are observed only at discrete times. Annals of statistics, 39(1).
Zhang, Z., Mei, H., and Xu, Y. (2023). Continuous-time decision transformer for healthcare applications. In International Conference on Artificial Intelligence and Statistics, pages 6245- 6262. PMLR.
14


Reviews:
[{"review_id": "GsHLJtTMOx", "reviewer": "Reviewer_vFET", "summary": "The authors propose a method for estimating the effect of a continuous-time treatment policy, defined as a treatment intensity function (when to treat) coupled with a mark distribution (what treatment to select), from observational data. Their approach, Earliest Disagreement Q-Evaluation (EDQ), is based on the principle that effects of observed versus proposed policies should not diverge until the point when the two policies disagree. The authors first review the identifiability of causal effects in this setting, then present the proposed estimator and associated algorithm. They construct a simulation setting in which a supposed vital sign is responsive to treatment, with diminishing returns. They show that EDQ estimates the effect of alternative (non-observed) treatment policies more accurately than comparator methods in this setting.", "questions": "- Line 89: Why do these assumptions imply that the process is self-exciting? Please elaborate.\n- How well does the algorithm scale? Could the authors comment on the computational complexity?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The authors tackle a novel problem and present an elegant solution. Identifiability of causal effects follows from previous results, and is presented clearly. The related work section is outstanding: comprehensive, highly informative, and well-written. The simulation setup is intuitive and clearly presented, and the proposed method outperforms alternatives.", "weaknesses": "- The key weakness is that the experimental results are underdeveloped. I would have liked to see more variations on the simulation results as well as application to at least one real dataset.\n\n- I was confused by the notation and presentation in Definition 5, which then made it difficult to understand the details of EDQ. Specifically, I am not clear on how $\\tilde{P}_t^a$ is defined or how to sample from it. I have lowered my score for this reason, but I'm happy to increase it if the authors clarify the presentation and/or if other reviewers did not have similar difficulty.\n\n- While defining the policy as an intensity (i.e. rate) is interesting, I have a hard time imagining a realistic scenario where it would make sense to sample treatment times rather than deciding whether/how to treat at fixed or given intervals.\n\n- (minor) The title (\"When and What\") is a bit misleading. I take the point that learning when to treat (i.e., the rate portion of the policy) is the challenging part, but saying both when and what in the title is misleading, since the paper and experimental results focus only on the former.\n\n- (minor) There are quite a few typos / minor errors, particularly in later sections of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "itmuzZuQf4", "reviewer": "Reviewer_jmvH", "summary": "The paper aims to tackle the temporal irregularity when estimating causal effects of policies, by proposing a method called EDQ (earliest disagreement Q-evaluation) based on dynamic programming. The simulation demonstrates more accurate estimations of EDQ.", "questions": "Please see Weaknesses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The work investigated an important problem, i.e., when to intervene with irregular times.\n\n- The work is well motivated, given many human-related backgrounds such as healthcare and finance requires a carefully evaluation for when to provide proper intervenes.\n\n- The paper is well structured and polished.", "weaknesses": "- Given the work is called into category of off-policy evaluation. Related works in off-policy evaluation should be thoroughly discussed in the paper, either in Introduction or Related work. I understand there can exist some key differences between traditional off-policy evaluation and EDQ, but should be carefully and thoroughly discussed and compared in experimental settings. Also, there exists work regarding when-to-treat problem (e.g., [1]). A further comparison and discussion regarding EDQ and those works would be great. \n\n- Since one major motivation of the work is human-related when-to-treat problem, and the paper used a lot of healthcare examples (which is comprehensive). I’m curious about whether the work can be examined on some related settings. It’s understandable that running real-world experiments would not be feasible and high-stake, but it would be more impressive to provide experiments on some empirical motivated settings, e.g., sepsis [2], autism [3], etc.\n\n\n\n\n- Minors:\n\nIt would be great if code and/or data can be released for reproducibility.\n\nLine 99. a was defined as action and then represented number of multivariate unobserved process\n\nLine 185. FQE needs careful citations, given it’s a well-established work in off-policy evaluation and optimization.\n\nLine 308, 355, 363. Lower cases for bolded words?\n\n[1] Learning When-to-Treat Policies\n\n[2] Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Models\n\n[3] Off-policy Policy Evaluation For Sequential Decisions\nUnder Unobserved Confounding\n\n**Update after rebuttal**:\n\nThe authors conducted further healthcare analysis to support their major contribution, and further discussions regarding related works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Lo1EjOXTMg", "reviewer": "Reviewer_5s6h", "summary": "The paper proposes a method for estimating the causal effect of policies using Earliest Disagreement Q-Evaluation (EDQ), where a policy determines both the action and its timing. Specifically, the authors aim to estimate the outcome of a target policy by leveraging dynamic programming, extending fitted Q-evaluation from discrete to continuous time. The update function is driven by the earliest point of disagreement between the behavior policy and the target policy.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper considers the problem of evaluating a policy using continuous-time causal inference, focusing on off-policy evaluation.\n- To tackle the challenges of policy evaluation in continuous time, where the conventional Q-function collapses, the paper leverages the property of countable decision points in a point process and introduces a simple yet efficient method based on earliest disagreement times.\n- The paper also provides identifiability conditions for the causal estimands, ensuring accurate estimation.", "weaknesses": "- Could the authors discuss the connection between the Bellman equation in discrete time and Theorem 1? Equation (1) is equivalent to the Bellman equation for a finite-horizon problem in discrete times, and Theorem 1 appears to be a direct extension using differential equations and stopping times.\n- The proposed framework allows the trajectory outcome to be a sum of time-specific outcomes (lines 79, 204-205). Are there any constraints on the number of observations in the trajectory? If the total number of $k$ is indefinite, the scale of $Y$ will vary with the number of observed time-specific outcomes, potentially affecting the outcome’s interpretation and stability.\n- Writing improvement suggestions\n  - In the second paragraph of Section 1, the motivation could be clearer by explicitly stating the challenges related to sequential treatment and irregular timing, with a particular emphasis on the primary challenge about decision times. In the current introduction, it is not immediately clear to me which difficulties are being discussed in lines 37-39.\n  - In the third paragraph of Section 1, it would help to introduce the components of an intervention. From my understanding, there are two key components: the timing of the intervention and the intervention itself. Otherwise, readers might not yet understand whether treatment timing in line 49 is determined by the environment or chosen by the policy.\n  - In line 79, $Y_k$ has not been defined yet.\n  - At the end of line 88, should $d N(t)$ be $d N^l (t)$?\n  - Could you clarify the distinction between a marked decision point process in Definition 1 and a marked temporal point process (Upadhyay et al. 2018)?\n  - A period is missing from line 103.\n  - In line 105, should each trajectory be indexed by $i$ or $m_i$? It seems that $m$ is the total number of trajectories and $i$ takes values from $[m]$.\n  - What are assumptions 1-3 in line 251?\n  - In equation (2), should the right-hand side condition on $\\mathcal{H}_t$ as in equation (1)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f6mx4DUETY", "reviewer": "Reviewer_u2Uo", "summary": "This paper introduces off-policy evaluation through the framework of stochastic point processes and presents Earliest Disagreement Q-evaluation (EDQ) as a method for estimation. Under the causal validity assumption, the authors demonstrate that EDQ can accurately identify the true policy value. Additionally, this assumption is elucidated using a local independence graph. Experiments conducted on simulated data confirm the effectiveness of the proposed approach.", "questions": "**1**. To my knowledge, consistency and the SUTVA assumptions are also essential in causal effect estimation. Why are these assumptions not required in your work?\n\n**2**. What is the impact of different sequential modeling architectures (such as Transformers, RNNs, regression models, etc.) on your method? Is the estimation sensitive to the choice of architecture and hyperparameters?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "**1**. The formulation of sequential causal effect estimation using point processes is novel, effectively addressing the irregular nature of treatments and covariates in various real-world applications.\n\n**2**. The EDQ method for estimation is compelling, with corresponding consistency results rigorously demonstrated in Theorem 1.\n\n**3**. The assumptions required for causal identification are elucidated through a local independence graph, providing an intuitive understanding of their implications.", "weaknesses": "Experiments should be conducted on real-world datasets or, at the very least, on simulated datasets generated from real-world data to provide more convincing validation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lYAUPHrrtU", "reviewer": "Reviewer_WaHo", "summary": "This paper presents a novel approach for handling irregular timing in off-policy evaluation within sequential decision-making frameworks. It extends the classical Bellman equation in fitted Q-evaluation (FQE) to account for cases with irregular time intervals and introduces the concept of measuring the \"earliest disagreement time\" in Q-function updates.\n\nOverall, the paper is well-written, with a natural flow that guides the reader from foundational ideas to specific methods. While reading the introduction, I was somehow expecting a real-data analysis that could demonstrate the approach, perhaps with an example like ASCVD control or another scenario where OPE might reveal insights under varied intervention timings. I understand the challenges of conducting OPE with observational data, so this is not a request for inclusion in the rebuttal. However, such an analysis could add an interesting dimension to showcase differences in performance in real-world applications and offer potential interpretations. Some terms and assumptions could benefit from clearer explanations, but overall, this paper provides a practical and effective method for handling irregular time in OPE without imposing heavy modeling or causal assumptions.", "questions": "1. How does the computational time of your algorithm compare with classical FQE? \n\n2. In cases where stages are regularly spaced in time, would EDQ reduce to classical FQE? A brief discussion or proof in the paper about how EDQ behaves in the limit as time intervals become regular would suffice.\n\n3. In practical applications, a common workaround for irregular timing is to include the interval between the $k-1$th and the $k$th stages as a component of the state $\\boldsymbol{X}$, then proceed with classical FQE for Q-function estimation. While this may lack methodological novelty, industry practitioners often find it to be a straightforward and effective way to account for \"irregular time\" in value estimation. Could the authors elaborate on the motivation for considering \"earliest disagreement times\" and explain situations where this approach might perform better?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper’s motivation is well-illustrated with examples. I appreciate the effort to clarify the setup in Section 2-3, which makes the new approach easier to understand. The comparison with classical FQE and the challenges outlined at the end of Section 3 are also helpful.\n\n2. The simulation studies demonstrate promising results, effectively showcasing the benefits of the proposed EDQ by comparing it with FQE and MC methods.", "weaknesses": "1. Some notations are unclear on first mention. Here are a few areas where clarification could improve readability (please correct me if I missed something):\n\n (a) In Definition 4, Line 170, does $\\ll$ in $P\\ll P_{obs}$ denote absolute continuity? It might be helpful to define this symbol for readers who may not be familiar with the concept.\n \n (b) In Theorem 1, Line 251, assumptions 1-3 don’t appear to be clearly defined or referenced in Section 2.2.\n\n (c) In Algorithm 2, Line 279, the loss function $l(Q_t,y)$ seems to be undefined. While it may be user-specified and flexible, adding a brief note about its role and possible choices could provide helpful context.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tn2j0ytVOm", "reviewer": "Reviewer_jyT3", "summary": "This paper proposes Earliest Disagreement Q-Evaluation (EDQ), a method for estimating causal effects of sequential treatments with irregular timing. The key contribution is adapting Q-learning approaches to continuous-time settings while maintaining model-free properties. The work focuses on healthcare and financial applications where treatment timing is crucial.", "questions": "1. Definition 5, which is the most important definition, is not clear. For instance, what does $\\tilde P_t^a$ mean? I cannot find its definition throughout the whole article including appendix.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper addresses a crucial and previously underexplored problem in causal inference: estimating the effect of interventions on treatment timing in sequential decision-making. This problem has practical implications in healthcare and finance, where treatment timing often has substantial impact on outcomes. The authors provide compelling motivation and clearly articulate why existing methods are insufficient.\n\n2. The proposed EDQ method demonstrates impressive scalability and flexibility by being compatible with modern deep learning architectures like transformers. This is a significant advancement over existing methods that either require time discretization or don't scale to large models, making the approach more practical for real-world applications.", "weaknesses": "1. The paper's title promises insights into \"When and What to Do,\" but primarily delivers a method for evaluating pre-specified timing policies. While the proposed EDQ method effectively handles off-policy evaluation of treatment timing effects, it provides no framework for discovering optimal timing strategies.  The experimental section further highlights this gap, focusing solely on estimation accuracy rather than demonstrating practical utility in finding better treatment schedules. Additionally, despite the title's suggestion, the paper explicitly omits treatment selection (the \"what\" aspect) to focus on timing, making the scope narrower than advertised.\n\n2. The experimental validation relies on overly simplified simulation settings, particularly in modeling treatment delays with an exponential distribution  $\\delta\\sim exp(\\lambda_a)$. Although the method's theoretical guarantees don't depend on specific distributional assumptions, the paper would be stronger with experiments using more realistic delay distributions and multiple vital signs. This limitation raises questions about the method's practical performance in real-world healthcare settings where treatment timing follows more complex patterns.\n\n3. The technical presentation has several gaps that affect its accessibility and reproducibility. The paper lacks analysis of computational complexity and clear guidelines for practical implementation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)