Model-agnostic meta-learners for estimating heterogeneous treatment effects over time
Authors: Dennis Frauen
Konstantin Hess
Stefan Feuerriegel
Venue: ICLR 2025 Poster (2025)
Keywords: causal inference
treatment effect estimation
CATE
doubly robust learning
Primary area: causal reasoning
TL;DR: We propose meta-learners for estimating HTEs over time, provide a theoretical analysis, and derive inverse-variance weights to obtain a stable and doubly robust learner.

Abstract: Estimating heterogeneous treatment effects (HTEs) over time is crucial in many disciplines such as personalized medicine. Existing works for this task have mostly focused on *model-based* learners that adapt specific machine-learning models and adjustment mechanisms. In contrast, model-agnostic learners - so-called *meta-learners* - are largely unexplored. In our paper, we propose several meta-learners that are model-agnostic and thus can be used in combination with arbitrary machine learning models (e.g., transformers) to estimate HTEs over time. We then provide a comprehensive theoretical analysis that characterizes the different learners and that allows us to offer insights into when specific learners are preferable. Furthermore, we propose a novel IVW-DR-learner that (i) uses a doubly robust (DR) and orthogonal loss; and (ii) leverages inverse-variance weights (IVWs) that we derive to stabilize the DR-loss over time. Our IVWs downweight extreme trajectories due to *products* of inverse-propensities in the DR-loss, resulting in a lower estimation variance. Our IVW-DR-learner achieves superior performance in our experiments, particularly in regimes with low overlap and long time horizons.

Published as a conference paper at ICLR 2025
MODEL-AGNOSTIC META- FOR ESTIMATING HETEROGENEOUS TREATMENT EFFECTS OVER TIME
Dennis Frauen,' Konstantin Hess & Stefan Feuerriegel LMU Munich
Munich Center of Machine Learning (MCML) (f (fraven, hess, feverriegel elmu..c. de
ABSTRACT
Estimating heterogeneous treatment effects (HTEs) over time is crucial in many disciplines such as personalized medicine. Existing works for this task have mostly focused on model-based learners that adapt specific machine-learning models and adjustment mechanisms. In contrast, model-agnostic learners so-called meta-learners are largely unexplored. In our paper, we propose several meta- learners that are model-agnostic and thus can be used in combination with arbitrary machine learning models (e.g., transformers) then provide a comprehensive theoretical analysis that characterizes the different learners and that allows us offer insights into when specific learners are preferable. Furthermore, we propose a novel WVW-DR-learner that (i) uses a doubly robust (DR) and orthogonal loss; and (ii) leverages envers-varriane weights (IVWs) that we derive to stabilize the DR-loss over time. Our IVWs downweight extreme trajectories due to products of dnverseprropensttie in the DR-loss, resulting in a lower estimation variance. Our IVW-DR-leame achieves superior performance in our experiments, particularly in regimes with low overlap and long time horizons.
estimate HTEs over time. We
INTRODUCTION
Estimating heterogeneous treatment effects (HTEs) from observational data over time is crucial when randomized experiments are costly or infeasible (Lim et al.) 2018 [ica et al. 2020a Melnychuk al. 2022). A typical example is personalized medicine (Feuerniegel et al. 2024). Here, medications are commonly prescribed over the course of several days, and it is nowadays standard that patient responses captured within electronic health records (Allam et al. 2021). Hence, clinicians can electronic health records to estimate HTES over time and thereby analyze the safety and effectiveness of treatments across different patient profiles (Hamburg & Collins 2010). For example, in cancer care, clinicians are commonly interested in which sequence of chemotherapy and immunotherapy has the largest expected benefit for patients with a specific age, gender, or medical history.
Estimating HTEs over time (also referred to as the time-varee varying setting) poses unique challenges due to the complex underlying fata-gener process and time-vat varying confounding structure. In particular, estimation is hard because of the following three challenges: (i) Time-varying confounding: the existence of post-treatment confounders biases naïve estimators and requires adjustment mechanisms tailored to the time-varying setting (Robins et al. 2000) van der Laan & Gruber 2012). (ii) Growing history: this means that one must adjust for entire patient histories growing over time (Bica et al.) 2020a). (ii) Low overlap: sequences of treatments are less likely observed in observational data from time-varying settings (as compared to single treatments in static settings), which thus increases the variance of estimators (Hess et al. 2024b). This problem increases with the length of the time-horizon.
Existing works for estimating HTEs over time are almost entirely model-based (see Table That is, these works propose specific model architectures and/or losses designed to address the challenges (i) from above, or at least some of them (Lim et al.) 2018 Bica et al. (2020a] MMenychuk et al.) 2022 Furthermore, these model-based learners build upon certain of adjustment mechanisms
(c.g., history adjustment, inverse propensity weighting). In contrast, other important adjustment
"Corresponding author

---

Published as a conference paper at ICLR 2025
mechanisms such doubly robust adjustment (which typically offers favorable theoretical properties) remain overlooked. This is surprising, since sadel-agnostic methods so-called meta-learned commonly used in the static setting (i.e., without time dimension) and implemented popular in software packages (Battocchi et al.] 2019). whereas a comprehensive toolbox of meta-learners for the time-varying setting is still missing.
In this paper, we take a model-agnostic view on estimating HTES over time. To this end, we propose several meta-learners for this task that can be used in combination with arbitrary machine-learning models and come with theoretical guarantees such as Peyman-orthogonaa Our meta--eerners address the challenges (i)- from above as follows: (i) they leverage adjustment strategies tailored to the time-series setting to ensure unbiased estimation, including regression adjustment (G-computation), propensity adjustment, and doubly robust adjustment; (ii) they can be instantiated using state-of-the-art models such as transformers to effectively learn representations from high-dimensional, time-varying data; and (iii) we show that certain meta-learners (e.g., the doubly robust learner) can suffer from high variance under violations of overlap due to division by propensity scores (or products thereof). As a remedy, we derive time-varying inverse-variance weights to stabilize the doubly robust loss, which yields the varaeeevvaiinwwweieeee doubly robust leamer (called WWWDR--leamee
and double robustness.
Our contributions are: (1) We propose several meta-learners for estimating HTEs over time by leveraging various adjustment mechanisms. They are model-agnostic; i.e., they can be instantiated using any machine learning model of choice such as transformers.
(2) We analyze the meta-learners theoretically and provide insights for when specific learners are preferable over others. We also structure the literature by showing that some existing model-based learners are instantiations of certain meta-learners, while model-based learners corresponding to other meta-learners have not been proposed yet (see Tables and [5). (3) We derive inverse variance weights to stabilize the loss of the DR-learner, yielding novel IV IVW-DR-lcane We also perform various experiments to confirm the effectiveness of our IVW-DR-learner as well as the validity of our theoretical results.
2 RELATED WORK
Related works on estimating HTEs can be grouped along two dimensions (see Table (a) whether method is model-based or modelaagnostic, and (b) whether a method is for the static or time-varying setting.
Learners in the static set- ting: Both (i) model-based and (ii) model-agnostic meta-learners have been proposed in the static setting, but without consid- ering the time dimension. (i) Prominent model-based learners include the causal forest (ii) Meta-learners in the static setting leverage different adjustment mechanisms. Examples include
Table I: Key learners for heterogeneous treatment effects.
Time-varying wetling
CFR Decunsenn 2016
RMSNs [im 2018)
Model-based
Casttl foter viney 2015
Stanneree
CRN [B paleryy
CT
Mess 2024a]
Planins RA, IPW Carth van der Schure 2011
Roizener
Lewis & Sarglanss 2021
Macklagravii
DR Kemsph 2003 2021
PL-HA- incs,,it,,,
R.lemne
DR. IVW-DR-lemer
(Wager & Athey 2018) and TARNet (Shalit al. 2017
the RA- or X-learner (Künzel et al.
2019; Curth & van der Schaar 2021I. the IPW-learner (Curth
& van der Schaar| 2021). the DR-
learner (Kennedy 2023), the R-learner (Nie & Wager 2021).
the EP-learner (van der Laan al.)
2024). and the recently proposed awvess-vvrriance-wwiiree
DR-leaner (lea-eRrlearmer) (Fisher 2024). Note that these learners focus on the static setting (and
not the time-narying setting). As a result, these are inherently biased for estimating HTEs over time because runtime confounding renders static adjustment mechanisms insufficient. Meta-learners are often based on semiparametric theory and come with theoretical guarantees (Chernozhukov et al.
2018:
Foster & Syrgkanis 2023). Despite being well well-established in the static setting (Acharki et al.
2023
Curth & van der Schaar [021] Kennedy| 2023 Nie & Wager 2021). similar learners for the
time-series setting are missing.
Model-based learners for HTEs in the time-varying setting: Several model-based learners have been proposed for estimating HTEs over time. These learners leverage (i) specific model architec-
Code is available at wttp:///thiiioom///mmm///aaaaaaa
2

---

Published as a conference paper at ICLR 2025
tures (e.g., recurrent neural networks or transformers with balancing representations (Bica et al. 2020al Hess et al.] 2024a; Meinychuk et al., 2022)) or (ii) specific adjustment mechanisms (e.g.. G-computation (Li et al. 2021) or inverse propensity weighting (Lim et al. 2018)). However, these learners are not model agnostic. To the best of our knowledge, none of these learners uses a doubly robust adjustment mechanism. Further, theoretical analysis comparing the different learners is missing.
Meta-learners for HTES in the time-varying setting: We are only aware of a single meta-learner for the time-varying setting: the R-learner proposed in (Lewis & Syrgkanis 202T). While it is model- agnostic, it imposes parametric assumptions on the data-generaing process (e.g., linear Markov assumptions). In contrast, our meta-learners are completely nonparametric. We provide detailed comparison of our meta-learners with the R-learner in Table 5 and Appendix [
Meta-learners in reinforcement learning (RL): The time-varying setting for estimating HTEEs can be viewed as model-free off off-policy learning problem RL, where we are interested in estimating advantage functions (A-learning (Murphy) 2003)) in non-Markovian decision process (NMDP) (Uehara et al. 2022). Recently, meta-learners have been proposed to estimate the average reward of policy in NMDPs (Kallus & Uehara 2019 2020 2022). However, these works differ from ours in two ways: (i) they do consider a different estimand and are not directly applicable for advantage functions/ HTEs; and (ii) to the best of our knowledge, no previous work in RL has leveraged inverse variance weights to deal with the widespread problem of low overlap.
3 PROBLEM SETUP
Data-generating process: We build upon the standard setting for estimating HTEs from ob- servational data over time (Bica et al. 2020a| Melnyccuk et al., 2022). That is, con- sider covariates X, E X, discrete treatments A, E A. and continuous outcomes Y, E R that are observed over several time periods t. In medical settings, X, may denote (time- varying) patient characteristics such as heart rate and blood pressure, Ar may denote whether drug is administered, and Y may denote health outcome such as the blood sugar level.
For each time period t, the treat- ment At is assigned based on the history H, where X₁ = 1XI..... X2). Y,-1 = A (Ai. 11-1). The outcome Y, depends on both the history H, and the assigned treatment A.. Furthermore, the time-varying covariates X, (e.g., blood pressure) and treatments As may depend on arbitrary past variables. A possible
(X, A, -
1,1)), and
A
A
A642
H4
Figure 1: Setting for estimating treatment effects over time.
causal graph is shown in Fig. [ Note that w.l.o.g. we do not consider separate static covariates (e.g., age, gender) as these can be part of X,. We are further given an observational dataset (t) (())
D = {(x))) ot Vr. Jt=l Ji=1 consisting of n E N patient trajectories sampled i.i.d. from an unknown probability distribution P of potentially different lengths 7()
Causal estimands: We build upon the potential outcomes framework (Rubin 1974) for time-varying settings (Robins et al.) 2000 Robins & Hernán 2009) to formally define HTES. For time consider a fixed sequence of treatment interventions 0p.++ = (oty...,
ar+r) . of length T + 1. We the r-step-ahead outcome that would be observed when
denote the potential outcome (g+r(au2+7)
intervening on the treatments Agsto and setting them to aeatr. We are then interested in estimating the conditional average potential outcome (CAPO)
Pa = E (Ve+r((re+t) H, he]
and the conditional average treatment effect (CATE)
Ta (T.) T.. (Ye+((arat7) Yotr (Teset7)
H, Tu]]
(2)
3

---

Published as a conference paper at ICLR 2025
where bus++ denotes another sequence of treatment interventions. Both CAPO (and CATE) functions that predict (differences in) the r-step-ahead potential outcomes based on the history at time period t.
Identifiability. We impose the following assumptions to ensure the identifiability of both CAPO and CATE from observational data (for proof refer to Appendix C These assumptions are standard in the causal inference literature (Bica et al.] 2020a Lim et al. 2018 Melnychuk et al. 2022)
Practically, they exclude spillover effects, deterministic treatment assignments, and unobserved confounders (Frauen et al. 2023b).
Assumption (Identifiability (Robins et al. 2000)). We assume that the following conditions hold for all a and he: (i) consistency: f++-(au:::: Yi+r whenever A Cta++; ; (ii) overlap: P(A, = dt He lte) > 0 whenever IHH = hx)) > 0; and (iii) sequential ignorability: A, L Y2+8(ae+0) H. he for all ó E (l,...,++.
Time-varying confounding and adjustments. A key challenge of the time-varying setting is that the unbiased estimation of CAPO and CATE is non-trivee non-trivial. Under Assumption I both CAPO and CATE are identified and can thus be expressed as function of the observational data distribution (Robins 1999 Robins et al. 2000). However, the time-varying nature of the confounding structure
gives rise to adjustment mechanisms that different from the static setting whenever 0. In particular, adjusting for only the history H, is generally insufficient for unbiased estimation
(van der
Laan & Gruber, 2012).
That is, for T > 0, it holds that
E Yo+r (ān.-r)
| He he] # E [Yi+z
Arc-r .c.t+r]
Importantly, Eq. (3) implies that static meta-learners that only condition on the history H, are biased in the time-varying setting. The reason is that the history adjustment in Eq. (3) ignores to adjust for the post-treatment confounders Xe for > t. which are not observed during inference time. This phenomenon is also known as time-varying confounding" Adjusting for post-treatment confounders requires leveraging custom adjustment formulas for the time-varying settings. This includes iterated conditional expectations (G-computation) (van der Laan & Gruber 2012) and products of propensity scores (inverse- (inverse-poppensity weighting) (Robins et al. 2000).. In the next section, we leverage such Line-varying adjustment formulas to construct unbiased meta-learners for estimating both CAPO and CATE.
4 META-LEARNERS FOR ESTIMATING HTES OVER TIME
In this section, we propose several meta-learners for estimating CAPO and CATE. The gen- eral idea of meta-learners is to first estimate so-called nuisance functions that capture impor- tant aspects of the observational data distribution (e.g., response functions or propensity scores).
Then, the estimated nuisance functions are used for estimating the quantity of interest (e.g.. CAPO or CATE). Both nuisance functions and meta-learners are defined in terms of conditional
+
expectations and probabilities that can esti- mated using arbitrary machine learning models.
Our meta- meta-learners can be grouped along two cri- teria: (a) which adjustment mechanism is lever- aged; and (b) whether the learner is a plug-in learner or two-stage learner. For (a), we lever- age different adjustment mechanisms, including history adjustment, regression adjustment (G- computation), propensity adjustment, and dou- bly robust adjustment. For (b), the difference between plug-in and two-stage learners is as follows: Plug-in learners "plug" the nuisance functions into the corresponding adjustment formula to estimate the quantity of interest. In contrast, two-stage learners estimate the treatment effect directly by performing second-stage regression pseudo-outcome constructed from the estimated nuisance
AAB
Chovectionl dois D
Nuleensn sctrestoon
Hate-kamar
Figure 2: Overview of the different nuisance esti-
mators and meta-learners proposed in this paper.
TThis is closely related to runtime confounding (ostonn et al. 2020). where confounders are unavailable during inference time.
4

---

Published as a conference paper at ICLR 2025
functions. Two-stage learners often outperform plug-in learners if the target quantity (e.g., CATE) is of simpler structure and thus easier to estimate than the nuisance parameters involved (Curth et al. 2020 Kennedy, 2023).
In the following, we group our learners according to (a) and propose both plug-in and two-stage learners for different adjustment mechanisms. We refer to Table 5 for detailed overview and comparison of our meta-leamnrs.
4.1 HISTORY ADJUSTMENT
PI-HA-learner. A naive approach to construct meta-learner is to use the history adjustment
(h+) = TE [Ye+r H, = h2 Aus+T = (es+r]. As a result, this learner controls only for the history
H2, while it thus ignores post-treatment confounders. Hence, for some arbitrary estimator 8a (Tt), learner (PI-HA-learner for CAPO and CATE as
define the plug-in history-adjustmee
ALLHA (ha) = (h,) and
2ā.6 "Pi-HA (h,) = 8 (he) - (hc).
(4)
Here, the estimator 8 (he) of the nuisance function 8 (he) ) is plugged into the formulas for CAPO and CATE, respectively. Note that both ÍpJ-HA (ht) and PP--A (Fe) are biased because the history adjustment is not sufficient for the time-varying setting (see Eq. (3) The reason why we nevertheless consider the PI-HA-learner is its low variance, which can lead to relatively good performance in
law-sample settings when estimation variance has larger impact than confounding bias.
The PI-HA-learner can also be interpreted as a plug-in learner from the static setting (i.e., ignoring time) (Curth et al. 2020), where the history H, acts as confounders and the intervention sequence Age+r acts as a discrete treatment. On this basis, one could, in principle, adopt other two-stage learners from the static setting that can handle discrete treatments (Acharki et al. (2023) (e.g.. doubly robust versions of history adjustment). Of note, such static two-stage learners suffer from the same bias as the PI-HA-learner they aim at estimating dd (h+) but not E [Y+++(une+)) H. he]
4.2 REGRESSION ADJUSTMENT (G- (G-COMPUTATION)
We now introduce an unbiased meta-learner for estimating CAPO and CATE based on regression adjustment, also known as G-comuutttion (Robins 1999). The nuisance functions used for this are the response functions, defined via
Ma+r (he+r) = [Ye+r H = hett, Actr = act7]
(5)
and iteratively via
14 2 (he) = E. (He+t) 1 Hl = he, Ar = ae]
(6)
for l E +....,,++111 1}.
PI-RA-learner. Given estimators of the response functions, we introduce the plug-in in regression- adjustment learner (PI-RA-learrer) for CAPO and CATE via
PPP-RA (f.) io. (Tx) and a..
TPi-RA (Tx) 10g (Tr.) 12 (he)
respectively. The plug-in learner is unbiased (see Sec. 5. This is because. under Assumption [
CAPO can be written as 7 (h,,) = ue (Tt) (van der Laan & Gruber 2012 Frauen et al. 2023a). In particular, AR PPIRRA (he) adjusts for the post-treatment covariates by successively integrating them out through an iterative regression process. This approach is also called iterative G-computation and has been leveraged for estimating average treatment effects (van der Laan & Gruber) 2012 Frauen et al. 2023a), while we extend the idea to estimating heterogenous treatment effects.
RA-learner. We now introduce the regression-addustmmnnt adjustment learner (RA-leamer), which is two-
stage learner for estimating CATE using regression adjustment. The motivation is that, in many applications, the CATE Ta. (he) is of simpler structure than the CAPO ra(h,) due to cancellation effects (Kennedy) 2023). Hence, two-stage learners that target CATE directly may outperform the plug-in learner in such scenarios. Given estimators jug (hz) of the response functions, we define the pseudo-outcomes
ÿ = 1 L(A, a.} (pi+1 (He+1) -A (H..) +11A, b.) ( (fi)) il44 (Bc.i))
+ 1(A, * at arl1AA # bet (pt (ho) p.. (T.))
(8)
5

---

Published as a conference paper at ICLR 2025
We define the RA-learner via the staon regression (RA(he) = [PRA He He= heg
4.3 PROPENSITY ADJUSTMENT
Another major type of adjustment is propensity adjustment (Curth et al. 2020). For this, let define propensity scores in the time-varying setting via
(a( he) - I(Ac - ar He he)
for l E it....,t-77 .J. Each propensity score Te(ae) he) characterizes the treatment assignment mechanism at time based on the history hi.
IPW-learner. We now propose the mmerrs--rropessitywweee two-stage learner for estimating CAPO and CATE based on propensity adjustment. Using estimators (al he) of the propensity scores, define the pscudo-outcomes
learners (IPW-learner), which is
H+
1(A = ac) Yo+r and #e(a; He)
YiPw = II
VIPW = Y Vipw YIPw-
(10)
t=1
Our IPW-learner is two-stage learner defined via the second-stage regressions
Miww = = ya H. = hr and FIw (Tt.
H. hr
(11)
deverse-propensee weighting in the time-varying setting has originally been proposed for estimating average treatment effects (e.g., via marginal structural models (Robins al. 2000)), however, not explicitly for heterogeneous treatment effects.
4.4 DOUBLY ROBUST ADJUSTMENT
The above learners (i.e., the PI-HA- PI-RA- RA-, and IPW-leaners) have a shortcoming in that they all utilize estimators of only some of the possible nuisance functions. While the PI-HA-, PI-RA- and RA-learners leverage only the response function estimators (or parts thereof), the IPW-leameer leverages only the propensity score estimators. This motivates us to construct meta-learners that leverage both nuisance estimators and thus all components informative of the data-generating process.
DR-learner. We propose the doubly robust learner (DR-learner), which is two-stage learner that can be used for estimating both CAPO and CATE, For this, we define the pseudo-outcomes
aa DR
Piww E (Hx) (1
1(Ax=ug) [a(
((4mme)) and He(az)ll))
y-u.b DR
P
(12)
DR
The pseudo-outcomer Y and d estimators of the (uncentered) efficient influence function of CAPO and CATE, respectively (van der Laan Gruber 2012). Similar to the static setting (Kennedy) 2023), introduce DR-learner for the time-varying setting via second-stage regressions based on the efficient influence functions. That is, we define
A (h+)
E
DR
H, hg and
a. DR (h,, = E
va. DR
H, = h1g
(13)
Due the of efficient influence functions, the DR-learner comes with theoretical properties such as double robustness (see Sec.
THEORETICAL ANALYSIS
In this section, we add theoretical analysis for the PI-HA- PI-RA-, RA-, IPW-, and DR-learners. For this, we employ mathematical tools from Kennedy (2023) and derive asymptotic bounds on the point-wise risk of the corresponding learners. We start by imposing the following assumptions.
Assumption (Boundedness). The response functions μu (-), the propensity scores म. (az +), and its estimated versions i (.) and #e(az) () bounded functions for all interventions

---

Published as a conference paper at ICLR 2025
Assumption 3 (Estimation). The regression estimators E,, admit rates rhe (n) for the response
functions Ag (.), of PPIHHA (n) for the history adjustments (-),, and (n)(n) for the second-stage estimators E (see Appendix D for a mathematical definition and examples). We also impose the stability assumptions from Kennedy (202)))02))))) (2023) all regression estimators E,, (see Appendix C. Finally, the propensity score estimators (ee :) admit rates > (n).
Assumption 4 (Sample splitting). For each learner, the observational dataset D admits a partition D = Df U U... U D U DPO, so that each response function estimator j is fitted on Djj all
propensity score estimators Re are fitted D \ DPO, and any second-stage regressor È is fitted on DPO
Boundedness assumptions such as Assumption 2 are standard in the causal inference literature to provide theoretical guarantees for treatment effect learners (Curth et al., 2020; Foster & Syrgkanis 2023 Frauen & Feuerriegel 2023] Kennedy (023). Assumption 3 on the estimators is in line with
Kennedy (2023). The rates The (n), " (n),, PI-HA (n), and TP (n) capture assumptions on the complexity of the corresponding estimands, i.e., nuisance parameters and CATE. The assumption can be instantiated via, e.g., enforcing smoothness or sparsity on the underlying model (Curth et al.. 2020)
Kennedy, (223). Finally, Assumption (i.e.. sample splitting) is standard in the literature on meta- learners for causal inference (Chernozhukov et al., 2018 Foster Syrgkanis 2023: Kennedy 2023).
While sample splitting is required for the validity of our theoretical results, we note that, in practice, all of our learners can also be used without sample splitting Avoiding the use of sample splitting can
result in superior saite-sample performance as more data is used to estimate nuisance parameters and second-stage regressions (Curth et al., 2020). In our experiments, we follow established literature (Curth & van der Schaar 2021 and refrain from using sample splitting.
The following result establishes asymptotic rates for the risk our meta-learners for CATE under treatment interventions à and b. Rates for CAPO can be obtained by removing all terms that include 5. We write g(n) & (n) whenever there exist C, ny > O, so that g(n) S Cf(mm) for all n > no-
Theorem 1. Under Assumptions
4 we obtain the following rates:
B[(+PiLH (T+)) Ta.E Th.6((.)) 2 p (PLHA( (n) (1)
(n) + bias
(14)
H+
B[(APIRA (h+) - Ta.ā(he)))² x
n Me
(n) + + (n((
(15)
l+r
E((+ (h+) - Ta,6((11)) (hi)) 2 TO (n) +
Σ
a (n) + +r (n)
(16)
B[(Fip (h.) - Ta, (k,)) X (m) +
a (n) + + (n)
(17)
t=7
1+ Σ
2t
B Bl(+Ab((h)) - (T.b(T))))) (m) +
Te (n) Σ re (n) + r (n)
Σ
(18)
me
bias,,5 = E [Yetr He = he, Auctr = ac:--]]
- E [Y++) H. = he Arstr = Bea-r] The.
Proof. See Appendix C
For 0, the rates of all learners coincide with the corresponding rates in the static setting (Curth & van der Schaar 202T Kennedy 2023).
Interpretation of Theorem I Theorem provides us with the following insights, assuming the rate a.b
(p)(n) vanishes faster than the nuisance rates (e.g., when CATE is of simpler structure): (i) The PI-HA-learner is biased, even asymptotically, when
co. In contrast, all other learners are unbiased (consistent) whenever n - oc, assuming the involved rates vanish. (ii) The PI-RA- and RA-learner are asymptotically equivalent. (iii) The rate of each meta-leamnr depends on the rate of the corresponding nuisance parameters. Hence, the PI-RA- and RA-learner will generally converge faster whenever the response functions are easier estimate, while the Iearrer will converge
7

---

Published as a conference paper at ICLR 2025
faster whenever the propensity scores easier estimate. (iv) The DR-leamer admits doubly robust rate. That is, the DR-leamer will converge at a fast rate if each f, either the propensity score (ale or all response functions p& (-) for & allow for fast rates (for both and b).
STABILIZING THE DR-LOSS VIA INVERSE VARIANCE WEIGHTING
A drawback of the DR-learner is the division by propensity scores in Eq. which can render the second-stage regression unstable whenever the propensity score estimates Ke (a. he) are small. While this is well-known issue in the static setting (Kennedy et al. 2024 Morzywolek 2023), is especially problematic in the time-varying setting where the DR-learner uses products of inverse-pree proppsspropensttyy scores (Bica et al. (2220a). Hence, we now propose method for reducing the variance of the DR-learner.
al.
Our idea is motivated by recent work from Fisher (2024) who proposed to stabilize the DR-learner from the static setting by using so-called inverse-variance weights (IVWs). This is analogous to the R-learner from the static setting, which can be written as inverse-variance weighted version of another learner (the U-learner (Künzel et al. 2211)). We now extend the aforementioned results from the static to the time-varying setting. The following theorem derives the inverse-variance weights for the DR-learner in the time-varying setting.
Theorem 2. We define the weights
(+
[[e aa) T2(( H.)
VR
EII
and V VØR + VDRR-
(19)
If it holds that Var (ul+t (HR+1) HR, Au) = o² for all (l....,6 J.. we obtain
Var(YDR Hi = h1) = 2 DR. Hl, = hi] and V H. = he) = d²E var Hl = he
Proof. See Appendix
C
-VW-DR- Motivated by Theorem we propose an inverse-var settsse-vriinne-weiihtee doubly- robust learner (IVW-DR-learner) for estimating CAPO and CATE. contrast to our DR-learner from above, the IPW-DR-learner is not defined via second-stage regression on pseudo-outcome.. Instead, we introduce it as a minimizer loss weighted with stabilized inverse-variance weights. Using estimators W
Vi
we now define the IPW-DR-learner as Hi and W É va, Hi..
DR.
/fvw = arg min, B
WBn' B[Waa")
(DBR 18in-r(Ce ((H)))) and fiv = arg min,, B
Wa
(ga. r'H.))
(20)
We emphasize that the assumption of constant variance in Theorem 2 is in line with the assumption from (Fisher 2024) interpret the weights of the R-learner (Nic & Wager 202T) in the static setting as inverse-variane weights. Even if the assumption is violated, the weights may still stabilize the doubly robust loss despite not being interpretable as unvereevvariane weights.
To gain insights into the weighting mechanism of the IVW-DR-learne, we make comparison to the static setting where 0. Here, we obtain E H, h4] = (r(ae h+)-1 + (ee h-)--
(=(o:lh))-sab((bby (h)) for 0. Hence, for binary treatments, which imply n (at h,)+m, h.) 1, (x(arlla))xx0b1lee)) the inverse variance weights reduce to the product of propensities Te(a; TT Te). These weights reduce the influence of samples with low overlap, which may lead to instabilities due to lack of data. This corresponds to the DR-learner with inverse-variance variance weights in the static setting. In particular, our IVWs are a generalization to both time-varying and multiple treatments.
EXPERIMENTS
In this section, we compare our proposed meta-learners empirically. The purpose of our experiments two-fold: (i) we verify the insights from theoretical results (Sec.) 5) and (ii) we show that our
8

---

Published as a conference paper at ICLR 2025
IVW-DR- DR-learner improves over the DR-learner and other meta-leamers, specifically in low-overlap regimes and for long time horizons т. Recall that all our results are model-agnostic and therefore hold for arbitrary machine learning instantiations. To ensure fair comparison, we estimate all nuisance functions and instantiate all meta-learners using the same transformer-
architecture. In particular, we refrain from comparing with model-based methods as these can be viewed as instantiations of certain meta-learners (see T using specific models. Of note, this is consistent with established literature on meta-learners in the static setting (Curth & van der Schaar 202T). To empirically verify robustness with respect to model choice, we performed additional experiments using LSTM--baee based instantiations of our meta-leamers Appendix H We also provide additional results in Appendix !]
Neural instantiations. In our experiments, we use transformer-basee instantiations of our meta- learners. That is, we estimate both our nuisance function estimators and second-stage regressions using deep neural networks, specifically transformer architectures. Transformers (Ashish Vaswani at. 2017) are effective at learning representations that achieve impressive performance on sequential learning problems, including causal inference tasks (Melnychuk et al. 2022). However, emphasize that our implementation should be seen as a "proof-of-concep'" This means that we do not claim that the architecture of our transformer instantiations is optimized for task-specific performance. Rather, practitioners will need to determine the preferred model architecture based on various factors such sample size or data dimensionality (Curth & van der Schaar 2021). Nevertheless, this also pinpoints a clear advantage of our meta-learners: they can be used with arbitrary machine learning models and allow various other extensions (e.g., balancing losses; see Appendix [F for a discussion. In Appendix H we provide additional results using LSTM-bass instantiations.
We build upon an encoder-only trans- former from Ashish Vaswani et al. (2017) with a causal mask to avoid look-ahead bias and non- -on-trainable po- sitional encodings, combined with a feed-forward neural network with lin- car or softmax activation. We perform training using the Adam optimizer (Kingma & Ba 2015
Table 2: Results for D1.
Pl LLA leamer
1.57 0.338
10.92.1 1:24
16:00 2.01
14.45 5,30
7.56 2:42
PLRA--aanerr
0.50
2.60
1.72 0:25
KA-caaner
0.95 0.446
0.30
2.13
0.53
2.29 + 0,80
1.00
IPW-earner
0.00+
1.13
0.51+
14.77
8.25+-1.40
5.00
DR-kaaner
0.055
0.70
14:00
6.82-22.55
5.50 3.84
IVW-DR-kkarser
2.36
1.28 0.36
Repurtal: Average R RMIEE stamont devinten (x100) over
5 midden weads (beai in hold).
Further details regarding architecture,
training, hyperparameters, and runtime are in Appendix E
Simulated datasets. We simulate three datasets Dj with j (1,2,3 from different data-generating processes. Of note, the use of simulated data is standard in the causal inference literature (Lim et al.) 2018 Bica al. 2020al Curth al. 2020; Frauen & Feuerriegel 2023) and enables measurement of the performance of causal inference methods due to known ground-truth. We report full details
regarding the generation of all datasets D, in Appendix G
Table 3: Results for D2.
At a high level, the datasets satisfy the following properties: (i) All datasets include tare-varying con- founders X,, which should render the
-=0
2-
r --3
7-4
PI JIA learner
2,52 0.36
1.50 1.44
2.65 0.34
2.26 1
3.18 1..39
PI-RA--eam
2:50 0.30
1.37 0.00
1.08 0.30
1.00 1 0:30
0.63 0.559
R..leamer
0.51 : 0.22
0.53
2 11.24
0.75
0.80 0.48
0.56 11.40
PI-HA-learner biased. (ii) The treat- ment effect mechanisms of all datasets are "simpler" than the response func- tion mechanisms, which should give two-stage meta-learners an advantage over plug-in meta-learners (see Sec. 5) (iii) The treatment assignment mechanism of Di (D2) is more (less) "complex" than the response function mechanisms, which should give learners based on regression adjustment an advantage (disadvantage) over learners based on propensity adjustment (see Sec. 5). (iv) Both Dj and D3 contain observations with low overlap (i.e., large or small propensity scores), which should increase the variance of the IPW- and the DR-learner and increase the relative performance of the Iea-DR-learner. Hence, we now verify whether we can confirm these properties in our numerical experiments.
JPW learner
0.20 1 0.07
0.42
0.133
0.58
0.16
1.66
2.02
11.34
DR leamner
0.34 1 0.07
0.42
1 0.32
0.52 0.00
1.84 0.39
2,00
0.71
WWW.DR-kaaner
0.11 + 0.07
0.92
+ 0.51
0.07
0.24
0.50 de 0.46
0.50
=: 0.35
Reponeel
RXISE hol)
Results for Dj and Dz. The results for Dj and D2 are reported in Table Table respectively. Here, we report the RMSE of the estimated CATES for E (0, 2, 4} using different interventions ā, and b, (see Appendix G for details). In line with the properties of the datasets, we observe the following: (i) For t 0, the PI-HA achieves the worst performance as compared to the other meta- meta-learners, indicating bias due to runtime confounding. (ii) The two-stage learners (RA, IPW, DR, and IVW-DR) outperform the plugin learners (PI-HA and PI-RA) on both datasets. (iii) The

---

Published as a conference paper at ICLR 2025
RA-learner outperforms the IPW-learner on D1 (except for 0), while the IPW-learner is better on D2. In particular, the IPa-learner performs better for smaller prediction horizons т. which is likely due to increasing variance when dividing by products of propensity scores. The DR- and IVW-DR learners tend to perform well on both datasets, which confirms the doubly robust convergence rate from [ (iv) The IPW- and DR-leamers unstable on Di for r = 2, which is likely due to low overlap and divisions by products of propensity scores, Notably, the IV WW-DR-learner improves over both by a large margin, indicating a stabilizing effect of the inverse variance weights on the DR-loss. The performance gain is particularly noticeable for larger where the overlap is inherently smaller. In sum, our empirical findings confirm our theoretical results and the effectiveness of our IVW-DR-leammr in scenarios with low overlup and long time horizons..
Sensitivity to low overlap (Dz). We further analyze the sensi- tivity of the DR-learner to low overlap and compare it with our IvW-DR-leamer. To do so, we report the RMSE (for = 1) for both learners on D3 across different levels of overlap (quantified via a parameter 7/, see Appendix G for details). The results are shown in Fig As expected, the DR-learner becomes unstable for lower levels of overlap (i.e., larger 7). In contrast, the WWWDR--leamer remains stable and outperforms the DR-learner by a large margin for small overlap. This conforms the effectiveness of our inverse variance weights (Theorem to stabilize the DR-loss, which is crucial in the time-varying setting that comes with inherently low overlap.
041
2.58
2.75
5.50
3.86
3,50
3.75
480
Figure 3: RMSE of DR- and IVW-DR-learner for different levels of overlap averaged over 5 random seeds. Larger Y implics lower overlap.
3000 patient trajectories electronic health records over up to T = 10 time points from the MIMIC III dataset (Johnson et al. 2016). Similar to Meleenyy al.) 2002), we include 25 time-varying covariates, gender and age as static covariates, ventilation as a binary treatment, and blood pressure as time-varying outcome (standardized with a mean of 61.09 and a standard deviation of 14.48). We train our meta-learners for CAPO and prediction horizons T E {0, 1, 2, 3) and for the treatment intervention that gives treatment at every time step. We use the factual RMSE (i.e., RMSE between model predictions and observed outcomes whenever observed treatments coincide with the intervention sequence) as proxy for performance on real-world data. However, note that a reliable evaluation of causal models on real-world data is notoriously hard due to the fundamental problem of causal inference (Shalit et al.] 2017]] Curth & van der Schaar Hence, our results should be seen as indications and interpreted with care.
Real-world dataset. We sample
2021
Results: Interestingly, both the PI-HA and the PI-RA learner perform similarly well, even though the PI-HA learner suffers from bias due to time-varying confounding (Theorem D]. We suspect that this is due to a larger finite-sample variance of the PI-RA learner, as it requires iter- ative fitting T models instead of only one. Both the IPW and DR-learner perform relatively badly and become unstable for larger time horizons.
Table 4: Results for woaldworld data.
2=2
PI-HA-leamer
0.36 + 0.12
0.65 0.02
0.73 + 0.03
0.79 + 0.02
PL-AA-eammrr
0.36 0.02
0.65 0.03
0.72
0.02
0.02
0.78
0,04
[PW-leammer
0.53 0.10
0.71 0.01
0.088
2,03
2.05
DR-leumer
0.72-008
0.043
0.44
IVw.DR-leamer
0.30
0.03
0.62
0.01
0.02
Reponed: Average factual RMSE standard devinioo random seeds.
This is likely due to limited overlap which leads to divisions by small (products) of propensities. In contrast, our IVW-DR-learner achieves the best overall performance. This confirms the effectiveness of our inverse variance weights real-world data.
Discussion. In this paper, we proposed model-agnostic meta-learners that can be used for estimating CAPO CATE over time using arbitrary machine learning models. Limitations. In this paper, we studied the theoretical foundations of meta-learners and provided practical insights. However, we did not study specific model architectures as this generally depends on the complexity of the data-generating process, overlap, and sample size. Future directions. We helieve the development of state-of-the thet model architectures that can be used as backbones for our meta-learners for specific applications to be a valuable direction for future research. Furthermore, future research may consider extending our meta-learners to the continuous time setting. Broader impact. Our meta-learners have the potential to improve various fields in which personalized sequential decision maaiinnnmaaing is of importance. However, careful implementation needed to ensure robustness practice.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Naoufal Acharki, Ramiro Lugo, Antoine Bertoncello, and Josselin Garnier. Comparison of meta- learners for estimating multi-valued treatment heterogeneous effects. In ICML, 2023.
Ahmed Allam, Stefan Feuerriegel, Michael Rebhan, and Michael Krauthammer. Analyzing patient trajectories with artificial intelligence. Journal of Medical Internet Research, 23(12):e29812, 2021.
Ashish Vaswani. Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurIPS, 2017.
Keith Battocchi, Eleanor Dillon, Maggie Hei, Greg Lewis, Paul Oka, Miruna Oprescu, and Vasilis Syrgkanis. EconML: A python package for ML-based heterogeneous treatment effects estimation, 2019.
Ioana Bica, Ahmed M. Alaa, James Jordon, and Mihaela van der Schaar. Estimating counterfactual treatment outcomes over time through adversarially balanced representations. In ICLR, 2020a.
Ioana Bica, James Jordon, and Mihaela van der Schaar. Estimating the effects of continuous-valuee interventions using generative adversarial networks. In NewrlPS, 2020b.
Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James M. Robins. Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1):C1-C68 2018. ISSN 1368-4221.
Victor Chernozhukov, Whitney Newey, Rahul Singh, and Vasilis Syrgkanis. Automatic debiased machine learning for dynamic treatment effects and general nested functionals. arXiv preprint, MrXiv:2203.13825 2023.
Amanda Coston, Edward H. Kennedy, and Alexandra Chouldechova. Counterfactual predictions under runtime confounding. In NeurlPS, 2020.
Alicia Curth and Mihaela van der Schaar. Nonparametric estimation of heterogeneous treatment effects: From theory to learning algorithms. AISTATS, 2021.
Alicia Curth, Ahmed M. Alaa, and Mihaela van der Schaar. Estimating structural target functions using machine learning and influence functions. arXiv preprint, arXiv:2008.0644 2020.
Alicia Curth, Changhee Lee, and Mihaela der Schaar. SurvITE: Learning heterogeneous treatment effects from time-to-event data. In NeuriPS, 2021.
Stefan Feuerriegel, Dennis Frauen, Valentyn Melnychuk, Jonas Schweisthal, Konstantin Hess, Alicia Curth, Stefan Bauer, Niki Kilbertus, Isaac S. Kohane, and Mihaela van der Schaar. Causal machine learning for predicting treatment outcomes. Nature Medicine, 2024.
Aaron Fisher. The connection between r-learning and inverse-variance weighting for estimation of heterogeneous treatment effects. In ICML, 2024.
Dylan J. Foster and Vasilis Syrgkanis. Orthogonal statistical learning. The Annals of Statistics, 53(3): 879-908, 2023. ISSN 0090-5364.
Dennis Frauen and Stefan Feuerriegel. Estimating individual treatment effects under unobserved confounding using binary instruments. In ICLR,, 2023.
Dennis Frauen, Tobias Hatt, Valentyn Melnychuk, and Stefan Feuerriegel. Estimating average causal effects from patient trajectories. In AAAI, 2023a.
Dennis Frauen, Valentyn Melnychuk, and Stefan Feuerriegel. Sharp bounds for generalized causal sensitivity analysis. In NeurIPS, 2023b.
Margaret A. Hamburg and Francis S. Collins. The path to personalized medicine. The New England Journal of Medicine, 361(4)3301-304, 2010.
Konstantin Hess and Stefan Feuerriegel. Stabilized neural prediction of potential outcomes in continuous time. In ICLR, 2025.
11

---

Published as a conference paper at ICLR 2025
Konstantin Hess, Dennis Frauen, Valentyn Melnychuk, and Stefan Feuerriegel. G-transformer for conditional average potential outcome estimation over time. arXiv preprint, arXiv:2405210012, 2024a.
Konstantin Hess, Valentyn Melnychuk, Dennis Frauen, and Stefan Feuerriegel. Bayesian neural controlled differential equations for treatment effect estimation. In ICLR, 2024b.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):
1735-1780, 1997. ISSN 0899-7667.
Fredrik D. Johansson, Uri Shalit, and David Sonntag, Learning representations for counterfactual inference. In ICML, 2016.
Alistair E. W. Johnson, Tom J. Pollard, Lu Shen, Li-wei H. Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G. Mark. MIMIC-III, a freely accessible critical care database. Scientific Data, 3(1):160035, 2016. ISSN 2052-4463.
Nathan Kallus and Masatoshi Uchara. Intrinsically efficient, stable, and bounded off-policy evaluation for reinforcement learning. In NeurIPS, 2019. 2019.
Nathan Kallus and Masatoshi Uehara. Double reinforcement learning for efficient off-policy evalua- tion in markov decision processes. Journal of Machine Learning Research, 21:1-63; 2020.
Nathan Kallus and Masatoshi Uehara. Efficiently breaking the curse of horizon in off off-policy evaluation with double reinforcement learning. Operations Research, 70(6):3282-3302, 2022. 2022.
Edward H. Kennedy. Towards optimal doubly robust estimation of heterogeneous causal effects. Electronic Journal of Statistics, 17(2):3088-3049, 2023.
Edward H. Kennedy, Sivaraman Balakrishnan, and Larry Wasserman. Minimax rates for heteroge- neous causal effect estimation. The Annals of Statistics, 52(2):793-816, 2024. ISSN 0090-5364.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Sören R. Künzel, Jasject S. Sekhon, Peter J. Bickel, and Bin Yu. Metalearners for estimating heterogeneous treatment effects using machine learning. Proceedings of the National Academy of Sciences (PNAS), 16610))4156-4115 2019.
Greg Lewis and Vasilis Syrgkanis. Double/debiased machine learning for dynamic treatment effects via g-estimation. In NeurlPS, 2021.
Rui Li, Stephanie Hu, Mingyu Lu, Yuria Utsumi, Prithwish Chakraborty, Daby M. Sow, Piyush Madan, Jun Li, Mohamed Ghalwash, Zach Shahn, and Li-wei Lehman. G-net: A recurrent network approach to g-computation for counterfactual prediction under dynamic treatment regime. In ML4H, 2021.
Bryan Lim, Ahmed M. Alaa, and Mihaela van der Schaar. Forecasting treatment responses over time using recurrent marginal structural networks. In NeurIPS, 2018.
Valentyn Melnychuk, Dennis Frauen, and Stefan Feuerriegel. Causal transformer for estimating counterfactual outcomes. In ICML, 2022.
Valentyn Melnychuk, Dennis Frauen, and Stefan Feuerriegel. Bounds on iiniseentttiionnnndeeel confounding bias for treatment effect estimation. In ICLR, 2024.
Pawel Morzywolek, Johan Decruyenaere, and Stijn Vansteelandt. On a general class of orthogonal learners for the estimation of heterogeneous treatment effects. arXiv preprint, 1tXiv:2303.12687 2023.
Susan A. Murphy, Optimal dynamic treatment regimes. Journal of the Royal Statistical Society: Series B, 65(2):331-355, 2003. ISSN 1467-9868.
Xinkun Nie and Stefan Wager. Quasi-oracle estimation of heterogeneous treatment effects. Biometrika, 108(2):299-319 2021. ISSN 0006-3444.
12

---

Published as a conference paper at ICLR 2025
James M. Robins. Correcting for non-compliance in randomized trials using structural nested mean models. Communications Statistics Theory and Methods, 23(8):2379-24111 1994. ISSN 0361-0926.
James M. Robins. Robust estimation in sequentially ignorable missing data and causal inference models. Proceedings of the American Statistical Association on Bayesian Statistical Science, pp. 6-10, 1999.
James M. Robins and Miguel Hernán. Estimation of the causal effects of time-varying exposures. Chapman & Hall/CRC handbooks of modern statistical methods. CRC Press, Boca Raton, 2009. ISBN 1784588886
James M. Robins, Miguel A. Hernán, and Babette Brumback. Marginal structural models and causal inference in epidemiology. Epidemiology, 11(5):550-560, 2000.
Donald B. Rubin. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5):688-701, 1974. ISSN 0022-0663.
Nabeel Seedat, Fergus Imrie, Alexis Bellot, Zhaozhi Qian, and Mihaela van der Schaar. Continuous- time modeling of counterfactual outcomes using neural controlled differential equations. In ICMI., 2022.
Uri Shalit, Fredrik D. Johansson, and David Sontag. Estimating individual treatment effect: General- ization bounds and algorithms. In ICML,, 2017.
Charles J. Stone. Optimal rates of convergence for nonparametric estimators. Annals of Statistics, (6), 1980. ISSN 0090-5364.
Masatoshi Uchara, Chengehun Shi, and Nathan Kallus. A review of off-policy evaluation rein- forcement learning. arXiv preprint, arXiv:2212.06355. 2022.
Lars der Laan, Marco Carone, and Alex Luedtke. Combining t-learning and dr-learning: A framework for oracle-efficient estimation causal contrasts. arXiv preprint, arXiv:2402.01972. 2024.
Mark J. van der Laan and Susan Gruber. Targeted minimum loss based estimation of causal effects of multiple time point interventions. The International Journal of Biostatistics, 8(1), 2012.
Toon Vanderschueren, Alicia Curth, Wouter Verbeke, and Mihaela van der Schaar. Accounting for informative sampling when learning to forecast treatment outcomes over time. In JCMI., 2023.
Stefan Wager and Susan Athey. Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association, 122852))::22-1244 2018.
Yun Yang and Surya T. Tokdar. Minimax-optimal nonparametric regression in high dimensions. The Annals of Statistics, 43(2):652-61) 674, 2015. ISSN 0090-5364.
13

---

Published as a conference paper at ICLR 2025
A COMPARISON OF META-LEARNERS FOR ESTIMATING HTES OVER TIME
Comparison of different meta-learners. Table contains an overview of both model-agnostic and model-based learners for estimating HTEs over time. Aside from our meta-learners, the only other model-agnostic learner is the R-learner from (Lewis & Syrgkanis 202I), which leverages Neyman-orthogonal loss but imposes parametric assumptions based on structural nested mean models (SNMMs; see Appendix B) (Robins) 1994). In contrast, our DR- and IVW-DR-leamer are the only nonparametric learners that leverage efficient influence functions.
Comparison with model-based learners. Existing model-based learners for estimating CAPO are instantiations of either (i) the PI-HA-, (ii) the PI-RA, or (iii) the IPW-learner (see Table right column). For (i), existing model-based learners that build upon history adjustments are CRN (Bical ctt al. 2020a) and the Causal Transformer (CT) (Meinychuk et al., 2022). For (ii), G-Net (Li et al. 2021 leverages a version of G-computation and can thus be considered an instantiation of the PI-RA
learner. For (iii,, RMSNs (Lim et al., 2018 are a model-based learner using inverse- prerreettoennsttyy weighting. To the best of our knowledge, no work has considered doubly robust adjustments for estimating CAPO or CATE in the time-varying setting, which is a novelty of our work.
Table 5: Overview of meta-learners for estimatinn heterogeneous treatment effects over (discrete) time.
Aypermenttppe type
Mes Hro-llomer
UnNaue
CARO
CATI
Rertime!!
IWw
Houvy aiposnent
F1-6A-kams
GKN
Akleydras
Regrosicn afpereent
RY
Paryenily atpurenn
IPWle
SMSN
Unmell 101
Daubly rébet afuarnor
IWWOR kamee
kumer
1
7
rite
puenees oeangireec
14

---

Published as a conference paper at ICLR 2025
B
EXTENDED RELATED WORK
B.1 MONSS-AGNOSTIC METHODS FOR ESTIMATING AVERAGE TREATMENT EFFECTS OVER TIME
Model-agnostic estimators for the tire-iarying setting have been proposed for average treatment effects. Here, prominent estimators build upon different time- varying adjustment mechanisms, including engressionaadjustmen (G-computation) (Robins, 1999), propensity adjustment (Robins) al. 2000), and doubly robust adjustment (van der Laan Gruber 2012 Chernozhukov al. 2023 Furthermore, additional estimators have been proposed that impose parametric assumptions on the data- getdreegnnrninggggg process, including estimators based on marginal structural models (MSMs (Robins al. 22000) and structural nested mean models (SNMMs (Robins) 1994)). We emphasize that these estimators are for average treatment effects but not heterogeneous treatment effects.
B.2 MODEL-BASED METHODS FOR CONTINUOUS TIME
Orthogonal our work is a literature stream that focuses on model- learners for uncertainty quantification or for estimating HTES over continuous time (Hess et al. 2024b Hess & Feuerriegel] 2025) Seedat et al. 2022; Vanderschueren et al., 2023). However, our paper focuses on discrete time, and we expect extensions of our meta-learners to continuous time to be an interesting direction for future research.
B.3 R-LEARNER FOR ESTIMATING HTES OVER TIME
The R-learner proposed in Lewis & Syrgkanis (2021) is a model-agnostic learner for estimating HTEs over time. In contrast to our meta-lcarners, it imposes parametric assumptions on the data- generating process. More precisely, the R-learner builds upon the structural nested mean model (SNNM) framework (Robins 1994). which defines the so-called blip functions
a.b (br, Teate, (tatte,deete) E Yust (deeee 0148+1447) Yex+ be, (44041247) H, h, Xe.+e = Tea-es Agc+e = d/,,+e]
(21)
(22)
The main assumption is then that the generaeneg process admits a linear blip function
"t a.b (Te, Brate, Clate) (he a. (Trate, Cgate)
(23)
where of a.b is known.
R-learner. The R-learner proceeds by estimating the nuisance functions
ge())) = E|Yi+ He = he].
(24)
2₁ = 3 (Xeetj+ Actts) (a (attp: f), and (he) = E[ H. hel.
(25)
(26)
Once initial estimators of the nuisance functions are obtained, the R-learner iteratively minimizes the loss
Se arg min E., v%
(Yiur Your (e(e)))
Σ
(Qie 10, (5(he) (Qr, (lee (e((he)
j=E+1
(27)
In the original paper (Lewis & Syrgkanis 2021), the authors consider estimating HTES under a linear Markovian assumption, which they show corresponds to setting 0; (Tretee d(c+e) d/fe. Then, they estimate CATE via
t+r
2.. (Tc) = Σ (Te) (ar - be).
(28)
t=1
15

---

Published as a conference paper at ICLR 2025
C PROOFS
C.1 IDENTIFIARILITY OF CATE AND CAPO
Lemma 1 (G-formula (Robins 1999)). Under Assumption [ both CAPO and CATE are identified and can be expressed in terms of the observational data distribution.
Proof. Under Assumption D we can write the CAPO as
E|Y++ (0e:++)
Hi = hel
Ely (#retT)
Hi he, A, arl
(29)
(a ) Me+1 = hetill | Hi h4, Ar = arl
(30)
E| (Qne+))
He+1 het1, Aut+1 = 0r:++1] H4, = he, = orl
(31)
=BI BI (+
(+r (aeatr) Hetr = hetr Alatr amatr]
(32)
H14r-1 h Auttr-1 ac14--1] .... He = h4, A: arl
(33)
(*e*) Bl. (1 Hitr Th4r,Arte = ap347l
(34)
H = h14--1, A = acst7-1] .... He = hr A = arl
(35)
(36)
where we used sequential ignorability in (+), the tower property in (*x), and consistency in (* *).
The CATE is thus identified as the difference of CAPOs.
C.2 UNBIASEDNESS OF PSEUDO-OUTCOMES
Here, we show the unbiasedness of the pseudo-outcomes that we consider in our main paper when used as a regression target.
PI-RA-learner. Follows directly from Appendix C.I
RA-learner. It holds that
E
a.5 RA
He hz
(37)
=E L(Ar a=} (pbe (Hl+11) - (((e) + 1 1{Ar be) (NL (Bi) - -k44 (Heri)
(38)
+ 1fAi + ai a)1fA + bi} (nt (hi) 1b (hi) Hi = nr
(39)
-ma(a=) hh) (E [ptt1 (H(+1) H, = -he, A, =o]]=
n (h.)) (be hx) (It (Th) E (p4t1 (Hr41) [H. Ju, A. b.)
(40)
(41)
+ (1 Te(au heb h()) (nt (h.) - -1h (Fr)
(42)
(ax 1 he)) (ne (hh) - pdd (fx) + [he hha) (1t (he) - 1h (Tr)
(43)
+ (1 (a (be R.)) (Lt (h.) - 11b (Tr))
(44)
=he (Te) 1 (hx)
(45)
16

---

Published as a conference paper at ICLR 2025
IPW-learner. It holds that
E
Viiww
He = he]
(46)
1(A = art
E
Y(+ He he
(47)
Te(ae l16)
[++
1(Ar ac)
=IE
E
Ye+T Ho+t -:1 H, hr
(48)
(a(
He)
C==t
[++-1
1 1(4+ ar)
[{A+7 acto] Yetr Hots
E
Ho+t-1
H. h
(e( He)
Tetr (actr Hetr)
t=1
(49)
+ 1
1 L(A = ar}
E
E
uett (H(tT) Hetr-1
He = he
(50)
<r(ax He)
(51)
=E [ul (He+1)
H. = hr]
(52)
=1e (ha)
(53)
and hence
E YPpw
H1, he] 1 (hu) pi (ha)
(54)
DR-learner. It holds that
E BBR H. lie]
(55)
c++
p (Hx) (1- 1{As = axk
k-1 1(A: = ar}
=E Yiww H1 = Tuo]
+ E
He h
(56)
(a Hb)
(a( H2)
k=
E=1
k (H&)) (1- 1(Ax =ax
}z) Te(a;
=/e (ha)
+ E
E
II
Hl+7
H4 hr
T<(ax Ha)
H₂)
b=
t=t
(57)
(----
Me (Hx) (1
1{Ax ax3 Th Ax)
1-1
1{A ae}
=/² (hi) +E
...E
Σ k=t
H
H. = h4
Te me(as Hp)
t=t
(58)
ui (Hl:) (1
1iA =ar}
(ar Hi) )
=/4 (Th) + E
Al. = Tre
(59)
= (hz)
(60)
and hence
E y
H. = Tu.]
= 11 (7x) - 11 (hh))
(61)
C.3 PROOF OF THEOREM 2
Proof. Without loss of generality, we show the result for YDR The result for YB follows analogously. By iteratively applying the law of total variance, we obtain
a
Var(x | H. = Tro)
(62)
= =E (var(rokk Metr,AA47 Hi = fgg
(63)
+ Var (E) Hetr, Actt] He = hu
(64)
E (var(y Herr, Artr)
He Tr
(65)
17

---

Published as a conference paper at ICLR 2025
+
Var
E
a. D
Hetr Actil 11
Ht = hg
(66)
+ Var
E van DR
Hetr,
Actol
H Ast+-1
Hr = he
(67)
(68)
Ht
Σ E Var E
[...ERY
(.....
E\YDR
Hetr: Actol...
| Abt1
Hk+1,
H Ar)
He= = hd
(69)
Emst
(B [..... E.R
Hetri Autrl... He, Ar] AI = -Ta)
+ Var
DR
(70)
For the variance term, obtain
Hetrs Astrl... H1, As)
(71)
++
Mi (Hiz) (1- 1fAx = ax)
k-1
1(A, aa)
E
TPW
+
(ak Hx)
II
Te(ae He)
(72)
k=
c=st
5 Mk (Hix) (1
1(A+ = bb}
k-1 1iAe be}
Hitts AptT
H A
(73)
nh (bx Ha)
Te(be H+)
k=t
t=t
H+ 1{A- ar) Te(ae Hi)
(MArr (Herr) - pett (HitT)
E
E
(74)
=0
1{A = be) Te(be He)
(MAtz (Blets
po+t (Hotr)
(75)
t=1
1{A= = ar)
+++-1 [{Ar = =be}
MEtt (H+++)
a
Mett (A+tT)
b
(76)
Me(az He)
abb He)
t=t
# (Hx) (1-
1{Ar ax)
h-1
((A az)
(77)
(x(ax H&)
Te(ac He)
r (Hx) (1
1(Ax = bz}
R-1
L( = =b²} ne(be Hz)
Herr-1, A
He A
nx (bk Hx)
k=t
8=1
(78)
----| 1{A = ar) t=t Te(at He)
(LÂT- (B1+t-1) - 1btt-1 ((
E
(79)
[++-1
1{Ar be}
(H - 5 1 (HltT-i))
(80)
Tbe Hi)
t=1
14T-2
1 [{A ar)
1+--2
1 [ bee
p (H+4T1))
II
Te He) pitt (Hetr-1)
Te(ae He)
(81)
(---
Σ H (Ax) (1- 11-
1(A; =ax)
k-1 1 (Ae = ae)
(82)
(ax Hx)
Te(ae He)
k=t
f=t
18

---

Published as a conference paper at ICLR 2025
147-22 2
Mk (Hx) (1- 1(A+ = bx}
R-1
1iAe = be}
Σ
II t=
2 Actr-
He, Ar
(bb Hx)
Te(be He)
k=t
(83)
_...
(84)
1(Ar ar3
(ue (Ht, - u4 (B.))
1 1(ee be}
(1i+1 (He) - 18 (Hi)
(85)
(a( Hi)
Te (be
He)
+ 1 (H.) - n (H+))
(86)
1 (H.) 1 (H.).
(87)
which implies
Var
E ... .....IDDR Aetrl. H2, As)
DR
H. he
(88)
Var
14; 3 (Ht))
168 (F..)
H1 hrg = 0.
(89)
Furthermore,
Var
"...EEYyi
He-r: ActTl...
HktI, Act1]
Ar, Ab
(90)
k+1
1 [(e de) Te(ae) He)
Var
(n3 11/-1 (H,-1))
(91)
[(A+ bet
(19 (A2) 11j-1 (A)-1))
H As
(92)
(be He)
1 (AA =ad)
Var
(nle (Hs+2) - 11 (H)))
(93)
Vust Te(ar He)
k
1(Ae = be}
(1reta (BR++1) - - (Bss) Hys Av
(94)
Te(be (He)
1{Ar = ar)
Var (n2-1 (FTRtta) Fl, Ac)
(95)
7. (ar (oo)
1=1
1fAe = bet
Var (piti (Hu+1) Ab)
(96)
(he He)
11A = ar)
h
1 1{A = be}
(97)
(a( He)
7be He)
t=t
where (+) follows from the same arguments from Eq. onwards.
Hence,
Var DR
H, he)
(98)
E Var [B (B(....EYysb) E.yy
Hesr, Actrl... ... HR Akt1]
Hn, Ax)
fl. Tus
(99)
1(As ax3 (a
2
1 1(Ax = =b&
(b Hx) )
o²B
+
H, = h4
(100)
H2)
but
Amt
19

---

Published as a conference paper at ICLR 2025
=O'E ViR He = h
(101)
C.4 PROOF OF THEOREM 1
Before proving Theorem we impose additional assumptions from Kennedy (202)).
Assumption 5 (from (Kennedy) 2023)). Let E,, be regression estimator for either the response functions or the pseudo-outcome regressions. Furthermore, we denote Year as the target variable and Z as the conditioning variables of the corresponding regression. We assume Ytar P Ytar (consistency of the target estimator) and that any regression estimator B. satisfies
E, PYar
Z 2] - Bn Yaar Z 2) E, Year
Year
Z 2]
P 0
(102)
(Bn
(Bar Year Z = 2] E z = 2))
V
E
and
E (Embb(2) =1
2|2] S E B(2)2]
(103)
where b(z) = E((tar Ytar
2 = =].
Kennedy 2023) showed that these assumptions hold, e.g., for linear smoothers. Assumption implies the following lemma.
Lemma 2. Under Assumption for any regression estimator Ens it holds that
(TT
7 2]
La 7 =1)² $ (p)(n) E [(2)}}
(104)
We now proceed with the proof of Theorem
Proof. We prove the rates for each learner separately.
PI-HA-learner. Let us define
20.5
PI-HA (h+) = E Y++ H, h4, AbstT = dro+T) - [Y H.= = hy, Aust = bestt
(105)
which is the target estimand of the PI-HA-learner in population. We then can write
E 2..6 PI-HA (he) [
(106)
E
(SpLL (he)
20.6
-PI-A (h+) + 20.5
TI--H (h4) Ta,
(107)
S E
(FrinA (Th) - 20.6
-aa -PLHA (hx)))² + bias
(108)
S E
(E (Ytr H. === h4, Arctt = ],str] - En (Ytt Fl = Tree A = ((:1+7))
(109)
[B YYtt He = he, A bestr]] En [Yetr He = hes Ariett = b |)2]
+ bias
2,6
(110)
STPL-AAT)) (n)) +
5 TPPHHA (n) + bias?
(111)
which proves the rate for the PI-HA-learner.
PI-HA-learner. By applying Lemma iteratively, we obtain
E (p2 -1 (hx)))
S i (n) + E
IE [pt+] (H++1) - (1444)) HI, =ho]² hr!
(112)
< r (m) + E E ( (Bir))))))) H. hh
(113)
v2
(114)
20

---

Published as a conference paper at ICLR 2025
Py (n).
(115)
The result for CATE follows by noting that
B[(SPIRA (hx) Ta, T,, (b.)) < ( + (ae(he) nd(Thi))
- (ha))
(116)
RA-learner.
We start by deriving a formula for the bias term via
BRA (he) Telas he) [ |pt+1 (H(+1) H. = he] - 14 (ha) + 1 (he) - P1B (hi))
(117)
+ (bb ht) (1g (Te) pe (Te) + p (Th) - E lilken (A++1) |,, = Tue)
(118)
+ (1 (at h₂) (bb Ih)) (pe (Tx) - 1 (The + 3 (7+) 1/b (h.))
(119)
Hence, we can apply Lemma 2 and obtain
[[((RiA (he) Th.)( ك (m) E [BnA
a.b
(120)
14- Σ
x (n)
a. (n)
zz(a ho)
(n)
Σ
t (n)
(121)
1-1+1
E=
1+7
+ m his)
Σ
a
(n)
+
mr (n)
(122)
ue
8=1
t=++
1+0
1+
+ (1 -- | he))
le(bel hz))
2
Σ
the (n) +
Σ
me (n)
8=
(123)
l+ S (n)
Σ (m)
(n)
+
Σ 8=1+1
b
(n).
(124)
t=st
IPW learner. The bias term for the IPW-learner can be written as
(125)
1 1{A art
1iAe = ac) (e( l1l)
E
II
Vetr
Fr F1, The
(126)
Ae(as
He)
E=t
+++
1(Ag ar)
1{Ae = art Te(ae He)
E
E
Yetr Hetr
H1 Tr
(127)
(a(
He)
E==
E=
/++T-1
L(A ae)
Tetr (artr (ltt)
E
(128)
He)
(att7 Hotr)
1 1{A = ar) Tet (att7 H(tT)
a (Bett)
Altt-11 ... Hl, = hg
(129)
Te(ae He) Te+ (ast7 Hetr)
/1477- 1fA = =e3 T447
Hotr)
E
E
(130)
* He) (1+77 H+++)
E=t
4+-11
1(A, art
kett (attr Herr)
s4--1 1 ar) Petr(ar47 Hetr)
II
(131)
#g(aE He) kett (attr H(-r)
( He) Netr (att7 Hetr)
H--1 1 1(A ae) Te 7447(att7 HetT)
Phar (Hetr) HetT-1
H, = hg
(132)
(e He) 104r(att Hetr)
21

---

Published as a conference paper at ICLR 2025
1 [A ae) Te+ ( H(+r) To 17--(0147
Hetr) 2 (Ho+r)
E
Hr = ht
#x(ae H₂)
#e4r (0c+T H(tr)
t=t
(133)
(toTT|
1 1(A = act
biT-1 1{Ae = act
+
E
E
Mitt (Hetr) Hotr-1
a
H₁ he
te(az Ht)
me(ae He)
E=t
(134)
(135)
H+T
1 [(A, = or) +
Hx) - #x(ak Hx)
E
H. = hh pik (H&)
(136)
#a(e
He)
#x(ak
Hi)
k=t
f=tt
Hence, Lemma]
yields
(+)
E
ba (hew((h.))
2 CE
E
(T
(Hg) (a Hx)) He = Rua)
(137)
A==t
E
E
(Tx(ax) Hy) #x Hx)) He hg
(138)
km
(n),
(139)
where (*) follows from the boundedness assumptions 2 and (*+) from applying Fubini's theorem.
We obtain the rate for CATE via
a.5 [[(Fip (h+) - (hh)( S rro (n) + E
Bipw (h.)² [he)
(140)
a. x
rr (n) + max
r (n)
max re(t....tr)
rb (n).
(141)
(E(.....)T))
DR learner. We derive the bias term for the DR-leamerr via BR
(ha)
(142)
1[A+ = az}
1fA: = ar) Me(ac He)
E
E
Yetr
(143)
Re(a: He)
E-tt
ig (Hx) (1-
1{Ag ax3
k-1
1(A ar}
(144)
(ax H2)
#t(aa He)
!=1
1 (Hx) (1
1 [AA = ax}
k-1
1 ((Ae = at}
Hetr ... He he
(145)
T& (ax Hx)
Te( (e))
Eut
1(A+ = ae) TR (ax
H2) lak
Hx),,
IE
pik (HIx)
(146)
Aee He)
in(ap Hk)
1R (Hlx) (1
1 1AA ax)
h-1 1(A, = ar}
(147)
(ap Hx)
ftc(a H₂)
t=1
p4 (HK) (1
1 1(Ag = a+}
1fAc = ac}
Hetr ... He ho
(148)
Tz(a H+)
Fe(as l1l)
k=t
E=t
-+--1
1fA: = ae) Me+r (at-r
H.++) me+r(actr Altt)) po+t (Herr)
E
(149)
te(aa He)
Petr(ar:) H++r)
nett (at+T He+r)) (1-
44 1(Ae = ae)
+ 2aar (HetT)
(150)
1utr(a0t Herr))
#e(ae He)
E=t
22

---

Published as a conference paper at ICLR 2025
(1-
Tetr (actr H(tr)
(+T-1
1fAe act Te(ar He)
pl+t (Hlrr)
II
Hetr-1
Hr hr
Tetr (attr H(tr)
=0
(151)
1 = ar) (akak
Hxx) #x(axx 1 H2))
E
E
Mk (Hx)
(152)
fe((a) He)
#x(au (x)
t=t
(7--1
1(Ax ax)
k-1
1(A = =ael
A
(Hx)
1
II
(153)
(ak H
#t(ae H4a)
Σ pk (Hx)
(1-
1(As = ax3
k-1
LiAe = ar}
Hetr-1 ... H, h2
(154)
Tx(ax Hx)
t=t ae(aee He)
T+T-1
1{A ar) Titt (a++7
Hotr)
(1tt Hetr)
E
(155)
Re((ae He)
(147 Hetr)
E=t
(pitr (HlrT) - Hitr (H(++)) He = =he]
(156)
[+ 1 k-1
1(A = ar) xx(as
Hxk) - #x(aa
HR) Me (Hx)
+E
E
(157)
#e( He)
Ax(ak H2)
E=t
(1
1 As ax)
k-1 1[Ar = =aa}
1g (Hx)
(158)
#e(at He)
{=1
(1- 1(A+ = ax3
k-1
1fA+ arl Me(az He)
# (Hx)
Hett-1
H. = hg
(159)
Tx (ak H&)
(160)
1
1 (A₁ = ar) mxaaa
Hx) #x(ak
H&))
E
(A% (itb) 12 (H&))
11, hs
(at He) Ax(ax
Hx)
(161)
where
(+) follows from the bias formula of the IPW-learner.
Hence,
BR [h+)
(162)
1++ 2 CE
(me [dg)) - (a (2)))² 2 (ng (B&)) - 1b (fle)) H1, :1=])h) hn]
E
(163)
A=t
E
(Tx(ax H+) #ax (H))) E (RL (Hss) p (H.)) H, hg
k=
(164)
b+T Σ k=f
(m)
(n)
Eraa
(165)
where (*) follows from the boundedness assumption 2 and (**) follows from Fubini's theorem and the sample splitting assumption 4h. For CATE, we obtain the corresponding rate via
E(((BB (hh)) Ta. (h+)))
AB
[B
(T.)
+ IF.
[ (h.)
(166)
(+
4+.
l+
S (n) +
a (n)
Me (n) + r (n)
Σ n (m)) k=
(167)
E=t
k=
23

---

Published as a conference paper at ICLR 2025
24

---

Published as a conference paper at ICLR 2025
D MATHEMATICAL DETAILS REGARDING ASSUMPTION
In this section, we provide background on the formal definitions of the convergence rates Assump-
tion 3 We follow Stone 1980). Let 0 be parameter and T(0) some target functional that want to estimate.
Definition 1. A sequence of estimators (0) of functional T(Ø) converges with (achievable) rate (r(n) if
lim lim inf sup (T T(0) > cre(n)) 0. c00
(168)
wEN
#ee
Definition 2. A rate (T(n) is called an upper bound to the rate convergence for estimators (0) it holds for all that
lim inf sup (f) -(()]] cro(n)) > 0
(169)
and
lim lim inf sup (T, - T(0)) > cra(n))
(170)
o-0 WEN
AE
(r( is called optimal it is both achievable and upper bound.
Optimal rates are known in a variety of settings and depend on the specific assumption of the underlying function class. In the following, we provide examples of optimal rates that could be used in our Theorem to obtain explicit rates if we impose the corresponding assumptions on our nuisance functions and second-stage regression functions (similar as in, e.g., Curth & van der Schaar (2021): Kennedy (202))).
Smoothness assumptions: Stone (9800) showed that for nonparametrie regression problem with an 2n 77-mmooh regression function, the optimal rate of convergence is 2ntp, where is the dimension of the covariate space.
We say that function f(x) is k-sparse, if it is linear in x E IR and it only depends on k < min(n,p) predictors. (Yang Tokdar 2015) showed, that in this case the optimal rate of the regression problem is given by klog(p) The linearity assumption can be relaxed to an additive structural assumption. Sparsity assumptions are often imposed in high-dimensional settings where n <
Sparsity assumptions:
25

---

Published as a conference paper at ICLR 2025
E IMPLEMENTATION AND HYPERPARAMETER DETAILS
General implementation of nuisance and second-stage regressions: We start by noting that each nuisance and second-stage estimator can be written as function f: H + R that uses the history H, to predict a k-step ahead target Y For example, for the P1-HA-learner, we Y Y We parametrize each learner via So (hxx) = Dr (he)), where 0 (1). v) are learnable parameters, yy H + TRP is a parametrized representation function, and On,- R >> R is parametrized output function. Finally, we learn the propensity scores (T(a)) |-) in joint architecture, while using separate models for the response functions f (-) (as they require nested regressions).
Transformer architecture. We build on encoder transformer as in (Ashish Vaswani et al., 2017). Each transformer consists of single transformer block and employs a causal mask to avoid look- ahead bias. First, the imput is passed through linear input layer. In order to ensure that the sequence order is preserved, we apply a non-trainable positional encoding. Then, the history is passed through the transformer blocks. Each block consists of (i) self-attention mechanism with three attention heads and hidden state dimension modd = 30, (ii) and a feed-forward network with hidden layer size du 20. Both the (i) self-attention mechanism and (ii) the feed-forward network employ residual connections, which are followed by dropout layers with dropout probabilities p = 0.1, respectively. For regularization, we apply post-normalization as in (Ashish Vaswani et al. 2017), that is, layer normalization after each residual connection. Finally, we send the learned representation through feed-forward neural network with one hidden layer of size d' = 20, ReLU nonlinearities, and with linear or softmax activation for regression and classification tasks, respectively.
Training:
We Wi (T(--x)
perform training by minimizing the loss C(0)
Σ T
(JEtr - fo 1 where (((() are either observations of the sta-
bilized inverse-variance weights from Eq. (20) w0! 1/n for all and t. If the target Y44, is discrete (i.e., for learning propensity scores), we use a cross-entropy instead of mean squared error. For nuisance estimators which additionally condition on treatments (c.g., history adjustments or response functions), we split the data and fit the estimator on the part corresponding to the treatment intervention (as done by the T-learner (Curth et al.) (220)) in the static setting). Note that, in our loss, we not only average over samples but also over time steps. This is standard in the time-varying setting and increases the effective sample size by leveraging the time dimension (Lim et al., 2018 Bica et al. 2020a).
Hyperparameters. To ensure a fair comparison, we choose the same hyperparameters (e.g., dimen- sions, learning rate) for all nuisance estimators and meta-learners. We employ additional weight decay for the two-stage learners to avoid overfitting during the pseudo-outcome regression (as the CATE is of simpler structure than the nuisance parameters). This is consistent with the literature meta-learners for CATE estimation (Curth et al. 2020). For reproducibility purposes, we report all
hyperparameters as yymm files Fl
Runtime, For each wranformmr-basee learner, training took approximately 90 seconds using n = 5000 samples and a standard computer with AMD Ryzen 7 Pro CPU and 32GB of RAM.
'Code available at http:///wwiiiii.//////////aaaaae
26

---

Published as a conference paper at ICLR 2025
F REMARKS ON BALANCED REPRESENTATIONS
Prior works CAPO/ CATE estimation propose to learn balanced representations that are non-predictive of the treatment (Bica et al. 2020b Melnychuk et al., 2022 Seedat et al. 2022). The idea is mimic randomized clinical trials and reduce finite-sample error due to estimation variance (Shalit et al. 2017.. However, for time-varying treatment effects, this approach has several drawbacks:
1. Learning guarantees only exist in static (i.e., con-time-vvryingg settings (Shalit et al. 2017
Thus, balanced representations do not resolve bias due to time-varying confounders. In particular, all mentioned works (Bica et al.] 2020b; Melnychuk al. 2022 Seedat et al.
2022) can be interpreted as PI-HA-leameers with an additional balancing loss component. As we have shown, special adjustment methods are necessary to remove bias from time-varying confounders, as done by the PI-RA, RA-, IPW- DR-, and
2. Even in static settings, methods based on balanced representations impose invertibility assumptions the learned representations (Shalit al. (017). This is often unrealistic in time-varying settings, where representations not only incorporate information about the current confounders but also about the full patient history.
3. Because invertibility is difficult to ensure, balanced representations may increase bias (a phenomenon called representation-indeee confounding (Melnychuk et al. (024). We refer to Curth et al. (202T) and Melnychuk et al. (2024) for detailed discussion this issue. Again, this is particularly detrimental in varying settings.
For the above reasons, we decided not to incorporate balanced representations into our model instan- tiations when running our experiments. This is consistent with previous works in the literature (Curth ct al. 2021 Vanderschueren et al. 2023). Nevertheless, we emphasize that balanced representations be easily integrated into meta-learners.
27

---

Published as a conference paper at ICLR 2025
G DETAILS REGARDING EXPERIMENTS ON SIMULATED DATA
Data-generating process. Our general data-generating process for simulating datasets is as follows: we start by simulating an initial confounder X1 (0, 1) from standard normal distribution. For the following time steps & E (2,...,5) 5], we simulate time-varying confounders via
X, = fY H1-1) + Ezi
(171)
for some function fz and where Ex 0.5.. Then, we simulate binary, time-varying treatments via
A. = Bernoulli (px) with P: = o (H.))
(172)
for some function fa and where (-) denotes the sigmoid function. Finally, we simulate time-varying, continuous outcomes via
Y Y=fo(An
fA He) + Egy
(173)
where By .N,,,,.3)
Dataset D1. For D1, we set the confounding function to fo f = 0.5A,, the treatment assignment function to (H+) = coss(..5X, - 0,5(A,,1 - 0.5)). and the outcome assignment function to (A,, H₂) cos( + 0.5 0.5). We sample training dataset of size
Traiin = 5000 and a test dataset of size Aeest = 1000.
Dataset D2- For D2, we set the confounding function to f = 0.5X, the treatment assignment function to fa (H,) = 0.5.X, - 0.5 - 0.5), and the outcome assignment function fu(A,, (A,, H.) c5s(5X,,) 0.5 A, 0.5). We sample training dataset of size Th 10000 and a test dataset of size Teest
1000.
Dataset D3. For D3,
the confounding function to AY,,,,A-1,, H1 = 0.5X, the
treatment assignment function to (H,, = 70.55, 0.5(A,-1 - 0.5)). where 7 is parameter controlling the overlap, and the outcome assignment function to (y H,) cos cos(X,) 0.5(A. 0.5). We sample training dataset of size Paain = 5000 and test dataset of size Restt = 1000.
Interventions. For D and Dr we 1, 01 = (0, 1), interventions a and b, as follows: b2 (1, 0, 0). Hence, the corresponding CATE measures the effect of intervening the last time step as compared to intervening at the first time step. For D2, we set 7 = 1, (0, 1), and b = (1.0).
E {0, 1, 1, and choose the corresponding treatment
(0, 0, 1), bo = 0. 6ι = (1, 0), and
28

---

Published as a conference paper at ICLR 2025
H ADDITIONAL EXPERIMENTS USING LSTM-BASED MODEL ARCHITECTURES
Our meta-learners are model agnostic and can thus be instantiated with any machine learning model. Here, we perform additional experiments for an instantiation using LSTM networks (Hochreiter Schmidhuber 1997). The results are shown in Tables 6 and 7 with the ones from our transformer-base based instantiations.
The results remain largely consistent
Table 6: Results for D
== 0
PI-HA-learner
1.99 + 0.15
8.37 + 0.34
8.83 + 0.44
PI-RA-learner
2.00 = 0.16
3.58 0.35
RA-learner
0.05 + 0.04
0.13 0.10
0.68 = 0.42
IPW-learner
0.10 + 0.06
0.20
0.32 0.16
DR-learner
0.09 0.04
0.13
0.35 0.21
IVW-DR
WW-De-learner learner
0.05 = 0.02
0.17 0.11
0.66 0.31
Results averaged over random seeds. Best learner in bold.
Table 7: Results for D2.
2
PI-HA-learner
3.23 0.61
14.89 1.92
19.43 + 2.25
PI-RA-learner
2.83 + 0.20
6.53 +
7,42 1 0.64
RA-learner
0,20 0.18
1.07
1.04
IPW-learner
0.21 0.16
0.80
6.96 2.64
DR- learnr
0.20 0.13
0.72 0.44
7.33 = 2.00
WW-DR-learner
0.13 + 0.11
0.95 = 0.46
4.33 = 1.27
Results averaged over 5 random seeds. Best learner in bold.
29

---

Published as a conference paper at ICLR 2025
I ADDITIONAL EXPERIMENTS USING REAL-WORLDD DATA
We provide an additional exemplary application of our meta-learners for a specific patient from the MIMIC data who was never treated (Fig. 4. The counterfactual outcomes predicted by the meta-lcarners for 2 6 indicate that providing the patient with ventilation would have decreased their expected blood pressure for future time steps. While evaluating the best counterfactual trajectories is not possible on real-world data, the results may indicate that PIHA and PIRA overestimate the treatment effect while IPW and DR underestimate the effect. In contrast, our IVW-DR-learme provides a balanced trajectory.
Mehoo
Factuals
PIHA
PIRA
DR
IWWDR
Val.o8
-06
Figure 4: Results for real-world medical data. Blood pressure predictions of meta-learners for single patient that was never treated.
30


Reviews:
[{"review_id": "alibiw43NU", "reviewer": "Reviewer_ytb1", "summary": "This paper proposes model-agnostic meta-learners to estimate temporal HTE. Specifically, authors address the main challenges of temproal HTE estimation from a unified view. They theoretically analyze when specific learners are preferable over others. They derive inverse variance weights to stabilize the loss of the DR-learner.", "questions": "Please refer to \"Weaknesses\".", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Authors provided a comprehensive related works section.\n2. The problem setting is clear and overall reasonable. \n3. The theoretical analysis is good and comprehensive, although I can't fully check the correctness.\n4. The empirical validation is comprehensive too.", "weaknesses": "1. I suggest authors explain their motivation in detail. I believe the current version \"Despite the fact that meta-learners are often considered state-of-the-art in the static setting, similar learners for the time-series setting are missing.\"  is not enough. How exactly meta-learners benefit or fit the time-varying setting? I think the reason: \"meta-learners are SOTA under static setting\" is not convincing. \n\n2.  The caption of Figure 1 can be more specific. I suggest authors add some explanation for the edges, for example, why X(t-2) has a direct causal influence on X(t+1)? What is the practical meaning of such causal path?\n\n3. I think some important concepts are not clearly defined. For example, does response surfaces equal response functions? I presume they are the same estimands.\n\n4. I understand the PI-RA-learner, which can be seen as the estimator version of Eq 5. However, can authors explain the pseudo-outcomes (Eq. 8) in detail? When At = at or bt, why the time index of history info (H) is t+1 rather than t?\n\n5. The contribution of methodology may be thin, since I believe IPW estimator for heterogeneous treatment effects or counterfactual outcomes estimation is already proposed by Papadogeorgou et al (2022). 《Causal inference with spatio-temporal data: estimating the effects of airstrikes on insurgent violence in iraq.》 Journal of the Royal Statistical Society\nSeries B: Statistical Methodology, 84(5):1969–1999.\n\n6. In eq.9, authors define the propensity scores in the time-varying setting. My concerns are about the implementation details about this propensity score. To be specific, the action (treatment) sequence Al, is a high-dimensional variable. So how do authors employ history info Hl to model the probability distribution of this high-dimensional variable? Please clarify my concerns.\n\n7. In the experiment section, I understand authors try to emphasize that their proposed estimators do not rely on any specific model architecture. However, to implement an estimator, one have to choose a network, and I believe different network architectures can have diverse performances. In addition, there may be a best model (e.g. Transformer or Mamba). So I think the contribution of \"MODEL-AGNOSTIC\" may not be appropriate. \n\n8. BTW, it seems that authors do not discuss the computation efficiency. I suggest authors add this section in the main text or Appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fhXLJUANKu", "reviewer": "Reviewer_1hT1", "summary": "The authors study the problem of estimating heterogeneous treatment effects in a time-dependent setting, where the treatment and measured covariates can change at each discrete step of a pre-specified time interval. The authors consider several possible meta-learning strategies for conditional average potential outcome (CAPO) and conditional average treatment effect (CATE) estimation which can be combined with any machine learning estimate of a base nuisance function, and they provide a theoretical analysis of the asymptotic error rate of each method. Based on the limitations of the initially proposed meta-learners, they propose the inverse variance weighted doubly robust learner to reduce instabilities that may arise due to low data availability. Finally, they confirm their theoretical findings and insights on both synthetic and real data.", "questions": "1. What is the precise definition for a regression estimator to admit a particular rate in Assumption 3?\n\n2. The real-world experiment reports RMSE on predicted normalized blood pressure. What is the normalization, and should the results of the proposed method be considered \"good\" or \"bad\" in absolute terms (not just relative to the other methods, over which it clearly obtains improvement)? It would be helpful to include an analysis of this sort to indicate where there is still room for future work.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The problem under study is of high practical relevance. While there has been a great proliferation of data availability in recent years, most of it is observational in nature, and collecting data from strict randomized controlled trials is still expensive. Methods that can take advantage of such data are critical. The time-varying aspect of the problem is also a natural feature of observational data collected \"in the wild,\" making the problem setting even more relevant.\n \nThe presentation of the technical results is excellent. I am not an expert on causal inference, and the bare statement of one of the main results of the paper (Theorems 1) is highly technical in nature. However, the authors provide helpful interpretations of their results which make the value of the theorem clear. This clarity of exposition can be seen in many other places in the paper:\n\n- An explanation for the lack of equality in equation (3) (failure to adjust for post-treatment confounders).\n- An explanation for the constant variance assumption in Theorem 2. I had planned to add a question about this assumption to my review, but the authors addressed it immediately.\n- Insight on the weighting mechanism of the proposed inverse variance weighted doubly-robust learner (lines 413-419), especially showing the connection to the simpler but more familiar binary treatment and static (not time-varying) settings.\n- Intuition for which methods should be expected to perform best on which synthetic datasets and why.\n\nThese additional remarks and insights make the results of the paper far more accessible than they would be otherwise.\n\nThe results themselves are also strong. The authors give a theoretical analysis of the asymptotic behavior of each of the studied methods and discuss the implications for when each method should be preferred in practice. They then confirm their theory with synthetic datasets whose qualitative differences mimic some of the possible variation one could encounter when applying these methods in practice. Finally, they also obtain improved performance on real world healthcare data, which is the primary motivating example for the problem under study.", "weaknesses": "While the intuitions that the authors have already provided are greatly appreciated, additional context for some of the quantities used in the paper would be helpful for a non-expert audience. For instance, additional explanation of the nuisance functions in an appendix would be helpful. Specific examples of the rate functions used in Theorem 1 for some particular settings would also make the theorem easier to parse.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "og2VPCxtjQ", "reviewer": "Reviewer_Mx4T", "summary": "This paper studies treatment effect estimation from observational data over a period of time. The observed data contains a (time-varying) covariate, treatment, and outcome at each time point. The proposed methods contain a set of meta-learners designed to be used with any machine learning algorithm as base learners. The meta-learners are specifically designed for four types of confounding adjustment, namely history adjustment, regression adjustment, propensity adjustment, and doubly robust adjustment. In other words, the paper extends existing adjustment methods under the static setting into the dynamic setting. The theoretical bound between each of the proposed estimators and the true effect is given under assumptions that are commonly used in treatment effect estimation. In the experiment section, the paper studied Transformer as one type of learner under one synthetic and the MIMICIII dataset.", "questions": "1. One strength claimed by the author is that the proposed learners can be used with \"arbitrary machine learning models\", and the title also emphasises \"model-agnostic meta-learners\". However, this is not supported by the empirical evaluation at all. I suggest the authors back up their claim with evidence. Also, some comparisons with different machine-learning models are essential for readers to understand the proposed meta-learner better. In its current form, the paper feels very unfinished. \n\n2. I think the paper is relatively well-written until section 7. Unfortunately, the rest of the paper seems very rushed, and the content is limited. In the only instantiation using a Transformer, the authors emphasized that the Transformer has not been optimized for the treatment effect estimation and then use quite a bit of space to give the details of their architecture. I think the space can be better used if more experiment results can be presented. The detailed architecture of Transformer can be in the supplementary. \n\n3. The discussion of results should also be enhanced. Currently, there is only a very brief description of the results of the experiment. There should at least be discussion of the results that can be linked back to the theoretical analysis. For example, in the theorem, it has been shown that the PI-HA learner is asymptotically biased. Still, in the real-world dataset, there are two occasions that PI-HA performs the best and by a significant margin. In fact, except for the IVW-DR-learner, other proposed meta-learners seldom perform better than the PI-HA learner despite the fact that PI-HA is biased while the others are not.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper's strength is that it provides a good organisation of the existing learners and discusses the meta-learners and their estimation bounds. Although the technique used to prove the bound is not novel, it is nice to see the combination of existing techniques to provide bounds for each of the meta-learner.", "weaknesses": "1. One key weakness is that the experiment section is weak. Currently, the content has only one synthetic dataset and one real-world dataset, and the compared methods only include the meta-learners proposed in this paper. Only one instantiation is experimentally studied for the meta-learners proposed in this paper. I think there should at least be some existing state-of-the-art comparison included, and there should also be other instantiations even for a \"proof-of-demonstration\" experiment section, especially since ICLR has a tradition of valuing comprehensive experiments.\n\n2. The later sections of the paper are not as well-written as the earlier ones. The latter sections seem rushed, and many sentences are incorrect. For example, \"Here, we section outline...\" should be this section outline; \"proof-of-demonstration\" should be proof-of-concept; Making things worse, the paper ended abruptly at section 7. There is not even a conclusion paragraph at the end of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6y2bybTQIp", "reviewer": "Reviewer_spHK", "summary": "The authors compare different causal metalearners estimating treatment outcomes over time. They also propose a novel variant of the temporal DR-Learner based on inverse-variance weights. These learners are described and then compared based on a transformer architecture.", "questions": "- I am not sure whether I understand the implications of eq. 3. The authors claim that other methods, such as CRN and CT, do not correctly adjust for this bias. However, as they build an unbiased representation iteratively in each step (based on $\\bar{H}_t$), it seems to me that future covariates get added correctly, and no bias should exist?\n\n- Do the learners with different estimands (e.g., nuisance parameters) use separate models for each, or are they built in a multitask framework? While a multitask framework is sensible for transformers, it does not apply to time series models generally.\n\n- Do you use sample splitting in the experiments? \n\n- Having CAPO as $\\mu$ and CATE as $\\tau$ would be more clear in my opinion, though I leave this up to the authors. Later, $\\mu$ and $\\delta$ are also introduced, though they seemingly refer to similar estimands. What is the reasoning behind this notation? \n\n- The RA-Learner is much more complex than its static variant, where only the final term in Eq. 8 is present. Some intuition here would be appreciated.\n\n___\n\n**Minor points:**\n- Line 290: \"major different\" -> difference", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The presented metalearners are a novel framework for thinking about these models and, as such, this work constitutes an important contribution to the literature. Given the popularity of metalearners in the static setting, these will certainly have wide applicability. Moreover, given the fast evolution in research on time series modeling, having these model-agnostic metalearners will be very valuable. The presented experiments clearly show the benefits of different approaches, with results corresponding to the hypotheses suggested in the theoretical analysis.", "weaknesses": "1. Several points are not entirely clear to me (possibly due to some misunderstandings):\n- If I understand correctly, the challenge of time-varying confounding is different compared to previous work (e.g., CRN). In my opinion, this needs to be stressed more and explained in more detail.\n\n- Some parts of the methodology are not entirely clear (see questions).\n\n- The notation is not always clear to me (see questions).\n\n\n2. There is no conclusion, or a discussion of limitations and possible directions for future work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)