Model-agnostic meta-learners for estimating heterogeneous treatment effects over time

## Abstract

Abstract Estimating heterogeneous treatment effects (HTEs) over time is crucial in many disciplines such as personalized medicine. For example, electronic health records are commonly collected over several time periods and then used to personalize treatment decisions. Existing works for this task have mostly focused on model-based learners (i.e., learners that adapt specific machine-learning models). In contrast, model-agnostic learners – so-called meta-learners – are largely unexplored. In our paper, we propose several meta-learners that are model-agnostic and thus can be used in combination with arbitrary machine learning models (e.g., transformers) to estimate HTEs over time. Here, our focus is on learners that can be obtained via weighted pseudo-outcome regressions, which allows for efficient estimation by targeting the treatment effect directly. We then provide a comprehensive theoretical analysis that characterizes the different learners and that allows us to offer insights into when specific learners are preferable. Finally, we confirm our theoretical insights through numerical experiments. In sum, while meta-learners are already state-of-the-art for the static setting, we are the first to propose a comprehensive set of meta-learners for estimating HTEs in the time-varying setting.

## 1 Introduction

Estimating heterogeneous treatment effects (HTEs) from observational data over time is crucial when randomized experiments are often costly or infeasible [, , ]. A typical example is personalized medicine []. Here, medications are commonly prescribed over the course of several days, and it is nowadays standard that patient responses are captured within electronic health records []. Hence, clinicians can use electronic health records to estimate HTEs over time and thereby analyze the safety and effectiveness of treatments across different patient profiles []. For example, in cancer care, clinicians are commonly interested in which sequence of chemotherapy and immunotherapy has the largest expected benefit for patients with a specific age, gender, or medical history.

Estimating HTEs *over time* (also referred to as the *time-varying setting*) poses unique challenges due to the complex underlying data-generating process and time-varying confounding structure. In particular, estimation is hard because of the following three challenges: 1 Runtime confounding: not all confounders are observed during inference time, which biases naïve estimators and requires adjustment mechanisms tailored to the time-varying setting [, ]. 2 Conditioning on high-dimensional covariates: this means that one must adjust for whole patient histories []. 3 Low overlap: sequences of treatments are less likely observed in observational data from the time-varying setting (as compared to single treatments in the static setting), which thus increases the variance of estimators [].

Existing works for estimating HTEs over time are almost entirely *model-based* (see Table 1). That is, these works propose specific model architectures and/or losses designed to address the challenges 1– 3 from above, or at least some of them [, , ]. Furthermore, existing model-based learners build upon a certain set of adjustment mechanisms (e.g., history adjustment, inverse propensity weighting) [, , ]. In contrast, other important adjustment mechanisms such as doubly robust adjustment (which typically offers favorable theoretical properties) remain overlooked. This is surprising, since model-agnostic methods – so-called *meta-learners* – are commonly used in the static setting (i.e., without time dimension) and implemented popular in software packages [], whereas a comprehensive toolbox of meta-learners for the time-varying setting is still missing.

In this paper, we take a model-agnostic view on estimating HTEs over time. To this end, we propose several *meta-learners* for this task that can be used in combination with arbitrary machine-learning models and come with theoretical guarantees such as Neyman-orthogonality and double robustness. Our meta-learners address the challenges 1– 3 from above as follows: 1 they leverage adjustment strategies tailored to the time-series setting to ensure unbiased estimation, including regression adjustment (G-computation), propensity adjustment, and doubly robust adjustment; 2 they can be instantiated using state-of-the-art models such as transformers to effectively learn representations from high-dimensional, time-varying data; and 3 we show that certain meta-learners (e.g., the doubly robust learner) can suffer from high variance under violations of overlap due to division by propensity scores (or products thereof). As a remedy, we derive time-varying inverse-variance weights to stabilize the doubly robust loss, which yields a novel inverse-variance-weighted doubly robust learner.

Our contributions(^1^11Code is available at [https://anonymous.4open.science/r/CATEMetaLearnersTime-A35B](https://anonymous.4open.science/r/CATEMetaLearnersTime-A35B).) are: (1) We propose several meta-learners for estimating HTEs over time. For this, we leverage different adjustment strategies including a novel, inverse-variance weighted doubly-robust loss. Our meta-learners are model-agnostic, i.e., they can be instantiated using any machine learning model of choice such as transformers. (2) We analyze our meta-learners theoretically and provide insights for when specific learners are preferable over others. We also structure the literature by showing that some existing model-based learners are instantiations of certain meta-learners, while model-based learners corresponding to other meta-learners have not been proposed yet (see Tables 1 and 4). (3) We perform various experiments using synthetic and semi-synthetic data to confirm our theoretical results empirically.

## 2 Related work

Related works on estimating HTEs can be grouped along two dimensions (see Table 1): (a) whether a method is model-based or model-agnostic, and (b) whether a method is for the static or time-varying setting.

**Table 1: Key learners for heterogeneous treatment effects. In yellow: learners proposed in this paper.**
|  | Static setting | Time-varying setting |
| --- | --- | --- |
| Model-based | TARNet [], causal forest [], etc. | RMSNs [], CRN [], CT [], G-Net [], GT [] |
| Model-agnostic | Plug-in-, RA []-, IPW []-, DR []-, | PI-HA-, PI-RA-, RA-, IPW-, DR-, |
| IVW-DR- [], R-learner [] | IVW-DR-, R-learner [] |  |

Learners in the static setting: Both (i) model-based and (ii) model-agnostic meta-learners have been proposed in the static setting, but without considering the time dimension. (i) Prominent model-based learners include the causal forest [] and TARNet []. (ii) Meta-learners in the static setting leverage different adjustment mechanisms. Examples include the RA- or X-learner [, ], the IPW-learner [], the DR-learner [], the R-learner [], and the recently proposed inverse-variance-weighted DR-leaner (IVW-DR-learner) []. Note that these learners focus on the static setting (and not the time-varying setting). As a result, these are *inherently biased for estimating HTEs over time* because runtime confounding renders static adjustment mechanisms insufficient. Despite the fact that meta-learners are often considered state-of-the-art in the static setting [, ], similar learners for the time-series setting are missing.

Model-based learners for HTEs in the time-varying setting: Orthogonal to our work is a literature stream that focuses on model-based learners for uncertainty quantification or for estimating HTEs over continuous time [, , ]. However, our paper focuses on discrete time, and we expect extensions of our meta-learners to continuous time to be an interesting direction for future research.

For discrete time, several model-based learners have been proposed for estimating HTEs. These learners leverage (i) specific model architectures (e.g., recurrent neural networks or transformers with balancing representations [, , ]) or (ii) specific adjustment mechanisms (e.g., G-computation [] or inverse propensity weighting []). However, these learners are *not* model agnostic. To the best of our knowledge, none of these learners uses a doubly robust adjustment mechanism. Further, a theoretical analysis comparing the different learners is missing.

Meta-learners for HTEs in the time-varying setting: We are only aware of a single meta-learner for the time-varying setting: the R-learner proposed in []. While it is model-agnostic, it imposes parametric assumptions on the data-generating process (e.g., linear Markov assumptions). In contrast, our meta-learners are completely nonparametric. We provide a detailed comparison of our meta-learners with the R-learner in Table 4 and Appendix B.

## 3 Problem setup

Data-generating process: We build upon the standard setting for estimating HTEs from observational data over time [, ]. That is, we consider covariates $X_{t}\in\mathcal{X}$, discrete treatments $A_{t}\in\mathcal{A}$, and continuous outcomes $Y_{t}\in\mathbb{R}$ that are observed over several time periods $t$. In medical settings, $X_{t}$ may denote (time-varying) patient characteristics such as heart rate and blood pressure, $A_{t}$ may denote whether a drug is administered, and $Y_{t}$ may denote a health outcome such as the blood sugar level.

Figure: Figure 1: Setting for estimating treatment effects over time.
Refer to caption: x1.png

For each time period $t$, the treatment $A_{t}$ is assigned based on the history $\bar{H}_{t}=\{\bar{X}_{t},\bar{A}_{t-1},\bar{Y}_{t-1}\}$, where $\bar{X}_{t}=(X_{1},\dots,X_{t})$, $\bar{Y}_{t-1}=(Y_{1},\dots,Y_{t-1})$, and $\bar{A}_{t-1}=(A_{1},\dots,A_{t-1})$. The outcome $Y_{t}$ depends on both the history $\bar{H}_{t}$ and the assigned treatment $A_{t}$. A possible causal graph is shown in Fig. 1. Note that w.l.o.g. we do not consider separate static covariates (e.g., age, gender) as we can always consider them to be part of $X_{t}$.
We are further given an observational dataset $\mathcal{D}=\big{\{}\{x_{t}^{(i)},a_{t}^{(i)},y_{t}^{(i)}\}_{t=1}^{T^{(i)}}
\big{\}}_{i=1}^{n}$ consisting of $n\in\mathbb{N}$ patient trajectories sampled i.i.d. from an unknown probability distribution $\mathbb{P}$ of potentially different lengths $T^{(i)}$.

Causal estimands: We build upon the potential outcomes framework [] for time-varying settings [, ] to formally define HTEs. For a time $t$, we consider a fixed sequence of treatment interventions $\bar{a}_{t:t+\tau}=(a_{t},\ldots,a_{t+\tau})$ of length $\tau+1$. We denote the potential outcome $Y_{t+\tau}(\bar{a}_{t:t+\tau})$ as the $\tau$-step-ahead outcome that would be observed when intervening on the treatments $\bar{A}_{t:t+\tau}$ and setting them to $\bar{a}_{t:t+\tau}$. We are then interested in estimating the *conditional average potential outcome (CAPO)*

$$ $\tau_{\bar{a}}(\bar{h}_{t})=\mathbb{E}\left[Y_{t+\tau}(\bar{a}_{t:t+\tau})\; \mid\;\bar{H}_{t}=\bar{h}_{t}\right]$ (1) $$

and the *conditional average treatment effect (CATE)*

$$ $\tau_{\bar{a},\bar{b}}(\bar{h}_{t})=\mathbb{E}\left[Y_{t+\tau}(\bar{a}_{t:t+ \tau})-Y_{t+\tau}(\bar{b}_{t:t+\tau})\;\mid\;\bar{H}_{t}=\bar{h}_{t}\right],$ (2) $$

where $\bar{b}_{t:t+\tau}$ denotes another sequence of treatment interventions. Both CAPO (and CATE) are functions that predict (differences in) the $\tau$-step-ahead potential outcomes based on the history at time period $t$.

Identifiability. We impose the following assumptions to ensure the identifiability of both CAPO and CATE from observational data. These assumptions are standard in the causal inference literature [, , ]. Practically, they exclude spillover effects, deterministic treatment assignments, and unobserved confounders [, , ].

###### Assumption 1 (Identifiability [ 31 ] ) .

We assume that the following conditions hold for all $\bar{a}_{t:t+\tau}$ and $\bar{h}_{t}$: (i) *consistency*: $Y_{t+\tau}(\bar{a}_{t:t+\tau})=Y_{t+\tau}$ whenever $\bar{A}_{t:t+\tau}=\bar{a}_{t:t+\tau}$ ; (ii) *overlap*: $\mathbb{P}(A_{t}=a_{t}\mid\bar{H}_{t}=\bar{h}_{t})>0$ whenever $\mathbb{P}(\bar{H}_{t}=\bar{h}_{t})>0$ ; and (iii) *sequential ignorability*: ${A_{t}\perp Y_{t}(a_{t})\mid\bar{H}_{t}=\bar{h}_{t}}$.

Runtime confounding and time-varying adjustments. A key challenge of the time-varying setting is that the unbiased estimation of CAPO and CATE is non-trivial. Under Assumption 1, both CAPO and CATE are identified and can thus be expressed as a function of the observational data distribution [, ]. However, the time-varying nature of the confounding structure gives rise to adjustment mechanisms that are different from the static setting whenever $\tau>0$. In particular, adjusting for only the history $H_{t}$ is generally insufficient for unbiased estimation []. That is, for $\tau>0$, it holds that

$$ $\mathbb{E}\left[Y_{t+\tau}(\bar{a}_{t:t+\tau})\;\mid\;\bar{H}_{t}=\bar{h}_{t} \right]\neq\mathbb{E}\left[Y_{t+\tau}\mid\bar{H}_{t}=\bar{h}_{t},\bar{A}_{t:t+ \tau}=\bar{a}_{t:t+\tau}\right].$ (3) $$

Surprisingly, Eq.
(3) implies that static meta-learners that only condition on the history $\bar{H}_{t}$ are *biased* in the time-varying setting.
The reason is that the history adjustment in Eq. (3) ignores to adjust for the *post-treatment confounders* $X_{\ell}$ for $\ell>t$, which are not observed during inference time. This phenomenon is also known as *runtime confounding* in the literature []. Adjusting for post-treatment confounders requires leveraging custom adjustment formulas for the time-varying settings. This includes iterated conditional expectations (G-computation) [] and products of propensity scores (inverse-propensity weighting) []. In the next section, we leverage such time-varying adjustment formulas to construct unbiased meta-learners for estimating both CAPO and CATE.

###### Assumption 1 (Identifiability [ 31 ] ) .

## 4 Meta-learners for estimating HTEs over time

In this section, we propose several meta-learners for estimating CAPO and CATE. The general idea of meta-learners is to first estimate so-called *nuisance functions* that capture important aspects of the observational data distribution (e.g., response functions or propensity scores). Then, the estimated nuisance functions are used for estimating the quantity of interest (e.g., CAPO or CATE).

Figure: Figure 2: Overview of the different nuisance estimators and meta-learners proposed in this paper.
Refer to caption: x2.png

Our meta-learners can be grouped along two criteria: A which *adjustment mechanism* is leveraged; and B whether the learner is a *plug-in learner* or a *two-stage learner*. For A, we leverage different adjustment mechanisms, including history adjustment, regression adjustment (G-computation), propensity adjustment, and doubly robust adjustment. For B, the difference between plug-in and two-stage learners is as follows: Plug-in learners “plug” the nuisance functions into the corresponding adjustment formula to estimate the quantity of interest. In contrast, two-stage learners estimate the treatment effect directly by performing a second-stage regression on a pseudo-outcome constructed from the estimated nuisance functions. Two-stage learners often outperform plug-in learners if the target quantity (e.g., CATE) is of simpler structure and thus easier to estimate than the nuisance parameters involved [, ].

In the following, we group our learners according to A and propose both plug-in and two-stage learners for different adjustment mechanisms. We refer to Table 4 for a detailed overview and comparison of our meta-learners.

### 4.1 History adjustment

$\bullet$ PI-HA-learner. A naïve approach to construct a meta-learner is to use the history adjustment $\delta^{\bar{a}}(\bar{h}_{t})=\mathbb{E}\left[Y_{t+\tau}\mid\bar{H}_{t}=\bar{h
}_{t},\bar{A}_{t:t+\tau}=\bar{a}_{t:t+\tau}\right]$. As a result, this learner controls *only* for the history $H_{t}$, while it thus ignores post-treatment confounders. Hence, for some arbitrary estimator $\hat{\delta}^{\bar{a}}(\bar{h}_{t})$, we define the *plug-in history-adjustment learner* (PI-HA-learner) for CAPO and CATE as

$$ $\hat{\tau}_{\mathrm{PI\mathchar 45\relax HA}}^{\bar{a}}(\bar{h}_{t})=\hat{ \delta}^{\bar{a}}(\bar{h}_{t})\quad\text{and}\quad\hat{\tau}_{\mathrm{PI \mathchar 45\relax HA}}^{\bar{a},\bar{b}}(\bar{h}_{t})=\hat{\delta}^{\bar{a}}( \bar{h}_{t})-\hat{\delta}^{\bar{b}}(\bar{h}_{t}).$ (4) $$

Here, the estimator $\hat{\delta}^{\bar{a}}(\bar{h}_{t})$ of the nuisance function $\delta^{\bar{a}}(\bar{h}_{t})$ is plugged into the formulas for CAPO and CATE, respectively. Note that both $\hat{\tau}_{\mathrm{PI\mathchar 45\relax HA}}^{\bar{a}}(\bar{h}_{t})$ and $\hat{\tau}_{\mathrm{PI\mathchar 45\relax HA}}^{\bar{a},\bar{b}}(\bar{h}_{t})$ are *biased* because the history adjustment is *not* sufficient for the time-varying setting (see Eq. (3)). The reason why we nevertheless consider the PI-HA-learner is its low variance, which can lead to a relatively good performance in low-sample settings when estimation variance has a larger impact than confounding bias.

The PI-HA-learner can also be interpreted as a plug-in learner from the static setting (i.e., ignoring time) [], where the history $\bar{H}_{t}$ acts as confounders and the intervention sequence $\bar{A}_{t:t+\tau}$ acts as a discrete treatment. On this basis, one could, in principle, adopt other two-stage learners from the static setting that can handle discrete treatments [] (e.g., doubly robust versions of history adjustment). Of note, such static two-stage learners suffer from the same bias as the PI-HA-learner as they aim at estimating $\delta^{\bar{a}}(\bar{h}_{t})$ and *not* $\mathbb{E}\left[Y_{t+\tau}(\bar{a}_{t:t+\tau})\;\mid\;\bar{H}_{t}=\bar{h}_{t}\right]$.

### 4.2 Regression adjustment (G-computation)

We now introduce an *unbiased* meta-learner for estimating CAPO and CATE based on regression adjustment, also known as G-computation []. The nuisance functions used for this are the response surfaces, defined via

$$ $\mu_{t+\tau}^{\bar{a}}\left(\bar{h}_{t+\tau}\right)=\mathbb{E}\left[Y_{t+\tau} \mid\bar{H}_{t+\tau}=\bar{h}_{t+\tau},A_{t+\tau}=a_{t+\tau}\right]$ (5) $$

and iteratively via

$$ $\mu_{\ell}^{\bar{a}}\left(\bar{h}_{\ell}\right)=\mathbb{E}\left[\mu_{\ell+1}^{ \bar{a}}(\bar{H}_{\ell+1})\mid\bar{H}_{\ell}=\bar{h}_{\ell},A_{\ell}=a_{\ell}\right]$ (6) $$

for $\ell\in\{t,\dots,t+\tau-1\}$.

$\bullet$ PI-RA-learner.
Given estimators of the response functions, we introduce the *plug-in regression-adjustment learner* (PI-RA-learner) for CAPO and CATE via

$$ $\hat{\tau}_{\mathrm{PI\mathchar 45\relax RA}}^{\bar{a}}(\bar{h}_{t})=\hat{\mu} _{t}^{\bar{a}}\left(\bar{h}_{t}\right)\quad\text{and}\quad\hat{\tau}_{\mathrm{ PI\mathchar 45\relax RA}}^{\bar{a},\bar{b}}(\bar{h}_{t})=\hat{\mu}_{t}^{\bar{a }}\left(\bar{h}_{t}\right)-\hat{\mu}_{t}^{\bar{b}}\left(\bar{h}_{t}\right),$ (7) $$

respectively. The plug-in learner is *unbiased* (see Sec. 5). This is because, under Assumption 1, CAPO can be written as $\tau^{\bar{a}}(\bar{h}_{t})=\mu_{t}^{\bar{a}}\left(\bar{h}_{t}\right)$ [, ]. In particular, $\hat{\tau}_{\mathrm{PI\mathchar 45\relax RA}}^{\bar{a}}(\bar{h}_{t})$ adjusts for the post-treatment covariates by successively integrating them out through an iterative regression process. This approach is also called *iterative G-computation* and has been leveraged for estimating *average* treatment effects [, ], while we extend the idea to estimating *heterogenous* treatment effects.

$\bullet$ RA-learner. We now introduce the *regression-adjustment learner* (RA-learner), which is a two-stage learner for estimating CATE using regression adjustment. The motivation is that, in many applications, the CATE $\tau_{\bar{a},\bar{b}}(\bar{h}_{t})$ is of simpler structure than the CAPO $\tau_{\bar{a}}(\bar{h}_{t})$ due to cancellation effects []. Hence, two-stage learners that target CATE directly may outperform the plug-in learner in such scenarios. Given estimators $\hat{\mu}_{\ell}^{\bar{a}}\left(\bar{h}_{\ell}\right)$ of the response functions, we define the pseudo-outcomes

$$ $\displaystyle\hat{Y}_{\mathrm{RA}}^{\bar{a},\bar{b}}$ $\displaystyle=\mathbbm{1}\{A_{t}=a_{t}\}\left(\hat{\mu}_{t+1}^{\bar{a}}\left( \bar{H}_{t+1}\right)-\hat{\mu}_{t}^{\bar{b}}\left(\bar{H}_{t}\right)\right)+ \mathbbm{1}\{A_{t}=b_{t}\}\left(\hat{\mu}_{t}^{\bar{a}}\left(\bar{H}_{t}\right )-\hat{\mu}_{t+1}^{\bar{b}}\left(\bar{H}_{t+1}\right)\right)$ $\displaystyle\quad+\mathbbm{1}\{A_{t}\neq a_{t}\}\mathbbm{1}\{A_{t}\neq b_{t} \}\left(\hat{\mu}_{t}^{\bar{a}}\left(\bar{h}_{t}\right)-\hat{\mu}_{t}^{\bar{b} }\left(\bar{h}_{t}\right)\right).$ (8) $$

We define the RA-learner via the second-stage regression $\hat{\tau}_{\mathrm{RA}}^{\bar{a},\bar{b}}(\bar{h}_{t})=\hat{\mathbb{E}}\left[
\hat{Y}_{\mathrm{RA}}^{\bar{a},\bar{b}}\;\middle|\;\bar{H}_{t}=\bar{h}_{t}\right]$.

### 4.3 Propensity adjustment

Another major type of adjustment is propensity adjustment []. For this, let us define propensity scores in the time-varying setting via

$$ $\pi_{\ell}(a_{\ell}\mid\bar{h}_{\ell})=\mathbb{P}(A_{\ell}=a_{\ell}\mid\bar{H} _{\ell}=\bar{h}_{\ell})$ (9) $$

for $\ell\in\{t,\dots,t+\tau\}$. Each propensity score $\pi_{\ell}(a_{\ell}\mid\bar{h}_{\ell})$ characterizes the treatment assignment mechanism at time $\ell$ based on the history $\bar{h}_{\ell}$.

$\bullet$ IPW-learner. We now propose the *inverse-propensity-weighted learners* (IPW-learner), which is a two-stage learner for estimating CAPO and CATE based on propensity adjustment. Using estimators $\hat{\pi}_{\ell}(a_{\ell}\mid\bar{h}_{\ell})$ of the propensity scores, we define the pseudo-outcomes

$$ $\hat{Y}_{\mathrm{IPW}}^{\bar{a}}=\prod_{\ell=t}^{t+\tau}\frac{\mathbbm{1}\{A_{ \ell}=a_{\ell}\}}{\hat{\pi}_{\ell}(a_{\ell}\mid\bar{H}_{\ell})}Y_{t+\tau}\quad \text{and}\quad\hat{Y}_{\mathrm{IPW}}^{\bar{a},\bar{b}}=\hat{Y}_{\mathrm{IPW}} ^{\bar{a}}-\hat{Y}_{\mathrm{IPW}}^{\bar{b}}.$ (10) $$

Our IPW-learner is a two-stage learner defined via the second-stage regressions

$$ $\hat{\tau}_{\mathrm{IPW}}^{\bar{a}}(\bar{h}_{t})=\hat{\mathbb{E}}\left[\hat{Y} _{\mathrm{IPW}}^{\bar{a}}\;\middle|\;\bar{H}_{t}=\bar{h}_{t}\right]\quad\text{ and}\quad\hat{\tau}_{\mathrm{IPW}}^{\bar{a},\bar{b}}(\bar{h}_{t})=\hat{\mathbb {E}}\left[\hat{Y}_{\mathrm{IPW}}^{\bar{a},\bar{b}}\;\middle|\;\bar{H}_{t}=\bar {h}_{t}\right].$ (11) $$

Inverse-propensity weighting in the time-varying setting has originally been proposed for estimating *average* treatment effects (e.g., via marginal structural models []), which is again a major different to our IPW-learner that aims at estimating *heterogeneous* treatment effects.

### 4.4 Doubly robust adjustment

The above learners (i.e., the PI-HA-, PI-RA-, RA-, and IPW-leaners) have a shortcoming in that they all utilize estimators of only some of the possible nuisance functions. While the PI-HA-, PI-RA-, and RA-learners leverage *only* (part of) the response function estimators, the IPW-learner leverages *only* the propensity score estimators. This motivates us to construct meta-learners that leverage *both* nuisance estimators and thus all components informative of the data-generating process.

$\bullet$ DR-learner. We propose the *doubly robust learner* (DR-learner), which is a two-stage learner that can be used for estimating both CAPO and CATE. For this, we define the pseudo-outcomes

$$ $\leavevmode\resizebox{390.25534pt}{}{$\hat{Y}_{\mathrm{DR}}^{\bar{a}}=\hat{Y}_ {\mathrm{IPW}}^{\bar{a}}+\sum_{k=t}^{t+\tau}\hat{\mu}_{k}^{\bar{a}}\left(\bar{ H}_{k}\right)\left(1-\frac{\mathbbm{1}\{A_{k}=a_{k}\}}{\hat{\pi}_{k}(a_{k}\mid \bar{H}_{k})}\right)\prod_{\ell=t}^{k-1}\frac{\mathbbm{1}\{A_{\ell}=a_{\ell}\} }{\hat{\pi}_{\ell}(a_{\ell}\mid\bar{H}_{\ell})}\quad\text{and}\quad\hat{Y}_{ \mathrm{DR}}^{\bar{a},\bar{b}}=\hat{Y}_{\mathrm{DR}}^{\bar{a}}-\hat{Y}_{ \mathrm{DR}}^{\bar{b}}$}.$ (12) $$

The pseudo-outcomes $\hat{Y}_{\mathrm{DR}}^{\bar{a}}$ and $\hat{Y}_{\mathrm{DR}}^{\bar{a,b}}$ are estimators of the (uncentered) efficient influence function of CAPO and CATE, respectively []. Similar to the static setting [], we introduce our DR-learner for the time-varying setting via second-stage regressions based on the efficient influence functions. That is, we define

$$ $\hat{\tau}_{\mathrm{DR}}^{\bar{a}}(\bar{h}_{t})=\hat{\mathbb{E}}\left[\hat{Y}_ {\mathrm{DR}}^{\bar{a}}\mid\bar{H}_{t}=\bar{h}_{t}\right]\quad\text{and}\quad \hat{\tau}_{\mathrm{DR}}^{\bar{a},\bar{b}}(\bar{h}_{t})=\hat{\mathbb{E}}\left[ \hat{Y}_{\mathrm{DR}}^{\bar{a},\bar{b}}\mid\bar{H}_{t}=\bar{h}_{t}\right].$ (13) $$

Due to the use of efficient influence functions, the DR-learner comes with theoretical properties such as Neyman-orthogonality and double robustness (see Sec. 5).

$\bullet$ IVW-DR-learner. A drawback of the DR-learner is the division by propensity scores in Eq. (12), which can render the second-stage regression unstable whenever the propensity score estimates $\hat{\pi}_{\ell}(a_{\ell}\mid\bar{h}_{\ell})$ are small. While this is a well-known issue in the static setting [, ], it is especially problematic in the time-varying setting where the DR-learner uses *products* of inverse-propensity scores []. Hence, we now propose a method for reducing the variance of the DR-learner.

Our idea is motivated by a recent work from Fisher [] that proposed to stabilize the DR-learner from the static setting by using so-called inverse-variance weights (IVWs). This is analogous to the R-learner from the static setting, which can be written as an inverse-variance weighted version of another learner (the U-learner []). We now extend the aforementioned results from the static to the time-varying setting. The following theorem derives the inverse-variance weights for the DR-learner in the *time-varying* setting.

###### Theorem 1 .

We define the weights

$$ $V^{\bar{a}}_{\mathrm{DR}}=\sum_{k=t}^{t+\tau}\prod_{\ell=t}^{k}\frac{\mathbbm{ 1}\{A_{\ell}=a_{\ell}\}}{\pi^{2}_{\ell}(a_{\ell}\mid\bar{H}_{\ell})}\quad\text {and}\quad V^{\bar{a},\bar{b}}_{\mathrm{DR}}=V^{\bar{a}}_{\mathrm{DR}}+V^{\bar {b}}_{\mathrm{DR}}.$ (14) $$

If it holds that $\text{{Var}}\left(\mu_{k+1}^{\bar{b}}\left(\bar{H}_{k+1}\right)\bigm{|}\bar{H}
_{k},A_{k}\right)=\sigma^{2}$ for all $k\in\{t,\dots,t+\tau\}$, we obtain

$$ $\text{{Var}}({Y}_{\mathrm{DR}}^{\bar{a}}\mid\bar{H}_{t}=\bar{h}_{t})=\sigma^{2 }\mathbb{E}\left[V^{\bar{a}}_{\mathrm{DR}}\mid\bar{H}_{t}=\bar{h}_{t}\right]\, \,\text{and}\,\,\text{{Var}}({Y}_{\mathrm{DR}}^{\bar{a},\bar{b}}\mid\bar{H}_{t }=\bar{h}_{t})=\sigma^{2}\mathbb{E}\left[V^{\bar{a},\bar{b}}_{\mathrm{DR}}\; \middle|\;\bar{H}_{t}=\bar{h}_{t}\right].$ $$

###### Proof.

See Appendix C.
∎

Motivated by Theorem 1, we propose an *inverse-variance-weighted doubly-robust learner* (IVW-DR-learner) for estimating CAPO and CATE. In contrast to our DR-learner from above, the IPW-DR-learner is not defined via a second-stage regression on a pseudo-outcome. Instead, we introduce it as a minimizer of a loss weighted with stabilized inverse-variance weights. Using estimators ${\hat{V}^{\bar{a}}_{\mathrm{DR}}}$ and ${\hat{V}^{\bar{a},\bar{b}}_{\mathrm{DR}}}$ and denoting $\hat{\mathbb{E}}$ as the empirical mean, we now define the IPW-DR-learner as

$$ $\leavevmode\resizebox{433.62pt}{}{$\hat{\tau}_{\mathrm{IVW}}^{\bar{a}}=\arg \min_{\tau^{\prime}}\hat{\mathbb{E}}\left[\frac{\hat{V}^{\bar{a}\,-1}_{\mathrm {DR}}}{\hat{\mathbb{E}}\left[\hat{V}^{\bar{a}\,-1}_{\mathrm{DR}}\right]}\left( \hat{Y}_{\mathrm{DR}}^{\bar{a}}-\tau^{\prime}(\bar{H}_{t})\right)^{2}\right]\, \,\text{and}\,\,\hat{\tau}_{\mathrm{IVW}}^{\bar{a},\bar{b}}=\arg\min_{\tau^{ \prime}}\hat{\mathbb{E}}\left[\frac{\hat{V}^{\bar{a}\,\bar{b}\,-1}_{\mathrm{DR }}}{\hat{\mathbb{E}}\left[\hat{V}^{\bar{a}\,\bar{b}\,-1}_{\mathrm{DR}}\right]} \left(\hat{Y}_{\mathrm{DR}}^{\bar{a},\bar{b}}-\tau^{\prime}(\bar{H}_{t})\right )^{2}\right]$}.$ (15) $$

We emphasize that the assumption of constant variance in Theorem 1 is in line with the assumption from [] to interpret the weights of the R-learner [] in the static setting as inverse-variance weights. Even if the assumption is violated, the weights may still stabilize the doubly robust loss despite not being interpretable as inverse-variance weights.

To gain insights into the weighting mechanism of the IVW-DR-learner, we make a comparison to the static setting where $\tau=0$. Here, we obtain $\mathbb{E}[W^{\bar{a},\bar{b}}_{\mathrm{DR}}\mid\bar{H}_{t}=\bar{h}_{t}]=\pi_{
t}(a_{\ell}\mid\bar{h}_{t})^{-1}+\pi_{t}(b_{\ell}\mid\bar{h}_{t})^{-1}=\frac{
\pi_{t}(a_{\ell}\mid\bar{h}_{t})+\pi_{t}(b_{\ell}\mid\bar{h}_{t})}{\pi_{t}(a_{
\ell}\mid\bar{h}_{t})\pi_{t}(b_{\ell}\mid\bar{h}_{t})}$ for $\tau=0$. Hence, for binary treatments, which imply $\pi_{t}(a_{\ell}\mid\bar{h}_{t})+\pi_{t}(b_{\ell}\mid\bar{h}_{t})=1$, the inverse variance weights reduce to the product of propensities $\pi_{t}(a_{\ell}\mid\bar{h}_{t})\pi_{t}(b_{\ell}\mid\bar{h}_{t})$. These weights reduce the influence of samples with low overlap, which may lead to instabilities due to a lack of data. This corresponds to the DR-learner with inverse-variance weights in the static setting. In particular, our IVWs are a generalization to *both* time-varying and multiple treatments.

###### Theorem 1 .

###### Proof.

## 5 Theoretical analysis

In this section, we add a theoretical analysis for the PI-HA-, PI-RA-, RA-, IPW-, and DR-learners. For this, we employ mathematical tools from Kennedy [] and derive *asymptotic bounds* on the point-wise risk of the corresponding learners. We start by imposing the following assumptions.

###### Assumption 2 (Boundedness) .

The response surfaces $\mu_{\ell}^{\bar{a}}\left(\cdot\right)$, the propensity scores $\pi_{\ell}(a_{\ell}\mid\cdot)$, and its estimated versions $\hat{\mu}_{\ell}^{\bar{a}}\left(\cdot\right)$ and $\hat{\pi}_{\ell}(a_{\ell}\mid\cdot)$ are bounded functions for all interventions $\bar{a}$.

###### Assumption 3 (Estimation) .

The regression estimators $\hat{\mathbb{E}}_{n}$ admit rates $r^{\bar{a}}_{\mu_{\ell}}(n)$ for the response functions $\hat{\mu}_{\ell}^{\bar{a}}\left(\cdot\right)$, $r^{\bar{a}}_{\mathrm{PI\mathchar 45\relax HA}}(n)$ for the history adjustments $\hat{\delta}^{\bar{a}}(\cdot)$, and $r^{\bar{a},\bar{a}}_{\mathrm{PO}}(n)$ for the second-stage estimators $\hat{\mathbb{E}}$.
We also impose the stability assumptions from Kennedy [] on all regression estimators $\hat{\mathbb{E}}_{n}$ (see Appendix C). Finally, the propensity score estimators $\hat{\pi}_{\ell}(a_{\ell}\mid\cdot)$ admit rates $r^{\bar{a}}_{\pi_{\ell}}(n)$.

###### Assumption 4 (Sample splitting) .

For each learner, the observational dataset $\mathcal{D}$ admits a partition $\mathcal{D}=\mathcal{D}^{\hat{\mu}}_{t}\cup\dots\cup\mathcal{D}^{\hat{\mu}}_{t
+\tau}\cup\mathcal{D}^{\hat{\pi}}\cup\mathcal{D}^{\mathrm{PO}}$, so that each response function estimator $\hat{\mu}_{\ell}^{\bar{a}}$ is fitted on $\mathcal{D}^{\hat{\mu}}_{\ell}$, all propensity score estimators $\hat{\pi}_{\ell}$ are fitted on $\mathcal{D}^{\hat{\pi}}$, and any second-stage regressor $\hat{\mathbb{E}}$ is fitted on $\mathcal{D}^{\mathrm{PO}}$.

Boundedness assumptions such as Assumption 2 are standard in the causal inference literature to provide theoretical guarantees for treatment effect learners [, , , ]. Assumption 3 on the estimators is in line with Kennedy[]. The rates $r^{\bar{a}}_{\mu_{\ell}}(n)$, $r^{\bar{a}}_{\pi_{\ell}}(n)$, $r^{\bar{a}}_{\mathrm{PI\mathchar 45\relax HA}}(n)$, and $r^{\bar{a},\bar{a}}_{\mathrm{PO}}(n)$ capture assumptions on the *complexity* of the corresponding estimands, i.e., nuisance parameters and CATE. The assumption can be instantiated via, e.g., enforcing smoothness or sparsity on the underlying model [, ]. Finally, Assumption 4 (i.e., sample splitting) is standard in the literature on meta-learners for causal inference [, , ]. While sample splitting is required for the validity of our theoretical results, we note that, in practice, all of our learners can also be used without sample splitting. Avoiding the use of sample splitting can result in superior finite-sample performance as more data is used to estimate nuisance parameters and second-stage regressions [].

The following result establishes asymptotic rates for the risk our meta-learners for CATE under treatment interventions $\bar{a}$ and $\bar{b}$. Rates for CAPO can be obtained by removing all terms that include $\bar{b}$. We write $g(n)\lesssim f(n)$ whenever there exist $C,n_{0}>0$, so that $g(n)\leq Cf(n)$ for all $n>n_{0}$.

###### Theorem 2 .

Under Assumptions 1–4, we obtain the following rates:

$$ $\displaystyle\mathbb{E}[(\hat{\tau}_{\mathrm{PI\mathchar 45\relax HA}}^{\bar{a },\bar{b}}(\bar{h}_{t})-\tau_{\bar{a},\bar{b}}(\bar{h}_{t}))^{2}]$ $\displaystyle\lesssim r^{\bar{a}}_{\mathrm{PI\mathchar 45\relax HA}}(n)+r^{ \bar{b}}_{\mathrm{PI\mathchar 45\relax HA}}(n)+\mathrm{bias}^{2}_{\bar{a},\bar {b}}$ (16) $\displaystyle\mathbb{E}[(\hat{\tau}_{\mathrm{PI\mathchar 45\relax RA}}^{\bar{a },\bar{b}}(\bar{h}_{t})-\tau_{\bar{a},\bar{b}}(\bar{h}_{t}))^{2}]$ $\displaystyle\lesssim\sum_{\ell=t}^{t+\tau}r^{\bar{a}}_{\mu_{\ell}}(n)+r^{\bar {b}}_{\mu_{\ell}}(n)$ (17) $\displaystyle\mathbb{E}[(\hat{\tau}_{\mathrm{RA}}^{\bar{a},\bar{b}}(\bar{h}_{t })-\tau_{\bar{a},\bar{b}}(\bar{h}_{t}))^{2}]$ $\displaystyle\lesssim r^{\bar{a},\bar{b}}_{\mathrm{PO}}(n)+\sum_{\ell=t}^{t+ \tau}r^{\bar{a}}_{\mu_{\ell}}(n)+r^{\bar{b}}_{\mu_{\ell}}(n)$ (18) $\displaystyle\mathbb{E}[(\hat{\tau}_{\mathrm{IPW}}^{\bar{a},\bar{b}}(\bar{h}_{ t})-\tau_{\bar{a},\bar{b}}(\bar{h}_{t}))^{2}]$ $\displaystyle\lesssim r^{\bar{a},\bar{b}}_{\mathrm{PO}}(n)+\sum_{\ell=t}^{t+ \tau}r^{\bar{a}}_{\pi_{\ell}}(n)+r^{\bar{b}}_{\pi_{\ell}}(n)$ (19) $\displaystyle\mathbb{E}[(\hat{\tau}_{\mathrm{DR}}^{\bar{a},\bar{b}}(\bar{h}_{t })-\tau_{\bar{a},\bar{b}}(\bar{h}_{t}))^{2}]$ $\displaystyle\lesssim r^{\bar{a},\bar{b}}_{\mathrm{PO}}(n)+\sum_{\ell=t}^{t+ \tau}\left(r^{\bar{a}}_{\pi_{\ell}}(n)\sum_{k=\ell}^{t+\tau}r^{\bar{a}}_{\mu_{ k}}(n)+r^{\bar{b}}_{\pi_{\ell}}(n)\sum_{k=\ell}^{t+\tau}r^{\bar{b}}_{\mu_{k}}( n)\right),$ (20) $$

$\mathrm{bias}_{\bar{a},\bar{b}}=\mathbb{E}\left[Y_{t+\tau}\mid\bar{H}_{t}=\bar
{h}_{t},\bar{A}_{t:t+\tau}=\bar{a}_{t:t+\tau}\right]-\mathbb{E}\left[Y_{t+\tau
}\mid\bar{H}_{t}=\bar{h}_{t},\bar{A}_{t:t+\tau}=\bar{b}_{t:t+\tau}\right]-\tau
_{\bar{a},\bar{b}}(\bar{h}_{t})$.

###### Proof.

See Appendix C.
∎

For $\tau=0$, the rates of all learners coincide with the corresponding rates in the static setting [, ].

Interpretation of Theorem 2. Theorem 2 provides us with the following insights, assuming the rate $r^{\bar{a},\bar{b}}_{\mathrm{PO}}(n)$ vanishes faster than the nuisance rates (e.g., when CATE is of simpler structure): (i) The PI-HA-learner is biased, even asymptotically, when $n\to\infty$. In contrast, all other learners are unbiased (consistent) whenever $n\to\infty$, assuming the involved rates vanish. (ii) The PI-RA- and RA-learner are asymptotically equivalent. (iii) The rate of each meta-learner depends on the rate of the corresponding nuisance parameters. Hence, the PI-RA- and RA-learner will generally converge faster whenever the response functions are easier to estimate, while the IPW-learner will converge faster whenever the propensity scores are easier to estimate. (iv) The DR-learner admits a *doubly robust rate*. That is, the DR-learner will converge at a fast rate if each $\ell$, either the propensity score $\pi_{\ell}(a_{\ell}\mid\cdot)$ or *all* response functions $\mu_{k}^{\bar{a}}\left(\cdot\right)$ for $k\geq\ell$ allow for fast rates (for both $\bar{a}$ and $\bar{b}$).

###### Assumption 2 (Boundedness) .

###### Assumption 3 (Estimation) .

###### Assumption 4 (Sample splitting) .

###### Theorem 2 .

###### Proof.

## 6 Neural instantiations using transformers

This section outlines possible instantiations of our meta-learners using deep neural networks, specifically transformer architectures. Transformers [] are effective at learning representations that achieve impressive performance on sequential learning problems, including causal inference tasks []. However, we emphasize that this section should be seen as a “proof-of-demonstration”. This means that we do not claim that the architecture of our transformer instantiations is optimized for task-specific performance. Rather, practitioners will need to determine the preferred model architecture based on various factors such as sample size or data dimensionality [].
Nevertheless, this also pinpoints a clear advantage of our meta-learners: they can be used with arbitrary machine learning models and allow various other extensions (e.g., balancing losses [, ]).

Architecture: We start by noting that each nuisance and second-stage estimator can be written as a function $f\colon\mathcal{H}\to\mathbb{R}$ that uses the history $\bar{H}_{t}$ to predict a $\kappa$-step ahead target $\widetilde{Y}_{t+\kappa}$. For example, for the PI-HA-learner, we set $\widetilde{Y}_{t+\tau}=Y_{t+\tau}$. We parametrize each learner via $f_{\theta}(\bar{h}_{t})=g_{\eta}(\Phi_{\nu}(\bar{h}_{t}))$, where $\theta=(\eta,\nu)$ are learnable parameters, $\Phi_{\nu}\colon\mathcal{H}\to\mathbb{R}^{p}$ is a parametrized representation function, and $g_{\eta}\colon\mathbb{R}^{p}\to\mathbb{R}$ is a parametrized output function.

Training: We then perform training by minimizing the loss

$$ $\leavevmode\resizebox{216.81pt}{}{$\mathcal{L}(\theta)=\sum_{i=1}^{n}\sum_{t=1 }^{T^{(i)}-\kappa}\frac{w_{t}^{(i)}}{\left(T^{(i)}-\kappa\right)}\left( \widetilde{y}_{t+\tau}^{(i)}-f_{\theta}\left(\bar{h}_{t}^{(i)}\right)\right)^{ 2}$},$ (21) $$

where $w_{t}^{(i)}$ are either observations of the stabilized inverse-variance weights from Eq. (15) or $w_{t}^{(i)}=1/n$ for all $i$ and $t$. If the target $\widetilde{Y}_{t+\tau}$ is discrete (i.e., for learning propensity scores), we use a cross-entropy loss in Eq. (21). For nuisance estimators which additionally condition on treatments (e.g., history adjustments or response functions), we split the data and fit the estimator on the part corresponding to the treatment intervention (as done by the T-learner [] in the static setting). Note that, in Eq. (21), we not only average over samples but also over time steps. This is standard in the time-varying setting and increases the effective sample size by leveraging the time dimension [, ].

Implementation details: We build upon an encoder-only transformer from [] with a causal mask to avoid look-ahead bias and non-trainable positional encodings. Here, we use one transformer block with three attention heads. For the output function $g_{\eta}$, we use a feed-forward neural network with one hidden layer, ReLU nonlinearities, and with linear or softmax activation for regression and classification tasks, respectively. We perform training using the Adam optimizer []. Further details regarding training, choice of hyperparameters, and runtime are provided in Appendix D.

## 7 Experiments

In this section, we compare our proposed meta-learners empirically. To ensure a fair comparison, we estimate all nuisance functions and instantiate all meta-learners using the same transformer-based architecture from Sec. 6. In particular, *we refrain from comparing with model-based methods* as these can be viewed as instantiations of certain meta-learners using different models (see Table 4). Furthermore, we do *not* aim at showing the superior performance of a specific meta-learner (which highly depends on the data-generating process), but rather *verify the insights from our theoretical results* (Sec. 5). The actual choice will eventually depend on the underlying setting (e.g., whether there are complex treatment assignments or whether there is low overlap).

**Table 2: Results for $\mathcal{D}_{1}$.**
|  | $\tau=0$ | $\tau=1$ | $\tau=2$ |
| --- | --- | --- | --- |
| PI-HA-learner | $1.57\pm 0.38$ | $10.92\pm 1.24$ | $16.00\pm 2.01$ |
| PI-RA-learner | $1.60\pm 0.36$ | $2.57\pm 0.53$ | $2.62\pm 0.57$ |
| RA-learner | $0.95\pm 0.46$ | $1.32\pm 0.59$ | $\boldsymbol{2.13\pm 0.83}$ |
| IPW-learner | $0.60\pm 0.28$ | $1.43\pm 0.81$ | $14.77\pm 6.78$ |
| DR-learner | $0.65\pm 0.35$ | $1.39\pm 0.70$ | $14.60\pm 6.50$ |
| IVW-DR-learner | $\boldsymbol{0.57\pm 0.14}$ | $\boldsymbol{1.29\pm 0.23}$ | $2.40\pm 0.43$ |
| Reported: Average RMSE $\pm$ standard deviation ($\times 10$) over |  |  |  |
| $5$ random seeds (best in bold). |  |  |  |

Datasets. We simulate three datasets $\mathcal{D}_{j}$ with $j\in\{1,2,3\}$ from different data-generating processes. Of note, the use of simulated data is standard in the causal inference literature [, , , ] and enables measurement of the performance of causal inference methods due to known ground-truth. We report full details regarding the generation of all datasets $\mathcal{D}_{j}$ in Appendix E.

At a high level, the datasets satisfy the following properties: 1  All datasets include time-varying confounders $X_{t}$, which should render the PI-HA-learner biased. 2  The treatment effect mechanisms of all datasets are “simpler” than the response function mechanisms, which should give two-stage meta-learners an advantage over plug-in meta-learners (see Sec. 5). 3  The treatment assignment mechanism of $\mathcal{D}_{1}$ ($\mathcal{D}_{2}$) is more (less) “complex” than the response function mechanisms, which should give learners based on regression adjustment an advantage (disadvantage) over learners based on propensity adjustment (see Sec. 5). 4  Both $\mathcal{D}_{1}$ and $\mathcal{D}_{3}$ contain observations with low overlap (i.e., large or small propensity scores), which should increase the variance of the IPW- and the DR-learner and increase the relative performance of the IVW-DR-learner. Hence, we now verify whether we can confirm these properties in our numerical experiments.

**Table 3: Results for $\mathcal{D}_{2}$.**
|  | $\tau=0$ | $\tau=1$ | $\tau=2$ |
| --- | --- | --- | --- |
| PI-HA-learner | $2.52\pm 0.56$ | $1.99\pm 0.44$ | $2.65\pm 0.34$ |
| PI-RA-learner | $2.33\pm 0.30$ | $1.37\pm 0.06$ | $1.08\pm 0.30$ |
| RA-learner | $0.51\pm 0.22$ | $0.53\pm 0.24$ | $0.75\pm 0.30$ |
| IPW-learner | $0.20\pm 0.07$ | $0.42\pm 0.13$ | $0.58\pm 0.16$ |
| DR-learner | $0.14\pm 0.07$ | $0.42\pm 0.32$ | $\boldsymbol{0.52\pm 0.09}$ |
| IVW-DR-learner | $\boldsymbol{0.11\pm 0.07}$ | $\boldsymbol{0.32\pm 0.31}$ | $0.57\pm 0.24$ |
| Reported: average RMSE $\pm$ standard deviation ($\times 10$) over |  |  |  |
| $5$ random seeds (best in bold). |  |  |  |

Results for $\mathcal{D}_{1}$ and $\mathcal{D}_{2}$. The results for $\mathcal{D}_{1}$ and $\mathcal{D}_{2}$ are reported in Table 2 and Table 3, respectively. Here, we report the RMSE of the estimated CATEs for $\tau\in\{0,1,2\}$ using different interventions $\bar{a}_{\tau}$ and $\bar{b}_{\tau}$ (see Appendix E for details). In line with the properties of the datasets, we observe the following: 1 For $\tau\in\{1,2\}$, the PI-HA achieves the worst performance as compared to the other meta-learners, indicating a bias due to runtime confounding. 2 The two-stage learners (RA, IPW, DR, and IVW-DR) outperform the plugin learners (PI-HA and PI-RA) on both datasets. 3  The RA-learner outperforms the IPW-learner on $\mathcal{D}_{1}$ (except for $\tau=0$), while the IPW-learner is better on $\mathcal{D}_{2}$. In particular, the IPW-learner performs better for smaller prediction horizons $\tau$, which is likely due to increasing variance when dividing by products of propensity scores. The DR- and IVW-DR learners tend to perform well on both datasets, which confirms the doubly robust convergence rate from Theorem 2. 4  The IPW- and DR-learners are unstable on $\mathcal{D}_{1}$ for $\tau=2$, which is likely due to low overlap and divisions by products of propensity scores. Notably, the IVW-DR-learner improves over both by a large margin, indicating a stabilizing effect of the inverse variance weights on the DR-loss. In sum, *our learners perform as expected, and the empirical findings thus confirm our theoretical results*.

Figure: Figure 3: RMSE of DR- and IVW-DR-learner for different levels of overlap averaged over $5$ random seeds. Larger $\gamma$ implies lower overlap.
Refer to caption: x3.png

Sensitivity to low overlap ($\mathcal{D}_{3}$). We further analyze the sensitivity of the DR-learner to low overlap and compare it with the IVW-DR-learner. To do so, we report the RMSE (for $\tau=1$) for both learners on $\mathcal{D}_{3}$ across different levels of overlap (quantified via a parameter $\gamma$; see Appendix E for details). The results are shown in Fig 3. As expected, the DR-learner becomes unstable for lower levels of overlap (i.e., larger $\gamma$). In contrast, the IVW-DR-learner remains stable and outperforms the DR-learner by a large margin for small overlap. *This confirms the effectiveness of our inverse variance weights (Theorem 1) to stabilize the DR-loss, which is crucial in the time-varying setting that comes with inherently low overlap.*

Discussion. In this paper, we proposed model-agnostic meta-learners that can be used for estimating CAPO or CATE over time using arbitrary machine learning models. Limitations. Our meta-learners fill the gap of model-agnostic estimation of HTEs over discrete time but do not address challenges arising from settings with continuous time. We believe the development of continuous-time meta-learners to be a valuable direction for future research. Broader impact. Our meta-learners have the potential to improve personalized medicine, but careful implementation is needed to ensure robustness in practice.

## Appendix A Comparison of meta-learners for estimating HTEs over time

Comparison of different meta-learners. Table 4 contains an overview of both model-agnostic and model-based learners for estimating HTEs over time. Aside from our meta-learners, the only other model-agnostic learner is the R-learner from [], which leverages a Neyman-orthogonal loss but imposes parametric assumptions based on structural nested mean models (SNMMs; see Appendix B) []. In contrast, our DR- and IVW-DR-learners are the only nonparametric learners that leverage efficient influence functions and thus achieve Neyman-orthogonality [].

Comparison with model-based learners. Existing model-based learners for estimating CAPO are instantiations of either (i) the PI-HA-, (ii) the PI-RA, or (iii) the IPW-learner (see Table 4, right column). For (i), existing model-based learners that build upon history adjustments are CRN [] and the Causal Transformer(CT) []. For (ii), G-Net [] leverages a version of G-computation and can thus be considered an instantiation of the PI-RA learner. For (iii), RMSNs [] are a model-based learner using inverse-propensity weighting. To the best of our knowledge, no work has considered doubly robust adjustments for estimating CAPO or CATE in the time-varying setting, which is a novelty of our work.

**Table 4: Overview of meta-learners for estimating heterogeneous treatment effects over (discrete) time. In yellow: Learners proposed in this paper.**
| Adjustment type | Meta-learners | Nonparametric | Unbiased | CAPO | CATE | Orthogonal | IVWs | Instantiations |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| History adjustment | PI-HA-learner | ✓ | ✗ | ✓ | $\text{({\color[rgb]{0,0.88,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.88,0<br>}\pgfsys@color@cmyk@stroke{0.91}{0}{0.88}{0.12}\pgfsys@color@cmyk@fill{0.91}{0<br>}{0.88}{0.12}\checkmark})}^{\ast}$ | ✗ | ✗ | CRN [], CT [] |
| Regression adjustment | PI-RA-learner | ✓ | ✓ | ✓ | $\text{({\color[rgb]{0,0.88,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.88,0<br>}\pgfsys@color@cmyk@stroke{0.91}{0}{0.88}{0.12}\pgfsys@color@cmyk@fill{0.91}{0<br>}{0.88}{0.12}\checkmark})}^{\ast}$ | ✗ | ✗ | G-Net [], GT [] |
| RA-learner | ✓ | ✓ | ✗ | ✓ | ✗ | ✗ | — |  |
| Propensity adjustment | IPW-learner | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ | RMSNs [] |
| Doubly robust adjustment | DR-learner | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | — |
| IVW-DR-learner | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | — |  |
| SNMM | R-learner [] | $\text{({\color[rgb]{0.72,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.72,0,0<br>}\pgfsys@color@cmyk@stroke{0}{0.89}{0.94}{0.28}\pgfsys@color@cmyk@fill{0}{0.89<br>}{0.94}{0.28}\char 55})}^{\ast\ast}$ | $\text{({\color[rgb]{0,0.88,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.88,0<br>}\pgfsys@color@cmyk@stroke{0.91}{0}{0.88}{0.12}\pgfsys@color@cmyk@fill{0.91}{0<br>}{0.88}{0.12}\checkmark})}^{\ast\ast\ast}$ | ✓ | ✓ | ✓ | ✓ | — |
| ^∗ Learners suffer from plug-in bias.  ^∗∗ Except when $\tau=0$ and the treatment is binary.  ^∗∗∗ If parametric assumptions hold. |  |  |  |  |  |  |  |  |

## Appendix B Extended related work

### B.1 Model-agnostic methods for estimating average treatment effects over time

Model-agnostic estimators for the time-varying setting have been proposed for *average* treatment effects. Here, prominent estimators build upon different time-varying adjustment mechanisms, including regression-adjustment (G-computation) [], propensity adjustment [], and doubly robust adjustment []. Furthermore, additional estimators have been proposed that impose parametric assumptions on the data-generating process, including estimators based on marginal structural models (MSMs []) and structural nested mean models (SNMMs []). We emphasize that these estimators are for average treatment effects but *not* heterogeneous treatment effects.

### B.2 R-learner for estimating HTEs over time

The R-learner proposed in [] is a model-agnostic learner for estimating HTEs over time. In contrast to our meta-learners, it imposes parametric assumptions on the data-generating process. More precisely, the R-learner builds upon the structural nested mean model (SNNM) framework [], which defines the so-called *blip functions*

$$ $\displaystyle\gamma_{\ell}^{\bar{a},\bar{b}}(\bar{h}_{t},\bar{x}_{t:t+\ell}, \bar{a}^{\prime}_{t:t+\ell})$ $\displaystyle=\mathbb{E}\left[Y_{t:t+\tau}(\bar{a}^{\prime}_{t:t+\ell},\bar{a} _{t+\ell+1:t+\tau})-Y_{t:t+\tau}(\bar{a}^{\prime}_{t:t+\ell-1},b_{\ell},\bar{a }_{t+\ell+1:t+\tau})\right.$ (22) $\displaystyle\left.\quad\mid\bar{H}_{t}=\bar{h}_{t},\bar{X}_{t:t+\ell}=\bar{x} _{t:t+\ell},\bar{A}_{t:t+\ell}=\bar{a}^{\prime}_{t:t+\ell}\right].$ (23) $$

The main assumption is then that the data-generating process admits a linear blip function

$$ $\gamma_{\ell}^{\bar{a},\bar{b}}(\bar{h}_{t},\bar{x}_{t:t+\ell},\bar{a}^{\prime }_{t:t+\ell})=\psi_{\ell}(\bar{h}_{t})\phi_{\ell}^{\bar{a},\bar{b}}(\bar{x}_{t :t+\ell},\bar{a}^{\prime}_{t:t+\ell}),$ (24) $$

where $\phi_{\ell}^{\bar{a},\bar{b}}$ is known.

R-learner. The R-learner proceeds by estimating the nuisance functions

$$ $\displaystyle q_{\ell}(\bar{h}_{\ell})$ $\displaystyle=\mathbb{E}[Y_{t+\tau}\mid\bar{H}_{\ell}=\bar{h}_{\ell}],$ (25) $\displaystyle Q_{j,\ell}$ $\displaystyle=\phi_{j}^{\bar{a},\bar{b}}(\bar{X}_{t:t+j},\bar{A}_{t:t+j})-\phi _{j}^{\bar{a},\bar{b}}(\bar{X}_{t:t+j},(\bar{a}_{t:t+j-1},a_{t+j}))\mathbbm{1} (j>\ell),\text{ and}$ (26) $\displaystyle p_{j,\ell}^{\bar{a}}(\bar{h}_{\ell})$ $\displaystyle=\mathbb{E}[Q_{j,\ell}\mid\bar{H}_{\ell}=\bar{h}_{\ell}].$ (27) $$

Once initial estimators of the nuisance functions are obtained, the R-learner iteratively minimizes the loss

$$ $\hat{\psi}_{\ell}=\arg\min_{\psi^{\prime}_{\ell}}\hat{\mathbb{E}}_{n}\left[ \left(Y_{t+\tau}-\hat{q}_{\ell}(\bar{H}_{\ell})-\sum_{j=\ell+1}^{t+\tau}\left( Q_{j,\ell}-\hat{p}_{j,\ell}^{\bar{a}}(\bar{H}_{\ell})\right)\hat{\psi}_{j}( \bar{h}_{t})-\left(Q_{\ell,\ell}-\hat{p}_{\ell,\ell}^{\bar{a}}(\bar{H}_{\ell}) \right)\psi^{\prime}_{\ell}(\bar{h}_{t})\right)^{2}\right]$ (28) $$

In the original paper [], the authors consider estimating HTEs under a linear Markovian assumption, which they show corresponds to setting $\phi_{\ell}^{\bar{a},\bar{b}}(\bar{x}_{t:t+\ell},\bar{a}^{\prime}_{t:t+\ell})=
a^{\prime}_{t+\ell}$. Then, they estimate CATE via

$$ $\hat{\tau}^{\bar{a},\bar{b}}_{\mathrm{R}}(\bar{h}_{t})=\sum_{\ell=t}^{t+\tau} \hat{\psi}_{\ell}(\bar{h}_{t})\left(a_{\ell}-b_{\ell}\right).$ (29) $$

## Appendix C Proofs

### C.1 Proof of Theorem 1

###### Proof.

Without loss of generality, we show the result for $Y_{\mathrm{DR}}^{\bar{a},\bar{b}}$. The result for $Y_{\mathrm{DR}}^{\bar{a}}$ follows analogously. By iteratively applying the law of total variance, we obtain

$$ $\displaystyle\quad\text{{Var}}(Y_{\mathrm{DR}}^{\bar{a},\bar{b}}\mid\bar{H}_{t }=\bar{h}_{t})$ (30) $\displaystyle=\mathbb{E}\left[\text{{Var}}(Y_{\mathrm{DR}}^{\bar{a},\bar{b}} \mid\bar{H}_{t+\tau},A_{t+\tau})\bigm{|}\bar{H}_{t}=\bar{h}_{t}\right]$ (31) $\displaystyle\quad+\text{{Var}}\left(\mathbb{E}[Y_{\mathrm{DR}}^{\bar{a},\bar{ b}}\mid\bar{H}_{t+\tau},A_{t+\tau}]\bigm{|}\bar{H}_{t}=\bar{h}_{t}\right)$ (32) $\displaystyle=\mathbb{E}\left[\text{{Var}}(Y_{\mathrm{DR}}^{\bar{a},\bar{b}} \mid\bar{H}_{t+\tau},A_{t+\tau})\bigm{|}\bar{H}_{t}=\bar{h}_{t}\right]$ (33) $\displaystyle\quad+\mathbb{E}\left[\text{{Var}}\left(\mathbb{E}[Y_{\mathrm{DR} }^{\bar{a},\bar{b}}\mid\bar{H}_{t+\tau},A_{t+\tau}]\bigm{|}\bar{H}_{t+\tau-1}, A_{t+\tau-1}\right)\bigm{|}\bar{H}_{t}=\bar{h}_{t}\right]$ (34) $\displaystyle\quad+\text{{Var}}\left(\mathbb{E}\left[\mathbb{E}[Y_{\mathrm{DR} }^{\bar{a},\bar{b}}\mid\bar{H}_{t+\tau},A_{t+\tau}]\bigm{|}\bar{H}_{t+\tau-1}, A_{t+\tau-1}\right]\bigm{|}\bar{H}_{t}=\bar{h}_{t}\right)$ (35) $\displaystyle=\dots$ (36) $\displaystyle=\sum_{\ell=t}^{t+\tau}\mathbb{E}\left[\text{{Var}}\left(\mathbb{ E}\left[\dots\mathbb{E}[Y_{\mathrm{DR}}^{\bar{a},\bar{b}}\mid\bar{H}_{t+\tau}, A_{t+\tau}]\dots\bigm{|}\bar{H}_{k+1},A_{k+1}\right]\bigm{|}\bar{H}_{k},A_{k} \right)\biggm{|}\bar{H}_{t}=\bar{h}_{t}\right]$ (37) $\displaystyle\quad+\text{{Var}}\left(\mathbb{E}\left[\dots\mathbb{E}[Y_{ \mathrm{DR}}^{\bar{a},\bar{b}}\mid\bar{H}_{t+\tau},A_{t+\tau}]\dots\bigm{|} \bar{H}_{t},A_{t}\right]\bigm{|}\bar{H}_{t}=\bar{h}_{t}\right)$ (38) $$

For the variance term, we obtain

$$ $\displaystyle\quad\mathbb{E}\left[\dots\mathbb{E}[Y_{\mathrm{DR}}^{\bar{a}, \bar{b}}\mid\bar{H}_{t+\tau},A_{t+\tau}]\dots\bigm{|}\bar{H}_{t},A_{t}\right]$ (39) $\displaystyle=\mathbb{E}\left[\dots\mathbb{E}\left[Y_{\mathrm{IPW}}^{\bar{a}, \bar{b}}+\sum_{k=t}^{t+\tau}\mu_{k}^{\bar{a}}\left(\bar{H}_{k}\right)\left(1- \frac{\mathbbm{1}\{A_{k}=a_{k}\}}{\pi_{k}(a_{k}\mid\bar{H}_{k})}\right)\prod_{ \ell=t}^{k-1}\frac{\mathbbm{1}\{A_{\ell}=a_{\ell}\}}{\pi_{\ell}(a_{\ell}\mid \bar{H}_{\ell})}\right.\right.$ (40) $\displaystyle\quad\left.\left.-\sum_{k=t}^{t+\tau}\mu_{k}^{\bar{b}}\left(\bar{ H}_{k}\right)\left(1-\frac{\mathbbm{1}\{A_{k}=b_{k}\}}{\pi_{k}(b_{k}\mid\bar{H }_{k})}\right)\prod_{\ell=t}^{k-1}\frac{\mathbbm{1}\{A_{\ell}=b_{\ell}\}}{\pi_ {\ell}(b_{\ell}\mid\bar{H}_{\ell})}\bigm{|}\bar{H}_{t+\tau},A_{t+\tau}\right] \dots\biggm{|}\bar{H}_{t},A_{t}\right]$ (41) $\displaystyle=\mathbb{E}\left[\dots\mathbb{E}\left[\underbrace{\left(\prod_{ \ell=t}^{t+\tau}\frac{\mathbbm{1}\{A_{\ell}=a_{\ell}\}}{\pi_{\ell}(a_{\ell} \mid\bar{H}_{\ell})}\right)\left(\mu_{t+\tau}^{\bar{A}}\left(\bar{H}_{t+\tau} \right)-\mu_{t+\tau}^{\bar{a}}\left(\bar{H}_{t+\tau}\right)\right)}_{=0}\right .\right.$ (42) $\displaystyle\quad-\underbrace{\left(\prod_{\ell=t}^{t+\tau}\frac{\mathbbm{1} \{A_{\ell}=b_{\ell}\}}{\pi_{\ell}(b_{\ell}\mid\bar{H}_{\ell})}\right)\left(\mu _{t+\tau}^{\bar{A}}\left(\bar{H}_{t+\tau}\right)-\mu_{t+\tau}^{\bar{b}}\left( \bar{H}_{t+\tau}\right)\right)}_{=0}$ (43) $\displaystyle\quad+\left(\prod_{\ell=t}^{t+\tau-1}\frac{\mathbbm{1}\{A_{\ell}= a_{\ell}\}}{\pi_{\ell}(a_{\ell}\mid\bar{H}_{\ell})}\right)\mu_{t+\tau}^{\bar{a }}\left(\bar{H}_{t+\tau}\right)-\left(\prod_{\ell=t}^{t+\tau-1}\frac{\mathbbm{ 1}\{A_{\ell}=b_{\ell}\}}{\pi_{\ell}(b_{\ell}\mid\bar{H}_{\ell})}\right)\mu_{t+ \tau}^{\bar{b}}\left(\bar{H}_{t+\tau}\right)$ (44) $\displaystyle\quad+\sum_{k=t}^{t+\tau-1}\mu_{k}^{\bar{a}}\left(\bar{H}_{k} \right)\left(1-\frac{\mathbbm{1}\{A_{k}=a_{k}\}}{\pi_{k}(a_{k}\mid\bar{H}_{k}) }\right)\prod_{\ell=t}^{k-1}\frac{\mathbbm{1}\{A_{\ell}=a_{\ell}\}}{\pi_{\ell} (a_{\ell}\mid\bar{H}_{\ell})}$ (45) $\displaystyle\quad\left.\left.-\sum_{k=t}^{t+\tau-1}\mu_{k}^{\bar{b}}\left( \bar{H}_{k}\right)\left(1-\frac{\mathbbm{1}\{A_{k}=b_{k}\}}{\pi_{k}(b_{k}\mid \bar{H}_{k})}\right)\prod_{\ell=t}^{k-1}\frac{\mathbbm{1}\{A_{\ell}=b_{\ell}\} }{\pi_{\ell}(b_{\ell}\mid\bar{H}_{\ell})}\bigm{|}\bar{H}_{t+\tau-1},A_{t+\tau- 1}\right]\dots\biggm{|}\bar{H}_{t},A_{t}\right]$ (46) $\displaystyle=\mathbb{E}\left[\dots\mathbb{E}\left[\underbrace{\left(\prod_{ \ell=t}^{t+\tau-1}\frac{\mathbbm{1}\{A_{\ell}=a_{\ell}\}}{\pi_{\ell}(a_{\ell} \mid\bar{H}_{\ell})}\right)\left(\mu_{t+\tau-1}^{\bar{A}}\left(\bar{H}_{t+\tau -1}\right)-\mu_{t+\tau-1}^{\bar{a}}\left(\bar{H}_{t+\tau-1}\right)\right)}_{=0 }\right.\right.$ (47) $\displaystyle\quad-\underbrace{\left(\prod_{\ell=t}^{t+\tau-1}\frac{\mathbbm{1 }\{A_{\ell}=b_{\ell}\}}{\pi_{\ell}(b_{\ell}\mid\bar{H}_{\ell})}\right)\left( \mu_{t+\tau-1}^{\bar{A}}\left(\bar{H}_{t+\tau-1}\right)-\mu_{t+\tau-1}^{\bar{b }}\left(\bar{H}_{t+\tau-1}\right)\right)}_{=0}$ (48) $\displaystyle\quad+\left(\prod_{\ell=t}^{t+\tau-2}\frac{\mathbbm{1}\{A_{\ell}= a_{\ell}\}}{\pi_{\ell}(a_{\ell}\mid\bar{H}_{\ell})}\right)\mu_{t+\tau-1}^{\bar {a}}\left(\bar{H}_{t+\tau-1}\right)-\left(\prod_{\ell=t}^{t+\tau-2}\frac{ \mathbbm{1}\{A_{\ell}=b_{\ell}\}}{\pi_{\ell}(b_{\ell}\mid\bar{H}_{\ell})} \right)\mu_{t+\tau-1}^{\bar{b}}\left(\bar{H}_{t+\tau-1}\right)$ (49) $\displaystyle\quad+\sum_{k=t}^{t+\tau-2}\mu_{k}^{\bar{a}}\left(\bar{H}_{k} \right)\left(1-\frac{\mathbbm{1}\{A_{k}=a_{k}\}}{\pi_{k}(a_{k}\mid\bar{H}_{k}) }\right)\prod_{\ell=t}^{k-1}\frac{\mathbbm{1}\{A_{\ell}=a_{\ell}\}}{\pi_{\ell} (a_{\ell}\mid\bar{H}_{\ell})}$ (50) $\displaystyle\quad\left.\left.-\sum_{k=t}^{t+\tau-2}\mu_{k}^{\bar{b}}\left( \bar{H}_{k}\right)\left(1-\frac{\mathbbm{1}\{A_{k}=b_{k}\}}{\pi_{k}(b_{k}\mid \bar{H}_{k})}\right)\prod_{\ell=t}^{k-1}\frac{\mathbbm{1}\{A_{\ell}=b_{\ell}\} }{\pi_{\ell}(b_{\ell}\mid\bar{H}_{\ell})}\bigm{|}\bar{H}_{t+\tau-2},A_{t+\tau- 2}\right]\dots\biggm{|}\bar{H}_{t},A_{t}\right]$ (51) $\displaystyle=\dots$ (52) $\displaystyle=\underbrace{\frac{\mathbbm{1}\{A_{t}=a_{t}\}}{\pi_{t}(a_{t}\mid \bar{H}_{t})}\left(\mu_{t}^{\bar{A}}\left(\bar{H}_{t}\right)-\mu_{t}^{\bar{a}} \left(\bar{H}_{t}\right)\right)-\frac{\mathbbm{1}\{A_{\ell}=b_{\ell}\}}{\pi_{ \ell}(b_{\ell}\mid\bar{H}_{\ell})}\left(\mu_{t+1}^{\bar{A}}\left(\bar{H}_{t} \right)-\mu_{t}^{\bar{b}}\left(\bar{H}_{t}\right)\right)}_{=0}$ (53) $\displaystyle+\quad\mu_{t}^{\bar{a}}\left(\bar{H}_{t}\right)-\mu_{t}^{\bar{b}} \left(\bar{H}_{t}\right)$ (54) $\displaystyle=\mu_{t}^{\bar{a}}\left(\bar{H}_{t}\right)-\mu_{t}^{\bar{b}}\left (\bar{H}_{t}\right),$ (55) $$

which implies

$$ $\displaystyle\quad\text{{Var}}\left(\mathbb{E}\left[\dots\mathbb{E}[Y_{\mathrm {DR}}^{\bar{a},\bar{b}}\mid\bar{H}_{t+\tau},A_{t+\tau}]\dots\bigm{|}\bar{H}_{t },A_{t}\right]\bigm{|}\bar{H}_{t}=\bar{h}_{t}\right)$ (56) $\displaystyle=\text{{Var}}\left(\mu_{t}^{\bar{a}}\left(\bar{H}_{t}\right)-\mu_ {t}^{\bar{b}}\left(\bar{H}_{t}\right)\bigm{|}\bar{H}_{t}=\bar{h}_{t}\right)=0.$ (57) $$

Furthermore,

$$ $\displaystyle\quad\text{{Var}}\left(\mathbb{E}\left[\dots\mathbb{E}[Y_{\mathrm {DR}}^{\bar{a},\bar{b}}\mid\bar{H}_{t+\tau},A_{t+\tau}]\dots\bigm{|}\bar{H}_{k +1},A_{k+1}\right]\bigm{|}\bar{H}_{k},A_{k}\right)$ (58) $\displaystyle\overset{(\ast)}{=}\text{{Var}}\left(\sum_{j=t}^{k+1}\left(\prod_ {\ell=t}^{j-1}\frac{\mathbbm{1}\{A_{\ell}=a_{\ell}\}}{\pi_{\ell}(a_{\ell}\mid \bar{H}_{\ell})}\right)\left(\mu_{j}^{\bar{a}}\left(\bar{H}_{j}\right)-\mu_{j- 1}^{\bar{a}}\left(\bar{H}_{j-1}\right)\right)\right.$ (59) $\displaystyle\quad-\left.\sum_{j=t}^{k+1}\left(\prod_{\ell=t}^{j-1}\frac{ \mathbbm{1}\{A_{\ell}=b_{\ell}\}}{\pi_{\ell}(b_{\ell}\mid\bar{H}_{\ell})} \right)\left(\mu_{j}^{\bar{b}}\left(\bar{H}_{j}\right)-\mu_{j-1}^{\bar{b}} \left(\bar{H}_{j-1}\right)\right)\bigm{|}\bar{H}_{k},A_{k}\right)$ (60) $\displaystyle=\text{{Var}}\left(\left(\prod_{\ell=t}^{k}\frac{\mathbbm{1}\{A_{ \ell}=a_{\ell}\}}{\pi_{\ell}(a_{\ell}\mid\bar{H}_{\ell})}\right)\left(\mu_{k+1 }^{\bar{a}}\left(\bar{H}_{k+1}\right)-\mu_{k}^{\bar{a}}\left(\bar{H}_{k}\right )\right)\right.$ (61) $\displaystyle\quad-\left.\left(\prod_{\ell=t}^{k}\frac{\mathbbm{1}\{A_{\ell}=b _{\ell}\}}{\pi_{\ell}(b_{\ell}\mid\bar{H}_{\ell})}\right)\left(\mu_{k+1}^{\bar {b}}\left(\bar{H}_{k+1}\right)-\mu_{k}^{\bar{b}}\left(\bar{H}_{k}\right)\right )\bigm{|}\bar{H}_{k},A_{k}\right)$ (62) $\displaystyle=\left(\prod_{\ell=t}^{k}\frac{\mathbbm{1}\{A_{\ell}=a_{\ell}\}}{ \pi^{2}_{\ell}(a_{\ell}\mid\bar{H}_{\ell})}\right)\text{{Var}}\left(\mu_{k+1}^ {\bar{a}}\left(\bar{H}_{k+1}\right)\bigm{|}\bar{H}_{k},A_{k}\right)$ (63) $\displaystyle\quad+\left(\prod_{\ell=t}^{k}\frac{\mathbbm{1}\{A_{\ell}=b_{\ell }\}}{\pi^{2}_{\ell}(b_{\ell}\mid\bar{H}_{\ell})}\right)\text{{Var}}\left(\mu_{ k+1}^{\bar{b}}\left(\bar{H}_{k+1}\right)\bigm{|}\bar{H}_{k},A_{k}\right)$ (64) $\displaystyle=\sigma^{2}\left(\prod_{\ell=t}^{k}\frac{\mathbbm{1}\{A_{\ell}=a_ {\ell}\}}{\pi^{2}_{\ell}(a_{\ell}\mid\bar{H}_{\ell})}+\prod_{\ell=t}^{k}\frac{ \mathbbm{1}\{A_{\ell}=b_{\ell}\}}{\pi^{2}_{\ell}(b_{\ell}\mid\bar{H}_{\ell})} \right),$ (65) $$

where $(\ast)$ follows from the same arguments from Eq. 39 onwards.

Hence,

$$ $\displaystyle\quad\text{{Var}}(Y_{\mathrm{DR}}^{\bar{a},\bar{b}}\mid\bar{H}_{t }=\bar{h}_{t})$ (66) $\displaystyle=\mathbb{E}\left[\sum_{\ell=t}^{t+\tau}\text{{Var}}\left(\mathbb{ E}\left[\dots\mathbb{E}[Y_{\mathrm{DR}}^{\bar{a},\bar{b}}\mid\bar{H}_{t+\tau}, A_{t+\tau}]\dots\bigm{|}\bar{H}_{k+1},A_{k+1}\right]\bigm{|}\bar{H}_{k},A_{k} \right)\bigm{|}\bar{H}_{t}=\bar{h}_{t}\right]$ (67) $\displaystyle=\sigma^{2}\mathbb{E}\left[\sum_{\ell=t}^{t+\tau}\left(\prod_{k=t }^{\ell}\frac{\mathbbm{1}\{A_{k}=a_{k}\}}{\pi^{2}_{k}(a_{k}\mid\bar{H}_{k})}+ \prod_{k=t}^{\ell}\frac{\mathbbm{1}\{A_{k}=b_{k}\}}{\pi^{2}_{k}(b_{k}\mid\bar{ H}_{k})}\right)\bigm{|}\bar{H}_{t}=\bar{h}_{t}\right]$ (68) $\displaystyle=\sigma^{2}\mathbb{E}\left[V^{\bar{a},\bar{b}}_{\mathrm{DR}}\mid \bar{H}_{t}=\bar{h}_{t}\right].$ (69) $$

∎

###### Proof.

### C.2 Proof of Theorem 2

Before proving Theorem 2, we impose additional assumptions from Kennedy [].

###### Assumption 5 (from [ 18 ] ) .

Let $\hat{\mathbb{E}}_{n}$ be a regression estimator for either the response surfaces or the pseudo-outcome regressions. Furthermore, we denote $Y_{\mathrm{tar}}$ as the target variable and $Z$ as the conditioning variables of the corresponding regression. We assume $\hat{Y}_{\mathrm{tar}}\overset{p}{\to}Y_{\mathrm{tar}}$ (consistency of the target estimator) and that any regression estimator $\hat{\mathbb{E}}_{n}$ satisfies

$$ $\frac{\hat{\mathbb{E}}_{n}[\hat{Y}_{\mathrm{tar}}\mid Z=z]-\hat{\mathbb{E}}_{n }[{Y}_{\mathrm{tar}}\mid Z=z]-\hat{\mathbb{E}}_{n}[\hat{Y}_{\mathrm{\mathrm{ tar}}}-{Y}_{\mathrm{\mathrm{tar}}}\mid Z=z]}{\sqrt{\mathbb{E}\left[\left(\hat{ \mathbb{E}}_{n}[{Y}_{\mathrm{tar}}\mid Z=z]-\mathbb{E}[{Y}_{\mathrm{tar}}\mid Z =z]\right)^{2}\right]}}\overset{p}{\to}0$ (70) $$

and

$$ $\mathbb{E}\left[\hat{\mathbb{E}}_{n}[\hat{b}(Z)\mid Z=z]^{2}\right]\lesssim \mathbb{E}\left[\hat{b}(z)^{2}\right],$ (71) $$

where $\hat{b}(z)=\mathbb{E}[\hat{Y}_{\mathrm{\mathrm{tar}}}-Y_{\mathrm{tar}}\mid Z=z]$.

Kennedy [] showed that these assumptions hold, e.g., for linear smoothers. Assumption 5 implies the following Lemma.

###### Lemma 1 .

Under Assumption 5, for any regression estimator $\hat{\mathbb{E}}_{n}$, it holds that

$$ $\mathbb{E}[(\hat{\mathbb{E}}_{n}[\hat{Y}_{\mathrm{tar}}\mid Z=z]-\mathbb{E}[Y_ {\mathrm{tar}}\mid Z=z])^{2}]\lesssim r_{\mathrm{PO}}(n)+\mathbb{E}\left[\hat{ b}(z)^{2}\right].$ (72) $$

We now proceed with the proof of Theorem 2.

###### Proof.

We prove the rates for each learner separately.

PI-HA-learner. Let us define

$$ $\widetilde{\tau}_{\mathrm{PI\mathchar 45\relax HA}}^{\bar{a},\bar{b}}(\bar{h}_ {t})=\mathbb{E}\left[Y_{t+\tau}\mid\bar{H}_{t}=\bar{h}_{t},\bar{A}_{t:t+\tau}= \bar{a}_{t:t+\tau}\right]-\mathbb{E}\left[Y_{t+\tau}\mid\bar{H}_{t}=\bar{h}_{t },\bar{A}_{t:t+\tau}=\bar{b}_{t:t+\tau}\right],$ (73) $$

which is the target estimand of the PI-HA-learner in population. We then can write

$$ $\displaystyle\quad\mathbb{E}[(\hat{\tau}_{\mathrm{PI\mathchar 45\relax HA}}^{ \bar{a},\bar{b}}(\bar{h}_{t})-\tau_{\bar{a},\bar{b}}(\bar{h}_{t}))^{2}]$ (74) $\displaystyle=\mathbb{E}\left[(\hat{\tau}_{\mathrm{PI\mathchar 45\relax HA}}^{ \bar{a},\bar{b}}(\bar{h}_{t})-\widetilde{\tau}_{\mathrm{PI\mathchar 45\relax HA }}^{\bar{a},\bar{b}}(\bar{h}_{t})+\widetilde{\tau}_{\mathrm{PI\mathchar 45 \relax HA}}^{\bar{a},\bar{b}}(\bar{h}_{t})-\tau_{\bar{a},\bar{b}}(\bar{h}_{t}) )^{2}\right]$ (75) $\displaystyle\leq\mathbb{E}\left[(\hat{\tau}_{\mathrm{PI\mathchar 45\relax HA} }^{\bar{a},\bar{b}}(\bar{h}_{t})-\widetilde{\tau}_{\mathrm{PI\mathchar 45 \relax HA}}^{\bar{a},\bar{b}}(\bar{h}_{t}))^{2}\right]+\mathrm{bias}^{2}_{\bar {a},\bar{b}}$ (76) $\displaystyle\leq\mathbb{E}\left[(\mathbb{E}\left[Y_{t+\tau}\mid\bar{H}_{t}= \bar{h}_{t},\bar{A}_{t:t+\tau}=\bar{a}_{t:t+\tau}\right]-\hat{\mathbb{E}}_{n} \left[Y_{t+\tau}\mid\bar{H}_{t}=\bar{h}_{t},\bar{A}_{t:t+\tau}=\bar{a}_{t:t+ \tau}\right])^{2}\right]$ (77) $\displaystyle\quad+\mathbb{E}\left[(\mathbb{E}\left[Y_{t+\tau}\mid\bar{H}_{t}= \bar{h}_{t},\bar{A}_{t:t+\tau}=\bar{b}_{t:t+\tau}\right]-\hat{\mathbb{E}}_{n} \left[Y_{t+\tau}\mid\bar{H}_{t}=\bar{h}_{t},\bar{A}_{t:t+\tau}=\bar{b}_{t:t+ \tau}\right])^{2}\right]+\mathrm{bias}^{2}_{\bar{a},\bar{b}}$ (78) $\displaystyle\lesssim r^{\bar{a}}_{\mathrm{PI\mathchar 45\relax HA}}(n)+r^{ \bar{b}}_{\mathrm{PI\mathchar 45\relax HA}}(n)+\mathrm{bias}^{2}_{\bar{a},\bar {b}},$ (79) $$

which proves the rate for the PI-HA-learner.

PI-HA-learner. By applying Lemma 1 iteratively, we obtain

$$ $\displaystyle\mathbb{E}\left[\left(\hat{\mu}_{t}^{\bar{a}}(\bar{h}_{t})-\mu_{t }^{\bar{a}}(\bar{h}_{t})\right)^{2}\right]$ $\displaystyle\lesssim r^{\bar{a}}_{\mu_{t}}(n)+\mathbb{E}\left[\mathbb{E}\left [\hat{\mu}_{t+1}^{\bar{a}}(\bar{H}_{t+1})-\mu_{t+1}^{\bar{a}}(\bar{H}_{t+1}) \mid\bar{H}_{t}=\bar{h}_{t}\right]^{2}\right]$ (80) $\displaystyle\leq r^{\bar{a}}_{\mu_{t}}(n)+\mathbb{E}\left[\mathbb{E}\left[ \left(\hat{\mu}_{t+1}^{\bar{a}}(\bar{H}_{t+1})-\mu_{t+1}^{\bar{a}}(\bar{H}_{t+ 1})\right)^{2}\right]\mid\bar{H}_{t}=\bar{h}_{t}\right]$ (81) $\displaystyle\lesssim\dots$ (82) $\displaystyle\lesssim\sum_{\ell=t}^{t+\tau}r^{\bar{a}}_{\mu_{\ell}}(n).$ (83) $$

The result for CATE follows by noting that

$$ $\mathbb{E}[(\hat{\tau}_{\mathrm{PI\mathchar 45\relax RA}}^{\bar{a},\bar{b}}( \bar{h}_{t})-\tau_{\bar{a},\bar{b}}(\bar{h}_{t}))^{2}]\leq\mathbb{E}\left[ \left(\hat{\mu}_{t}^{\bar{a}}(\bar{h}_{t})-\mu_{t}^{\bar{a}}(\bar{h}_{t}) \right)^{2}\right]+\mathbb{E}\left[\left(\hat{\mu}_{t}^{\bar{b}}(\bar{h}_{t})- \mu_{t}^{\bar{b}}(\bar{h}_{t})\right)^{2}\right].$ (84) $$

RA-learner. We start by deriving a formula for the bias term via

$$ $\displaystyle\hat{b}_{\mathrm{RA}}(\bar{h}_{t})$ $\displaystyle=\pi_{t}(a_{t}\mid\bar{h}_{t})\left(\mathbb{E}\left[\hat{\mu}_{t+ 1}^{\bar{a}}\left(\bar{H}_{t+1}\right)\mid\bar{H}_{t}=\bar{h}_{t}\right]-\mu_{ t}^{\bar{a}}\left(\bar{h}_{t}\right)+\mu_{t}^{\bar{b}}\left(\bar{h}_{t}\right) -\hat{\mu}_{t}^{\bar{b}}\left(\bar{h}_{t}\right)\right)$ (85) $\displaystyle\quad+\pi_{t}(b_{t}\mid\bar{h}_{t})\left(\hat{\mu}_{t}^{\bar{a}} \left(\bar{h}_{t}\right)-\mu_{t}^{\bar{a}}\left(\bar{h}_{t}\right)+\mu_{t}^{ \bar{b}}\left(\bar{h}_{t}\right)-\mathbb{E}\left[\hat{\mu}_{t+1}^{\bar{b}} \left(\bar{H}_{t+1}\right)\mid\bar{H}_{t}=\bar{h}_{t}\right]\right)$ (86) $\displaystyle\quad+\left(1-\pi_{t}(a_{t}\mid\bar{h}_{t})-\pi_{t}(b_{t}\mid\bar {h}_{t})\right)\left(\hat{\mu}_{t}^{\bar{a}}\left(\bar{h}_{t}\right)-\mu_{t}^{ \bar{a}}\left(\bar{h}_{t}\right)+\mu_{t}^{\bar{b}}\left(\bar{h}_{t}\right)- \hat{\mu}_{t}^{\bar{b}}\left(\bar{h}_{t}\right)\right).$ (87) $$

Hence, we can apply Lemma 1 and obtain

$$ $\displaystyle\mathbb{E}[(\hat{\tau}_{\mathrm{RA}}^{\bar{a},\bar{b}}(\bar{h}_{t })-\tau_{\bar{a},\bar{b}}(\bar{h}_{t}))^{2}]$ $\displaystyle\lesssim r^{\bar{a},\bar{b}}_{\mathrm{PO}}(n)+\mathbb{E}\left[ \hat{b}_{\mathrm{RA}}(\bar{h}_{t})^{2}\right]$ (88) $\displaystyle\lesssim r^{\bar{a},\bar{b}}_{\mathrm{PO}}(n)+\pi^{2}_{t}(a_{t} \mid\bar{h}_{t})\left(\sum_{\ell=t+1}^{t+\tau}r^{\bar{a}}_{\mu_{\ell}}(n)+\sum _{\ell=t}^{t+\tau}r^{\bar{b}}_{\mu_{\ell}}(n)\right)$ (89) $\displaystyle\quad+\pi^{2}_{t}(b_{t}\mid\bar{h}_{t})\left(\sum_{\ell=t}^{t+ \tau}r^{\bar{a}}_{\mu_{\ell}}(n)+\sum_{\ell=t+1}^{t+\tau}r^{\bar{b}}_{\mu_{ \ell}}(n)\right)$ (90) $\displaystyle\quad+\left(1-\pi_{t}(a_{t}\mid\bar{h}_{t})-\pi_{t}(b_{t}\mid\bar {h}_{t})\right)^{2}\left(\sum_{\ell=t}^{t+\tau}r^{\bar{a}}_{\mu_{\ell}}(n)+ \sum_{\ell=t+1}^{t+\tau}r^{\bar{b}}_{\mu_{\ell}}(n)\right)$ (91) $\displaystyle\lesssim r^{\bar{a},\bar{b}}_{\mathrm{PO}}(n)+\sum_{\ell=t}^{t+ \tau}r^{\bar{a}}_{\mu_{\ell}}(n)+\sum_{\ell=t+1}^{t+\tau}r^{\bar{b}}_{\mu_{ \ell}}(n).$ (92) $$

IPW learner. The bias term for the IPW-learner can be written as

$$ $\displaystyle\quad{\hat{b}_{\mathrm{IPW}}}^{\bar{a}}(\bar{h}_{t})$ (93) $\displaystyle=\mathbb{E}\left[\left(\prod_{\ell=t}^{t+\tau}\frac{\mathbbm{1}\{ A_{\ell}=a_{\ell}\}}{\hat{\pi}_{\ell}(a_{\ell}\mid\bar{H}_{\ell})}-\prod_{\ell =t}^{t+\tau}\frac{\mathbbm{1}\{A_{\ell}=a_{\ell}\}}{{\pi}_{\ell}(a_{\ell}\mid \bar{H}_{\ell})}\right)Y_{t+\tau}\biggm{|}\bar{H}_{t}=\bar{h}_{t}\right]$ (94) $\displaystyle=\mathbb{E}\left[\dots\mathbb{E}\left[\left(\prod_{\ell=t}^{t+ \tau}\frac{\mathbbm{1}\{A_{\ell}=a_{\ell}\}}{\hat{\pi}_{\ell}(a_{\ell}\mid\bar {H}_{\ell})}-\prod_{\ell=t}^{t+\tau}\frac{\mathbbm{1}\{A_{\ell}=a_{\ell}\}}{{ \pi}_{\ell}(a_{\ell}\mid\bar{H}_{\ell})}\right)Y_{t+\tau}\bigm{|}\bar{H}_{t+ \tau}\right]\dots\biggm{|}\bar{H}_{t}=\bar{h}_{t}\right]$ (95) $\displaystyle=\mathbb{E}\left[\dots\mathbb{E}\left[\left(\prod_{\ell=t}^{t+ \tau-1}\frac{\mathbbm{1}\{A_{\ell}=a_{\ell}\}}{\hat{\pi}_{\ell}(a_{\ell}\mid \bar{H}_{\ell})}\frac{{\pi}_{t+\tau}(a_{t+\tau}\mid\bar{H}_{t+\tau})}{\hat{\pi }_{t+\tau}(a_{t+\tau}\mid\bar{H}_{t+\tau})}\right.\right.\right.$ (96) $\displaystyle\quad\left.\left.\left.-\prod_{\ell=t}^{t+\tau-1}\frac{\mathbbm{1 }\{A_{\ell}=a_{\ell}\}}{{\pi}_{\ell}(a_{\ell}\mid\bar{H}_{\ell})}\frac{{\pi}_{ t+\tau}(a_{t+\tau}\mid\bar{H}_{t+\tau})}{{\pi}_{t+\tau}(a_{t+\tau}\mid\bar{H}_ {t+\tau})}\right)\mu_{t+\tau}^{\bar{a}}\left(\bar{H}_{t+\tau}\right)\biggm{|} \bar{H}_{t+\tau-1}\right]\dots\biggm{|}\bar{H}_{t}=\bar{h}_{t}\right]$ (97) $\displaystyle=\mathbb{E}\left[\dots\mathbb{E}\left[\left(\prod_{\ell=t}^{t+ \tau-1}\frac{\mathbbm{1}\{A_{\ell}=a_{\ell}\}}{\hat{\pi}_{\ell}(a_{\ell}\mid \bar{H}_{\ell})}\frac{{\pi}_{t+\tau}(a_{t+\tau}\mid\bar{H}_{t+\tau})}{\hat{\pi }_{t+\tau}(a_{t+\tau}\mid\bar{H}_{t+\tau})}\right.\right.\right.$ (98) $\displaystyle\quad-\prod_{\ell=t}^{t+\tau-1}\frac{\mathbbm{1}\{A_{\ell}=a_{ \ell}\}}{\hat{\pi}_{\ell}(a_{\ell}\mid\bar{H}_{\ell})}\frac{{\pi}_{t+\tau}(a_{ t+\tau}\mid\bar{H}_{t+\tau})}{{\pi}_{t+\tau}(a_{t+\tau}\mid\bar{H}_{t+\tau})}+ \prod_{\ell=t}^{t+\tau-1}\frac{\mathbbm{1}\{A_{\ell}=a_{\ell}\}}{\hat{\pi}_{ \ell}(a_{\ell}\mid\bar{H}_{\ell})}\frac{{\pi}_{t+\tau}(a_{t+\tau}\mid\bar{H}_{ t+\tau})}{{\pi}_{t+\tau}(a_{t+\tau}\mid\bar{H}_{t+\tau})}$ (99) $\displaystyle\quad\left.\left.\left.-\prod_{\ell=t}^{t+\tau-1}\frac{\mathbbm{1 }\{A_{\ell}=a_{\ell}\}}{{\pi}_{\ell}(a_{\ell}\mid\bar{H}_{\ell})}\frac{{\pi}_{ t+\tau}(a_{t+\tau}\mid\bar{H}_{t+\tau})}{{\pi}_{t+\tau}(a_{t+\tau}\mid\bar{H}_ {t+\tau})}\right)\mu_{t+\tau}^{\bar{a}}\left(\bar{H}_{t+\tau}\right)\bigm{|} \bar{H}_{t+\tau-1}\right]\dots\biggm{|}\bar{H}_{t}=\bar{h}_{t}\right]$ (100) $\displaystyle=\mathbb{E}\left[\prod_{\ell=t}^{t+\tau-1}\frac{\mathbbm{1}\{A_{ \ell}=a_{\ell}\}}{\hat{\pi}_{\ell}(a_{\ell}\mid\bar{H}_{\ell})}\frac{{\pi}_{t+ \tau}(a_{t+\tau}\mid\bar{H}_{t+\tau})-\hat{\pi}_{t+\tau}(a_{t+\tau}\mid\bar{H} _{t+\tau})}{\hat{\pi}_{t+\tau}(a_{t+\tau}\mid\bar{H}_{t+\tau})}\mu_{t+\tau}^{ \bar{a}}\left(\bar{H}_{t+\tau}\right)\bigm{|}\bar{H}_{t}=\bar{h}_{t}\right]$ (101) $\displaystyle\quad+\mathbb{E}\left[\dots\mathbb{E}\left[\left(\prod_{\ell=t}^{ t+\tau-1}\frac{\mathbbm{1}\{A_{\ell}=a_{\ell}\}}{\hat{\pi}_{\ell}(a_{\ell}\mid \bar{H}_{\ell})}-\prod_{\ell=t}^{t+\tau-1}\frac{\mathbbm{1}\{A_{\ell}=a_{\ell} \}}{{\pi}_{\ell}(a_{\ell}\mid\bar{H}_{\ell})}\right)\mu_{t+\tau}^{\bar{a}} \left(\bar{H}_{t+\tau}\right)\bigm{|}\bar{H}_{t+\tau-1}\right]\dots\biggm{|} \bar{H}_{t}=\bar{h}_{t}\right]$ (102) $\displaystyle=\dots$ (103) $\displaystyle=\sum_{k=t}^{t+\tau}\mathbb{E}\left[\prod_{\ell=t}^{k-1}\frac{ \mathbbm{1}\{A_{\ell}=a_{\ell}\}}{\hat{\pi}_{\ell}(a_{\ell}\mid\bar{H}_{\ell}) }\frac{{\pi}_{k}(a_{k}\mid\bar{H}_{k})-\hat{\pi}_{k}(a_{k}\mid\bar{H}_{k})}{ \hat{\pi}_{k}(a_{k}\mid\bar{H}_{k})}\mu_{k}^{\bar{a}}\left(\bar{H}_{k}\right) \bigm{|}\bar{H}_{t}=\bar{h}_{t}\right].$ (104) $$

Hence, Lemma 1 yields

$$ $\displaystyle\mathbb{E}\left[{\hat{b}_{\mathrm{IPW}}}^{\bar{a}}(\bar{h}_{t})^{ 2}\right]$ $\displaystyle\overset{(\ast)}{\leq}C\,\mathbb{E}\left[\sum_{k=t}^{t+\tau} \mathbb{E}\left[\left({\pi}_{k}(a_{k}\mid\bar{H}_{k})-\hat{\pi}_{k}(a_{k}\mid \bar{H}_{k})\right)^{2}\bigm{|}\bar{H}_{t}=\bar{h}_{t}\right]\right]$ (105) $\displaystyle\overset{(\ast\ast)}{=}C\,\sum_{k=t}^{t+\tau}\mathbb{E}\left[ \mathbb{E}\left[\left({\pi}_{k}(a_{k}\mid\bar{H}_{k})-\hat{\pi}_{k}(a_{k}\mid \bar{H}_{k})\right)^{2}\right]\bigm{|}\bar{H}_{t}=\bar{h}_{t}\right]$ (106) $\displaystyle\lesssim\sum_{\ell=t}^{t+\tau}r^{\bar{a}}_{\pi_{\ell}}(n),$ (107) $$

where $(\ast)$ follows from the boundedness assumptions (2) and $(\ast\ast)$ from applying Fubini’s theorem. We obtain the rate for CATE via

$$ $\displaystyle\mathbb{E}[(\hat{\tau}_{\mathrm{IPW}}^{\bar{a},\bar{b}}(\bar{h}_{ t})-\tau_{\bar{a},\bar{b}}(\bar{h}_{t}))^{2}]$ $\displaystyle\lesssim r^{\bar{a},\bar{b}}_{\mathrm{PO}}(n)+\mathbb{E}\left[{ \hat{b}_{\mathrm{IPW}}}^{\bar{a}}(\bar{h}_{t})^{2}\right]+\mathbb{E}\left[{ \hat{b}_{\mathrm{IPW}}}^{\bar{b}}(\bar{h}_{t})^{2}\right]$ (108) $\displaystyle\lesssim r^{\bar{a},\bar{b}}_{\mathrm{PO}}(n)+\max_{\ell\in\{t, \dots,t+\tau\}}r^{\bar{a}}_{\pi_{\ell}}(n)+\max_{\ell\in\{t,\dots,t+\tau\}}r^{ \bar{b}}_{\pi_{\ell}}(n).$ (109) $$

DR learner. We derive the bias term for the DR-learner via

$$ $\displaystyle\quad{\hat{b}_{\mathrm{DR}}}^{\bar{a}}(\bar{h}_{t})$ (110) $\displaystyle=\mathbb{E}\left[\dots\mathbb{E}\left[\left(\prod_{\ell=t}^{t+ \tau}\frac{\mathbbm{1}\{A_{\ell}=a_{\ell}\}}{\hat{\pi}_{\ell}(a_{\ell}\mid\bar {H}_{\ell})}-\frac{\mathbbm{1}\{A_{\ell}=a_{\ell}\}}{{\pi}_{\ell}(a_{\ell}\mid \bar{H}_{\ell})}\right)Y_{t+\tau}\right.\right.$ (111) $\displaystyle\quad+\sum_{k=t}^{t+\tau}\hat{\mu}_{k}^{\bar{a}}\left(\bar{H}_{k} \right)\left(1-\frac{\mathbbm{1}\{A_{k}=a_{k}\}}{\hat{\pi}_{k}(a_{k}\mid\bar{H }_{k})}\right)\prod_{\ell=t}^{k-1}\frac{\mathbbm{1}\{A_{\ell}=a_{\ell}\}}{\hat {\pi}_{\ell}(a_{\ell}\mid\bar{H}_{\ell})}$ (112) $\displaystyle\quad\left.\left.-\sum_{k=t}^{t+\tau}{\mu}_{k}^{\bar{a}}\left( \bar{H}_{k}\right)\left(1-\frac{\mathbbm{1}\{A_{k}=a_{k}\}}{{\pi}_{k}(a_{k} \mid\bar{H}_{k})}\right)\prod_{\ell=t}^{k-1}\frac{\mathbbm{1}\{A_{\ell}=a_{ \ell}\}}{{\pi}_{\ell}(a_{\ell}\mid\bar{H}_{\ell})}\biggm{|}\bar{H}_{t+\tau} \right]\dots\bigm{|}\bar{H}_{t}=\bar{h}_{t}\right]$ (113) $\displaystyle\overset{(\ast)}{=}\mathbb{E}\left[\dots\mathbb{E}\left[\sum_{k=t }^{t+\tau}\prod_{\ell=t}^{k-1}\frac{\mathbbm{1}\{A_{\ell}=a_{\ell}\}}{\hat{\pi }_{\ell}(a_{\ell}\mid\bar{H}_{\ell})}\frac{{\pi}_{k}(a_{k}\mid\bar{H}_{k})- \hat{\pi}_{k}(a_{k}\mid\bar{H}_{k})}{\hat{\pi}_{k}(a_{k}\mid\bar{H}_{k})}\mu_{ k}^{\bar{a}}\left(\bar{H}_{k}\right)\right.\right.$ (114) $\displaystyle\quad+\sum_{k=t}^{t+\tau}\hat{\mu}_{k}^{\bar{a}}\left(\bar{H}_{k} \right)\left(1-\frac{\mathbbm{1}\{A_{k}=a_{k}\}}{\hat{\pi}_{k}(a_{k}\mid\bar{H }_{k})}\right)\prod_{\ell=t}^{k-1}\frac{\mathbbm{1}\{A_{\ell}=a_{\ell}\}}{\hat {\pi}_{\ell}(a_{\ell}\mid\bar{H}_{\ell})}$ (115) $\displaystyle\quad\left.\left.-\sum_{k=t}^{t+\tau}{\mu}_{k}^{\bar{a}}\left( \bar{H}_{k}\right)\left(1-\frac{\mathbbm{1}\{A_{k}=a_{k}\}}{{\pi}_{k}(a_{k} \mid\bar{H}_{k})}\right)\prod_{\ell=t}^{k-1}\frac{\mathbbm{1}\{A_{\ell}=a_{ \ell}\}}{{\pi}_{\ell}(a_{\ell}\mid\bar{H}_{\ell})}\bigm{|}\bar{H}_{t+\tau} \right]\dots\biggm{|}\bar{H}_{t}=\bar{h}_{t}\right]$ (116) $\displaystyle=\mathbb{E}\left[\dots\mathbb{E}\left[\prod_{\ell=t}^{{t+\tau}-1} \frac{\mathbbm{1}\{A_{\ell}=a_{\ell}\}}{\hat{\pi}_{\ell}(a_{\ell}\mid\bar{H}_{ \ell})}\frac{{\pi}_{t+\tau}(a_{t+\tau}\mid\bar{H}_{t+\tau})-\hat{\pi}_{t+\tau} (a_{t+\tau}\mid\bar{H}_{t+\tau})}{\hat{\pi}_{t+\tau}(a_{t+\tau}\mid\bar{H}_{t+ \tau})}\mu_{t+\tau}^{\bar{a}}\left(\bar{H}_{t+\tau}\right)\right.\right.$ (117) $\displaystyle\quad+\hat{\mu}_{t+\tau}^{\bar{a}}\left(\bar{H}_{t+\tau}\right) \left(1-\frac{{\pi}_{t+\tau}(a_{t+\tau}\mid\bar{H}_{t+\tau})}{\hat{\pi}_{t+ \tau}(a_{t+\tau}\mid\bar{H}_{t+\tau})}\right)\prod_{\ell=t}^{{t+\tau}-1}\frac{ \mathbbm{1}\{A_{\ell}=a_{\ell}\}}{\hat{\pi}_{\ell}(a_{\ell}\mid\bar{H}_{\ell})}$ (118) $\displaystyle\quad\left.\left.-\underbrace{{\mu}_{t+\tau}^{\bar{a}}\left(\bar{ H}_{t+\tau}\right)\left(1-\frac{{\pi}_{t+\tau}(a_{t+\tau}\mid\bar{H}_{t+\tau}) }{{\pi}_{t+\tau}(a_{t+\tau}\mid\bar{H}_{t+\tau})}\right)\prod_{\ell=t}^{{t+ \tau}-1}\frac{\mathbbm{1}\{A_{\ell}=a_{\ell}\}}{{\pi}_{\ell}(a_{\ell}\mid\bar{ H}_{\ell})}}_{=0}\bigm{|}\bar{H}_{t+\tau-1}\right]\dots\biggm{|}\bar{H}_{t}= \bar{h}_{t}\right]$ (119) $\displaystyle\quad+\mathbb{E}\left[\dots\mathbb{E}\left[\sum_{k=t}^{t+\tau-1} \prod_{\ell=t}^{k-1}\frac{\mathbbm{1}\{A_{\ell}=a_{\ell}\}}{\hat{\pi}_{\ell}(a _{\ell}\mid\bar{H}_{\ell})}\frac{{\pi}_{k}(a_{k}\mid\bar{H}_{k})-\hat{\pi}_{k} (a_{k}\mid\bar{H}_{k})}{\hat{\pi}_{k}(a_{k}\mid\bar{H}_{k})}\mu_{k}^{\bar{a}} \left(\bar{H}_{k}\right)\right.\right.$ (120) $\displaystyle\quad+\sum_{k=t}^{t+\tau-1}\hat{\mu}_{k}^{\bar{a}}\left(\bar{H}_{ k}\right)\left(1-\frac{\mathbbm{1}\{A_{k}=a_{k}\}}{\hat{\pi}_{k}(a_{k}\mid\bar {H}_{k})}\right)\prod_{\ell=t}^{k-1}\frac{\mathbbm{1}\{A_{\ell}=a_{\ell}\}}{ \hat{\pi}_{\ell}(a_{\ell}\mid\bar{H}_{\ell})}$ (121) $\displaystyle\quad\left.\left.-\sum_{k=t}^{t+\tau-1}{\mu}_{k}^{\bar{a}}\left( \bar{H}_{k}\right)\left(1-\frac{\mathbbm{1}\{A_{k}=a_{k}\}}{{\pi}_{k}(a_{k} \mid\bar{H}_{k})}\right)\prod_{\ell=t}^{k-1}\frac{\mathbbm{1}\{A_{\ell}=a_{ \ell}\}}{{\pi}_{\ell}(a_{\ell}\mid\bar{H}_{\ell})}\bigm{|}\bar{H}_{t+\tau-1} \right]\dots\biggm{|}\bar{H}_{t}=\bar{h}_{t}\right]$ (122) $\displaystyle=\mathbb{E}\left[\prod_{\ell=t}^{{t+\tau}-1}\frac{\mathbbm{1}\{A_ {\ell}=a_{\ell}\}}{\hat{\pi}_{\ell}(a_{\ell}\mid\bar{H}_{\ell})}\frac{{\pi}_{t +\tau}(a_{t+\tau}\mid\bar{H}_{t+\tau})-\hat{\pi}_{t+\tau}(a_{t+\tau}\mid\bar{H }_{t+\tau})}{\hat{\pi}_{t+\tau}(a_{t+\tau}\mid\bar{H}_{t+\tau})}\right.$ (123) $\displaystyle\quad\left.\left(\hat{\mu}_{t+\tau}^{\bar{a}}\left(\bar{H}_{t+ \tau}\right)-{\mu}_{t+\tau}^{\bar{a}}\left(\bar{H}_{t+\tau}\right)\right)\bigm {|}\bar{H}_{t}=\bar{h}_{t}\right]$ (124) $\displaystyle\quad+\mathbb{E}\left[\dots\mathbb{E}\left[\sum_{k=t}^{t+\tau-1} \prod_{\ell=t}^{k-1}\frac{\mathbbm{1}\{A_{\ell}=a_{\ell}\}}{\hat{\pi}_{\ell}(a _{\ell}\mid\bar{H}_{\ell})}\frac{{\pi}_{k}(a_{k}\mid\bar{H}_{k})-\hat{\pi}_{k} (a_{k}\mid\bar{H}_{k})}{\hat{\pi}_{k}(a_{k}\mid\bar{H}_{k})}\mu_{k}^{\bar{a}} \left(\bar{H}_{k}\right)\right.\right.$ (125) $\displaystyle\quad+\sum_{k=t}^{t+\tau-1}\hat{\mu}_{k}^{\bar{a}}\left(\bar{H}_{ k}\right)\left(1-\frac{\mathbbm{1}\{A_{k}=a_{k}\}}{\hat{\pi}_{k}(a_{k}\mid\bar {H}_{k})}\right)\prod_{\ell=t}^{k-1}\frac{\mathbbm{1}\{A_{\ell}=a_{\ell}\}}{ \hat{\pi}_{\ell}(a_{\ell}\mid\bar{H}_{\ell})}$ (126) $\displaystyle\quad\left.\left.-\sum_{k=t}^{t+\tau-1}{\mu}_{k}^{\bar{a}}\left( \bar{H}_{k}\right)\left(1-\frac{\mathbbm{1}\{A_{k}=a_{k}\}}{{\pi}_{k}(a_{k} \mid\bar{H}_{k})}\right)\prod_{\ell=t}^{k-1}\frac{\mathbbm{1}\{A_{\ell}=a_{ \ell}\}}{{\pi}_{\ell}(a_{\ell}\mid\bar{H}_{\ell})}\bigm{|}\bar{H}_{t+\tau-1} \right]\dots\biggm{|}\bar{H}_{t}=\bar{h}_{t}\right]$ (127) $\displaystyle=\dots$ (128) $\displaystyle=\sum_{k=t}^{t+\tau}\mathbb{E}\left[\prod_{\ell=t}^{k-1}\frac{ \mathbbm{1}\{A_{\ell}=a_{\ell}\}}{\hat{\pi}_{\ell}(a_{\ell}\mid\bar{H}_{\ell}) }\frac{{\pi}_{k}(a_{k}\mid\bar{H}_{k})-\hat{\pi}_{k}(a_{k}\mid\bar{H}_{k})}{ \hat{\pi}_{k}(a_{k}\mid\bar{H}_{k})}\left(\hat{\mu}_{k}^{\bar{a}}\left(\bar{H} _{k}\right)-{\mu}_{k}^{\bar{a}}\left(\bar{H}_{k}\right)\right)\biggm{|}\bar{H} _{t}=\bar{h}_{t}\right],$ (129) $$

where $(\ast)$ follows from the bias formula of the IPW-learner.

Hence,

$$ $\displaystyle\quad\mathbb{E}\left[{\hat{b}_{\mathrm{DR}}}^{\bar{a}}(\bar{h}_{t })^{2}\right]$ (130) $\displaystyle\overset{(\ast)}{\leq}C\,\mathbb{E}\left[\sum_{k=t}^{t+\tau} \mathbb{E}\left[\left({\pi}_{k}(a_{k}\mid\bar{H}_{k})-\hat{\pi}_{k}(a_{k}\mid \bar{H}_{k})\right)^{2}\left(\hat{\mu}_{k}^{\bar{a}}\left(\bar{H}_{k}\right)-{ \mu}_{k}^{\bar{a}}\left(\bar{H}_{k}\right)\right)^{2}\bigm{|}\bar{H}_{t}=\bar{ h}_{t}\right]\right]$ (131) $\displaystyle\overset{(\ast\ast)}{=}C\,\sum_{k=t}^{t+\tau}\mathbb{E}\left[ \mathbb{E}\left[\left({\pi}_{k}(a_{k}\mid\bar{H}_{k})-\hat{\pi}_{k}(a_{k}\mid \bar{H}_{k})\right)^{2}\right]\mathbb{E}\left[\left(\hat{\mu}_{k}^{\bar{a}} \left(\bar{H}_{k}\right)-{\mu}_{k}^{\bar{a}}\left(\bar{H}_{k}\right)\right)^{2 }\right]\bigm{|}\bar{H}_{t}=\bar{h}_{t}\right]$ (132) $\displaystyle\lesssim\sum_{\ell=t}^{t+\tau}r^{\bar{a}}_{\pi_{\ell}}(n)\left( \sum_{k=\ell}^{t+\tau}r^{\bar{a}}_{\mu_{k}}(n)\right),$ (133) $$

where $(\ast)$ follows from the boundedness assumption (2) and $(\ast\ast)$ follows from Fubini’s theorem and the sample splitting assumption (4). For CATE, we obtain the corresponding rate via

$$ $\displaystyle\mathbb{E}[(\hat{\tau}_{\mathrm{DR}}^{\bar{a},\bar{b}}(\bar{h}_{t })-\tau_{\bar{a},\bar{b}}(\bar{h}_{t}))^{2}]$ $\displaystyle\lesssim r^{\bar{a},\bar{b}}_{\mathrm{PO}}(n)+\mathbb{E}\left[{ \hat{b}_{\mathrm{DR}}}^{\bar{a}}(\bar{h}_{t})^{2}\right]+\mathbb{E}\left[{\hat {b}_{\mathrm{DR}}}^{\bar{b}}(\bar{h}_{t})^{2}\right]$ (134) $\displaystyle\lesssim r^{\bar{a},\bar{b}}_{\mathrm{PO}}(n)+\sum_{\ell=t}^{t+ \tau}\left(r^{\bar{a}}_{\pi_{\ell}}(n)\sum_{k=\ell}^{t+\tau}r^{\bar{a}}_{\mu_{ k}}(n)+r^{\bar{b}}_{\pi_{\ell}}(n)\sum_{k=\ell}^{t+\tau}r^{\bar{b}}_{\mu_{k}}( n)\right).$ (135) $$

∎

###### Assumption 5 (from [ 18 ] ) .

###### Lemma 1 .

###### Proof.

## Appendix D Implementation and hyperparameter details

Transformer architecture. We build on an encoder transformer as in []. Each transformer consists of a single transformer block and employs a causal mask to avoid look-ahead bias. First, the input is passed through a linear input layer. In order to ensure that the sequence order is preserved, we apply a non-trainable positional encoding. Then, the history is passed through the transformer blocks. Each block consists of (i) a self-attention mechanism with three attention heads and hidden state dimension $d_{\text{model}}=30$, (ii) and a feed-forward network with hidden layer size $d_{\text{ff}}=20$. Both the (i) self-attention mechanism and (ii) the feed-forward network employ residual connections, which are followed by dropout layers with dropout probabilities $p=0.1$, respectively. For regularization, we apply post-normalization as in [], that is, layer normalization after each residual connection. Finally, we send the learned representation through a feed-forward neural network with one hidden layer of size $d_{\text{ff}}^{\prime}=20$, ReLU nonlinearities, and with linear or softmax activation for regression and classification tasks, respectively.

Hyperparameters. To ensure a fair comparison, we choose the same hyperparameters (e.g., dimensions, learning rate) for all nuisance estimators and meta-learners. We employ additional weight decay for the two-stage learners to avoid overfitting during the pseudo-outcome regression (as the CATE is of simpler structure than the nuisance parameters). This is consistent with the literature on meta-learners for CATE estimation []. For reproducibility purposes, we report all hyperparameters as .yaml files.(^2^22Code is available at [https://anonymous.4open.science/r/CATEMetaLearnersTime-A35B](https://anonymous.4open.science/r/CATEMetaLearnersTime-A35B).)

Runtime. For each transformer-based learner, training took approximately 90 seconds using $n=5000$ samples and a standard computer with AMD Ryzen 7 Pro CPU and 32GB of RAM.

## Appendix E Details regarding experiments on simulated data

Data-generating process. Our general data-generating process for simulating datasets is as follows: we start by simulating an initial confounder $X_{1}\sim\mathcal{N}(0,1)$ from a standard normal distribution. For the following time steps $t\in\{2,\dots,5\}$, we simulate time-varying confounders via

$$ $X_{t}=f_{x}(Y_{t-1},A_{t-1},\bar{H}_{t-1})+\varepsilon_{x},$ (136) $$

for some function $f_{x}$ and where $\varepsilon_{x}\sim\mathcal{N}(0,0.5)$. Then, we simulate binary, time-varying treatments via

$$ $A_{t}=x\sim\textrm{Bernoulli}\left(p_{t}\right)\quad\text{with}\quad p_{t}= \sigma(f_{a}(\bar{H}_{t}))$ (137) $$

for some function $f_{a}$ and where $\sigma(\cdot)$ denotes the sigmoid function. Finally, we simulate time-varying, continuous outcomes via

$$ $Y=f_{y}(A_{t},\bar{H}_{t})+\varepsilon_{y},$ (138) $$

where $\varepsilon_{y}\sim\mathcal{N}(0,0.3)$.

$\bullet$ Dataset $\mathcal{D}_{1}$. For $\mathcal{D}_{1}$, we set the confounding function to $f_{x}(Y_{t-1},A_{t-1},\bar{H}_{t-1})=0.5X_{t-1}$, the treatment assignment function to $f_{a}(\bar{H}_{t})=4\cos(0.5X_{t}-0.5(A_{t-1}-0.5))$, and the outcome assignment function to $f_{y}(A_{t},\bar{H}_{t})=\cos(X_{t})+0.5(A_{t}-0.5)$. We sample a training dataset of size $n_{\text{train}}=5000$ and a test dataset of size $n_{\text{test}}=1000$.

$\bullet$ Dataset $\mathcal{D}_{2}$. For $\mathcal{D}_{2}$, we set the confounding function to $f_{x}(Y_{t-1},A_{t-1},\bar{H}_{t-1})=0.5X_{t-1}$, the treatment assignment function to $f_{a}(\bar{H}_{t})=0.5X_{t}-0.5(A_{t-1}-0.5)$, and the outcome assignment function to $f_{y}(A_{t},\bar{H}_{t})=\cos(5X_{t})+0.5(A_{t}-0.5)$. We sample a training dataset of size $n_{\text{train}}=10000$ and a test dataset of size $n_{\text{test}}=1000$.

$\bullet$ Dataset $\mathcal{D}_{3}$. For $\mathcal{D}_{3}$, we set the confounding function to $f_{x}(Y_{t-1},A_{t-1},\bar{H}_{t-1})=0.5X_{t-1}$, the treatment assignment function to $f_{a}(\bar{H}_{t})=\gamma(0.5X_{t}-0.5(A_{t-1}-0.5))$, where $\gamma$ is a parameter controlling the overlap, and the outcome assignment function to $f_{y}(A_{t},\bar{H}_{t})=\cos(X_{t})+0.5(A_{t}-0.5)$. We sample a training dataset of size $n_{\text{train}}=5000$ and a test dataset of size $n_{\text{test}}=1000$.

Interventions. For $\mathcal{D}_{1}$ and $\mathcal{D}_{2}$ we set $\tau\in\{0,1,2\}$ and choose the corresponding treatment interventions $a_{\tau}$ and $b_{\tau}$ as follows: $a_{0}=1$, $a_{1}=(0,1)$, $a_{2}=(0,0,1)$, $b_{0}=0$, $b_{1}=(1,0)$, and $b_{2}=(1,0,0)$. Hence, the corresponding CATE measures the effect of intervening at the last time step as compared to intervening at the first time step. For $\mathcal{D}_{2}$, we set $\tau=1$, $a_{1}=(0,1)$, and $b_{1}=(1,0)$.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Please refer to "Weaknesses".
soundness: 3
strengths: 1. Authors provided a comprehensive related works section.
2. The problem setting is clear and overall reasonable. 
3. The theoretical analysis is good and comprehensive, although I can't fully check the correctness.
4. The empirical validation is comprehensive too.
summary: This paper proposes model-agnostic meta-learners to estimate temporal HTE. Specifically, authors address the main challenges of temproal HTE estimation from a unified view. They theoretically analyze when specific learners are preferable over others. They derive inverse variance weights to stabilize the loss of the DR-learner.
weaknesses: 1. I suggest authors explain their motivation in detail. I believe the current version "Despite the fact that meta-learners are often considered state-of-the-art in the static setting, similar learners for the time-series setting are missing."  is not enough. How exactly meta-learners benefit or fit the time-varying setting? I think the reason: "meta-learners are SOTA under static setting" is not convincing. 

2.  The caption of Figure 1 can be more specific. I suggest authors add some explanation for the edges, for example, why X(t-2) has a direct causal influence on X(t+1)? What is the practical meaning of such causal path?

3. I think some important concepts are not clearly defined. For example, does response surfaces equal response functions? I presume they are the same estimands.

4. I understand the PI-RA-learner, which can be seen as the estimator version of Eq 5. However, can authors explain the pseudo-outcomes (Eq. 8) in detail? When At = at or bt, why the time index of history info (H) is t+1 rather than t?

5. The contribution of methodology may be thin, since I believe IPW estimator for heterogeneous treatment effects or counterfactual outcomes estimation is already proposed by Papadogeorgou et al (2022). 《Causal inference with spatio-temporal data: estimating the effects of airstrikes on insurgent violence in iraq.》 Journal of the Royal Statistical Society
Series B: Statistical Methodology, 84(5):1969–1999.

6. In eq.9, authors define the propensity scores in the time-varying setting. My concerns are about the implementation details about this propensity score. To be specific, the action (treatment) sequence Al, is a high-dimensional variable. So how do authors employ history info Hl to model the probability distribution of this high-dimensional variable? Please clarify my concerns.

7. In the experiment section, I understand authors try to emphasize that their proposed estimators do not rely on any specific model architecture. However, to implement an estimator, one have to choose a network, and I believe different network architectures can have diverse performances. In addition, there may be a best model (e.g. Transformer or Mamba). So I think the contribution of "MODEL-AGNOSTIC" may not be appropriate. 

8. BTW, it seems that authors do not discuss the computation efficiency. I suggest authors add this section in the main text or Appendix.

---

confidence: 2
contribution: 4
initial_rating: 8
presentation: 4
questions: 1. What is the precise definition for a regression estimator to admit a particular rate in Assumption 3?

2. The real-world experiment reports RMSE on predicted normalized blood pressure. What is the normalization, and should the results of the proposed method be considered "good" or "bad" in absolute terms (not just relative to the other methods, over which it clearly obtains improvement)? It would be helpful to include an analysis of this sort to indicate where there is still room for future work.
soundness: 4
strengths: The problem under study is of high practical relevance. While there has been a great proliferation of data availability in recent years, most of it is observational in nature, and collecting data from strict randomized controlled trials is still expensive. Methods that can take advantage of such data are critical. The time-varying aspect of the problem is also a natural feature of observational data collected "in the wild," making the problem setting even more relevant.
 
The presentation of the technical results is excellent. I am not an expert on causal inference, and the bare statement of one of the main results of the paper (Theorems 1) is highly technical in nature. However, the authors provide helpful interpretations of their results which make the value of the theorem clear. This clarity of exposition can be seen in many other places in the paper:

- An explanation for the lack of equality in equation (3) (failure to adjust for post-treatment confounders).
- An explanation for the constant variance assumption in Theorem 2. I had planned to add a question about this assumption to my review, but the authors addressed it immediately.
- Insight on the weighting mechanism of the proposed inverse variance weighted doubly-robust learner (lines 413-419), especially showing the connection to the simpler but more familiar binary treatment and static (not time-varying) settings.
- Intuition for which methods should be expected to perform best on which synthetic datasets and why.

These additional remarks and insights make the results of the paper far more accessible than they would be otherwise.

The results themselves are also strong. The authors give a theoretical analysis of the asymptotic behavior of each of the studied methods and discuss the implications for when each method should be preferred in practice. They then confirm their theory with synthetic datasets whose qualitative differences mimic some of the possible variation one could encounter when applying these methods in practice. Finally, they also obtain improved performance on real world healthcare data, which is the primary motivating example for the problem under study.
summary: The authors study the problem of estimating heterogeneous treatment effects in a time-dependent setting, where the treatment and measured covariates can change at each discrete step of a pre-specified time interval. The authors consider several possible meta-learning strategies for conditional average potential outcome (CAPO) and conditional average treatment effect (CATE) estimation which can be combined with any machine learning estimate of a base nuisance function, and they provide a theoretical analysis of the asymptotic error rate of each method. Based on the limitations of the initially proposed meta-learners, they propose the inverse variance weighted doubly robust learner to reduce instabilities that may arise due to low data availability. Finally, they confirm their theoretical findings and insights on both synthetic and real data.
weaknesses: While the intuitions that the authors have already provided are greatly appreciated, additional context for some of the quantities used in the paper would be helpful for a non-expert audience. For instance, additional explanation of the nuisance functions in an appendix would be helpful. Specific examples of the rate functions used in Theorem 1 for some particular settings would also make the theorem easier to parse.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. One strength claimed by the author is that the proposed learners can be used with "arbitrary machine learning models", and the title also emphasises "model-agnostic meta-learners". However, this is not supported by the empirical evaluation at all. I suggest the authors back up their claim with evidence. Also, some comparisons with different machine-learning models are essential for readers to understand the proposed meta-learner better. In its current form, the paper feels very unfinished. 

2. I think the paper is relatively well-written until section 7. Unfortunately, the rest of the paper seems very rushed, and the content is limited. In the only instantiation using a Transformer, the authors emphasized that the Transformer has not been optimized for the treatment effect estimation and then use quite a bit of space to give the details of their architecture. I think the space can be better used if more experiment results can be presented. The detailed architecture of Transformer can be in the supplementary. 

3. The discussion of results should also be enhanced. Currently, there is only a very brief description of the results of the experiment. There should at least be discussion of the results that can be linked back to the theoretical analysis. For example, in the theorem, it has been shown that the PI-HA learner is asymptotically biased. Still, in the real-world dataset, there are two occasions that PI-HA performs the best and by a significant margin. In fact, except for the IVW-DR-learner, other proposed meta-learners seldom perform better than the PI-HA learner despite the fact that PI-HA is biased while the others are not.
soundness: 2
strengths: This paper's strength is that it provides a good organisation of the existing learners and discusses the meta-learners and their estimation bounds. Although the technique used to prove the bound is not novel, it is nice to see the combination of existing techniques to provide bounds for each of the meta-learner.
summary: This paper studies treatment effect estimation from observational data over a period of time. The observed data contains a (time-varying) covariate, treatment, and outcome at each time point. The proposed methods contain a set of meta-learners designed to be used with any machine learning algorithm as base learners. The meta-learners are specifically designed for four types of confounding adjustment, namely history adjustment, regression adjustment, propensity adjustment, and doubly robust adjustment. In other words, the paper extends existing adjustment methods under the static setting into the dynamic setting. The theoretical bound between each of the proposed estimators and the true effect is given under assumptions that are commonly used in treatment effect estimation. In the experiment section, the paper studied Transformer as one type of learner under one synthetic and the MIMICIII dataset.
weaknesses: 1. One key weakness is that the experiment section is weak. Currently, the content has only one synthetic dataset and one real-world dataset, and the compared methods only include the meta-learners proposed in this paper. Only one instantiation is experimentally studied for the meta-learners proposed in this paper. I think there should at least be some existing state-of-the-art comparison included, and there should also be other instantiations even for a "proof-of-demonstration" experiment section, especially since ICLR has a tradition of valuing comprehensive experiments.

2. The later sections of the paper are not as well-written as the earlier ones. The latter sections seem rushed, and many sentences are incorrect. For example, "Here, we section outline..." should be this section outline; "proof-of-demonstration" should be proof-of-concept; Making things worse, the paper ended abruptly at section 7. There is not even a conclusion paragraph at the end of the paper.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: - I am not sure whether I understand the implications of eq. 3. The authors claim that other methods, such as CRN and CT, do not correctly adjust for this bias. However, as they build an unbiased representation iteratively in each step (based on $\bar{H}_t$), it seems to me that future covariates get added correctly, and no bias should exist?

- Do the learners with different estimands (e.g., nuisance parameters) use separate models for each, or are they built in a multitask framework? While a multitask framework is sensible for transformers, it does not apply to time series models generally.

- Do you use sample splitting in the experiments? 

- Having CAPO as $\mu$ and CATE as $\tau$ would be more clear in my opinion, though I leave this up to the authors. Later, $\mu$ and $\delta$ are also introduced, though they seemingly refer to similar estimands. What is the reasoning behind this notation? 

- The RA-Learner is much more complex than its static variant, where only the final term in Eq. 8 is present. Some intuition here would be appreciated.

___

**Minor points:**
- Line 290: "major different" -> difference
soundness: 4
strengths: The presented metalearners are a novel framework for thinking about these models and, as such, this work constitutes an important contribution to the literature. Given the popularity of metalearners in the static setting, these will certainly have wide applicability. Moreover, given the fast evolution in research on time series modeling, having these model-agnostic metalearners will be very valuable. The presented experiments clearly show the benefits of different approaches, with results corresponding to the hypotheses suggested in the theoretical analysis.
summary: The authors compare different causal metalearners estimating treatment outcomes over time. They also propose a novel variant of the temporal DR-Learner based on inverse-variance weights. These learners are described and then compared based on a transformer architecture.
weaknesses: 1. Several points are not entirely clear to me (possibly due to some misunderstandings):
- If I understand correctly, the challenge of time-varying confounding is different compared to previous work (e.g., CRN). In my opinion, this needs to be stressed more and explained in more detail.

- Some parts of the methodology are not entirely clear (see questions).

- The notation is not always clear to me (see questions).


2. There is no conclusion, or a discussion of limitations and possible directions for future work.

Meta-review: **Summary**: The paper proposes new meta-learners for time-varying settings. Time-varying confounding makes statistical estimation of heteregeneous treatment effects (HTE) very challenging due to exacerbated issues of overlap, justifying the need for estimators that can appropriately account for bias-variance tradeoffs in finite samples. Drawing on existing literature on meta-learners in the static setting, the paper develops new meta/dr-estimators and uses standard techniques to show consistency of proposed estimators. Empirical evaluation on synthetic and one real data demonstrate reasonable performance on heterogeneous treatment effect estimation.

**Strengths**:
1. HTE in time-varying settings is an important problem and prior work has so far not discussed robust estimators using sequential ML models as nuisance estimators in full detail. Therefore reviewers found the contribution useful to the community.
2. The theoretical results, while drawing on standard literature in the area was considered an important advance.
3. The paper is overall well written and the results are interpreted for a broad audience.
4. Paper contributes both theoretical and empirical insights for doubly-robust meta-learners.

**Weakness**:
1. Empirical evaluation is very weak considering the limited evaluation in real world data
2. Several claims were not precisely explained, specifically around runtime efficiency, some writing flaws, no conclusion section, etc. Reviewers pointed out many of these issues and were subsequently addressed in the revisions.
3. My read on the author response on discussion and interpretation of real world empirical performance of the proposed estimators was weak in the rebuttal response and mainly provided a generic response as opposed to concrete ablations on possible reasons of why a specific estimator dominates despite theoretical demonstration of bias.
4. Interesting that authors remove sample splitting in empirical evaluation, I understand the temptation for longitudinal data but the claim that this is not necessary for experiments but only necessary for the theory to go through is a bit weird, and it could change confidence intervals. Nonetheless authors may want to temper the claim on Donsker class and choice of sample-fitting vs not or provide a very concrete citation in justification of these choices.

**Justification**:
After seeing the updates to the main draft, and the responses to the reviewer concerns, the manuscript is definitively stronger, and crisper. Reviewers also improved their scores based on author response and most of their concerns were addressed modulo empirical evaluations. Considering the strengths and weakness of the paper, while I think statistically the contribution is somewhat incremental, it is important for the community and therefore I am recommending an accept.

Decision: Accept (Poster)