Using Surrogates in Covariate-adjusted Response-adaptive Randomization Experiments with Delayed Outcomes

Authors: lei  shi
waverly  wei
jingshen  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 19:09:07

Abstract: Covariate-adjusted response-adaptive randomization (CARA) designs are gaining increasing attention. These designs combine the advantages of randomized experiments with the ability to adaptively revise treatment allocations based on data collected across multiple stages, enhancing estimation efficiency. Yet, CARA designs often assume that primary outcomes are immediately observable, which is not the case in many clinical scenarios where there is a delay in observing primary outcomes. This assumption can lead to significant missingness and inefficient estimation of treatment effects. To tackle this practical challenge, we propose a CARA experimental strategy integrating delayed primary outcomes with immediately observed surrogate outcomes. Surrogate outcomes are intermediate clinical outcomes that are predictive or correlated with the primary outcome of interest. Our design goal is to improve the estimation efficiency of the average treatment effect (ATE) of the primary outcome utilizing surrogate outcomes. From a methodological perspective, our approach offers two benefits: First, we accommodate arm and covariates-dependent delay mechanisms without imposing any parametric modeling assumptions on the distribution of outcomes. Second, when primary outcomes are not fully observed, surrogate outcomes can guide the adaptive treatment allocation rule. From a theoretical standpoint, we prove the semiparametric efficiency bound of estimating ATE under delayed primary outcomes while incorporating surrogate outcomes. We show that the ATE estimator under our proposed design strategy attains this semiparametric efficiency bound and achieves asymptotic normality. Through theoretical investigations and a synthetic HIV study, we show that our design is more efficient than the design without incorporating any surrogate information.

Main paper: 
 introduction 1.motivation: 
 In recent years, covariate-adjusted response-adaptive randomization (CARA) designs have become increasingly prominent in clinical research for evaluating the effects of treatment plans or drugs on primary outcomes of interest [43, 36, 49] . Specifically, CARA designs are experiment strategies that adaptively revise treatment allocation based on observed outcomes and covariate information accumulated during multi-stage clinical trials [22] . Inheriting the advantage of traditional clinical trial designs, CARA designs can also help with providing reliable causal evidence to support clinical discoveries. Nevertheless, different from traditional clinical trial designs, CARA designs offer several major benefits. On the one hand, CARA designs enhance the precision of medicine by improving the statistical power to estimate treatment effects. On the other hand, CARA designs allow for adaptive modifications of treatment allocation based on intermediate findings. As highlighted in the recent Food and Drug Administration (FDA)'s adaptive design clinical trials for drugs and biologics guidance for industry [43] , the key advantage of adaptive clinical trial designs is their ability to adapt to new information that emerges during the trial. This fundamental characteristic fosters more flexible and efficient clinical trials. In sum, CARA designs can utilize clinical trial data efficiently by performing sequentially adaptive subgroup-specific optimization.While CARA designs are beneficial, they face a practical challenge when outcomes are observed with delays. Such delays make it impossible to sequentially revise the randomization probability to optimize trial objectives. Delayed outcomes are common in clinical practice, as treatments may take time to show their effects, especially for chronic conditions or diseases with slow progression. For instance, consider an HIV study conducted in Tanzania that investigates the effect of cash incentives on viral load suppression [15] . The primary outcome is the viral load measured six months after starting antiretroviral therapy (ART) because achieving viral suppression (below 1000 copies per mL) is critical for preventing HIV transmission [33, 31] . However, measuring viral load depends on regular clinic visits by study subjects, which are often delayed due to factors such as limited access to care, poverty, or social stigma [19] . These delays lead to significant missingness in primary outcome data, which undermines the efficiency of estimating treatment effects on the primary outcome.In situations where primary outcomes are delayed, surrogate outcomes-intermediate or substitute biomarkers that can be measured immediately and are predictive of primary outcomes-emerge as a natural remedy [20, 29, 5] . As mentioned in the FDA's surrogate endpoint resources for drug and biologic development, from 2010 to 2012, the FDA granted approval to 45% of new drugs based on a surrogate endpoint [44] . Using surrogate outcomes provides early indicators of a treatment's efficacy, which accelerates the decision-making process and can lead to faster delivery of beneficial treatments to patients. Surrogate outcomes are particularly useful in CARA designs because they enable modifications of treatment allocation without waiting for delayed primary outcomes during the interim stages.As such, there is a pressing need to develop new CARA designs to answer the following questions: How can both delayed primary outcomes and immediately observed surrogate outcomes be used simultaneously to guide treatment allocation revisions, optimizing the estimation efficiency of the primary outcome's treatment effect? In this manuscript, we propose a multi-stage CARA design with the goal of improving the estimation efficiency of primary outcome treatment effects by utilizing surrogate outcomes in the presence of delayed primary outcomes.Our contributions can be summarized as follows:1. From a methodological perspective, we propose a novel CARA design strategy that significantly improves the estimation efficiency of the primary outcome's ATE by incorporating surrogate outcomes. First, our approach uniquely integrates both primary and surrogate outcomes synergistically, unlike traditional clinical trial designs that depend exclusively on one or the other. Second, diverging from some adaptive experimental strategies that incorporate both surrogate and primary outcomes, we explore strategies that tackle the arm-dependent delay mechanism of primary outcomes and optimize the statistical efficiency in the presence of temporary or permanent missingness.2. From a theoretical perspective, we prove the semiparametric efficiency bound under armdependent delayed primary outcomes while incorporating surrogate outcomes for estimating the ATE (Theorem 3). We show that our proposed design strategy converges to the oracle design, which represents the optimal design assuming perfect knowledge of the data-generating distribution. We also prove the statistical validity of our inference procedure by establishing the asymptotic normality of our proposed ATE estimator (Theorem 1). Furthermore, the proposed ATE estimator attains the semiparametric efficiency bound (Theorem 2), which verifies the efficiency gain of adopting the proposed design. Theorem 1 further provides practical guidance for the general setup of the design in terms of the number of experimentation stages and the portion of enrolled units within each stage to achieve optimality. Lastly, we illustrate that our design largely enhances statistical efficiency compared to the design that only uses delayed primary outcomes.1.2 Related literature CARA designs. When covariate information is available to experimenters, CARA design can utilize both outcome information and covariate information to optimize for the experimental goals [37, 47] . CARA designs originate from response-adaptive randomization designs, yet they further utilize covariate information to assist the design [34, 21, 45] . [22] introduced a family of CARA designs that consider both efficiency and ethics. Generalizations of CARA to incorporate semiparametric estimates are discussed by [49] . Related CARA designs are also explored in studies such as [35, 1, 48] . However, unlike conventional CARA designs that assume immediate observation of outcomes, our approach accommodates delayed outcomes.Criteria of surrogacy. [32] proposes the statistical surrogacy criterion, which states that the observed primary outcome should be conditionally independent of the assigned treatment given the observed value of the post-treatment variable (surrogate outcome). The proposal is further extended by [17] and [9] . [16] argues that [32] 's proposal does not satisfy the causal necessity properties and proposes to study the principal surrogate. [27] introduces the notion of a strong surrogate using a graphical model, which states that treatment serves as an instrument for the effect of the surrogate on the primary outcome. [12] makes a detailed comparison of different notions of surrogacy. The authors propose (strictly) consistent surrogacy to avoid the surrogacy paradox and evaluate several conditions for the criteria to hold. [24] studies criteria of surrogacy based on distributional average causal effects.Surrogates in randomized experiments. Existing literature focuses on using surrogates can be roughly divided into two lines of work. The first line of work utilizes surrogates from a data fusion perspective. [4] proposes to use the surrogate index to estimate long-term causal effects when primary outcomes are missing in experimental data and only observed in observational data. This work relies on the statistical surrogacy assumption. [3] proposes a data combination framework to incorporate surrogate outcomes under the assumption of latent unconfoundedness. [23] proposes methods to use surrogate outcomes as proxy variables for persistent confounders. The second line of work uses surrogates as supplements for the primary outcomes. [13] investigates using surrogate data to improve efficiency when inferring parameters from estimating functions. [25] proposes to view surrogates as supplements instead of replacements for the primary outcome and investigates how this proposal can improve the efficiency of treatment effect estimation. [10] proposes to combine primary data and auxiliary data where both datasets are assumed to satisfy ignorability. [2] uses surrogate outcomes to assist with the design of adaptive clinical trials and show their relative and absolute benefits. Diverging from the usage of surrogate outcomes in randomized experiments, we explicitly model the arm and covariates delay mechanisms of primary outcomes while incorporating surrogates. 
problem setup: 
 Suppose we are designing an adaptive experiment with a total of T stages. Let X it denote the covariates, S it denote the surrogate outcome, and Y it denote the primary outcome for subject i at stage t = 1, . . . , T . Let A it denote the treatment assignment status, where A it = 1 corresponds to the treatment arm, while A it = 0 corresponds to the control. Following the Neyman-Rubin causal inference framework [30, 38] , we let Y it (a) and S it (a) represent the primary potential outcome and surrogate potential outcomes under treatment a, where a ∈ {0, 1}. Assume that (Y it (1) , Y it (0), S it (1), S it (0)) are i.i.d. copies of a population tuple (Y (1), Y (0), S(1), S(0)). Our parameter of interest is the average treatment effect corresponding to the primary outcome:τ = E[Y (1) -Y (0)].We are in a setting where, at the end of Stage t, we cannot observe the primary outcome for all subjects due to possible delay in the data collection process, but we can observe the surrogate outcomes for all subjects. The surrogate outcomes can inform the primary outcomes. We assume the primary outcome is observed after D it stages and denote the conditional cumulative distribution function of D it asρ a (d|x) = P(D it ≤ d|A it = a, X it = x), d = {0, 1, 2, . . . , T -1} ∪ {∞}, a ∈ {0, 1}.D it represents the number of stages for which an observation is delayed at stage t. For example, in a two-stage experiment, D i1 = 0 means no delay for the i-th observation. D i1 = 1 means that the i-th primary outcome will be observed at stage 2. When D i1 = ∞, we never collect the i-th primary outcome back. A primary outcome can only be observed by the end of the experiment if D it ≤ T -t. Moreover, we allow D it = ∞, which means that the primary outcome Y it is missing (censored) at the end of the experiment.The observed primary outcome can be written asY it = A it Y it (1) + (1 -A it )Y it (0), D it ≤ T -t; missing, D it > T -t.The observed surrogate outcome can be written as S it = A it S it (1) + (1 -A it )S it (0). Therefore, our observed data has the following structure:{(X it , A it , S it , Y it , D it ) nt i=1 } T t=1 .In our designs, experimenters can sequentially revise the treatment assignment based on the historical data. We define the treatment assignment probability for subjects in stage t ase t = P(A it = 1|X it , H t-1 ), t = 1, . . . , T, where H t-1 = {(X is , A is , D is , S is , Y is ) ns i=1 } t-1 s=1is the historical information collected up to Stage t -1. In our manuscript, we consider a multi-stage clinical trial setting where T < ∞ and n t → ∞ for t = 1, . . . , T .We have the following unconfounded assumption for arm-dependent delayed outcomes: A ⊥ ⊥ (Y (1), Y (0), S(1), S(0)) | X. We also assume the delay is arm and covariates dependent:D ⊥ ⊥ (Y (1), Y (0), S(1), S(0)) | X, A.As we are in the randomized experiment setting, the unconfoundedness assumption holds by design. The delay mechanism says that the delay variable is independent of potential primary outcomes and potential surrogate outcomes when conditional one covariates and treatment arms. This rules out the arrow between the delay and the outcome, which simplifies the technical discussion of the paper. Moreover, such a setup aims to reflect real-world randomized experiments where the delay is mainly due to protocol or structural factors instead of the realized outcome (such as the severity of some diseases). For example, if we are testing the effect of a vitamin supplement on blood vitamin levels and ask patients for regular check-ins, the delay is less likely to depend on the outcome since the outcome does not cause severe symptoms that affect patient check-in.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: Please see Section 7.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
design objective: 
 In this section, we shall formulate our design objective. Our goal of the CARA design is to improve the estimation efficiency of the ATE. Heuristically, we aim to find the optimal treatment allocation that minimizes the semiparametric efficiency bound (SPEB) of estimating the ATE.The formulation of the design objective relies on the derivation of the SPEB of the ATE under the delayed outcome setting while incorporating surrogate information. In Theorem 3 of the Appendix, we establish the SPEB for estimating τ . If we introduce the following notations on the conditional expectation of the outcomes:τ (a, x, s) = E[Y |A = a, X = x, S = s], τ (a, x) = E[τ (A, X, S)|A = a, X = x],our design objective can be formally defined as follows:min e∈[δ,1-δ] T V SPEB = min e∈[δ,1-δ] T E (Y -τ (1, X, S)) 2 T t=1 r t e t (X)ρ 1 (T -t|X) + (Y -τ (0, X, S)) 2 T t=1 r t (1 -e t (X))ρ 0 (T -t|X) + E (τ (1, X, S) -τ (1, X)) 2 T t=1 r t e t (X) + (τ (0, X, S) -τ (0, X)) 2 T t=1 r t (1 -e t (X)).The SPEB is a theoretical result derived from the work of [28] . Our design objective says that we aim to find the sequence of optimal treatment allocation that minimizes the semiparametric efficiency bound. Furthermore, we require that the treatment allocation should be bounded by δ and 1 -δ for δ ∈ (0, 1/2). Note that our design objective depends on the conditional expectation of the primary outcome on covariates and surrogates and the delay mechanism. However, in practical settings, both the conditional expectations and the delay mechanisms are unknown to practitioners. Therefore, we propose to use CARA designs that allow us to adaptively learn the unknown parameters based on sequentially collected data. We shall introduce our proposed design strategy in the following section.4 Proposed CARA design strategyIn this section, we shall formally introduce our proposed design strategy. We consider a delay variable D with finite support: D = {0, 1, . . . , D * } ∪ {∞}, where D * is pre-specified based on domain knowledge. Also, we allow D to take ∞, meaning the outcome is permanently missing. For simplicity, we assume that the enrollment proportion at each stage is r t = 1/T , for t = 1, . . . , T , T < ∞. We assume both X and S take discrete values to avoid unnecessary technical complexity.From a high level, our proposal consists of four steps.Step 1 consists of D ⋆ + 1 stages and serves as an exploration step that learns parameters like the delay distribution, outcome model, etc.Step 2 is a policy optimization step after Stage D ⋆ + 1, to compute the optimal allocation with the learned parameters.Step 3 is a policy calibration step from Stage D ⋆ + 1 to T , where we calibrate the allocation to match the optimal strategy. Step 4 constructs point and variance estimators based on the data. The full procedure is summarized in Algorithm 1.Stage 1 (Initialization): 1: Enroll n1 participants, and assign treatments with e1 = 1 2 ; 2: Estimate ρ a (l|x) = ρ (1) a (0|x), l = 1, . . . , T -1. Stage 2 to D * + 1 (Learn delay mechanism): 3: for t → 2 to D * + 1 do 4:Enroll nt subjects and assign treatment with probability et = 1 2 ; 5:Updateρ (t)a (l|x) for l ≤ t -1. 6: end for 7: Obtain the sequence of optimal treatment allocation e (D * +1) by solving (1) .Stage D * + 2 to T (Calibration of treatment allocation): 8: for t → D * + 2 to T do 9:Enroll subjects and assign treatments with calibrated probability e * t . 10: end for After Stage T (Inference): 11: Update τ and V. 12: Construct a two-sided α-level confidence interval for τ .Below we take a deeper dive into the details and motivations for each step in Algorithm 1.In line 1 -2, we initialize the experiment by assigning treatments with probability e 1 = 1 2 . After Stage 1, we are able to observe all the surrogate outcomes for n 1 subjects and part of the primary outcomes. One can estimate arm-dependent delay with ρ a (0|x) by counting the portion of the observed sample:ρ (1) a (0|x) = n1 i=1 1 (Di1≤0) • 1 (Xi1=x) 1 (Ai1=a) n1 i=1 1 (Xi1=x) 1 (Ai1=a)As we only observe outcomes whose delay variable D i1 ≤ 0 at the end of Stage 1, we let ρ(1)a (l|x) = ρ (1)a (0|x), l ≥ 1. We then estimate τ (1) (a, x, s) and τ (1) (a, x) by taking a sample average of the observed outcome stratified by (a, x, s) and (a, x), respectively:τ (1) (a, x, s) = n1 i=1 1 (Di1≤0) Y i1 1 (Ai1=a) 1 (Xi1=x) 1 (Si1=s) n1 i=1 1 (Di1≤0) 1 (Ai1=a) 1 (Xi1=x) 1 (Si1=s) , τ (1) (a, x) = n1 i=1 1 (Di1≤0) Y i1 1 (Ai1=a) 1 (Xi1=x) n1 i=1 1 (Di1≤0) 1 (Ai1=a) 1 (Xi1=x).Stage 2 to D * + 1. In line 4 to 5, we keep enrolling subjects and assign subjects to the treatment arm with probability e t = 1/2. Update the estimates of the delay mechanism by counting portions of the observed sample up to the current stage:ρ (t) a (l|x) = t-1 s=1 ns i=1 1 (Dis≤l) • 1 (Xis=x) 1 (Ais=a) t-1 s=1 ns i=1 1 (Xis=x) 1 (Ais=a), for l = 0, . . . , t -1, and let ρ(t) a (l|x) = ρ (t)a (t -1|x) for l ≥ t. Then update the estimates τ (t) (a, x, s) and τ (t) (a, x) as the new stratified sample averages:τ (t) (a, x, s) = t-1 l=1 n l i=1 Y il 1 (Dis≤t-1) 1 (A il =a) 1 (X il =x) 1 (S il =s) t-1 l=1 n l i=1 1 (Dis≤t-1) 1 (A il =a) 1 (X il =x) 1 (S il =s) , τ (t) (a, x) = t-1 l=1 n l i=1 Y il 1 (Dis≤t-1) 1 (A il =a) 1 (X il =x) t-1 l=1 n l i=1 1 (Dis≤t-1) 1 (A il =a) 1 (X il =x).At the end of Stage D * + 1, we can obtain estimates of all the delay mechanisms: ρ(D * +1) a (0|x), . . . , ρ (D * +1) a (D * |x).We then solve the following optimization problem to find the optimal sequence of e = ( e 1 , . . . , e T ):EQUATIONIn particular, confronted with the possibility that multiple minima could exist for the oracle optimization program, we introduced the penalty term λ N ∥e∥ 2 2 , which regularizes the solution. In Theorem 2, we will showcase that, theoretically, if we choose the penalty level to be λ N = C log N/N , e will converge in probability to an oracle minima of the population program. In practice, we propose to choose the tuning parameter based on cross-validation, where we split the collected data into multiple folds to tune the best scaling factor C for λ N = C log N/N . After Stage T . We obtain the final ATE estimate τ based on the efficient influence curve derived in Theorem 3 asτ = 1 N T t=1 nt i=1 A it 1 (Dit≤T -t) T t=1 r t • e t (X it ) ρ t,1 (D it ≤ T -t|X) Y it -τ (1, X it , S it ) - (1 -A it )1 (Dit≤T -t) T t=1 r t • (1 -e t (X it ) ρ t,0 (D it ≤ T -t|X) Y it -τ (0, X it , S it ) + A it T t=1 r t • e t (X it ) τ (1, X it , S it ) -τ (1, X it ) - (1 -A it ) T t=1 r t • (1 -e t (X it ) τ (0, X it , S it ) -τ (0, X it ) + τ (1, X it ) -τ (0, X it ) ,and the variance estimates asV = 1 N T t=1 nt i=1 1 (Dit≤T -t) A it (Y it -τ (1, X it , S it )) 2 { T t=1 r t • e t (X it ) ρ 1 (T -t|X it )} 2 + 1 (Dit≤T -t) (1 -A it )(Y it -τ (0, X it , S it )) 2 { T t=1 r t • 1 -e t (X it ) ρ 0 (T -t|X it )} 2 + 1 N T t=1 nt i=1 A it ( τ (1, X it , S it ) -τ (1, X it )) 2 { T t=1 r t • e t (X it )} 2 + (1 -A it )( τ (0, X it , S it ) -τ (0, X it )) 2 { T t=1 r t • 1 -e t (X it ) } 2 + 1 N T t=1 nt i=1 τ (1, X it ) -τ (0, X it ) -τ 2 .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
theoretical investigation: 
 In this section, we investigate the theoretical properties of our proposed design strategy. In Theorem 1, we demonstrate the asymptotic normality of the ATE estimator with any sequence of propensity scores that are generated sequentially from history and converge to some fixed scores. Furthermore, we show that our design strategy asymptotically converges to the oracle design strategy in Theorem 2, which suggests that the constructed estimator is semi-parametrically efficient.Theorem 1 (Asymptotic normality). Assume the delays are conditionally independent of the surrogate and potential outcomes, given covariates and historical data. Assume the following conditions hold:1. Bounded conditional moments:max{m 4 (0, s, x), m 4 (1, s, x)} ≤ M 4 , where m 4 (a, s, x) = E |Y (a)| 4 | S(a) = s, X = x , a = 0, 1.2. Convergent scores: there exists a set of propensity scores e, such thatEQUATIONLet n t = r t N with r t > 0, t r t = 1. Then when N → ∞, we have√ N ( τ -τ ) → N (0, V(e)).Moreover, the variance estimator satisfiesV(e) -V(e) = O p ( 1 √ N ).Theorem 1 is a general theoretical result that works for any sequence of historic-dependent treatment allocation strategy e, as long as e converges to an oracle policy in probability asymptotically. For example, one can apply complete randomization through the experiment, which corresponds to e t,cr = e t,cr = 1/2 and establish the asymptotic normality of τ with variance V(e cr ). In particular, if the proposed adaptive designing strategy (Algorithm 1) is implemented, the treatment allocation probability will converge to one set of optimal policies that minimizes the semiparametric efficiency bound (4), which is established in Theorem 2 below. Theorem 2 (Convergence of the proposed design strategy). Let n t = r t N with r t > 0, t r t = 1.Assume the bounded conditional moments hold as in Theorem 1. Assume sufficient enrollment at the first (T -D ⋆ ) (non-delayed) stages: (b) ee * * = o p (1) , where e * * is a minima of L(e) defined as follows:EQUATION) Set λ N = C log N /N . When N → ∞,e * * t =      1/2, t ≤ D * + 1; T -D ⋆ s=1 rs•e * s -D * +1 s=1 rs•1/2 T -D * s=D * +2 rs , D * + 2 ≤ t ≤ T -D * ; e * t , t ≥ T -D * + 1.Theorem 2 formally established the convergence property of the propensity scores generated by Algorithm 1. More concretely, Theorem 2 showed that two sets of estimated treatment allocation probability converge to the minima of the population optimal variance: one is the scores e generated by the optimization program (1) , and the other is the sequence of propensity scores e upon calibration which is implemented through the experiment. Therefore, Theorem 1 and Theorem 2 together imply that the point estimator based on the proposed treatment strategy attains the semiparametric efficient bound asymptotically.Theorem 2 imposes an additional condition (3), which requires a sufficient number of enrollment in the non-delayed stages. This is an important sufficient condition for the optimality of the adaptive allocation strategy, which ensures the optimization program has multiple global minima and enables experimenters to calibrate the propensity scores based on the history to achieve optimality. In practice, Condition (3) can also provide guidance on the design of the experiment to decide the length of the study and the portion of units to recruit in the initial stages. 
synthetic case study: hiv trial: 
 In this synthetic case study, we calibrate our data-generating process using data collected from an HIV randomized controlled trial (RCT) conducted in Tanzania [15] . In this study, n = 530 participants are initially enrolled and randomized to one of the three treatment arms: (1) receive no cash, (2) receive 10,000 TZS, and (3) receive 22,500 TZS. In our case study, we combine the latter two arms and define the control arm A = 0 as "receive no cash transfer" and the treatment arm A = 1 as "receive any cash transfer." For the treatment arm, the cash incentives were administered six times from May 21, 2018, to May 31, 2019. In our case study, we consider June 1, 2019, as the starting point of our trial because all the delays observed after June 1, 2019, are not due to the delay in cash transfer administration. After all the cash incentives were administered, n = 511 subjects were still alive on June 1, 2019, with n = 177 subjects in the control arm and n = 334 in the treatment arm.As the very last test was taken on June 29, 2021, we consider June 29, 2021, as the last time point where we gathered outcome data.Motivated by the original HIV trial, we also consider a multi-stage trial where each stage consists of six months, with the primary outcome Y being the viral load measured at the end of a stage. The viral load can be delayed to be measured because HIV patients may miss their scheduled test appointments. Therefore, we consider the World Health Organization (WHO) stage as the surrogate outcome S ∈ {1, 2, 3} because the WHO stage is a clinical indicator of HIV infection progression and it is measured without delay, where S = 1, 2, 3 indicates the asymptomatic, mild, and advanced symptoms respectively. We use "sex" as the covariate X in our study, where X = 1 indicates male and X = 0 indicates female. We further observe that the delay mechanism depends on both the treatment arm and the covariate. We summarize the delay mechanism observed from the real data in Table B . We summarize the expectation and standard deviation of the primary outcome under arm A given X and S in Table 2 .In our synthetic data generation, we generate the surrogate outcome from a multinomial distribution. The delay mechanism is also generated following multinomial distribution using the parameters in Table B . We generate X as X it ∼ Bernoulli(0.64), and the primary outcome variable Y asY it |A it = a, X it = x, S it = s ∼ N (τ (a, x, s)).The true average treatment effect of the primary outcome is τ = 0.04. Following our proposed design strategy, we set the total number of experimental stages as T = 2(D * + 1) = 8. In this case study, We compare three designs: (1) Our proposed design strategy; (2) the complete randomization design which sets e t = 1/2 for t = 1, . . . , T ; (3) CARA design that only utilizes primary outcomes but not surrogate outcomes.We present the case study results in Figure 1 . In terms of bias, Figure 1 (A) suggests that the point estimators from all three designs have a vanishing bias as the sample size grows, validating the asymptotic unbiasedness of all strategies. Nevertheless, in terms of variance, Figure 1 (B) demonstrates that our proposed design yields smaller standard deviations and thus higher estimation efficiency for estimating the ATE. The synthetic case study results verify our efficiency comparison in Section 5 and demonstrate the efficiency gain of our proposed design strategy. Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: Please see Section 6 and Supplementary Materials.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
discussion: 
 In this work, we proposed a CARA design strategy that improves the efficiency of ATE estimation in the presence of delay by incorporating surrogate outcomes. The strategy leads to a point estimator that achieves semiparametric efficiency under the arm-covariate-dependent delay setting. It also demonstrates efficiency gain compared with the strategy that uses only primary endpoints. There are some future directions to explore. First, we mainly considered an arm-dependent delay setting [18, 41] . A more general setup is to consider the delay mechanism that is also dependent on the outcomes [42] . Another possible framework is to consider outcomes that are also impacted by delays, for which the delay variable serves as a special set of surrogate variables [11] . Moreover, the objective of this work focuses on statistical efficiency gain for ATE estimation. It is also important to incorporate realistic constraints for practical implementation, such as fairness among subgroups [46] , safety concerns [6] , the discrepancy between algorithmic results and human knowledge [14] , etc. More generally, it is also interesting to draw connection to other relevant problems in causal inference and adaptive experiments, such as dealing with many treatment arms [26, 39, 40] , targeting other causal parameters [7] , incorporating instrumental variables [8] , among others. We leave these questions as future research work.We use R i ∈ {1, . . . , T } to record the stage at which the subject i is enrolled, such that P(R i = t) = r t , T t=1 r t = 1. Theorem 3. Under the unconfoundedness assumption and the arm-dependent delay assumption:(a) The efficient influence function for estimating τ isφ(X, A, S, Y, R, D) = τ (1, X) -τ (0, X) -τ + A T t=1 1 (R=t) T t=1 r t e t (X) τ (1, X, S) -τ (1, X) - (1 -A) T t=1 1 (R=t) T t=1 r t (1 -e t (X))τ (0, X, S) -τ (0, X)+ A T t=1 1 (R=t) 1 (D≤T -t) T t=1 r t e t (X)ρ 1 (T -t|X) Y -τ (1, X, S) - (1 -A) T t=1 1 (R=t) 1 (D≤T -t) T t=1 r t (1 -e t (X))ρ 0 (T -t|X) Y -τ (0, X, S) .(b) The semiparametric efficiency bond for estimating τ isEQUATIONProof. (a) Proof of Efficient influence functionStep 1. First, we write out the joint density the observed sample as f η (x, a, d, v, s, y), indexed by η.f η (x, a, d, v, s, y) = f X,η (x) T t=1 r t,η [e t,η (x)g S(1)|X,η (s|x) ρ a,η (T -t|x)f Y (1)|S,X,η (y|s, x) 1 (d≤T -t) ] (1 -ρ a,η (T -t|x)) 1 (d>T -t) ] a • [(1 -e t,η (x)) g S(0)|X,η (s|x) ρ 1-a,η (T -t|x)f Y (0)|S,X,η (y|s, x) 1 (d≤T -t) ] (1 -ρ 1-a,η (T -t|x)) 1 (d>T -t) ] 1-a 1 (v=t) , = f X,η (x) × r T t=1 1 (v=t) t,η × T t=1 (e t,η (x)) a1 (v=t) (1 -e t,η (x)) (1-a)1 (v=t) × g S(1)|X,η (s|x) T t=1 a1 (v=t) g S(0)|X,η (s|x) T t=1 (1-a)1 (v=t) × T t=1 (ρ a,η (T -t|x)) a1 (d≤T -t) 1v=t (1 -ρ a,η (T -t|x)) a1 (d>T -t) 1 (v=t) × T t=1 (ρ 1-a,η (T -t|x)) (1-a)1 (d≤T -t) 1 (v=t) (1 -ρ 1-a,η (T -t|x)) (1-a)1 (d>T -t) 1 (v=t) × f Y (1)|S,X,η (y|s, x) T t=1 a1 (d≤T -t) 1 (v=t) × f Y (0)|S,X,η (y|s, x) T t=1 (1-a)1 (d≤T -t) 1 (v=t) .Step 2. Next, we derive the tangent spaces. We usef(•), g(•), e t (•), p t (•) to denote ∂ ∂η log f (•), ∂ ∂η log g(•), ∂ ∂η log e(•), and ∂ ∂η log ρ(•). T X = {f X (x) : E[f X (X i )] = 0} , T Y (1) = T t=1 a1 (d≤T -t) 1 (v=t) • f 1 (y|s, x) : E[f 1 (Y i (1)|S i , X i )|S i , X i ] = 0 , T Y (0) = T t=1 (1 -a)1 (d≤T -t) 1 (v=t) • f 0 (y|s, x) : E[f 0 (Y i (0)|S i , X i )|S i , X i ] = 0 , T S(1) = T t=1 a1 (v=t) g 1 (s|x) : E[g 1 (S i (1)|X i )|X i ] = 0 T S(0) = T t=1 (1 -a)1 (v=t) g 0 (s|x) : E[g 0 (S i (0)|X i )|X i ] = 0 T A,t = a1 (v=t) e t (x) + (1 -a)1 (v=t) 1 -e t (x) = 1 (v=t) a e t (x) - 1 -a 1 -e t (x) e ′ t (x) , T D,a,t = a1 (v=t) 1 (d≤T -t) p a (T -t|x) + 1 (d>T -t) 1 -p a (T -t|x) = a1 (v=t) 1 (d≤T -t) ρ a (T -t|x) - 1 (d>T -t) 1 -ρ a (x) ρ ′ a (T -t|x) , T D,1-a,t = (1 -a)1 (v=t) 1 (d≤T -t) p 1-a (T -t|x) + 1 (d>T -t) 1 -p 1-a (T -t|x) = (1 -a)1 (v=t) 1 (d≤T -t) ρ 1-a (T -t|x) - 1 (d>T -t) 1 -ρ 1-a (x) ρ ′ 1-a (T -t|x) .Therefore, the tangent space isT = T X ⊕ T Y (1) ⊕ T Y (0) ⊕ T S(1) ⊕ T S(0) ⊕ (T A,1 ⊕ T D,A,1 ⊕ T 1-D,A,1 ) ⊕ . . . ⊕ (T A,T ⊕ T D,A,T ⊕ T 1-D,A,T ).Step 3. We consider the estimation of the target parameter E[Y (1)] via a parametric submodel indexed by η. Under identifiability assumptions, the target parameter can be written asE η [Y (1)] = E η E η E η [Y |A = 1, V = t, D ≤ T -t, S, X]|X .Taking pathwise derivative of E η [Y (1)] at η = 0, we have∂E η [Y (1)] ∂η | η=0 = ∂ ∂η E η E E[Y |A = 1, V = t, D ≤ T -t, S, X]|X η=0 (I) + E ∂ ∂η E η E[Y |A = 1, V = t, D ≤ T -t, S, X]|X η=0 (II) + E E ∂ ∂η E η [Y |A = 1, V = t, D ≤ T -t, S, X]| η=0 |X (III)Let s(X, A, D, V, S, Y ) denote the score function of f (x, a, d, v, s, y).(I) = E τ (1, X)f X (X) = E (τ (1, X) -τ 1 )f X (X) = E (τ (1, X) -τ 1 )s(X, A, D, V, S, Y ) , where τ (1, X) = E E[Y |A = 1, V = t, D ≤ T -t, S, X]|X . (II) = E E τ (1, X, S)g(S|X, A = 1)|X = E E τ (1, X, S) -τ (1, X) g(S|X, A = 1)|X = E E τ (1, X, S) -τ (1, X) s(X, A, D, V, S, Y )|X = E A • 1 (V =t) e t (X) • r t τ (1, X, S) -τ (1, X) s(X, A, D, V, S, Y ) , where τ (1, S, X) = E[Y |A = 1, V = t, D ≤ T -t, S, X]. (III) = E E E[Y f Y (Y |A = 1, S, X, V, D)|A = 1, V = t, D ≤ T -t, S, X]|X = E E E[ Y -τ (1, S, X) f Y (Y |A = 1, S, X, V, D)|A = 1, V = t, D ≤ T -t, S, X]|X = E A • 1 (V =t) • 1 (D≤T -t) e t (X)r t ρ a (T -t|X) Y -τ (1, S, X) f Y (Y |A = 1, S, X, V, D) = E A • 1 (V =t) • 1 (D≤T -t) e t (X) • r t • ρ a (T -t|X) Y -τ (1, S, X) s(Y, A, S, X, V, D) . As a result, ∂E η [Y (1)] ∂η | η=0 = E τ (1, X) -τ 1 + A • 1 (V =t) e t (X)r t τ (1, X, S) -τ (1, X) + A • 1 (V =t) • 1 (D≤T -t) e t (X)r t ρ a (T -t|X) Y -τ (1, S, X) s(Y, A, S, X, V, D) = E φ 1 (X) + φ 2 (A, S, X, V ) + φ 3 (Y, A, S, X, V, D) s(Y, A, S, X, V, D) ,whereφ 1 (X) = τ (1, X) -τ 1 , φ 2 (A, S, X, V ) = A • 1 (V =t) e t (X)r t τ (1, X, S) -τ (1, X) , φ 3 (Y, A, S, X, V, D) = A • 1 (V =t) • 1 (D≤T -t) e t (X)r t ρ a (T -t|X) Y -τ (1, S, X) .Step 4. Now we project the score functions to the corresponding tangent spaces. First, we project φ 1 (X) onto T X .E φ 1 (X) -f(X) × f(X) = 0.We obtainf(X) = φ 1 (X).Next, we project φ 2 (A, S, X, V ) onto T S(1) .E φ 2 (A, S, X, V ) -A T t=1 1 (V =t) f(S|X) × A T t=1 1 (V =t) f(S|X) = 0. E φ 2 (A, X, V, D) × A i T t=1 1 (V =t) f(S|X) = E A • 1 (V =t) e t (X)r t τ (1, S, X) -τ (1, X) × A T t=1 1 (V =t) f(S|X) = E E A • 1 (V =t) e t (X)r t τ (1, S, X) -τ (1, X) × A T t=1 1 (V =t) f(S|X)|X = E τ (1, S, X) -τ (1, X) f(S|X) .Then,E A T t=1 1 (V =t) f(S|X) × A T t=1 1 (V =t) f(S|X) = E A T t=1 1 (V =t) f(S|X) 2 = E E A T t=1 1 (V =t) f(S|X) 2 |X = E T t=1 e t (X)r t E f(S|X) 2 |X . Because E τ (1, S, X) -τ (1, X) f(S|X) = E T t=1 e t (X)r t E f(S|X) 2 |X , we obtain f(Y |X, S) = τ (1, S, X) -τ (1, X) T t=1 e t (X)r t .Lastly, we project φ 3 (X) onto T Y (1) .E φ 3 (Y, A, S, X, V, D) -A T t=1 1 (V =t) 1 D≤T -t f(Y |X, S) × A T t=1 1 (V =t) 1 D≤T -t f(Y |X, S) = 0. E φ 3 (Y, A, S, X, V, D) × A i T t=1 1 (V =t) 1 (D≤T -t) f(Y |X, S) = E A • 1 (V =t) • 1 (D≤T -t) e t (X)r t ρ a (T -t|X) Y -τ (1, S, X) × A T t=1 1 (V =t) 1 (D≤T -t) f(Y |X, S) = E E A • 1 (V =t) • 1 (D≤T -t) e t (X)r t ρ a (T -t|X) Y -τ (1, S, X) × A T t=1 1 (V =t) 1 D≤T -t f(Y |X, S)|X = E Y -τ (1, S, X) f(Y |X, S) . E A T t=1 1 (V =t) 1 D≤T -t f(Y |X, S) × A T t=1 1 (V =t) 1 D≤T -t f(Y |X, S) = E A T t=1 1 (V =t) 1 D≤T -t f(Y |X, S) 2 = E E A T t=1 1 (V =t) 1 D≤T -t f(Y |X, S) 2 |X = E T t=1 e t (X)r t ρ 1 (T -t|X)E f(Y |X, S) 2 |X . Because E Y -τ (1, S, X) f(Y |X, S) = E T t=1 e t (X)r t ρ 1 (T -t|X)E f(Y |X, S) 2 |X , we obtain f(Y |X, S) = Y -τ (1, S, X) T t=1 e t (X)r t ρ 1 (T -t|X)In sum, the efficient influence function isφ(Y, A, S, X, V, D) = A • T t=1 1 (V =t) • 1 (D≤T -t) T t=1 e t (X)r t ρ 1 (T -t|X) Y -τ (1, S, X) - (1 -A) • T t=1 1 (V =t) • 1 (D≤T -t) T t=1 (1 -e t (X))r t ρ 0 (T -t|X) Y -τ (0, S, X) + A • T t=1 1 (V =t) T t=1 e t (X)r t τ (1, X, S) -τ (1, X) - (1 -A) • T t=1 1 (V =t) T t=1 (1 -e t (X))r t τ (0, X, S) -τ (0, X) + (τ (1, X) -τ (0, X) -τ ).Proof.V[φ(X, A, S, Y, R, D)]= V τ (1, X) -τ (0, X) -τ + A T t=1 1 (R=t) T t=1 r t e t (X) τ (1, X, S) -τ (1, X) - (1 -A) T t=1 1 (R=t) T t=1 r t (1 -e t (X))τ (0, X, S) -τ (0, X)+ A T t=1 1 (R=t) 1 (D≤T -t) T t=1 r t e t (X)ρ 1 (T -t|X) Y -τ (1, X, S) - (1 -A) T t=1 1 (R=t) 1 (D≤T -t) T t=1 r t (1 -e t (X))ρ 0 (T -t|X) Y -τ (0, X, S) = V τ (1, X) -τ (0, X) -τ (A) + V A T t=1 1 (R=t) T t=1 r t e t (X) τ (1, X, S) -τ (1, X) - (1 -A) T t=1 1 (R=t) T t=1 r t (1 -e t (X))τ (0, X, S) -τ (0, X)EQUATION.We work with each term separately.(A) = V τ (1, X) -τ (0, X) -τ = E τ (1, X) -τ (0, X) -τ 2 . (B) = V A T t=1 1 (R=t) T t=1 r t e t (X) τ (1, X, S) -τ (1, X) - (1 -A) T t=1 1 (R=t) T t=1 r t (1 -e t (X)) τ (0, X, S) -τ (0, X) = V A T t=1 1 (R=t) T t=1 r t e t (X) τ (1, X, S) -τ (1, X) + V (1 -A) T t=1 1 (R=t) T t=1 r t (1 -e t (X)) τ (0, X, S) -τ (0, X) -2Cov A T t=1 1 (R=t) T t=1 r t e t (X) τ (1, X, S) -τ (1, X) , (1 -A) T t=1 1 (R=t) T t=1 r t (1 -e t (X)) τ (0, X, S) -τ (0, X) = E A T t=1 1 (R=t) T t=1 r t e t (X) τ (1, X, S) -τ (1, X) 2 + E (1 -A) T t=1 1 (R=t) T t=1 r t (1 -e t (X)) τ (0, X, S) -τ (0, X) 2 -2E A T t=1 1 (R=t) T t=1 r t e t (X) τ (1, X, S) -τ (1, X) • E (1 -A) T t=1 1 (R=t) T t=1 r t (1 -e t (X)) τ (0, X, S) -τ (0, X) = E A T t=1 1 (R=t) T t=1 r t e t (X) τ (1, X, S) -τ (1, X) 2 + E (1 -A) T t=1 1 (R=t) T t=1 r t (1 -e t (X)) τ (0, X, S) -τ (0, X) 2 . (C) = V A T t=1 1 (R=t) 1 (D≤T -t) T t=1 r t e t (X)ρ 1 (T -t|X) Y -τ (1, X, S) - (1 -A) T t=1 1 (R=t) 1 (D≤T -t) T t=1 r t (1 -e t (X))ρ 0 (T -t|X) Y -τ (0, X, S) = V A T t=1 1 (R=t) 1 (D≤T -t) T t=1 r t e t (X)ρ 1 (T -t|X) Y -τ (1, X, S) + V (1 -A) T t=1 1 (R=t) 1 (D≤T -t) T t=1 r t (1 -e t (X))ρ 0 (T -t|X) Y -τ (0, X, S) -2Cov A T t=1 1 (R=t) 1 (D≤T -t) T t=1 r t e t (X)ρ 1 (T -t|X) Y -τ (1, X, S) , (1 -A) T t=1 1 (R=t) 1 (D≤T -t) T t=1 r t (1 -e t (X))ρ 0 (T -t|X) Y -τ (0, X, S) = E A T t=1 1 (R=t) 1 (D≤T -t) T t=1 r t e t (X)ρ 1 (T -t|X) Y -τ (1, X, S) 2 + E (1 -A) T t=1 1 (R=t) 1 (D≤T -t) T t=1 r t (1 -e t (X))ρ 0 (T -t|X) Y -τ (0, X, S) 2 -2E A T t=1 1 (R=t) 1 (D≤T -t) T t=1 r t e t (X)ρ 1 (T -t|X) Y -τ (1, X, S) E (1 -A) T t=1 1 (R=t) 1 (D≤T -t) T t=1 r t (1 -e t (X))ρ 0 (T -t|X) Y -τ (0, X, S) = E A T t=1 1 (R=t) 1 (D≤T -t) T t=1 r t e t (X)ρ 1 (T -t|X) Y -τ (1, X, S) 2 + E (1 -A) T t=1 1 (R=t) 1 (D≤T -t) T t=1 r t (1 -e t (X))ρ 0 (T -t|X) Y -τ (0, X, S) 2 . (D) = 2Cov τ (1, X) -τ (0, X) -τ , A T t=1 1 (R=t) T t=1 r t e t (X) τ (1, X, S) -τ (1, X) - (1 -A) T t=1 1 (R=t) T t=1 r t (1 -e t (X)) τ (0, X, S) -τ (0, X) = 2Cov τ (1, X) -τ (0, X) -τ , A T t=1 1 (R=t) T t=1 r t e t (X) τ (1, X, S) -τ (1, X) D1 -2Cov τ (1, X) -τ (0, X) -τ , (1 -A) T t=1 1 (R=t) T t=1 r t (1 -e t (X)) τ (0, X, S) -τ (0, X) D2 = D 1 -D 2 . D 1 = 2E τ (1, X) -τ (0, X) -τ A T t=1 1 (R=t) T t=1 r t e t (X) τ (1, X, S) -τ (1, X) -2E τ (1, X) -τ (0, X) -τ E A T t=1 1 (R=t) T t=1 r t e t (X) τ (1, X, S) -τ (1, X) = 2E τ (1, X) -τ (0, X) -τ A T t=1 1 (R=t) T t=1 r t e t (X) τ (1, X, S) -τ (1, X) , D 2 = 2E τ (1, X) -τ (0, X) -τ (1 -A) T t=1 1 (R=t) T t=1 r t (1 -e t (X)) τ (0, X, S) -τ (0, X) -2E τ (1, X) -τ (0, X) -τ E (1 -A) T t=1 1 (R=t) T t=1 r t (1 -e t (X)) τ (0, X, S) -τ (0, X) = 2E τ (1, X) -τ (0, X) -τ (1 -A) T t=1 1 (R=t) T t=1 r t (1 -e t (X))τ (0, X, S) -τ (0, X) .Therefore,(D) = D 1 -D 2 = 2E τ (1, X) -τ (0, X) -τ A T t=1 1 (R=t) T t=1 r t e t (X) τ (1, X, S) -τ (1, X) -2E τ (1, X) -τ (0, X) -τ (1 -A) T t=1 1 (R=t) T t=1 r t (1 -e t (X)) τ (0, X, S) -τ (0, X) = 0. (E) = 2Cov τ (1, X) -τ (0, X) -τ , A T t=1 1 (R=t) 1 (D≤T -t) T t=1 r t e t (X)ρ 1 (T -t|X) Y -τ (1, X, S) - (1 -A) T t=1 1 (R=t) 1 (D≤T -t) T t=1 r t (1 -e t (X))ρ 0 (T -t|X) Y -τ (0, X, S) = 2Cov τ (1, X) -τ (0, X) -τ , A T t=1 1 (R=t) 1 (D≤T -t) T t=1 r t e t (X)ρ 1 (T -t|X) Y -τ (1, X, S) E1 -2Cov τ (1, X) -τ (0, X) -τ , (1 -A) T t=1 1 (R=t) 1 (D≤T -t) T t=1 r t (1 -e t (X))ρ 0 (T -t|X) Y -τ (0, X, S) E2 = E 1 -E 2 . E 1 = 2E τ (1, X) -τ (0, X) -τ A T t=1 1 (R=t) 1 (D≤T -t) T t=1 r t e t (X)ρ 1 (T -t|X) Y -τ (1, X, S) -2E τ (1, X) -τ (0, X) -τ E A T t=1 1 (R=t) 1 (D≤T -t) T t=1 r t e t (X)ρ 1 (T -t|X) Y -τ (1, X, S) = 0. E 2 = 2E τ (1, X) -τ (0, X) -τ (1 -A) T t=1 1 (R=t) 1 (D≤T -t) T t=1 r t (1 -e t (X))ρ 0 (T -t|X) Y -τ (0, X, S) -2E τ (1, X) -τ (0, X) -τ E (1 -A) T t=1 1 (R=t) 1 (D≤T -t) T t=1 r t (1 -e t (X))ρ 0 (T -t|X) Y -τ (0, X, S) = 0. Therefore, (E) = E 1 -E 2 = 0. (F ) = 2Cov A T t=1 1 (R=t) T t=1 r t e t (X) τ (1, X, S) -τ (1, X) - (1 -A) T t=1 1 (R=t) T t=1 r t (1 -e t (X)) τ (0, X, S) -τ (0, X) , A T t=1 1 (R=t) 1 (D≤T -t) T t=1 r t e t (X)ρ 1 (T -t|X) Y -τ (1, X, S) - (1 -A) T t=1 1 (R=t) 1 (D≤T -t) T t=1 r t (1 -e t (X))ρ 0 (T -t|X) Y -τ (0, X, S) = 2E A T t=1 1 (R=t) T t=1 r t e t (X) τ (1, X, S) -τ (1, X) - (1 -A) T t=1 1 (R=t) T t=1 r t (1 -e t (X))τ (0, X, S) -τ (0, X)• A T t=1 1 (R=t) 1 (D≤T -t) T t=1 r t e t (X)ρ 1 (T -t|X) Y -τ (1, X, S) - (1 -A) T t=1 1 (R=t) 1 (D≤T -t) T t=1 r t (1 -e t (X))ρ 0 (T -t|X) Y -τ (0, X, S) -2E A T t=1 1 (R=t) T t=1 r t e t (X) τ (1, X, S) -τ (1, X) - (1 -A) T t=1 1 (R=t) T t=1 r t (1 -e t (X))τ (0, X, S) -τ (0, X)• E A T t=1 1 (R=t) 1 (D≤T -t) T t=1 r t e t (X)ρ 1 (T -t|X) Y -τ (1, X, S) - (1 -A) T t=1 1 (R=t) 1 (D≤T -t) T t=1 r t (1 -e t (X))ρ 0 (T -t|X) Y -τ (0, X, S) = 2E A T t=1 1 (R=t) T t=1 r t e t (X) τ (1, X, S) -τ (1, X) • A T t=1 1 (R=t) 1 (D≤T -t) T t=1 r t e t (X)ρ 1 (T -t|X) Y -τ (1, X, S) + 2E (1 -A) T t=1 1 (R=t) T t=1 r t (1 -e t (X)) τ (0, X, S) -τ (0, X) (1 -A) T t=1 1 (R=t) 1 (D≤T -t) T t=1 r t (1 -e t (X))ρ 0 (T -t|X) Y -τ (0, X, S) = 0.In sum, terms(D) = (E) = (F ) = 0, V[φ(X, A, S, Y, R, D)] = (A) + (B) + (C): V[φ(X, A, S, Y, R, D)] = E τ (1, X) -τ (0, X) -τ 2 + E A T t=1 1 (R=t) T t=1 r t e t (X) τ (1, X, S) -τ (1, X) 2 + E (1 -A) T t=1 1 (R=t) T t=1 r t (1 -e t (X)) τ (0, X, S) -τ (0, X) 2 + E A T t=1 1 (R=t) 1 (D≤T -t) T t=1 r t e t (X)ρ 1 (T -t|X) Y -τ (1, X, S) 2 + E (1 -A) T t=1 1 (R=t) 1 (D≤T -t) T t=1 r t (1 -e t (X))ρ 0 (T -t|X) Y -τ (0, X, S) 2 .A.2 Proof of Theorem 1Proof. Our proposed ATE estimator isEQUATION) τ 0 = 1 N T t=1 nt i=1 (1 -A it )1 (Dit≤T -t) T t=1 r t • (1 -e t (X it )) ρ t,0 (D it ≤ T -t|X) Y it -τ (0, X it , S it ) + (1 -A it ) T t=1 r t • (1 -e t (X it )) τ (0, X it , S it ) -τ (0, X it ) + τ (0, X it ) .Step 0. A basic decomposition.τ 1 -τ 1 = 1 N T t=1 nt i=1 A it 1 (Dit≤T -t) T t=1 r t • e t (X it ) ρ 1 (D it ≤ T -t|X) Y it -τ (1, X it , S it ) (A.1) + A it 1 (Dit≤T -t) T t=1 r t • e t (X it ) ρ 1 (D it ≤ T -t|X) -1 τ (1, X it , S it ) -τ (1, X it , S it ) (A.2) + A it T t=1 r t • e t (X it ) τ (1, X it , S it ) -τ (1, X it ) (B.1) + A it T t=1 r t • e t (X it ) -1 τ (1, X it , S it ) -τ (1, X it , S it ) -( τ (1, X it ) -τ (1, X it )) (B.2) + τ (1, X it ) -τ 1 (C.1).Note that for τ 2 , we have a similar decomposition; we omit the presentation for conciseness.Step 1. Asymptotics for the nuisance terms.Step 1.1. Convergence of weighted propensity scores. Under Condition (2), we prove the following results:T t=1 r t e t (X it ) ρ 1 (D it ≤ T -t|X) = T t=1 r t e t (X it )ρ 1 (D it ≤ T -t|X) + o p (1), T t=1 r t (1 -e t (X it )) ρ 0 (D it ≤ T -t|X) = T t=1 r t (1 -e t (X it ))ρ 0 (D it ≤ T -t|X) + o p (1), T t=1 r t e t (X it ) = T t=1 r t e t (X it ) + o p (1), T t=1 r t (1 -e t (X it )) = T t=1 r t (1 -e t (X it )) + o p (1).WLOG, we only prove the first one.T t=1 r t e t (X it ) ρ 1 (D it ≤ T -t|X) = T t=1 r t ( e t (X it ) -e t (X it )) ρ 1 (D it ≤ T -t|X) + T t=1 r t e t (X it )( ρ 1 (D it ≤ T -t|X) -ρ 1 (D it ≤ T -t|X) + T t=1 r t e t (X it )ρ 1 (D it ≤ T -t|X).By Condition (2), we haveT t=1 r t e t (X it ) -e t (X it ) ρ 1 (D it ≤ T -t|X) ≤ max t,x | e t (x) -e t (x)| • T t=1 r t ρ 1 (D it ≤ T -t|X) = o p (1). Meanwhile, T t=1 r t e t (x)( ρ 1 (D it ≤ T -t|x) -ρ 1 (D it ≤ T -t|x)) = T t=1 1 n t (x) nt i=1 r t e t (x) 1 (Dit≤T -t) A it e t (x) -ρ 1 (D it ≤ T -t|x) 1 (Xit=x) .The summands in the outer layer, indexed by the t, form a martingale difference sequence. Hence, its variance is given byT t=1 r 2 t n t • E 1 (Dit≤T -t) A t -ρ 1 (D it ≤ T -t|x) e t (x) 2 | X t = x p(x) • E n t n t (x) ≤C • max t r 2 t • T t=1 p(x) n t • n t p(x)(n t + 1)where we applied E1 1 + Bin(n, p) ≤ 1 (1 + n)p =O( 1 n ).Therefore, we haveT t=1 r t e t (X it ) ρ 1 (D it ≤ T -t|X) = T t=1 r t e t (X it )ρ 1 (D it ≤ T -t|X) + o p (1).Step 1.2. Convergence of the mean functionals. In this part, we prove the following results:τ (1, x) -τ (1, x) = O p ( 1 √ N ), τ (0, x) -τ (0, x) = O p ( 1 √ N ), τ (1, x, s) -τ (1, x, s) = O p ( 1 √ N ), τ (0, x, s) -τ (0, x, s) = O p ( 1 √ N ).WLOG, we check the first result.τ (1, x) -τ (1, x) = N -1 T t=1 nt i=1 (Y it (1) -τ (1, x))A it 1 (Dit≤T -t) 1 (Xit=x) N -1 T t=1 nt i=1 A it 1 (Dit≤T -t) 1 (Xit=x).The numerator forms a martingale difference triangular array:W (n) it = (Y it (1) -τ (1, x))A it 1 (Dit≤T -t) 1 (Xit=x) , F (n) it = σ{(Y js (•), S js (•), X js , A js , D js ) s≤t,j≤i }.For convenience we denoteF (n) 0s = F (n)ns-1(s-1) , then we can verify thatE W (n) it | F (n) (i-1)t = 0, for 1 ≤ t ≤ T, 1 ≤ i ≤ n t .The variance of the numerator is then given by1 N T t=1 r t σ 2 (1, x)E { e t (x)} ρ ⋆ 1 (T -t | x)p(x) = σ 2 (1, x)p(x) T t=1 r t e t (x)ρ ⋆ 1 (T -t | x) + o(1) N .Therefore,N -1 T t=1 nt i=1 (Y it (1) -τ (1, x))A it 1 (Dit≤T -t) 1 (Xit=x) = O p ( 1 √ N ).For the denominator, similarly, we can show thatN -1 T t=1 nt i=1 A it 1 (Dit≤T -t) 1 (Xit=x) = p(x) T t=1 r t e t (x)ρ ⋆ 1 (T -t | x) + O p ( 1 √ N ).Therefore,τ (1, x) -τ (1, x) = O p ( 1 √ N ).Step 1.3. Concentration of propensity weights.We prove that1 N T t=1 nt i=1 1 (Xit=x) 1 (Dit≤T -t) A it T t=1 r t e t (x) ρ 1 (T -t|x) -1 = O p ( 1 √ N ), 1 N T t=1 nt i=1 1 (Xit=x) 1 (Dit≤T -t) (1 -A it ) T t=1 r t (1 -e t (x)) ρ 0 (T -t|x) -1 = O p ( 1 √ N ), 1 N T t=1 nt i=1 1 (Xit=x) A it T t=1 r t • e t (x) -1 = O p ( 1 √ N ), 1 N T t=1 nt i=1 1 (Xit=x) 1 -A it T t=1 r t • (1 -e t (x)) -1 = O p ( 1 √ N ).WLOG, we prove the first result. We can compute1 N T t=1 nt i=1 1 (Xit=x) 1 (Dit≤T -t) A it T t=1 r t e t (x) ρ 1 (T -t|x) -1 = T t=1 r t e t (x)ρ ⋆ 1 (T -t | x)p(x) + O p ( 1 √ N ) T t=1 r t e t (x)ρ ⋆ 1 (T -t | x) + O p ( 1 √ N ) -p(x) + O p ( 1 √ N ) =O p ( 1 √ N ).Step 2. Approximate the main terms. We prove that the sum of (A.1) and (B.1) can be approximated by the following:EQUATION1 N T t=1 nt i=1 A it T t=1 r t • e t (X it ) τ (1, X it , S it ) -τ (1, X it ) = 1 N T t=1 nt i=1 A it T t=1 r t • e t (X it ) τ (1, X it , S it ) -τ (1, X it ) + o p ( 1 √ N ).The first approximation can be done by simply noticing that1 N T t=1 nt i=1 A it 1 (Xit=x,Sit=s) 1 (Dit≤T -t) Y it -τ (1, x, s) • 1 T t=1 r t • e t (x) ρ 1 (D it ≤ T -t|X it ) - 1 T t=1 r t • e t (x) ρ 1 (D it ≤ T -t|X it ) = O p ( 1 √ N ) • o p (1) = o p ( 1 √ N ).The second approximation can be done similarly.Step 3. Compute the variance of the main part.Step 0-2, we have the following approximation for τ -τ :τ -τ = τ 1 -τ 1 -( τ 0 -τ 0 ) = τ 1,apr -τ 1 -( τ 0,apr -τ 0 ) + o p ( 1 √ N ),whereτ 1,apr = 1 N T t=1 nt i=1 A it 1 (Dit≤T -t) T t=1 r t • e t (X it )ρ 1 (D ≤ T -t|X it ) Y it -τ (1, X it , S it ) + A it T t=1 r t • e t (X it ) τ (1, X it , S it ) -τ (1, X it ) + τ (1, X it ) , τ 0,apr = 1 N T t=1 nt i=1 (1 -A it )1 (Dit≤T -t) T t=1 r t • (1 -e t (X it ))ρ 0 (D it ≤ T -t|X it ) Y it -τ (0, X it , S it ) + (1 -A it ) T t=1 r t • (1 -e t (X it )) τ (0, X it , S it ) -τ (0, X it ) + τ (0, X it ) .We can verify that the variance of τ 1,apr -τ 0,apr is given by the following form:V( τ apr ) = 1 N • E σ 2 (1, X, S) T t=1 r t • e t (X)ρ 1 (D ≤ T -t|X) + σ 2 (0, X, S) T t=1 r t • (1 -e t (X))ρ 0 (D ≤ T -t|X) + σ 2 (1, X) T t=1 r t • e t (X) + σ 2 (0, X) T t=1 r t • (1 -e t (X)) + (τ (1, X) -τ (0, X) -τ ) 2 + o( 1 N ).To sum up,EQUATION).Step 4. Prove the asymptotic normality of the approximation τ apr upon standardization.We check that the Lyapunov condition holds:1 N 2 V(e) 2 T t=1 nt i=1 E    A it 1 (Dit≤T -t) T t=1 r t • e t (X it )ρ 1 (D ≤ T -t|X it ) Y it -τ (1, X it , S it ) 4    ≤ CM 4 N V(e) 2 E 1 { T t=1 r t • e t (X it )ρ 1 (D ≤ T -t|X it )} 3 ≤ CM 4 N V(e) 2 δ 3 .Similarly, we haveEQUATION1 N 2 V(e) 2 T t=1 nt i=1 E    A it T t=1 r t • e t (X it ) τ (1, X it , S it ) -τ (1, X it ) 4    ≤ CM 4 N V(e) 2 δ 3 , 1 N 2 V(e) 2 T t=1 nt i=1 E    1 -A it T t=1 r t • (1 -e t (X it )) τ (0, X it , S it ) -τ (0, X it ) 4    ≤ CM 4 N V(e) 2 δ 3 , 1 N 2 V(e) 2 T t=1 nt i=1 E |τ (1, X it ) -τ (0, X it ) -τ | 4 ≤ CM 4 N V(e) 2 .Therefore, we have proved that the Lyapunov condition holds and √ N ( τ -τ ) ⇝ N (0, V(e)).Step 5. Prove the convergence of the variance estimator.First, we can prove that1 N T t=1 nt i=1 1 (Dit≤T -t) A it (Y it -τ (1, X it , S it )) 2 { T t=1 r t • e t (X it ) ρ 1 (D it ≤ T -t|X it )} 2 = 1 N T t=1 nt i=1 1 (Dit≤T -t) A it (Y it -τ (1, X it , S it ) -( τ (1, X it , S it ) -τ (1, X it , S it ))) 2 { T t=1 r t • e t (X it ) ρ 1 (D it ≤ T -t|X it )} 2 = 1 N T t=1 nt i=1 1 (Dit≤T -t) A it (Y it -τ (1, X it , S it )) 2 π 1T (X it ) 2 + 1 N T t=1 nt i=1 1 (Dit≤T -t) A it ( τ (1, X it , S it ) -τ (1, X it , S it )) 2 { T t=1 r t • e t (X it ) ρ 1 (D it ≤ T -t|X it )} 2 - 2 N T t=1 nt i=1 1 (Dit≤T -t) A it (Y it -τ (1, X it , S it ))( τ (1, X it , S it ) -τ (1, X it , S it )) π 1T (X it ) 2 =E σ 2 (1, X, S) T t=1 r t • e t (X)ρ 1 (D ≤ T -t|X) + O p ( 1 √ N ).The results follow from the martingale structure and the approximation given in Steps 1.1 and 1.2.Analogously, with similar ways of decomposition, we can prove that1 N T t=1 nt i=1 1 (Dit≤T -t) (1 -A it )(Y it -τ (0, X it , S it )) 2 { T t=1 r t • (1 -e t (X it )) ρ 0 (D it ≤ T -t|X it )} 2 =E σ 2 (0, X, S) T t=1 r t • (1 -e t (X))ρ 0 (D ≤ T -t|X) + O p ( 1 √ N ), 1 N T t=1 nt i=1 1 (Dit≤T -t) A it ( τ (1, X it , S it ) -τ (1, X it )) 2 { T t=1 r t • e t (X it )} 2 =E σ 2 (1, X) T t=1 r t e t (X) + O p ( 1 √ N ), 1 N T t=1 nt i=1 1 (Dit≤T -t) (1 -A it )( τ (0, X it , S it ) -τ (0, X it )) 2 { T t=1 r t • (1 -e t (X it ))} 2 =E σ 2 (0, X) T t=1 r t • (1 -e t (X)) + O p ( 1 √ N ),Moreover, we have1 n T t=1 nt i=1 ( τ (1, X it ) -τ (0, X it ) -( τ 1,OR -τ 0,OR )) 2 = 1 n T t=1 nt i=1 ( τ (1, X it ) -τ (0, X it ) -(τ (1, X it ) -τ (0, X it ))) 2 + 1 n T t=1 nt i=1 (τ (1, X it ) -τ (0, X it ) -(τ 1 -τ 0 )) 2 + 2 n T t=1 nt i=1 (τ (1, X it ) -τ (0, X it ) -(τ 1 -τ 0 ))( τ (1, X it ) -τ (0, X it ) -(τ (1, X it ) -τ (0, X it ))) -(( τ 1,OR -τ 0,OR ) -(µ 1 -µ 0 )) 2 =E (τ (1, X) -τ (0, X) -τ ) 2 + O p ( 1 √ N ).Summarizing the above parts, we can concludeV(e) -V(e) = O p ( 1 √ N ).Proof. The proof is delivered in two steps:Step 1. A sufficient condition for the existence of non-trivial oracle propensity scores in the non-delay stages.For the oracle optimization problem 3, by considering covariates X and S with finite discrete support values, we are essentially solving the following program:min e σ 2 (1, x, s) t r t e t (x)ρ 1 (T -t | x) + σ 2 (0, x, s) t r t (1 -e t (x))ρ 0 (T -t | x) + σ 2 (1, x) t r t e t (x) + σ 2 (0, x) t r t (1 -e t (x)).For simplicity, we can omit the dependence on x and s values and focus instead on the following:min δ≤et≤1-δ σ 2 1 t r t e t ρ 1 (T -t) + σ 2 0 t r t (1 -e t )ρ 0 (T -t) + σ 2 1 t r t e t + σ 2 0 t r t (1 -e t ).We want to compute how many stages it takes to obtain a nontrivial solution for the adaptive experiment. The last D * stages will just come with delay; the first T -(D * + 1) stages will have ρ 1 (T -t) = ρ 1,∞ and ρ 0 (T -t) = ρ 0,∞ . We can condense the first T -(D * + 1) variables as a single one by representing it as an average e:min δ≤et≤1-δ σ 2 1 t≤T -D * -1 r t eρ 1,∞ + t>T -D * -1 r t e t ρ 1 (T -t) + σ 2 0 t≤T -D * -1 r t (1 -e)ρ 0,∞ + t>T -D * -1 r t (1 -e t )ρ 0 (T -t) + σ 2 1 t≤T -D * -1 r t e + t>T -D * -1 r t e t + σ 2 0 t≤T -D * -1 r t (1 -e) + t>T -D * -1 r t (1 -e t ).We want to introduce a solution for e that is bounded away from the boundary. By KKT conditions, for the optimal solution variable e ⋆ , we have ∂L(e; e t ) ∂ee=e ⋆ = - σ 2 1 t≤T -D * -1 r t ρ 1,∞ { t≤T -D * -1 r t e ⋆ ρ 1,∞ + t>T -D * -1 r t e t ρ 1 (T -t)} 2 + σ 2 0 t≤T -D * -1 r t ρ 0,∞ { t≤T -D * -1 r t (1 -e ⋆ )ρ 0,∞ + t>T -D * -1 r t (1 -e t )ρ 0 (T -t)} 2 - σ 2 1 t≤T -D * -1 r t { t≤T -D * -1 r t e ⋆ + t>T -D * -1 r t e t } 2 + σ 2 0 t≤T -D * -1 r t { t≤T -D * -1 r t (1 -e ⋆ ) + t>T -D * -1 r t (1 -e t )} 2 = 0.To sum up, we have proved∥e * N ∥ 2 2 - 2E N λ N ≤ ∥ e N ∥ 2 2 ≤ ∥e * ∥ 2 2 + 2E N λ N .Now letting N → ∞, we know that2E N λ N = O p 1 √ log N .Hence we can conclude thatEQUATIONOn the other hand, EQUATIONIn this section, we provide the true parameters generated from the real data adopted in our synthetic case study. The parameters are summarized in Tables B and 2 .Table 1 : Delay mechanism from the real data Justification: Please see Section 1.EQUATIONGuidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: Please see Section 6.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The creators or original owners of the original dataset used in the paper have been properly credited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.38th Conference on Neural Information Processing Systems (NeurIPS 2024). 
 
 Appendix:  


Decision: I concur with the reviewers' positive assessment of the paper. In particular, I think the problem is very well motivated from practice and merits attention, and I think the paper takes a very reasonable approach and studies it very nicely. Well done! It is thus a great addition to the literature on the subject, and I am very glad to recommend acceptance. There are some edits discussed in the rebuttal that I request the authors undertake. Also, while the authors do a nice job of reviewing the landscape of the literature, they could be more transparent in connections in their technical results. In particular, as one reviewer highlights, the connection of Theorem 1 to Theorems 2.1 and 2.2 of [23] and any significant differences should be discussed upfront (you mention some non-specific technical difficulties in the rebuttal but I did not follow since the decomposition seems to just follow from the ancillarity of the design, no?). Moreover, drawing the connection between an adaptive experiment and a sequence of different experiments and discussing the recent works of Bibaut et al. "Learning the Covariance ..." and "Nonparametric Jackknife ..." would add to the (already very nice) discussion of the landscape. I fully trust that the authors can undertake all this appropriately within the preparation of a camera ready version as this mostly involves adding a couple sentences here and there.

Reviews: review: paper_summary: paper_summary: The paper considers the covariate-adjusted response-adaptive randomization design for settings with delayed outcomes and surrogate outcomes. The paper first characterizes the efficient influence function for estimating the primary outcome under the delayed setting with surrogate outcomes. They then characterize the semiparametric efficiency bound of the estimate. Using the semi parametric efficiency bound they devise an optimization approach to construct an adaptive randomization design that optimizes the semi-parametric efficient bound. They provide a synthetic case study to demonstrate the effectiveness of their approach.
strength_weakness: strength_weakness: The paper leaves out some details which may be obvious but are worth including for clarity. For example, the unconfoundedness and arm-dependent delay assumptions are briefly mentioned in passing, but are not formally defined even though they are used in the statements of the theorems. The authors also do not comment on the tractability of optimization problem that determines the treatment allocation probabilities. Other details omitted include details of the cross-validation set-up used to tune the regularization penalty parameter and the the details of the "No Surrogate" approach in the case study. 

Discussion on the settings where the proposed method has the most benefit is also lacking. The numeric case study primarily studies the impact on sample size on different approaches for estimating the primary outcome. The results show that while there is benefit from using the proposed method, the most naive method with complete randomization also performs better than other adaptive randomization approaches. Adaptive randomization design may be more challenging to implement so a deeper study on settings that benefit the most from the proposed method would shed more light on the benefits and robustness of the proposed approach.
questions: questions: 1. In the numerics, the "No Surrogate" approach seems to perform worse than complete randomization. Is there any insight on why this adaptive approach performs worse than the complete randomization approach? Does the "No Surrogate" approach also account for delays or does it utilize an existing adaptive randomization design approach?

2. The proposed method suggests applying regularization to the oracle optimization program to deal with potential multiple optima. Does the procedure tune the regularization parameter so it produces calibrated treatment allocation probabilities with better estimation error? Does this align cross-validation set-up briefly mentioned in the paper? If so, what is the exact cross-validation set-up and if not how is the the tuning parameter chosen?
limitations: limitations: The authors have adequately addressed limitations.
paper_summary: paper_summary: This paper addresses the problem of covariate adaptive experimental design in the presence of time delayed outcomes. More specifically, we assume that participants are enrolled in waves, and the probability of treatment is given conditional on available user covariates. The target estimand is the average treatment effect for a long term/delayed outcome. To circumvent the issue of covariate adaptive randomization in the absence of observed outcomes, the authors consider a surrogate metrics. To derive the assignment procedure, the authors derive the influence function and semi-parametric efficiency bound, and then derive an objective based on them. A small number of empirical evaluations is provided which show the improvement in variance of the proposed estimator over complete randomization and approaches which do not incorporate surrogate information.
strength_weakness: strength_weakness: The paper focuses on asymptotic results, which is sensible and provides good empirical performance. However, it would be useful to have finite sample analysis as well, since many of the applications of experimental design are sample-starved. It would have also been nice to have seen a slightly larger set of empirical results. The authors provide a small demonstration, but ideally the behavior of the propose algorithm is more rigorously evaluated empirically.
questions: questions: It wasn't clear to me what is being assumed about the size of each wave of participants. I assume that the authors are assuming some conditions on the number of arriving participants? Also that these participants are arriving i.i.d. over time?  Does the proposed procedure give any guidance to the setting where the experimenter can choose the number of participants to enroll at each stage?
limitations: limitations: Yes. See above for questions.
paper_summary: paper_summary: This paper introduces an approach to optimizing treatment allocation for variance reduction, in the context of adaptive clinical trials.  The particular setting is one where there are delays in observing outcomes. These delays that are independent of the outcomes themselves, but they impact efficiency of estimation.  Moreover, there are short-term surrogate variables that can be used to improve estimation efficiency.  In this setting, this paper proposes to estimate the variance of different treatment allocation schemes from an initial stage of experimentation, and then use those estimates to choose an optimal treatment allocation scheme for the remainder of experimentation.

---------------
REBUTTAL UPDATE: I have updated my score from 5->6
strength_weakness: strength_weakness: I noted a few areas of the paper that appeared a bit weaker, or at least warrant some clarification.  I'll number my points to make it easier to respond during the rebuttal period, and put them in rough order of priority.

(W1) Significance / Novelty of Theorem 1: I understood Theorem 1 to be a necessary pre-cursor to the proposed adaptive strategy, but not the main contribution of the paper, so this concern is not as pressing as it could be.  However, it seems that Theorem 1 is essentially the same as Theorem 2.1 of Kallus & Mao 2020 [1], with the minor complication that different observations appear at different times, which leads to some additional notation to ensure the correct $e_{t}(X)$ is used in the relevant denominators.  Since the paper cites [1] (as [23] in the paper), it may be worth clarifying the similarity in the main text, and framing the contributions appropriately (e.g., my read is that Theorem 2 and 3 are the novel theoretical thrust of the paper).  I'm open to clarifications / corrections on this front.

[1] https://arxiv.org/abs/2003.12408

(W2) Significance of the proposed problem setting: From a practical perspective, I struggle to think of a setting where we would expect the causal graph in Figure 1 to hold, where the delay itself provides no information on the outcome $Y$, i.e., delays are not informative. E.g., in the HIV application, the outcome is viral load, and delay is in coming in for a visit (Lines 274-275).  It seems to be a classic case where delay could be caused by more severe illness, giving us some indication of $Y$.

(W3) I found it slightly hard to parse the optimality claim (Theorems 2 and 3 and subsequent discussion).  It's clear that the estimated treatment allocation converges in probability to the optimal treatment allocation, but does that imply that this approach yields the optimal variance?  Could there be other ways to estimate the treatment allocation that also converges to the optimal one, but has better variance properties along the way, or is that somehow ruled out?

(W4) Overall, I would have preferred to see more discussion of intuition and insights, rather than just going through the technical steps to derive the results.  It seemed that some parts could be made shorter to make space for this, e.g., Section 4 could be trimmed down a bit (e.g., define $\hat{\tau}^t(a, x, s), \hat{\tau}^t(a, x)$ and then use that notation in both Stage 1 and Stage 2 to $D^*+1$, or even just give the high-level comment that you estimate via empirical counts among the population where $Y$ is observed.)

In terms of missing intuition, I still lack some intuition for the "signal" being used in the optimization of the treatment allocation, and why we should expect it to vary across time periods.  I suppose the variance reduction comes from some understanding of which combinations of $X, A$ are likely to have missing data by the end due to delay, and so should be prioritized earlier in the treatment allocation process?

(W5) There is a slight mismatch between the discussion in the introduction and the actual setting where the algorithm can be applied, particularly the restriction that the delay must be upper-bounded (see 181-182).  In contrast, Line 75-76 refers to the contribution "optimize the statistical efficiency in the presence of temporary or permanent missingness", where I would have interpreted "permanent missingness" to be $D = \infty$. 

(W6) Interaction (or lack thereof) between delay and surrogates: This is a subjective / aesthetic point, so I place it last, but I was expecting more of a nuanced interaction between the delay and the surrogates.  In particular, the problem seems to factorize cleanly into (a) deriving an efficient estimator and variance lower-bound for any treatment allocation, (b) optimizing the allocation in a backward-looking period of data, and then (c) applying that allocation to the next period of data.  It's not clear the role of surrogates in this recipe, other than the influence they have on (a), because observing surrogates gives you no insight into future delays and therefore the optimal allocation going forward.

Minor nits:
* Algorithm 1 mentions "Problem B" on line 7, but I think that's really just Equation (2), Page 6, line 199.
questions: questions: 1. (W1) Could you clarify the similarity/difference of Theorem 1 to Theorem 2.1 of Kallus & Mao 2020?  It's fine if they are essentially the same, in my view, b/c the main contribution of the paper seems to lie in Theorems 2 & 3, but I just wanted to get some clarification in case I'm missing something.
2. (W2) Is there a real-world scenario you have in mind where you would expect this causal structure to hold (i.e., delays are not informative of outcomes)?
3. (W3) Could you clarify the optimality claim?  Is it just that the allocation converges in probability to the optimal allocation?  Or is there a stronger claim being made about the optimality of the procedure which uses the estimated allocation?
4. (W4) Is my intuition correct, for where the variance reduction is coming from?  Or do you have other intuition on this point?

The authors are welcome to react to my points (W5) and (W6), but I don't think they need to do so, and they factored less into my score.
limitations: limitations: N/A
paper_summary: paper_summary: This article studies how to use interminate surrogate outcomes to estimate causal effects when the primary outcome is delayed in clinical trials. The author first proposes a novel Covariate-adjusted Response-adaptive (CARA)  design that supports efficient estimation of ATE using both surrogate and primary outcomes.  The authors prove the semiparametric efficiency bound for their ATE estimator and demonstrate the proposed design's efficiency through theoretical analyses and a synthetic HIV study.
strength_weakness: strength_weakness: The main weakness of this article is that the presentation is too messy and lacks explanations in multiple places. 
1. In section 3, from line 159 to 179, the author introduces the efficient influence function (EIF) and variance bound in estimating $\tau$. However, it seems like the main motivation of CARA is that the expectation of the delay outcomes and the delay mechanisms are unknown.   It is unclear why EIF and variance bound are presented at the beginning of section 3.

2. Section 4 is also presented in a poor way. The authors go through the design steps with dense notation. The authors should provide more explanation of the steps taken in the design. The current Section 4 only shows that the design allows us to estimate the unknown quantity. But it is unclear how the design improves efficiency.
questions: questions: N.A.
limitations: limitations: N.A.
paper_summary: paper_summary: Clinical trials try to achieve the highest statistical precision using the fewest number of enrolled participants. One way to do this is by assigning more participants to the {covariate, arm} combinations with the highest outcome variance. We don't know the outcome variance before running the trial, which motivates the adaptive CARA method to adaptively assign more units with high outcome variance to treatment. This paper considers the setting where we don't directly observe the outcome, but instead observe a noisy surrogate along with a delayed outcome. The paper proposes a design objective (minimizing the semiparametric efficiency bound) and an explore-then-commit style algorithm for learning then applying a covariate-adaptive randomization using surrogate outcomes. The paper shows that this algorithm attains the semiparametric efficiency bound, and provides synthetic experiments based on an HIV study comparing this algorithm to complete randomization and CARA on observed outcomes only.
strength_weakness: strength_weakness: The empirical results are missing error bars. These should be added to show variation in the results across many replicates. This is especially important for two reasons: (a) to support the claim that the proposed method has lower bias than complete randomization, when the two are extremely close on the plots, and (b) to understand whether the "no surrogate" method is really worse in both bias and variance than complete randomization, as suggested by figure 2. If the "no surrogate" method is actually worse than complete randomization, it would be interesting to have a comment on this in the text.
questions: questions: I had trouble understanding the setup of the real-world study. If I understand correctly, measurements are taken every 6 months. The actual outcome Y is viral load (measured every 6 months), while the surrogate outcome is WHO stage (also measured every 6 months). I'm confused by the claim that WHO stage is available immediately, which I took to mean that the experimenters receive the WHO stage for every patient every 6 months. Does that not require the patient to visit a clinic, which might also lead to missingness? This isn't a question you need to address in your response, but if I have misunderstood something it might be worth clarifying.

Request: Neurips style is for citations to be by (Author, Year) instead of numerical. Please fix this in the next revision.

The introduction states that "we allow D_it = infty" (Line 139), but the algorithm only works with finite, known upper bound on D. It would be helpful for the authors to address the sensitivity of their results to misspecification of D*.
limitations: limitations: Yes

Comments: title: Request for clarification
comment: Thank you for your responses. Overall, I am quite positive on this paper (as my initial score indicates). I will leave my score unchanged.
Great, thank you for the discussion.

The authors have addressed my concerns, and I will raise my score to a 6.
Thanks for the fast response. 

- You are right about the intuition. The idea is just to learn the observed delays and do a subtraction to learn the permanent missing (or censoring) probability. This gives an easier theoretical discussion and also provides a fully nonparametric framework. Putting a parametric form works too and is also a great idea to extend the framework to allow infinite delays. On the algorithm side, it does not change the general pipeline too much except for the delay estimation part. We also need to modify the theoretical argument accordingly.

- Oh we see your point. Yes, based on the confidence band, there is no significant difference in the mean of the bias. **We do not claim that the mean of bias of the proposed method has superior properties.** Our point is that the proposed design and estimator leads to a reduction of variance while maintaining the asymptotical unbiasedness property, and we show this both theoretically and empirically. This is the main message we want to convey, combining both figures.

Re your request, we hope to modify the wording as follows: 

> Figure (A) suggests that the point estimators from all three designs have a vanishing bias as the sample size grows, validating the asymptotic unbiasedness of all strategies. Nevertheless, in terms of variance, Figure (B) demonstrates that our proposed design yields smaller standard deviations and thus higher estimation efficiency for estimating the ATE.

Thanks again for these questions. All the points are well taken and will be incorporated in our next round of discussion.
Thanks for the response.

## Definition of $D^*$
Okay, interesting -- so I'm guessing you can learn the delay mechanism in $D^*$ steps by assuming that every unit which was not observed after $D^*$ steps has infinite delay. I believe you that this makes the theory easier, although it doesn't align with common delay distributions like the exponential. I suppose another approach could be to assume a parametric form for the delay and fit that on the data from Stage 1.

You don't need to respond to this; I'm just sharing some thoughts.

## Bias in rebuttal Figure 2A.

I think we're still not understanding one another. I also agree that all the methods are asymptotically unbiased, but this is not what I meant when I referred to the mean of the bias. Instead, I meant that in Figure 2A, the blue squares (complete randomization) and orange triangles (proposed method) are plotted exactly on top of each other. I expect you computed those points by running finite-sample simulations with N="sample size" observations, computing the actual bias for each simulation, and averaging those biases. Therefore it looks to me like the finite-sample mean bias is the same between complete randomization and the proposed method.

I want to make sure that you're not claiming the proposed method has superior bias properties to complete randomization on the basis of Figure 2A. The proposed method has a _slightly_ wider confidence band on the bias than complete randomzation, but adding variance to the bias cannot be a basis for claiming that the bias is improved.

My final request: Can you tell me how you would rewrite the sentence on line 292-293 in the paper, given the confidence intervals you have presented in this rebuttal? (Specifically, the line "Figure 2(A) suggests...estimating the ATE.")
Thanks for your response.

- Thank you for pointing this out. Apologize that we did not state the definition of $D^\star$ clearly in the paper. For the definition of $D^\star$, it is the upper bound for the finite values that delay can take. In the equation following Line 134, we briefly stated the values that $D$ can take. Moreover, Line 182 of the paper is not correct - it should be stated as $\mathcal{D} = \\{0,1,\dots,D^\star\\}\cup\\{\infty\\}$. 

   From a high level, when the delay takes its value on finite support (no need to be finitely bounded), we can more easily estimate the delay distribution by taking empirical means and avoid unnecessary technicality in our theoretical discussion. 

- Apologize for the ambiguity in our wording. By "unbiased" we want to make the claim of asymptotic unbiasedness of the three approaches. We fully agree that all three methods are asymptotically unbiased or asymptotically consistent for the truth, or in your words, "the mean of the bias is identical". Our claim wants to highlight a bit on the finite sample performance of the estimators in this example; or, how fast can these bias converge to zero. The confidence band suggests the "No surrogate" approach converges slightly slower, which makes sense because the surrogate information is not used in the implementation to mitigate the missingness in primary outcomes. The proposed method and complete randomization behave quite similar. Therefore, while asymptotically all three methods are unbiased, incorporating surrogates can slightly improve the finite sample convergence rate of the final estimator.

Hope the above clarification resolves your concerns. We will also incorporate the updates in the next revision.
Got it. We agree with you about the variance claim - if we have a general $\hat{g}$ there is an additional layer of variance due to the estimation of $g$. Nevertheless, there is some specialty in the IPW and AIPW cases. Here the propensity score and the outcome models are nuisance parameters. For the outcome model, we can achieve $O(N^{-1/2})$ convergence because of the bounded moment condition on the potential outcomes and discrete covariates. Also for the delay distribution $\rho(t\mid X)$, we can achieve $O(N^{-1/2})$ because the delay indicators are bounded, and the covariate and delay have finite support. Then for the propensity score modeling, we only require $o_p(1)$ convergence. The proof is similar to the techniques adopted in double machine learning literature. To see this, consider the following math in the classical AIPW case:
\begin{align}
&\frac{1}{n}\sum_{i} \frac{(Y_i - \hat{\mu}(X_i)) A_i}{\hat{e}(X_i)} + \frac{1}{n}\sum_{i} \hat{\mu}(X_i)-\mu\\\\
=&\frac{1}{n}\sum_{i} \frac{{e}(X_i)}{\hat{e}(X_i)} \frac{(Y_i - {\mu}(X_i)) A_i}{{e}(X_i)} + \frac{1}{n}\sum_{i}\mu(X_i) - \mu \tag{1}\\\\
+&\frac{1}{n}\sum_{i} \\{\hat{\mu}(X_i) - \mu(X_i) \\}\\{ 1 - \frac{A_i}{\hat{e}(X_i)}\\}. \tag{2} \\\\
\end{align}

For (1), we show this part gives the asymptotic minimal variance. We have
\begin{align}
&\frac{1}{n}\sum_{i} \frac{{e}(X_i)}{\hat{e}(X_i)} \frac{(Y_i - {\mu}(X_i)) A_i}{{e}(X_i)} + \frac{1}{n}\sum_{i}\mu(X_i) - \mu \\\\
=& \frac{1}{n}\sum_{i} (1 + o_p(1)) \frac{(Y_i - {\mu}(X_i)) A_i}{{e}(X_i)} + \frac{1}{n}\sum_{i}\mu(X_i) - \mu \\\\
\asymp & \frac{1}{n}\sum_{i} \frac{(Y_i - {\mu}(X_i)) A_i}{{e}(X_i)} + \frac{1}{n}\sum_{i}\mu(X_i) - \mu + o_p(n^{-1/2}).
\end{align}

For (2), we show it's of small order $o(n^{-1/2})$. 
\begin{align}
&\frac{1}{n}\sum_{i} \\{\hat{\mu}(X_i) - \mu(X_i) \\}\\{ 1 - \frac{A_i}{\hat{e}(X_i)}\\}\\\\
= & \sum_{x\in\mathcal{X}}\frac{\hat{\mu}(x) - \mu(x) }{\hat{e}(x)}\frac{1}{n}\sum_{i} \\{A_i - \hat{e}(x)\\}1\\{X_i = x\\}\\\\
= & \sum_{x\in\mathcal{X}}\frac{\hat{\mu}(x) - \mu(x) }{\hat{e}(x)} \cdot \frac{1}{n}\sum_{i} \\{A_i - {e}(x)\\}1\\{X_i = x\\}
+\sum_{x\in\mathcal{X}}\frac{(\hat{\mu}(x) - \mu(x))(e(x) - \hat{e}(x)) }{\hat{e}(x)}\frac{1}{n}\sum_{i}1\\{X_i = x\\}\\\\
= & \sum_{x\in\mathcal{X}} \frac{O_p(n^{-1/2})}{e(x) + o_p(1)} \cdot O_p(n^{-1/2})
+\sum_{x\in\mathcal{X}} \frac{O_p(n^{-1/2}) \cdot o_p(1)}{e(x) + o_p(1)} \cdot \\{p(x) + O_p(n^{-1/2})\\}\\\\
= & o_p(n^{-1/2}). 
\end{align}

Therefore, we can see that (1) plays the dominant role in the asymptotic regime and gives minimal variance. Again, we highlight that the specialty is due to we can estimate the outcome model in square root order so we do not need this order for the propensity score models. 

Hope this clarifies your concern regarding the variance part.
I think again you're focused on consistency while I'm asking about something else - the implications for the asymptotic variance.  Continuing in this simple example, consider discrete X, as you do.  Suppose that $|g(x) - \hat{g}(x)| \rightarrow 0$ in probability, but that for each value of $x$ the convergence is at $O_p(\sqrt{n})$ rates, i.e., $ \sqrt{n} (g(X) - \hat{g}(X)) \rightarrow N(0, \sigma^2)$

Then if you ask about $\hat{\theta} := E\_n[\hat{g}(X)]$ and $\theta\_0 := E[g(X)]$, the analysis I was looking for is something like 
$$\sqrt{n} (\hat{\theta} - \theta) \rightarrow N(0, \sigma\_\theta^2),$$ and showing that the variance term is minimal.  Here there are two sources of variation, because $$\sqrt{n}(\hat{\theta} - \theta) = \sqrt{n}(E\_n[\hat{g}(X)] - E\_n[g(X)]) + \sqrt{n}(E\_n[g(X)] - E[g(X)])$$  Then the second term is what you're talking about:  It has minimal variance, because you're using the right $g(X)$.  But the first term still contributes to the variance.  Feel free to correct me if I'm thinking about this incorrectly.
Thank you for your responses. I have two remaining questions:

* $D^* = \infty$: Your response suggests that D* can be infinite, as long as the delay takes only a finite number of values. Line 183 says that the algorithm requires at least T=(2D* + 2) steps. If D* is infinite, how would the algorithm ever move to Stage 2, which happens after D*+2 steps?

* Error bars. Thank you for adding these. My read of Figure 2 in your pdf is that complete randomization and the proposed method have identical bias, given the width of the confidence intervals. In other words, it's not possible to say that the bias of one method is better or worse than the other. I'm not sure what you mean by the claim "the complete randomization design becomes unbiased when the sample size is over 800, while our proposed design is unbiased at a much smaller sample size." I suspect this claim might be because the _confidence interval_ for the bias crosses zero slightly earlier for the proposed method than the completely randomized design. However, I think that interpretation of the plot is incorrect. What we actually care about is the mean of the bias, which is essentially identical for the two methods. I think the claims about smaller bias are not justified by the plots. The claims about variance (Fig 2B) are well-supported.
Thanks very much for your quick response and for raising the score!

Your concern regarding the estimation problem for $E[g(X)]$ is correct for a general $X$. But in our case, we take a simplification and consider just **discrete** $X$ with finite support $\mathcal{X}$ (first paragraph of Section 4) to avoid unnecessary technical complexity. To see why this makes things go through, suppose we have $|\hat{g}(x) - g(x)| \to 0$ for every $x\in\mathcal{X}$. We can compute
\begin{align}
 & \frac{1}{n}\sum_{i=1}^n \hat{g}(X_i) \\\\
& = \frac{1}{n}\sum_{x\in\mathcal{X}}\sum_{i=1}^n \hat{g}(x) 1(X_i = x) \\\\
& = \frac{1}{n}\sum_{x\in\mathcal{X}}\sum_{i=1}^n \\{{g}(x) + o_p(1)\\} 1(X_i = x) \\\\
& = \sum_{x\in\mathcal{X}}g(x)p(x) + o_p(1)\\\\
& = E(g(X)) + o_p(1). 
\end{align}
Therefore, when the support of $X$ is finite, the technical part is greatly simplified. Nevertheless, for continuous $X$, we will need more conditions as you suggested, such as root-$n$ convergence of the nuisance, and more steps to achieve the optimal variance such as sample splitting. 

We hope this short discussion clarifies your concern and will include all the points in the next revision.
Thank you for the thoughtful response, a quick point-by-point of my reactions.  Overall I'm positive on the paper, and I'm going to raise my score to a 6.  

The one point I'm still fuzzy on is (W3):

Do you need any other conditions to say something about the asymptotic variance being optimal?  To clarify my question with an example:  If I want to estimate $E[g(X)]$ for some deterministic function g, then obviously the empirical mean $E_n[g(X)]$ has optimal variance, but it's not usually the case that the plug-in $E_n[\hat{g}(X)]$ has optimal variance, even if $\hat{g}$ is consistent/converges in probability to $g$.  Similarly, you're saying that the allocation converges in probability to the optimal one, and that if you had the optimal allocation, it would minimize variance, but I struggle to see how that's different from my example, where I could say "$\hat{g}$ converges to $g$, and using $g$ gives optimal variance, therefore my procedure is also optimal" but obviously that's incorrect.  That's why I was wondering if there are other conditions, e.g., in my example above you might require that the stronger condition that $\hat{g}$ converges to $g$ at $o_p(n^{-1/2))$ rates so it doesn't contribute to the asymptotic variance, etc.

For the remaining points, thank you for the clarifications, my quick takes:
* (W1) That seems reasonable enough as a distinction, and I would clarify that point in the paper, as you state.
* (W2) Fair enough, I would suggest some discussion in the paper as space permits, to at least highlight reasons why the assumption might be considered a strong one.
* (W4) I like the example!  Would be nice to include somewhere for intuition, if space permits.
* (W5-6) Makes sense, understood, good clarifications all around.