Causal Inference in the Closed-Loop: Marginal Structural Models for Sequential Excursion Effects
Authors: Alexander W. Levis
Gabriel Loewinger
Francisco Pereira
Venue: NeurIPS 2024 poster (2024)
Keywords: marginal structural models
optogenetics
excursion effects
neuroscience
dynamic treatment regimes
micro-randomized trials
sequentially randomized experiments
Primary area: causal_inference
TL;DR: We propose a non-parametric causal inference framework for closed-loop optogenetics behavioral experiments to enable excursion effect estimation for treatment sequences greater than length one in the presence of positivity violations.

Abstract: Optogenetics is widely used to study the effects of neural circuit manipulation on behavior. However, the paucity of causal inference methodological work on this topic has resulted in analysis conventions that discard information, and constrain the scientific questions that can be posed. To fill this gap, we introduce a nonparametric causal inference framework for analyzing "closed-loop" designs, which use dynamic policies that assign treatment based on covariates. In this setting, standard methods can introduce bias and occlude causal effects. Building on the sequentially randomized experiments literature in causal inference, our approach extends history-restricted marginal structural models for dynamic regimes. In practice, our framework can identify a wide range of causal effects of optogenetics on trial-by-trial behavior, such as, fast/slow-acting, dose-response, additive/antagonistic, and floor/ceiling. Importantly, it does so without requiring negative controls, and can estimate how causal effect magnitudes evolve across time points. From another view, our work extends "excursion effect" methods---popular in the mobile health literature---to enable estimation of causal contrasts for treatment sequences greater than length one, in the presence of positivity violations. We derive rigorous statistical guarantees, enabling hypothesis testing of these causal effects. We demonstrate our approach on data from a recent study of dopaminergic activity on learning, and show how our method reveals relevant effects obscured in standard analyses.

Causal Inference in the Closed-Loop: Marginal Structural Models for Sequential Excursion Effects
Alexander W. Levis" Carnegie Mellon University alevis cmu.edu
Gabriel Loewinger" National Institutes of Health gloewinger cmm
Francisco Pereira National Institutes of Health francisco.pereira @nih.gov
Abstract
Optogenetics is widely used study the effects of neural circuit manipulation on behavior. However, the paucity of causal inference methodological work on this topic has resulted in analysis conventions that discard information, and constrain the scientific questions that can be posed. To fill this gap, we introduce nonpara- metric causal inference framework for analyzing "closed-loop" designs, which use dynamic policies that assign treatment based on covariates. In this setting, standard methods can introduce bias and occlude causal effects. Building on the sequen- tially randomized experiments literature in causal inference, approach extends history-restricee marginal structural models for dynamic regimes. In practice, our framework can identify wide range of causal effects of optogenetics trial-by- trial behavior, such as, fast/slow-acting. dose-response, Aaditive/antagonistiic,, and floor/ceiling. Importantly, it does without requiring negative controls, and estimate how causal effect magnitudes evolve across time points. From another view, work extends "excursion effect" methods-poople literature-to enable estimation of causal contrasts for treatment sequences greater than length one, in the presence of positivity violations. We derive rigorous statisti- cal guarantees, enabling hypothesis testing of these causal effects. We demonstrate our approach on data from a recent study of dopaminergic activity on learning, and show how our method reveals relevant effects obscured in standard analyses.
in the mobile health
Introduction
Optogenetics is neuroscience technique "turn on/off" neurons vivo in real-time, with mil- lisecond time resolution. It works by shining lasers on neurons that have been genetically modified through viral infection to express a light-sensitive protein. is one of the most popular assays with roughly 700 references to it in 2023 alone Optogenetics is often applied to study the causal effect of manipulating specific brain circuits while animals (e.g., mice) perform behavioral tasks to study, for example, learning and decision decision-making These tasks typically composed of sequence of trials, E 21, ..., T), each of which involves presentation of stimuli and an opportunity for a behavioral response. For example, trial might begin with a cue (e.g., light), which indicates that a lever press will trigger delivery of a food reward. Investigators might want to know, for instance, whether applying optogenetic stimulation which mice press the lever. On trial f, an animal's behavioral outcome, Y,, time-varying covari- ates, X,, and optogenetic (treatment) indicator, A,, observed. Experiments often include both treatment (G 1) and negative-control (G = 0) groups, with animals assigned randomly to each. While the laser (i.e., the sequential treatment, A,) is often applied on randon subset of trials in
random subset of trials alters the rate
Authors contributed equally, names alphabetized
2 2699 papers on Web of Science mention "optogenetics" in 2023 and between 526 796 in years 2015-2023, Some studies deterministically laser/no-lase trials but focus "stochastic" experimental designs.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Offect
D. Closedi Loop (Dynamic Regime) Design DAG
Pija Xio
Effect Divalpation
X
Bliver
hewf
Marginal
X
G.Dose Rexponse
Yara
No P
P4 Rio
[0
(0
Past
Fiad
1
Y;
Margfnaiize over all data through triall 6-á
Determiniaticalyy "sot" policy for & trieta
Estimate coussel effect of intervening last trials
Effect Size
Figure 1: Sequential Excursion Effects. [A][[]] The left panels show one setting where a sequence of laser simulations do or do not have the indicated effect on the outcome. The middle panel shows deterministic static policies that could be used to construct a causal contrast to probe the effect. The right panel shows what the anticipated effect size (darker is larger) of the contrast might be if the effect was was not present. [A] Blip Effeet: the effect of single stimulation vs. no treatment on a recent trial. [B] Effect Dissipation: Whether the effect of single stimulation causes an effect that rises and dissipates after a few trials, or persists. [C] Dose Response: Do successive simulations increase the response in dose-dependent fashion? [D] Closed-loop design DAG for trials. U is an unmeasured variable. [E] HR-MSM illustration inspired by figure in (6)
both groups, only treatment group animals (G 1) express the protein that enables the laser to trigger the target neural response. The control group thus controls for toff-target" effects such as the laser heating the brain, and the optogenetic insertion surgery. To answer the question above, investigators often estimate the effect of optogenetic manipulation through comparisons such as common to test whether 1h = at specific timepoints like the end of the study (t T), or to conduct inference on summaries (e.g., v + E. 1/2). These between- between-group comparisons assess the intervention impact based on simple long-term, or "macro""""lloba" longitudinal effects. When studies randomly deliver treatment at each trial (i.e., with stochastic policies), within-group comparisons between laser and no-laser trials are also common (e.g., 30 E. [E A, = 1, = R A: = 0, G G=11 =11]).
th = B[y G == E G = Oj.
Importantly, such comparisons do not lend themselves to testing within-group "micro"/"locall longitu- dinal effects related to specific treatment sequence patterns. For example, one might ask whether there is a dose-dependent relationship between the outcome and the number of stimulations in the last five trials, or whether stimulation on two consecutive trials has a synergistic effect that is greater than stimulation instead occurred on two non- conccsseeeutive trials. Figures [ shows some representative micro longitudinal effects that are identifiable in many optogenetics studies, yet typically are not explored. Critically, such effects may be present even in studies in which one fails to detect the macro effects commonly tested. However, no formal causal inference framework has been applied to these studies, resulting in analysis conventions that limit the scope of questions researchers can ask.
Furthermore, certain experimental designs can complicate the use and interpretation of even standard analysis approaches. In "closed-loop" (referred to as "dynamic regimes" in the causal inference literature) designs, stimulation is applied depending on the behavior of the animal. For example, say study tests if lever pressing for food, Y,, decreases if optogenetic stimulation (A, = 1) is applied, with positive probability, only when animals approach the lever (X, = 1). Since A, is randomized conditional on Xt, one must incorporate X, into their analysis, but standard strategies like including X, as a covariate in regression can obscure effects and induce bias. This because 1) X, influences the probability of both the outcome and treatment, and thus can be cast time-varying confounder, and 2) X, also mediates the effect of prior treatments 77: see the illustrative DAG in Figure DP, though note that generalize this setting later on to allow treatment to depend on the

---

complete history of previously measured variables at each time point. In this case, since treatment also influences both Y, and X, on subsequent trials, closed-loop designs induce treatment-coofooudder feedback" 77. which can lead to bias with standard analyses. We include an example in Appendix B to show how, when the treatment has opposing effects on Y, and X,, treatment and control groups can exhibit identical average (observed) outcome levels even if the laser causes a large immediate effect. Furthermore, standard regression approaches can actually induce collider-bias, and block mediators of the treatment effect 77. Finally, if treatment policies deterministically rule out treatment (e.g., when X, = 0), certain effects are not identifiable: the positivity violation inherent to these designs precludes estimation of certain counterfactual distributions. Closed-loop designs therefore require specialized algorithms for valid causal inference.
More broadly, there have been a number of high profile calls for more rigorous definitions of causality and causal inference in neuroscience
[6] However, to the best of our knowledge, existing methodologie oogicall work [25 10 14 focuses on instrumental variable-basee
approaches to estimate causal effects of optogenetics on neural activity. Unlike our setting, these methods are restricted to datasets that include both measurements of the activity of the neurons stimulated by optogenetics, and the neurons those cells interact with. One can then conceptualize the neural activity of the stimulated neurons as treatment variables, and the optogenetics sequence as instruments. In addition to focusing on behavioral outcomes, we explicitly deal with sequentially randomized (and closed-loop) designs, exchangeable draw, ignoring the sequential nature of trials.
whereas prior work treats each trial
Our contributions are (1) proposing the first formal counterfactuall-assee causal framing of these behavioral optogenetics designs, (2) developing an analysis framework based on history-restricted marginal structural models that enables the estimation of "sequential excursion effects" that capture the local causal contrasts described above, (3) expanding excursion effect methodology to account for positivity violations, and to accommodate treatment sequences greater than length one, (4) providing estimators with efficient computational implementations and strong theoretical guarantees under minimal nonparametric conditions (verified in simulations), and (5) applying our methods to data from a high profile Nature paper, and showing how they reveal effects obscured by standard methods.
2 Notation and Related Work
In this section, we (i) provide the necessary notation and a brief review of relevant work, and (ii) describe the key methodological gap the current literature: existing methods cannot estimate causal effects of proximal treatment sequences longer than one timepoint in closed-loop designs.
Notation Let O. = (X,, Ar, Yr be the vector of observed variables for an animal on trial t. We denote T as the number of trials and [T]] as the set [1, 2.,,...; T.. A sample of subjects i == 1, 2..., collected but, as subjects are exchangeable, we often suppress indices to reduce notational burden. We express counterfactual variables, or potential outcomes, with parentheses. For example, Ya:)). represents the potential outcome that would be observed at trial if subject received the treatment sequence, a, (a1, "........... at). Overbars represent all history up to and including a given trial. For example, B, = (B1 By), for any sequence of variables (B, ) and any j E [T]. Finally, define H, (X,, At-1, V.-1), H, includes all information prior to the treatment "decision" at I.
Relevant Literature Marginal structural models (MSMs) are often used to model the mean counterfactuals E[Y, [ 30 29] in sequentially randomized experiments, though these typically do not perform well 21 when there are a large number of time points (e.g.. as in many optogenetics studies): the variance of the model coefficients can grow prohibitively large. Iiittor--rstrriteed MSMs (1 (HR-MSMs) model EaY,(a,,+)) for some a.a,- = (a: 11 ae), typically with A &< t. That is, HR-MSMs model the mean counterfactual outcome at time I, under an intervention defined on a proximal (often short) treatment sequence. As
= (A, ass:)), by consistency assumption, these estimands implicitly marginalize over the observed treatment sequence, Ā,-A- prior to the first point of intervention. However, any Markov-like assumptions made by the causal framework follow directly from the experimental design: HR-MSMs (and, by extension, our proposed methods) allow for X,, Y, to be causally affected by all prior trials (i.e., O, for j E [t- - 1)). By placing structure on E)),(a the HR-MSM can borrow strength across treatment sequences aa.t. which can increase power when there are many trials. Figure [E provides graphical illustration of -SMM. HR-MSMs are typically fit by using generalized estimating equations (GEE) with
3

---

inverse probability of treatment weighting (IPW). IPW resolves the dilemma with standard regression techniques sequentially randomized experiments, outlined above, where failure to condition on time-varying confounders, X₁, biases estimates (as treatment is randomized conditional on X in closed-loop designs), but conditioning on X, induces confounding (X; are colliders on the path between past treatments and subsequent outcomes, through unmeasured confounders, U, as shown in the DAG in Figure [[]) 77. HR-MSMs can also incorporate time-varying effect modifiers (e.g., see [33 and references), to test, for example, whether causal effects vary across trials, or animal-specific covariate levels.
The gaps: Sequential effects and positivity violations In designs that assign treatment randomly at each trial, HR-MSMs can be used to estimate the causal effect of specific deterministic treatment sequences as. that may differ from the observed sequence A, close to trial t, and are compatible with the experimental treatment rule ("policy")). Importantly, this enables estimation of interpretable causal parameters, such as the effect of treatment on the most recent trial, E [az = 1) - Yj(a, = O)]. These causal contrasts have grown popular recently in the analysis of mobile health studies (5). where they are referred to as "excursion effects." " However, current methods are restricted to estimating excursion effects for the A = 1 case in experimental designs like ours, and thus preclude estimation of effects defined only for A > 1 (e.g., the micro longitudinal effects in Figures I and Mobile health studies often include treatment rules with positivity violations: due ethical practical constraints, treatment must be withheld in certain cases (e.g., no phone notifications while driving). [5] use the notation that treatment is withheld when the time-varying "availability" indicator, 4t- equals zero. Similarly, in "closed-loop" optogenetics experiments, 4, = when the conditions are met such that neural manipulation may occur (e.g., when the animal approaches the lever in the example in Section []. There have been proposals for methods intended to account for estimand [5]: E (a, 1) - Ya(a; 0) I 1]. However, estimands proposed for these settings are defined only for = Thus, the presence of these positivity violations, there is currently no methodology to conduct causal inference for longer proximal treatment sequences. We note that machine learning based causal methods including causal transformers [8], counterfactual recurrent networks (3] and recurrent marginal structural networks [15] are comparable to HR-MSMs that condition on all measured variables prior to the first intervention timepoint (t A + 1). These methods target effects of static treatment sequences and require a positivity assumption, and thus cannot be applied in closed-loop designs. They also do not provide tools for statistical inference.
such implied positivity violations 19
5 26 such as the cnnilaablity-conditinn
3 Methods
To fill the gaps identified above, we propose HR-MSMs for proximal sequences of dynamic treatment regimes, designed to be compatible with treatment availability restrictions in this scientific context. These estimands are defined for any A 2 1, can incorporate time-varying effect modifiers, and can dissect more intricate patterns of treatment over time, compared to standard excursion effects.
3.1 HR-MSMs for Dynamic Treatment Regimes
Adopting the notation from 55, we define I, == L(P[A, = 1 H. > 0) an "availability indicator", i..., I, = 0 if and only if active treatment (e.g., laser stimulation) prohibited by design. Define D. [de: H. {0, 1) de = 0 if I, = 0}. for any I, to be the class of treatment rules at time compatible with I. In particular, we will consider the deterministic rules D; = (d.) d')) C Dr, where d(()) 0. d.2) = Jr. In words, d(0) fixes A, 0, and d(1) sets A, equal to 7,. The treatment rules d()) ((")) E D, represent the two most extreme policies whose effects remain identifiable. We can combine these time-specific rules to construct multiple time-point analogs of excursion effects compatible with availability restrictions: for A E N, we let DA. be a subset of D; X ... x D;, taking da,: (di-At).... d,) DA, to be sequence of A treatment rules (compatible with availability restrictions) for trials j E (t A + 1,....). l). The counterfactual outcome under this policy sequence is defined to be
Yi(ds.) Y((A). A d (HI, ..... . (H.(())-
(I)

---

That is, Ye(ds.e) is the counterfactual outcome under an intervention that leaves the natural value of treatment for the first t A trials, then sequentially determines treatment by applying di-A+5 to H (dj-,,--tt--)- for j [ where djj1,t-Ati-1
(d+ -on-A+1...--- d.-At----)) -+)-1)-
Letting V2 H, be a set of effect modifiers at trial t, we seek to estimate EdY;(da,)) IV, V1- the counterfactual mean outcome, conditional on effect modifiers that are observed before the treatment decision of trial f - A + 1. By construction, these estimands are identifiable under standard causal assumptions (see Section 3.2 We discuss their interpretation, and compare with existing proposals in Appendix C.T When A sequence combinations. We thus propose to estimate effects of these interventions with an MSM on the (conditional) means of the counterfactuals (I): m(t d V 3) z (dy,(d,,)) V,-A+1l; where m is a fixed known function. We aim to conduct inference on the MSM parameters, B, but we do not assume that the model is necessarily well specifid,, and thus treat the MSM parameters as projections onto the working model [2 32):
and studies have many trials, there may be many potential treatment rule
B. arg min BERO Σ
Σ dA..E
E da, V2 a1) (Yi(ds.) - m(t, da VL- (=a+1:3)))) (2)
for some fixed non-negative weight function h. This projection approaches lies between fully parametric strategy, that assumes m correctly specified, and a fully nonparametric approach, that places no structure across the target causal quantities. The target Bo is defined as the parameter of the best fitting working model m (i.e., closest in (P)). In practice, the choice between considering working model or as correctly specified model amounts to trade-off between bias and variance- see the discussions in [2] where analogous projection parameters are proposed.
3.2 Identification and Estimation
In this section, we first describe the causal assumptions under which the effects of interest are identified. We then develop an inverse babbability-weighhee estimator of the MSM parameters, and derive their asymptotic properties. While we focus on dynamic regime HR-MSMs below, our results also apply to static regime HR-MSMs in the case that there are no availability issues (i.e., I = 1). There, the treatment rule da.t r reduces to corresponding static sequence as,r.
For each I, define the treatment probability function 5e (a H₂) P(A, a H.). We make the following standard assumptions, which are expected to hold in many optogenetics designs:
Assumption 3.1. Consistency: (ids,.) = Yú, whenever A₁ = dy (H,)), for all j E (t-A+1,.... Assumption 3.2. Positivity: For all t E (A...., T), and d; E Dj. ne (H,) H₂) c, w.p.
Assumption 3.3. Sequential randomization: A. IL (d(d,,) H for all t E {A...., . N....,T,. (1 - A + 1,....,}
We provide a detailed discussion of these assumptions in practice in Appendix The following result says that these three assumptions are sufficient for identification of the counterfactual means (ds,() Vi-A+1). and of the MSM parameters By-
Proposition 3.4. Under Assumptions 3.7
3.3
we have
B(Y((da.)) 1 = R.
II
IA) H,,
Yi Vi-An)
J=t-A+1
Recall that Z; = (Or,iT is the totality of data observed on subject i; suppressing spejecc-specific index for clarity, define o( (Z. R2 + R29 via
(L,
Σ
1(1, VI M(, da,, V.l.,,V-AH. Vi-A+199)
da.tEDs
II J-i-A+1
d( <s(As; Hj)
(Yt-m(t, m(t, da.,, V=-
where (L,, da.., V Vam(l da V;,++;; unique, and the working model m differentiable in B, the MSM parameters Go are identified through the estimating equation 0- Ep (o(Z, Bo)).
3). Then, assuming the solution to 2 is
5

---

The result of Proposition 3.4 is population inverse probability wwenghte estimating equation for the target parameters Pp. This estimating equation motivates corresponding IPW estimator, B, solving the empirical IPW estimating equation IP,, (p[[((Z, ((, 3) O. In the optogenetics applications of interest, the propensity scores F. are known by design, and can be plugged in when estimating B.
We now prove asymptotic normality of the our estimator, 3., under mild conditions. We require the following notation: define A(3) (Z (), 33)]] and ( E (),
Theorem 3.5. Suppose Assumptions 3.7 3.3 and the following conditions hold:
(i) The minimizer Bu in is unique;
(ii) m(t, V is Donsker in B, continuously differentiable at Go. uniformly in Vi-++1i
(ii) In neighborhood around Bo. A(B) and ((3) are finte-valled, and B(3) is non-singular;
(iv) 3 2 Bo.
Then vn(l3 Bo) 4 No v(B)),, where V(3) B(B))
Theorem 3.5 gives the asymptotic distribution of the estimator 3. The conditions (i)-(iv) are relatively mild; see Appendix C.3 for discussion. Theorem 3.5 provides a strategy construct asymptotically valid Wald-based confidence intervals for the MSM parameters By: for any ,3 we can take
A(3) = P),
B(3) I. (Z,
and define V = B(3) - A(3) which is consistent for V ((3)). Then, for E [g], an (1-a)
confidence interval for 3,0 given by B, 21-a/2 Yie, where ø1-a/2 is the (1 - -12)aa/--uannlll
of the standard normal distribution, and V3J is the j-th diagonal element of V. Confidence intervals for any linear combination of the 33 parameters can be constructed in similar fashion.
We provide implementation that builds the necessary dataset (with each observation copied once for every regime in Da.)). calculates the corresponding IPW weights, and estimates the HR-MSM parameters 3 by solving the estimating equation in expression] 2 using the rootSolve R package The process takes about 10 seconds standard laptop, for > 100, 000 total (pre-copy) trials.
11.
4
Experiments
4.1 Simulation Studies
Experimental Setup We sought to assess performance of the proposed estimator, and identify
variance estimators that yield nominal coverage in the small settings common in optogenetics studies. To evaluate the accuracy of our framework in estimating mean counterfactuals, we designed the simulations such that the target cecimannds-contarn to regression coefficients from the true HR-MSM. The data were simulated to mimic closed-loop optogenetics designs with positivity violations: we drew i) Xo Bernoulli(1/2); ii) A, X, Bernoulli( XX),, for E {0.... T); lii) X₂ Ar-1 ~ Bernoulli + 0.4A,-1), for t E [T]: and iv) Y. Xt-1,A1-1 X,, A, ~ N(ax + A2 + Xx + A, o?), for t E [T]. where (a1, 022, a3, .4) = (0.25, 1..55,..55 and o for all These set availability indicator I = X,, for all t, and result in marginal probabilities P = 1] 1. P(A, 1] 7. for all t. We obtain a closed form for the parameters of the saturated two time-point dynamic treatment regime
of mean
HR-MSM: letting d2 = (d-1,, dc) D2, be arbitrary, and defining J,1 = [d4-1 = d
J = (dd = d(²), we derive in Appendix D. that ((Y = Bu + J- 32J + JJJ+-1Jh where So 0.5an 0.403, Bn 0.5cz 0.20%,, B2 0.404,, and B3 0.204. Aggregating
3 = (3o. 31 B2, Ba), the HR-MSM given by
m(t, dz B) Bu 31J-1 + B2Je + B3J1-1Jh
(3)
correctly specified under this data generating process, and we can evaluate the performance of the proposed estimator relative these true values.

---

10
30
100
10
30
100
0.5
1.0
Releive Dies: 6-6
0.0
7
90%, CI Cavarage
0.9
0.9
0.2
0.9
0.0
009
-0.2
HC HC2 HC3 LS
HC HC2 HC3 LS
HC HC2 MC3 LS
Robust Variance Estimator
Excursion Effect
Blip
Dissipation
Dose
Dose 1
Dose 2
Figure 2: Simulation Results Panel columns indicate sample sizes, (10, 30, or 100), and panel rows indicate number of trials, T (cluster sizes, 50 500). [Left]] Relative bias associated with each sequential excursion effect. These results show that our estimator consistent the target parameters, [Right] 95% Confidence interval (CI) coverage for the sequential excursion effects. The coverage of 95% CIs constructed using one of three established robust variance estimators and our robust large sample (shown LS) variance estimator. The nominal coverage is reached for either large or large t for all estimators.
To show we can conduct valid inference on sequential excursion effects, we estimated the three estimands illustrated in Figure 1 (1) the "blip" effect of an additional exposure oppor- tunity at the more proximal trial t, while keeping treatment at t - fixed at the control con- dition (32- (d d(²) - (d 1. d I): (2) "effect dissipation," comparing the effeet of an exposure opportunity at one versus two trials prior to the measurement of the outcome E[ (d df") (1 1,')): and (3) the "dose response" curve of exposure opportunities, where the two possible sequences for a single opportunity are averaged (the sequence (3₂
(8o. 30 + [ + }. Σ 3
B,)). This setup also illustrates how HR-MSMs are easily specified such that sequential excursion effects can be calculated as linear combinations of the 3 parameters.
Although the HR- HR-MSM (3) is correctly specified, still estimated the B coefficients as projection parameters. We proceeded as if we started by defining 3 as the minimizers in (2). with V = (i.c.. effect modifiers), and htt,, da.:) = =1 (i.c., constant weight function). We applied the IPW point estimator the HR-MSM parameters, 3,, described Section 3.2 We assessed the coverage of 95% Cls constructed from our large sample variance estimator, and the small sample size-adjusted HC, HC2, and HC3 variance estimators (using the sandwich ch package in R [36].. We tested performance with sample size {6, 10, 30, 100), and number of trials T {10, 50, 500).
Results Simulation results in Figures and Appendix Figure show that our estimator is unbiased for the target sequential excursion effects. We present 95% CI coverage in Table [ and HR-MSM coefficient estimate bias and MSE in Appendix Tables and 3 Sandwich estimators can yield small-sample bias [36), but in small n and 7 settings, the sample size-adjusted HC3-based Cls achieve 95% coverage. All CIs achieve 95% coverage when n is large, showing that we can conduct valid inference in all settings.
4.2 Application: Optogenetic Study
Data [17] tested whether optogenetically stimulating dopamine (DA) release in the dorsolateral striatum while an animal engaged specific "pose" (c.g., exploring, rearing, grooming) could "teach" mice to exhibit that movement more frequently. This study was foundational in identifying the role this region plays in learning. To that end, the researchers implanted mice with optogenetics machinery, and filmed them reely-moving in a behavioral chamber. They used pre-trained hidden Markov model to estimate an animal's pose online real-time. They first measured the animals' target pose frequency on a baseline session without optogenetics. Then, on subsequent treatment session, they applied the laser on a random subset the target pose occurrences. They repeated this experiment for six target poses, in both the optogeneties and control (laser has effect) groups
7

---

50
500
Effect
C1
A
8
10
R 30
100
8
R
R = 100
Blip
HC 0.01
0.01
0,95
0.01
0.95 0.01
0.99
0.00
0.97
4: 0.01
0.95 0.01
0.01
LS 0.86 - 0.01
0.00
0.93 = 0.01
0.95
0.86
+ 0.01
0.89
0.01
0.93
0.93
0.91
Dissip
HC 0.93 0.01
0.94
0.95 = 0.01
0.00
0.97 + 0.01
0.96
0.01
0.91
IS U.85 0.01
0.89
0,94 0.01
0.95
0.01
0.88 + 0.01
0.91
+ 0.01
0.93
0 0.01
0.94 0.00
Dose 0
HC 0.92 0.01
0.01
0.94 = 0.01
0.94
0,94
0.01
0.94
- 0.01
0.96
0.95 0.01
LS 0,11
0.88
0,92 = 0.01
0.94
0.91
0.86 0.01
0.91
0.01
0,95
0.00
Dose
HC 0.95 0.01
0.05
0.96 0.01
0.95
1.00
0.97
+ 0.01
0.94
0.00
LS 0.58 = 0.01
0.01
0.95 = 0.01
0.95 + 0.01
0.56 + 0.01
0.90
0.01
0.92 +
0.96
Dose
0.95 0.01
0.94
0.01
0.96
0.96
1.00
0.00
0.98
+ 0.00
0.96
0.01
0.96 0.01
LS 0.86 0.01
0.89
+ 0.01
0.95 0.01
0.95
+ 0.01
0.87 + 0.01
0.90
+ 0.01
0.94 + 0.01
0.96 + 0.01
Table 1: Simulation Results: CI Coverage We achieve 95% confidence interval (CI) coverage using
either small sample size-adjusted HC3 (shown as HC), or our large sample (shown as LS) sandwich variance estimators. Mean of R = 1000 replicates is shown (= standard error). We recommend HC3 when n is low. When is high, LS achieves nominal coverage, confirming our asymptotic theory.
To define "trials," the authors spliced the time-series of estimated pose classifications into intervals of consecutive timepoints with the same pose classification. If mice exhibited the target pose on trial t, they were considered "available" for optogenetic stimulation, I 1, and were "unavailable" otherwise, = O. The laser was applied (A, = 1) with the dynamic policy, P(A, 1 I)) = 0. 0.754,. Denoting Y and Y, as a binary indicator that an animal engaged in the target pose on trial t of the baseline and treatment sessions, respectively, the authors estimated treatment effects of the form &0 = (E[ == E G = 1)) (R[YY) G = 0] R[ G
0 where yo
Els r,, 1'.'-=E- E Ye, and T, Tu E N are the trial numbers in treatment and baseline sessions, respectively
P There were n1 = 28 and no = 12 animals in the optogenetics and control groups, respectively. T' ranged across animals/sessions from 1207-4876, with mean of 3612 and IQR [3341, 3940]. The authors reported a (pooled across target poses) positive optogenetics treatment effect estimate akin to e,, suggesting DA stimulation causes an increase in target pose frequency.
We argue this analysis procedure leaves many scientific questions untested. Conceptualizing opto- genetics like a "study drug," we question whether stimulation immediately "taught" the animal the target pose, or whether the treatment effect on learning had lagged onset. Similarly, did the effect of a single stimulation persist or dissipate across trials? Did more treatments lead to more learning monotonically, or is there an antagonistic effect or non-monotonic dose-response curve in learning?
Application Methods We applied our framework to provide a nuanced trial-by- -rial characterization of the causal effects of DA stimulation, and formally answer the questions above. Specifically, we tested the causal effect of specific sequences of deterministic dynamic policies, da,, (occurring on trials t E (t- A + 1, .!). on the mean counterfactual E Y (dd..)))) We defined the outcome Y, as an indicator that the mouse exhibited the target pose on trial + 2, the next trial on which mice could exhibit the target pose if they were available for stimulation on trial t. We fit a set of HR-MSMs illustrating that we can reliably estimate the types of excursion effects in Figure [ We describe the models we below, and relegate code, data and pre-processing details to Appendix Section
Results Our first question was whether standard methods reveal significant treatment effects when assessed with the estimands commonly tested in optogenetics studies. We applied a GEE with mean model, log [E|Y* G = S s]) = 70 + 7ig + 728 + 73g X s, where S E {0, 1} indicates baseline and optogenetics sessions, respectively. The estimate 73. shown in Figure 3F. thus provides treatment effect estimate for the observed stochastic dynamic policy in [17. We adopted a Poisson working model, since [77 analyzed y y E N. We tested these (macro longitudinal) effects for each pose individually, rather than pooling over them, in [7]. The model yielded no significant effects for any individual pose. We show boxplots in Appendix Figure 10 of the subject-level summary y1 y that is compared across groups in this model. Outcome levels are similar across groups for most poses, further highlighting how standard outcome summaries can obscure effects.
To assess an analogous "local" treatment effect using our method, we tested the impact of a single stimulation opportunity. We further evaluated whether the effect had a lagged onset and/or dissipated
'The authors used Mann Whitney U Test applied to a summary across poses but, in keeping with the mean counterfactual based causal estimands, we describe it in terms of means (not medians) and individual poses.
8

---

Piazo Longltuded elllect
Group woophtecchhaaaacaca
Opto Group Skin
Prior
Pose
Pose
Figure 3: Optogenetics Analyses. Plots show coefficient estimates (error bars show 95% Cls). Columns/colors indicate larget pose. [A] Interaction term between and sequential excursion effect single "dose" occuring r = 1, 2, or 3 trials prior to the proximal outcome that the "dose" occurred on. The excursion effects are significant for poses 1-5 (at, at Icast, one lag level). [B] Availability conditional estimate of interaction A x G: opportunity" policy; this reflects
laser x group interaction. [C] Main effect of under the average causal effect group among a population that has received laser opportunities the last trials. [D] Macro longitudinal analysis, similar to original paper, identifies no significant effects.
recent-treattue
across trials. We included group, G, as an effect modifier, to test whether the causal effect of this treatment opportunity was larger in of the groups. Setting A and restricting the regimes interest to those with at most one treatment opportunity "dose", da, E (id--,,d-1,40
Ej (d,)) s 1}, where (d(d)) 1(d, - df"),, we fit the HR-MSM
logit (da), G g)) = 3o Ei 1r (de-r) Bag Ei=o Ps+19 or (de-r) (4)
Thus, 3, with r [3] is an estimate of the log odds ratio comparing the mean counterfactual of Y; under a treatment sequence with a single dose (on r = 2, or 3 trials prior) vs. treatment sequence with zero dose. This permits assessment of effect dissipation or persistence. Figure [B illustrates the analogous effect under a static regime. The interaction terms, 3, with r E {5, 6, 7]. quantify how these causal effects of recent treatment opportunity differ between the two groups.
The results from our model (4) reveal that stimulation opportunities in the treatment group tend to reduce the odds of the outcome, compared to the control group. As shown in Fic these effects are significantly negative for at least one lag level in five out of six target poses. In personal communications, the authors of [17] stated that this result appeared consistent with their finding that animal exploration increased right after stimulation (quantified as higher pose entropy). Figure 3D shows the main effect of group under treatment sequence of dose zero. In essence, this provides an estimate of the "long-term" effect of DA stimulation: 34 is the log odds ratio of treatment group under regime of dose zero (i.e., a "no recent stimulation" policy). We fit comparable models for sequences long A and found results were similar across A values.
Next, we fit the analogous model for the availability wwwbbblity-condiitiin conditional estimand [5] to determine whether current excursion effect methods (i.c., those confined to A policies) identify the same treatment effects: logit (E{Y)(a.) 4e = 1, = g]) = ag + ara; + a:2 + agg a, Figure shows that the effect estimates, 0g significant in only one pose. These results highlight how approach can uncover a greater number significant effects.
Finally, in Appendix Section E we include analyses showing that the laser exhibits dose-response curve in both groups: more treatment opportunities (on some poses) in the last = 5 trials causes the animal to exhibit the target pose more often. Additionally, there is significant effect modification by baseline responding: the laser has a larger effect in animals who exhibited high baseline pose frequeney. Together, these results show reliably estimate sequential excursion effects.
5 Discussion
We propose the first, to the best of our knowledge, formal causal inference framework for closed-loop optogenetics behavioral studies. We introduce a nonparametric excursion effect framework, an associated IPW estimator (with valid Cls), with scalable implementation, and proved its consistency

---

and asymptotic normality under mild assumptions. Methodologically, our proposed sequential excursion effects represent an expansion of the conditional estimands proposed in [5] to longitudinal policies (A 2 1), in the presence of positivity violations. Our methods also directly apply to "open-loop" (static policies) designs, as they arise as a special case when I, for all E [T]-
HR-MSMs are powerful and useful models, but have their limitations. As has been discussed in the causal inference literature, these estimands marginalize over all treatments for trials t [t A]. and thus depend on the protocol used in the design [3 6. Moreover, while contrasts of our estimands are null under the sharp null of no causal effect of treatment (e-g.. optogenetic laser stimulation), effects should generally still be interpreted in terms of treatment opportunities. Finally, while our implementation is computationally efficient, we anticipate computational challenges for very large A.
The model m and the number of intervention timepoints A represent key choices for practitioners. Our inferential results (i.e., Theorem (.5 are valid for large class of working mean models m, and notably do not rely on any distributional assumptions. The "Donsker" requirement (condition (ii) in Theorem 3.5 satisfied outright by generalized linear models such as those we use in our application [35]. as well as some formulations of random forests (34] and kernel estimators (2). In future work, we will study how inference can be obtained for more flexible models that do not satisfy the Donsker assumption. Likewise, the value of A plays significant role as it determines the nature of the effects being estimated, and should be chosen on the basis of subject matter expertise. That said, we found that 2-3 intervention timepoints are often sufficient to capture a rich set of sequential excursion effects, and in our application that results were relatively stable across a range of A values.
The application highlights the drawbacks of standard optogenetics analysis methods. Our finding that macro longitudinal estimates of individual target poses show almost no effect between groups highlights how "treatment-confounde" designs, even when inspecting simple averages of observed outcomes. Our methods account for this by careful causal adjustment with IPW. In personal communications with the authors of [17]. they agreed with our findings and remarked how these methods reveal collection of causal effects that are difficult to uncover without sophisticated causal inference methods.
feedback can obscure strong treatment effects in closed-loop
Our analyses reveal immediate negative effects (detectable on the next trial) and positive slower effects of DA stimulation (i.e., in treatment relative to control animals). We also find the control group exhibits positive, off-target effects of the laser. Together the opposing signs of these "sloo""""low" and on/off-target causal effects may further dilute the magnitude of macro longitudinal effects that summarize the outcome across many trials (e.g., total pose counts). Finally, by enabling estimation of sequential excursion effects (i.e., A >> 1), we can reveal effect profiles (e.g.. dose-response curves) not possible with waillbblityyconnditonn estimands whose definition is confined A regimes. As we observed, the optogenetics group sometimes exhibits an excursion effect not present in the control group. Thus, by combining different sequential excursion effects, analysts can, for example, disentangle laser on on-target from off-target effects. When off-target effects are not major concern, our framework enables estimation of causal effects withou having to collect data in a control group. thereby potentially reducing the number of animals required in a study.
Although we focus on optogenetics here, our proposed methods are relevant for wide range of mobile health, neuroscience and psychology experiments for which the "local/micro" longitudinal structure of scientific interest. Indeed, "closed-loop" designs are common in many behavioral studies in human neuroimaging and cognitive sciences (e.g., when stimuli are conditionally randomized). We hope our methods constitute useful methodological contribution the causal inference literature, and will help applied researchers exploit the rich information contained in their experiments.
Acknowledgements
This research was supported by the Intramural Research Program of the National Institute of Men- tal Health (NIMH), project NIC-M9002968. This study utilized the seegh-erffrmncc computa- tional capabilities of the Biowulf Linux cluster at the National Institutes of Health, Bethesda, MD (http://boowlf..mbb
First, we would like to thank the authors of "Spontaneous behaviour is structured by reinforcement without explicit reward," Drs. Jeffrey Markowitz and Sandeep Robert Datta, for sharing their data, helping us conduct analyses, and interpret the results. This work would not have been possible without their generosity, commitment to open science and scientific rigor. We would also like to thank the NIMH Machine Learning Team for helpful feedback on our project.
10

---

References
[1] D. L. Barack, E. K. Miller, C. Moore, A. M. Packer, L. Pessoa, L. N. Ross, and N. C. Rust. A call for more clarity around causality neuroscience. Trends in neurosciences, 45(9):654-66 655, 2022.
[2] E. Beutner and H. Zähle. Donsker results for the empirical process indexed by functions of locally bounded variation and applications to the smoothed empirical process. Bernoulli, 29(1):205 228, 2023.
[3] I. Bica, A. M. Alaa, J. Jordon, and M. van der Schaar. Estimating counterfactual treat- ment outcomes over time through adversarially balanced representations. arXiv preprint arXiv:2002.04083, 2020.
[4] R. Biswas and Shlizerman. Statistical perspective functional and causal neural connec- tomics: A comparative study. Frontiers in Systems Neuroxcience, 16:817962, 2022.
[5] A. Boruvka, D. Almirall, K. Witkiewitz, and S. A. Murphy. Assessing time-varying causal effect moderation in mobile health. Journal of the American Statistical Association, 113(523): (1:523))1112 112- 1121, 2018.
[6] F. R. Guo, T. S. Richardson, and J. M. Robins. Discussion of 'Estimating time-varying causal excursion effects in mobile health with binary outcomes'. Biometrika, 108(3):541-550, 08 2021.
[7] M. Hernan and J. Robins. Causal Inference: What If. Chapman & Hall/CRC Monographs Statistics & Applied Probab. CRC Press, 2023.
[8] J. Huber. Robust estimation of location parameter. The Annals of Mathematical Statistics, pages 73-101, 1964.
[9] P. J. Huber et al. The behavior of maximum likelihood estimates under nonstandard conditions. In Proceedings of the fifth Berkeley symposium on mathematical statistics and probability, volume 1, pages 221-233. Berkeley, CA: University of California Press, 1967.
[10] Z. Jiang, S. Chen, and P. Ding. An instrumental variable method for point processes: generalized wald estimation based deconvolution. Biometrika, page asad005, 2023.
[11] Karline Soetaert. rootSolve: Nonlinear root finding. equilibrium and steady-state analysis of ordinary differential equations, 2009. R package 1.6.
[12] E. Kennedy, S. Balakrishnan, and L. Wasserman. Semiparametric counterfactual density estimation. Biometrika, page asad017, 2023.
[13] E. H. Kennedy, S. Lorch, D. S. Small, al. Robust causal inference with continuous instruments using the local instrumental variable curve. Journal of the Royal Statistical Society Series B 8 81(1):121-143, 2019.
[14] M. E. Lepperød, T. Stöber, T. Hafting, M. Fyhn, and K. P. Kording, Inferring causal connectivity from pairwise recordings and optogenetics. PLoS Computational Biology. 19(1 1))e11)e111111 1574, 2023.
[15] B. Lim. Forecasting treatment responses over time using recurrent marginal structural networks. Advances in Neural Information Processing Systems, 31, 2018.
[16] I. Marinescu, N. Lawlor, and K. Kording, Quasi-experimenntl causality in neuroscience and behavioural research. Nature Human Behaviour, 2112):81-8898 2018
[17] E. Markowitz, W. F. Gillis, M. Jay, Wood, R. W. Harris, R. Cieszkowski, R. Scott, D. Brann, D. Koveal, T. Kula, C. Weinreb, M. A. M. Osman, S. R. Pinto, N. Uchida, S. W. Linderman, B. Sabatini, and R. Datta. Spontaneous behaviour structured by reinforcement without explicit reward. Nature, 61447946):008-117, 2023.

---

[18] V. Melnychuk, D. Frauen, and S. Feuerriegel. Causal transformer for estimating counterfactual outcomes. In ICML, pages 15293-15329. PMLR, 2022.
[19] K. Moore, R. Neugebauer, Lurmann, Hall, V. Brajer, S. Alcor, and Tager. Ambient ozone concentrations cause increased hospitalizations for asthma in children: An 18-year study in southern california. Enviroumental Health Perspectives, 116(():1063-10 1070, 2008.
[20] R. Neugebauer and M. van der Laan. Nonparametric causal effects based on marginal structural models. Journal of Statistical Planning and Inference, 137(():419 434, 2007.
[21] R. Neugebauer, M. van der Laan, M. M. Joffe, and Tager. Causal inference in longitudinal studies with history-restricee marginal structural models. Electronic of statistics, 1:119, 2007.
[22] W. K. Newey. Uniform convergence in probability and stochastic equicontinuity. Econometrica: Journal of the Econometric Society, pages 1161-1167, 1991.
[23] M. L. Petersen, S. G. Deeks, J. N. Martin, and M. J. van der Laan. History-adjusted Marginal Structural Models for Estimating Time-varying Effect Modification. American Journal of Epidemiology, (66(9)9955999)) 09 2007.
[24] D. Pollard. Convergence of stochastic processes. Springer Science & Business Media, 2012.
[25] D. Pospisil, M. Aragon, and J. Pillow. From connectome to effectome: learning the causal interaction map of the fly brain. bioRxiv, 2023.
[26] T. Qian, H. Yoo, P. Klasnja, D. Almirall, and S. A. Murphy. Estimating time-varying causal excursion effects in mobile health with binary outcomes. Biometrika, 108(3):507-527, 09 2020.
[27] Robins. A new approach to causal inference in mortality studies with a sustained exposure period application control of the healthy worker survivor effect. Mathematical modelling, 7(9- 12):119331111111 1986.
[28] J. M. Robins. Marginal structural models versus structural nested models as tools for causal inference. In Statistical models in epidemiology, the environment, and clinical trials, pages 95-133. Springer, 2000.
[29] J. M. Robins, S. Greenland, and F.-C. Hu. Estimation of the causal effect of a time-varying exposure on the marginal mean of a repeated binary outcome. Journal of the American Statistical Association, 94(447):687-70, 1999.
[30] J. M. Robins, M. A. Herman, and B. Brumback. Marginal structural models and causal inference in epidemiology. Epidemiology, pages 550-560, 2000.
[31] J. M. Robins, M. A. Hernán, and A. Rotnitzky. Invited Commentary: Effect Modification by Time-varying Covariates. American Journal of Epidemiology. 166(9):99-1002, 09 2007.
[32] M. Rosenblum and M. van der Laan. Targeted maximum likelihood estimation of the parameter of a marginal structural model. The international journal of biostatistics, 6(2), 2010.
[33] L. N. Ross and D. S. Bassett. Causation in neuroscience: keeping mechanism meaningful. Nature Reviews Neuroscience, pages 1-10, 2024.
[34] E. Scornet. On the asymptotics of random forests. Journal of Multivariate Analysis, 146:72-83, 2016.
[35] A. W. Van der Vaart. Asymptotic statistics, volume 3. Cambridge university press, 2000.
[36] A. Zeileis, S. Köll, and N. Graham. Various versatile variances: An object-oriented implemen- tation of clustered covariances r. Journal of Statistical Software, 95(1):1-36, 2020.
12

---

A Micro Longitudinal Effects
In Appendix Figure 4 we illustrate additional micro longitudinal effects that can be probed with our sequential excursion effect framework. This figure has the same layout TigureTA-C.
B Illustration of Treatment- Feedback
We provide this section a synthetic example which two independent groups exhibit identical mean outcome patterns over time, but where the treatment (e.g., turning laser in the brain) has substantial effect in one group but not the other. As our construction will demonstrate, this phenomenon manifests due to reaamennt-conoonnde feedback leading to effects canceling out. In similar fashion, one can similarly construct scenarios where effects are exaggerated.
Suppose G E {0, } represents an experimentally manipulable marker (e.g., animals expressing opsin the brain), and counterfactual outcomes under G are denoted Y We will suppose that potential outcomes generated in the active setting (G 1) are given by
y N(oor + X-1 + 7/ + 73 X + A,, o?);
and potential outcomes in the control condition (G = 0) are given by
y NYoor + X + X3X,, of),
i... the treatment (e.g., laser) has an effect when G 1, but not when = 0.
Suppose further that behavior X, is measured at all time points I, and determines whether or not treatment will be administered with positive probability. Like the outcomes, this behavior will be affected by the laser only when G 1:
X² Bernoulli - 0.5 A.-5)) for t E (1...., TT.
and X Bernoulli ( at baseline.
Now we consider a study where animals are randomly assigned at baseline to either G = 1 or G = At each time point t, the behavior X, is measured, and treatment then drawn according to
A, ~ Bernoulli
By induction, E(A, G = 1) 0.4 and E(X, | G = g) = 0.7 0.2g, for
all follows that
[YY G = 9)
=Nor + 71 g) + G = +g B(x, G G
G) 9) 7dB((A G 1)g
{To: + 0.7(71 + 73)) + (-022( + 73) 0.4(12 + 74)]g.
Thus, the "macro"""globba" between-grooe group mean difference trajectory is given by
E(Y, C E(Y, G = 0) -0.2(m + +a) + 0.4(72 + 74),
which will be null if 72 + 74 = 0.5(71 + 73). Notice that this cancellation is possible even if the immediate effect of treatment on the outcome is quite strong, say if 72 and 74 are large and positive. The cancellation is made possible through the opposing effects of treatment on the intermediate behavior and the outcome: when generally, these X,-A, feedback loops can lead to dilution or exaggeration of the actual effect of treatments when only analyzing observed mean outcomes.
1, A negatively impacts X, but positively impacts Y, More
We note that in the data generating scenario described in this appendix, the proposed dynamic treatment regime HR-MSM methodology would pick out non-null effects of treatment within the active group (G' = 1). and show differing effects between groups, even if the condition above held such that observed mean outcomes were identical. This example thus serves illustrate both the challenges with closed-loop designs and, despite these challenges, the ability of the proposed methodology to elucidate effects.
Example Analysis on Synthetic Data illustrate the above, we provide an example on a simulated dataset, taking 100, T 500, 71 73 1, 72 74 0.5.
13

---

A. Sufficient Dose
Suffisient Aohiewed
Sufticlent Achloaed
PolicyOraf
El k(a
Paley
E[y(Gae0n k((
EMAAA K(a
ElNaça ()) V,(a
P[Vi(a 7.("0))
3
Trolay a
Y Y
PakaySsa
Palayy 2
B. Repeated Stimulation Antagonism
Offecz Magrituda Maimainad
Antagoniam Effect
RaloyOralt
3 Y,aa )
PI1 > K/a
EIVAGA () Viaa
) V,(a 11)
Pulay
((y(a ) 100--%.0 k(a
Polccy 2h
Pullcy
E.Effect Additivity
Callin Bllest
Additiv Blfest
Effect Lag
Uben Corrercce aoon
Inmredisto Onret
Lagged
Polloy 1.
(a )
E(Y(a k((a
E[ (a 0 (
E[ V.(a
(a((( Y,(a
ELY El %((a
Policy a!
lime-Dependent Effect Synergism
Am
Ope P30
Folicy fraftt
X
Tirse-Daperdent
deeeleedeenddett Effect
Pulicy 2M
(JA) XX
3[6G4) V. (1)
Pollyy 2
Effect Size
None
Low
Med
High
Figure 4: Example Sequential Excursion Effeect.. The left panels show one setting where sequence of laser simulations do do not have the indicated effect on the outcome. The middle panel shows deterministic static policies that could be used to construct a causal contrast to probe the effect. The right panel shows what the anticipated effect size (darker is larger) of those contrasts might be if there is is not the indicated effect profile. [A] Sufficient dose. The red line shows how three successive stimulations is required whereas effect profile blue shows that sufficient dose not been reached. [B] Repeated stimulation anatagonism. The red line shows a negative dose-response, and the blue line shows a stable effect size. [C] Effect additivity. The red line shows second stimulation triggers larger response, whereas the blue shows that the second stimulation does not increase the response substantially beyond that of the first stimulation. [D] Effect Lag. The line shows that causal effect of stimulation is not visible until after a lag period. The blue line shows setting where shows a setting where the effect is additive provided the stimulations occur close enough together (red line), hut if stimulations occur far apart, this synergism does not occur (blue line).
trigger large effect,
effect immediate. [F] Time-dependemt effect synergism. The red line
14

---

Averaged scoms Subjects
Awaraged Timepers
Group 10-di
02
Sca Excursion veluc
0.3
Moann Dutzomen:
Outcome Pudled
0.3
0.7
1-
100
250
360
400
Contral
Treammett
Excursion Effect
Group Contoll
Treatment
Effect
Bip
Dissication
Dose vs. Dosa
Dose 2 vs. Dose
Figure 5: Preatment-Confounder Feedback Example Sequential excursion effects reveal causal effects obscured in "macro" summaries. Analysis results from simulated dataset following the argument above (in Appendix B}. taking n 100, T = 500, 71 7 1, 72 7 0.5. [A] Each dot is an outcome value, Y.G. for subject at timepoint / from "control" (G from "treatment" (G 1) groups. Lines are timepoint-speciin specific means (averaged across subjects), estimated using linear smoother (loess). (B) Same data (A), but each point in boxplot is subject's mean outcome value (averaged across timepoints). (A)-BB) "macro" summaries show no differences due to treatment-confounde feedback: mean outcome values (averaged across subjects or timepoints) are nearly identical in both groups. (C)-(D) Point estimates and 95% Cls (error bars) of sequential excursion effects reveal "local" causal effects (in Treatment group only), obscured in "macro" " summaries (shown in (A)-BB)).
0), or
15

---

C Additional Details for Section 3
C.1 Interpretation of Sequential Excursion Effects
The interpretation of the mean counterfactual quantity Edy;(da.:) | Vi-A+1] somewhat subtle, and warrants further discussion. When A 1, we can express a contrast of these estimands in terms of the effect of exposure in a certain subgroup:
(4) V. - ( V.] = [a = 1) (= 0) V,, 7, = 11] = 1 Vil.
That is, the mean contrast in counterfactual outcomes for df versus d.)) is the mean effect of A, = versus A, = O among those with I, = 1-the intilaaiiiiycconntttin by [5] [5]-diluted by the probability of availability. Even in this case with A 1, it may not always be clear for whom the wvailability-conditin highly idiosyneratic and not of particular interest. On the other hand, the parameters are proposing summarize the effects of plausible interventions on the whole population, acknowledging that for some individuals active treatment (i.e., A, 1) not possible.
estimand proposed
estimand generalizes to, i.e., the group I, = 1 may be
The comparison just described somewhat akin to the duality in clinical trials of per-protocol (or cumplier-specifeer effects, and intention- cosooon-tt-ttrattt to-treat effects. Thus, in practice when A = would recommend assessing both the availability-cone conditional estimand, as in [5]. as well as our proposed population-lewe level effect. When A 1, it is not clear whether analogous estimand exists; our approach is viable for arbitrary A. In general, our estimands have the population- level (possibly conditional on effect modifiers) interpretation of summarizing how outcomes would be affected if the experimental protocol were changed to match da,t for the A trials leading up to the outcome. Finally, we note that, as for all excursion effects or history-restricted marginal structural models, the estimands under study are dependent on the treatment protocol [6].
wwwalaabliiy-condition
C.2 Discussion of Causal Assumptions
Consistency (Assumption 3.1 states that for any of the regimes da, under study, the counterfactual outcome Yi (dda,,) equals the observed outcome Yg when observed treatment values correspond to assignment under da Positivity (Assumption 3.2) states that treatment probabilities are bounded away from zero this is required for the asymptotic analysis of the proposed estimator later on. Note that, by definition of the availability indicator I, and the regimes Dj in Section 3.1 we are allowing P(A, = 1 H,) 0 in some cases (i.e., when 1, 0), but Assumption 3.2 rules out [P H.] = 1. This positivity assumption holds in many open- and closed-loop optogentic studies. In practice, in such experiments, one can ensure that Assumption 3.2 holds by design when choosing the treatment assignment probabilities. Finally, Assumption 3.3 says that treatments are randomly assigned at each time f, based on all previously measured data Hr, In the sequential optogenetic experiments that motiviate this work, this assumption would hold by design. In observational studies, one will have to assess the plausibility of Assumption 3.3 (as well as Assumptions 3.1 and 3.2) on a basse basis, ideally based on subject matter knowledge; it may be harder to justify Assumption 3.3 due to the possible presence of unmeasured confounders.
C.3 Conditions of Theorem 3.5
Conditions (i) through (iv) are standard conditions for asymptotic normality of M-estimators g 9 For condition (ii). expect the working model m be differentiable in (3 for most common models. Moreover, for standard generalized linear models, m will be appropriately Donsker-see 35 for formal definitions. Condition (iii) is satisfied under mild conditions, e.g., if the weight functions h. the model m and its derivative M, and the outcomes Y, are uniformly bounded, and no haphazard degeneracy in B exists that could cause singularity. Lastly, condition (iv) is also quite weak,
only requiring convergence 3 at an arbitrarily slow rate, and would hold under some stochastic equicontinuity conditions 22 24
16

---

C.4 Proofs of results in Section 3.2
Proof of Proposition 3.4
This result follows the usual g-formula identification argument (27] defin-
ing As.t = (Al-Ati:...... Ad), da)((H))) = ( (d=A-1(H++A+t).... (H.)).
B(Yi(da,)) Vi-A-1)
B(E(Y((da..)) He-a+1) VI-A+1) (T(E(Y(ds,:)) He-a+1: A+ = de- Vi-a+1)
-B(E(--- B(Yi(da.)) HG, Aa, = da.((H:)) ... | H1 A+1 di (4-4 V-
where repeatedly invoke iterated expectations and Assumption 3.3 (justified by Assumption 3.2
then use Assumption 3.1 the last equality. We can then rewrite this formula in an equivalent TPW form:
E(B( ((Yi(da,)) As, (H,))
HH- 4-4-1 dd - Vo- 1 (At- = 4- (H,- ...E 1(4, d(HH.) Yi(da.)) H1) m((A; Ho)
E
H Hi-A41) VV-
Te T 2+1 H1-2+1)
144
d, (H,ر) Ty (A3; H,)
II
Ye V2-A+1
j=t-A-1
where the last equality is achieved again by iterated expectations. The second statement in Proposi-
tion is obtained by differentiating (2 with respect to B, setting this to zero, then invoking the first statement of Proposition 3.4 (which we have just proved).
Proof of Theorem
3.5
This is immediate application of Theorem 5.31 in 35
D Additional Simulation Details and Results
D.1 HR-MSM for Simulation Data-Gengg Mechanism
In this section, we derive the form of the HR-MSM in (3). and show that it is implied by the data-generating mechanism of the simulation study, First, observe that for t > 2,
(d,.1, de) a4 X, +
+ a3 (, + (X,(dr1)))) + 4.
for some exogenous & NN, of). where X, (d+-1) is the potential X, value under the intervention setting A,1 to d dx-1 Note that (x,1 J and by our structural equations. 0.4d, (X,-1)), so that (d,,( that [((x-1) 0.5. Finally, de (X,
Xe(d,-) 1) Bernoulli
0.4 + 0.2J, 1, recalling J, Bernoulli + 0.4d,-1
which gives E(d (X,((x = (0.4 0.2J Putting everything together, we obtain
(d, 0.5a, + 0.5.J (0.4 0.2J,- J,as]
=00.5a + 0.40. (0.5a, + 0.20g 0.404 Jr + 0.204 J.
= fo J1 + J2J. + 33J4-14.
as claimed.
D.2 Further Simulation Results
In Appendix Figure we present the same results as in Figure but with more sample sizes, n and trials. T. We also present these same simulation results in terms of the 3 coefficients of HR-MSM In the main text we presented results in terms of sequential excursion effect parameters, which are linear combinations of these HR-MSM regression coefficients.
E Additional Application Results
E.1 Dose-Response Excursion Effeett
History-Restricttr MSM We fit HR-MSM within the treatment group (G = 1) to estimate the causal effect of "dose," the number of treatment opportunities in the previous A 5 trials:
17

---

10-
Heaeon F-3.15
HC VollecVerlncce Ectiesator
Escursion Offect
Bllp
Diccipadon
Doee0
Done
Dona
Figure 6: Simulation Results Panel columns indicate sample sizes, and panel indicate number of trials, (cluster sizes). [Left] Relative bias associated with each sequential excursion effect. These results show that our estimator consistent for the targel parameters, [Right] 95% Confidence interval (CI) coverage for the the sequential excursion effects. The coverage of 95% CIs constructed using one of three established robust variance estimators and our robust large sample (shown LS) variance estimator. The nominal coverage reached for either large n or large for all estimators.
3.0
Be
1-3.9
Conficiint
Rollas Essieatoo
factoricet
Figure 7: Simulation Results in Terms of MSM Coefficients Relative bias and 95% Confidence Interval (CI) coverage of regression coefficients of history marginal structural model (MSM) Panel columns indicate sample sizes, n. panel rows indicate number of trials, (cluster sizes). [Left] Relative bias associated with each HR-MSM regression coefficient. These results show that our estimator is consistent for the target parameters. [Right] 95% CI coverage for the MSM coefficients. The coverage of 95% Cls constructed using of three established robust variance estimators and our robust large sample (shown LS) variance estimator. The nominal coverage is reached for either large large for all estimators.
logit (E(Y,(da.:) G 1]) 80 Σ 8,1
Σ 1+t-A+1
o,(d,) =r)
(5)
where (d,d,) (d, ) The coefficient B, is an estimate of the log odds ratio comparing the mean counterfactual of Y, for a treatment sequence of dose E [3] compared to sequence of dose zero (see Figure TC for an illustration of the static regime analogue). A dose of three is the maximum feasible dose for A 5 since the same pose cannot occur on two consecutive trials.
The dose-response effect estimates, {3.) from HR-MSM are shown in Figure 8A. This illustrates the capacity of our approach to identify clear dose-response effect: within the past A trials, each additional opportunity for stimulation causes increase in the odds of engaging the
18

---

50
T = 500
Effect
==6
11 10
R=
0= 100
n=
18 10
1 30
N 100
Blip
1.91 0.09
=1100.00
0.30 0.02
0.03 0.01
0.01
0.03
0.00
0.00
0.00
Dissip
0.33
0.04
0.00
Dosc 0
0.73 = 0.04
0.40 0.00
0.08
= 0.01
0.00
0.0000000
0.02
0.00
0.00 0.00
0.00 = 0.00
Dose
0.90 = 0.04
0.51 = 0.03
0.10
= 0.01
0.00 0.00
0.00
0.00 0.00
0.00
=.00 0.00
0.00 = 0.00
Dose
3.52 0.15
2.05 0.09
0.58
0.08
0.12 + 0.01 0.25+00.08
0.11 0.01
0.01 0.00
0.00 0.00
Table 2: Simulation Results: MSE Our estimator's MSE decreases to 0 T grows. Denoting
the estimated effect j (e.g.. j = "Blip") for replicate
fy.r. we show MSE, = = Σ (3,,r
9,)² for = 1000 simulation replicates (+: SE) for a sample size, n, and timepoints, Values are scaled by 100 for readability (e.g., 0.01 is shown in the table as 1.0). Thus 0 indicates value 4.
T
Effect
==
==
n= 30
= 100
==6
n=
100
Blip
0.23 = 0.55
0.43
0.24
0.10
0.07 0.17
0.08 0.13
0.03 = 0.04
Dissip
0.82 = 0.93
1.0
0.45 = 0.39
0.22
0.27
0.21
0.10
0.00 = 0.07
Dosc
0.28 1.48
0.15
1.17
0.12
= 0.68
0.42
0.28
0.23 0.35
0.34
0.13 0.11
0.32
0.06
0.10
0.06
0.01
Dose 2 0.41 0.35
0.09
0.27
0.12
1 0.15
0.10
0.08
0.03
0.05 +: 0.08 0.03 + 0.05 0.01 + 0.03
Table 3: Simulation Results: Bias Our estimator is unbiased. Moreover, the absolute relative bias
decreases 0 as and/or grows. Denoting the estimated effect j (e.g.. "Blip") for replicate as Bj.r. we show Absolute Relative Bias, 1/2 E B;)/B,] for 1000 replicates (+ standard error). Values are scaled by 100 for readability (e.g., 0.01 is shown in the table as 1.0).
target pose on the next trial. The effects are significant for at least one dose value in all but two target poses. Interestingly, the effect is also significantly negative for one target pose.
Conditional Excursion Effect We next estimate availabilitycconditinaaa mine if existing excursion effect methods have the capacity to reveal the effects identified with our method. We estimate this in the MSM
conditiooal estimand (5) deter-
G, logit (B()((a)) 4 = 1, = 1]) = an + ag
(6)
Figure 3 shows the availability-conddiion treatment effect estimates, âj estimated in model (6). identifies no significant effects for any target pose. The conditional estimand, often referred to as "blip effect" (see Figure A for an illustration) is only defined for the effect of applying the laser on the most recent trial (i.e., dose of 1), and thus cannot estimate dose-response profiles. In contrast, our approach can test sequential excursion effects (i.e., for policies with A > 1), enabling the estimation of rose-response profile that reveals treatment effects here. Importantly, the effect estimates 3, and ãj have different interpretations because 31 reflects a causal effect of a single treatment opportunity for any of the last A 5 trials, and 31 is not interpreted conditional on availability.
19

---

Dose -Resporse Tresiment Effects: B. 6.
Conditional Treatment Effect. G1
1.0
0.05
0.
-0.0
-15
Dose
Pose
5
Pose
Figure Our method enables estimation of dose effects. Plots show cocfficient estimates (crror bars show 95% CIs) as a function of dose. Columns and colors indicate the dose. [A] Main effects of stimulation opportunity from HR-MSM (5) [B] Availablity-ccondiional effects of treatment estimated in MSM (6).
E.1.1 Effect Modification by Baseline Behavior
Marginal Effeet Modification Parameter Next we show the capacity our method to estimate effect modification in the form of interactions between covariates and functions of deterministic policies (i.e., treatment opportunity dose). We estimated effect modification total target pose counts on baseline sessions, Yod (defined in Section because [17] estimated the treatment effect of the laser by comparing the mean change outcome levels between treatment and baseline sessions.
Augmenting the HR-MSM in the previous section with effect modifier, Y we fit the HR-MSM
logit (R]()(d>)) (19,
an -)) + +ave 2 a((()=)))))) -1)
(7)
Similarly, we fit an analogous model for the conditional estimand,
logit (E(Y((a)) | -
= as + taj + az8 + az880 x ar.
(8)
We centered and scaled the effect modifier to make the coefficients easier to interpret.
Figure -B shows how our method can be used to probe frfeet-modiiication. The figures illustrate that our approach estimates that for two target poses, there is statistically significant effect modifica- tion by baseline responding at, at least, one of the doses. Specifically, animals who exhibited higher levels of responding on baveline sessions, exhibited larger effect of dose. Figure shows that the maallbblliy-cooditiina estimator does not identify any significant main effect of stimulation or interactions with baseline responding. These results show how our framework enables one to probe effect modification of sequential excursion effects.
20

---

Ai.
Conditaaa Troatment Effect
Reepording A.
IntodontCadttoo
Dose
Po68
Figure 9: Our method enables estimation of effect modification of baseline behavioral responding, Plots show coefficient estimates (crror bars show 95% Cls) as function of dose. Columns and colors indicate pose. [A] Main effects of treatment opportunity for doses 1-3 mean baseline responding levels with marginal HR-MSM (our approach),
l [B] Interaction with baseline pre-stimulation session) responding levels with marginal HR-MSM (our approach). (C) Main effects stimulation on past at mean baseline responding levels with availablityy-coddiion approach. [D) Interaction between stimulation and baseline responding with ovailability-condiionn approach.
F Data, Pre-processing and Code Availability
We provide code reproduce all analyses and figures anonymous GitHub Repo: https / //anonymous 4open sci http://wwweeeeeaaaaaa.aaaa////// source dataset from [ from bttmmmttss:///eeeoo
md. We downloaded the open- wwwp://oor////////ggeeeee/
We used the open-source pre-processing code provided by the authors on Github repo https: // thub. com/ dattalab/ dopamine- reinf orces- spontaneous- behavior We analyzed data from all animals the on- line dataset (both ChR2 and Chrimson animals). We constructed trials as described in [17]. That is, we defined trials consecutive timepoints when the animal was classified be in a given pose. For our HR- HRM analyses of the treatment (opto) sessions, we classified "target pose" trials only if they met the criteria [17,, which required that the hidden Markov model predictions had sufficiently high forward algorithm probabilities of the latent states. This indicator was provided in the opto session dataset provided by the authors. The baseline session data did not, however, include this indicator since optogenetic stimulation was applied. Thus when recreating the "standard" between-gnn group (macro longitudinal) analyses that compared bascline and opto session data, we did not classify target pose trials based whether it met this criteria: classified the pose based the most likely latent state prediction but did not require the forward algorithm probabilities met the threshold set by the authors (for either baseline opto sessions to be consistent). We corresponded closely with the authors to ensure we pre- preppeceesedd the data correctly.
There was a small percentage of trials that the authors described eliminating because they were deemed too short. We did not eliminate these trials because this created inconsistencies in the pattern of trials: it allowed two consecutive trials to be of the same trial type which broke with the pattern in the remainder of the dataset. This was very small percentage the dataset. We compared results with and without this criteria and the decision appeared to have negligible effects analysis results.
21

---

Finally, to the best of our understanding, the original authors' hypothesis tests were conducted on further processed version of the data that first calculated the number of target pose occurrences in each 30 second bin of the experiment (period). From our understanding, this was done to provide smoothed time-series of outcome frequency across the course of the opto sessions. We conducted similar analyses to make sure our pre-processing yielded comparable results, but we did not use these pre-pro pre-processing steps in our HR-MSM analyses or replication of the "standard" between-group (macro longitudinal) analyses it appeared to "double-count" target pose occurrences that began before the end of one second bin and ended after the start the subsequent bin.
Finally, as described in [17], the experiment included two 30 minute replicates of both opto and baseline sessions. We constructed trials on each replicate separately (to account for the discontinuity in time between replicates) and then pooled the replicate datasets together to be consistent with the analysis procedures 17 We accounted for the longitudinal structure by using sandwich variance estimators in all of our hypothesis tests.
F1 Replication of Original Author Analysis
Finally, Figure shows that the "standard" macro longitudinal effects exhibit significant changes, emphasizing how estimands that marginalize over the stochastic dynamic glosed-loop policies can obscure effects. We visualize the within-subject differences in target pose counts between treatment (opto) and baseline sessions Figure 10 Σ Y 1 -! YO, where Y,, Y, E {0, 1} are indicators that the animal exhibited the target pose on trial t of the treatment and baseline sessions, respectively; T. To E denote the total number of trials in treatment and baseline sessions, respectively. Because of the trial definition, the total number of trials usually differed within- between treatment and baseline sessions (i.e., T' * To), but we found comparable analyses of Σi Y, and + > Y
yielded similar results to analyses of total counts Ei, -E Y For that reason, present results in terms of total outcome counts to be consistent with the analyses presented in [17]. We showed these results the authors of [17 and they confirmed that these analyses aligned with theirs.
1
2
4
5
600
:
Opto -Booelliee Session Difference:
400
200
-200
Cpri
Ciri
Ctrl Opto
Ctrl Opto
Cirl Opto
Cpro
Group
Figure 10: Difference between target pose counts within-subject between baseline and treatment (opto)
sessions. Each point in the boxplot shows ET= Y, animal exhibited the target pose on trial f of the treatment and baseline sessions, respectively. T, To E Z were the total number of trials in treatment and baseline sessions, respectively. Columns and colors indicate target pose. Ctrl and Opto indicates control and treatment group subjects, respectively.
Σ/e, Y where Y,, Y E {0, }} are indicators that the
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The methodological, theoretical, and experimental contributions described in the abstract are each addressed in their own main text sections. We only make claims the abstract that carefully justify and explain in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:[Yed Yes]
The Discussion section includes limitations of our proposed method and the interpretation of the method's results. We state the mathematical assumptions clearly in the main text and discuss their justification the Appendix. We also discuss method scalability in the Methods (Section B. Our simulation results highlight the limitations of our approach in small sample settings: small sample bias, and the requirement of sample size-adjusted covariance estimators achieve nominal coverage. We intentionally include very small sample sizes in our simulations to emphasize the trade-offs in these settings even though our data application has datasets with a much larger sample size. Our work focuses on animal data and thus does not include problems of privacy or fairness.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they seale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
23

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Complete proofs of Proposition 3.4 and Theorem 3.5 are provided Ap-
pendix C.4 All remaining technical details are discussed in the paper, with additional details provided the appendices when relevant.
Guidelines:
The answer NA means that the paper does include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide all details needed to reproduce our results in the main text and appendix. We include all code used to produce results, pre-process data and make figures. We provide links to the open-source datasets that we analyze.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., s., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
24

---

(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We include data pre-processing and data availability section (data from our application is open-source) in the Appendix section. We also include a link to anonymous GitHub repository with an implementation results presented the main text and Appendix.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
hotee//////iieeee/eeeubbbiiiiiii Stt////oooSssssiiiiiiaaaa wtw.himmmssiioooooocccoo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We discuss the optimizer used. Our method has no tuning parameters. No data splits were used anywhere in the paper since our method is focused on statistical inference and hypothesis testing (not on prediction).
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
25

---

7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We include 95% confidence intervals (CIs) for our results, e.g., represented by error bars in figures. We verify in simulations the validity of these CIs (i.e., that they achieve the nominal coverage).
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: For the vast majority of applications, our methods are not computationally intensive and can be run quickly with little memory. We briefly discuss but we do not provide details since
everything standard laptop with no GPUs or parallelization.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 'ttntt make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:////haaaaeccccppppc///caaaa
Answer: [Yes]
Justification: The authors have reviewed the NeurlPS Code of Ethics, and have ensured that the paper conforms in all respects with these guidelines.
Guidelines:
26

---

The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss that our method has the capacity to reduce the number of animal subjects used in experients.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper focused on methodology for animal studies, and we do not foresee our approaches being misused or used irresponsibly. The data used are publicly available, coming from a high profile Nature paper.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
27

---

12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly cite existing packages and datasets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification:
We are not releasing any packages, benchmarks, or other assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Data used the paper did not involve crowdsoureing nor human subjects. Data were publicly available-se see Appendix For details.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
28

---

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Data used the paper did not involve crowdsourcing nor human subjects. Data were publicly available-see Appendix for details.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
29


Reviews:
[{"review_id": "jspElFoTzW", "reviewer": "Reviewer_pQ5n", "summary": "This work investigate the causal effect estiamtion in the Optogenetics, which is interestimg in causality in science. The paper proposed a nonparametric causal inference framework for analyzing “closed-loop” designs in sequential setting and  extends “excursion effect” methods to enable estimation of causal contrasts for treatment sequences. The writing is good, and the toptic is important for the causality and biologic community.", "questions": "1.I am kind of confused for the definition and notation of treatment, what's the difference of G and A? I thought they are the same but the description in intro confuses me. optogenetic manipulation is not about the laser and no-laser.\n\n2. The experimental evaluation is weak. Indeed there are only one baseline method, i.e., HC to compare with the proposed estimator, which is is too limited, and it is not convincing that the proposed method is effective.\n\n3.There are many sequential models (e.g., Causal Transformer, CRN, RMSN ,etc) which can handle this kind of time-dependent causal inference task.", "limitations": "N/A", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The investigated problem is important.\n\n2. The presentation and writing is good.", "weaknesses": "1. Some claims are not clear, which makes the paper unreadable, especially for those who are not familiar on biologic and optogenetics\n\n2. The experimental evaluation is weak.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and questions. Most of the major concerns are, we believe, a result of misunderstandings of the goals and claims of our work. We hope that our revisions will make them clearer, and address the questions below:\n\n1. \"Some claims are not clear, which makes the paper unreadable, especially for those who are not familiar on biologic and optogenetics.\"\n\n- The reviewers unanimously felt that a strength of our work is the importance of the methods for applications in neuroscience, thus we feel we should maintain some domain-specific language. We make a significant effort to provide the necessary background to motivate our methods. Moreover, the problem setup and counterfactual notation we use follow the conventions of the sequentially randomized experiments causal inference literature (e.g., MSMs, HR-MSMs) as well as much of the causal machine learning research (e.g., RMSNs, CRNs, CTs). Thus, no domain-specific knowledge should be necessary to understand the key methodological and theoretical development (Sections 2, 3, 4.1). Our contributions to the field of counterfactual-based causal inference are general and extend well-established statistical theory.\n\n2. \"I am kind of confused for the definition and notation of treatment, what's the difference of G and A? I thought they are the same but the description in intro confuses me. optogenetic manipulation is not about the laser and no-laser.\"\n\n- We are sympathetic that the biology may be unfamiliar to some readers and, in our revision, we will further emphasize the distinction between group ($G$) and laser ($A$). The group, $G$, and laser stimulation, $A$, are indeed very different. Optogenetics studies often include randomly assigned groups of “light-sensitive” animals ($G = 1$), and light-insensitive negative controls ($G = 0$). Animals in the $G = 1$ group express a protein that causes their neurons to be activated in response to laser stimulation, whereas negative control animals ($G = 0$) do not express this protein, and thus do not exhibit this response to the laser. Animals in *both* groups can receive the laser treatment ($A_t = 1$), or not ($A_t = 0$), on each trial, $t$, but the application of the laser will not influence the neuronal activity in the control group ($G=0$) . This is a fundamental distinction, which we make a significant effort to explain in the paper. Indeed, on line 32 we state: “Experiments often include both treatment ($G=1$) and negative-control ($G=0$) groups, with animals assigned randomly to each. While the laser is often applied on a random subset of trials in both groups, only treatment animals express the protein that enables the laser to trigger the target neural response. The control group thus controls for “off-target” effects such as the laser heating the brain…” \n\n3. \"The experimental evaluation is weak. Indeed there are only one baseline method, i.e., HC to compare with the proposed estimator,\"\n\n- See global “Author Rebuttal for a detailed response. A primary contribution of our work is the introduction of **new target estimands** (i.e., they allow us to ask new causal questions that previous methods cannot): there are *no existing estimators* against which we can compare our proposed methods.\n\n- We believe that the reviewer’s description of our empirical evaluation as “weak” may be a result of misunderstanding. For example, “HC” is *not* the “only one baseline method.” Rather “HC” is a family of variance estimators that can be used to estimate uncertainty for our point estimators, i.e., they can be used as components of our proposed framework. We describe in the Results Section 4.1 that: “the sample size-adjusted HC3-based CIs achieve 95% coverage. All CIs achieve 95% coverage when n is large, showing that we can conduct valid inference in all settings.” Moreover, in the Figure 2 caption, where the “HC” results are presented, we describe that the results show “[t]he coverage of 95% CIs constructed using one of three established robust variance estimators and our robust variance estimator. The nominal coverage is reached for either large $n$ or large $T$ for all estimators.”\n\n- As we emphasize in the global response, we evaluated our methods across a wide range of simulation settings (see Tables 1-3 in the attached 1-page PDF with updated numerical results), and a very thorough application for which the results were confirmed and commended by the original neuroscientists who authored the Nature paper. \n\n4. \"There are many sequential models (e.g., Causal Transformer, CRN, RMSN ,etc) which can handle this kind of time-dependent causal inference task.\"\n\n- As we detail in the global response, we omitted these methods because they are not applicable in our setting. In our revision, we will expand our literature review to include them, and explain the reasons why they are not suitable for this problem: \n\n“While RMSNs (Lim et al., 2018), CRNs (Bica et al., 2020) and CTs (Melnychuk et al., 2022) can be used to estimate causal effects over time, these methods have important limitations: (a) these methods all target the effects of **static** (i.e., fixed) treatment sequences and depend *heavily* a positivity assumption, and thus cannot be applied in closed-loop designs; (b) the target estimands for these methods are *always* conditional on complete history of covariates/outcomes before treatment initiation, $H_{t - \\Delta + 1}$, whereas our estimands may depend on all history variables if desired, but may be averaged over some or all of these variables (often yielding greater statistical power) depending on the user’s preference; and (c) unlike for our methods, there are no rigorous statistical guarantees (e.g., consistency, asymptotic normality) that have been proven for RMSNs, CRNs or CTs, and correspondingly, no inferential tools (e.g., confidence intervals) developed for uncertainty quantification.”", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y24J1mKK1B", "reviewer": "Reviewer_z4yk", "summary": "This paper introduces a causal inference framework for analyzing close-loop optogenetics experiments. The authors propose using HR-MSMs to estimate the sequential excursion effects (causal effects of specific neural stimulation sequences on behavioral outcomes). Specifically, they develop an IPW estimator with a scalable implementation and provide the theoretical guarantee of consistency and asymptotic normality under mild assumptions. The proposed framework extends the existing approaches to handle longer treatment sequences and addresses their limitations such as positivity violations. Through both simulation studies and an application to a real-world optogenetics study, the authors show that the proposed framework can reveal longitudinal causal effects that are often obscured by standard optogenetics analysis methods due to \"treatment-confounder\" feedback in closed-loop designs.", "questions": "1. Please see the questions in the Weaknesses part.\n\n2. I wonder how is $\\Delta$ determined and How would the value of $\\Delta$ affects the sequential excursion effects?", "limitations": "The authors adequately addressed the limitations of their work.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper proposes the first formal causal inference framework based on HR-MSMs for closed-loop optogenetics behavioral studies, addressing the challenges (e.g., positivity violations and \"treatment-confounder\" feedback) in the standard methods.\n\n2. The authors prove the consistency and asymptotic normality of the proposed IPW estimator, under mild assumptions.\n\n3. The proposed framework is evaluated on both simulated and real optogenetics studies, and demonstrates its effectiveness in analyzing more complex causal effects in closed-loop designs than standard analyses.", "weaknesses": "1. The focus of this paper centers on an application study that refines causal effect estimation within a specialized causal graph, akin to Figure 1E, rather than introducing broad methodological advancements in causal inference. The authors adapt Marginal Structural Models (MSMs) to dynamic treatment regimes and introduce History-Restricted MSMs (HR-MSMs) for estimating sequential excursion effects. This setup closely relates to estimating causal effects in temporal sequences. It is unclear what is the difference/gap between the proposed framework and the existing temporal causal inference methods. I wonder if the authors could discuss this in more detail. For instance, can any temporal causal inference framework estimate sequential excursion effects accurately? What are the challenges in performing causal inference the closed-loop optogenetics behavioral studies compared to more general (spatial-)temporal studies?\n\n2.  I think it would be better if the authors explicitly introduce the problem settings in Section 2 rather than mention it together with relevant literature. The current relevant literature part involves many technical details, which makes it slightly difficult for readers to understand the gap between the proposed method and existing works.\n\n3. The authors should consider adding more baselines in experiments. In this paper, the authors did not use any baselines, and thus it is unclear whether the proposed framework outperforms existing methods.\n\n4. The settings and evaluation metrics of the experiments need to be revised. \n\na. I think one of the advantages of the proposed method is to handle more complex local causal effects or longitudinal effects, whereas the standard methods only estimate macro effects. Intermediate causal effects can be canceled out in some cases, so only estimating macro effects can be biased and inaccurate. However, the authors did not conduct any simulation studies to demonstrate the advantage of the proposed method. \n\nb. The current presentation of experimental results primarily relies on descriptive analysis, and I think the authors should use some quantitative metrics to evaluate the performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their helpful questions, comments and feedback on our work. We believe that the proposed revisions and clarifications in line with the responses below will improve the strength of the paper.\n\n1. “The focus of this paper centers on an application study that refines causal effect estimation within a specialized causal graph, akin to Figure 1E, rather than introducing broad methodological advancements in causal inference.”\n\n- We used the causal graph in Figure 1E to provide an illustration, which may have suggested limited applicability. To clarify, the proposed methods are applicable in **all sequentially randomized settings**, going far beyond the optogenetics studies that motivated our work. In particular, treatment at each time point may depend on the *complete* history of observed variables, $H_{t}$ (as opposed to just $X_t$ as in the illustrative causal graph). In the revision, we will emphasize that this is an illustrative causal graph, but elaborate on the generality of the methodology to causal effect estimation within a far more general set of causal graphs. The methodological advancement is the introduction of specific estimands that are valid in the presence of positivity violations, paired with powerful statistical approaches. Such positivity violations occur regularly across many application areas beyond neuroscience, for instance in mobile health studies where active treatment may be systematically withheld for ethical reasons (e.g., see discussion of treatment “availability” in Section 4 in Boruvka et al. (2018; JASA)).\n\n2. “[...] This setup closely relates to estimating causal effects in temporal sequences. It is unclear what is the difference/gap between the proposed framework and the existing temporal causal inference methods. I wonder if the authors could discuss this in more detail. [...]”\n\n- Please see global “Author Rebuttal” for a detailed response: the major challenges in this case are the large number of timepoints (for which traditional MSMs are unstable) and the inherent positivity violations in the design (where no existing methods can handle treatment sequences longer than length one). Further, existing machine learning-based causal methodology (RMSNs, CRNs, CTs) rely on positivity, and do not provide statistical guarantees or inferential tools (e.g., confidence intervals).\n\n3. “I think it would be better if the authors explicitly introduce the problem settings in Section 2 rather than mention it together with relevant literature…”\n\n- We thank the reviewer for this suggestion and agree with their point. In the revision, we will split Section 2 into two sections. The first will be a less technical summary of existing methods, highlighting the gaps. The second will then explicitly introduce the problem and notation.\n\n4. “The authors should consider adding more baselines in experiments. In this paper, the authors did not use any baselines, and thus it is unclear whether the proposed framework outperforms existing methods.”\n\n- Please see global “Author Rebuttal” for a detailed response. Briefly, our paper introduces new target causal estimands. Consequently, no existing estimators could be used to compare against our approach. That said, we will add a simulation in the appendix (see Figure 1 in attached 1-page PDF) to show how the proposed methods uncover effects when standard “macro” approaches yield null results (see Point 6 below).\n\n5. “The settings and evaluation metrics of the experiments need to be revised.”\n\n- We thank the reviewer for their suggestion; we believe this criticism reflects our choice to present our findings graphically as opposed to numerically (e.g., in tables). In the revision, we will provide the numerical bias and coverage results in a table, and display mean-squared error (MSE) of the proposed estimator across the simulation settings (see Table 1 in attached 1-page PDF). \n\n6. “[...] so only estimating macro effects can be biased and inaccurate. However, the authors did not conduct any simulation studies to demonstrate the advantage of the proposed method. ”\n\n- In Appendix 2, we show mathematically that macro effects may be misleading. Specifically, we demonstrate that, even if the magnitude of local causal effects are large, the macro effects can be exactly zero. In the revision, we will include a simulation that demonstrates this more concretely: macro approaches will show null results, whereas the proposed methods will uncover more nuanced non-null effects (see Figure 1 in attached 1-page PDF for preliminary results).\n\n7. “The current presentation of experimental results primarily relies on descriptive analysis, and I think the authors should use some quantitative metrics to evaluate the performance.”\n\n- See response to Point 5 above: we will provide explicit numerical tables (in addition to our figures) to demonstrate bias, MSE, and coverage properties (see Tables 1-3 in attached 1-page PDF).\n\n8. “I wonder how is $\\Delta$ determined and How would the value of $\\Delta$ affects the sequential excursion effects?”\n\n- The number of intervention timepoints in the counterfactual outcome of interest ($\\Delta$) determines the nature of the effects being estimated. However, 2-3 intervention timepoints are often sufficient to capture a rich collection of sequential excursion effects. We found that, in our application, conclusions were stable across a range of $\\Delta$ values (e.g., in Section 4.2, we mention that “[w]e fit comparable models for sequences as long as $\\Delta=7$ and found results were similar across $\\Delta$ values.”). In practice, the value of $\\Delta$ may be constrained, as the variance of parameter estimates can grow if $\\Delta$ is set too large (likely for the same reason that the parameters of non-history-restricted MSMs grows prohibitively large as the number of timepoints increases). In our revision, we will emphasize that $\\Delta$ should be chosen on the basis of subject matter knowledge.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sL4jBXbV9n", "reviewer": "Reviewer_manL", "summary": "The paper proposes novel causal inference framework for closed-loop optogenetics behavioral studies, develops causal inference estimation method for sequential excursion effects that capture local causal contrasts within the same treatment group. The proposed method is robust to accommodate both positivity violations and longer treatment sequences. The paper provided a practical computational implementation and analyze the method with theoretical guarantees. More interestingly, it empirically corroborates their studies with real-applications.", "questions": "- Proposition 3.4 $Z_i$ is introduced but the proposition only uses $Z$ throughout. would be good to specify the relationship between $Z_i$ and $Z$.", "limitations": "Yes.", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The utility and implications of counterfactual causal effect under dynamic treatment policies are well-described by authors. I found it especially interesting that the authors performs empirical validation through applying their method on existing Nature paper and found their computational result is corroborated with the original authors of the existing paper. This shows a thorough post-hoc validation of the proposed method.", "weaknesses": "The largest weakness I see here is both experimental results uses linear or generalized linear model for $m$. I see the advantages of well-developed generalized linear model in experimental sciences for good interpretation and statistical guarantees, though would be curious to find out how the authors see the method would work beyond linear models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive feedback on our work. The reviewer raises a very interesting methodological/modeling question, and identifies a notational issue, both of which we address below.\n\n1. “The largest weakness I see here is both experimental results uses linear or generalized linear model for $m$. I see the advantages of well-developed generalized linear model in experimental sciences for good interpretation and statistical guarantees, though would be curious to find out how the authors see the method would work beyond linear models.”\n\n- The projection parameter specification, as laid out in Section 3, is entirely generic, and remains well-specified even for more complicated models $m$ (e.g., random forests, kernel-based methods, neural networks). However, as you rightly point out, we adopt linear or generalized linear models (GLMs) as our working models in both simulations and the applied study. For the questions of scientific interest in the applied optogenetics study, we felt that the target causal effects (e.g., dose response effects and effect dissipation, pooled across time) could be well approximated using GLMs.\n\n- The theory applies for a large class of working mean models $m$, and does not rely on any distributional/likelihood-based assumptions. Beyond their familiarity to most readers, the technical reason for opting for GLMs in our applied examples is—as you again correctly identify—for statistical guarantees (e.g., convergence properties and inferential tools). In particular, Theorem 3.5 stipulates that $m$ must be “Donsker” in $\\beta$: GLMs automatically satisfy this technical requirement (see, e.g., Examples 19.7 and 19.8 in van der Vaart (2000)). Whether a model class is “Donsker” is a non-trivial query regarding its “complexity” or “size”, but can be investigated using machinery and results from empirical process theory (van der Vaart & Wellner (1996)). The situation is indeed more complicated for more flexible models, and this Donsker requirement is not always satisfied. However, we note that it *is* satisfied for some formulations of random forests (see, e.g., Theorem 3.2 in Scornet (2016)) and kernel estimators (see, e.g., Theorems 3.7 and 3.13 in Beutner & Zähle (2023)). In our revised Discussion section, we will discuss this issue in more detail, and note that future research will focus on applying random forests and kernel estimators (when they satisfy the Donsker requirement), and the more challenging problem of studying the statistical properties of flexible model specifications when the Donsker requirement is not satisfied.\n\n2. “Proposition 3.4 $Z_i$ is introduced but the proposition only uses $Z$ throughout. would be good to specify the relationship between $Z_i$ and $Z$.”\n\n- We thank the reviewer for noticing this notational issue. As we mention in the beginning of Section 2, we adopt a common statistical shorthand in that “we often suppress [subject-specific] indices to reduce notational burden”. In this case, $Z$ represents a generic observation (i.e., from an arbitrary subject), whereas $Z_i$ represents that same data, but specifically from subject $i$. In the revision, we will specifically reiterate in Proposition 3.4 that we omit the subject-specific index for clarity.\n\n**References**:\n- van der Vaart AW, Wellner JA. Weak convergence. Springer New York; 1996.\n- van der Vaart AW. Asymptotic statistics. Cambridge University Press; 2000 Jun 19.\n- Scornet E. On the asymptotics of random forests. Journal of Multivariate Analysis. 2016 Apr 1;146:72-83.\n- Beutner E, Zähle H. Donsker results for the empirical process indexed by functions of locally bounded variation and applications to the smoothed empirical process. Bernoulli. 2023 Feb;29(1):205-28.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank the reviewers for their thoughtful feedback, and were pleased to see agreement on the following positive points:

**Importance & Novelty**:
- z4yk: “This paper proposes the first formal causal inference framework…for closed-loop optogenetics behavioral studies addressing the challenges (e.g., positivity violations and "treatment-confounder" feedback) in the standard methods.” 
- manL: “The paper proposes novel causal inference framework for closed-loop optogenetics behavioral studies.”
- pQ5n: “Causal effect estimation in [o]ptogenetics…is interesting in causality in science...and the topic is important for the causality and biologic community.”

**Methodological Contributions**:
- z4yk: “The ... framework extends the existing approaches to handle longer treatment sequences and addresses their limitations...and [provides a] scalable implementation and… theoretical guarantee of consistency and asymptotic normality under mild assumptions.”   
- manL: “The paper develops causal inference estimation method for sequential excursion effects that capture local causal contrasts within the same treatment group. The ... method is robust to ... positivity violations and longer treatment sequences.The paper provided a practical computational implementation and analyze the method with theoretical guarantees.“

**Experimental Evaluation**:
- manL:  “... the authors perform empirical validation through applying their method on existing Nature paper and found their computational result is corroborated with the original authors of the existing paper. This shows a thorough post-hoc validation of the proposed method.”
- z4yk: “The proposed framework is evaluated on both simulated and real optogenetics studies, and demonstrates its effectiveness in analyzing more complex causal effects in closed-loop designs than standard analyses.”

**Presentation & Writing**:
- manL: “The utility and implications of counterfactual causal effect under dynamic treatment policies are well-described.”
- pQ5n: “The presentation and writing is good.”


There were also concerns raised by more than one reviewer, which we address below, leaving other points for individual reviewer responses.

**Experimental Evaluation**:

- z4yk: “The authors did not use any baselines, and thus it is unclear whether the proposed framework outperforms existing methods.”
- pQ5n: “The experimental evaluation is weak. ... there are only one baseline method, i.e., HC to compare with the proposed estimator, which is is too limited.”

**Response**

A key contribution of our work is the introduction of new target estimands. The motivation for this work is precisely the fact that no existing estimands or estimators were suitable for closed-loop designs, so there is no appropriate method to compare against. There are also no consensus benchmark datasets to analyze.

Given this, we respectfully disagree that the empirical evaluation of the methodology is “weak”. We followed two evaluation strategies:
1. application to realistic simulated data, with known ground truth
2. application to real data, then validating the results with domain experts

For 1., we simulate data across six sample sizes and three total timepoint scenarios (Apdx. Fig. 5), which cover all optogenetics applications we have seen in the literature. We show that our estimators are unbiased and that nominal coverage (e.g., 95%) can be achieved in nearly all scenarios, thus verifying our theoretical statistical results. 
For 2., we analyzed a landmark optogenetics study, and have replicated and expanded the authors’ findings.

The “HC” standard error estimators are **not** competitors—these provide uncertainty quantification for our proposed point estimator. Beyond the large sample variance we derived, these alternatives can improve coverage in small sample sizes. To avoid the confusion from labeling the variance estimator $\hat{V}$ from Section 3 as “Ours”, we will relabel it as “Large Sample” in the revision. 

**Distinction from Existing Methods**

- z4yk: “It is unclear what is the difference/gap between the proposed framework and the existing temporal causal inference methods.”
- pQ5n: “There are many sequential models (e.g., Causal Transformer, CRN, RMSN ,etc) which can handle this kind of time-dependent causal inference task.”

**Response**

While our methods estimate causal effects in sequential settings, standard approaches like traditional MSMs often do not perform well when there are a large number of timepoints (e.g., as in many optogenetics studies). In our revision, we will emphasize that MSM parameter variances grow prohibitively large as the number of timepoints rise, an issue that our method does not suffer from for reasonable values of $\Delta$. We will also describe how existing history-restricted MSMs rely on a positivity assumption which is violated in closed-loop designs, and/or cannot accommodate longer treatment sequences. 

RMSNs, CRNs and CTs can be used to estimate causal effects over time, but have limitations that preclude their use in our setting: 

- no statistical guarantees (e.g., asymptotic normality) or inferential tools (e.g., confidence intervals)
- these methods target the effects of **static** (i.e., fixed) treatment sequences and depend heavily on a positivity assumption, and thus cannot be applied in closed-loop designs
- the target estimands are *always* conditional on complete history before treatment initiation, $H_{t - \Delta + 1}$, whereas our estimands may depend on all history or averaged over some/all of these variables (often improving statistical power), if desired 

To address this, we will describe these methods and their lack of suitability for closed-loop optogenetic studies in our revision.

**1-page PDF**

We include:
- New simulation results showing how treatment-confounder feedback can obscure effects
- Updated bias results from simulations, showing unbiasedness
- Tables of simulation results: MSE, bias & coverage

Meta-review: This submission carefully develops novel methodology for a concrete application area. I consider the fact that it is specifically tailored to what could be viewed as a "narrow application area" a strength rather than a weakness. This led the authors to develop a new, suitable target estimand in their setting taking into account the real-world nuances (e.g., positivity violations). Both the theoretical underpinnings as well as the experimental validation are overall convincing to me (and most reviewers). Given that it is difficult to squeeze a detailed domain/problem description for non-experts into the limited space, the paper is overall well written and structured and was mostly considered fairly easy to follow and read. Ultimately, I believe this paper is going to be relevant and of interest to a substantial fraction of the NeurIPS community and the ideas developed here may be of interest also outside the specific application area. I encourage the authors to work in some of the elucidating clarifications from the rebuttal to further strengthen the paper and make it as widely accessible as possible.
Decision: Accept (poster)