Title: Causal Inference in the Closed-Loop: Marginal Structural Models for Sequential Excursion Effects
Venue: NeurIPS 2024
Accepted: yes
Keywords: marginal structural models

optogenetics

excursion effects

neuroscience

dynamic treatment regimes

micro-randomized trials

sequentially randomized experiments
Abstract: Optogenetics is widely used to study the effects of neural circuit manipulation on behavior. However, the paucity of causal inference methodological work on this topic has resulted in analysis conventions that discard information, and constrain the scientific questions that can be posed. To fill this gap, we introduce a nonparametric causal inference framework for analyzing "closed-loop" designs, which use dynamic policies that assign treatment based on covariates. In this setting, standard methods can introduce bias and occlude causal effects. Building on the sequentially randomized experiments literature in causal inference, our approach extends history-restricted marginal structural models for dynamic regimes. In practice, our framework can identify a wide range of causal effects of optogenetics on trial-by-trial behavior, such as, fast/slow-acting, dose-response, additive/antagonistic, and floor/ceiling. Importantly, it does so without requiring negative controls, and can estimate how causal effect magnitudes evolve across time points. From another view, our work extends "excursion effect" methods---popular in the mobile health literature---to enable estimation of causal contrasts for treatment sequences greater than length one, in the presence of positivity violations. We derive rigorous statistical guarantees, enabling hypothesis testing of these causal effects. We demonstrate our approach on data from a recent study of dopaminergic activity on learning, and show how our method reveals relevant effects obscured in standard analyses.

Causal Inference in the Closed-Loop: Marginal Structural Models for Sequential Excursion Effects

Abstract

Optogenetics is widely used to study the effects of neural circuit manipulation on behavior. However, the paucity of causal inference methodological work on this topic has resulted in analysis conventions that discard information, and constrain the scientific questions that can be posed. To fill this gap, we introduce a nonparametric causal inference framework for analyzing 'closed-loop' designs, which use dynamic policies that assign treatment based on covariates. In this setting, standard methods can introduce bias and occlude causal effects. Building on the sequentially randomized experiments literature in causal inference, our approach extends history-restricted marginal structural models for dynamic regimes. In practice, our framework can identify a wide range of causal effects of optogenetics on trial-bytrial behavior, such as, fast/slow-acting, dose-response, additive/antagonistic, and floor/ceiling. Importantly, it does so without requiring negative controls, and can estimate how causal effect magnitudes evolve across time points. From another view, our work extends 'excursion effect' methods-popular in the mobile health literature-to enable estimation of causal contrasts for treatment sequences greater than length one, in the presence of positivity violations. We derive rigorous statistical guarantees, enabling hypothesis testing of these causal effects. We demonstrate our approach on data from a recent study of dopaminergic activity on learning, and show how our method reveals relevant effects obscured in standard analyses.

1 Introduction

Optogenetics is a neuroscience technique to 'turn on/off' neurons in vivo in real-time, with millisecond time resolution. It works by shining lasers on neurons that have been genetically modified through viral infection to express a light-sensitive protein. It is one of the most popular assays with roughly 700 references to it in 2023 alone. 2 Optogenetics is often applied to study the causal effect of manipulating specific brain circuits while animals (e.g., mice) perform behavioral tasks to study, for example, learning and decision-making. These tasks are typically composed of a sequence of trials, t ∈ { 1 , 2 , ..., T } , each of which involves presentation of stimuli and an opportunity for a behavioral response. For example, a trial might begin with a cue (e.g., a light), which indicates that a lever press will trigger delivery of a food reward. Investigators might want to know, for instance, whether applying optogenetic stimulation on a random subset of trials alters the rate at which mice press the lever. On trial t , an animal's behavioral outcome, Y t , time-varying covariates, X t , and optogenetic (treatment) indicator, A t , are observed. Experiments often include both treatment ( G = 1 ) and negative-control ( G = 0 ) groups, with animals assigned randomly to each. While the laser (i.e., the sequential treatment, A t ) is often applied on a random 3 subset of trials in

Policy 0:

𝒂

(")

1

𝑌

#

0

𝑌

#

Policy 1:

𝒂

($)

𝔼[𝑌

#

(𝒂

%

) - 𝑌

#

(𝒂

&

)]

Blip Excursion Effect

No Effect

𝔼[𝑌

#

(𝒂

%

) - 𝑌

#

(𝒂

&

)]

Δ = 1

Opto (t)

0

Trial Number

Mean Counterfactual:

E





Yt

(

a

(

1

)

)





𝑌

!

measured

Policy 0:

𝒂

(")

1

0

0

𝑌

#

0

1

0

𝑌

#

0

0

1

𝑌

#

0

0

0

𝑌

#

Policy 1:

𝒂

($)

Policy 2:

𝒂

(%)

Policy 3:

𝒂

(&)

𝔼[𝑌

#

(𝒂

*

) - 𝑌

#

(𝒂

&

)]

𝔼[𝑌

#

(𝒂

+

) - 𝑌

#

(𝒂

&

)]

Effect Dissipation

Effect Persistence

𝔼[𝑌

#

(𝒂

*

) - 𝑌

#

(𝒂

&

)]

𝔼[𝑌

#

(𝒂

+

) - 𝑌

#

(𝒂

&

)]

𝔼[𝑌

#

(𝒂

%

) - 𝑌

#

(𝒂

&

)]

𝔼[𝑌

#

(𝒂

%

) - 𝑌

#

(𝒂

&

)]

Δ = 3

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

0

Trial Number

Mean Counterfactual:

E





Yt

(

a

(

2

)

)





𝑌

!

measured

0

0

0

𝑌

#

1

0

0

𝑌

#

0

1

0

𝑌

#

0

0

1

𝑌

#

1

0

1

𝑌

#

1

1

0

𝑌

#

0

1

1

𝑌

#

1

1

1

𝑌

#

𝔼[𝑌

#

(𝒜

%

) - 𝑌

#

(𝒂

&

)]

𝔼[𝑌

#

(𝒜

*

) - 𝑌

#

(𝒂

&

)]

𝔼[𝑌

#

(𝒂

-

) - 𝑌

#

(𝒂

&

)]

Dose-Response

Ceiling Effect

𝔼[𝑌

#

(𝒜

*

) - 𝑌

#

(𝒂

&

)]

𝔼[𝑌

#

(𝒂

-

) - 𝑌

#

(𝒂

&

)]

Δ = 3

Policy 0:

𝒂

(")

Policy 1:

𝒂

($)

Policy 2:

𝒂

(%)

Policy 3:

𝒂

(&)

Policy 4:

𝒂

(')

Policy 5:

𝒂

(()

Policy 6:

𝒂

())

Policy 7:

𝒂

(*)

𝒜

(%)

𝒜

(*)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t)

0

Trial Number

Mean Counterfactual:

E





Yt

(

a

(

7

)

)





𝑌

!

measured

B. Effect Dissipation

𝔼[𝑌

#

(𝒜

%

) - 𝑌

#

(𝒂

&

)]

None

Low

Med

High

E. Introduction to History-Restricted Marginal Structural Models

D. Closed-Loop (Dynamic Regime) Design DAG

U

Yi

Y2

X1

A1

X2

A2

X

Y

{0;3²=1

[03-

ut-△+

Yt

Marginalizeoverall

Deterministically“set"

Estimatecausal

datathroughtrial t-△

policyfor△trials

effectofintervening

onlast△trials

both groups, only treatment group animals ( G = 1 ) express the protein that enables the laser to trigger the target neural response. The control group thus controls for 'off-target' effects such as the laser heating the brain, and the optogenetic insertion surgery. To answer the question above, investigators often estimate the effect of optogenetic manipulation through comparisons such as ψ t = E [ Y t | G = 1] -E [ Y t | G = 0] . It is common to test whether ψ t = 0 at specific timepoints like the end of the study ( t = T ), or to conduct inference on summaries (e.g., ¯ ψ = 1 T ∑ t ψ t ). These between-group comparisons assess the intervention impact based on simple long-term, or 'macro'/'global' longitudinal effects. When studies randomly deliver treatment at each trial (i.e., with stochastic policies), within-group comparisons between laser and no-laser trials are also common (e.g., ˜ ψ = ∑ t { E [ Y t | A t = 1 , G = 1] -E [ Y t | A t = 0 , G = 1] } ).

Importantly, such comparisons do not lend themselves to testing within-group 'micro'/'local' longitudinal effects related to specific treatment sequence patterns. For example, one might ask whether there is a dose-dependent relationship between the outcome and the number of stimulations in the last five trials, or whether stimulation on two consecutive trials has a synergistic effect that is greater than if stimulation instead occurred on two non-consecutive trials. Figures 1A-C shows some representative micro longitudinal effects that are identifiable in many optogenetics studies, yet typically are not explored. Critically, such effects may be present even in studies in which one fails to detect the macro effects commonly tested. However, no formal causal inference framework has been applied to these studies, resulting in analysis conventions that limit the scope of questions researchers can ask.

Furthermore, certain experimental designs can complicate the use and interpretation of even standard analysis approaches. In 'closed-loop' (referred to as 'dynamic regimes' in the causal inference literature) designs, stimulation is applied depending on the behavior of the animal. For example, say a study tests if lever pressing for food, Y t , decreases if optogenetic stimulation ( A t = 1 ) is applied, with positive probability, only when animals approach the lever ( X t = 1 ). Since A t is randomized conditional on X t , one must incorporate X t into their analysis, but standard strategies like including X t as a covariate in a regression can obscure effects and induce bias. This is because 1) X t influences the probability of both the outcome and treatment, and thus can be cast as a time-varying confounder, and 2) X t also mediates the effect of prior treatments [7]; see the illustrative DAG in Figure 1D, though note that we generalize this setting later on to allow treatment to depend on the

complete history of previously measured variables at each time point. In this case, since treatment also influences both Y t and X t on subsequent trials, closed-loop designs induce 'treatment-confounder feedback' [7], which can lead to bias with standard analyses. We include an example in Appendix B to show how, when the treatment has opposing effects on Y t and X t , treatment and control groups can exhibit identical average (observed) outcome levels even if the laser causes a large immediate effect. Furthermore, standard regression approaches can actually induce collider-bias, and block mediators of the treatment effect [7]. Finally, if treatment policies deterministically rule out treatment (e.g., when X t = 0 ), certain effects are not identifiable: the positivity violation inherent to these designs precludes estimation of certain counterfactual distributions. Closed-loop designs therefore require specialized algorithms for valid causal inference.

More broadly, there have been a number of high profile calls for more rigorous definitions of causality and causal inference in neuroscience [1, 33, 4, 16]. However, to the best of our knowledge, existing methodological work [25, 10, 14] focuses on instrumental variable-based approaches to estimate causal effects of optogenetics on neural activity. Unlike our setting, these methods are restricted to datasets that include both measurements of the activity of the neurons stimulated by optogenetics, and the neurons those cells interact with. One can then conceptualize the neural activity of the stimulated neurons as treatment variables, and the optogenetics sequence as instruments. In addition to focusing on behavioral outcomes, we explicitly deal with sequentially randomized (and closed-loop) designs, whereas prior work treats each trial as an exchangeable draw, ignoring the sequential nature of trials.

Our contributions are (1) proposing the first formal counterfactual-based causal framing of these behavioral optogenetics designs, (2) developing an analysis framework based on history-restricted marginal structural models that enables the estimation of 'sequential excursion effects' that capture the local causal contrasts described above, (3) expanding excursion effect methodology to account for positivity violations, and to accommodate treatment sequences greater than length one, (4) providing estimators with efficient computational implementations and strong theoretical guarantees under minimal nonparametric conditions (verified in simulations), and (5) applying our methods to data from a high profile Nature paper, and showing how they reveal effects obscured by standard methods.

2 Notation and Related Work

In this section, we (i) provide the necessary notation and a brief review of relevant work, and (ii) describe the key methodological gap in the current literature: existing methods cannot estimate causal effects of proximal treatment sequences longer than one timepoint in closed-loop designs.

Notation Let O t = { X t , A t , Y t } be the vector of observed variables for an animal on trial t . We denote T as the number of trials and [ T ] as the set { 1 , 2 , ..., T } . A sample of subjects i = 1 , 2 , ..., n is collected but, as subjects are exchangeable, we often suppress indices to reduce notational burden. We express counterfactual variables, or potential outcomes , with parentheses. For example, Y t ( a t ) , represents the potential outcome that would be observed at trial t if a subject received the treatment sequence, a t = ( a 1 , . . . , a t ) . Overbars represent all history up to and including a given trial. For example, B j = ( B 1 , . . . , B j ) , for any sequence of variables { B t } T t =1 , and any j ∈ [ T ] . Finally, we define H t = ( X t , A t -1 , Y t -1 ) , so H t includes all information prior to the treatment 'decision' at t .

Relevant Literature Marginal structural models (MSMs) are often used to model the mean counterfactuals E [ Y t ( a t )] [28, 30, 29] in sequentially randomized experiments, though these typically do not perform well [21] when there are a large number of time points (e.g., as in many optogenetics studies): the variance of the model coefficients can grow prohibitively large. History-restricted MSMs [21] (HR-MSMs) model E [ Y t ( a ∆ ,t )] for some a ∆ ,t = ( a t -∆+1 , . . . , a t ) , typically with ∆ ≪ t . That is, HR-MSMs model the mean counterfactual outcome at time t , under an intervention defined on a proximal (often short) treatment sequence. As E [ Y t ( a ∆ ,t )] = E [ Y t ( A t -∆ , a ∆ ,t )] , by a consistency assumption, these estimands implicitly marginalize over the observed treatment sequence, A t -∆ , prior to the first point of intervention. However, any Markov-like assumptions made by the causal framework follow directly from the experimental design: HR-MSMs (and, by extension, our proposed methods) allow for X t , Y t to be causally affected by all prior trials (i.e., O j for j ∈ [ t -1] ). By placing structure on E [ Y t ( a ∆ ,t )] , the HR-MSM can borrow strength across treatment sequences a ∆ ,t , which can increase power when there are many trials. Figure 1E provides a graphical illustration of HR-MSMs. HR-MSMs are typically fit by using generalized estimating equations (GEE) with

inverse probability of treatment weighting (IPW). IPW resolves the dilemma with standard regression techniques in sequentially randomized experiments, outlined above, where failure to condition on time-varying confounders, X t , biases estimates (as treatment is randomized conditional on X t in closed-loop designs), but conditioning on X t induces confounding ( X t are colliders on the path between past treatments and subsequent outcomes, through unmeasured confounders, U , as shown in the DAG in Figure 1D)) [7]. HR-MSMs can also incorporate time-varying effect modifiers (e.g., see [23] and references), to test, for example, whether causal effects vary across trials, or animal-specific covariate levels.

The gaps: Sequential effects and positivity violations In designs that assign treatment randomly at each trial, HR-MSMs can be used to estimate the causal effect of specific deterministic treatment sequences a ∆ ,t that may differ from the observed sequence A t close to trial t , and are compatible with the experimental treatment rule ('policy'). Importantly, this enables estimation of interpretable causal parameters, such as the effect of treatment on the most recent trial, E [ Y t ( a t = 1) -Y t ( a t = 0)] . These causal contrasts have grown popular recently in the analysis of mobile health studies [5], where they are referred to as 'excursion effects.' However, current methods are restricted to estimating excursion effects for the ∆ = 1 case in experimental designs like ours, and thus preclude estimation of effects defined only for ∆ > 1 (e.g., the micro longitudinal effects in Figures 1 and 4). Mobile health studies often include treatment rules with positivity violations: due to ethical or practical constraints, treatment must be withheld in certain cases (e.g., no phone notifications while driving). [5] use the notation that treatment is withheld when the time-varying 'availability' indicator, I t , equals zero. Similarly, in 'closed-loop' optogenetics experiments, I t = 1 when the conditions are met such that neural manipulation may occur (e.g., when the animal approaches the lever in the example in Section 1). There have been proposals for methods intended to account for such implied positivity violations [19, 5, 26], such as the availability-conditional estimand [5]: E [ Y t ( a t = 1) -Y t ( a t = 0) | I t = 1] . However, estimands proposed for these settings are defined only for ∆ = 1 . Thus, in the presence of these positivity violations, there is currently no methodology to conduct causal inference for longer proximal treatment sequences. We note that machine learning based causal methods including causal transformers [18], counterfactual recurrent networks [3], and recurrent marginal structural networks [15] are comparable to HR-MSMs that condition on all measured variables prior to the first intervention timepoint ( t -∆+1 ). These methods target effects of static treatment sequences and require a positivity assumption, and thus cannot be applied in closed-loop designs. They also do not provide tools for statistical inference.

3 Methods

To fill the gaps identified above, we propose HR-MSMs for proximal sequences of dynamic treatment regimes, designed to be compatible with treatment availability restrictions in this scientific context. These estimands are defined for any ∆ ≥ 1 , can incorporate time-varying effect modifiers, and can dissect more intricate patterns of treatment over time, compared to standard excursion effects.

3.1 HR-MSMs for Dynamic Treatment Regimes

Adopting the notation from [5], we define I t := 1 ( P [ A t = 1 | H t ] > 0) as an 'availability indicator', i.e., I t = 0 if and only if active treatment (e.g., laser stimulation) is prohibited by design. Define D t = { d t : H t →{ 0 , 1 } | d t ( H t ) = 0 if I t = 0 } , for any t , to be the class of treatment rules at time t compatible with I t . In particular, we will consider the deterministic rules D ∗ t = { d (0) t , d (1) t } ⊂ D t , where d (0) t ≡ 0 , d (1) t ≡ I t . In words, d (0) t fixes A t = 0 , and d (1) t sets A t equal to I t . The treatment rules d (0) t , d (1) t ∈ D t represent the two most extreme policies whose effects remain identifiable. We can combine these time-specific rules to construct multiple time-point analogs of excursion effects compatible with availability restrictions: for ∆ ∈ N , we let D ∆ ,t be a subset of D ∗ t -∆+1 ×··· × D ∗ t , taking d ∆ ,t = ( d t -∆+1 , . . . , d t ) ∈ D ∆ ,t to be a sequence of ∆ treatment rules (compatible with availability restrictions) for trials j ∈ { t -∆+1 , ..., t } . The counterfactual outcome under this policy sequence is defined to be

That is, Y t ( d ∆ ,t ) is the counterfactual outcome under an intervention that leaves the natural value of treatment for the first t -∆ trials, then sequentially determines treatment by applying d t -∆+ j to H t -∆+ j ( d j -1 ,t -∆+ j -1 ) , for j ∈ [∆] , where d j -1 ,t -∆+ j -1 = ( d t -∆+1 , . . . , d t -∆+ j -1 ) .

Letting V t ⊆ H t be a set of effect modifiers at trial t , we seek to estimate E [ Y t ( d ∆ ,t ) | V t -∆+1 ] , the counterfactual mean outcome, conditional on effect modifiers that are observed before the treatment decision of trial t -∆+1 . By construction, these estimands are identifiable under standard causal assumptions (see Section 3.2). We discuss their interpretation, and compare with existing proposals in Appendix C.1. When ∆ > 1 and studies have many trials, there may be many potential treatment rule sequence combinations. We thus propose to estimate effects of these interventions with an MSM on the (conditional) means of the counterfactuals (1): m ( t, d ∆ ,t , V t -∆+1 ; β ) ≈ E [ Y t ( d ∆ ,t ) | V t -∆+1 ] , where m is a fixed known function. We aim to conduct inference on the MSM parameters, β , but we do not assume that the model is necessarily well-specified, and thus treat the MSM parameters as projections onto the working model m [20, 32]:

for some fixed non-negative weight function h . This projection approaches lies between a fully parametric strategy, that assumes m is correctly specified, and a fully nonparametric approach, that places no structure across the target causal quantities. The target β 0 is defined as the parameter of the best fitting working model m (i.e., closest in L 2 ( P ) ). In practice, the choice between considering m as a working model or as a correctly specified model amounts to a trade-off between bias and variance-see the discussions in [13, 12] where analogous projection parameters are proposed.

3.2 Identification and Estimation

In this section, we first describe the causal assumptions under which the effects of interest are identified. We then develop an inverse probability-weighted estimator of the MSM parameters, and derive their asymptotic properties. While we focus on dynamic regime HR-MSMs below, our results also apply to static regime HR-MSMs in the case that there are no availability issues (i.e., I t ≡ 1 ). There, the treatment rule d ∆ ,t reduces to a corresponding static sequence a ∆ ,t .

For each t , define the treatment probability function π t ( a ; H t ) := P [ A t = a | H t ] . We make the following standard assumptions, which are expected to hold in many optogenetics designs:

Assumption 3.1.

Assumption 3.2.

Consistency:

Y

t

(

d

∆

Positivity: For all

t

,t

) =

∈ {

Y

∆

t

, whenever

, . . . , T

}

A

, and

j

d

t

=

d

j

(

∈ D

H

∗

t

,

j

π

)

t

, for all

(

d

t

(

H

t

j

);

∈ {

H

t

)

t

-

∆+1

ϵ

, . . . , t

, w.p. 1

Assumption 3.3. Sequential randomization: A s ⊥ ⊥ Y t ( d ∆ ,t ) | H s , for all t ∈ { ∆ , . . . , T } , s ∈ { t -∆+1 , . . . , t }

We provide a detailed discussion of these assumptions in practice in Appendix C.2. The following result says that these three assumptions are sufficient for identification of the counterfactual means E [ Y t ( d ∆ ,t ) | V t -∆+1 ] , and of the MSM parameters β 0 .

Proposition 3.4. Under Assumptions 3.1-3.3, we have

Recall that Z i = {O t,i } T t =1 is the totality of data observed on subject i ; suppressing subject-specific index for clarity, define ϕ ( Z, · ) : R q → R q via

where M ( t, d ∆ ,t , V t -∆+1 ; β ) = ∇ β m ( t, d ∆ ,t , V t -∆+1 ; β ) . Then, assuming the solution to (2) is unique, and the working model m is differentiable in β , the MSM parameters β 0 are identified through the estimating equation 0 = E P ( ϕ ( Z, β 0 )) .

}

The result of Proposition 3.4 is a population inverse probability-weighted estimating equation for the target parameters β 0 . This estimating equation motivates a corresponding IPW estimator, ̂ β , solving the empirical IPW estimating equation P n [ ϕ ( Z, ̂ β )] = 0 . In the optogenetics applications of interest, the propensity scores π t are known by design, and can be plugged in when estimating ̂ β .

We now prove asymptotic normality of the our estimator, ̂ β , under mild conditions. We require the following notation: define A ( β ) = E [ ϕ ( Z, β ) ϕ ( Z, β ) T ] and B ( β ) = E [ ∇ β ϕ ( Z, β )] .

Theorem 3.5. Suppose Assumptions 3.1-3.3 and the following conditions hold:

(i) The minimizer β 0 in (2) is unique;

(ii) m ( t, d ∆ ,t , V t -∆+1 ; β ) is Donsker in β , continuously differentiable at β 0 , uniformly in V t -∆+1 ;

(iii) In a neighborhood around β 0 , A ( β ) and B ( β ) are finite-valued, and B ( β ) is non-singular;

(iv) ̂ β p → β 0 .

Theorem 3.5 gives the asymptotic distribution of the estimator ̂ β . The conditions (i)-(iv) are relatively mild; see Appendix C.3 for a discussion. Theorem 3.5 provides a strategy to construct asymptotically valid Wald-based confidence intervals for the MSM parameters β 0 : for any β we can take

and define ̂ V = ̂ B ( ̂ β ) -1 ̂ A ( ̂ β ) ̂ B ( ̂ β ) -1 , which is consistent for V ( β 0 ) . Then, for j ∈ [ q ] , an (1 -α ) confidence interval for β j, 0 is given by ̂ β j ± z 1 -α/ 2 √ ̂ V jj n , where z 1 -α/ 2 is the (1 -α/ 2) -quantile of the standard normal distribution, and ̂ V jj is the j -th diagonal element of ̂ V . Confidence intervals for any linear combination of the β parameters can be constructed in a similar fashion.

We provide an implementation that builds the necessary dataset (with each observation copied once for every regime in D ∆ ,t ), calculates the corresponding IPW weights, and estimates the HR-MSM parameters β by solving the estimating equation in expression 2 using the rootSolve R package [11]. The process takes about 10 seconds on a standard laptop, for > 100 , 000 total (pre-copy) trials.

4 Experiments

4.1 Simulation Studies

Experimental Setup We sought to assess performance of the proposed estimator, and identify variance estimators that yield nominal coverage in the small n settings common in optogenetics studies. To evaluate the accuracy of our framework in estimating mean counterfactuals, we designed the simulations such that the target estimands--contrasts of mean counterfactuals--corresponded to regression coefficients from the true HR-MSM. The data were simulated to mimic closed-loop optogenetics designs with positivity violations: we drew i) X 0 ∼ Bernoulli(1 / 2) ; ii) A t | X t ∼ Bernoulli( 1 2 X t ) , for t ∈ { 0 , . . . , T } ; iii) X t | A t -1 ∼ Bernoulli(0 . 4 + 0 . 4 A t -1 ) , for t ∈ [ T ] ; and iv) Y t | X t -1 , A t -1 , X t , A t ∼ N ( α 1 X t -1 + α 2 A t -1 + α 3 X t + α 4 A t , σ 2 t ) , for t ∈ [ T ] , where ( α 1 , α 2 , α 3 , α 4 ) = (0 . 25 , 2 , 1 . 75 , 0 . 5) , and σ 2 t = 1 for all t . These set availability indicator I t ≡ X t , for all t , and result in marginal probabilities P [ X t = 1] = 1 2 , P [ A t = 1] = 1 4 , for all t . We obtain a closed form for the parameters of the saturated two time-point dynamic treatment regime HR-MSM: letting d 2 ,t = ( d t -1 , d t ) ∈ D 2 ,t be arbitrary, and defining J t -1 := 1 ( d t -1 ≡ d (1) t -1 ) , J t := 1 ( d t ≡ d (1) t ) , we derive in Appendix D.1 that E ( Y t ( d 2 ,t )) = β 0 + β 1 J t -1 + β 2 J t + β 3 J t -1 J t , where β 0 = 0 . 5 α 1 + 0 . 4 α 3 , β 1 = 0 . 5 α 2 + 0 . 2 α 3 , β 2 = 0 . 4 α 4 , and β 3 = 0 . 2 α 4 . Aggregating β = ( β 0 , β 1 , β 2 , β 3 ) , the HR-MSM given by

is correctly specified under this data generating process, and we can evaluate the performance of the proposed estimator relative to these true values.

10

30

100

50

500

-1.0

-0.5

0.0

0.5

1.0

-0.2

0.0

0.2

Relative Bias:

Ł

b

^

- b

GLYPH<246>

ł

b

10

30

100

50

500

HC

HC2

HC3

LS

HC

HC2

HC3

LS

HC

HC2

HC3

LS

0.9

0.9

1.0

0.9

0.9

1.0

Robust Variance Estimator

95% CI Coverage

Excursion Effect

Blip

Dissipation

Dose 0

Dose 1

Dose 2

To show we can conduct valid inference on sequential excursion effects, we estimated the three estimands illustrated in Figure 1: (1) the 'blip' effect of an additional exposure opportunity at the more proximal trial t , while keeping treatment at t -1 fixed at the control condition ( β 2 = E [ Y t ( d (0) t -1 , d (1) t ) -Y t ( d (0) t -1 , d (0) t )] ); (2) 'effect dissipation,' comparing the effect of an exposure opportunity at one versus two trials prior to the measurement of the outcome ( β 2 -β 1 = E [ Y t ( d (0) t -1 , d (1) t ) -Y t ( d (1) t -1 , d (0) t )] ); and (3) the 'dose response' curve of exposure opportunities, where the two possible sequences for a single opportunity are averaged (the sequence ( β 0 , β 0 + 1 2 { β 1 + β 2 } , ∑ 3 j =0 β j ) ). This setup also illustrates how HR-MSMs are easily specified such that sequential excursion effects can be calculated as linear combinations of the β parameters.

Although the HR-MSM (3) is correctly specified, we still estimated the β coefficients as projection parameters. We proceeded as if we started by defining β as the minimizers in (2), with V t -∆+1 = ∅ (i.e., no effect modifiers), and h ( t, d ∆ ,t ) ≡ 1 (i.e., constant weight function). We applied the IPW point estimator for the HR-MSM parameters, ̂ β , described in Section 3.2. We assessed the coverage of 95% CIs constructed from our large sample variance estimator, and the small sample size-adjusted HC , HC2 , and HC3 variance estimators (using the sandwich package in R [36]). We tested performance with sample size n ∈ { 6 , 10 , 30 , 100 } , and number of trials T ∈ { 10 , 50 , 500 } .

Results Simulation results in Figures 2 and Appendix Figure 6 show that our estimator is unbiased for the target sequential excursion effects. We present 95% CI coverage in Table 1, and HR-MSM coefficient estimate bias and MSE in Appendix Tables 2 and 3. Sandwich estimators can yield small-sample bias [36], but in small n and T settings, the sample size-adjusted HC3 -based CIs achieve 95% coverage. All CIs achieve 95% coverage when n is large, showing that we can conduct valid inference in all settings.

4.2 Application: Optogenetic Study

Data [17] tested whether optogenetically stimulating dopamine (DA) release in the dorsolateral striatum while an animal engaged in a specific 'pose' (e.g., exploring, rearing, grooming) could 'teach' mice to exhibit that movement more frequently. This study was foundational in identifying the role this region plays in learning. To that end, the researchers implanted mice with optogenetics machinery, and filmed them freely-moving in a behavioral chamber. They used a pre-trained hidden Markov model to estimate an animal's pose online in real-time. They first measured the animals' target pose frequency on a baseline session without optogenetics. Then, on a subsequent treatment session, they applied the laser on a random subset of the target pose occurrences. They repeated this experiment for six target poses, in both the optogenetics and control (laser has no effect) groups.

To define 'trials,' the authors spliced the time-series of estimated pose classifications into intervals of consecutive timepoints with the same pose classification. If mice exhibited the target pose on trial t , they were considered 'available' for optogenetic stimulation, I t = 1 , and were 'unavailable' otherwise, I t = 0 . The laser was applied ( A t = 1 ) with the dynamic policy, P ( A t = 1 | I t ) = 0 . 75 I t . Denoting Y 0 t and Y t as a binary indicator that an animal engaged in the target pose on trial t of the baseline and treatment sessions, respectively, the authors estimated treatment effects of the form ψ = ( E [ ¯ Y 1 | G = 1] -E [ ¯ Y 0 | G = 1] ) -( E [ ¯ Y 1 | G = 0] -E [ ¯ Y 0 | G = 0] ) where ¯ Y 0 = ∑ T 0 t =1 Y 0 t , ¯ Y 1 = ∑ T t =1 Y t , and T, T 0 ∈ N are the trial numbers in treatment and baseline sessions, respectively. 4 There were n 1 = 28 and n 0 = 12 animals in the optogenetics and control groups, respectively. T ranged across animals/sessions from 1207-4876, with a mean of 3612 and IQR = [3341 , 3940] . The authors reported a (pooled across target poses) positive optogenetics treatment effect estimate akin to ̂ ψ , suggesting DA stimulation causes an increase in target pose frequency.

We argue this analysis procedure leaves many scientific questions untested. Conceptualizing optogenetics like a 'study drug,' we question whether stimulation immediately 'taught' the animal the target pose, or whether the treatment effect on learning had a lagged onset. Similarly, did the effect of a single stimulation persist or dissipate across trials? Did more treatments lead to more learning monotonically, or is there an antagonistic effect or non-monotonic dose-response curve in learning?

Application Methods Weapplied our framework to provide a nuanced trial-by-trial characterization of the causal effects of DA stimulation, and formally answer the questions above. Specifically, we tested the causal effect of specific sequences of deterministic dynamic policies, d ∆ ,t (occurring on trials t ∈ { t -∆+1 ,. . . , t } ), on the mean counterfactual E [ Y t ( d ∆ ,t )] . We defined the outcome Y t as an indicator that the mouse exhibited the target pose on trial t +2 , the next trial on which mice could exhibit the target pose if they were available for stimulation on trial t . We fit a set of HR-MSMs illustrating that we can reliably estimate the types of excursion effects in Figure 1. We describe the models we fit below, and relegate code, data and pre-processing details to Appendix Section F.

Results Our first question was whether standard methods reveal significant treatment effects when assessed with the estimands commonly tested in optogenetics studies. We applied a GEE with mean model, log ( E [ ¯ Y s | G = g, S = s ] ) = γ 0 + γ 1 g + γ 2 s + γ 3 g × s , where S ∈ { 0 , 1 } indicates baseline and optogenetics sessions, respectively. The estimate ̂ γ 3 , shown in Figure 3F, thus provides a treatment effect estimate for the observed stochastic dynamic policy in [17]. We adopted a Poisson working model, since [17] analyzed ¯ Y 1 , ¯ Y 0 ∈ N . We tested these (macro longitudinal) effects for each pose individually, rather than pooling over them, as in [17]. The model yielded no significant effects for any individual pose. We show boxplots in Appendix Figure 10 of the subject-level summary ¯ Y 1 -¯ Y 0 that is compared across groups in this model. Outcome levels are similar across groups for most poses, further highlighting how standard outcome summaries can obscure effects.

To assess an analogous 'local' treatment effect using our method, we tested the impact of a single stimulation opportunity. We further evaluated whether the effect had a lagged onset and/or dissipated

1

2

3 1

2

3 1

2

3 1

2

3 1

2

3 1

2

3

-0.10

-0.05

0.00

0.05

Trials Prior

Group Effect Modification:

{

b

r }

r

=

5

7

A

-1.0

0.0

1.0

2.0

1

2

3

4

5

6

Pose

Conditional Treatment Effect:

a

3

B

-0.8

-0.4

0.0

0.4

1

2

3

4

5

6

Pose

Opto Group (No Stim) Effect:

b

4

C

-0.2

0.0

0.2

0.4

1

2

3

4

5

6

Pose

Macro Longitudinal Effect:

g

3

D

Pose

1

2

3

4

5

6

across trials. We included group, G , as an effect modifier, to test whether the causal effect of this treatment opportunity was larger in one of the groups. Setting ∆ = 3 , and restricting the regimes of interest to those with at most one treatment opportunity 'dose', d 3 ,t ∈ { ( d t -2 , d t -1 , d t ) :

Thus, ̂ β r with r ∈ [3] is an estimate of the log odds ratio comparing the mean counterfactual of Y t under a treatment sequence with a single dose (on r = 1 , 2 , or 3 trials prior) vs. a treatment sequence with zero dose. This permits assessment of effect dissipation or persistence. Figure 1B illustrates the analogous effect under a static regime. The interaction terms, ̂ β r with r ∈ { 5 , 6 , 7 } , quantify how these causal effects of a recent treatment opportunity differ between the two groups.

The results from our model (4) reveal that stimulation opportunities in the treatment group tend to reduce the odds of the outcome, compared to the control group. As shown in Figure 3C, these effects are significantly negative for at least one lag level in five out of six target poses. In personal communications, the authors of [17] stated that this result appeared consistent with their finding that animal exploration increased right after stimulation (quantified as higher pose entropy). Figure 3D shows the main effect of group under a treatment sequence of dose zero. In essence, this provides an estimate of the 'long-term' effect of DA stimulation: ̂ β 4 is the log odds ratio of treatment group under a regime of dose zero (i.e., a 'no recent stimulation' policy). We fit comparable models for sequences as long as ∆ = 7 and found results were similar across ∆ values.

Next, we fit the analogous model for the availability-conditional estimand [5] to determine whether current excursion effect methods (i.e., those confined to ∆ = 1 policies) identify the same treatment effects: logit ( E [ Y t ( a t ) | I t = 1 , G = g ]) = α 0 + α 1 a t + α 2 g + α 3 g × a t . Figure 3C shows that the effect estimates, ̂ α 3 , are significant in only one pose. These results highlight how our approach can uncover a greater number of significant effects.

Finally, in Appendix Section E, we include analyses showing that the laser exhibits a dose-response curve in both groups: more treatment opportunities (on some poses) in the last ∆ = 5 trials causes the animal to exhibit the target pose more often. Additionally, there is significant effect modification by baseline responding: the laser has a larger effect in animals who exhibited high baseline pose frequency. Together, these results show we can reliably estimate sequential excursion effects.

5 Discussion

We propose the first, to the best of our knowledge, formal causal inference framework for closed-loop optogenetics behavioral studies. We introduce a nonparametric excursion effect framework, an associated IPW estimator (with valid CIs), with a scalable implementation, and proved its consistency

and asymptotic normality under mild assumptions. Methodologically, our proposed sequential excursion effects represent an expansion of the conditional estimands proposed in [5] to longitudinal policies ( ∆ ≥ 1 ), in the presence of positivity violations. Our methods also directly apply to 'open-loop' (static policies) designs, as they arise as a special case when I t = 1 for all t ∈ [ T ] .

HR-MSMs are powerful and useful models, but have their limitations. As has been discussed in the causal inference literature, these estimands marginalize over all treatments for trials t ∈ [ t -∆] , and thus depend on the protocol used in the design [31, 6]. Moreover, while contrasts of our estimands are null under the sharp null of no causal effect of treatment (e.g., optogenetic laser stimulation), effects should generally still be interpreted in terms of treatment opportunities . Finally, while our implementation is computationally efficient, we anticipate computational challenges for very large ∆ .

The model m and the number of intervention timepoints ∆ represent key choices for practitioners. Our inferential results (i.e., Theorem 3.5) are valid for a large class of working mean models m , and notably do not rely on any distributional assumptions. The 'Donsker' requirement (condition (ii) in Theorem 3.5) is satisfied outright by generalized linear models such as those we use in our application [35], as well as some formulations of random forests [34] and kernel estimators [2]. In future work, we will study how inference can be obtained for more flexible models that do not satisfy the Donsker assumption. Likewise, the value of ∆ plays a significant role as it determines the nature of the effects being estimated, and should be chosen on the basis of subject matter expertise. That said, we found that 2-3 intervention timepoints are often sufficient to capture a rich set of sequential excursion effects, and in our application that results were relatively stable across a range of ∆ values.

The application highlights the drawbacks of standard optogenetics analysis methods. Our finding that macro longitudinal estimates of individual target poses show almost no effect between groups highlights how 'treatment-confounder' feedback can obscure strong treatment effects in closed-loop designs, even when inspecting simple averages of observed outcomes. Our methods account for this by careful causal adjustment with IPW. In personal communications with the authors of [17], they agreed with our findings and remarked at how these methods reveal a collection of causal effects that are difficult to uncover without sophisticated causal inference methods.

Our analyses reveal immediate negative effects (detectable on the next trial) and positive slower effects of DA stimulation (i.e., in treatment relative to control animals). We also find the control group exhibits positive, off-target effects of the laser. Together the opposing signs of these 'fast'/'slow' and on/off-target causal effects may further dilute the magnitude of macro longitudinal effects that summarize the outcome across many trials (e.g., total pose counts). Finally, by enabling estimation of sequential excursion effects (i.e., ∆ > 1 ), we can reveal effect profiles (e.g., dose-response curves) not possible with availability-conditional estimands whose definition is confined to ∆ = 1 regimes. As we observed, the optogenetics group sometimes exhibits an excursion effect not present in the control group. Thus, by combining different sequential excursion effects, analysts can, for example, disentangle laser on-target from off-target effects. When off-target effects are not a major concern, our framework enables estimation of causal effects without having to collect data in a control group, thereby potentially reducing the number of animals required in a study.

Although we focus on optogenetics here, our proposed methods are relevant for a wide range of mobile health, neuroscience and psychology experiments for which the 'local/micro' longitudinal structure is of scientific interest. Indeed, 'closed-loop' designs are common in many behavioral studies in human neuroimaging and cognitive sciences (e.g., when stimuli are conditionally randomized). We hope our methods constitute a useful methodological contribution to the causal inference literature, and will help applied researchers exploit the rich information contained in their experiments.

References

D. L. Barack, E. K. Miller, C. I. Moore, A. M. Packer, L. Pessoa, L. N. Ross, and N. C. Rust. A call for more clarity around causality in neuroscience. Trends in neurosciences , 45(9):654-655, 2022.

E. Beutner and H. Zähle. Donsker results for the empirical process indexed by functions of locally bounded variation and applications to the smoothed empirical process. Bernoulli , 29(1):205-228, 2023.

I. Bica, A. M. Alaa, J. Jordon, and M. van der Schaar. Estimating counterfactual treatment outcomes over time through adversarially balanced representations. arXiv preprint arXiv:2002.04083 , 2020.

R. Biswas and E. Shlizerman. Statistical perspective on functional and causal neural connectomics: A comparative study. Frontiers in Systems Neuroscience , 16:817962, 2022.

A. Boruvka, D. Almirall, K. Witkiewitz, and S. A. Murphy. Assessing time-varying causal effect moderation in mobile health. Journal of the American Statistical Association , 113(523):11121121, 2018.

F. R. Guo, T. S. Richardson, and J. M. Robins. Discussion of 'Estimating time-varying causal excursion effects in mobile health with binary outcomes'. Biometrika , 108(3):541-550, 08 2021.

M. Hernan and J. Robins. Causal Inference: What If . Chapman & Hall/CRC Monographs on Statistics & Applied Probab. CRC Press, 2023.

P. J. Huber. Robust estimation of a location parameter. The Annals of Mathematical Statistics , pages 73-101, 1964.

P. J. Huber et al. The behavior of maximum likelihood estimates under nonstandard conditions. In Proceedings of the fifth Berkeley symposium on mathematical statistics and probability , volume 1, pages 221-233. Berkeley, CA: University of California Press, 1967.

Z. Jiang, S. Chen, and P. Ding. An instrumental variable method for point processes: generalized wald estimation based on deconvolution. Biometrika , page asad005, 2023.

Karline Soetaert. rootSolve: Nonlinear root finding, equilibrium and steady-state analysis of ordinary differential equations , 2009. R package 1.6.

E. Kennedy, S. Balakrishnan, and L. Wasserman. Semiparametric counterfactual density estimation. Biometrika , page asad017, 2023.

E. H. Kennedy, S. Lorch, D. S. Small, et al. Robust causal inference with continuous instruments using the local instrumental variable curve. Journal of the Royal Statistical Society Series B , 81(1):121-143, 2019.

M. E. Lepperød, T. Stöber, T. Hafting, M. Fyhn, and K. P. Kording. Inferring causal connectivity from pairwise recordings and optogenetics. PLoS Computational Biology , 19(11):e1011574, 2023.

B. Lim. Forecasting treatment responses over time using recurrent marginal structural networks. Advances in Neural Information Processing Systems , 31, 2018.

I. E. Marinescu, P. N. Lawlor, and K. P. Kording. Quasi-experimental causality in neuroscience and behavioural research. Nature Human Behaviour , 2(12):891-898, 2018.

J. E. Markowitz, W. F. Gillis, M. Jay, J. Wood, R. W. Harris, R. Cieszkowski, R. Scott, D. Brann, D. Koveal, T. Kula, C. Weinreb, M. A. M. Osman, S. R. Pinto, N. Uchida, S. W. Linderman, B. L. Sabatini, and S. R. Datta. Spontaneous behaviour is structured by reinforcement without explicit reward. Nature , 614(7946):108-117, 2023.

V. Melnychuk, D. Frauen, and S. Feuerriegel. Causal transformer for estimating counterfactual outcomes. In ICML , pages 15293-15329. PMLR, 2022.

K. Moore, R. Neugebauer, F. Lurmann, J. Hall, V. Brajer, S. Alcorn, and I. Tager. Ambient ozone concentrations cause increased hospitalizations for asthma in children: An 18-year study in southern california. Environmental Health Perspectives , 116(8):1063-1070, 2008.

R. Neugebauer and M. van der Laan. Nonparametric causal effects based on marginal structural models. Journal of Statistical Planning and Inference , 137(2):419-434, 2007.

R. Neugebauer, M. J. van der Laan, M. M. Joffe, and I. B. Tager. Causal inference in longitudinal studies with history-restricted marginal structural models. Electronic journal of statistics , 1:119, 2007.

W. K. Newey. Uniform convergence in probability and stochastic equicontinuity. Econometrica: Journal of the Econometric Society , pages 1161-1167, 1991.

M. L. Petersen, S. G. Deeks, J. N. Martin, and M. J. van der Laan. History-adjusted Marginal Structural Models for Estimating Time-varying Effect Modification. American Journal of Epidemiology , 166(9):985-993, 09 2007.

D. Pollard. Convergence of stochastic processes . Springer Science & Business Media, 2012.

D. Pospisil, M. Aragon, and J. Pillow. From connectome to effectome: learning the causal interaction map of the fly brain. bioRxiv , 2023.

T. Qian, H. Yoo, P. Klasnja, D. Almirall, and S. A. Murphy. Estimating time-varying causal excursion effects in mobile health with binary outcomes. Biometrika , 108(3):507-527, 09 2020.

J. Robins. A new approach to causal inference in mortality studies with a sustained exposure period-application to control of the healthy worker survivor effect. Mathematical modelling , 7(9-12):1393-1512, 1986.

J. M. Robins. Marginal structural models versus structural nested models as tools for causal inference. In Statistical models in epidemiology, the environment, and clinical trials , pages 95-133. Springer, 2000.

J. M. Robins, S. Greenland, and F.-C. Hu. Estimation of the causal effect of a time-varying exposure on the marginal mean of a repeated binary outcome. Journal of the American Statistical Association , 94(447):687-700, 1999.

J. M. Robins, M. A. Hernan, and B. Brumback. Marginal structural models and causal inference in epidemiology. Epidemiology , pages 550-560, 2000.

J. M. Robins, M. A. Hernán, and A. Rotnitzky. Invited Commentary: Effect Modification by Time-varying Covariates. American Journal of Epidemiology , 166(9):994-1002, 09 2007.

M. Rosenblum and M. J. van der Laan. Targeted maximum likelihood estimation of the parameter of a marginal structural model. The international journal of biostatistics , 6(2), 2010.

L. N. Ross and D. S. Bassett. Causation in neuroscience: keeping mechanism meaningful. Nature Reviews Neuroscience , pages 1-10, 2024.

E. Scornet. On the asymptotics of random forests. Journal of Multivariate Analysis , 146:72-83, 2016.

A. W. Van der Vaart. Asymptotic statistics , volume 3. Cambridge university press, 2000.

A. Zeileis, S. Köll, and N. Graham. Various versatile variances: An object-oriented implementation of clustered covariances in r. Journal of Statistical Software , 95(1):1-36, 2020.

A Micro Longitudinal Effects

In Appendix Figure 4, we illustrate additional micro longitudinal effects that can be probed with our sequential excursion effect framework. This figure has the same layout as Figure 1A-C.

B Illustration of Treatment-Confounder Feedback

We provide in this section a synthetic example in which two independent groups exhibit identical mean outcome patterns over time, but where the treatment (e.g., turning on laser in the brain) has a substantial effect in one group but not the other. As our construction will demonstrate, this phenomenon manifests due to treatment-confounder feedback leading to effects canceling out. In a similar fashion, one can similarly construct scenarios where effects are exaggerated.

Suppose G ∈ { 0 , 1 } represents an experimentally manipulable marker (e.g., animals expressing opsin in the brain), and counterfactual outcomes under G = g are denoted Y g t . We will suppose that potential outcomes generated in the active setting ( G = 1 ) are given by

and potential outcomes in the control condition ( G = 0 ) are given by

i.e., the treatment (e.g., laser) has an effect when G = 1 , but not when G = 0 .

Suppose further that a behavior X t is measured at all time points t , and determines whether or not treatment will be administered with positive probability. Like the outcomes, this behavior will be affected by the laser only when G = 1 :

and X g 0 ∼ Bernoulli( 1 2 ) at baseline.

Now we consider a study where animals are randomly assigned at baseline to either G = 1 or G = 0 . At each time point t , the behavior X t is measured, and treatment is then drawn according to A t ∼ Bernoulli(0 . 8 X t ) . By induction, E ( A t | G = 1) = 0 . 4 and E ( X t | G = g ) = 0 . 7 -0 . 2 g , for all t . It follows that

Thus, the 'macro'/'global' between-group mean difference trajectory is given by

which will be null if γ 2 + γ 4 = 0 . 5( γ 1 + γ 3 ) . Notice that this cancellation is possible even if the immediate effect of treatment on the outcome is quite strong, say if γ 2 and γ 4 are large and positive. The cancellation is made possible through the opposing effects of treatment on the intermediate behavior and the outcome: when G = 1 , A t -1 negatively impacts X t but positively impacts Y t . More generally, these X t -A t feedback loops can lead to dilution or exaggeration of the actual effect of treatments when only analyzing observed mean outcomes.

We note that in the data generating scenario described in this appendix, the proposed dynamic treatment regime HR-MSM methodology would pick out non-null effects of treatment within the active group ( G = 1 ), and show differing effects between groups, even if the condition above held such that observed mean outcomes were identical. This example thus serves to illustrate both the challenges with closed-loop designs and, despite these challenges, the ability of the proposed methodology to elucidate effects.

Example Analysis on Synthetic Data To illustrate the above, we provide an example on a simulated dataset, taking n = 100 , T = 500 , γ 1 = γ 3 = 1 , γ 2 = γ 4 = 0 . 5 .

Antagonism

𝔼[𝑌 𝑡(𝒜 1

) - 𝑌 𝑡(𝒂

0

)]

𝔼[𝑌 𝑡(𝒜 2

) - 𝑌 𝑡(𝒂

0

)]

𝔼[𝑌 𝑡(𝒂

7

) - 𝑌 𝑡(𝒂

0

)]

Not

Achieved

𝔼[𝑌 𝑡(𝒜 1

) - 𝑌 𝑡(𝒂

0

)]

𝔼[𝑌 𝑡(𝒜 2

) - 𝑌 𝑡(𝒂

0

)]

𝔼[𝑌 𝑡(𝒂

7

) - 𝑌 𝑡(𝒂

0

)]

Opto (t-2)

Opto (t-1)

Opto (t)

0

Trial Number

Mean Counterfactual:

E

é

ë

Yt

(

a

(

7

)

) ù

û

𝑌 𝑡

measured

0

0

0

𝑌 𝑡

1

0

0

𝑌 𝑡

0

1

0

𝑌 𝑡

0

0

1

𝑌 𝑡

1

0

1

𝑌 𝑡

1

1

0

𝑌 𝑡

0

1

1

𝑌 𝑡

1

1

1

𝑌 𝑡

Δ = 3

Policy 0:

𝒂 (0)

Policy 1:

𝒂 (1)

Policy 2:

𝒂

(2)

Policy 3:

𝒂

(3)

Policy 4:

𝒂 (4)

Policy 5:

𝒂

(5)

Policy 6:

𝒂

(6)

Policy 7:

𝒂

(7)

𝒜 (1)

𝒜 (2)

𝔼[𝑌 𝑡(𝒜 1

) - 𝑌 𝑡(𝒂

0

)]

𝔼[𝑌 𝑡(𝒜 2

) - 𝑌 𝑡(𝒂

0

)]

𝔼[𝑌 𝑡(𝒂

7

) - 𝑌 𝑡(𝒂

0

)]

Antagonism Effect

𝔼[𝑌 𝑡(𝒜 1

) - 𝑌 𝑡(𝒂

0

)]

Opto (t-2)

Opto (t-1)

Opto (t)

0

Trial Number

Mean Counterfactual:

E

é

ë

Yt

(

a

(

7

)

) ù

û

𝑌 𝑡

measured

0

0

0

𝑌 𝑡

1

0

0

𝑌 𝑡

0

1

0

𝑌 𝑡

0

0

1

𝑌 𝑡

1

0

1

𝑌 𝑡

1

1

0

𝑌 𝑡

0

1

1

𝑌 𝑡

1

1

1

𝑌 𝑡

Δ = 3

Policy 0:

𝒂 (0)

Policy 1:

𝒂 (1)

Policy 2:

𝒂

(2)

Policy 3:

𝒂

(3)

Policy 4:

𝒂 (4)

Policy 5:

𝒂

(5)

Policy 6:

𝒂

(6)

Policy 7:

𝒂

(7)

𝒜 (1)

𝒜 (2)

𝔼[𝑌 𝑡(𝒜 2

) - 𝑌 𝑡(𝒂

0

)]

𝔼[𝑌 𝑡(𝒂

7

) - 𝑌 𝑡(𝒂

0

)]

C. Effect Additivity

D. Effect Lag

E. Time-Dependent Effect Synergism

0

1

𝑌

#

1

1

𝑌

#

Policy 1:

𝒂

($)

Policy 2:

𝒂

(%)

𝔼[𝑌

#

(𝒂

*

) - 𝑌

#

(𝒂

%

)]

Ceiling Effect

Additive Effect

𝔼[𝑌

#

(𝒂

*

) - 𝑌

#

(𝒂

%

)]

Δ = 2

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t)

0

Trial Number

Mean Counterfactual:

E





Yt

(

a

(

2

)

)





𝑌

!

measured

1

0

0

0

𝑌

#

0

1

0

0

𝑌

#

0

0

1

0

𝑌

#

0

0

0

1

𝑌

#

Policy 1:

𝒂

($)

Policy 2:

𝒂

(%)

Policy 3:

𝒂

(&)

Policy 3:

𝒂

(')

𝔼[𝑌

#

(𝒂

*

) - 𝑌

#

(𝒂

%

)]

𝔼[𝑌

#

(𝒂

+

) - 𝑌

#

(𝒂

%

)]

𝔼[𝑌

#

(𝒂

1

) - 𝑌

#

(𝒂

%

)]

Immediate Onset

Lagged Effect

𝔼[𝑌

#

(𝒂

*

) - 𝑌

#

(𝒂

%

)]

𝔼[𝑌

#

(𝒂

+

) - 𝑌

#

(𝒂

%

)]

𝔼[𝑌

#

(𝒂

1

) - 𝑌

#

(𝒂

%

)]

Δ = 4

Opto (t

-

3)

Opto (t

-

3)

Opto (t

-

3)

Opto (t

-

3)

Opto (t

-

3)

Opto (t

-

3)

Opto (t

-

3)

Opto (t

-

3)

Opto (t

-

3)

Opto (t

-

3)

Opto (t

-

3)

Opto (t

-

3)

Opto (t

-

3)

Opto (t

-

3)

Opto (t

-

3)

Opto (t

-

3)

Opto (t

-

3)

Opto (t

-

3)

Opto (t

-

3)

Opto (t

-

3)

0

Trial Number

Mean Counterfactual:

E





Yt

(

a

(

1

)

)





𝑌

!

measured

1

0

1

𝑌

#

1

1

0

𝑌

#

0

1

1

𝑌

#

Policy 1:

𝒂

($)

𝔼 𝑌

#

(𝒜 -𝑌

#

(𝒂

%

)]

Time-Dependent

Effect Synergism

Time-Independent

Effect Synergism

𝔼[𝑌

#

(𝒜) - 𝑌

#

(𝒂

%

)]

𝒜

Δ = 3

Policy 2:

𝒂

(%)

Policy 3:

𝒂

(&)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

Opto (t

-

1)

0

Trial Number

Mean Counterfactual:

E





Yt

(

a

(

2

)

)





Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t

-

2)

Opto (t)

0

Trial Number

Mean Counterfactual:

E





Yt

(

a

(

1

)

)





𝑌

!

measured

𝑌

!

measured

None

Low

Med

High

1

1

1

2

2

0

100

200

300

400

500

Timepoint

Mean Outcome:

1

n

GLYPH<229>

i

Y

i

,

t

G

Outcomes Averaged across Subjects

A

1

1

1

1

2

Control

Treatment

Group

Outcome Pooled Across Time:

1

T

GLYPH<229>

t

Y

i

,

t

G

Outcomes Averaged across Timepoints

B

Group

Control

Treatment

0.0

0.1

0.2

0.3

Seq. Excursion Effect Type

Seq. Excursion Effect Value:

b

j

Control Group (G=0)

C

0.0

0.1

0.2

0.3

Seq. Excursion Effect Type

Seq. Excursion Effect Value:

b

j

Treatment Group (G=1)

D

Effect

Blip

Dissipation

Dose 1 vs. Dose 0

Dose 2 vs. Dose 0

C Additional Details for Section 3

C.1 Interpretation of Sequential Excursion Effects

The interpretation of the mean counterfactual quantity E [ Y t ( d ∆ ,t ) | V t -∆+1 ] is somewhat subtle, and warrants further discussion. When ∆ = 1 , we can express a contrast of these estimands in terms of the effect of exposure in a certain subgroup:

That is, the mean contrast in counterfactual outcomes for d (1) t versus d (0) t is the mean effect of A t = 1 versus A t = 0 among those with I t = 1 -the availability-conditional estimand proposed by [5]-diluted by the probability of availability. Even in this case with ∆ = 1 , it may not always be clear for whom the availability-conditional estimand generalizes to, i.e., the group I t = 1 may be highly idiosyncratic and not of particular interest. On the other hand, the parameters we are proposing summarize the effects of plausible interventions on the whole population, acknowledging that for some individuals active treatment (i.e., A t = 1 ) is not possible.

The comparison just described is somewhat akin to the duality in clinical trials of per-protocol (or complier-specific) effects, and intention-to-treat effects. Thus, in practice when ∆ = 1 , we would recommend assessing both the availability-conditional estimand, as in [5], as well as our proposed population-level effect. When ∆ > 1 , it is not clear whether an analogous availability-conditional estimand exists; our approach is viable for arbitrary ∆ . In general, our estimands have the populationlevel (possibly conditional on effect modifiers) interpretation of summarizing how outcomes would be affected if the experimental protocol were changed to match d ∆ ,t for the ∆ trials leading up to the outcome. Finally, we note that, as for all excursion effects or history-restricted marginal structural models, the estimands under study are dependent on the treatment protocol [6].

C.2 Discussion of Causal Assumptions

Consistency (Assumption 3.1) states that for any of the regimes d ∆ ,t under study, the counterfactual outcome Y t ( d ∆ ,t ) equals the observed outcome Y t when observed treatment values correspond to assignment under d ∆ ,t . Positivity (Assumption 3.2) states that treatment probabilities are bounded away from zero-this is required for the asymptotic analysis of the proposed estimator later on. Note that, by definition of the availability indicator I t , and the regimes D ∗ t in Section 3.1, we are allowing P [ A t = 1 | H t ] = 0 in some cases (i.e., when I t = 0 ), but Assumption 3.2 rules out P [ A t = 1 | H t ] = 1 . This positivity assumption holds in many open- and closed-loop optogentic studies. In practice, in such experiments, one can ensure that Assumption 3.2 holds by design when choosing the treatment assignment probabilities. Finally, Assumption 3.3 says that treatments are randomly assigned at each time t , based on all previously measured data H t . In the sequential optogenetic experiments that motiviate this work, this assumption would hold by design. In observational studies, one will have to assess the plausibility of Assumption 3.3 (as well as Assumptions 3.1 and 3.2) on a case-by-case basis, ideally based on subject matter knowledge; it may be harder to justify Assumption 3.3 due to the possible presence of unmeasured confounders.

C.3 Conditions of Theorem 3.5

Conditions (i) through (iv) are standard conditions for asymptotic normality of M-estimators [8, 9]. For condition (ii), we expect the working model m to be differentiable in β for most common models. Moreover, for standard generalized linear models, m will be appropriately Donsker-see [35] for formal definitions. Condition (iii) is satisfied under mild conditions, e.g., if the weight functions h , the model m and its derivative M , and the outcomes Y t are uniformly bounded, and no haphazard degeneracy in B exists that could cause singularity. Lastly, condition (iv) is also quite weak, only requiring convergence of ̂ β at an arbitrarily slow rate, and would hold under some stochastic equicontinuity conditions [22, 24].

C.4 Proofs of results in Section 3.2

Proof of Proposition 3.4. This result follows the usual g -formula identification argument [27]: defining A ∆ ,t = ( A t -∆+1 , . . . , A t ) , d ∆ ,t ( H t )) = ( d t +∆+1 ( H t +∆+1 ) , . . . , d t ( H t )) ,

E

(

Y

t

(

d

∆

,t

)

|

V

t

-

∆+1

)

= E ( E ( Y t ( d ∆ ,t )) | H t -∆+1 ) | V t -∆+1 )

. . .

where we repeatedly invoke iterated expectations and Assumption 3.3 (justified by Assumption 3.2), then use Assumption 3.1 in the last equality. We can then rewrite this formula in an equivalent IPW form:

where the last equality is achieved again by iterated expectations. The second statement in Proposition 3.4 is obtained by differentiating (2) with respect to β , setting this to zero, then invoking the first statement of Proposition 3.4 (which we have just proved).

Proof of Theorem 3.5. This is an immediate application of Theorem 5.31 in [35].

D Additional Simulation Details and Results

D.1 HR-MSM for Simulation Data-Generating Mechanism

In this section, we derive the form of the HR-MSM in (3), and show that it is implied by the data-generating mechanism of the simulation study. First, observe that for t ≥ 2 ,

for some exogenous ϵ t ∼ N (0 , σ 2 t ) , where X t ( d t -1 ) is the potential X t value under the intervention setting A t -1 to d t -1 ( X t -1 ) . Note that d t -1 ( X t -1 ) = J t -1 X t -1 , and by our structural equations, X t ( d t -1 ) ∼ Bernoulli(0 . 4 + 0 . 4 d t -1 ( X t -1 )) , so that E ( X t ( d t -1 )) = 0 . 4 + 0 . 2 J t -1 , recalling that E ( X t -1 ) = 0 . 5 . Finally, d t ( X t ( d t -1 )) = J t · Bernoulli(0 . 4 + 0 . 4 d t -1 ( X t -1 )) , which gives E ( d t ( X t ( d t -1 ))) = { 0 . 4 + 0 . 2 J t -1 } J t . Putting everything together, we obtain

as claimed.

D.2 Further Simulation Results

In Appendix Figure 6 we present the same results as in Figure 2 but with more sample sizes, n and trials, T . We also present these same simulation results in terms of the β coefficients of HR-MSM 3. In the main text we presented results in terms of sequential excursion effect parameters, which are linear combinations of these HR-MSM regression coefficients.

E Additional Application Results

E.1 Dose-Response Excursion Effects

History-Restricted MSM We fit an HR-MSM within the treatment group ( G = 1 ) to estimate the causal effect of 'dose,' the number of treatment opportunities in the previous ∆ = 5 trials:

6

10

30

100

10

50

500

-2

0

2

-2

-1

0

1

-0.5

-0.2

0.0

0.2

0.5

Relative Bias:

Ł

b

^

- b

GLYPH<246>

ł

b

6

10

30

100

10

50

500

HC

HC2

HC3

LS

HC

HC2

HC3

LS

HC

HC2

HC3

LS

HC

HC2

HC3

LS

0.9

0.9

1.0

1.0

0.9

0.9

1.0

1.0

0.9

0.9

1.0

1.0

Robust Variance Estimator

95% CI Coverage

Excursion Effect

Blip

Dissipation

Dose 0

Dose 1

Dose 2

4

6

8

10

30

100

10

50

500

0

1

2

3

0

1

2

3

0

1

2

3

0

1

2

3

0

1

2

3

0

1

2

3

-5

-2

0

2

-2

-1

0

1

-0.6

-0.3

0.0

0.3

0.6

Coefficient Index

Ł

b

^

- b

GLYPH<246>

ł

b

4

6

8

10

30

100

10

50

500

HC

HC2

HC3

LS

HC

HC2

HC3

LS

HC

HC2

HC3

LS

HC

HC2

HC3

LS

HC

HC2

HC3

LS

HC

HC2

HC3

LS

0.8

0.8

0.9

0.9

1.0

0.8

0.8

0.9

0.9

1.0

0.8

0.8

0.9

0.9

1.0

Robust Variance Estimator

95% CI Coverage

factor(coef_idx)

0

1

2

3

where σ j ( d j ) = 1 ( d j = d (1) j ) . The coefficient ̂ β r is an estimate of the log odds ratio comparing the mean counterfactual of Y t for a treatment sequence of dose r ∈ [3] compared to a sequence of dose zero (see Figure 1C for an illustration of the static regime analogue). A dose of three is the maximum feasible dose for ∆ = 5 since the same pose cannot occur on two consecutive trials.

The dose-response effect estimates, { ̂ β r } 3 r =1 , from HR-MSM (5) are shown in Figure 8A. This illustrates the capacity of our approach to identify a clear dose-response effect: within the past ∆ = 5 trials, each additional opportunity for a stimulation causes an increase in the odds of engaging in the

target pose on the next trial. The effects are significant for at least one dose value in all but two target poses. Interestingly, the effect is also significantly negative for one target pose.

Conditional Excursion Effect We next estimate an availability-conditional estimand [5], to determine if existing excursion effect methods have the capacity to reveal the effects identified with our method. We estimate this in the MSM

Figure 8B shows the availability-conditional treatment effect estimates, ̂ α 1 estimated in model (6). It identifies no significant effects for any target pose. The conditional estimand, often referred to as a 'blip effect' (see Figure 1A for an illustration) is only defined for the effect of applying the laser on the most recent trial (i.e., a dose of 1), and thus cannot estimate dose-response profiles. In contrast, our approach can test sequential excursion effects (i.e., for policies with ∆ > 1 ), enabling the estimation of a dose-response profile that reveals treatment effects here. Importantly, the effect estimates ̂ β 1 and ̂ α 1 have different interpretations because ̂ β 1 reflects a causal effect of a single treatment opportunity for any of the last ∆ = 5 trials, and ̂ β 1 is not interpreted as conditional on availability.

1

2

3

1

2

3

1

2

3

1

2

3

1

2

3

1

2

3

-0.05

0.00

0.05

Dose

Dose-Response Treatment Effects:

{

b

r }

r

=

3

1

A

-0.5

0.0

0.5

1.0

1

2

3

4

5

6

Pose

Conditional Treatment Effect:

a

1

B

Pose

1

2

3

4

5

6

E.1.1 Effect Modification by Baseline Behavior

Marginal Effect Modification Parameter Next we show the capacity of our method to estimate effect modification in the form of interactions between covariates and functions of deterministic policies (i.e., treatment opportunity dose). We estimated effect modification of total target pose counts on baseline sessions, ¯ Y 0 (defined in Section 4.2), because [17] estimated the treatment effect of the laser by comparing the mean change in outcome levels between treatment and baseline sessions.

Augmenting the HR-MSM in the previous section with effect modifier, ¯ Y 0 , we fit the HR-MSM

Similarly, we fit an analogous model for the conditional estimand,

We centered and scaled the effect modifier to make the coefficients easier to interpret.

Figure 9A-B shows how our method can be used to probe effect-modification. The figures illustrate that our approach estimates that for two target poses, there is a statistically significant effect modification by baseline responding at, at least, one of the doses. Specifically, animals who exhibited higher levels of responding on baseline sessions, exhibited a larger effect of dose. Figure 9C-D shows that the availability-conditional estimator does not identify any significant main effect of stimulation or interactions with baseline responding. These results show how our framework enables one to probe effect modification of sequential excursion effects.

1

2

3

1

2

3

1

2

3

1

2

3

1

2

3

1

2

3

-0.1

0.0

Dose-Response Treatment Effects:

{

b

r }

r

=

3

1

A

0

2

4

1

2

3

4

5

6

Conditional Treatment Effect:

a

1

C

1

2

3

1

2

3

1

2

3

1

2

3

1

2

3

1

2

3

-0.1

0.0

0.1

Dose

{

b

r }

r

=

7

5

B

0

2

4

6

8

1

2

3

4

5

6

Pose

Interaction (Conditional):

a

3

D

Pose

1

2

3

4

5

6

F Data, Pre-processing and Code Availability

We provide code to reproduce all analyses and figures on anonymous GitHub Repo: https: //anonymous.4open.science/r/causal_opto-52CD/README.md . We downloaded the opensource dataset from [17] from https://zenodo.org/records/7274803 . Weused the open-source pre-processing code provided by the authors on Github repo https://github.com/dattalab/ dopamine-reinforces-spontaneous-behavior . We analyzed data from all animals in the online dataset (both ChR2 and Chrimson animals). We constructed trials as described in [17]. That is, we defined trials as consecutive timepoints when the animal was classified to be in a given pose. For our HR-MSM analyses of the treatment (opto) sessions, we classified 'target pose' trials only if they met the criteria of [17], which required that the hidden Markov model predictions had sufficiently high forward algorithm probabilities of the latent states. This indicator was provided in the opto session dataset provided by the authors. The baseline session data did not, however, include this indicator since no optogenetic stimulation was applied. Thus when recreating the 'standard' between-group (macro longitudinal) analyses that compared baseline and opto session data, we did not classify target pose trials based on whether it met this criteria: we classified the pose based on the most likely latent state prediction but did not require the forward algorithm probabilities met the threshold set by the authors (for either baseline or opto sessions to be consistent). We corresponded closely with the authors to ensure we pre-processed the data correctly.

There was a small percentage of trials that the authors described eliminating because they were deemed too short. We did not eliminate these trials because this created inconsistencies in the pattern of trials: it allowed two consecutive trials to be of the same trial type which broke with the pattern in the remainder of the dataset. This was a very small percentage of the dataset. We compared results with and without this criteria and the decision appeared to have negligible effects on analysis results.

Finally, to the best of our understanding, the original authors' hypothesis tests were conducted on further processed version of the data that first calculated the number of target pose occurrences in each 30 second bin of the experiment (period). From our understanding, this was done to provide a smoothed time-series of outcome frequency across the course of the opto sessions. We conducted similar analyses to make sure our pre-processing yielded comparable results, but we did not use these pre-processing steps in our HR-MSM analyses or replication of the 'standard' between-group (macro longitudinal) analyses as it appeared to 'double-count' target pose occurrences that began before the end of one 30 second bin and ended after the start of the subsequent bin.

Finally, as described in [17], the experiment included two 30 minute replicates of both opto and baseline sessions. We constructed trials on each replicate separately (to account for the discontinuity in time between replicates) and then pooled the replicate datasets together to be consistent with the analysis procedures in [17]. We accounted for the longitudinal structure by using sandwich variance estimators in all of our hypothesis tests.

F.1 Replication of Original Author Analysis

̸

Finally, Figure 3D shows that the 'standard' macro longitudinal effects exhibit no significant changes, emphasizing how estimands that marginalize over the stochastic dynamic (closed-loop) policies can obscure effects. We visualize the within-subject differences in target pose counts between treatment (opto) and baseline sessions in Figure 10: ∑ T t =1 Y t -∑ T 0 t =1 Y 0 t , where Y t , Y 0 t ∈ { 0 , 1 } are indicators that the animal exhibited the target pose on trial t of the treatment and baseline sessions, respectively; T, T 0 ∈ Z denote the total number of trials in treatment and baseline sessions, respectively. Because of the trial definition, the total number of trials usually differed within-subject between treatment and baseline sessions (i.e., T = T 0 ), but we found comparable analyses of 1 T ∑ T t =1 Y t and 1 T 0 ∑ T 0 t =1 Y 0 t yielded similar results to analyses of total counts ∑ T t =1 Y t -∑ T 0 t =1 Y 0 t . For that reason, we present results in terms of total outcome counts to be consistent with the analyses presented in [17]. We showed these results to the authors of [17] and they confirmed that these analyses aligned with theirs.

1

2

3

4

5

6

CtrlOpto

CtrlOpto

CtrlOpto

CtrlOpto

CtrlOpto

CtrlOpto

-200

0

200

400

600

Group

Opto - Baseline Session Difference:

GLYPH<229>

t

=

1

T

Y t

- GLYPH<229>

t

=

1

T0

Y

t

0

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The methodological, theoretical, and experimental contributions described in the abstract are each addressed in their own main text sections. We only make claims in the abstract that we carefully justify and explain in the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer:[Yes]

Justification: The Discussion section includes limitations of our proposed method and the interpretation of the method's results. We state the mathematical assumptions clearly in the main text and discuss their justification in the Appendix. We also discuss method scalability in the Methods (Section 3). Our simulation results highlight the limitations of our approach in small sample settings: small sample bias, and the requirement of sample size-adjusted covariance estimators to achieve nominal coverage. We intentionally include very small sample sizes in our simulations to emphasize the trade-offs in these settings even though our data application has datasets with a much larger sample size. Our work focuses on animal data and thus does not include problems of privacy or fairness.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Complete proofs of Proposition 3.4 and Theorem 3.5 are provided in Appendix C.4. All remaining technical details are discussed in the paper, with additional details provided in the appendices when relevant.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide all details needed to reproduce our results in the main text and appendix. We include all code used to produce results, pre-process data and make figures. We provide links to the open-source datasets that we analyze.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We include a data pre-processing and data availability section (data from our application is open-source) in the Appendix section. We also include a link to an anonymous GitHub repository with an implementation of all results presented in the main text and Appendix.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We discuss the optimizer used. Our method has no tuning parameters. No data splits were used anywhere in the paper since our method is focused on statistical inference and hypothesis testing (not on prediction).

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We include 95% confidence intervals (CIs) for our results, e.g., represented by error bars in figures. We verify in simulations the validity of these CIs (i.e., that they achieve the nominal coverage).

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: For the vast majority of applications, our methods are not computationally intensive and can be run quickly with little memory. We briefly discuss but we do not provide details since we ran everything on a standard laptop with no GPUs or parallelization.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The authors have reviewed the NeurIPS Code of Ethics, and have ensured that the paper conforms in all respects with these guidelines.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss that our method has the capacity to reduce the number of animal subjects used in experients.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper is focused on methodology for animal studies, and we do not foresee our approaches being misused or used irresponsibly. The data used are publicly available, coming from a high profile Nature paper.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We properly cite existing packages and datasets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [No]

Justification: We are not releasing any packages, benchmarks, or other assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Data used in the paper did not involve crowdsourcing nor human subjects. Data were publicly available-see Appendix F for details.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Data used in the paper did not involve crowdsourcing nor human subjects. Data were publicly available-see Appendix F for details.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper proposes novel causal inference framework for closed-loop optogenetics behavioral studies, develops causal inference estimation method for sequential excursion effects that capture local causal contrasts within the same treatment group. The proposed method is robust to accommodate both positivity violations and longer treatment sequences. The paper provided a practical computational implementation and analyze the method with theoretical guarantees. More interestingly, it empirically corroborates their studies with real-applications.

STRENGTHS:
The utility and implications of counterfactual causal effect under dynamic treatment policies are well-described by authors. I found it especially interesting that the authors performs empirical validation through applying their method on existing Nature paper and found their computational result is corroborated with the original authors of the existing paper. This shows a thorough post-hoc validation of the proposed method.

WEAKNESSES:
The largest weakness I see here is both experimental results uses linear or generalized linear model for $m$. I see the advantages of well-developed generalized linear model in experimental sciences for good interpretation and statistical guarantees, though would be curious to find out how the authors see the method would work beyond linear models.

QUESTIONS:
- Proposition 3.4 $Z_i$ is introduced but the proposition only uses $Z$ throughout. would be good to specify the relationship between $Z_i$ and $Z$.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a causal inference framework for analyzing close-loop optogenetics experiments. The authors propose using HR-MSMs to estimate the sequential excursion effects (causal effects of specific neural stimulation sequences on behavioral outcomes). Specifically, they develop an IPW estimator with a scalable implementation and provide the theoretical guarantee of consistency and asymptotic normality under mild assumptions. The proposed framework extends the existing approaches to handle longer treatment sequences and addresses their limitations such as positivity violations. Through both simulation studies and an application to a real-world optogenetics study, the authors show that the proposed framework can reveal longitudinal causal effects that are often obscured by standard optogenetics analysis methods due to \"treatment-confounder\" feedback in closed-loop designs.\n\nSTRENGTHS:\n1. This paper proposes the first formal causal inference framework based on HR-MSMs for closed-loop optogenetics behavioral studies, addressing the challenges (e.g., positivity violations and \"treatment-confounder\" feedback) in the standard methods.\n\n2. The authors prove the consistency and asymptotic normality of the proposed IPW estimator, under mild assumptions.\n\n3. The proposed framework is evaluated on both simulated and real optogenetics studies, and demonstrates its effectiveness in analyzing more complex causal effects in closed-loop designs than standard analyses.\n\nWEAKNESSES:\n1. The focus of this paper centers on an application study that refines causal effect estimation within a specialized causal graph, akin to Figure 1E, rather than introducing broad methodological advancements in causal inference. The authors adapt Marginal Structural Models (MSMs) to dynamic treatment regimes and introduce History-Restricted MSMs (HR-MSMs) for estimating sequential excursion effects. This setup closely relates to estimating causal effects in temporal sequences. It is unclear what is the difference/gap between the proposed framework and the existing temporal causal inference methods. I wonder if the authors could discuss this in more detail. For instance, can any temporal causal inference framework estimate sequential excursion effects accurately? What are the challenges in performing causal inference the closed-loop optogenetics behavioral studies compared to more general (spatial-)temporal studies?\n\n2.  I think it would be better if the authors explicitly introduce the problem settings in Section 2 rather than mention it together with relevant literature. The current relevant literature part involves many technical details, which makes it slightly difficult for readers to understand the gap between the proposed method and existing works.\n\n3. The authors should consider adding more baselines in experiments. In this paper, the authors did not use any baselines, and thus it is unclear whether the proposed framework outperforms existing methods.\n\n4. The settings and evaluation metrics of the experiments need to be revised. \n\na. I think one of the advantages of the proposed method is to handle more complex local causal effects or longitudinal effects, whereas the standard methods only estimate macro effects. Intermediate causal effects can be canceled out in some cases, so only estimating macro effects can be biased and inaccurate. However, the authors did not conduct any simulation studies to demonstrate the advantage of the proposed method. \n\nb. The current presentation of experimental results primarily relies on descriptive analysis, and I think the authors should use some quantitative metrics to evaluate the performance.\n\nQUESTIONS:\n1. Please see the questions in the Weaknesses part.\n\n2. I wonder how is $\\Delta$ determined and How would the value of $\\Delta$ affects the sequential excursion effects?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
This work investigate the causal effect estiamtion in the Optogenetics, which is interestimg in causality in science. The paper proposed a nonparametric causal inference framework for analyzing “closed-loop” designs in sequential setting and  extends “excursion effect” methods to enable estimation of causal contrasts for treatment sequences. The writing is good, and the toptic is important for the causality and biologic community.

STRENGTHS:
1. The investigated problem is important.

2. The presentation and writing is good.

WEAKNESSES:
1. Some claims are not clear, which makes the paper unreadable, especially for those who are not familiar on biologic and optogenetics

2. The experimental evaluation is weak.

QUESTIONS:
1.I am kind of confused for the definition and notation of treatment, what's the difference of G and A? I thought they are the same but the description in intro confuses me. optogenetic manipulation is not about the laser and no-laser.

2. The experimental evaluation is weak. Indeed there are only one baseline method, i.e., HC to compare with the proposed estimator, which is is too limited, and it is not convincing that the proposed method is effective.

3.There are many sequential models (e.g., Causal Transformer, CRN, RMSN ,etc) which can handle this kind of time-dependent causal inference task.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13