SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Marcus A. Triplett $^{1,2,\dagger}$
Marta Gajowa<sup>3</sup>
Hillel Adesnik<sup>3</sup>
Liam Paninski $^{1,2}$
$^{1}$ Department of Statistics, Columbia University
$^{2}$ Zuckerman Mind Brain Behavior Institute, Columbia University
$^{3}$ Department of Molecular and Cell Biology, UC Berkeley
marcus.triplett@columbia.edu
Two-photon optogenetics has transformed our ability to probe the structure and function of neural circuits. However, achieving precise optogenetic control of neural ensemble activity has remained fundamentally constrained by the problem of off-target stimulation (OTS): the inadvertent activation of nearby non-target neurons due to imperfect confinement of light onto target neurons. Here we propose a novel computational approach to this problem called Bayesian target optimisation. Our approach uses nonparametric Bayesian inference to model neural responses to optogenetic stimulation, and then optimises the laser powers and optical target locations needed to achieve a desired activity pattern with minimal OTs. We validate our approach in simulations and using data from in vitro experiments, showing that Bayesian target optimisation considerably reduces OTS across all conditions we test. Together, these results establish our ability to overcome OTS, enabling optogenetic stimulation with substantially improved precision.
A key technological goal in neuroscience is to gain precise control over the activity of neurons. Currently, one of the most promising approaches for achieving such control is two-photon optogenetics [1-4]. This technique relies on the use of two-photon excitation to activate opsin molecules expressed in the somatic membrane, allowing individual neurons to be targeted for stimulation. Holographic optogenetics further extends this technique by focusing two-photon excitation into  $\sim 10\mu \mathrm{m}$  disks of light that illuminate all of a neuron's opsin molecules in parallel [5-9]. Replicating this illumination pattern across many neurons at once then grants simultaneous optogenetic control of entire neural ensembles [10-12]. However, the spatial precision of holographic optogenetics has remained fundamentally limited by the problem of off-target stimulation (OTS): if multiple opsin-expressing neurons lie in close proximity (in the most extreme case, if their membranes are in physical contact), two-photon excitation will frequently activate opsin molecules in each neuron when attempting to stimulate only one. This can inadvertently elicit spikes in non-target neurons, compromising the precision and specificity of the optogenetic manipulation [13].
Previous attempts at overcoming OTS have been either optical or molecular in nature (e.g. by using temporal focusing [2, 6, 7, 14] or improving the soma-targeting of the opsin [15, 16, 10]), but have not yet achieved "true" single-cell precision under many experimental conditions. Frequently, experimenters will simply express opsin sparsely in a population to avoid OTS, as this reduces the number of nearby opsin-expressing neurons that could be mistakenly activated [17]. However, sparse opsin expression reduces the number of neurons that can potentially be probed during an
experiment, and therefore limits the kinds of scientific questions that can be addressed using two-photon optogenetics. Conversely, in an ideal experiment, one could precisely stimulate any neuron despite a moderate or high density of opsin-expressing cells.
Here, we develop a novel computational strategy for all-optical experiments [18-21] that reduces (and in some cases entirely eliminates) OTS (Figure 1a, b). The key insight is that heterogeneity in opsin expression and intrinsic excitability causes neurons to have different sensitivities to optical stimulation. This implies that in some cases laser power could be safely turned down to prevent the spiking of a nearby non-target neuron that is weakly driven by stimulation while still spiking a target neuron that is strongly driven by stimulation. Further, the spatial arrangement of neurons can be exploited by moving a holographic target off of a neuron's nucleus (the typical stimulation point) towards a less dense area (e.g. towards neuropil or less photosensitive neurons) while still exciting a sufficient number of the target neuron's opsin molecules to elicit a spike. While some prior experimental research has considered cell-specific tuning of laser powers [22], our advance is to identify both the optimal laser powers and target locations automatically using efficient Bayesian techniques.
To achieve highly precise optogenetic ensemble control, we propose a two-phase process called Bayesian target optimisation (Bataro). In the first phase, we map the "optogenetic receptive field" (ORF) of each neuron by stimulating at various locations around cell nuclei and at a range of laser powers to test whether they elicit spikes. To infer these ORFs without exhaustively testing every combination of location and power, we use Gaussian processes (GPs) to encode prior knowledge about how neurons respond to optogenetic stimulation. Further, we propose to use holographic stimulation to test multiple locations around one or many neurons simultaneously, ensuring that the mapping phase completes quickly. This also allows us to model how neurons integrate two-photon excitation from multiple nearby holograms. In the second phase, we then exploit knowledge of the ORFs to optimise holographic targets. We use properties of the GP to infer gradients of an objective function at a spatial resolution that exceeds the original sampling resolution, enabling us to optimise for the exact holographic targets and laser powers needed to evoke a specific neural activity pattern.
Our work is related to prior research in three areas: GP-based receptive field inference, optimal stimulus design, and statistical methods for estimating connectivity using optogenetic stimulation. Since ref. [23], many studies have used GPs to infer the relationship between neural spiking and multi-dimensional covariates. This includes using GPs to infer place fields [24, 25] and orientation preference maps [26], as well as with Bayesian active learning techniques for inferring sensory receptive fields [27, 28]. However, none to our knowledge have used GP inference to model responses to optogenetic stimulation. Closed-loop methods for optimal control of neural activity using one-photon or electrical stimulation have been recently explored [29-34], though these approaches lack the necessary spatial specificity for highly precise ensemble control. Ref. [35] considers optimising spike-timing using two-photon stimulation (in addition to electrical stimulation), but focused on single neurons and did not use GP estimation methods. Finally, a number of papers have developed statistical models of optogenetic data to infer functional or synaptic connectivity [36-43], but do not consider identifying the exact stimulation parameters needed to evoke specific activity patterns. Here, we unify these three approaches to develop a novel computational framework for optimal holographic stimulation of neural ensembles.
3.1 Optogenetic receptive field model
Given a population of  $N$  neurons, we first wish to learn ORF functions  $g_{n}$  that model how each neuron responds to optogenetic stimulation at different laser powers and locations near the cell soma. Ideally, ORF mapping should be completed quickly, stimulating as few times as possible, so that stimulus optimisation and the desired experiment can begin. Therefore, an ORF model should leverage prior knowledge of how neurons respond to stimulation. Importantly, spike probability should increase monotonically with laser power until saturation [42], and the ORF should approximately match the shape of the somatic membrane (though enlarged to account for the  $\sim 10~\mu \mathrm{m}$  diameter of the
Figure 1: Off-target stimulation with two-photon optogenetics. (a) Two-photon holographic optogenetics can be used to elicit spikes in specific ensembles of experimenter-selected neurons. (b) OTS arises due to an inability to confine two-photon excitation to the soma of a target neuron. By repositioning the hologram away from the soma, OTS could be avoided while still activating the target neuron. (c) Data from real two-photon optogenetics experiment showing that at high power (e.g. 40 mW) a neuron can be activated from  $15 - 20\mu \mathrm{m}$  away, though this depends on the specific pattern of opsin expression at the soma and proximal dendrites. Red and gray circles indicate locations where stimulation resulted in successful or unsuccessful spikes. Colour map shows the inferred probability of spiking (i.e., the ORF) using the log-barrier Newton method from Subsection 3.2. As the number of sampled locations and laser powers increases, the GP model adapts to the particular ORF shape (see supplementary material for the prior mean). This shape can then be exploited to precisely optimise holographic stimuli. Data from PV-neuron in L2/3 of V1 expressing the soma-targeted, excitatory opsin ChroME2f [44].
holographic disk [22]). However, any residual expression of opsin molecules in the proximal dendrites of a neuron creates unique differences in its ORF shape compared to other neurons (Figure 1c). Further, neurons can express opsin in variable concentrations and vary in their intrinsic excitability, leading to an unpredictable dependence of spiking on power. Thus, we need an approach to modelling an ORF that roughly describes the typical shape and photosensitivity of a cell, but that can flexibly adapt to variation across neurons.
To simultaneously account for these factors, we use a novel variant of the GP-Bernoulli model. Let  $y_{nt} \in \{0,1\}$  denote the response of neuron  $n$  on trial  $t$  to a holographic stimulus  $\mathbf{x}_t \in \mathbb{R}^{J \times 3}$ . As our approach is designed for all-optical experiments, the response to stimulation is assumed to be observed for all neurons at once using calcium or voltage imaging. Each stimulus delivers two-photon excitation to  $J \geq 1$  different target locations, with  $\mathbf{x}_t^j = (c_{1t}^j, c_{2t}^j, I_t^j) \in \mathbb{R}^3$  representing the two-dimensional coordinates and laser power of the  $j$ th target (though note that we can also handle three-dimensional hologram coordinates without any substantial changes to the model). While in some cases the targets will be far enough apart in space to not interact with each other, frequently a neuron will integrate two-photon excitation from multiple nearby targets, increasing the risk of OTS. To account for this, we model the probability of evoking a spike in neuron  $n$  by summing across all  $J$  points on that neuron's ORF,
$$
y _ {n t} \sim \operatorname {B e r n o u l l i} \left(\sigma \left(\gamma_ {n} \left(\mathbf {x} _ {t}\right) - \theta_ {n}\right)\right), \quad \gamma_ {n} \left(\mathbf {x} _ {t}\right) = \sum_ {j = 1} ^ {J} g _ {n} \left(\mathbf {x} _ {t} ^ {j}\right), \tag {1}
$$
where  $\sigma$  is the logistic sigmoid function and  $\theta_{n} \in \mathbb{R}$  is a scalar threshold. We model each ORF using a three-dimensional GP prior that describes the effect of stimulating at a given location and power:  $g_{n} \sim \mathcal{GP}(m_{n}(\cdot), k(\cdot, \cdot))$ , where  $m_{n}$  and  $k$  are respectively the mean and covariance functions of the GP. During inference (as discussed below) we constrain  $g_{n}$  to be non-negative, which ensures that stimulation cannot have an inhibitory effect. Further, using a non-negativity constraint rather than (e.g.) applying a rectifying nonlinearity to  $g_{n}$  guarantees that the posterior distribution remains log-concave, facilitating tractable identification of the global optimum.
To encode prior knowledge about optogenetic stimulation, the mean function should have the property that the probability of evoking a spike increases with laser power but decays quickly as the hologram
is positioned further away from the nucleus. To this end, we set
$$
m _ {n} (\mathbf {x}) = \rho I \exp \left(- \| \mathbf {c} - L _ {n} \| ^ {2} / \sigma_ {m} ^ {2}\right) \tag {2}
$$
for  $n = 1, \dots, N$  (note that we suppress reference to the target dimension  $j$  for notational clarity here and in the definition of the covariance function below). Here  $\mathbf{c} = (c_1, c_2)$  denotes the coordinates of the holographic target,  $L_n$  is the location of neuron  $n$ , and  $\rho$  can be loosely interpreted as the average opsin expression level, determining how laser power affects the probability of spiking. While we treat  $\rho$  and  $\sigma_m^2$  as static hyperparameters, in future work one could consider a hierarchical model where the mean function is also learned, with its accuracy improving as additional ORFs are probed.
We model the covariance between points on the ORF using the radial basis function (RBF) kernel  $k$ , which regularises the ORF by encouraging it to vary smoothly through space and power. However, we note that other covariance kernels could be used here provided that they are differentiable with respect to  $\mathbf{x}$ . We define
$$
k \left(\mathbf {x}, \mathbf {x} ^ {\prime}\right) = \alpha^ {2} \exp \left(- \frac {1}{2} \left(\mathbf {x} - \mathbf {x} ^ {\prime}\right) ^ {\top} \boldsymbol {\Lambda} \left(\mathbf {x} - \mathbf {x} ^ {\prime}\right)\right) + \sigma_ {d} ^ {2} \delta_ {\mathbf {x}, \mathbf {x} ^ {\prime}}, \tag {3}
$$
where  $\Lambda = \mathrm{diag}(\lambda_s,\lambda_s,\lambda_I)$  is a diagonal matrix with each diagonal term giving the characteristic lengthscale for the corresponding GP dimension. We assume that the lengthscales  $\lambda_{s}$  in the two spatial dimensions are equal (a typical assumption in holography, see e.g. ref. [6]), but take a different lengthscale  $\lambda_I$  for the laser power dimension. Finally, we denote the hyperparameters of the ORF prior by  $\phi = (\alpha^{2},\sigma_{d}^{2},\lambda_{s},\lambda_{I},\rho ,\sigma_{m}^{2})$
During an ORF mapping experiment we first collect calibration data by performing stimulation in a series of trials  $\mathbf{x}_1, \ldots, \mathbf{x}_T$ , where each  $\mathbf{x}_t$  targets  $J$  different locations and laser powers near a random set of neurons. Then we combine these stimuli with the corresponding observations of any evoked activity  $\mathbf{y}_{:,1}, \ldots, \mathbf{y}_{:,T}$  to estimate the ORFs. Here each  $\mathbf{y}_{:,t} \in \{0,1\}^N$  describes the population response to the stimulus  $\mathbf{x}_t$ . Due to the non-negativity constraints on  $g_n$ , the posterior  $p(g_n \mid \mathbf{y}_n, \{\mathbf{x}_t\}_{t=1}^T, \theta_n, \phi)$  is not itself a GP, and hence we perform maximum a posteriori (MAP) inference rather than attempting to obtain a complete description of posterior uncertainty (though see the supplementary material for a full treatment of posterior uncertainty in the  $J = 1$  case). To this end, the MAP estimates are obtained by maximising the log-posterior, which, via Bayes rule, is equivalent to computing
$$
\hat {g} _ {n}, \hat {\theta} _ {n} = \operatorname * {a r g m a x} _ {g _ {n}, \theta_ {n}} \left\{\sum_ {t = 1} ^ {T} \ln p \left(y _ {n t} \mid \mathbf {x} _ {t}, g _ {n}, \theta_ {n}\right) + \ln p \left(g _ {n} \left(\mathbf {x} _ {1}\right), \dots , g _ {n} \left(\mathbf {x} _ {T}\right) \mid \phi\right) \right\} \tag {4}
$$
such that  $g_{n}(\mathbf{x}_{t})\geq 0$  for  $t = 1,\ldots ,T$
Note that we have abbreviated  $\hat{g}_n(\mathbf{X})$  (the MAP version of the ORF) as  $\hat{g}_n$ . To solve Equation 4 we alternate between using Newton's method with a log-barrier to update  $g_{n}$  (such that the optimised ORFs respect the non-negativity constraints), and performing gradient descent steps for updating  $\theta_{n}$ . We also adaptively set the Newton step-size using a standard backtracking line-search method.
3.3 Optimisation approach
Given the estimated ORFs, we then aim to identify holographic stimuli that evoke specific neural activity patterns as accurately as possible (i.e., minimising off-target activation). To do this, we introduce a novel optimisation approach that exploits properties of the GP to compute gradients of an error function between a target activity pattern and the (predicted) evoked activity. To this end, let  $\mathcal{G} = \{\hat{g}_n,\hat{\theta}_n\}_{n = 1}^N$  be the MAP-estimated ORFs, and define the predicted evoked activity as
$$
\hat {y} (\mathbf {x}, \mathcal {G}) = \left(\sigma \left(\hat {\gamma} _ {1} (\mathbf {x}) - \hat {\theta} _ {1}\right), \dots , \sigma \left(\hat {\gamma} _ {N} (\mathbf {x}) - \hat {\theta} _ {N}\right)\right) \in \mathbb {R} ^ {N}, \tag {5}
$$
where  $\hat{\gamma}_n(\mathbf{x}) = \sum_{j=1}^{J} \hat{g}_n(\mathbf{x}^j)$ . Let  $\Omega \in \{0,1\}^N$  denote a target activity pattern. The optimal holographic stimulus is then
$$
\mathbf {x} _ {\text {o p t i m a l}} = \underset {\mathbf {x}} {\operatorname {a r g m i n}} \| \boldsymbol {\Omega} - \hat {y} (\mathbf {x}, \mathcal {G}) \| ^ {2} \text {s u c h t h a t} 0 \leq I \leq I _ {\max } \tag {6}
$$
Note that we include  $I_{\mathrm{max}}$  as an upper bound on the laser power, which can be set to match the power practically deliverable by the microscopy system or to prevent tissue damage due to excess heating.
We propose to solve Equation 6 using a projected gradient descent algorithm. While for the ORF mapping phase we only need to compute the ORFs  $\hat{g}_n$  at a small set of stimulation points  $\mathbf{x}_t$ , for the optimisation phase we must evaluate the gradients of Equation 6 (and therefore of the ORFs) at a series of new, unobserved points constituting an optimisation path. However, it is not immediately obvious how to evaluate these gradients for a general nonparametric function  $g_n$ . The key idea for our approach is that at each step, gradient descent will provide an updated estimate  $\mathbf{x}^*$  of the optimal stimulus, which will act as a "test point" commonly used in GP regression. We then leverage properties of the GP to perform inference of the ORF gradients at  $\mathbf{x}^*$ , and subsequently perform a gradient descent update that reduces the value of the objective function in Equation 6.
Inferring the ORF gradients relies on the fact that a GP and its derivative are jointly GP-distributed, and hence the derivative can be inferred just from a small number of observations of the GP itself. To this end, note that the covariance between a GP and its derivative is given by [45, Sec 9.4]
$$
\operatorname {c o v} \left(g _ {n} \left(\mathbf {x} _ {t}\right), \frac {\partial}{\partial x _ {d} ^ {*}} g _ {n} \left(\mathbf {x} ^ {*}\right)\right) = \frac {\partial k \left(\mathbf {x} _ {t} , \mathbf {x} ^ {*}\right)}{\partial x _ {d} ^ {*}}, \tag {7}
$$
where  $d$  is any dimension of  $\mathbf{x}^*$  and where we have suppressed reference to the hologram target  $j$  for notational clarity here and in the following equation. The expression in Equation 7 can be used to obtain a predictive distribution over derivative functions consistent with the observed data points, which we use to evaluate the derivatives of  $\hat{g}_n$  at the test point  $\mathbf{x}^*$  via
$$
\frac {\partial \hat {g} _ {n} \left(\mathbf {x} ^ {*}\right)}{\partial x _ {d} ^ {*}} = \frac {\partial m _ {n} \left(\mathbf {x} ^ {*}\right)}{\partial x _ {d} ^ {*}} + \operatorname {c o v} \left(g _ {n} (\boldsymbol {\mathcal {X}}), \frac {\partial g _ {n} \left(\mathbf {x} ^ {*}\right)}{\partial x _ {d} ^ {*}}\right) ^ {\top} \mathbf {K} ^ {- 1} \left(\hat {g} _ {n} (\boldsymbol {\mathcal {X}}) - m _ {n} (\boldsymbol {\mathcal {X}})\right), \tag {8}
$$
where  $\mathcal{X} \in \mathbb{R}^{JT \times 3}$  is the collection of  $JT$  unique points on the ORF that were probed during ORF mapping,  $g_{n}(\mathcal{X}), m_{n}(\mathcal{X}) \in \mathbb{R}^{JT}$  give the value of the GP and mean function evaluated at each such point, and where  $\mathbf{K}$  is the corresponding GP covariance matrix obtained by evaluating the covariance kernel  $k$  at every pair of elements of  $\mathcal{X}$ . Further details are provided in the supplementary material.
Equation 8 allows us to define a closed-form gradient of the objective function, which we use in a projected gradient descent algorithm (Algorithm 1). Note, however, that the objective function is not convex in  $\mathbf{x}$ , and therefore such updates are only guaranteed to converge to a locally optimal solution. We therefore typically run Algorithm 1 with several random initialisations and select the optimised target with the lowest predicted error.
Algorithm 1: Bayesian target optimisation (Bataro).
1 Compute MAP estimates of ORFs  $\{\hat{g}_n,\hat{\theta}_n\}_{n = 1}^N$  from calibration data  $\{\mathbf{y}_n\}_{n = 1}^N,\{\mathbf{x}_t\}_{t = 1}^T$  using Newton's method with log-barrier.
2. Initialise targets  $\mathbf{x} \in \mathbb{R}^{J \times 3}$  to random locations near the somas of the  $J$  target neurons and with random laser powers.
3 while target not converged do
4 Construct gradient vectors  $\nabla_{\mathbf{x}}\hat{\gamma}_n(\mathbf{x})$  for  $n = 1,\dots ,N$  using inference of ORF derivatives (Equation 8).
5 Set  $\delta_{\mathbf{x}} = -2\sum_{n = 1}^{N}(\Omega_{n} - \sigma (\hat{\gamma}_{n}(\mathbf{x}) - \hat{\theta}_{n}))\sigma^{\prime}(\hat{\gamma}_{n}(\mathbf{x}) - \hat{\theta}_{n})\nabla_{\mathbf{x}}\hat{\gamma}_{n}(\mathbf{x}).$
6 Perform gradient descent update  $\mathbf{x}\gets \mathbf{x} + \beta \delta_{\mathbf{x}}$  with step-size  $\beta$
7 | Project laser power onto feasible domain,  $I_{j} \gets \min (I_{j}, I_{\max})$  for  $j = 1, \dots, J$ .
8 end
4.1 Simulations
We first tested Bataro by simulating an optical "write-in" experiment, where we attempted to write specific activity patterns into a hypothetical neural population. To do this, we positioned 50 neurons randomly in a  $250\mu \mathrm{m}\times 250\mu \mathrm{m}$  field of view (FOV), sampled each neuron's ORF from its GP
Figure 2: Minimising off-target stimulation using Bayesian target optimisation. (a) Direct nuclear stimulation at  $70\mathrm{mW}$  successfully activates the target neurons with high probability, but also activates nearby non-target neurons due to OTS. Triangles indicate target neurons. Shading indicates probability of spiking. Optical write-in error (bottom) given as the sum of squared errors between the evoked and desired activity patterns. (b) Optimised stimulation using Algorithm 1 repositions the hologram locations away from the nuclei of non-target neurons, resulting in a substantial reduction in off-target activation. (c) Optimisation trajectory of the 6 different target laser powers. Initial laser powers selected randomly between 50 and  $70\mathrm{mW}$ . (d) Optimising holographic targets for 20 different random ensembles shows a robust reduction of optical write-in error (average reduction,  $75\%$ ).
prior, and generated responses to ensemble stimulation. To quantify the optical write-in error, we used the sum of squared errors  $\sum_{n} (\Omega_{n} - y_{n}(\mathbf{x}))^{2}$ , where  $y_{n}(\mathbf{x})$  is the ground truth probability of neuron  $n$  spiking in response to stimulus  $\mathbf{x}$ . This error takes the interpretable value of  $\sim m$  if  $m$  non-target neurons are inadvertently recruited due to OTS. In this simulation, holographic stimulation of an example ensemble consisting of 6 selected neurons resulted in a substantial recruitment of activity from non-target neurons due to OTS, especially when multiple holograms converged on nearby non-target neurons (Figure 2a; optical write-in error = 7.84).