SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To optimally reposition the holographic targets, we probed the population's ORFs using 10-target ensemble stimulation, and then used Bataro (Algorithm 1). This successfully repositioned holographic targets an appropriate distance from the nuclei of the target neurons, such that while minimal excitation was applied to the non-target neurons, the target neurons were still driven to spike with high probability (Figure 2b,c; write-in error  $= 1.61$ ). To confirm that the improvement in write-in precision was robust, we repeated the optimisation for 20 additional random 6-target ensembles and found that off-target effects were substantially reduced in every case, with the write-in error reducing by  $75\%$  on average compared to naively stimulating the nucleus of each neuron (Figure 2d).
Next, we investigated two variables known to critically constrain the ability to evoke specific neural activity patterns: (1) the density of opsin-expressing neurons in the stimulation FOV, and (2) the size of the ensemble that one is attempting to activate. Figure 3a illustrates how increasing the number of opsin-expressing neurons introduces many more potential non-target neurons that could be inadvertently activated. This causes errors induced by direct nuclear stimulation of (e.g.) 5-neuron ensembles to grow rapidly (Figure 3b, grey curve), though note that the magnitude of the error depends on many experimental factors including the size of the ORF, numerical aperture of the microscope, and the soma-targeting efficacy. By optimising holographic target locations and laser powers, the average write-in error was reduced by  $76\%$  relative to nuclear stimulation (Figure 3b, red curve). Similarly, in these simulations, nuclear stimulation resulted in an error that grew almost
Figure 3: Performance of Bayesian target optimisation in increasingly difficult contexts. (a) Two example scenarios with low (left) and high (right) density of opsin-expressing neurons. Triangles indicate example 5-target ensemble to be stimulated. At low density, the risk of OTS is low because neurons are often spaced far apart. However, at high density, OTS arising from direct nuclear stimulation at high power is unavoidable. (b) Optical write-in error for nuclear and optimised stimulation of 5-target ensembles. Error bars show the mean error  $\pm 1$  s.d. over 10 different simulations. For each simulation we averaged the write-in error over 20 random ensembles. (c) Same as (b), but for a fixed population size of 50 and varying ensemble size.
Figure 4: Performance of Bayesian target optimisation using simulations based on two-photon holographic optogenetics data. (a) Left: field of view from a real in vitro optogenetics experiment showing every optimised single-target hologram. Opsin fused to red fluorescent protein mRuby3 so that opsin-expressing neurons can be visualised. Unfilled white circles represent putative neurons detected by automated cell segmentation method. Smaller filled white circles represent optimised holographic targets, shown in relation to the cell nucleus by a straight white line. Right: zoomed view of optimised targets, corresponding to dashed region in the left panel. (b) Optimised laser powers relative to their initialised values show how Bataro exploits differences in photosensitivity to avoid OTS. Each circle represents the power delivered to a single holographic target. (c) Reduction of optical write-in error using optimised holographic targets. Each circle represents the error when attempting to stimulate a single neuron.
directly proportionally to the size of the stimulated ensemble, indicating that almost as many non-target neurons were activated as target neurons. However, Bataro reduced the average write-in error by  $69\%$  across ensembles of size 1-15 (Figure 3c).
4.2 Application to holographic optogenetics experiments
To test whether Bataro could eliminate OTS in realistic settings, we created synthetic optogenetics experiments involving  $>100$  neurons from a small number  $(n = 4)$  of detailed cell-attached recordings in slice [42]. Briefly, each slice experiment was performed by first establishing a cell-attached electrophysiological recording of a single L2/3 interneuron from mouse V1 expressing the somatargeted opsin ChroME2f. Then, the recorded neuron was optogenetically stimulated at a dense grid
Figure 5: Optimisation of holographic targets in three-dimensional space. (a) Example target neuron and optimised holographic stimulus (plane  $3,25\mu \mathrm{m}$ ). By repositioning the hologram (primarily in the x/y dimensions), off-target activation is entirely eliminated (bottom spike probability plots, inset numbers show write-in error). Deepest plane labelled as  $0\mu \mathrm{m}$  by convention. Solid white circle indicates target neuron. Dashed white circles indicate nearby non-target neurons that must be avoided. (b) Similar to (a), but for a neuron in plane 2 ( $50\mu \mathrm{m}$ ) and with repositioning of the hologram primarily in the z dimension. (c) All optimised single-target holograms over four stimulation planes. Targets shown at their original depth for visualisation. Average displacement of optimised holographic targets relative to nuclei,  $8.3\mu \mathrm{m}$ . Scale bar,  $20\mu \mathrm{m}$ . (d) Max-projection over z-planes simultaneously showing locations of all segmented opsin-expressing neurons. (e) Optimised depths of holographic targets corresponding to neurons in (c). Depths are shown in comparison to one of four depths that the target neuron was segmented at during the experiment. (f) Optimised laser powers relative to their initialised values. (g) Reduction of optical write-in error for all neurons, across multiple depths.
of locations surrounding the cell and at a range of laser powers to comprehensively map its ORF (see Figure 1c, "100% sampled" column, also see supplementary material for additional examples). We used this data to create a set of four "ground truth" ORFs by fitting the GP-Bernoulli model from Equation 1, where the hyperparameters of the GP covariance kernel were selected using the cross-validated predictive log-likelihood. Next, we used a fluorescence image from a separate experiment to extract the locations of 116 putative opsin-expressing neurons, and randomly assigned each putative neuron one of four ground-truth ORFs. Together, this enabled us to simulate responses to optogenetic stimulation at arbitrary locations and laser powers and for a large number of neurons that were realistically distributed in space.
We sampled responses to optogenetic stimulation at a sparse grid of locations surrounding each neuron to map their ORFs in a hypothetical experiment (see supplementary methods for details). Then, we used Bataro to compute single-target stimuli that activated each neuron individually and minimised OTs (Figure 4a). We found that, on the one hand, a subset of neurons required no change from nuclear stimulation at high power due to being spatially isolated. However, regions with large numbers of closely-packed, opsin-expressing neurons required repositioning of the holographic targets (Figure 4a, right) and adjustment to the laser powers (Figure 4b) to eliminate OTS (average distance between nuclear and optimised stimuli,  $4.9\mu \mathrm{m}$ ). Across all single-target holograms, target optimisation reduced the average write-in error by  $85\%$  (Figure 4c). We also performed a control
analysis by matching the nuclear stimulation laser power to the average optimised laser power and obtained similar results (Figure S3).
Some holographic microscopy systems cannot position holograms at arbitrary continuous depths, instead either requiring holograms to be positioned in two dimensions or on a preselected number of discrete stimulation planes. Hence, thus far, we focused our efforts on the much more common case of two-dimensional optogenetics experiments. However, OTS is generally observed to be even more prevalent in three dimensions, as the optical point spread function is elongated in the "z"-axis [15, 6, 13, 17]. We therefore tested Bataro in the more general setting of three spatial dimensions and laser power (i.e., four dimensions in total), under the assumption that some microscopy systems may be able to flexibly reposition holograms continuously across depths.
We used the fact that the cell-attached recordings were repeated at multiple depths to generate synthetic optogenetics experiments with four-dimensional ORFs (Figure 5a,b). Optimisation of holographic targets successfully reduced OTs by exploiting both the x/y dimensions (Figure 5a), as well the z dimension (Figure 5b), in addition to automatically adjusting the laser power where needed. Repeating the optimisation for every single-target hologram reduced the average optical write-in error by  $89\%$  (from 0.7 to 0.08, Figure 5c-g; average distance between nuclear and optimised targets, 8.3  $\mu \mathrm{m}$ ), demonstrating the feasibility of largely eliminating OTs in optogenetics experiments involving all three spatial dimensions.
We developed Bataro, a novel computational framework for two-photon optogenetics that optimises holographic stimuli to evoke neural activity patterns with minimal off-target activation. Our preliminary in vitro experiments predict that specific adjustments to holographic target placements of  $5 - 10\mu \mathrm{m}$  (on average) could substantially reduce OTS, even in the more challenging regime of three-dimensional optogenetic stimulation. Our key idea was to simultaneously exploit variability in photoexcitability, residual opsin expression in proximal dendrites (due to imperfections in opsin soma-targeting), and the specific spatial arrangement of neurons in the stimulation FOV. Optimisation using these three factors ultimately led to reductions in the optical write-in error of  $85 - 89\%$  for single-target holograms using data from in vitro two-photon optogenetics experiments.
Whether achieving such stimulus precision is worth spending valuable experiment time mapping ORFs is, of course, highly dependent on the application. While for some experiments minimising off-target activation may not be critical, there are many cases where stimulating with as close to "true" single-cell precision as possible is essential to testing a scientific hypothesis. For example, "hub" neurons in the hippocampus are believed to orchestrate network-level function through their unilateral activity [46, 47]. Thus, using two-photon optogenetics to test whether an individual neuron represents a functional hub by stimulating that neuron alone requires true single-cell precision. In such cases, we believe devoting experiment time to ORF mapping and stimulus optimisation is a necessary trade-off.
Throughout our analysis we have assumed that ORFs are stable over time (i.e., are stationary). However, desensitisation of opsins following prolonged illumination has previously been observed [11], indicating that some nonstationarity could be encountered in real experiments. To account for this, we note that after the ORFs have been mapped and holographic stimuli optimised, it is straightforward to periodically recompute the MAP estimate of the ORF as the experiment proceeds. This would provide an updated estimate of the laser power required to maintain optimal precision. We hypothesise that this would not require repeating the entire ORF mapping phase as we do not expect the shape of the ORF to change drastically beyond desensitisation.
Standard GP regression methods scale cubically with the number of locations and laser powers probed, and therefore estimation of each neuron's ORF could encroach on experiment time if ORFs are mapped at high detail just due to required computation time. However, computational efficiency could be improved by appealing to modern GP techniques such as inducing point methods [48], or by adaptively mapping ORFs using Bayesian active learning to minimise the number of probed locations and powers [49, 50]. While in this work we have focused primarily on demonstrating the feasibility of overcoming OTs, in future work we plan to minimise both required experiment and computation time in order to deploy Bataro online.
We thank Darcy Peterka, Benjamin Antin, Cole Hurwitz, Shizhe Chen, and Ben Shababo for helpful discussions and suggestions. This work was funded by NIH awards 1RF1MH120680 and 1U19NS107613-01 to HA and LP. MAT and LP were supported by the Gatsby Charitable Foundation and NSF NeuroNex award 1707398.
[1] John Peter Rickgauer and David W Tank. Two-photon excitation of channelrhodopsin-2 at saturation. Proceedings of the National Academy of Sciences, 106(35):15025–15030, 2009.
[2] Eirini Papagiakoumou, Francesca Anselmi, Aurélien Bègue, Vincent De Sars, Jesper Glückstad, Ehud Y Isacoff, and Valentina Emiliani. Scanless two-photon excitation of channelrhodopsin-2. Nature Methods, 7(10):848–854, 2010.
[3] Adam M Packer, Darcy S Peterka, Jan J Hirtz, Rohit Prakash, Karl Deisseroth, and Rafael Yuste. Two-photon optogenetics of dendritic spines and neural circuits. Nature Methods, 9(12):1202-1205, 2012.
[4] Rohit Prakash, Ofer Yizhar, Benjamin Grewe, Charu Ramakrishnan, Nancy Wang, Inbal Goshen, Adam M Packer, Darcy S Peterka, Rafael Yuste, Mark J Schnitzer, et al. Two-photon optogenetic toolbox for fast inhibition, excitation and bistable modulation. Nature Methods, 9(12):1171-1179, 2012.
[5] Oscar Hernandez, Eirini Papagiakoumou, Dimitrii Tanese, Kevin Fidelin, Claire Wyart, and Valentina Emiliani. Three-dimensional spatiotemporal focusing of holographic patterns. Nature Communications, 7(1):1-11, 2016.
[6] Nicolas C Pégard, Alan R Mardinly, Ian Antón Oldenburg, Savitha Sridharan, Laura Waller, and Hillel Adesnik. Three-dimensional scanless holographic optogenetics with temporal focusing (3d-shot). Nature Communications, 8(1):1-14, 2017.
[7] Emiliano Ronzitti, Cathie Ventalon, Marco Canepari, Benoit C Forget, Eirini Papagiakoumou, and Valentina Emiliani. Recent advances in patterned photostimulation for optogenetics. Journal of Optics, 19(11):113001, 2017.
[8] Nicolò Accanto, Clément Molinier, Dimitrii Tanese, Emiliano Ronzitti, Zachary L Newman, Claire Wyart, Ehud Isacoff, Eirini Papagiakoumou, and Valentina Emiliani. Multiplexed temporally focused light shaping for high-resolution multi-cell targeting. Optica, 5(11):1478-1491, 2018.
[9] Weijian Yang, Luis Carrillo-Reid, Yuki Bando, Darcy S Peterka, and Rafael Yuste. Simultaneous two-photon imaging and two-photon optogenetics of cortical circuits in three dimensions. *elife*, 7:e32671, 2018.
[10] Alan R Mardinly, Ian Antón Oldenburg, Nicolas C Pégard, Savitha Sridharan, Evan H Lyall, Kirill Chesnov, Stephen G Brohawn, Laura Waller, and Hillel Adesnik. Precise multimodal optical control of neural ensemble activity. Nature Neuroscience, 21(6):881-893, 2018.
[11] James H Marshall, Yoon Seok Kim, Timothy A Machado, Sean Quirin, Brandon Benson, Jonathan Kadmon, Cephra Raja, Adelaida Chibukhchyan, Charu Ramakrishnan, Masatoshi Inoue, et al. Cortical layer-specific critical dynamics triggering perception. Science, 365(6453):eaaw5202, 2019.
[12] Luis Carrillo-Reid, Shuting Han, Weijian Yang, Alejandro Akrouh, and Rafael Yuste. Controlling visually guided behavior by holographic recalling of cortical ensembles. Cell, 178(2):447-457, 2019.
[13] Hillel Adesnik and Lamiae Abdeladim. Probing neural codes with two-photon holographic optogenetics. Nature Neuroscience, 24(10):1356-1366, 2021.
[48] Michalis Titsias. Variational learning of inducing variables in sparse gaussian processes. In Artificial Intelligence and Statistics (AISTATS), pages 567-574. PMLR, 2009.
[49] Jeremy Lewi, Robert Butera, and Liam Paninski. Sequential optimal design of neurophysiology experiments. *Neural Computation*, 21(3):619-687, 2009.
[50] Jonathan W Pillow and Mijung Park. Adaptive bayesian methods for closed-loop neurophysiology. Closed Loop Neuroscience, pages 3-18, 2016.
[51] Andrew James McHutchon et al. Nonlinear modelling and control using Gaussian processes. PhD thesis, CiteSeer, 2015.
6 Supplementary material
6.1 Animal ethics statement
All experiments on animals were conducted with approval of the Animal Care and Use Committee of the University of California, Berkeley.
6.2 Compute
All computational procedures were performed either on a desktop workstation running Ubuntu 18.04 with an Intel Xeon E5-2620 v4 CPU, four GTX 1080 Ti GPUs, and 112GB RAM, or on the Axon computer cluster based at the Zuckerman Institute (Columbia University) using nodes comprised of two Xeon E5-2660 v4 CPUs, eight GTX 1080 Ti GPUs, and 125GB RAM.
6.3 Broader societal impact
Our work is significant for interventional approaches to studying the brain and its connection to disease. By minimising off-target activation, Bayesian target optimisation could enable (e.g.) more precise synaptic connectivity mapping, improving our understanding of neural circuitry. This advancement has potential implications for understanding brain disorders like epilepsy, where abnormal synaptic connections are central to seizure generation and propagation. Deepening our understanding of these diseases can lead to enhanced targeted interventions and more effective therapeutic strategies, benefiting individuals with neurological disorders.
6.4 Code availability
An open-source implementation of Bayesian target optimisation is available in Python at https://github.com/marcustriplett/bataro.
6.5 Single-target holographic stimulus optimisation with posterior uncertainty
Here we provide further mathematical details for optimising holographic stimuli. First, we develop the approach for single optogenetic targets, as this is most closely related to existing GP-based receptive field inference techniques. The single-target case also allows us to have a full treatment of posterior uncertainty (unlike for optimising ensemble stimuli) which may be desired in certain applications.
Optogenetic receptive field model. We use a GP-Bernoulli approach to model the response  $y_{nt}$  of neuron  $n$  on trial  $t$  to a single-target stimulus  $\mathbf{x}_t$ ,
$$
y _ {n t} \sim \operatorname {B e r n o u l l i} \left(\sigma \left(g _ {n} \left(\mathbf {x} _ {t}\right)\right)\right), \tag {9}
$$
where the stimulus  $\mathbf{x}_t = (c_{1t}, c_{2t}, I_t) \in \mathbb{R}^3$  represents the two-dimensional coordinates and laser power of the  $t$ -th hologram. Each ORF follows a three-dimensional GP prior  $g_n \sim \mathcal{GP}(m_n(\cdot), k(\cdot, \cdot))$ , where  $m_n$  and  $k$  again are the mean and covariance functions of the GP.
Posterior inference. Unlike for ensemble stimulation, for single-target stimulation we do not require that the ORF  $g_{n}$  is non-negative. This is because now if a point on the ORF becomes inhibitory (by taking a negative value), it will not conflict with excitation from any other hologram. Consequently, the posterior of  $g_{n}$  is a GP, which allows us to work with a full description of posterior uncertainty. To compute the posterior, we use the conventional Laplace approximation. Briefly, this consists of approximating the posterior using a multivariate normal  $q(g_{n} \mid \boldsymbol{\mu}_{n}, \boldsymbol{\Sigma}_{n}) = \mathrm{Normal}(g_{n} \mid \boldsymbol{\mu}_{n}, \boldsymbol{\Sigma}_{n}) \approx p(g_{n} \mid \mathbf{y}_{n}, \mathbf{X}, \boldsymbol{\phi})$ . The mean  $\boldsymbol{\mu}_{n}$  is obtained by maximising the log-posterior, given by the expression
$$
\ln p \left(g _ {n} \mid \mathbf {y} _ {n}, \mathbf {X}, \phi\right) = \sum_ {t = 1} ^ {T} \ln p \left(\mathbf {y} _ {n t} \mid \mathbf {x} _ {t}, g _ {n}\right) + \ln p \left(g _ {n} \left(\mathbf {x} _ {1}\right), \dots , g _ {n} \left(\mathbf {x} _ {T}\right) \mid \phi\right) + \text {c o n s t}, \tag {10}
$$
where  $\mathbf{X} = (\mathbf{x}_1, \dots, \mathbf{x}_T)$  and where const does not depend on  $g_{n}$ . Since the posterior is log-concave in  $g_{n}$ , we use Newton's method to identify the global optimum of Equation 10, and adaptively set the Newton step-size using a standard backtracking line-search method. Letting  $\mathbf{H} = \nabla \nabla_{g_n} \ln p(g_n|$
$\mathbf{y}_n, \mathbf{X}, \phi)$  be the Hessian of the log-posterior, the posterior covariance matrix is obtained by setting  $\boldsymbol{\Sigma}_n = -\mathbf{H}^{-1}\big|_{g_n = \mu_n}$ .
Target optimisation. Let  $G = (g_{1},\ldots ,g_{N})$ , and define the predicted evoked activity for single holographic targets as  $\hat{y} (\mathbf{x},G) = (\sigma (g_1(\mathbf{x})),\dots,\sigma (g_N(\mathbf{x})))$ . To minimise the error between a target binary activity pattern  $\Omega \in \{0,1\} ^N$  and the predicted evoked activity, we solve an optimisation problem that accounts for the uncertainty in the ORF estimates:
$$
\mathbf {x} _ {\text {o p t i m a l}} = \underset {\mathbf {x}} {\operatorname {a r g m i n}} \mathbb {E} _ {q (G | \boldsymbol {\mu}, \boldsymbol {\Sigma})} \left[ \| \boldsymbol {\Omega} - \hat {y} (\mathbf {x}, G) \| ^ {2} \right] \quad \text {s u c h t h a t} \quad 0 \leq I \leq I _ {\max }, \tag {11}
$$
where  $q(G \mid \boldsymbol{\mu}, \boldsymbol{\Sigma}) = \prod_{n=1}^{N} q(g_n \mid \boldsymbol{\mu}_n, \boldsymbol{\Sigma}_n)$  gives the joint posterior across all ORFs. To solve Equation 11, we first sample ORFs  $g_n^{(s)}$  (for  $s = 1, \dots, S$ ) from their posterior distributions to approximate the expected error at the current estimate  $\mathbf{x}^*$ ,
$$
\mathbb {E} _ {q (G | \boldsymbol {\mu}, \boldsymbol {\Sigma})} \left[ \| \boldsymbol {\Omega} - \hat {y} (\mathbf {x} ^ {*}, G) \| ^ {2} \right] \approx \frac {1}{S} \sum_ {s = 1} ^ {S} \sum_ {n = 1} ^ {N} \left(\Omega_ {n} - \sigma \left(g _ {n} ^ {(s)} (\mathbf {x} ^ {*})\right)\right) ^ {2}. \tag {12}
$$
Then, we compute the partial derivative (in dimension  $d$ ) of the expected error by differentiating through the Monte Carlo approximation,
$$
\frac {\partial}{\partial x _ {d} ^ {*}} \mathbb {E} _ {q (G | \boldsymbol {\mu}, \boldsymbol {\Sigma})} \left[ \| \boldsymbol {\Omega} - \hat {y} (\mathbf {x} ^ {*}, G) \| ^ {2} \right] \approx - \frac {2}{S} \sum_ {s = 1} ^ {S} \sum_ {n = 1} ^ {N} \left(\Omega_ {n} - \sigma \left(g _ {n} ^ {(s)} (\mathbf {x} ^ {*})\right) \sigma^ {\prime} \left(g _ {n} ^ {(s)} (\mathbf {x} ^ {*})\right)\right) \frac {\partial}{\partial x _ {d} ^ {*}} g _ {n} ^ {(s)} \left(\mathbf {x} ^ {*}\right). \tag {13}
$$
Next we must evaluate the partial derivative on the right-hand side of Equation 13. We use the fact that a GP and its derivative are jointly GP-distributed, and hence infer the derivative from observations of the ORF. The covariance between a GP and its derivative is given by [45, Sec 9.4]
$$
\operatorname {c o v} \left(g _ {n} (\mathbf {x} _ {t}), \frac {\partial}{\partial x _ {d} ^ {*}} g _ {n} \left(\mathbf {x} ^ {*}\right)\right) = \frac {\partial k \left(\mathbf {x} _ {t} , \mathbf {x} ^ {*}\right)}{\partial x _ {d} ^ {*}} = \frac {\alpha^ {2}}{\lambda_ {d} ^ {2}} \left(x _ {d t} - x _ {d} ^ {*}\right) \exp \left(- \frac {\left\| \mathbf {x} _ {t} - \mathbf {x} ^ {*} \right\| ^ {2}}{2 \lambda_ {d} ^ {2}}\right), \tag {14}
$$
where the second equality is specific to the RBF covariance. Thus, we can use Equation 14 to obtain the posterior predictive mean for the derivative GPs in closed form as [51, Sec 2.7]
$$
\mathbb {E} _ {q \left(g _ {n} \mid \boldsymbol {\mu} _ {n}, \boldsymbol {\Sigma} _ {n}\right)} \left[ \frac {\partial g _ {n} \left(\mathbf {x} ^ {*}\right)}{\partial x _ {d} ^ {*}} \right] = \frac {\partial m _ {n} \left(\mathbf {x} ^ {*}\right)}{\partial x _ {d} ^ {*}} + \operatorname {c o v} \left(g _ {n} (\mathbf {X}), \frac {\partial g _ {n} \left(\mathbf {x} ^ {*}\right)}{\partial x _ {d} ^ {*}}\right) ^ {\top} \mathbf {K} ^ {- 1} \left(\boldsymbol {\mu} _ {n} - m _ {n} (\mathbf {X})\right). \tag {15}
$$