SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Here  $\mathbf{X} = (\mathbf{x}_1, \dots, \mathbf{x}_T)$  is the collection of unique points on the ORF probed during calibration. If Equation 15 is combined with an expression for the posterior predictive variance, one obtains a full predictive distribution over derivative functions consistent with the observed neural responses. However, rather than working with this full distribution, we instead use Equation 15 to approximate the derivatives of the Monte Carlo samples by replacing the posterior mean  $\pmb{\mu}_n$  with a Monte Carlo sample,
$$
\frac {\partial g _ {n} ^ {(s)} \left(\mathbf {x} ^ {*}\right)}{\partial x _ {d} ^ {*}} \approx \frac {\partial m _ {n} \left(\mathbf {x} ^ {*}\right)}{\partial x _ {d} ^ {*}} + \operatorname {c o v} \left(g _ {n} (\mathbf {X}), \frac {\partial g _ {n} \left(\mathbf {x} ^ {*}\right)}{\partial x _ {d} ^ {*}}\right) ^ {\top} \mathbf {K} ^ {- 1} \left(g _ {n} ^ {(s)} (\mathbf {X}) - m _ {n} (\mathbf {X})\right). \tag {16}
$$
Equation 16 then allows us to define a closed-form approximate gradient  $\tilde{\nabla}_{\mathbf{x}^*}g_n^{(s)}$  at test point  $\mathbf{x}^*$ , defined as
$$
\tilde {\nabla} _ {\mathbf {x} ^ {*}} g _ {n} ^ {(s)} = \left[ \frac {\partial g _ {n} ^ {(s)} (\mathbf {x} ^ {*})}{\partial x _ {1} ^ {*}}, \dots , \frac {\partial g _ {n} ^ {(s)} (\mathbf {x} ^ {*})}{\partial x _ {D} ^ {*}} \right] ^ {\top}, \tag {17}
$$
which we use in the single-target projected gradient descent algorithm (Algorithm 2). Note that one could also consider a quadrature approach to solving Equation 12, which may be more efficient than Monte Carlo sampling. However, the presentation of the Monte Carlo approach is instructive for deriving the optimisation of ensemble stimuli below.
Algorithm 2: Projected Monte Carlo gradient descent algorithm for optimising single-target holograms  
1 Infer ORF posterior  $q(G \mid \mu, \Sigma)$  from calibration data  $\{\mathbf{y}_n\}_{n=1}^N$ ,  $\mathbf{X}$  using the Laplace approximation.  
2 Precompute the negative of the Hessian  $\mathbf{W}_n = -\nabla \nabla \ln p(\mathbf{y}_n \mid \mathbf{X}, g_n) \big|_{g_n = \mu_n}$  for each  $n$ .  
3 Initialise  $\mathbf{x}$  to random location near soma of target neuron and with random laser power.  
4 while target not converged do  
5 for  $n = 1, \ldots, N$  do  
6 Compute mean and variance of posterior predictive distribution at current target estimate  $\mathbf{x}$  via  $\mu_n(\mathbf{x}) = m_n(\mathbf{x}) + k(\mathbf{X}, \mathbf{x})^\top \mathbf{K}^{-1} (\boldsymbol{\mu}_n - m_n(\mathbf{X}))$ , and  $\sigma_n^2(\mathbf{x}) = k(\mathbf{x}, \mathbf{x}) - k(\mathbf{X}, \mathbf{x})^\top (\mathbf{K} + \mathbf{W}_n^{-1})^{-1} k(\mathbf{X}, \mathbf{x})$ .  
7 Sample ORFs at the current target estimate,  $g_n^{(s)}(\mathbf{x}) \sim \text{Normal}(\mu_n(\mathbf{x}), \sigma_n^2(\mathbf{x}))$  for  $s = 1, \ldots, S$ .  
8 Construct approximate gradients  $\tilde{\nabla}_{\mathbf{x}} g_n^{(s)}$  for  $s = 1, \ldots, S$  using Equation 17.  
9 end  
10 Set  $\delta_{\mathbf{x}} = -\frac{2}{S} \sum_{s=1}^{S} \sum_{n=1}^{N} (\Omega_n - \sigma(g_n^{(s)}(\mathbf{x})) \sigma'(g_n^{(s)}(\mathbf{x})) \tilde{\nabla}_{\mathbf{x}} g_n^{(s)}(\mathbf{x})$  as per Equation 13.  
11 Perform gradient descent update,  $\mathbf{x} \gets \mathbf{x} + \beta \delta_{\mathbf{x}}$  with step-size  $\beta$ .  
12 Project laser power onto feasible domain,  $I \gets \min(I, I_{\max})$ .  
13 end
6.6 Additional details on ensemble stimulus optimisation approach
The approach for optimising holographic ensemble stimuli is based on the approach for single-target optimisation, but modified to account for differences in the ORF model and inference. In particular, we again seek to minimise the error between a target activity pattern  $\Omega$  and the predicted evoked activity, but now using the MAP estimates  $\mathcal{G} = \{\hat{g}_n,\hat{\theta}_n\}_{n = 1}^N$  in place of the full posterior distributions. Let  $\hat{y} (\mathbf{x},\mathcal{G}) = (\sigma (\hat{\gamma}_1(\mathbf{x}) - \hat{\theta}_1),\dots ,\sigma (\hat{\gamma}_N(\mathbf{x}) - \hat{\theta}_N))$  be the predicted population response to an ensemble stimulus, where  $\hat{\gamma}_n(\mathbf{x}) = \sum_{j = 1}^{J}\hat{g}_n\left(\mathbf{x}^j\right)$ . The optimal ensemble stimulus is now
$$
\mathbf {x} _ {\text {o p t i m a l}} = \underset {\mathbf {x}} {\operatorname {a r g m i n}} \| \boldsymbol {\Omega} - \hat {y} (\mathbf {x}, \mathcal {G}) \| ^ {2} = \underset {\mathbf {x}} {\operatorname {a r g m i n}} \sum_ {n = 1} ^ {N} \left(\Omega_ {n} - \sigma \left(\hat {\gamma} _ {n} (\mathbf {x}) - \hat {\theta} _ {n}\right)\right) ^ {2} \tag {18}
$$
such that  $0 \leq I \leq I_{\max}$ . Evaluating the partial derivative of Equation 18 with respect to dimension  $d$  of a test point  $\mathbf{x}^*$  yields,
$$
\frac {\partial}{\partial x _ {d} ^ {*}} \| \boldsymbol {\Omega} - \hat {y} (\mathbf {x} ^ {*}, \mathcal {G}) \| ^ {2} = - 2 \sum_ {n = 1} ^ {N} \left(\Omega_ {n} - \sigma \left(\hat {\gamma} _ {n} \left(\mathbf {x} ^ {*}\right) - \hat {\theta} _ {n}\right)\right) \sigma^ {\prime} \left(\hat {\gamma} _ {n} \left(\mathbf {x} ^ {*}\right) - \hat {\theta} _ {n}\right) \frac {\partial}{\partial x _ {d} ^ {*}} \hat {\gamma} _ {n} \left(\mathbf {x} ^ {*}\right). \tag {19}
$$
The derivative on the right-hand side of Equation 19 is given by  $\frac{\partial}{\partial x_d^*}\hat{\gamma}_n(\mathbf{x}) = \sum_{j=1}^{J}\frac{\partial}{\partial x_d^*}\hat{g}_n(\mathbf{x}^j)$ , which requires computing the derivative of  $\hat{g}_n(\mathbf{x}^j)$ . To evaluate this derivative, we use a similar trick to Equation 16, but substituting the MAP estimate in place of the posterior mean or Monte Carlo sample,
$$
\frac {\partial}{\partial x _ {d} ^ {*}} \hat {g} _ {n} \left(\mathbf {x} ^ {*}\right) = \frac {\partial}{\partial x _ {d} ^ {*}} m _ {n} \left(\mathbf {x} ^ {*}\right) + \operatorname {c o v} \left(g _ {n} (\mathbf {X}), \frac {\partial}{\partial x _ {d} ^ {*}} g _ {n} \left(\mathbf {x} ^ {*}\right)\right) ^ {\top} \mathbf {K} ^ {- 1} \left(\hat {g} _ {n} (\mathbf {X}) - m _ {n} (\mathbf {X})\right). \tag {20}
$$
This expression can also be arrived at by first evaluating the posterior predictive mean of  $g_{n}(\mathbf{x}^{*})$ , and then differentiating with respect to  $x_{d}^{*}$ .
We use Equation 20 to define a closed-form gradient  $\nabla_{\mathbf{x}^*}\hat{\gamma}_n$  at test point  $\mathbf{x}^*$  via
$$
\nabla_ {\mathbf {x} ^ {*}} \hat {\gamma} _ {n} = \left[ \frac {\partial \hat {\gamma} _ {n} \left(\mathbf {x} ^ {*}\right)}{\partial x _ {1} ^ {*}}, \dots , \frac {\partial \hat {\gamma} _ {n} \left(\mathbf {x} ^ {*}\right)}{\partial x _ {D} ^ {*}} \right] ^ {\top}. \tag {21}
$$
Finally, Equation 21 is used in the projected gradient descent algorithm for optimising ensemble stimuli (Algorithm 1).
Figure S1: Effect of reducing the number of points at which each ORF is probed. (a) In the high coverage case (left), each ORF is probed by stimulating at a  $5 \times 5$  grid of points near the soma (grid points separated by  $10\mu \mathrm{m}$ ), at three different laser powers. In the low coverage case (right), this reduces to stimulating at just a  $3 \times 3$  grid (grid points separated by  $12\mu \mathrm{m}$ ) at three powers. However, as the density of opsin-expressing neurons increases, ORFs are probed at high density even in the low coverage case as the grids from different neurons increasingly overlap. (b) Minimal performance difference between the high and low coverage cases in simulations with 50 neurons. (c) Reduction in optical write-in error using cell-attached recordings as in Figure 4, but with low coverage. Reduction in average write-in error,  $74\%$  (c.f.  $85\%$  with high coverage, Figure 4c).
6.7 Further details on simulations and "synthetic" optogenetics experiments
Simulations consisted of both ORF mapping and stimulus optimisation phases. ORF mapping required probing responses to stimulation at a range of laser powers and stimulus locations. We defined a grid of stimulation points surrounding each neuron. In the spatial dimensions, the grid ranged from  $-20\mu \mathrm{m}$  to  $20~\mu \mathrm{m}$  relative to the centroid of the neuron in steps of  $10~\mu \mathrm{m}$ , and powers ranged from  $30\mathrm{mW}$  to  $70\mathrm{mW}$  in steps of  $20\mathrm{mW}$ . The complete grid was thus given by the Cartesian product  $\{-20, - 10,0,10,20\} \times \{-20, - 10,0,10,20\} \times \{30,50,70\}$ . For opsin-expressing neurons that were spaced far apart, this coarse-resolution grid was sufficient because risk of OTS was low, and therefore ORF mapping was not needed at high detail. On the other hand, as the density of opsin-expressing neurons increased, the grids surrounding each neuron increasingly overlapped with each other, resulting in much denser sampling of the ORFs.
For the synthetic optogenetics experiments (based on the cell-attached recordings), we used the same spatial grid spacing but used laser powers of 10, 25, and  $40\mathrm{mW}$  to match the range of powers used in the underlying slice experiment, though note that the slice experiment had a denser spacing than our chosen  $15\mathrm{mW}$  (see example loose-patch recordings below), which we chose to reduce the ORF mapping time. For the optogenetics experiments involving three spatial dimensions, we extended the grid sampling to include depths of  $-60~\mu \mathrm{m}$  to  $60~\mu \mathrm{m}$  in steps of  $30~\mu \mathrm{m}$ . We also explored the effect of reducing the number of probed grid points to further reduce the time spent mapping ORFs, and found that Bayesian target optimisation maintained high performance when probing with a  $3\times 3$  spatial grid of  $\{-12,0,12\} \times \{-12,0,12\}$  (Figure S1).
We selected the parameters of the GP covariance kernel using 5-fold cross-validation on a separate set of recordings that were made on the same set of four cells, ensuring the hyperparameter selection was using out-of-sample data. Cross-validation was performed using a grid search over a set of possible hyperparameters: the possible radial lengthscales were 2, 4, 8, 16, the power lengthscales were 2, 4, 8, 16, and the amplitudes were 1, 2, 4, 8, 16. For each hyperparameter combination  $\theta$  and for each cell, we used Newton's method to fit the GP-Bernoulli model to  $80\%$  of the trials in the loose-patch data, yielding an ORF estimate  $\hat{g}_{\theta}$ . On the remaining  $20\%$  of the trials (denoted as  $\mathcal{T}_{\mathrm{held - out}}$ ), we evaluated the log-likelihood,  $\sum_{t\in \mathcal{T}_{\mathrm{held - out}}}\{y_t\ln (\sigma (\hat{g}_\theta (\mathbf{x}_t))) + (1 - y_t)\ln (1 - \sigma (\hat{g}_\theta (\mathbf{x}_t)))\}$ . We averaged the log-likelihood across all five folds and across all four cells, and chose the hyperparameter combination  $\theta$  that yielded the largest average log-likelihood, resulting in a radial lengthscale of 8, a power lengthscale of 16, and a kernel amplitude of 8.
The GP parameters for generating the simulations in Figure 3, inferring the resulting ORFs, and generating synthetic optogenetics experiments with two and three spatial dimensions are given in Table S1. For reference, a typical ORF mean function is given in Figure S2.
Figure S2: Example mean function (shown at three powers) used for simulations (left column). Also shown are four samples from the ORF prior corresponding to this mean function (right four columns). Parameters given in Table S1.
Figure S3: Performance of Bayesian target optimisation compared to nuclear stimulation when average laser powers are matched. Each repeat consists of randomly reassigning neurons different optogenetic receptive fields while keeping their spatial positions fixed, and performing a full mapping and target optimisation sequence. Within each repeat, nuclear stimulation was performed at the average optimised power. Left: average write-in error for nuclear stimulation remains substantially higher than with optimised stimulation despite having matched average powers. Each circle represents a single neuron. Right: powers used within each repetition. Note that these powers are lower than the  $40\mathrm{mW}$  used to benchmark performance in Figures 4 and 5 in the main text.
Parameter	Symbol	Value
Simulations (data generation)
Mean function excitability	ρ	0.125
Mean function width	σm2	3 × 102μm
Spike threshold	θ	3.5
Kernel radial lengthscale	λs	8 μm
Kernel power lengthscale	λI	20 mW
Kernel amplitude	α2	0.2
Kernel marginal variance	σd2	10-5
Simultaneously stimulated neurons during ORF mapping	J	10
Simulations (ORF inference)
Mean function excitability	ρ	0.125
Mean function width	σm2	3 × 102μm
Kernel radial lengthscale	λs	5 μm
Kernel power lengthscale	λI	16 mW
Kernel amplitude	α2	1
Kernel marginal variance	σd2	10-5
Learning rate for spike thresholds (θnNn=1)	-	5
Number of random initialisations	-	5
Synthetic optogenetics experiments (two spatial dimensions)
Mean function excitability	ρ	0.175
Mean function width	σm2	3 × 102μm
Kernel radial lengthscale	λs	8 μm
Kernel power lengthscale	λI	16 mW
Kernel amplitude	α2	8
Kernel marginal variance	σd2	10-5
Learning rate for spike thresholds (θnNn=1)	-	5
Number of random initialisations	-	5
Synthetic optogenetics experiments (three spatial dimensions)
Mean function excitability	ρ	0.175
Mean function width (x/y dimensions)	σm2	3 × 102μm
Mean function width (z dimension)	-	3 × 103μm
Kernel radial lengthscale (x/y dimensions)	λs	8 μm
Kernel axial lengthscale (z dimension)	λz	32 μm
Kernel power lengthscale	λI	16 mW
Kernel amplitude	α2	8
Kernel marginal variance	σd2	10-5
Learning rate for spike thresholds (θnNn=1)	-	5
Number of random initialisations	-	5
Table S1: Parameters used for simulations and generating synthetic optogenetics experiments.
6.8 Additional examples of optogenetic receptive fields from cell-attached recordings
Figures S4 to S7 show examples of four ORFs that have been comprehensively mapped using two-photon optogenetic stimulation and cell-attached recordings of evoked spikes. Note the unpredictable differences in ORF shape across laser powers and depths, motivating a nonparametric approach.
Figure S4: Loose-patch recording and inferred ORF (experiment 1/4).
Figure S5: Loose-patch recording and inferred ORF (experiment 2/4)
Figure S6: Loose-patch recording and inferred ORF (experiment 3/4)
Figure S7: Loose-patch recording and inferred ORF (experiment 4/4)