SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Data and Metrics We used MNIST images as target visual percepts throughout the experiments. Images were resized to be the same size as the output of  $f$  ( $49 \times 49$  pixels), and scaled to have a maximum brightness of 2 (aligned with range( $f$ )). Inspired by [12, 14], we used a perceptual similarity metric designed to capture higher-level differences between images [44]. Let  $v_{l}(\mathbf{t})$  be a function that extracts the downstream representations of target  $\mathbf{t}$  input to a VGG19 network pretrained on ImageNet [45]. The perceptual metric is then given by equation 8.
$$
d (\mathbf {t}, \hat {\mathbf {t}}) = \frac {1}{| t |} \left(\left\| \mathbf {t} - \hat {\mathbf {t}} \right\| _ {2} ^ {2} + \beta \left\| v _ {l} (\mathbf {t}) - v _ {l} (\hat {\mathbf {t}}) \right\| _ {2} ^ {2}\right) \tag {8}
$$
This metric was used by the deep stimulus encoder as a training objective, by the simulated patient to choose a duel winner, and throughout HILO as an evaluation metric.  $\beta = 2.5\mathrm{e - }5$  was selected via cross-validation (see [14] App. B). To aid in interpretability, we also report a secondary metric based on how identifiable the predicted percepts were. We first pretrained a separate deep net to  $99\%$  test accuracy on MNIST classification. We then measured the accuracy of this classifier on the predicted percepts at every iteration of HILO.
Figure 3: Percepts resulting from a naive encoder and the trained DSE for two example target images across 25 randomly selected patients.
4.1 Phosphene Model
To verify that our phosphene model's predictions line up with observed results from real prosthesis users, we repeated analyses from previous state-of-the-art models, evaluating how phosphene appearance changes with electrode location [8] and stimulus parameters [10, 11, 19, 37]. We used the same datasets, consisting of thousands of phosphene drawings and brightness and size ratings collected across multiple epiretinal prosthesis [3] patients over several years. To evaluate phosphene appearances with electrode location, we calculated the correlation between predicted and observed phosphenes  $(R_{i})$  for three shape descriptors: area, eccentricity, and orientation. The final score reported is  $1 - \sum_{i} R_{i}^{2}$  [8]. To evaluate how phosphene appearance was modulated by stimulus parameters, we calculated the mean squared error between the size and brightness of predicted percepts and patient ratings as amplitude, frequency, and pulse duration were varied. The reported values correspond to Figures 4a-c and 5 in [11].
Evaluation results are presented in Table 2. Our model significantly outperforms all baselinse on the Beyeler et al. evaluation, and matches SOTA on the Granley et al. evaluations. Additionally, our model is on average  $45\mathrm{x}$  faster, and uses  $120\mathrm{x}$  less memory than [11]. A detailed description of evaluation methods and additional analysis can be found in Appendix A.2.
Table 2: Evaluation of Phosphene Model
Model	Electrode Location [8]	Stimulus Parameters [11]
S1	S2	S3	4A	4B	4C	5
Nanduri et al. [19]	37.3	19.5	294.8	11.1	72.9	.2	160.9
Beyeler et al. [8]	2.43	7.07	1.15	215.6	108.7	5.52	190.2
Granley et al. [11]	2.43	7.07	1.15	0.9	2.1	0.16	49.5
Proposed	0.28	0.57	0.38	0.73	2.3	0.1	48.6
4.2 Deep Stimulus Encoder
We trained a deep stimulus encoder (DSE) to invert our phosphene model (decoder). The encoder was trained across 13 patient-specific parameters, randomly sampled at every epoch, including for the first time implant position and rotation. This is in contrast to previous DSEs, which either require retraining for every new patient [12, 13, 27], or can only vary two patient-specific parameters [14].
We compared the performance of the DSE to a traditional ('naive') encoder [14] currently used by retinal prostheses [3], illustrated in Fig. 3. The DSE achieved a test perceptual loss of 0.05 and a MNIST accuracy of  $95.6\%$ , significantly outperforming the naive encoder (5.68 and  $51\%$  respectively). Note that this performance is when the true patient-specific parameters are known.
D
Figure 4: Human-in-the-loop optimization of a deep stimulus encoder. A: Two example duels, from which patient preferences are learned. B: Reconstruction error throughout optimization across 100 simulated patients. Insets show the predicted percept resulting from stimulation with various encoders. Note the y axis is on a log scale. C: MNIST accuracy of a pretrained classifier on reconstructed phosphenes. Both plots show smoothed median (window size of 3), with error bars denoting IQR. D: Example percepts after optimization for Naive, DSE without HILO, and HILO encoders for 6 random patients. Note, brightness is capped for display for the Naive encoder.
This performance was slightly better than the values reported in [14] (L1 loss of .108 vs .12 in [14]) despite training across 11 additional patient-specific parameters.
4.3 Human-in-the-Loop Optimization
We ran deep learning-based HILO for 100 randomly sampled simulated patients. After every duel, we evaluated the DSE parameterized by the current prediction of patient-specific parameters on a subset of the MNIST test set. The performance of the learned encoder over time ('HILO') is illustrated in Figure 4, which plots the joint perceptual loss (Figure 4.B) and MNIST accuracy (Fig. 4C).
As baselines for comparison we used a naive encoder, a non-personalized DSE where the patient-specific parameters are guessed (DSE- $\phi_{Guess}$ ), and an ideal DSE using the true  $\phi$  (DSE- $\phi_{True}$ ). To guess  $\phi$ , we consider two approaches, one which selects the midpoint from the ranges in Table 1, and another where performance was averaged across random selections of  $\phi$  from the same ranges. In real patients, we estimate that the performance of a deep stimulus encoder without patient-specific optimization would likely fall somewhere between these two methods, since the distribution of real patients might deviate slightly from Table 1. We therefore plot the region bounded by the performance of a DSE with either of these approaches for guessing  $\phi$ . Example percepts after optimization are shown in Fig. 4D.
The HILO encoder starts with random predictions, but, after a short initial exploration period, quickly surpassed the baselines. After about 75 iterations, performance approached the ideal DSE encoder, however the HILO encoder still resulted in high-quality percepts after as few as 20 iterations. Averaged across patients, the final reconstruction error of the HILO encoder was .071 ± .0031 (SEM)
Number of Duels
Figure 5: Reconstruction error through optimization for noisy patient responses (upper left) and for various misspecifications in the forward model assumed by the DSE. Noise level denotes the percentage of duels where the decision was essentially random ( $p \in [0.35, 0.65]$ ), corresponding to  $\sigma$  of 1e-4, 0.005, 0.01, 0.02, and 0.05, respectively. All y axes are on log scales. Naive encoders and some error bars omitted for clarity.
and MNIST accuracy was  $92\% \pm 1.0\%$ . DSE- $\phi_{Guess}$  had an error of between .25 and 1.1 and MNIST accuracy between  $58.6\%$  and  $78.3\%$ , and the DSE with true  $\phi$  had an error of .05 ± .001 and accuracy of  $95.5\% \pm .1\%$ .
4.4 Robustness
In reality, it is likely that a patient's perceptions will not be perfectly captured by the phosphene model. Further, patient responses for visual prostheses are notoriously noisy [7, 46]. To test HILO's resiliency to these variations, we conducted additional robustness experiments, each with the same 25 simulated patients (Figure 5). First, we varied the noise parameter  $\sigma$  in simulated patients' decision making (Figure 5A). Next we constructed various 'misspecified' forward models, where the ground-truth model used to decode stimuli differed from the forward model assumed by the DSE. First, we varied the trajectories of the simulated axon bundles [47], thereby changing the orientation of phosphenes (Figure 5C). Second, threshold amplitudes for stimulation are notoriously hard to predict, and have been shown to drift by up to  $300\%$  over time [48]. Therefore, we tested a variant where the threshold assumed by the encoder was incorrect by up to  $300\%$  (Figure 5B). Lastly, we used the same forward model, but with patient-specific parameters outside the ranges in 1 (Figure 5D).
At  $\sigma = 1\mathrm{e - }4$  , the patient response was noiseless. For  $\sigma$  equal to .005, .01, and .02, HILO performed similarly to the noiseless model, despite the patient on average making 'random'  $(p\in [0.35,0.65])$  decisions in  $26\%$ $38\%$  ,and  $48\%$  of duels. At  $\sigma = 0.05$  , the decision was random'  $2 / 3$  of the time, and HILO performed similarly or slightly better than the baseline DSE-  $\phi_{Guess}$  . The DSE itself is very resilient to misspecifications in axon trajectory, so HILO performs similarly for this misspecification to the original patients. When thresholds varied, HILO still outperformed the baselines, but converged to slightly worse encodings than without misspecification. Further, HILO surpassed the DSE encoded with the ground-truth  $\phi$  . This demonstrates that even when the forward model assumed by the DSE is incorrect, HILO still tends to converge to a set of patient-specific parameters that work well for the
misspecified patient. For out-of-distribution  $\phi$ , HILO again outperformed both the baseline and true DSEs, but performed worse than in-distribution patients.
Our experiments show that HILO optimization of a deep stimulus encoder led to high-quality, personalized stimulation strategies that outperformed previous state-of-the-art techniques. HILO led to an increase in percept quality compared to using a non-personalized DSE for 99 out of 100 simulated patients, demonstrating the viability of our approach. To enable our HILO algorithm, we also developed a new phosphene model, which is computationally simpler and matches patient data better than previous models, and trained a new DSE, which is able to produce high-quality encodings across all 13 dimensions of patient-specific variations included in our phosphene model. Together, these significantly advance state-of-the-art in patient-specific stimulus encoding, and are important steps towards practically-feasible personalized prosthetic vision in real patients.
The proposed framework combining Bayesian optimization and deep stimulus encoding offers significant improvements over both components in isolation. Use of a DSE allows us to incorporate prior information, reducing the dimensionality of the Bayesian optimization search space from the large stimulus space to the much smaller model parameter space. Our results demonstrate that even when the DSE's predictions are incorrect, this parameterization is still useful for Bayesian optimization based on patient preferences. Additionally, DSEs are able to invert highly nonlinear forward models, enabling encoder-parameterized Bayesian optimization to be applied to a much larger set of problems. Lastly, the learned encoder can be applied for any target percept, without needing additional optimization. On the other hand, without adaptive feedback from HILO, deep stimulus encoders have no method for learning the individual differences of a new patient, which we show leads to suboptimal stimuli. DSEs rely on the accuracy of their assumed forward model over the entire stimulus space. We show that our approach produces stimuli that work well for the patient, even when the forward model is misspecified, or when the patient's responses are noisy.
This approach is practical for stimulus optimization in the wild. The encoder learned during optimization is lightweight, and once deployed, can predict individual stimuli in less than  $5\mathrm{ms}$  on CPU, allowing for high frame rates for prosthetic stimulation. During HILO, updating the Gaussian process model and producing new stimuli on average took 3 seconds, meaning that stimulus optimization could be performed in a matter of minutes. A HILO strategy could be bundled with future visual prostheses, allowing for patients to periodically re-calibrate their devices when they feel the device is not performing adequately, without requiring a clinical professional.
Broader Impacts Although we demonstrate this approach in the context of visual prostheses, our framework is general and could be applied to a variety of sensory devices. Our approach is applicable when the stimulus search space is large and there exists a forward model mapping stimuli to responses. Forward models [49-52] and deep stimulus encoders [53-55] have been successfully used across multiple sensory modalities, and could potentially be adapted for personalization with HILO.
Limitations Although promising, our approach is not without limitations. We assumed that the preference of patients for different stimuli is related to the distance metric used to measure perceptual similarity, which may not be true in practice. However, results by [35] suggest that PBO is robust to a mismatch between the distance metric used to invert the forward model and the preference of patients. Another limitation is that evaluation of our approach was only performed on simulated patients with a simulated perceptual model. However, this is mitigated by the fact that HILO showed robustness to model inaccuracies. Still, since it is difficult to predict the behavior of deep learning models, using a deep stimulus encoder in real patients could raise safety concerns. It may be possible for a deep encoder to produce unconventional stimuli, potentially leading to adverse effects. However, most devices come with firmware responsible for ensuring stimuli stay within FDA-approved safety limits.
In conclusion, our results suggest that combining the strengths of deep learning and Bayesian optimization could significantly improve the perceptual experience of patients fitted with visual prostheses and may prove a viable solution for a range of neuroprosthetic technologies.
Research reported in this publication was supported by the National Library of Medicine of the National Institutes of Health under Award Number DP2LM014268. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
A Phosphene Model
A.1 Methods
This section describes the phosphene model used to simulate patient's perception resulting from stimulation. The model takes in a stimulus vector  $s \in \mathbb{R}^{n_e \times 3}$  specifying the frequency (freq), amplitude (amp), and pulse duration (pdur) of a biphasic pulse train on each electrode. In addition, the model also takes in a vector of patient-specific parameters  $\phi$  (see Table 1). We break these parameters down into implant parameters  $(x, y, rot)$ , global parameters  $(\rho, \lambda, \omega, OD_x, OD_y)$ , and stimulus-related parameters  $(a_0 - a_4)$ ; all explained below.
Exact implant locations vary patient-to-patient. The three implant parameters allow our model to account for these changes. We used a simulated implant inspired by designs of real epiretinal implants [3, 56] and those used in previous simulation studies [14]. It consists of 225 disk electrodes (radius  $75\mu \mathrm{m}$  arranged onto a square,  $15\times 15$  grid with  $400\mu \mathrm{m}$  spacing, initially centered over the fovea. The three implant-related parameters translate and rotate the initial implant to be centered at  $(x,y)$ , and to be rotated by rot degrees. The implant used is depicted in Figure A.2, overlaid on top of a simulated map of axon nerve fiber bundles [57].
Figure A.1: The implant used for optimization, and an example implant after rotation and translation based on patient-specific parameters  $\phi$ . The white circle on the right is the optic disc. Arced lines depict simulated axon nerve fiber bundles.
The remaining model parameters are inspired by various psychophysical and electrophysiological studies [8, 10, 18, 19, 58], and are summarized in the following list:
-  $\rho$ : Average phosphene size. This will be modified locally based on stimulus parameters.
-  $\lambda$ : Average phosphene eccentricity (a measure of phosphene elongation; not to be confused with retinal eccentricity). This will be modified locally based on stimulus parameters.
-  $\omega$ : Orientation scaling factor. The orientation of phosphenes will be the orientation of the underlying axon bundle, scaled by  $\omega$  (Eq. 12).
-  $OD_{x}, OD_{y}$ : The x and y location of the patient's optic disc, into which axon nerve fiber bundles terminate.
-  $a_0 - a_2$ : Coefficients to modulate phosphene brightness with stimulus parameters (Eq. 9).
-  $a_3$ : Coefficient to modulate phosphene size with stimulus parameters (Eq. 10).
-  $a_4$ : Coefficient to modulate phosphene eccentricity with stimulus parameters (Eq. 11).
Each electrode's location on the retina can be determined from the implant parameters. The corresponding location in the visual field  $(\mu_{e})$  is determined using the retinotopic map described in
Watson et al. [59]. Each electrode's phosphene orientation is then  $\theta_{e} = \omega \theta_{axon}$ , where  $\theta_{axon}$  is the orientation of the axon nerve fiber bundle (NFB) underlying the cell (pixel). Axon NFBs are modeled as spirals originating at the optic disc and terminating at each simulated cell. These spirals follow a simulated axon map [47] based on tracings of axon trajectories in 55 human eyes. In summary, phosphene size, eccentricity, brightness, and orientation are modulated based on stimulus parameters and implant location according to the following equations:
$$
b _ {e} = a _ {0} \left(\operatorname {a m p} _ {e}\right) ^ {a _ {1}} + a _ {2} (\text {f r e q} _ {e}) \tag {9}
$$
$$
\rho_ {e} = \rho * a _ {3} * a m p _ {e} \tag {10}
$$
$$
\lambda_ {e} = \lambda \left(\frac {p d u r}{0 . 4 5}\right) ^ {a _ {4}} \tag {11}
$$
$$
\theta_ {e} = \omega * \theta_ {\text {a x o n}} \tag {12}
$$
The phosphene for each electrode is a multivariate Gaussian blob, centered at the electrodes location in visual field  $(\mu_{e})$ , and with covariance matrix  $\Sigma_{e}$  constructed such that the resulting phosphene will have brightness  $b_{e}$ , size  $\rho_{e}$ , eccentricity  $\lambda_{e}$ , and orientation  $\theta_{e}$ , as shown in the following equations (repeated from main text for convenience):
$$
b (x, y) = 2 \pi b _ {e} \det  \left(\boldsymbol {\Sigma} _ {\mathbf {e}}\right) \mathcal {N} ([ x, y ] ^ {\top} | \mu_ {e}, \boldsymbol {\Sigma} _ {\mathbf {e}}), \tag {13}
$$
The covariance matrix  $\boldsymbol{\Sigma}_{\mathrm{e}} = \mathbf{R}\boldsymbol{\Sigma}_{0}\mathbf{R}^{T}$  is calculated from the eigenvalue matrix  $\boldsymbol{\Sigma}_{0}$  and a rotation matrix  $\mathbf{R}$ :
The eigenvalues  $\alpha_{e}$  and  $\beta_{e}$  depend on the intended phosphene area  $(\rho_{e})$  elongation  $(\lambda_{e})$ , and a constant  $\epsilon$  (set to  $e^{-2}$ ):
$$
\alpha_ {e} ^ {2} = - \frac {\rho_ {e} \sqrt {1 - \lambda_ {e} ^ {2}}}{2 \pi \ln \epsilon}, \beta_ {e} ^ {2} = - \frac {\rho_ {e}}{2 \pi \ln \epsilon \sqrt {1 - \lambda_ {e} ^ {2}}}.
$$
This formulation guarantees that the Gaussian blob, when thresholded using  $\epsilon$ , will have the intended area, orientation, eccentricity, and brightness.
Blobs from individual electrodes are summed into a global percept. This linear summation is supported by recent studies, which have shown that percepts from multi-electrode stimulation are often linearly related to the percepts from stimulation on the individual electrodes [36]. Although the sum across electrodes is linear, modulating the size and eccentricity of phosphenes makes the final result a nonlinear function of stimulus parameters, preventing analytic inversion.
A.2 Evaluation
Our model is motivated by similar anatomical and psychophysical phenomena as the previous state-of-the-art model for epiretinal prostheses [11], but its formulation allows for favorable computational properties. In comparison, our model is on average  $45\mathrm{x}$  faster to run, and consumes about  $120\mathrm{x}$  less GPU memory. These computational benefits are the main reason a new model was necessary, and enables a more advanced deep stimulus encoder by allowing training with larger encoder models, longer training duration, and larger batch sizes.
Nonetheless, we also verified that the model produces state-of-the-art predictions, as described in Section 4.1. Despite its similar design, our model achieves much better scores on the Beyeler et al. [8] evaluation for phosphene shape. This is likely because our formulation allows much tighter control of phosphene shape attributes (e.g., size, eccentricity), allowing (for the first time) positive  $R^2$  on shape descriptors for held-out electrodes. Our model performs similarly to the previous state-of-the-art model on the Granley et al. [11] evaluation, which is to be expected given that the equations modulating phosphene appearance with stimulus parameters in both models are very similar.
Fig. A.2 reproduces the plots from Figures 4A-C and 5 of [11], but with our proposed model included. These figures show the brightness or size rating from Argus II patient(s) as stimulus parameters vary [19, 37], in subjective units. For brightness, '10' means the same as the reference pulse, '20' means twice as bright, etc. For size, '1' means the same as reference pulse, '2' means twice as large (notation matching [11, 19, 37]).
Figure A.2: Evaluation of phosphene brightness and size as stimulus parameters vary. Reproduced from [11], but with our proposed phosphene model included. State-of-the-art denotes the phosphene model from [11]. Units are subjective, in comparison to a reference pulse (i.e. brightness of 20 means twice as bright, size of 3 means 3 times as bright) [19]
B Deep Stimulus Inversion
B.1 Architecture
The encoder architecture described in Section 3 is illustrated in Fig. B.1.
Figure B.1: Deep stimulus encoder architecture. FC: fully connected layer, BN: batch normalization layer, B: block of layers (shown at top). Two arrows merging into one fully connected layer denotes concatenation. 45M total parameters.
C Human-in-the-Loop Optimization
C.1 Kernel Selection and Hyperparameters
This section gives more details on fitting the hyperparameters for the Gaussian process (GP) kernel used in preferential Bayesian optimization (PBO). As stated previously, the performance of PBO crucially depends on the GP kernel and its hyperparameters, which encode our prior assumptions about the latent preference function. To select hyperparameters, we used a transfer learning strategy, selecting hyperparameters that generalized best within a small validation group of simulated patients (see Appendix C.2 for discussion).
We simulated 600 random duels ( $\phi_{1}$  and  $\phi_{2}$  chosen randomly) on each of 10 simulated patients. For each patient, we fit four commonly used kernels (Squared Exponential, Squared Exponential with Automatic Relevance Determination (ARD), Matérn 3/2, and Matérn 5/2) and inferred hyperparameters using type II maximum likelihood estimation [60]. The bounds for each hyperparameter were  $[\exp(-10), \exp(10)]$ . For each of these candidate kernel-hyperparameter pairs, we fit a GP with the corresponding kernel and hyperparameters to 50 training duels for each of the other 9 patients. Then, the performance of the candidate GP was evaluated on the remaining 550 data points using Brier score [43], a commonly used metric measuring the accuracy of probabilistic predictions:
$$
B S = \frac {1}{n} \sum_ {i = 1} ^ {n} \left(y _ {\text {t r u e}} - y _ {\text {p r e d}}\right) ^ {2}, \tag {14}
$$
where  $y_{true}$  is the true duel outcome (1 or 0) as decided by the simulated patient, and  $y_{pred}$  is the probability of  $\phi_1$  being selected as the winner (corresponding to outcome of 1), as predicted by the Gaussian process.