SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jacob Granley
Department of Computer Science  
University of California, Santa Barbara  
jgranley@ucsb.edu
Tristan Fauvel
Institut de la Vision, Sorbonne Université 17 rue Moreau, F-75012 Paris, France Now with Quinten Health t.fauvel@quinten-health.com
Matthew Chalk
Institut de la Vision, Sorbonne Université 17 rue Moreau, F-75012 Paris, France matthew.chalk@inserm.fr
Department of Computer Science  
Department of Psychological & Brain Sciences  
University of California, Santa Barbara  
mbeyeler@ucsb.edu
Neuroprostheses show potential in restoring lost sensory function and enhancing human capabilities, but the sensations produced by current devices often seem unnatural or distorted. Exact placement of implants and differences in individual perception lead to significant variations in stimulus response, making personalized stimulus optimization a key challenge. Bayesian optimization could be used to optimize patient-specific stimulation parameters with limited noisy observations, but is not feasible for high-dimensional stimuli. Alternatively, deep learning models can optimize stimulus encoding strategies, but typically assume perfect knowledge of patient-specific variations. Here we propose a novel, practically feasible approach that overcomes both of these fundamental limitations. First, a deep encoder network is trained to produce optimal stimuli for any individual patient by inverting a forward model mapping electrical stimuli to visual percepts. Second, a preferential Bayesian optimization strategy utilizes this encoder to optimize patient-specific parameters for a new patient, using a minimal number of pairwise comparisons between candidate stimuli. We demonstrate the viability of this approach on a novel, state-of-the-art visual prosthesis model. We show that our approach quickly learns a personalized stimulus encoder, leads to dramatic improvements in the quality of restored vision, and is robust to noisy patient feedback and misspecifications in the underlying forward model. Overall, our results suggest that combining the strengths of deep learning and Bayesian optimization could significantly improve the perceptual experience of patients fitted with visual prostheses and may prove a viable solution for a range of neuroprosthetic technologies.
Sensory neuroprostheses are devices designed to restore or enhance perception in individuals with sensory deficits. They often interface with the nervous system by electrically stimulating neural tissue in order to provide artificial sensory feedback to the user [1, 2]. For instance, visual prostheses have the potential to restore vision to people living with incurable blindness by bypassing damaged parts of the visual system and directly stimulating the remaining cells in order to evoke visual percepts (phosphenes) [3-6]. However, patient outcomes with current technologies are limited. Patients
Figure 1: Left: Deep stimulus encoder (DSE). A forward model  $(f)$  is used to approximate the perceptual response to electrical stimuli, subject to patient-specific parameters  $\phi$ . An encoder  $(f^{-1})$  is then learned to minimize the perceptual error between predicted and target percept. Right: Human-in-the-loop optimization (HILO). Patient-specific parameters  $\phi$  of the DSE are optimized with user preferences: the patient performs a series of binary comparisons between percepts evoked with different encoders. New pairs of parameters to compare are adaptively selected so as to efficiently find the parameters maximizing the patient's preference. The target changes each iteration.
require extensive training to learn to interpret the evoked percepts, which are typically described as "fundamentally different" from natural vision [7]. Moreover, phosphene appearance varies widely across patients [8], making personalized stimulus optimization a major outstanding challenge [9].
Much work has gone into developing computational models that can predict the neuronal or perceptual response to an electrical stimulus [8, 10, 11] (often called forward models). Once the forward model is known, a deep neural network can approximate its inverse, thereby identifying the required stimulus to elicit a desired percept [12-14]. However, these inverse models typically assume perfect knowledge of the patient-specific mapping from stimulation to perception (which is not practically feasible) and are heavily reliant on the forward model's accuracy over the entire stimulus space.
Alternatively, Bayesian optimization has been successful in personalizing stimulation strategies for many existing neural interfaces [15, 16]. However, this approach is limited in practice because it requires the stimulus dimension to be small (typically  $< 30$  [17], which is orders of magnitudes smaller than the number of stimulus parameters in current implants), and optimization must be repeated for every stimulus. Moreover, visual prosthesis users can typically only give indirect feedback (e.g., verbal phosphene descriptions), unsuitable for traditional Bayesian optimization.
To address these challenges, we propose a novel framework that integrates deep learning-based stimulus inversion into a preferential Bayesian optimization strategy to learn a patient-specific stimulus encoder (Fig. 1). First, a deep stimulus encoder (DSE) is trained to optimize stimuli assuming perfect knowledge of a set of patient-specific parameters (Fig. 1, left). Second, we embed the DSE within a human-in-the-loop optimization (HILO) strategy based on preferential Bayesian optimization, which iteratively learns the ground-truth patient-specific parameters through a series of 'duels', where the patient is repeatedly asked their preference between two candidate stimuli. The resulting DSE can then be deployed as a personalized stimulation strategy.
To this end, we make the following contributions:
- We introduce a forward model for retinal implants that achieves state-of-the-art response predictions. Unlike previous models, this allows us to train a deep stimulus encoder to predict optimal stimuli across 13 dimensions of patient-specific parameters.
- We propose a personalized stimulus optimization strategy for visual prostheses, where a human-in-the-loop optimization (HILO) Bayesian optimization algorithm iteratively learns the optimal patient-specific parameters for a deep stimulus encoder.
- We demonstrate the viability of our approach by conducting a comprehensive series of evaluations on a population of simulated patients. We show HILO quickly learns a personalized stimulus encoder and leads to dramatic improvements in the quality of restored vision, outperforming existing encoding strategies. Importantly, HILO is resilient to noise in patient feedback and performs well even when the forward model is misspecified. Code for the forward model, DSE, and HILO algorithm is available at https://github.com/bionicvisionlab/2023-NeurIPS-HILO.
Visual Neuroprostheses Numerous groups worldwide are pursuing a visual prosthesis that stimulates viable neuronal tissue in the hope of restoring a rudimentary form of vision to people who are blind (Fig. 2, left) [3-6]. Analogous to cochlear implants, these devices electrically stimulate surviving cells in the visual pathway to evoke visual percepts (phosphenes). Existing devices generally provide an improved ability to localize high-contrast objects and perform basic mobility tasks.
Much work has focused on characterizing phosphene appearance as a function of stimulus and neuroanatomical parameters [2, 10, 18-21]. In epiretinal implants, phosphenes often appear distorted due to inadvertent activation of nerve fiber bundles in the optic fiber layer of the retina [8], causing elongated percepts (Fig. 2, center). In addition, the exact brightness and shape of these elicited percepts depends on the applied stimulus [19] and differs widely across patients (Fig. 2, right). Granley et al. [11] captured these individual differences by including a set of patient-specific parameters in their phosphene model, denoted by  $\phi$ , which includes both neuroanatomical (e.g., implant location) and stimulus-related parameters (e.g., how brightness scales with amplitude).
Deep Stimulus Encoding Many works attempt to mitigate distortions in prosthetic vision, but do not describe comprehensive stimulation strategies [22-24]. Those that describe strategies in detail typically require simplification [25] or strong assumptions [26] to be used in practice. Due to the complexities of optimization, deep learning-based stimulus encoders have risen in popularity [12-14]. In [12], authors proposed an innovative approach where the latent representations of an autoencoder are treated as stimuli and decoded with a phosphene model. However, they used an unrealistic binary phosphene model. Their approach has since been adapted for cortical models [27], and for non-differentiable forward models [13]. Granley et al. [14] generalized the approach, showing it could work with realistic forward models across a small range of patients without needing to retrain.
Given a forward (phosphene) model  $f$  (mapping stimuli to percepts given  $\phi$ ), it is straightforward to show that the optimal stimulus encoder (mapping target images to stimuli) is the pseudoinverse of  $f$  [14]. However, to account for the wide range of individual differences in phosphene perception, most realistic forward models are highly nonlinear and not analytically invertible. Thus, previous works have proposed to use the forward model [12, 14] as a fixed decoder within a deep autoencoder trained to minimize the reconstruction error between target images and the predicted percepts. After training, the encoder can be extracted and used to encode target visual inputs in real time. Deep
Figure 2: Left: Visual prosthesis. Incoming target images are transmitted from a camera to an implant in the retina, which encodes the image as an electrical stimulus pattern. Center: Electrical stimulation (red disc) of a nerve fiber bundle (gray lines) leads to elongated tissue activation (gray shaded region) and a phosphene (bottom). Right: The same stimulus parameters may lead to widely varying visual perceptions in different patients. Adapted with permission from [14].
stimulus encoders trained using this approach produce high quality stimuli, but assume knowledge of  $\phi$ . Additionally, if the forward model  $f$  is not extremely accurate over the whole stimulus space, then the encoder network might learn to exploit inaccuracies in the model, producing stimuli that don't generalize to real patients [14]. We utilize an enhanced variant of this approach in our experiments.
Preferential Bayesian Optimization Preferential Bayesian optimization (PBO) is an efficient method for optimizing expensive black-box functions based on binary comparisons [28, 29]. Since the subject's response to stimulation cannot be directly observed, PBO instead builds a Bayesian model of the subject's preferences,  $g$ , typically modeled using a Gaussian process. An approximate inference algorithm (expectation propagation; [30, 31]) is used to infer the posterior distribution of the preference function given binary comparison data,  $p(g|\mathcal{D})$ , which is then used to select new configurations for the next trial according to an acquisition rule. The acquisition rule must balance the exploration-exploitation trade-off inherent to any black-box optimization problem [32].
PBO was previously used to tune BCI stimulation parameters for transcranial [33] and spinal cord stimulation [34]. However, these works directly optimized only a handful of stimulation parameters and cannot translate to visual prostheses, where complex and varying visual inputs have to be mapped to high-dimensional stimuli. To this end, Fauvel & Chalk [35] reduced optimization complexity by inverting a model of phosphene perception, then used PBO to generate encodings preferred by sighted subjects viewing simulated prosthetic vision. However, a linear approximation was used to invert the perception model, which is unrealistic for real-world applications.
Summary We identify 3 main limitations of previous work that this study aims to address:
- Generalizability of deep stimulus encoders. Autoencoder-like deep stimulus encoders can accurately optimize stimuli, but require perfect knowledge of patient-specific parameters [14], which can be difficult or impossible to determine in practice [8, 11]. Further, these approaches heavily rely on the accuracy of the forward model [13, 14], while real patients will likely deviate from the forward model. We overcome this limitation by optimizing the learned stimulus encoder based on patients' preferences, which we show is not bounded by a misspecified forward model.
- Applicability of Bayesian optimization. Bayesian optimization is ideally suited for optimizing stimulation parameters based on limited, noisy measurements, but can only optimize a small number of parameters [17]. We train a deep stimulus encoder to output optimal stimuli for any specific patient with known parameters, reducing the search space from the entire stimulus space (hundreds or thousands of parameters) to the low-dimensional space of patient-specific parameters (13 parameters with our model), enabling Bayesian optimization.
- Simplistic models of perception. Most previous approaches use overly simplified forward models that do not match empirical data [8, 19]. More accurate models [11] are too computationally expensive to support deep stimulus optimization over a wide range of patients. We overcome this by developing a new phosphene model that matches patient data better than existing models, with significantly reduced memory and time complexity
General Framework We consider a system attempting to optimize stimuli for a new patient, specified by a set of (unknown) parameters  $\phi$ . The goal of optimization is a patient-specific stimulus encoder mapping target perceptual responses  $t$  (e.g., visual percepts) to stimuli  $s$ .
We assume there exists a forward model  $f$  which predicts the patient's perceptual response to stimulation:  $\hat{\mathbf{t}} = f(\mathbf{s};\phi)$ . It follows that the optimal stimulus encoder is the (pseudo)inverse of  $f$ . The inverse can be approximated using an autoencoder-like deep neural network, with  $f$  as the fixed decoder and  $\hat{f}^{-1}$  as the learned stimulus encoder [14]. The encoder's weights  $w$  are updated using gradient descent to minimize the reconstruction error, measured by some distance function  $d$ , between the target  $\mathbf{t}$  and the predicted response  $\hat{\mathbf{t}}$  averaged across patients and a dataset of targets (Eq. 1):
$$
w \approx \underset {w} {\arg \min } \mathbb {E} _ {\mathbf {t}, \phi} [ d (\mathbf {t}, \hat {\mathbf {t}}) ] \tag {1}
$$
$$
\hat {\mathbf {t}} = f \left(\hat {f} _ {w} ^ {- 1} (\mathbf {t}, \phi); \phi\right) \tag {2}
$$
Table 1: Patient-Specific Parameters  $\phi$
ρ (dva)	λ	ω	a0	a1	a2	a3	a4	ODx(μm)	ODy(μm)	Implant Parameters
x (μm)	y (μm)	rot (deg)
Lower	1.5	.45	.9	.27	.42	.005	.2	-0.5	3700	0	-500	-500	-30
Upper	8	.98	1.1	.57	.62	.025	.7	-0.1	4700	1000	500	500	30
Once trained, the encoder can accurately predict stimuli, but requires knowledge of the patient-specific parameters  $\phi$ . For a new patient, Bayesian optimization is used to optimize  $\phi$  based on user feedback, thereby learning a personalized DSE. Since the patient's response cannot be directly measured for visual prostheses, the user is presented with a 'duel', i.e. a binary comparison, where they are asked to decide which of two candidate stimuli they prefer [35]. A Gaussian process model of preferences is updated based on the patient's response, and an acquisition function is used to generate new candidate stimuli. The process can be repeated to iteratively tune the DSE to the patient's preferences.
Phosphene Model The phosphene model is a differentiable approximation of the underlying biological system (also called a forward model [14]), which maps an electrical stimulus to a visual percept. Although phosphene models exist for visual prostheses, current models either do not match patient data well [8, 10, 19], or are too computationally expensive [11].
Thus, we developed a new phosphene model for epiretinal prostheses (See Appendix A.1 for full details). The model takes in a stimulus vector  $\mathbf{s} \in \mathbb{R}^{n_e \times 3}$  specifying the frequency, amplitude, and pulse duration of a biphasic pulse train on each of  $n_e$  electrodes. The output phosphene for each electrode is a Gaussian blob centered over the electrode's location  $\mu_e(\phi)$  with covariance matrix  $\Sigma_{\mathbf{e}}(\mathbf{s}, \phi)$  constructed so that the resulting percept will have area  $\rho_e(\mathbf{s}, \phi)$ , eccentricity  $\lambda_e(\mathbf{s}, \phi)$  and orientation  $\theta_e(\phi)$ . These functions allow phosphene properties to vary locally with stimulus (e.g., current spread) and anatomical parameters (e.g., electrode location, underlying axon nerve fiber bundle trajectory). The percept for an electrode located at  $\mu_e$ , is a multivariate Gaussian blob, renormalized to have maximum brightness  $b_e(\mathbf{s}, \phi)$ :
$$
b (x, y) = 2 \pi b _ {e} \det  \left(\boldsymbol {\Sigma} _ {\mathbf {e}}\right) \mathcal {N} ([ x, y ] ^ {\top} | \mu_ {e}, \boldsymbol {\Sigma} _ {\mathbf {e}}), \tag {3}
$$
where  $b_{e}, \mu_{e}$ , and  $\Sigma_{e}$  are implicitly parametrized by  $s$  and  $\phi$ . The covariance matrix  $\mathbf{\Sigma}_{\mathbf{e}} = \mathbf{R}\mathbf{\Sigma}_{\mathbf{0}}\mathbf{R}^{T}$  is calculated from the eigenvalue matrix  $\mathbf{\Sigma}_{\mathbf{0}}$  and a rotation matrix  $\mathbf{R}$ :
$$
\Sigma_ {0} = \left[ \begin{array}{c c} \alpha_ {e} ^ {2} & 0 \\ 0 & \beta_ {e} ^ {2} \end{array} \right], \hskip 1 4. 2 2 6 3 7 8 p t R = \left[ \begin{array}{c c} \cos \theta_ {e} & - \sin \theta_ {e} \\ \sin \theta_ {e} & \cos \theta_ {e} \end{array} \right].
$$
The eigenvalues  $\alpha_{e}$  and  $\beta_{e}$  depend on the intended phosphene area  $(\rho_{e})$  and elongation  $(\lambda_{e})$ :
$$
\alpha_ {e} ^ {2} = - \frac {\rho_ {e} \sqrt {1 - \lambda_ {e} ^ {2}}}{2 \pi}, \beta_ {e} ^ {2} = - \frac {\rho_ {e}}{2 \pi \sqrt {1 - \lambda_ {e} ^ {2}}}.
$$
Blobs from individual electrodes are summed into a global percept [36]. Although the sum across electrodes is linear, modulating the size and eccentricity of phosphenes with stimulus parameters makes the final result a nonlinear function of stimulus parameters, preventing analytic inversion. Motivated by previous studies, we used a square  $15 \times 15$  array of  $150\mu \mathrm{m}$  electrodes, spaced  $400\mu \mathrm{m}$  apart [14]. In total, the model is parameterized by 13 patient specific parameters, shown in Table 1. The ranges for each parameter were chosen to conservatively encompass all observed patients, centered on the mean value across patients [8, 10, 11, 19, 37].
Deep Stimulus Inversion A deep stimulus encoder (DSE) is a deep neural network responsible for approximately inverting the forward model to produce the optimized stimulus for a target image and a specific patient  $(\mathbf{s}_{\phi} = \hat{f}^{-1}(\mathbf{t},\phi))$  [14]. We used a network (45M parameters) consisting of blocks, each containing 3 fully connected layers, batch normalization, and a residual connection. The architecture is illustrated in Appendix B.1. The flattened target image and the patient specific parameters were passed separately through one block each, concatenated, and passed through another block, after which the amplitude is predicted. The amplitudes were concatenated to the prior intermediate representation, fed through a final block, after which frequency and pulse duration were predicted. The output layers use ReLU activation; all others use leaky ReLU. During training,  $\phi$  were randomly sampled from the range of allowed parameters (Table 1). Tensorflow 2.12, an NVIDIA RTX 3090, Adam optimizer, and batch size of 256 [38, 39] were used to train the network.
Human-in-the-Loop Optimization We propose using preferential Bayesian optimization (PBO) to optimize the patient-specific parameters  $\phi$  of the pretrained DSE. Given two sets of patient-specific parameters,  $\phi_{1}$  and  $\phi_{2}$ , we assume that the probability of a subject preferring  $\phi_{1}$  to  $\phi_{2}$  (returning a response  $\phi_{1} \succ \phi_{2}$ ) depends on a preference function  $g(\phi)$ , modeled using a Gaussian process model:
$$
P \left(\phi_ {1} \succ \phi_ {2} | g\right) = \Phi \left(g \left(\phi_ {1}\right) - g \left(\phi_ {2}\right)\right), \tag {4}
$$
where  $\Phi$  is the normal cumulative distribution [40, 41]. The larger the value of  $g(\phi_1)$  relative to  $g(\phi_2)$ , the higher the likelihood that the subject reports preferring  $\phi_1$  over  $\phi_2$ .
We used the Maximally Uncertain Challenge [42] to select new comparisons to query, although other popular acquisitions performed similarly (Appendix C.3). Searching within the bounds in Table 1, this acquisition function selects a 'champion',  $\phi_1$ , which maximizes the expectation of  $g$ , and a 'challenger',  $\phi_2$ , for which subjects' preferences are most uncertain:
$$
\phi_ {1} \rightarrow \underset {\phi} {\arg \max } \mathbb {E} _ {p (g | \mathcal {D})} [ g (\phi) ], \tag {5}
$$
$$
\phi_ {2} \rightarrow \underset {\phi} {\arg \max } \mathbb {V} _ {p (g | \mathscr {D})} [ \Phi (g (\phi) - g (\phi_ {1})) ], \tag {6}
$$
where  $\mathbb{V}$  denotes the variance. This algorithm is designed to balance exploitation (values of  $\phi$  that maximize  $g$ ) and exploration (values of  $\phi$  for which the response is uncertain).
The performance of PBO crucially depends on the Gaussian process kernel and its hyperparameters, which encode our prior assumptions about the latent preference function. Inferring the kernel's hyperparameters online would slow down the algorithm and could lead to overfitting. Thus, we adopted a transfer learning strategy, which could also be applied to real-life patients (see Appendices C.1 and C.2 for full details and discussion of alternatives). In brief, for each of 10 patients (with parameters different from those used in the following PBO experiment), we simulated 600 random duels and fit candidate hyperparameters for each of 4 commonly used kernels. We then selected the kernel and hyperparameters that generalized best to the other 9 patients (measured using Brier score [43] on a held-out test set). The 5/2 Matérn kernel performed best, and was used for all subsequent experiments.
Simulated Patients In silico experiments on simulated patients were used to demonstrate the viability of our approach. Each patient was assigned a set of patient-specific parameters  $\phi$ , uniformly sampled from the ranges specified in Table 1. When challenged with a duel between two candidate stimuli  $\mathbf{s}_{\phi_1}$  and  $\mathbf{s}_{\phi_2}$ , the simulated patient ran each stimulus through the phosphene model (using ground-truth patient-specific parameters), obtaining the predicted percepts  $\hat{\mathbf{t}}_{\phi_1} = f(\mathbf{s}_{\phi_1};\phi)$  and  $\hat{\mathbf{t}}_{\phi_2} = f(\mathbf{s}_{\phi_2};\phi)$ . The users' preferences were modeled with a Bernoulli distribution, with probability  $p$  modulated by the difference in reconstruction error between each percept and the target image:
$$
p = \frac {1}{1 + \exp \left(- \frac {1}{\sigma} \left(d \left(\hat {\mathbf {t}} _ {\phi_ {2}}, \mathbf {t}\right) - d \left(\hat {\mathbf {t}} _ {\phi_ {1}}, \mathbf {t}\right)\right)\right)} \tag {7}
$$
Here,  $\sigma$  is a configurable parameter that scales the width of the sigmoid, introducing noise into the response. We set  $\sigma$  to be 0.01, chosen empirically based on a conservative estimate: when the error difference was greater than 0.01 it was obvious which percept was better to human observers.