Title: Human-in-the-Loop Optimization for Deep Stimulus Encoding in Visual Prostheses
Venue: NeurIPS 2023
Accepted: yes
Keywords: Brain Computer Interfaces

BCI

Stimulus Encoding

Visual Prostheses

Bayesian Optimization

Preferential Bayesian Optimization

Human-in-the-loop Optimization

Sensory Neuroprostheses

Neuroprostheses

Patient-Specific Optimization

Latent Space Bayesian Optimization
Abstract: Neuroprostheses show potential in restoring lost sensory function and enhancing human capabilities, but the sensations produced by current devices often seem unnatural or distorted. Exact placement of implants and differences in individual perception lead to significant variations in stimulus response, making personalized stimulus optimization a key challenge. Bayesian optimization could be used
to optimize patient-specific stimulation parameters with limited noisy observations, but is not feasible for high-dimensional stimuli. Alternatively, deep learning models can optimize stimulus encoding strategies, but typically assume perfect knowledge of patient-specific variations. Here we propose a novel, practically feasible approach that overcomes both of these fundamental limitations. First, a deep encoder network is trained to produce optimal stimuli for any individual patient by inverting a forward model mapping electrical stimuli to visual percepts. Second, a preferential Bayesian optimization strategy utilizes this encoder to learn the optimal patient-specific parameters for a new patient, using a minimal number of pairwise comparisons between candidate stimuli. We demonstrate the viability of this approach on a novel, state-of-the-art visual prosthesis model. Our approach quickly learns a personalized stimulus encoder and leads to dramatic improvements in the quality of restored vision, outperforming existing encoding strategies. Further, this approach is robust to noisy patient feedback and misspecifications in the underlying forward model. Overall, our results suggest that combining the strengths of deep learning and Bayesian optimization could significantly improve the perceptual experience of patients fitted with visual prostheses and may prove a viable solution for a range of neuroprosthetic technologies

Human-in-the-Loop Optimization for Deep Stimulus Encoding in Visual Prostheses

Abstract

Neuroprostheses show potential in restoring lost sensory function and enhancing human capabilities, but the sensations produced by current devices often seem unnatural or distorted. Exact placement of implants and differences in individual perception lead to significant variations in stimulus response, making personalized stimulus optimization a key challenge. Bayesian optimization could be used to optimize patient-specific stimulation parameters with limited noisy observations, but is not feasible for high-dimensional stimuli. Alternatively, deep learning models can optimize stimulus encoding strategies, but typically assume perfect knowledge of patient-specific variations. Here we propose a novel, practically feasible approach that overcomes both of these fundamental limitations. First, a deep encoder network is trained to produce optimal stimuli for any individual patient by inverting a forward model mapping electrical stimuli to visual percepts. Second, a preferential Bayesian optimization strategy utilizes this encoder to optimize patient-specific parameters for a new patient, using a minimal number of pairwise comparisons between candidate stimuli. We demonstrate the viability of this approach on a novel, state-of-the-art visual prosthesis model. We show that our approach quickly learns a personalized stimulus encoder, leads to dramatic improvements in the quality of restored vision, and is robust to noisy patient feedback and misspecifications in the underlying forward model. Overall, our results suggest that combining the strengths of deep learning and Bayesian optimization could significantly improve the perceptual experience of patients fitted with visual prostheses and may prove a viable solution for a range of neuroprosthetic technologies.

1 Introduction

Sensory neuroprostheses are devices designed to restore or enhance perception in individuals with sensory deficits. They often interface with the nervous system by electrically stimulating neural tissue in order to provide artificial sensory feedback to the user [1, 2]. For instance, visual prostheses have the potential to restore vision to people living with incurable blindness by bypassing damaged parts of the visual system and directly stimulating the remaining cells in order to evoke visual percepts (phosphenes) [3-6]. However, patient outcomes with current technologies are limited. Patients

Deep Stimulus Encoder (DSE)

Human-in-the-Loop Optimization (HILO)

Encoder

Encoder

I prefer...

3

Φ1

f

3

fixed

learned

Perceptual

Patient Φ

Error

Parameters

Stimulus

一

1

VPU

fixed

3

f

Acquisition

Gaussianprocess

fixed

function

model ofpreference

Decoder

A

Preferential BayesianOptimization

require extensive training to learn to interpret the evoked percepts, which are typically described as 'fundamentally different' from natural vision [7]. Moreover, phosphene appearance varies widely across patients [8], making personalized stimulus optimization a major outstanding challenge [9].

Much work has gone into developing computational models that can predict the neuronal or perceptual response to an electrical stimulus [8, 10, 11] (often called forward models). Once the forward model is known, a deep neural network can approximate its inverse, thereby identifying the required stimulus to elicit a desired percept [12-14]. However, these inverse models typically assume perfect knowledge of the patient-specific mapping from stimulation to perception (which is not practically feasible) and are heavily reliant on the forward model's accuracy over the entire stimulus space.

Alternatively, Bayesian optimization has been successful in personalizing stimulation strategies for many existing neural interfaces [15, 16]. However, this approach is limited in practice because it requires the stimulus dimension to be small (typically < 30 [17], which is orders of magnitudes smaller than the number of stimulus parameters in current implants), and optimization must be repeated for every stimulus. Moreover, visual prosthesis users can typically only give indirect feedback (e.g., verbal phosphene descriptions), unsuitable for traditional Bayesian optimization.

To address these challenges, we propose a novel framework that integrates deep learning-based stimulus inversion into a preferential Bayesian optimization strategy to learn a patient-specific stimulus encoder (Fig. 1). First, a deep stimulus encoder (DSE) is trained to optimize stimuli assuming perfect knowledge of a set of patient-specific parameters (Fig. 1, left ). Second, we embed the DSE within a human-in-the-loop optimization (HILO) strategy based on preferential Bayesian optimization, which iteratively learns the ground-truth patient-specific parameters through a series of 'duels', where the patient is repeatedly asked their preference between two candidate stimuli. The resulting DSE can then be deployed as a personalized stimulation strategy.

To this end, we make the following contributions:

We introduce a forward model for retinal implants that achieves state-of-the-art response predictions. Unlike previous models, this allows us to train a deep stimulus encoder to predict optimal stimuli across 13 dimensions of patient-specific parameters.

We propose a personalized stimulus optimization strategy for visual prostheses, where a humanin-the-loop optimization (HILO) Bayesian optimization algorithm iteratively learns the optimal patient-specific parameters for a deep stimulus encoder.

We demonstrate the viability of our approach by conducting a comprehensive series of evaluations on a population of simulated patients. We show HILO quickly learns a personalized stimulus encoder and leads to dramatic improvements in the quality of restored vision, outperforming existing encoding strategies. Importantly, HILO is resilient to noise in patient feedback and performs well even when the forward model is misspecified. Code for the forward model, DSE, and HILO algorithm is available at https://github.com/bionicvisionlab/2023-NeurIPS-HILO .

2 Background and Related Work

Visual Neuroprostheses Numerous groups worldwide are pursuing a visual prosthesis that stimulates viable neuronal tissue in the hope of restoring a rudimentary form of vision to people who are blind (Fig. 2, left ) [3-6]. Analogous to cochlear implants, these devices electrically stimulate surviving cells in the visual pathway to evoke visual percepts (phosphenes). Existing devices generally provide an improved ability to localize high-contrast objects and perform basic mobility tasks.

Much work has focused on characterizing phosphene appearance as a function of stimulus and neuroanatomical parameters [2, 10, 18-21]. In epiretinal implants, phosphenes often appear distorted due to inadvertent activation of nerve fiber bundles in the optic fiber layer of the retina [8], causing elongated percepts (Fig. 2, center ). In addition, the exact brightness and shape of these elicited percepts depends on the applied stimulus [19] and differs widely across patients (Fig. 2, right ). Granley et al. [11] captured these individual differences by including a set of patient-specific parameters in their phosphene model, denoted by ϕ , which includes both neuroanatomical (e.g., implant location) and stimulus-related parameters (e.g., how brightness scales with amplitude).

Deep Stimulus Encoding Many works attempt to mitigate distortions in prosthetic vision, but do not describe comprehensive stimulation strategies [22-24]. Those that describe strategies in detail typically require simplification [25] or strong assumptions [26] to be used in practice. Due to the complexities of optimization, deep learning-based stimulus encoders have risen in popularity [12-14]. In [12], authors proposed an innovative approach where the latent representations of an autoencoder are treated as stimuli and decoded with a phosphene model. However, they used an unrealistic binary phosphene model. Their approach has since been adapted for cortical models [27], and for non-differentiable forward models [13]. Granley et al. [14] generalized the approach, showing it could work with realistic forward models across a small range of patients without needing to retrain.

Given a forward (phosphene) model f (mapping stimuli to percepts given ϕ ), it is straightforward to show that the optimal stimulus encoder (mapping target images to stimuli) is the pseudoinverse of f [14]. However, to account for the wide range of individual differences in phosphene perception, most realistic forward models are highly nonlinear and not analytically invertible. Thus, previous works have proposed to use the forward model [12, 14] as a fixed decoder within a deep autoencoder trained to minimize the reconstruction error between target images and the predicted percepts. After training, the encoder can be extracted and used to encode target visual inputs in real time. Deep

Retinal implant

Opticnerve

SimulatedProstheticVision

ExamplePatientVariations

to brain

Receiver

Implant tacked

toretina

Camera

optic

disc

Area of

photoreceptors

array

destroyedby

disease

Photoreceptors

Tack

(rods and

cones)

Ganglion cells

stimulus encoders trained using this approach produce high quality stimuli, but assume knowledge of ϕ . Additionally, if the forward model f is not extremely accurate over the whole stimulus space, then the encoder network might learn to exploit inaccuracies in the model, producing stimuli that don't generalize to real patients [14]. We utilize an enhanced variant of this approach in our experiments.

Preferential Bayesian Optimization Preferential Bayesian optimization (PBO) is an efficient method for optimizing expensive black-box functions based on binary comparisons [28, 29]. Since the subject's response to stimulation cannot be directly observed, PBO instead builds a Bayesian model of the subject's preferences, g , typically modeled using a Gaussian process. An approximate inference algorithm (expectation propagation; [30, 31]) is used to infer the posterior distribution of the preference function given binary comparison data, p ( g | D ) , which is then used to select new configurations for the next trial according to an acquisition rule. The acquisition rule must balance the exploration-exploitation trade-off inherent to any black-box optimization problem [32].

PBO was previously used to tune BCI stimulation parameters for transcranial [33] and spinal cord stimulation [34]. However, these works directly optimized only a handful of stimulation parameters and cannot translate to visual prostheses, where complex and varying visual inputs have to be mapped to high-dimensional stimuli. To this end, Fauvel & Chalk [35] reduced optimization complexity by inverting a model of phosphene perception, then used PBO to generate encodings preferred by sighted subjects viewing simulated prosthetic vision. However, a linear approximation was used to invert the perception model, which is unrealistic for real-world applications.

Summary We identify 3 main limitations of previous work that this study aims to address:

Generalizability of deep stimulus encoders. Autoencoder-like deep stimulus encoders can accurately optimize stimuli, but require perfect knowledge of patient-specific parameters [14], which can be difficult or impossible to determine in practice [8, 11]. Further, these approaches heavily rely on the accuracy of the forward model [13, 14], while real patients will likely deviate from the forward model. We overcome this limitation by optimizing the learned stimulus encoder based on patients' preferences, which we show is not bounded by a misspecified forward model.

Applicability of Bayesian optimization. Bayesian optimization is ideally suited for optimizing stimulation parameters based on limited, noisy measurements, but can only optimize a small number of parameters [17]. We train a deep stimulus encoder to output optimal stimuli for any specific patient with known parameters, reducing the search space from the entire stimulus space (hundreds or thousands of parameters) to the low-dimensional space of patient-specific parameters (13 parameters with our model), enabling Bayesian optimization.

Simplistic models of perception. Most previous approaches use overly simplified forward models that do not match empirical data [8, 19]. More accurate models [11] are too computationally expensive to support deep stimulus optimization over a wide range of patients. We overcome this by developing a new phosphene model that matches patient data better than existing models, with significantly reduced memory and time complexity

3 Methods

General Framework We consider a system attempting to optimize stimuli for a new patient, specified by a set of (unknown) parameters ϕ . The goal of optimization is a patient-specific stimulus encoder mapping target perceptual responses t (e.g., visual percepts) to stimuli s .

We assume there exists a forward model f which predicts the patient's perceptual response to stimulation: ˆ t = f ( s ; ϕ ) . It follows that the optimal stimulus encoder is the (pseudo)inverse of f . The inverse can be approximated using an autoencoder-like deep neural network, with f as the fixed decoder and ˆ f -1 as the learned stimulus encoder [14]. The encoder's weights w are updated using gradient descent to minimize the reconstruction error, measured by some distance function d , between the target t and the predicted response ˆ t averaged across patients and a dataset of targets (Eq. 1):

Once trained, the encoder can accurately predict stimuli, but requires knowledge of the patient-specific parameters ϕ . For a new patient, Bayesian optimization is used to optimize ϕ based on user feedback, thereby learning a personalized DSE. Since the patient's response cannot be directly measured for visual prostheses, the user is presented with a 'duel', i.e. a binary comparison, where they are asked to decide which of two candidate stimuli they prefer [35]. A Gaussian process model of preferences is updated based on the patient's response, and an acquisition function is used to generate new candidate stimuli. The process can be repeated to iteratively tune the DSE to the patient's preferences.

Phosphene Model The phosphene model is a differentiable approximation of the underlying biological system (also called a forward model [14]), which maps an electrical stimulus to a visual percept. Although phosphene models exist for visual prostheses, current models either do not match patient data well [8, 10, 19], or are too computationally expensive [11].

Thus, we developed a new phosphene model for epiretinal prostheses (See Appendix A.1 for full details). The model takes in a stimulus vector s ∈ R n e × 3 specifying the frequency, amplitude, and pulse duration of a biphasic pulse train on each of n e electrodes. The output phosphene for each electrode is a Gaussian blob centered over the electrode's location µ e ( ϕ ) with covariance matrix Σ e ( s , ϕ ) constructed so that the resulting percept will have area ρ e ( s , ϕ ) , eccentricity λ e ( s , ϕ ) and orientation θ e ( ϕ ) . These functions allow phosphene properties to vary locally with stimulus (e.g., current spread) and anatomical parameters (e.g., electrode location, underlying axon nerve fiber bundle trajectory). The percept for an electrode located at µ e , is a multivariate Gaussian blob, renormalized to have maximum brightness b e ( s , ϕ ) :

where b e , µ e , and Σ e are implicitly parametrized by s and ϕ . The covariance matrix Σ e = RΣ 0 R T is calculated from the eigenvalue matrix Σ 0 and a rotation matrix R :

The eigenvalues α e and β e depend on the intended phosphene area ( ρ e ) and elongation ( λ e ) :

Blobs from individual electrodes are summed into a global percept [36]. Although the sum across electrodes is linear, modulating the size and eccentricity of phosphenes with stimulus parameters makes the final result a nonlinear function of stimulus parameters, preventing analytic inversion. Motivated by previous studies, we used a square 15 × 15 array of 150 µ melectrodes, spaced 400 µ m apart [14]. In total, the model is parameterized by 13 patient specific parameters, shown in Table 1. The ranges for each parameter were chosen to conservatively encompass all observed patients, centered on the mean value across patients [8, 10, 11, 19, 37].

Deep Stimulus Inversion A deep stimulus encoder (DSE) is a deep neural network responsible for approximately inverting the forward model to produce the optimized stimulus for a target image and a specific patient ( s ϕ = ˆ f -1 ( t , ϕ ) ) [14]. We used a network (45M parameters) consisting of blocks, each containing 3 fully connected layers, batch normalization, and a residual connection. The architecture is illustrated in Appendix B.1. The flattened target image and the patient specific parameters were passed separately through one block each, concatenated, and passed through another block, after which the amplitude is predicted. The amplitudes were concatenated to the prior intermediate representation, fed through a final block, after which frequency and pulse duration were predicted. The output layers use ReLU activation; all others use leaky ReLU. During training, ϕ were randomly sampled from the range of allowed parameters (Table 1). Tensorflow 2.12, an NVIDIA RTX 3090, Adam optimizer, and batch size of 256 [38, 39] were used to train the network.

Human-in-the-Loop Optimization We propose using preferential Bayesian optimization (PBO) to optimize the patient-specific parameters ϕ of the pretrained DSE. Given two sets of patient-specific parameters, ϕ 1 and ϕ 2 , we assume that the probability of a subject preferring ϕ 1 to ϕ 2 (returning a response ϕ 1 ≻ ϕ 2 ) depends on a preference function g ( ϕ ) , modeled using a Gaussian process model:

where Φ is the normal cumulative distribution [40, 41]. The larger the value of g ( ϕ 1 ) relative to g ( ϕ 2 ) , the higher the likelihood that the subject reports preferring ϕ 1 over ϕ 2 .

We used the Maximally Uncertain Challenge [42] to select new comparisons to query, although other popular acquisitions performed similarly (Appendix C.3). Searching within the bounds in Table 1, this acquisition function selects a 'champion', ϕ 1 , which maximizes the expectation of g , and a 'challenger', ϕ 2 , for which subjects' preferences are most uncertain:

where V denotes the variance. This algorithm is designed to balance exploitation (values of ϕ that maximize g ) and exploration (values of ϕ for which the response is uncertain).

The performance of PBO crucially depends on the Gaussian process kernel and its hyperparameters, which encode our prior assumptions about the latent preference function. Inferring the kernel's hyperparameters online would slow down the algorithm and could lead to overfitting. Thus, we adopted a transfer learning strategy, which could also be applied to real-life patients (see Appendices C.1 and C.2 for full details and discussion of alternatives). In brief, for each of 10 patients (with parameters different from those used in the following PBO experiment), we simulated 600 random duels and fit candidate hyperparameters for each of 4 commonly used kernels. We then selected the kernel and hyperparameters that generalized best to the other 9 patients (measured using Brier score [43] on a held-out test set). The 5/2 Matérn kernel performed best, and was used for all subsequent experiments.

Simulated Patients In silico experiments on simulated patients were used to demonstrate the viability of our approach. Each patient was assigned a set of patient-specific parameters ϕ , uniformly sampled from the ranges specified in Table 1. When challenged with a duel between two candidate stimuli s ϕ 1 and s ϕ 2 , the simulated patient ran each stimulus through the phosphene model (using ground-truth patient-specific parameters), obtaining the predicted percepts ˆ t ϕ 1 = f ( s ϕ 1 ; ϕ ) and ˆ t ϕ 2 = f ( s ϕ 2 ; ϕ ) . The users' preferences were modeled with a Bernoulli distribution, with probability p modulated by the difference in reconstruction error between each percept and the target image:

Here, σ is a configurable parameter that scales the width of the sigmoid, introducing noise into the response. We set σ to be 0 . 01 , chosen empirically based on a conservative estimate: when the error difference was greater than 0.01 it was obvious which percept was better to human observers.

Data and Metrics We used MNIST images as target visual percepts throughout the experiments. Images were resized to be the same size as the output of f ( 49 × 49 pixels), and scaled to have a maximum brightness of 2 (aligned with range( f )). Inspired by [12, 14], we used a perceptual similarity metric designed to capture higher-level differences between images [44]. Let v l ( t ) be a function that extracts the downstream representations of target t input to a VGG19 network pretrained on ImageNet [45]. The perceptual metric is then given by equation 8.

This metric was used by the deep stimulus encoder as a training objective, by the simulated patient to choose a duel winner, and throughout HILO as an evaluation metric. β = 2 . 5 e-5 was selected via cross-validation (see [14] App. B). To aid in interpretability, we also report a secondary metric based on how identifiable the predicted percepts were. We first pretrained a separate deep net to 99% test accuracy on MNIST classification. We then measured the accuracy of this classifier on the predicted percepts at every iteration of HILO.

Target

Target

7

Naive

DSE

Naive

DSE

7

7

7

7

777

7

7

7

7

7

7

77

7

7

727

4 Results

4.1 Phosphene Model

To verify that our phosphene model's predictions line up with observed results from real prosthesis users, we repeated analyses from previous state-of-the-art models, evaluating how phosphene appearance changes with electrode location [8] and stimulus parameters [10, 11, 19, 37]. We used the same datasets, consisting of thousands of phosphene drawings and brightness and size ratings collected across multiple epiretinal prosthesis [3] patients over several years. To evaluate phosphene appearances with electrode location, we calculated the correlation between predicted and observed phosphenes ( R i ) for three shape descriptors: area, eccentricity, and orientation. The final score reported is 1 -∑ i R 2 i [8]. To evaluate how phosphene appearance was modulated by stimulus parameters, we calculated the mean squared error between the size and brightness of predicted percepts and patient ratings as amplitude, frequency, and pulse duration were varied. The reported values correspond to Figures 4a-c and 5 in [11].

Evaluation results are presented in Table 2. Our model significantly outperforms all baselinse on the Beyeler et al. evaluation, and matches SOTA on the Granley et al. evaluations. Additionally, our model is on average 45x faster, and uses 120x less memory than [11]. A detailed description of evaluation methods and additional analysis can be found in Appendix A.2.

4.2 Deep Stimulus Encoder

We trained a deep stimulus encoder (DSE) to invert our phosphene model (decoder). The encoder was trained across 13 patient-specific parameters, randomly sampled at every epoch, including for the first time implant position and rotation. This is in contrast to previous DSEs, which either require retraining for every new patient [12, 13, 27], or can only vary two patient-specific parameters [14].

We compared the performance of the DSE to a traditional ('naive') encoder [14] currently used by retinal prostheses [3], illustrated in Fig. 3. The DSE achieved a test perceptual loss of 0.05 and a MNIST accuracy of 95.6%, significantly outperforming the naive encoder (5.68 and 51% respectively). Note that this performance is when the true patient-specific parameters are known.

A

Early Duel

Target

Φ

Target

1

I prefer...

I prefer...

VPU

VPU

B

C

10

100%

DSE-Φ

Naive

MNIST Accuracy

75%

Error

1

DSE-Φ

Guess

50%

Naive

0.1

25%

DSE-Φ

True

0

50

100

150

0

50

100

150

Number of Duels

Number of Duels

D

HILO

DSE-Φ

Guess

Naive

This performance was slightly better than the values reported in [14] (L1 loss of .108 vs .12 in [14]) despite training across 11 additional patient-specific parameters.

4.3 Human-in-the-Loop Optimization

We ran deep learning-based HILO for 100 randomly sampled simulated patients. After every duel, we evaluated the DSE parameterized by the current prediction of patient-specific parameters on a subset of the MNIST test set. The performance of the learned encoder over time ('HILO') is illustrated in Figure 4, which plots the joint perceptual loss (Figure 4.B) and MNIST accuracy (Fig. 4C).

As baselines for comparison we used a naive encoder, a non-personalized DSE where the patientspecific parameters are guessed (DSEϕ Guess ), and an ideal DSE using the true ϕ (DSEϕ True ). To guess ϕ , we consider two approaches, one which selects the midpoint from the ranges in Table 1, and another where performance was averaged across random selections of ϕ from the same ranges. In real patients, we estimate that the performance of a deep stimulus encoder without patient-specific optimization would likely fall somewhere between these two methods, since the distribution of real patients might deviate slightly from Table 1. We therefore plot the region bounded by the performance of a DSE with either of these approaches for guessing ϕ . Example percepts after optimization are shown in Fig. 4D.

The HILO encoder starts with random predictions, but, after a short initial exploration period, quickly surpassed the baselines. After about 75 iterations, performance approached the ideal DSE encoder, however the HILO encoder still resulted in high-quality percepts after as few as 20 iterations. Averaged across patients, the final reconstruction error of the HILO encoder was .071 ± .0031 (SEM)

Noise Level

B

— Thresholds 300% Off -- Original

0%26%—38%—48%—66%

DSE-ΦGuess

DSE-Φ

Guess

DSE-Φ

True

0.11

Error

DSE-

True

C

D

4

—Axon Trajectories Off -- Original

— Out of Distribution-- Original

DSE-ΦGuess

DSE-Φ

Guess

DSE-

True'

0.1

DSE-Φ

True

25

50

75

100

1

25

50

75

100

Number of Duels

and MNIST accuracy was 92% ± 1.0%. DSEϕ Guess had an error of between .25 and 1.1 and MNIST accuracy between 58.6% and 78.3%, and the DSE with true ϕ had an error of .05 ± .001 and accuracy of 95.5% ± .1%.

4.4 Robustness

In reality, it is likely that a patient's perceptions will not be perfectly captured by the phosphene model. Further, patient responses for visual prostheses are notoriously noisy [7, 46]. To test HILO's resiliency to these variations, we conducted additional robustness experiments, each with the same 25 simulated patients (Figure 5). First, we varied the noise parameter σ in simulated patients' decision making (Figure 5A). Next we constructed various 'misspecified' forward models, where the ground-truth model used to decode stimuli differed from the forward model assumed by the DSE. First, we varied the trajectories of the simulated axon bundles [47], thereby changing the orientation of phosphenes (Figure 5C). Second, threshold amplitudes for stimulation are notoriously hard to predict, and have been shown to drift by up to 300% over time [48]. Therefore, we tested a variant where the threshold assumed by the encoder was incorrect by up to 300% (Figure 5B). Lastly, we used the same forward model, but with patient-specific parameters outside the ranges in 1 (Figure 5D).

At σ =1e-4, the patient response was noiseless. For σ equal to .005, .01, and .02, HILO performed similarly to the noiseless model, despite the patient on average making 'random' ( p ∈ [0 . 35 , 0 . 65] ) decisions in 26%, 38%, and 48% of duels. At σ =0.05, the decision was 'random' 2/3 of the time, and HILO performed similarly or slightly better than the baseline DSEϕ Guess . The DSE itself is very resilient to misspecifications in axon trajectory, so HILO performs similarly for this misspecification to the original patients. When thresholds varied, HILO still outperformed the baselines, but converged to slightly worse encodings than without misspecification. Further, HILO surpassed the DSE encoded with the ground-truth ϕ . This demonstrates that even when the forward model assumed by the DSE is incorrect, HILO still tends to converge to a set of patient-specific parameters that work well for the

misspecified patient. For out-of-distribution ϕ , HILO again outperformed both the baseline and true DSEs, but performed worse than in-distribution patients.

5 Discussion

Our experiments show that HILO optimization of a deep stimulus encoder led to high-quality, personalized stimulation strategies that outperformed previous state-of-the-art techniques. HILO led to an increase in percept quality compared to using a non-personalized DSE for 99 out of 100 simulated patients, demonstrating the viability of our approach. To enable our HILO algorithm, we also developed a new phosphene model, which is computationally simpler and matches patient data better than previous models, and trained a new DSE, which is able to produce high-quality encodings across all 13 dimensions of patient-specific variations included in our phosphene model. Together, these significantly advance state-of-the-art in patient-specific stimulus encoding, and are important steps towards practically-feasible personalized prosthetic vision in real patients.

The proposed framework combining Bayesian optimization and deep stimulus encoding offers significant improvements over both components in isolation. Use of a DSE allows us to incorporate prior information, reducing the dimensionality of the Bayesian optimization search space from the large stimulus space to the much smaller model parameter space. Our results demonstrate that even when the DSE's predictions are incorrect, this parameterization is still useful for Bayesian optimization based on patient preferences. Additionally, DSEs are able to invert highly nonlinear forward models, enabling encoder-parameterized Bayesian optimization to be applied to a much larger set of problems. Lastly, the learned encoder can be applied for any target percept, without needing additional optimization. On the other hand, without adaptive feedback from HILO, deep stimulus encoders have no method for learning the individual differences of a new patient, which we show leads to suboptimal stimuli. DSEs rely on the accuracy of their assumed forward model over the entire stimulus space. We show that our approach produces stimuli that work well for the patient, even when the forward model is misspecified, or when the patient's responses are noisy.

This approach is practical for stimulus optimization in the wild. The encoder learned during optimization is lightweight, and once deployed, can predict individual stimuli in less than 5 ms on CPU, allowing for high frame rates for prosthetic stimulation. During HILO, updating the Gaussian process model and producing new stimuli on average took 3 seconds, meaning that stimulus optimization could be performed in a matter of minutes. A HILO strategy could be bundled with future visual prostheses, allowing for patients to periodically re-calibrate their devices when they feel the device is not performing adequately, without requiring a clinical professional.

Broader Impacts Although we demonstrate this approach in the context of visual prostheses, our framework is general and could be applied to a variety of sensory devices. Our approach is applicable when the stimulus search space is large and there exists a forward model mapping stimuli to responses. Forward models [49-52] and deep stimulus encoders [53-55] have been successfully used across multiple sensory modalities, and could potentially be adapted for personalization with HILO.

Limitations Although promising, our approach is not without limitations. We assumed that the preference of patients for different stimuli is related to the distance metric used to measure perceptual similarity, which may not be true in practice. However, results by [35] suggest that PBO is robust to a mismatch between the distance metric used to invert the forward model and the preference of patients. Another limitation is that evaluation of our approach was only performed on simulated patients with a simulated perceptual model. However, this is mitigated by the fact that HILO showed robustness to model inaccuracies. Still, since it is difficult to predict the behavior of deep learning models, using a deep stimulus encoder in real patients could raise safety concerns. It may be possible for a deep encoder to produce unconventional stimuli, potentially leading to adverse effects. However, most devices come with firmware responsible for ensuring stimuli stay within FDA-approved safety limits.

In conclusion, our results suggest that combining the strengths of deep learning and Bayesian optimization could significantly improve the perceptual experience of patients fitted with visual prostheses and may prove a viable solution for a range of neuroprosthetic technologies.

References

Caterina Cinel, Davide Valeriani, and Riccardo Poli. Neurotechnologies for Human Cognitive Augmentation: Current State of the Art and Future Prospects. Frontiers in Human Neuroscience , 13:13, January 2019.

Eduardo Fernandez. Development of visual Neuroprostheses: trends and challenges. Bioelectronic Medicine , 4(1):12, December 2018.

Yvonne Hsu-Lin Luo and Lyndon da Cruz. The Argus® II Retinal Prosthesis System. Progress in Retinal and Eye Research , 50:89-107, January 2016.

Katarina Stingl, Ruth Schippert, Karl U. Bartz-Schmidt, Dorothea Besch, Charles L. Cottriall, Thomas L. Edwards, Florian Gekeler, Udo Greppmaier, Katja Kiel, Assen Koitschev, Laura Kühlewein, Robert E. MacLaren, James D. Ramsden, Johann Roider, Albrecht Rothermel, Helmut Sachs, Greta S. Schröder, Jan Tode, Nicole Troelenberg, and Eberhart Zrenner. Interim Results of a Multicenter Trial with the New Electronic Subretinal Implant Alpha AMS in 15 Patients Blind from Inherited Retinal Degenerations. Frontiers in Neuroscience , 11, 2017. Publisher: Frontiers.

D. Palanker, Y. Le Mer, S. Mohand-Said, and J. A. Sahel. Simultaneous perception of prosthetic and natural vision in AMD patients. Nature Communications , 13(1):513, December 2022.

Eduardo Fernández, Arantxa Alfaro, Cristina Soto-Sánchez, Pablo Gonzalez-Lopez, Antonio M. Lozano, Sebastian Peña, Maria Dolores Grima, Alfonso Rodil, Bernardeta Gómez, Xing Chen, Pieter R. Roelfsema, John D. Rolston, Tyler S. Davis, and Richard A. Normann. Visual percepts evoked with an intracortical 96-channel microelectrode array inserted in human occipital cortex. The Journal of Clinical Investigation , 131(23), December 2021. Publisher: American Society for Clinical Investigation.

Cordelia Erickson-Davis and Helma Korzybska. What do blind people 'see' with retinal prostheses? Observations and qualitative reports of epiretinal implant users. preprint, Neuroscience, February 2020.

Michael Beyeler, Devyani Nanduri, James D. Weiland, Ariel Rokem, Geoffrey M. Boynton, and Ione Fine. A model of ganglion axon pathways accounts for percepts elicited by retinal implants. Scientific Reports , 9(1):1-16, June 2019.

Fabrizio Grani, Cristina Soto-Sánchez, Antonio Fimia, and Eduardo Fernández. Toward a personalized closed-loop stimulation of the visual cortex: Advances and challenges. Frontiers in Cellular Neuroscience , 16, 2022.

Alan Horsager, Scott H. Greenwald, James D. Weiland, Mark S. Humayun, Robert J. Greenberg, Matthew J. McMahon, Geoffrey M. Boynton, and Ione Fine. Predicting Visual Sensitivity in Retinal Prosthesis Patients. Investigative Opthalmology & Visual Science , 50(4):1483, April 2009.

Jacob Granley and Michael Beyeler. A Computational Model of Phosphene Appearance for Epiretinal Prostheses. In 2021 43rd Annual International Conference of the IEEE Engineering in Medicine Biology Society (EMBC) , pages 4477-4481, November 2021. ISSN: 2694-0604.

Jaap de Ruyter van Steveninck, Umut Güçlü, Richard van Wezel, and Marcel van Gerven. End-to-end optimization of prosthetic vision. Journal of Vision , 22(2):20, February 2022.

Lucas Relic, Bowen Zhang, Yi-Lin Tuan, and Michael Beyeler. Deep Learning&#x2013;Based Perceptual Stimulus Encoder for Bionic Vision. In Augmented Humans 2022 , AHs 2022, pages 323-325, New York, NY, USA, March 2022. Association for Computing Machinery.

Jacob Granley, Lucas Relic, and Michael Beyeler. Hybrid Neural Autoencoders for Stimulus Encoding in Visual and Other Sensory Neuroprostheses. In Advances in Neural Information Processing Systems , October 2022.

Sumedh Sopan Nagrale, Ali Yousefi, Theoden Netoff, and Alik S Widge. In silico development and validation of Bayesian methods for optimizing deep brain stimulation to enhance cognitive control. Journal of Neural Engineering , April 2023.

Ghazaleh Soleimani, Michael A. Nitsche, Til Ole Bergmann, Farzad Towhidkhah, Ines Violante, Romy Lorenz, Rayus Kuplicki, Aki Tsuchiyagaito, Beni Mulyana, Ahmad Mayeli, Peyman Ghobadi-Azbari, Mohsen Mosayebi Samani, Anna Zilverstand, Martin P Paulus, Marom Bikson, and Hamed Ekhtiari. Closing the loop between brain and electrical stimulation: Towards precision neuromodulation treatments. preprint, PsyArXiv, July 2022.

Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P. Adams, and Nando de Freitas. Taking the Human Out of the Loop: A Review of Bayesian Optimization. Proceedings of the IEEE , 104(1):148-175, January 2016.

Alan Horsager, Robert J. Greenberg, and Ione Fine. Spatiotemporal Interactions in Retinal Prosthesis Subjects. Investigative Opthalmology & Visual Science , 51(2):1223, February 2010.

Devyani Nanduri, Ione Fine, Alan Horsager, Geoffrey M. Boynton, Mark S. Humayun, Robert J. Greenberg, and James D. Weiland. Frequency and Amplitude Modulation Have Different Effects on the Percepts Elicited by Retinal Stimulation. Investigative Ophthalmology & Visual Science , 53(1):205-214, January 2012.

Ione Fine and Geoffrey M. Boynton. Pulse trains to percepts: the challenge of creating a perceptually intelligible world with sight recovery technologies. Philosophical Transactions of the Royal Society B: Biological Sciences , 370(1677):20140208, September 2015.

Jacob Granley, Alexander Riedel, and Michael Beyeler. Adapting brain-like neural networks for modeling cortical visual prostheses. arXiv preprint arXiv:2209.13561 , 2022.

Dorsa Haji Ghaffari, Akwasi Darkwah Akwaboah, Ehsan Mirzakhalili, and James D. Weiland. Real-Time Optimization of Retinal Ganglion Cell Spatial Activity in Response to Epiretinal Stimulation. IEEE Transactions on Neural Systems and Rehabilitation Engineering , 29:2733-2741, 2021.

Nishal P. Shah, Sasidhar Madugula, Lauren Grosberg, Gonzalo Mena, Pulkit Tandon, Pawel Hottowy, Alexander Sher, Alan Litke, Subhasish Mitra, and E.J. Chichilnisky. Optimization of Electrical Stimulation for a High-Fidelity Artificial Retina. In 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER) , pages 714-718, San Francisco, CA, USA, March 2019. IEEE.

Sasidhar S. Madugula, Alex R. Gogliettino, Moosa Zaidi, Gorish Aggarwal, Alexandra Kling, Nishal P. Shah, Ramandeep Vilkhu, Madeline R. Hays, Huy Nguyen, Victoria Fan, Eric G. Wu, Pawel Hottowy, Alexander Sher, Alan M. Litke, Ruwan A. Silva, and E.J. Chichilnisky. Focal Electrical Stimulation of Human Retinal Ganglion Cells for Vision Restoration. preprint, Neuroscience, August 2020.

Martin J. Spencer, Tatiana Kameneva, David B. Grayden, Hamish Meffin, and Anthony N. Burkitt. Global activity shaping strategies for a retinal implant. Journal of Neural Engineering , 16(2):026008, January 2019. Publisher: IOP Publishing.

Nishal P. Shah, Sasi Madugula, A.J. Phillips, Lauren Grosberg, Alex R. Gogliettino, Aditya Dusi, Pulkit Tandon, Jeff Brown, Pawel Hottowy, Wladyslaw Dabrowski, Alexander Sher, Alan M. Litke, Subhasish Mitra, and E.J. Chichilnisky. Precise control of neural activity using temporally dithered and spatially multiplexed electrical stimulation. preprint, Neuroscience, July 2022.

Maureen van der Grinten, Jaap de Ruyter van Steveninck, Antonio Lozano, Laura Pijnacker, Bodo Rückauer, Pieter Roelfsema, Marcel van Gerven, Richard van Wezel, Umut Güçlü, and Ya˘ gmur Güçlütürk. Biologically plausible phosphene simulation for the differentiable optimization of visual cortical prostheses, December 2022. Pages: 2022.12.23.521749 Section: New Results.

Eric Brochu, Tyson Brochu, and Nando De Freitas. A Bayesian Interactive Optimization Approach to Procedural Animation Design. Eurographics/ ACM SIGGRAPH Symposium on Computer Animation , page 10 pages, 2010. Artwork Size: 10 pages ISBN: 9783905674279 Publisher: The Eurographics Association.

Javier Gonzalez, Zhenwen Dai, Andreas Damianou, and Neil D. Lawrence. Preferential Bayesian Optimization, April 2017. arXiv:1704.03651 [stat].

Thomas P. Minka. Expectation Propagation for approximate Bayesian inference. 2013. Publisher: arXiv Version Number: 1.

Matthias Seeger. Expectation Propagation for Exponential Families. 2008.

Roman Garnett. Bayesian optimization . Cambridge University Press, Cambridge, United Kingdom ; New York, NY, 2023.

Romy Lorenz, Laura E. Simmons, Ricardo P. Monti, Joy L. Arthur, Severin Limal, Ilkka Laakso, Robert Leech, and Ines R. Violante. Efficiently searching through large tACS parameter spaces using closed-loop Bayesian optimization. Brain Stimulation , 12(6):1484-1489, November 2019.

Zixi Zhao, Aliya Ahmadi, Caleb Hoover, Logan Grado, Nicholas Peterson, Xinran Wang, David Freeman, Thomas Murray, Andrew Lamperski, David Darrow, and Theoden I. Netoff. Optimization of Spinal Cord Stimulation Using Bayesian Preference Learning and Its Validation. IEEE Transactions on Neural Systems and Rehabilitation Engineering , 29:1987-1997, 2021.

Tristan Fauvel and Matthew Chalk. Human-in-the-loop optimization of visual prosthetic stimulation. Journal of Neural Engineering , 19(3):036038, June 2022. Publisher: IOP Publishing.

Yuchen Hou, Devyani Nanduri, Jacob Granley, James D Weiland, and Michael Beyeler. Axonal stimulation affects the linear summation of single-point perception in three argus ii users. medRxiv , pages 2023-07, 2023.

Scott H. Greenwald, Alan Horsager, Mark S. Humayun, Robert J. Greenberg, Matthew J. McMahon, and Ione Fine. Brightness as a Function of Current Amplitude in Human Retinal Electrical Stimulation. Investigative Opthalmology & Visual Science , 50(11):5017, November 2009.

Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mane, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viegas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems. 2016. Publisher: arXiv Version Number: 2.

Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. 2014. Publisher: arXiv Version Number: 9.

Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian Active Learning for Classification and Preference Learning, December 2011. arXiv:1112.5745 [cs, stat].

Wei Chu and Zoubin Ghahramani. Preference learning with Gaussian processes. In Proceedings of the 22nd international conference on Machine learning - ICML '05 , pages 137-144, Bonn, Germany, 2005. ACM Press.

Tristan Fauvel and Matthew Chalk. Efficient Exploration in Binary and Preferential Bayesian Optimization, October 2021. arXiv:2110.09361 [cs, math, stat].

Glenn W Brier. Verification of forecasts expressed in terms of probability. Monthly weather review , 78(1):1-3, 1950.

Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Ming-Hsuan Yang. Universal Style Transfer via Feature Transforms. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition , pages 248-255, Miami, FL, June 2009. IEEE.

Nick Barnes, Adele F Scott, Paulette Lieby, Matthew A Petoe, Chris McCarthy, Ashley Stacey, Lauren N Ayton, Nicholas C Sinclair, Mohit N Shivdasani, Nigel H Lovell, Hugh J McDermott, Janine G Walker, and for the BVA Consortium. Vision function testing for a suprachoroidal retinal prosthesis: effects of image filtering. Journal of Neural Engineering , 13(3):036013, June 2016.

N.M. Jansonius, J. Nevalainen, B. Selig, L.M. Zangwill, P.A. Sample, W.M. Budde, J.B. Jonas, W.A. Lagrèze, P.J. Airaksinen, R. Vonthein, L.A. Levin, J. Paetzold, and U. Schiefer. A mathematical description of nerve fiber bundle trajectories and their variability in the human retina. Vision Research , 49(17):21572163, August 2009.

Lan Yue, Paulo Falabella, Punita Christopher, Varalakshmi Wuyyuru, Jessy Dorn, Paulo Schor, Robert J. Greenberg, James D. Weiland, and Mark S. Humayun. Ten-Year Follow-up of a Blind Patient Chronically Implanted with Epiretinal Prosthesis Argus I. Ophthalmology , 122(12):2545-2552.e1, December 2015.

Milana P. Mileusnic, Ian E. Brown, Ning Lan, and Gerald E. Loeb. Mathematical Models of Proprioceptors. I. Control and Transduction in the Muscle Spindle. Journal of Neurophysiology , 96(4):1772-1788, October 2006. Publisher: American Physiological Society.

Elizaveta V. Okorokova, Qinpu He, and Sliman J. Bensmaia. Biomimetic encoding model for restoring touch in bionic hands through a nerve interface. Journal of Neural Engineering , 15(6):066033, October 2018. Publisher: IOP Publishing.

Hannes P. Saal, Benoit P. Delhaye, Brandon C. Rayhaun, and Sliman J. Bensmaia. Simulating tactile signals from the whole hand with millisecond precision. Proceedings of the National Academy of Sciences , 114(28), July 2017.

Michael F. Dorman, Anthony J. Spahr, Philipos C. Loizou, Cindy J. Dana, and Jennifer S. Schmidt. Acoustic Simulations of Combined Electric and Acoustic Hearing (EAS). Ear and Hearing , 26(4):371, August 2005.

Fotios Drakopoulos and Sarah Verhulst. A Neural-Network Framework for the Design of Individualised Hearing-Loss Compensation. IEEE/ACM Transactions on Audio, Speech, and Language Processing , pages 1-15, 2023. Conference Name: IEEE/ACM Transactions on Audio, Speech, and Language Processing.

Fotios Drakopoulos and Sarah Verhulst. A Differentiable Optimisation Framework for The Design of Individualised DNN-based Hearing-Aid Strategies. In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 351-355, May 2022. ISSN: 2379-190X.

Fotios Drakopoulos, Arthur Van Den Broucke, and Sarah Verhulst. A DNN-Based Hearing-Aid Strategy For Real-Time Processing: One Size Fits All. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 1-5, June 2023.

Paola Vagni, Marta Jole Ildelfonsa Airaghi Leccardi, Charles-Henri Vila, Elodie Geneviève Zollinger, Golnaz Sherafatipour, Thomas J. Wolfensberger, and Diego Ghezzi. POLYRETINA restores light responses in vivo in blind Göttingen minipigs. Nature Communications , 13(1):3678, June 2022. Number: 1 Publisher: Nature Publishing Group.

M. Beyeler, G. M. Boynton, I. Fine, and A. Rokem. pulse2percept: A Python-based simulation framework for bionic vision. In K. Huff, D. Lippa, D. Niederhut, and M. Pacer, editors, Proceedings of the 16th Science in Python Conference , pages 81-88, 2017.

Andrew C. Weitz, Devyani Nanduri, Matthew R. Behrend, Alejandra Gonzalez-Calle, Robert J. Greenberg, Mark S. Humayun, Robert H. Chow, and James D. Weiland. Improving the spatial resolution of epiretinal implants by increasing stimulus pulse duration. Science Translational Medicine , 7(318), December 2015.

Andrew B. Watson. A formula for human retinal ganglion cell receptive field density as a function of visual field location. Journal of Vision , 14(7):15, June 2014.

Matthias Seeger. Gaussian processes for machine learning. International Journal of Neural Systems , 14(02):69-106, April 2004. Publisher: World Scientific Publishing Co.

Kenneth H. Louie, Matthew N. Petrucci, Logan L. Grado, Chiahao Lu, Paul J. Tuite, Andrew G. Lamperski, Colum D. MacKinnon, Scott E. Cooper, and Theoden I. Netoff. Semi-automated approaches to optimize deep brain stimulation parameters in Parkinson's disease. Journal of NeuroEngineering and Rehabilitation , 18(1):83, December 2021.

Maegan Tucker, Noel Csomay-Shanklin, Wen-Loong Ma, and Aaron D. Ames. Preference-Based Learning for User-Guided HZD Gait Generation on Bipedal Walking Robots, March 2021. arXiv:2011.05424 [cs].

Jens Brehm Bagger Nielsen, Jakob Nielsen, and Jan Larsen. Perception-Based Personalization of Hearing Aids Using Gaussian Processes and Active Learning. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 23(1):162-173, January 2015. Conference Name: IEEE/ACM Transactions on Audio, Speech, and Language Processing.

Alessio Benavoli, Dario Azzimonti, and Dario Piga. Preferential Bayesian optimisation with skew gaussian processes. In Proceedings of the Genetic and Evolutionary Computation Conference Companion , GECCO '21, pages 1842-1850, New York, NY, USA, July 2021. Association for Computing Machinery.

Yanan Sui, Vincent Zhuang, Joel W Burdick, and Yisong Yue. Multi-dueling bandits with dependent arms. arXiv preprint arXiv:1705.00253 , 2017.

Galen Pogoncheff, Zuying Hu, Ariel Rokem, and Michael Beyeler. Explainable Machine Learning Predictions of Perceptual Sensitivity for Retinal Prostheses, February 2023. Pages: 2023.02.09.23285633.

Appendix

A Phosphene Model

A.1 Methods

This section describes the phosphene model used to simulate patient's perception resulting from stimulation. The model takes in a stimulus vector s ∈ R n e × 3 specifying the frequency ( freq ), amplitude ( amp ), and pulse duration ( pdur ) of a biphasic pulse train on each electrode. In addition, the model also takes in a vector of patient-specific parameters ϕ (see Table 1). We break these parameters down into implant parameters ( x , y , rot ), global parameters ( ρ , λ , ω , OD x , OD y ), and stimulus-related parameters ( a 0 -a 4 ); all explained below.

Exact implant locations vary patient-to-patient. The three implant parameters allow our model to account for these changes. We used a simulated implant inspired by designs of real epiretinal implants [3, 56] and those used in previous simulation studies [14]. It consists of 225 disk electrodes (radius 75 µ m arranged onto a square, 15 × 15 grid with 400 µ m spacing, initially centered over the fovea. The three implant-related parameters translate and rotate the initial implant to be centered at ( x , y ), and to be rotated by rot degrees. The implant used is depicted in Figure A.2, overlaid on top of a simulated map of axon nerve fiber bundles [57].

Original Implant

ExampleRotationand Offset

4000

(microns)

>

-2000-

-4000

2000

0

nRoral

tem

boooooooooooooouadns

ooooooooooooo

6

6

a

o

Q

6

9

temporal

superior

o

a

o

o

a

Q

nasa

Q

O

6

o

6

Q

9

a

a

o

O

6

6

Q

。

Q

6

o

o

9

a

a

。

6

α

o

Q

6

Q

O

o

。

0

a

6

Q

0

a

O

Q

Q

。

_inferior

_ inferior

-4000

-2000

0

2000

4000

-4000

-2000

0

2000

4000

x(microns)

x(microns)

The remaining model parameters are inspired by various psychophysical and electrophysiological studies [8, 10, 18, 19, 58], and are summarized in the following list:

ρ : Average phosphene size. This will be modified locally based on stimulus parameters.

λ : Average phosphene eccentricity (a measure of phosphene elongation; not to be confused with retinal eccentricity). This will be modified locally based on stimulus parameters.

ω : Orientation scaling factor. The orientation of phosphenes will be the orientation of the underlying axon bundle, scaled by ω (Eq. 12).

OD x , OD y : The x and y location of the patient's optic disc, into which axon nerve fiber bundles terminate.

a 0 -a 2 : Coefficients to modulate phosphene brightness with stimulus parameters (Eq. 9).

a 3 : Coefficient to modulate phosphene size with stimulus parameters (Eq. 10).

a 4 : Coefficient to modulate phosphene eccentricity with stimulus parameters (Eq. 11).

Each electrode's location on the retina can be determined from the implant parameters. The corresponding location in the visual field ( µ e ) is determined using the retinotopic map described in

Watson et al. [59]. Each electrode's phosphene orientation is then θ e = ωθ axon , where θ axon is the orientation of the axon nerve fiber bundle (NFB) underlying the cell (pixel). Axon NFBs are modeled as spirals originating at the optic disc and terminating at each simulated cell. These spirals follow a simulated axon map [47] based on tracings of axon trajectories in 55 human eyes. In summary, phosphene size, eccentricity, brightness, and orientation are modulated based on stimulus parameters and implant location according to the following equations:

The phosphene for each electrode is a multivariate Gaussian blob, centered at the electrodes location in visual field ( µ e ), and with covariance matrix Σ e constructed such that the resulting phosphene will have brightness b e , size ρ e , eccentricity λ e , and orientation θ e , as shown in the following equations (repeated from main text for convenience):

The covariance matrix Σ e = RΣ 0 R T is calculated from the eigenvalue matrix Σ 0 and a rotation matrix R :

The eigenvalues α e and β e depend on the intended phosphene area ( ρ e ) elongation ( λ e ) , and a constant ϵ (set to e -2 ):

This formulation guarantees that the Gaussian blob, when thresholded using ϵ , will have the intended area, orientation, eccentricity, and brightness.

Blobs from individual electrodes are summed into a global percept. This linear summation is supported by recent studies, which have shown that percepts from multi-electrode stimulation are often linearly related to the percepts from stimulation on the individual electrodes [36]. Although the sum across electrodes is linear, modulating the size and eccentricity of phosphenes makes the final result a nonlinear function of stimulus parameters, preventing analytic inversion.

A.2 Evaluation

Our model is motivated by similar anatomical and psychophysical phenomena as the previous stateof-the-art model for epiretinal prostheses [11], but its formulation allows for favorable computational properties. In comparison, our model is on average 45x faster to run, and consumes about 120x less GPU memory. These computational benefits are the main reason a new model was necessary, and enables a more advanced deep stimulus encoder by allowing training with larger encoder models, longer training duration, and larger batch sizes.

Nonetheless, we also verified that the model produces state-of-the-art predictions, as described in Section 4.1. Despite its similar design, our model achieves much better scores on the Beyeler et al. [8] evaluation for phosphene shape. This is likely because our formulation allows much tighter control of phosphene shape attributes (e.g., size, eccentricity), allowing (for the first time) positive R 2 on shape descriptors for held-out electrodes. Our model performs similarly to the previous state-of-the-art model on the Granley et al. [11] evaluation, which is to be expected given that the equations modulating phosphene appearance with stimulus parameters in both models are very similar.

4A

35

22

State-of-the-Art,MSE:0.93,R2:0.91

State-of-the-Art,MSE:2.07,R2:0.97

Proposed Model,MSE:0.73,R2:0.94

Prop0sed Model, MSE:2.31, R2:0.96

20

30

18

25

Brightness

Brightness

16

20

T

一

14

15

12

10

10

2

3

4

5

6

20

40

60

80

100

120

Amplitude

Frequency

4C

5

50

7

State-of-the-Art,MSE:0.16,R2:0.97

State-of-the-Art,MSE:49.48,R2:0.59

Prop0sed Model,MSE:0.10, R2:0.98

Proposed Model, MSE:48.63,R2:0.62

6

40

5

Brightness

30

Size

4

20

3

10

2

1

T

0

2

3

4

5

6

0.0

2.5

5.0

7.510.012.515.017.5

Amplitude

Amplitude

4B

B Deep Stimulus Inversion

B.1 Architecture

The encoder architecture described in Section 3 is illustrated in Fig. B.1.

Block (B)

FC

→

FC

→

FC

→

BN

1024

1024

1024

3

FC

B

FC

1024

Target Response

FC

B

FC

FC

B

1024

1024

Stimulus

FC

BN

FC

B

Pdur

1024

Patient-Specific

Parameters

C Human-in-the-Loop Optimization

C.1 Kernel Selection and Hyperparameters

This section gives more details on fitting the hyperparameters for the Gaussian process (GP) kernel used in preferential Bayesian optimization (PBO). As stated previously, the performance of PBO crucially depends on the GP kernel and its hyperparameters, which encode our prior assumptions about the latent preference function. To select hyperparameters, we used a transfer learning strategy, selecting hyperparameters that generalized best within a small validation group of simulated patients (see Appendix C.2 for discussion).

We simulated 600 random duels ( ϕ 1 and ϕ 2 chosen randomly) on each of 10 simulated patients. For each patient, we fit four commonly used kernels (Squared Exponential, Squared Exponential with Automatic Relevance Determination (ARD), Matérn 3/2, and Matérn 5/2) and inferred hyperparameters using type II maximum likelihood estimation [60]. The bounds for each hyperparameter were [exp( -10) , exp(10)] . For each of these candidate kernel-hyperparameter pairs, we fit a GP with the corresponding kernel and hyperparameters to 50 training duels for each of the other 9 patients. Then, the performance of the candidate GP was evaluated on the remaining 550 data points using Brier score [43], a commonly used metric measuring the accuracy of probabilistic predictions:

where y true is the true duel outcome (1 or 0) as decided by the simulated patient, and y pred is the probability of ϕ 1 being selected as the winner (corresponding to outcome of 1), as predicted by the Gaussian process.

The kernel and hyperparameters with the lowest Brier score, averaged across all 9 other patients, were selected (Matérn 5/2). To verify that this kernel performed well, we also ran human-in-the-loop optimization (HILO) for 20 random simulated patients, using the best hyperparameters for each of the four previously mentioned kernels. The results are shown in Figure C.1. The Matérn 5/2 kernel performed slightly better than the Matérn 3/2 kernel, and significantly better than the ARD kernel. While performance was similar to the Squared Exponential kernel, we ultimately selected Matérn 5/2 due to its lower Brier score.

ARD

■Squared Exponential

■Matern 3/2Matern 5/2

Random

0.25

Mean

0.1

True

0

20

40

60

80

100

NumberofDuels

C.2 Hyperparameter Optimality

The Gaussian process kernel hyperparameters selected with our transfer learning strategy performed well in our simulations, leading to higher-quality patient-specific stimulus encodings (Figure 4). This transfer learning approach was chosen to match a clinical setting, where limited human data availability. However, it is certainly possible that better performing hyperparameters exist. To investigate the optimality of our hyperparameters, we examine two other strategies: an ideal case where 'patient-optimal' hyperparameters are used for each patient, and an online strategy where hyperparameters are updated during optimization with each patient,

Patient-optimal Hyperparameter Selection In this strategy, hyperparameters were precomputed for each patient by simulating 200 random duels. Kernel parameters were again fit using type II maximum likelihood estimation [60]. These 'patient-optimal' hyperparameters were then used during HILO for the patient.

transfer

patient-optimal

1

DSE-ΦGuess

Error

0.1

DSE-pTrue

0

25

50

75

100

Number of Duels

Optimization results for 20 random patients are shown in Figure C.2. Both the transfer and the patient-optimal settings led to similar performance, and there was no significant difference between the final reconstruction errors ( p > . 05 , two-sided paired t-test). Thus it seems the transfer learning strategy indeed selected hyperparameters that generalize well for new patients. Note that in general the transfer learning selection strategy is more practically applicable than the patient-optimal strategy since it does not require a long calibration period, but the patient-optimal strategy is an alternative that could be used for the first human subjects when no human data (only simulated data) is available.

Online hyperparameters selection While it is common to keep kernel hyperparameters constant during optimization [33, 61, 62], online optimization is an alternative, where kernel parameters are periodically re-fit to the patient data during optimization. We tested online optimization where hyperparameters were recalculated with an update period of 1, 5, 10, or 20 duels, or were never updated. In all settings, the initial hyperparameters were chosen using the transfer learning strategy.

The results for online hyperparameters optimization are shown in Figure C.3. All update periods eventually converged to similar performance as with the transfer learning strategy ( i.e never updating hyperparameters), but since recalculating hyperparameters is costly, online hyperparameter updates increased the optimization time required to reach a desired performance (Figure C.3, right ). This suggests that just using the transfer learning hyperparameters is a better strategy than online updates.

HILOPerformance

TimeNeededtoReachPerformance

2

350

Update Period

UpdatePeriod

None

None

1

1

1

5

250

5

10

DSE-ΦGuess

time

10

20

20

Error

Optimization

150

0.1

50

DSE-pTrue

0

25

50

75

100

0.21

0.18

0.15

0.12

0.09

0.06

NumberofDuels

Error

C.3 Acquisition function

The acquisition function is responsible for choosing ϕ 1 and ϕ 2 for each duel, and must balance exploration of the search space with exploiting values of ϕ that are expected to work well. The Maximally Uncertain Challenge (MUC) [42] acquisition presented in the main text was initially compared against 2 other top-ranking [42] acquisition functions: Bivariate Expected Improvement (Bivariate EI) [63], and Dueling Upper Credibility Bound (Dueling UCB) [64]. In addition, we also compare against a baseline acquisition, where ϕ 1 and ϕ 2 are chosen randomly for each duel. We ran HILO for the same 20 random patients for each acquisition.

The joint perceptual loss throughout optimization for each acquisition is presented in Figure C.4. All of the tested acquisition functions dramatically outperformed the random baseline, which converged to a value near the mean DSE without HILO. Although MUC and Dueling UCB performed similarly, we ultimately selected MUC due to its slightly lower final loss.

Acquisition Performance

Random

Bivariate EI

■Dueling UCB

MUC

Random

LOSS

Mean

0.25

0.1

True

20

40

60

80

100

C.4 Batch Bayesian Optimization

While the proposed Bayesian optimization strategy of sequentially presenting the user with stimuli would likely not be prohibitively time consuming (about 17 minutes with 100 duels, 10 seconds per duel), it is possible that batch Bayesian optimization could speed up optimization. We consider two formulations of batch Bayesian optimizations: 1) the user selects their preference from a batch of stimuli in each iteration, and 2) each duel still has only two options, but batches of duels are precomputed to save on updated the Gaussian process posterior and the acquisition function time between duels.

Option 1 is theoretically more ideal, since more information acquired in each comparison would hopefully allow for fewer comparisons. However, this would require the patient to remember an entire batch of stimuli before making a comparison, and in practice phosphenes can be very difficult to discriminate [7]. It is difficult to evaluate the effect of this cognitive burden on simulated patients, and we thus leave it to future work to consider whether the parallelization of data acquisition would make up for the increased difficulty of the task.

Therefore, option 2 was tested by precomputing batches of 1 ( i.e. original ), 3, 6, or 10 duels at a time. We used a batch variant of the maximally-uncertain challenge acquisition function [42]. Results for 20 simulated patients are shown in Figure C.5. Precomputing batches reduced the optimization time required to reach a desired performance, but at the cost of requiring more duels. In practice, the optimal batch size could be determined by balancing the time required per duel with the time required for optimization. Faster acquisitions ( e.g. KernelSelfSparring [65]) could further reduce optimization time.

HILOPerformance

TimeNeeded toReachPerformance

2

350

BatchSize

BatchSize

1

1

1

3

3

6

250

6

10

DSE-ΦGuess

time

10

Error

Optimization

150

0.1

50

"DSE-ΦTrue

0

25

50

75

100

0.24

0.18

0.12

0.06

NumberofDuels

Error

C.5 Baselines and Robustness

In this section, we provide further details for the naive encoder we compared against, and the robustness experiments.

Naive Encoder The naive encoder is the encoding strategy currently used in commercial epiretinal prostheses [3]. This encoder operates under the assumption that each electrode can be thought of as a pixel in an image. The optimal stimulus under this assumption is therefore simply a downsampled version of the target image. The frequency and pulse duration are constant across all electrodes. This naive encoder has been previously shown to be suboptimal [14], but we still include it as a comparison to the currently used encoding strategy.

Robustness In section 4.4 we evaluate the robustness of human-in-the-loop optimization (HILO) to misspecifications in the forward model. Here, we provide specific details on the implementation of these misspecifications, and how we adapted the baseline encoders to the misspecified scenarios.

Axon Trajectories : The simulated axon map from [47] has two parameters, β inf and β sup , which control the axon trajectories in the inferior and superior retina, respectively. [47] also reports the observed ranges for these parameters: β sup ∈ [ -2 . 5 , -1 . 3] and β inf ∈ [0 . 1 , 1 . 3] . The unmodified model uses the centers of these ranges. Under misspecification, we randomly set both β sup and β inf to one of these bounds for each patient.

Thresholds : Threshold is the amplitude at which a phosphene becomes visible to a patient 50 % of the time. In epiretinal prostheses, thresholds are notoriously noisy, and vary significantly across electrodes, patients, and over time [48]. While some progress has been made towards predicting these thresholds [66], most state-of-the-art models assume that thresholds are known.

With this misspecification, the assumed threshold on each electrode was modified by a random but systematic amount. Specifically, the threshold on each electrode was randomly selected to be between 1 2 x and 2x its original value for the 100% condition and between 1 4 x and 4x its original value for the 300% condition.

Out of Distribution : It is also possible that a new patient does not fall within our assumed ranges. Thus, we tested a variant where the true patient-specific parameters ϕ were sampled from outside the ranges in Table 1. Specifically, each parameter was sampled to be 0-50% above or below the specified range (some parameters were clipped to stay within defined ranges, e.g. , λ cannot be outside of [0 , 1) ).

During HILO, the acquisition functions have specified bounds that constrain candidate ϕ . We therefore tested two variants, one where PBO was allowed to expand the bounds, and another where it was confined to within its original bounds. The end results were similar in terms of DSE performance, so the variant with its original bounds is presented in the main text.

DSEϕ Guess is our best approximation of what a DSE would guessed patient-specific parameters would perform, and is bounded by the performance of a DSE with mean ϕ , and random ϕ from the ranges in Table 1. For each of the misspecifications, the mean and random ϕ baseline DSEs are still encoded with the same ϕ as in the unchanged patient, but since the phosphene model for the patient is changed, the resulting loss is different. DSEϕ True is still parameterized with the patients true ϕ , however, the true ϕ are no longer a perfect description of the misspecified patient. This is shown by the fact that HILO surpasses the true encoders performance: under the misspecified model, there exists some other ϕ which leads to percepts with improved perceptual quality when decoded using the misspecified phosphene model, compared to those encoded with the true ϕ . This highlights the robustness of optimization based on user preferences.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a method to improve perceptions for patients using visual prostheses using a combination of a deep learning encoding model and a patient-specific tuned set of parameters for stimulation learned via preferential Bayesian optimization. They use simulated data and patient choices to demonstrate the method’s performance across a range of possible patient types and noise considerations. \n\nETA: After review and rebuttal, raising my score from a 5 to a 6.\n\nSTRENGTHS:\nOriginality: This work presents a nice combination of deep learning and Bayesian optimization. Each component of their approach (phosphene model, encoding model, and parameter optimization) has in general been done before, but here the authors modify each element and combine them to overall improve performance. The phosphene model is described as a new model with performance exceeding current SOTA. This was then combined with a deep stimulus encoder (DSE) to yield performance comparable to other methods. Finally, their human-in-the-loop (HILO) method to tune parameters using PBO was used to iteratively improve the DSE per patient. \n\nQuality: In general the work is well-sourced and supported from many previous works. The benchmarking and evaluation of performance was good, and the authors compared to similar methods. The code in particular was very well written and documented.\n\nClarity: Overall well-written, well-sourced, nicely explained work. There were some points of confusion or ambiguity mentioned below. \n\nSignificance: This kind of patient-specific tuning of stimulation for visual prostheses is certainly needed, and the authors do good job of demonstrating the significance of their contribution. The authors also suggest it could be used in other areas beyond visual stimulation, but with the results being tied closely to the patient-specific parameters in Table 1 it is unclear how broadly this method might be applied.\n\nWEAKNESSES:\nThe primary weakness is the reliance on simulated data, and it is not clear that simulated patient choice or patient parameters to generate the data was sufficiently grounded in experimentally collected datasets. There is some discussion of how the ranges for the parameters were chosen in their simulated data, but nothing about the expected variability and how these parameters were determined in the first place (is it considered ground truth? were patient preferences included?). If the method is sufficiently generalizable to many populations of patients, as expected, then why not simulated patients outside of those limited ranges in Table 1? Lines 263-265 appear to suggest that their ranges of parameters are unlikely to be same as in the real world. A similar concern is for simulating patient preference, but here the authors' simulations (up to 33% incorrect decisions from the patient) appear like a robust demonstration. \n\nOne other weakness not discussed is the feasibility of their approach for actual human patients. Even if it only takes 3 seconds to update the GP and generate new stimuli, the patient must sit through and compare difficult to discern visual percepts for 100 duels. Is it likely the patient would persist? Does their judgement change over time? If this could be changed every so often, does the system need re-tuning for all ~100 duels at once? Additionally, humans see visual scenes with certain image statistics -- so MNIST may not be a reasonable or sufficient dataset for testing human perception on. Could this approach be extended to natural images, including color?\n\nQUESTIONS:\ni.\tLine 43 says <30 is too few parameters, but then the authors used only 13 dimensions of parameters? \nii.\tLine 129-130 appears to have a sentence cut off abruptly.\niii.\tLine 127: BO can’t optimize more than a small number of parameters -- clarify please? BO gets difficult in high dimensions, but this is true of many methods. How specifically does the DSE reduce this? \niv.\tClarify the issue in lines 154-158. The logic is not clear what the exact problem is and how it is solved here. \nv.\tIf you didn’t restrict the range of parameters in Table 1 as heavily, would it still find the optimal values? E.g. if you increased the range by a factor of 2. \nvi.\tLine 232 first mention of Argus II, needs explanation and citation. \nvii.\tWhat does it mean to outperform the true DSE? Line 289\nviii.\tLine 296 how does the approach fail?\n\nLIMITATIONS:\nThe authors addressed some limitations in the discussion, noting some drawbacks of predicting the behavior of deep learning algorithms but that hardware-engineered safety constraints would provide patient protection.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors proposed a pipeline for optimizing the deep neural network based encoder, which is to generate visual stimulus for neuroprostheses. The pipeline considers human-in-the-loop optimization. The authors use preferential Bayesian optimization techniques to reduce the number of queries to the human making the decision. The major contributions claimed by the authors are as follows:\n1. Introduced a forward model for retinal implants suitable for fast simulation experiments.\n2. Proposed a personalized human-in-the-loop (HILO) patient-specific stimulus parameter optimization strategy using preferential Bayesian optimization.\n3. Demonstrated HILO can quickly learn personalized stimulus encoder and the robustness of HILO, on simulated patients.\n\nSTRENGTHS:\nOriginality and significance:\nAs far as I know, this work is the first work in considering the patient-specific stimulation parameter calibration problem for visual neuroprostheses that does not assume direct access to the output of the forward model. Instead, the authors introduce the technique of human-in-the-loop training with preferential Bayesian optimization in the pipeline. This setting is steps more realistic than previous works. \n\nQuality and clarity:\nThe writing in general is very structured and easy to follow.\n\nWEAKNESSES:\n1. It is not made super clear whether and how the parameters w of the encoder is being updated in the manuscript. \n2. The authors could have demonstrated a small amount of humans with sight to show the effectiveness of this optimization method.\n\nQUESTIONS:\n1. The simulated patient has access to t, which is the target percept or ground truth. Is it realistic to assume the blind person could have access to that? \n2. In line 219 it’s written that the metric in equation (7) is also used to train the deep stimulus encoder (DSE). So the encoder’s parameters are getting updated in a rather conventional way as in the visual prosthesis literature. How to distinguish the contribution between the improvement of the encoder and the selection of patient-specific parameters? Could it be that the update of the encoder contribute the most? Ablation studies might be needed.\n\nLIMITATIONS:\nOther than the concerns listed in the weakness and question sections, the limitations mentioned in the work are adequate.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n* This paper proposes a flexible framework addressing personalized stimulus optimization predominantly seen in _visual prostheses_. \n* The authors propose integrating the state-of-the-art deep learning with a preferential Bayesian Optimization (BO) strategy to learn optimal patient-specific parameters in fewer trials. \n* The proposed approach is a two-stage process, with the first stage aiming at learning a Deep Stimulus Encoder (DSE) to optimize stimuli, and the second stage aiming at embedding the DSE learnt in the first stage in a sample-efficient preferential Bayesian optimization strategy.\n\nSTRENGTHS:\n* This paper aims at advancing the state-of-the-art in sensory neuroprostheses that have a significant impact on the present world.\n* The authors propose to overcome the strong assumptions made on the accuracy of the knowledge on patient-specific parameters, thus aiming to improvise the accuracy of the deep stimulus encoder by optimizing the stimuli.\n* The authors use a sample-efficient Bayesian optimisation (Human-in-the-loop Optimization (HILO)) method to optimally select the patient-specific parameters used in deep stimulus encoder. \n* Authors have empirically evaluated the proposed approach in optimizing the personalized stimulus encoder for several simulated patients.\nFurther, the authors evaluate the performance of their method when the preferential inputs are noisy, thereby proving its robustness.\n\nWEAKNESSES:\n* The main weakness of this paper lies in its _limited novelty_, as the proposed approach is merely a _combination_ of the state-of-the-art of forward models and sample-efficient Bayesian optimization to optimally find personalized stimuli. \n* Although this paper improvises the techniques used in the visual prostheses domain, the overall contribution of this paper is in the _application of the existing ML/DL techniques_ in neuroprostheses, visual prostheses in particular, thus may not advance the state-of-the-art in ML/DL community. \n* Though the paper reads well in many parts, it demands significant knowledge/domain expertise in visual prosthesis for a thorough understanding of the paper. \n* Authors have missed defining/describing a few important mathematical quantities or parameters that break the flow of understanding and also raise questions about its reproducibility (Please refer to the Questions section).\n* The details provided in _Section 4_ give the impression that the proposed framework is tightly coupled with the problems in visual prostheses. \n* The authors have discussed a few experimental settings in Section 4 that are apt to discuss in Section 5, though they do not contribute to the proposed method.\n* The visualization schemes used to show the empirical results do not effectively demonstrate the superiority of the proposed approach. Also, the authors do not discuss in brief the trends and the possible reasons behind them.\n\nQUESTIONS:\n* Line 159: What does $n_e$ stand for? number of electrodes?\n* Line 169: What is the significance of threshold $\\epsilon$ and why authors have set it to $e^{-2}$? \n* Line 198: Why do authors considered only 10 simulated patients for transfer learning? is that not sparse? \n* Line 200: The hyperparameters considered and their bounds are neither mentioned in the main paper nor in the supplementary material. Could authors confirm the hyperparameter set of each kernel?\n* Line 214: The noise parameter s is set to be $\\frac{1}{0.01}=100$, whereas in Line 285, s is set to 1e-4, is this a typo or it is $\\frac{1}{1e-4}$?\n* Line 221: What is the significance of $\\beta$ in equation (7)?\n* Line 235: What does $R_i$ stands for? Is $i \\in$ {$ 1,...,n_e$}?\n* Line 239: Why authors have not considered comparing their empirical results with Beyeler et al. 2019?\n* Line 250: How do the results compare with Granley et al. 2022 if L1 is used in computing the test loss for DSE?\n* Line 302: Could Batch-BO strategies further reduce the latency and thus provide a quicker solution by parallelizing the data acquisition?\n* Figure 3: Why there is an increasing (or decreasing) trend in loss (or accuracy) plot during the initial phase of the optimization?\n* Figure 4: Why there is a lot of variance in the bottom right plot for \"Out of Distribution\" misspecification?\n\n\nFollowing are the minor (typographical) errors, that needs to be fixed.\n\n* Inconsistent referencing style and hyperlinks are missing throughout the paper. It is strongly suggested to not modify the style files provided by NeurIPS.\n* Line 59: \"...visual prostheses that, where ....\" \n* Figure 2: The current font size in the Leftmost plot is illegible.\n* Line 151: The section/Sub-section number has to be clearly mentioned. \n* Line 204: Provide a quick reference to the Brier score in the main paper, even though it is mentioned in the supplementary.\n* Line 237: \"ratings as amplitude, frequency,....\"\n* Line 259: :...  perceptual loss (figure 3.B)....(figure 3.C)\"\n\n**References**\n\n* Michael Beyeler, Devyani Nanduri, James D. Weiland, Ariel Rokem, Geoffrey M. Boynton, and Ione Fine. A model of ganglion axon pathways accounts for percepts elicited by retinal implants. Scientific Reports, 9(1):1–16, June 2019.\n\nLIMITATIONS:\n* The optimization performance of PBO crucially depends on the _ generalization performance of the inherent GP surrogate model_. Authors reduce latency by avoiding online inference and instead use a transfer learning strategy. However, the source considered in the adopted transfer learning uses sparse data generated from 10 patients, thus, raising concerns about the optimality of kernel parameters and generalizability of the GP surrogates.\n* Although the authors discuss the empirical results, the lack of sufficient experiments and competing baselines (Beyeler et al. 2019, Granley et al. 2022) in the empirical results makes it hard to conclude the superiority of the proposed approach. (Please refer to the Questions section).\n* Sequential PBO may not be very suitable in time-critical situations, thus, more efficient strategies such as batch-BO could be of great use to reduce the possible latency in finding the optimal stimuli on-the-fly.\n\n**References**\n\n* Michael Beyeler, Devyani Nanduri, James D. Weiland, Ariel Rokem, Geoffrey M. Boynton, and Ione Fine. A model of ganglion axon pathways accounts for percepts elicited by retinal implants. Scientific Reports, 9(1):1–16, June 2019.\n\n* Jacob Granley, Lucas Relic, and Michael Beyeler. Hybrid Neural Autoencoders for Stimulus Encoding in Visual and Other Sensory Neuroprostheses. October 2022.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-24"}