Hybrid Neural Autoencoders for Stimulus Encoding in Visual and Other Sensory Neuroprostheses
Authors: Jacob Granley
Lucas Relic
Michael Beyeler
Venue: NeurIPS 2022 Accept (2022)
Keywords: Sensory Neuroprostheses
Bionic Vision
Brain Computer Interfaces
BCI
Stimulus Encoding
Autoencoder
Perception
Computational Modeling
Inverse Problem
Retinal Prostheses
Argus
Cortical Prostheses
Vision

Abstract: Sensory neuroprostheses are emerging as a promising technology to restore lost sensory function or augment human capabilities. However, sensations elicited by current devices often appear artificial and distorted. Although current models can predict the neural or perceptual response to an electrical stimulus, an optimal stimulation strategy solves the inverse problem: what is the required stimulus to produce a desired response? Here, we frame this as an end-to-end optimization problem, where a deep neural network stimulus encoder is trained to invert a known and fixed forward model that approximates the underlying biological system. As a proof of concept, we demonstrate the effectiveness of this Hybrid Neural Autoencoder (HNA) in visual neuroprostheses. We find that HNA produces high-fidelity patient-specific stimuli representing handwritten digits and segmented images of everyday objects, and significantly outperforms conventional encoding strategies across all simulated patients. Overall this is an important step towards the long-standing challenge of restoring high-quality vision to people living with incurable blindness and may prove a promising solution for a variety of neuroprosthetic technologies.

Hybrid Neural Autoencoders for Stimulus Encoding in Visual and Other Sensory Neuroprostheses
Jacob Granley
Lucas Relic
Department of Computer Science University of California, Santa Barbara jgranley@ucsb. edu
Department of Computer Science
University of California, Santa Barbara lucasreli edu
Michael Beyeler
Department of Computer Science
Department of Psychological & Brain Sciences University of California, Santa Barbara mbeyeler@ucsb. .du
Abstract
Sensory neuroprostheses are emerging as promising technology to restore lost sensory function or augment human capabilities. However, sensations elicited by current devices often appear artificial and distorted. Although current models can predict the neural or perceptual response to an electrical stimulus, an optimal stimulation strategy solves the inverse problem: what is the required stimulus produce desired response? Here, we frame this as an end-to-end optimization problem, where deep neural network stimulus encoder is trained to invert a known and fixed forward model that approximates proof of concept, (HNA) in visual neuroprostheses. We find that HNA produces high- fidelity patient- specific stimuli representing handwritten digits and segmented images of everyday objects, and significantly outperforms conventional encoding strategies across all simulated patients. Overall this is important step towards the long-standing challenge of restoring high-quality vision to people living with incurable blindness and may prove promising solution for variety of neuroprosthetic technologies.
underlying biological system. As a demonstrate the effectiveness of this hybrid neural autoencoder
Introduction
Sensory neuroprostheses emerging as promising technology restore lost sensory function augment human capacities [1, 2]. In such devices, diminished sensory modalities (e.g., hearing [3], vision [4, 5], cutaneous touch [6]) re-enacted through streams of artificial input to the nervous system. For example, visual neuroprostheses electrically stimulate neurons in the early visual system to elicit neuronal responses that the brain interprets visual percepts. Such devices have the potential to restore a rudimentary form of vision to millions of people living with incurable blindness.
However, all of these technologies face the challenge of interfacing with highly nonlinear system of biological neurons whose role in perception not fully understood. Due to the limited resolution of electrical stimulation, prostheses often create neural response patterns foreign to the brain. Conse- quently, sensations elicited by current sensory neuroprostheses often appear artificial and distorted [7, 8). A major outstanding challenge is thus identify stimulus encoding that leads to perceptually intelligible sensations. Often there exists a forward model, f (Fig. 1A), constrained by empirical data, that can predict neuronal or (ideally) perceptual response to the applied stimulus (see [9] for recent review). To find the stimulus that will elicit a desired response, one essentially needs to find
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

A
C
HNA Training Phase
Stimulus
3
3
Target Percept
Enzoder
Stimuuus
Decoder
Predicted Percept
Resporise
Percsptual Loes
Prosthesis Recipient
B
Hybrid Neural Autoencoder (HNA)
D
Bionic Vision: Deployment
Target Resporse
Predicted Response
3
Irared
Tuod
Fund
YPU
Encoder
Dacodar
Targel Parospt
Encoder
Blanic Recipient
Reconstrutton Error
Figure 1: A) Sensory neuroprosthesis. A forward model (f) is used to approximate the neuronal or, ideally, perceptual response electrical stimuli. B) Hybrid neural autoencoder (HNA). A deep neural encoder (f--) is trained to predict the patterns of electrical stimulation that elicit responses closest the target. C) Visual neuroprostheses are one prominent application of HNA, where an encoder can be trained to predict the electrical stimulation needed to elicit a desired visual percept. D) The trained encoder is deployed on vision processing unit (VPU), predicting stimuli in real-time that are decoded by the patient's 's visual cortex.
the inverse of the forward model, f-1 However, realistic forward models are rarely analytically invertible, making this challenging open problem for neuroprostheses.
Here we propose approach this an end-to-end optimization problem, where a deep neural network (DNN) (encoder) is trained to invert a known, fixed forward model (decoder, Fig. IB). The encoder is trained to predict the patterns of electrical stimulation patterns that elicit perception (e.g. vision, audition) or neural responses (e.g.. firing rates) closest to the target. This hybrid neural autoencoder (HNA) could in theory be used optimize stimuli for any open-loop neuroprosthesis with known forward model that approximates the underlying biological system.
In order optimize end-to-end, the forward model must be differentiable and computationally efficient. When this is not the case, an alternative approach to train surrogate neural network, f, to approximate the forward model [10-13]. However, even rell-trained surrogates may result in errors when included in our end-to-ent framework, due the encoders' ability learn to exploit the surrogate model. We therefore also evaluate whether a surrogate approach to HNA is suitable.
To this end, make the following contributions:
We propose hybrid neural autoencoder (HNA) consisting deep neural encoder trained to invert fixed, numerical or symbolic forward model (decoder) to optimize stimulus selection. Our framework is general and addresses an important challenge with sensory neuroprostheses.
As proof of concept, we demonstrate the utility of HNA for visual neuroprostheses, where we predict electrode activation patterns required to produce desired visual percept. We show that the HNA is able to produce high-fidelity, patient-specific stimuli representing handwritten digits and segmented images of everyday objects, drastically outperforming conventional approaches.
We evaluate replacing a computationally expensive or nondifferentiable forward model with a surrogate, highlighting benefits and potential dangers of popular surrogate techniques.
Background
Sensory Neuroprostheses Recent advances in neural understanding, wearable electronics, and biocompatible materials have accelerated the development of sensory neuroprostheses to restore perceptual function to people with impaired sensation. Use of neuroprostheses typically requires invasive implants that elicit neural responses via electrical, magnetic, or optogenetic stimulation. Two of the most promising applications are cochlear implants, which stimulate the auditory nerve to elicit sounds [3], and visual implants (see next subsection) to restore vision to the blind. However, variety of other devices are development; for instance, to restore touch [6, 14] or motor function [15]. The latter differ from other sensory neuroprostheses in that they generate stimuli using motor

---

Simulated Prosthetic Vision
Predicted Visual Percepts
imrpoatt
OAts neve
A= 100
A 2,000
p= 106
3
Camera
20000
000
Masselecuude
Aroa of
o= 500
Tack
Insh and
Gangles
Figure 2:
Left: Visual prosthesis. Incoming target images are transmitted from camera to an implant the retina, which encodes the image as an electrical stimulus pattern. Center: Electrical stimulation (red disc) of nerve fiber bundle (gray lines) leads to elongated tissue activation (gray shaded region) and phosphenes whose shape can be described by two parameters, À (axonal spread) and p (radial spread). Right: Predicted percepts for an MNIST digit using varying P and À values.
feedback (closed loop) [16, 17]. the absence of feedback (open loop), a proper stimulus encoding is paramount to the success of these devices.
Restoring Vision to the Blind For millions of people who are living with incurable blindness, visual prostheses (bionic eye, Fig. 2, left) may be the only treatment option [18]. Analogous to cochlear implants, these devices electrically stimulate surviving cells in the visual pathway to evoke visual percepts (phosphenes), which can support simple behavioral tasks [5, 19, 20].
common misconception is that each electrode in the array can be thought of as pixel in an image; to generate complex visual experience, one then simply needs to pixels [21]. However, recent evidence suggests that phosphenes often appear distorted (e-g., as lines, wedges, and blobs) and vary drastically across subjects and electrodes [4. 7].
the right combination of
Phosphene appearance has been best characterized in epiretinal implants, where inadvertent activation of nerve fiber bundles (NFBs) in the optic fiber layer of the retina leads to elongated phosphenes [22, 23] (Fig. 2, center). To this end, Granley et. al [24] developed a forward model to predict phosphene shape function of both neuroanatomical parameters (i.e., location of the stimulating electrode) and stimulus parameters (i.e., pulse frequency, amplitude, and duration). With this model, phosphenes are primarily characterized by two main parameters, p and X, which dictate the size and elongation of the elicited phosphene, respectively (Fig. 2, right). These parameters can be determined using psychophysical tasks (e.g., drawings, brightness ratings) [22, 24], and although they vary drastically across patients [22], they do not change much over time [25, 26J. Stimulation from multiple electrodes is nonlinearly integrated into combined perception, and if two electrodes happen to activate the same NFB, they might not generate two distinct phosphenes.
3 Related Work
The conventional 'naive" encoding strategy sets the amplitude of each electrode to the brightness of the corresponding pixel in the target image [5, 27), making the stimulus a down-sampled version of the target. Although simple, this strategy only works with near-linear forward models, cannot account for real phosphene data, and often leads to unrecognizable percepts (Fig. 2, right) [7, 22].
Many alternative stimulation strategies have been proposed [28]. Shah et al. [29] used a greedy approach iteratively select the stimuli that best recreate desired neural activity pattern over given temporal window, assuming that the brain would integrate them into a coherent visual percept. Ghaffari et al. [30] used neural network surrogate model combined with an interior point algorithm to optimize for localized, circular neural activation patterns for individual electrodes. Fauvel et al. [31] used human in-the-loop Bayesian optimization to achieve encodings perceptually favored by the simulated patient. Spencer el al. [32] proposed framing stimulus encoding as inversion of forward model of neural activation patterns, but to approximate the inverse, their approach either requires simplification is Na-hard [32].
3

---

Van Steveninck et al. [33] proposed an end-to-end optimization strategy with a fixed phosphene model, similar to HNA. However, their approach crucially differs from ours in its inclusion of a secondary DNN to post-process the predicted phosphenes. This is critical limitation, because low reconstruction loss does not reveal whether a high-fidelity encoder was learned or whether the secondary decoder network simply learned to correct for the encoder's is mistakes. In addition, they used an unrealistic phosphene model that simply upscales and smooths a binary stimulus pattern. is therefore not clear whether their results would generalize to real visual prosthesis patients.
Relic er al. [10] also utilized the end-to approach, but without the secondary decoder network used in [33]. They used a more realistic phosphene model, which accounts for some spatial distortions (e.g.. axonal streaks), but not the effects of stimulus parameters. Since including realistic phosphene model in the loop is not straightforward, they instead trained surrogate neural network to approximate the forward model. We re-implemented Relic's 's surrogate approach in this paper as a baseline method to compare against, as described in Section 4.
Taken together, we identified three main limitations of previous work that this study aims address:
1) Unrealistic forward models. Most previous approaches (e.g. s. [29, 32, 33J) use an overly simpli- fied forward model that cannot account for empirical data [7. 22]. We overcome this limitation by developing (and inverting) a differentiable version of a meurophysiologicall informed and psychophysically validated phosphene model [24] that can account for the effects of stimulus amplitude, frequency, and pulse duration on phosphene appearance.
2) Optimization of neural responses. Most previous approaches (e.g., [29, 32]) focus optimiz- ing neural activation patterns in the retina response to electrical stimulation ( ("bottom-up"). However, the visual system undergoes extensive remodeling during blinding diseases such as retinitis pigmentosa [34]. Thus the link between neural activity and visual perception is unclear. We overcome this limitation by inverting phenomenological ("top-down") model constrained by behavioral data that predicts visual perception directly from electrical stimuli [22, 24].
3) Objective function Most previous approaches rely on minimizing mean squared error (MSE) between the target and reconstructed image. Although simple and efficient, MSE is known to be a poor measure of perceptual dissimilarity for images [35] and does not align well with human assessments of image quality [36]. We overcome this limitation by proposing a joint perceptual metric that combines mean absolute error (MAE), VGG, and Laplacian smoothing losses.
4 Methods
Problem Formulation We consider system where there is some known forward process f mapping stimuli to responses f : S R. f(S) C R. In the case of visual prostheses, f may map stimuli to visual percepts. may optionally be parameterized by patient-specific parameters ø.
Finding the best stimulus for an arbitrary target response E is equivalent to finding the inverse of f. However, since not every response can be perfectly reproduced by stimulus, the true inverse of not well defined. We therefore seek the pseudoinverse (still denoted as f-1 for simplicity) instead, which outputs the stimuli that produce the closest response to t. under some distance metric d:
f-' (t, Φ) arg min d( d[((s; (s)
sES
This problem is straightforward to solve using autoencoder approach, where a learned encoder f-1 is trained to invert the fixed decoder of (i.e., forward model).
Encoder We approximate the pseudoinverse f-1 with a DNN encoder f-1 Φ; 8) with weights 8, trained to minimize the distance d between the target image t and predicted image t:
min d(t,t) t)
i= s(j Φb 0); Φ),
(3)
where @ is sampled from a uniform random distribution spanning the empirically observed range of patient-specific parameters [22, 24].

---

We use a simple architecture consisting solely of fully connected (FC) and batch normalization (BN) [37] layers (1.4M trainable parameters). First, the target t is flattened and input to FC layer. In parallel, the patient parameters & are input to BN layer and two hidden FC layers. Then, the outputs of these two paths are concatenated, and the combined vector fed through two FC layers, producing an intermediate representation x. Amplitudes are predicted from x with FC layer. The amplitudes are then concatenated with x, put through a BN layer, and used to predict frequency and pulse duration, each with FC layer. The outputs are merged into stimulus matrix 8. All layers use leaky ReLU activation, except for stimulus outputs, which use ReLU to enforce nonnegativity.
Decoder The HNA decoder is a differentiable approximation of the underlying biological system, and describes the transform from stimulus to response. For our decoder f. we use a reformulated but equivalent version of the model described in [24]. This model is specific to epiretinal prostheses; analogous models exist for other neuroprostheses (e.g., auditory [38-43], tactile and somatosensory
[44-4]]), and could potentially be adapted for use with HNA. We use a square 15 x 15 array of 150mm electrodes, spaced 400 apart and centered on the fovea. The size and scale of this device are motivated by similar designs in real epiretinal implants.
I takes as input a stimulus matrix s E 12203 where the stimulus on each electrode (8₂) is a biphasic pulse train described by its frequency, amplitude, and pulse duration. A simulated map of retinal NFBs is used to predict phosphene shape. Following [22], each retinal ganglion cells' activation is assumed to be the maximum stimulation intensity along its axon. Axon sensitivity is assumed to
decay exponentially with i) distance to the stimulating electrode (radial decay rate, p) and distance to the soma along the curved axon (axonal decay rate,
A). To account for stimulus properties [24), p. X, and the per-electrode brightness are scaled by three simple equations Fsizo 6), Fstreak (Se: 6), and Fhiight (Se, Φ), respectively.
The brightness of pixel located the point R² in the output image is given by
e + -x, a)2
f(s; Φ) max aeA eE
Σ
Foright (Se: o) exp F (se: の) 2X Fstreak の)
(4)
where A is the cells' axon trajectory, E is the set of electrodes, & = {o, A, } is a set of 12 patient-specific parameters, and d, is the path length along the axon trajectory [49]from a x:
dx(x, a) = f.a VAAOOO- |10)2
(dA(O) d8
d0.
This model (f) can be fit to individual patients; however, it is computationally slow and not differen- tiable. For more details on these equations, see [24]. We therefore considered two approaches:
Differentiable Model: We reformulated equations 4 and 5 into an equivalent set of parallelized matrix operations, implemented in Tensorflow [50]. Significant efforts were put towards de- veloping a model optimized for XLA engines on GPU, resulting in speedups of up to 5000x compared to the model as presented in [24], enabling large-scale gradient descent. To enforce differentiability, we numerically approximated d. using |-|| = 500 axon segments per axon.
Surrogate Model: We also implemented the surrogate approach from [10] as baseline method, where f is approximated with another DNN fo 0y) with weights 0,. To achieve this generated 50,000 percepts using randomly selected stimuli passed through f and fit a DNN patient-specific parameters ø, generated new data and fit a separate surrogate model for each ø in our experimental set. Specific implementation details of the surrogate are presented in Appendix A. Our implementation improves upon [10] by using the more advanced phosphene model described above, which accounts for effects of stimulus properties and allows for optimization of stimulus frequency in addition to amplitude.
produce identical images. As is highly dependent
Metrics To measure perceptual similarity. we use joint perceptual objective consisting of a VGG [51] similarity term, a mean absolute error (MAE) term, and a smoothness regularization term. The E/|1-
MAE term is given by LMAE [
The VGG term aims to capture higher-level differences between images [33, 52]. The target image and reconstructed phosphene are input to VGG-19 pretrained on ImageNet [53], and the MSE between
5

---

the activations on downstream convolutional layer is computed. Let Vi be function that extracts the activations of the I-th convolutional layer for a given image. The VGG loss then defined as LVGG | [V((()) 2
We also include a smoothing regularization term. This term imposes a loss on the second spatial derivative of the predicted image. A low second derivative implies that where the target image does change, changes slowly. We found this encouraged smoother, more connected phosphenes. To approximate the second derivative, we convolve the image with a Laplacian filter [54] of size k. denoted by Lap(-,k), k)) and compute the mean. The smoothness loss given by:
Lsmooth
22
Σ Lap(t, k); [
1
(6)
Our final objective is the weighted sum of the three individual losses, given by Eq. 7, where a and B hyperparameters controlling the relative weighting of the three terms.
d = MAAE + aLLsmoot + 3LvGG.
(7)
We also implement a secondary metric to quantify phosphene recognizabilityy applicable only for the MNIST reconstruction task. We first pre-train a classifier network on the MNIST targets until it reaches 99% test accuracy, and then fix the weights. The relative accuracy (RA) is then defined as the ratio of the classifiers accuracy on the reconstructed images to its accuracy on the targets RA = ACC/ACC(t) A perfect encoder would result in RA = 100%. A similar process was not possible for the COCO task due to the possibility of having multiple objects in each target image.
Training/Optimmzzation We trained using Tensorflow 2.7 [50] on a single NVIDIA RTX 3090 with 24GB memory. Stochastic gradient descent with Nesterov momentum was used to minimize the joint perceptual loss. We used a batch size of 16 due to memory limitations imposed by f. The amplitude, frequency predictions are scaled by 2, 20 respectively, while the pulse duration predictions were shifted by 1e-3 prior to being fed through the decoder. This encourages the initial predictions of the network to be in a reasonable range. The Laplacian filter size l is set to 5. We choose to be first convolutional layer in the last block using cross validation (see Appendix B). Similarly, we perform cross validation to find the best values for a and B. Instead of using one value, we found that incrementally increasing the weighting of the VGG loss (3) from O while simultaneously decreasing the initially high weight on the smoothing constraint (a) was crucial for performance, especially when the range of allowed 6 values was large (see Appendix B).
Datasets We first evaluated on handwritten digits from MNIST [55], enabling comparison to previous works [10]. Images preprocessing consisted of resizing the target images to the same shape as the output of (49x49). We also evaluate more realistic images of common objects from the MS-COCO [56] dataset. We selected a subset of 25 of the MS-COCO object categories deemed more likely be encountered by blind individuals (e.g. people, household objects), and use only images that contain at least one instance of these objects. We further filter out images by various other criteria, such as being too cluttered or too dim. This process results in a total of approximately 47K training images and 12K test images. See Appendix C for a full description of the selection process.
Natural images often contain too much detail to be encoded with prosthetic vision. While scene simplification strategies exist [57]. we focus on the encoding algorithm, so we simply used the ground-truth segmentation masks segment out the objects of interest. The images were then converted to grayscale, and resized to 49 x 49 pixels.
Results
5.1 MNIST
The phosphenes produced from the HNA, surrogate, and naive encoders on the MNIST test set are shown in Fig. 3 and performance is summarized in Table 1. For each MNIST sample, the target image is input to the encoder, which predicts stimulus. The stimulus is fed through the true forward model f, and the predicted phosphene is shown. Since the surrogate method must be retrained for each Φb, results are only shown for simulated patients. Our proposed approach outperformed the baselines across all metrics (sec Appendix D for a comparison of stimuli).

---

Target images:
2
3
4
5
6
7
8
9
x= 100
150, A= 1500
HNA:
3
4
5
6
7
8
9
6
/
3
_
Surrogate:
6 6
/
77
+.. 5
Naive:
2
3
4
7
8
9
D 800 X= 100
p 800 A 1500
HNA:
/
3
4
5
6 7
8 9
6
/ 2
3
7
5
6
Surrogate:
_
:
4
Nalve:
Figure 3: Reconstructed MNIST targets for HNA, surrogate, and naive encoders across 4 specific simulated patients. Note that the brightness the naive encoder is clipped for display
Table 1: MNIST performance
Encoding
p= 150 A=100
150
A=1500
p=800
A=100
p=800
A=
Joint Loss
MAE
RA
Joint Loss
MAE
RA
Joint Loss
MAE
RA
Joint Loss
MAE
RA
Naive
1.161
0.1855
90.3
1.442
0.214
78.1
8.152
1.500
34.8
8,780
1.726
28.8
Surrogate
2.509
0.1351
53.8
3.118
0.2431
30.7
1.692
0.2135
19.9
1.694
0.2237
18.1
HNA
0.559
0.064
98.1
1.029
0.1412
89.3
0.913
0.113
95.9
0.957
0.126
94.8
5.2 COCO
The phosphenes produced by HNA and the naive encoder for the segmented COCO dataset are shown in Fig. 4. We omit the surrogate results due to its poor perceptual performance on MNIST. Averaged across all Φ, HNA had a joint loss of 0.713 on the test set and MAE O. 1408, while the naive encoder had joint loss of 1.873 and MAE of 0.2830.
5.3 Modeling Patient-to-Patient Variations
MNIST encoder performance across simulated patients (#) is shown in Fig. 5. Since the surrogate encoder has to be retrained for each patient, comparison is infeasible. To visualize the effects of changing / and on the produced phosphenes, Fig. 5A shows the result of encoding two example MNIST digits, both using the naive method and encoder. As À increases, the naive phosphenes appear increasingly elongated, and as p increases, the phosphenes become increasingly large and blurry. The phosphenes from HNA are slightly too dim and disconnected at low p, but are relatively stable across other values of and X.
To compare performance across the entire dataset, we computed the average test set loss across the same range of p and (Fig. 5B). The encoder performs well across a wide range simulated patients, with larger loss only at low p. The naive method performs well only on limited set of ó, with small À and p xe 200. The naive loss was higher than the learned encoder at every simulated point. Random sampling of and À for each image results in a joint loss of 0.921, MAE of O. 120, and RA of 94.0% for HNA, while the naive encoder results in a joint loss of 3.17, MAE of 0.596, and RA of 63.6%. The same analysis yielded similar results COCO (Appendix E). An analysis across other parameters is presented in Appendix F.
In order for prosthetic vision to be useful, different instances of the same objects would ideally produce similar phosphenes, allowing for consistent perception. To evaluate whether our model achieves this, we cluster the target images and resulting phosphenes using t-SNE [58] shown in Fig. SC. The ground truth images form clusters corresponding to the digits 0-9. The phosphenes from our encoder roughly form similar, slightly less separated groupings, whereas the naive phosphenes do not. To ensure that this was not the result bad 1-SNE hyperparameters, we repeated the clustering across different perplexities and learning rates, obtaining similar worse results.
7

---

Target COCO
Nake HNA
Figure 4: Original (top row), segmented (second row), and reconstructed targets for the COCO dataset, for both HNA (third row) and naive encoders (bottom row). Left to right within each block of 4 images, P takes values of 200, 400, 600, 800. Left to right across blocks, À takes values of 250, 750, 1250, 2000, Note that the brightness of the naive method is clipped for display.
A
3
6
B
HINA
Naive
A
600
1060
1525 2000
100
575
1050
1525
2000
SO0C Bot
100
3
3
3
400
275
3
3
3
3
6
HNA
3
3
3
3
6
250
750
1250
1/50
2b0
/30
1250
1/50
60
3
3
3
3
6
80
3
3
3
3
6
MNIST
HNA
Naive
Nalve Approach
100
3 3
3
3
275
3 3 3
100 025 1d0
Figure 5: Encoder performance across simulated patients (varying P and A) on the MNIST dataset. A:
Target, HNA encoder, and naive encoder phosphenes for two example digits. B: Heatmaps showing the log joint loss across p and X for HNA and naive encoders. C: T-SNE clusterings on original MNIST targets, HNA reconstructed phosphenes, and naive reconstructed phosphenes.
5.4 Joint Perceptual Error Ablation Study
To show that the joint perceptual metric performs better than any of its individual components, train models using just the VGG loss and just MAE loss. Shown are values for p=150 and A=600. As mentioned previously, encoders trained using just VGG loss fail to converge, thus pretrain the VGG encoder using MAE and smoothing loss, then transition to using only VGG. We do not consider ablating the smoothing term (Eq. 6) because it is simply regularization term. Fig. 6 shows the phosphenes produced by HNA trained on the joint, VGG-only, and MAE-only loss.
The VGG encoder had test VGG loss of 4% lower than the joint model, but its produced phosphenes are oversmoothed and blurry. The MAF encoder had a final test MAE of 9% lower than the joint model, but its produced phosphenes are disconnected and low-quality. The joint model had a RA of 99.0%, the VGG encoder had a RA of 95.9%, and the joint model had a RA of 77.6%
Discussion
Visual Prostheses We found that HNA is able to produce high-fidelity stimuli from the MNIST and COCO datasets that outperform conventional encoding strategies across all tested conditions. Importantly, HNA produces phosphenes that are consistent across representations of the same object (Fig. 5C), which is critical to allowing prosthesis users to learn to associate certain visual patterns with specific objects. On the MNIST task, HNA produced high quality reconstructions, nearly matching the targets (Figure 3). On the harder COCO task, HNA significantly outperformed the naive encoder, but was still unable to capture all of the detail in the images. Appendix G, we demonstrate that this is largely due to the implant's 's limited spatial resolution and not fundamental limitation of HNA.
8

---

VGG Joint Target
9
2
1
3
X
3
3
6
7
2
8
9
2
3
!
X
3
3
6
7
8
9
3
1
3
5
3
6
7
MAE
:
3
Figure 6: MNIST images for HNA encoders trained using the joint, VGG-only, and MAE-only loss.
Another advantage of the HNA is that it can be trained to predict stimuli across a wide range of patient-specific parameter values é, whereas the conventional naive encoder works well only for small values of and A. This may be one reason why the naive encoding strategy has been shown to lead to substantial individual differences in visual outcomes [19, 59]. Our results suggest that stimuli produced with HNA may be able to reduce at least some amount of this patient-to-patien variabilityy
Furthermore, HNA also proved superior surrogate forward model. The latter offer alternative when the forward model is computationally expensive or not differentiable. Understandably, any inaccuracies in the surrogate model will propagate the learned encoder during training. However, we observed that even for well trained surrogates, the encoder may still learn to exploit the inexact surrogate instead of learning invert the true model (see Appendix A). It is possible that this exploitation could be mitigated to some extent by adversarially- robuus training techniques [60]. We suspect that the surrogate method's inferior performance here compared to [10] can be explained by our larger stimulus search space. Thus, we cannot currently suggest HNA for surrogate forward models, unless the forward model is sufficiently simple or has small stimulus space.
Deployment HNA encoders must be lightweight enough to be deployed in resource-limice neu- roprosthetic environments. Our encoder's s single image inference time was 1.2ms on GPU and 4ms on CPU. Future work could reduce these numbers through network pruning, mixed precision, and architecture search. Low-power Edge Al accelerators (e.g., Intel's Neural Compute Stick) and dedicated neuromorphic hardware (e.g., BrainChip's Akida SoC) may provide another solution.
Broader Impacts While our work is presented in the context of visual prostheses, the HNA framework may apply any sensory neuroprosthesis where stimulus selection can be informed by a numeric or symbolic forward model. For example, HNA could be used in cochlear implants [3] to choose stimuli that result desired sound, and in spinal cord implants [15] to find the best way to relay neural signals through damaged section of the spinal cord. Conveniently, the forward models required by HNA have already been developed for a range of applications [38-48]. However, HNA might not apply to all neural interfaces, such as systems without clear neural or perceptual target (e.g., deep brain stimulation for the treatment of Parkinson's 's [61]) or closed-loop systems [16, 62].
Limitations Despite HNA's 's potential, the current implementation has number of limitations. First, as presented the HNA encoder only applies to static targets. Hence dynamic targets must be split into individual frames and encoded separately. However, one approach might be encode entire stimulus sequences (instead of frames) that are optimized to reconstruct the dynamic target sequence.
Second, HNA works best if there is accurate forward model mapping from stimulus space to perception. However, Appendix H shows that HNA may still give benefits over a naive encoding even when patient-specific parameters are unknown mis-specified. In general, if a prosthesis elicits similar results across patients, then a non-patient-specific model would suffice.
Third, the current works deals only with simulated patients. The use of DNN for stimulus encoding in real patients may raise safety concerns. Since we cannot examine the process by which stimuli are chosen, it is possible that HNA might produce harmful stimuli that could lead to serious adverse events (e.g., seizures). However, this concern is mitigated by the fact that most neuroprostheses are equipped with firmware responsible for ensuring stimuli stay within FDA-approved safety limits.

---

Conclusion
In summary, this paper proposes hybrid autoencoder structure as general framework for stimulus optimization in sensory neuroprostheses and, as proof of concept, demonstrates its utility on the prominent example of visual neuroprostheses, drastically outperforming conventional encoding strategies. This constitutes an important step towards the quality vision to people living with incurable blindness and may prove a promising solution for variety of neuroprosthetic technologies.
challenge of restoring high-
8 Acknowledgements
This work was supported by the National Institutes of Health (NIH ROO EY-029329 to MB).
10

---

References
[1] Caterina Cinel, Davide Valeriani, and Riccardo Poli. Neurotechnologies for Human Cognitive Augmen- tation: Current State of the Art and Future Prospects. Frontiers in Human Neuroscience, 13:13, January 2019.
[2] Eduardo Feraandcz. Development of visual Neuroprostheses: trends and challenges. Bioelectronic Medicine, 4(1):12, August 2018.
[3] Blake S. Wilson, Charles C. Finley, Dewey T. Lawson, Robert Wolford, Donald K. Eddington, and William M. Rabinowitz. Better speech recognition with cochlear implants. Nature, 352163332)2236-2388 July 1991. Number: 6332 Publisher: Nature Publishing Group.
[4] Eduardo Fernández, Aramtxa Alfaro, Cristina Soto-Sánchez, Pablo Gonzalez-Lopez, Antonio M. Lozano, Scbastian Peña, Maria Dolores Grima, Alfonso Rodil, Bernardeta Gómez, Xing Chen, Picter R. Roelfsema, John D. Rolston, Tyler S. Davis, and Richard A. Normann. Visual percepts evoked with an intracortical 96-channel microelectrode array inserted in human occipital cortex. Journal of Clinical Invesigation, 131(23)ee111331 December 2021.
[5] Yvonne Hsu-Linn Luo and Lyndon da Cruz. The Argus@ Retinal Prosthesis System. Progress in Retinal and Eye Research, 50:89-107, January 2016.
[6] Daniel W. Tan, Matthew A. Schiefer, Michael W. Keith, James Robert Anderson, Joyce Tyler, and Dustin J. Tyler. A neural interface provides long-term stable natural touch perception. Science Transla- tiomal Medicine, (1257)2577a18-2257aall October 2014. Publisher: American Association for the Advancement of Science.
[7] Cordelia Erickson- Brickson-Davis and Helma Korzybska. What do blind people "see" with retinal prostheses? Observations and qualitative reports of epiretinal implant users. PLOS ONE, 16(2):e022118, February 2021. Publisher: Public Library of Science.
[8] Craig D. Murray. Embodiment and Prosthetics. In Pamela Gallagher, Deirdre Desmond, and Malcolm MacLachlan, editors, Psychoprosthetics, pages 119-129 Springer, London, 2008.
Bingni W. Brunton and Michael Beyeler. Data-driven models in human neuroscience and neuroengineering. Current Opinion in Neurobiology. 58:21-29, October 2019.
[10] Lucas Relic, Bowen Zhang, Yi-Lin Tuan, and Michael Beyeler. Deep Learning-Basel Perceptual Stimulus Encoder for Bionic Vision. In Augmented Humans 2022, AHS 2022, pages 323-325, New York, NY, USA, March 2022. Association for Computing Machinery.
[11] David Montes Oca Zapiain, James A. Stewart, and Rémi Dingreville. Accelerating phase-field-based microstructure evolution predictions via surrogate models trained by machine learning methods. upj Computational Materials, 7(1):1111, January 2021. Number: 1 Publisher: Nature Publishing Group.
[12] Mohammad Amin Nabian and Hadi Meidani. A Deep Neural Network Surrogate for High- lingh-Dimessional Random Partial Differential Equations. Probabilistic Engineering Mechanics, 57:14-25, July 2019. arXiv:1806.002. [physics, stat].
[13] Stefanos Nikolopoulos, Ioannis Kalogeris, and Vissarion Papadopoules. Non-intrusive surrogate mod- eling for parametrized time-dependent partial differential equations using convolutional autoencoders. Engineering Applications of Artificial Intelligence, 109:104652, March 2022.
[14] Gregg A. Tabot, John F. Dammann, Joshua A. Berg, Francesco V. Tenore, Jessica I.. Boback, R. Jacob Vogelstein, and Sliman J. Bensmaia. Restoring the sense of touch with prosthetic hand through a brain interface. Proceedings of the National Academy Sciences, 110045):182991888 18284, November 2013.
[15] Marco Capogrosso, Tomislav Milekovic, David Borton, Fabien Wagner, Eduardo Martin Moraud, Jean- Baptiste Mignardot, Nicolas Buse, Jerome Gandar, Quentin Barraud, David Xing. Elodie Rey, Simone Duis. Yang Jianzhong. Wai Kin D. Ko, Qin Li, Peter Detemple, Tim Denison, Silvestro Micera, Erwan Bezard, Jocelyne Bloch, and Grégoire Courtine. A brain spine interface alleviating gait deficits after spinal cord injury primates. Nature, 539(7628):244 288, November 2016.
[16] Christopher A. R. Chapman, Noah Goshi, and Erkin Seker. Multifunctional Neural Interfaces for Closed- Loop Control of Neural Activity. Advanced Functional Materials, 28(12):1703523, 2018. _eprint: http://ollwelbbbbayyyyweooooo//b http://wwoooooioooooo////o//m... 201703333.

---

[17] Fabien B. Wagner, Jean- CaattBtp Mignardot, Camille G. Le Goff-Mignardot. Robin Demesmaeker, Salif Komi, Marco Capogrosso, Andreas Rowald, Ismael Seáñeez, Miroslav Caban, Elvira Pirondini, Molywan Vat, Laura A. McCracken, Roman Heimgartner. Isabelle Fodor, Anne Watrin, Perrine Seguin, Edoardo Paoles, Katrien Van Den Keybus, Grégoire Eberle, Brigitte Schurch, Etienne Pralong, Fabio Becce, John Prior, Nicholas Buse, Rik Buschman, Esra Neufeld, Niels Kuster, Stefano Carda, Joachim von Zitzewitz, Vincent Delattre, Tim Denison, Hendrik Lambert, Karen Minassian, Jocelyne Bloch, and Grégoire Courtine. Targeted neurotechnology restores walking in humans with spinal cord injury. Nature, 563(7729):65-71, November 2018
[18] Lauren N. Ayton, Nick Barnes, Gislin Dagnelie, Takashi Fujikado, Georges Gaetz, Ralf Hornig, Bryan W. Jones, Mahiul M. K. Muqit, Daniel L. Rathbun, Katarina Stingl. James D. Weiland, and Matthew A. Petoe. An update on retinal prostheses. Clinical Nearophysiology 13):113383319 June 2020.
[19] Katarina Stingl. Ruth Schippert, Karl U. Bartz-Schmidi Dorothea Besch, Charles L. Cottriall, Thomas Edwards, Florian Gekeler. Udo Greppmaier, Katja Kiel, Assen Koitschev, Laura Küihlewein, Robert E. MacLaren. James D. Ramsden, Johann Roider, Albrecht Rothermel, Helmut Sachs. Greta S. Schröder. Jan Tode, Nicole Troelenberg, and Eberhart Zrenner. Interim Results of a Multicenter Trial with the New Electronic Subretinal Implant Alpha AMS in 15 Patients Blind from Inherited Retinal Degenerations. Frontiers in Neuroscience, 11, 2017. Publisher: Frontiers.
[20] Lewis Karapanos, Carla J. Abbott, Lauren N. Ayton, Maria Kolic, Myra B. McGuinness, Elizabeth K. Baglin, Samuel A. Titchener, Jessica Kvansakul, Dean Johnson, William G. Kentler, Nick Barnes, David A. X. Nayagam, Penelope J. Allen, and Matthew A. Petoe. Functional Vision in the Real-World Environ- ment With a Second- Recond-Generationn (44-Channe) Suprachoroidal Retinal Prosthesis. Translational Vision Science & Technology, 10(10)7777, August 2021. Publisher: The Association for Research in Vision and Ophthalmology.
[21] Wm H. Dobelle. Artificial Vision for the Blind by Connecting a Television Camera to the Visual Cortex. ASAIO Journal, 46(1):3-9, February 2000.
[22] Michacl Beyeler, Devyani Nanduri, James D. Weiland, Ariel Rokem, Geoffrey M. Boynton, and lonc Fine. A model of ganglion axon pathways accounts for pereepts elicited by retinal implants. Scientific Reports, 9(1):1-16, 16, June 2019.
[23] J. Rizzo, Wyatt, Loewenstein, Kelly, and D. Shire. Perceptual efficacy of electrical stimulation of human retina with a microelectrode array during short-term surgical trials. Invest Ophthalmol Vis Sci, 44(12):5362-9, December 2003.
[24] Jacob Granley and Michael Beyeler. A Computational Model of Phosphene Appearance for Epiretinal Prostheses. In 202/ 43rd Annual International Conference of the IEEE Engineering in Medicine Biology Society (EMBC). pages 4477-4481, November 2021. ISSN: 2694-0604.
[25] Yvonne H-L. Luo, Joe Jiangjian Zhong, Monica Clemo, and Lyndon da Cruz. Long-term Repeatability and Reproducibility of Phosphene Characteristics in Chronically Implanted Argus II Retinal Prosthesis Subjects. American Journal of Ophthalmology, 170:100 109, October 2016.
[26] M. Beyeler, A. Rokem, G. M. Boynton. and I. Fine, Learning to see again: biological constraints on cortical plasticity and the implications for sight restoration technologies. J NNeural Eng, 14(5):05 1003, June 2017.
[27] S. C. Chen, G. J. Suaning, J. W. Morley, and N. H. Lovell. Simulating prosthetic vision: I. Visual models of phosphenes. Vision Research, 49( (12)) 1493-566, June 2009.
[28] Wei Tong, Hamish Meftin, David Garrett, and Michael R Ibbotson. Stimulation strategies for improving the resolution of retinal prostheses. Frontiers in neuroscience, 14:262, 2020.
[29] Nishal P. Shah, Sasidhar Madugula, Lauren Grosberg, Gonzalo Mena, Pulkit Tandon, Pawel Hottowy, Alexander Sher, Alan Litke, Subhasish Mitra, and E.J. Chichilnisky. Optimization of Electrical Stimulation for a High-Fidelity Artificial Retina. In 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER), pages 714-718, March 2019. ISSN: 1948-3554.
[30] Dorsa Haji Ghaffari, Yao-Chuan Chang, Ehsan Mirzakhalili, and James D. Weiland. Closed-loop Opti- mization of Retinal Ganglion Cell Responses to Epiretinal Stimulation: A Computational Study. In 202/ 10th International JEEE/EMBS Conference Neural Engineering (NER), pages 597-600, May 2021. ISSN: 1948-3554,
[31] Tristan Fauvel and Matthew Chalk. Human-in ieetan--i-the-looe optimization of visual prosthetic stimulation. preprint, Neuroscience, November 2021.
12

---

[32] Martin J. Spencer, Tatiana Kameneva, David B. Grayden, Hamish Meffin, and Anthony N. Burkitt. Global activity shaping strategies for retinal implant. Journal of Neural Engineering, 16(2):026008, January 2019, Publisher: IOP Publishing.
[33] Jaap de Ruyter van Steveninck, Umut Güşlü, Richard van Wezel, and Marcel van Gerven. End-to-end optimization of prosthetic vision. Journal of Vision, 22(2):20, February 2022.
[34] Robert E Mare, Bryan W Jones, Carl B Watt, and Enrica Strettoi. Neural remodeling in retinal degeneration. Progress in Retinal and Eye Research, 22(5):607-655 2003.
[35] Zhou Wang and Alan C. Bovik. Mean squared error: Love it or leave it? A new look at Signal Fidelity Measures. IEEE Signal Processing Magazine, 26(1 26(1):98-4)) ):91-117, January 2009. Conference Name: IEEE Signal Processing Magazine.
[36] Guangtao Zhai and Xiongkuo Min. Perceptual image quality assessment: a survey. Science China Information Sciences, 63(1 1):213301, November 2020.
[37] Sergey loffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reduc- ing Internal Covariate Shift. Technical Report [es] type: article.
arXiv, March 2015. arXiv:1502.03167
[38] Michael F Dorman, Anthony J Spahr, Philipos C Loizon, Cindy J Dana, and Jennifer S Schmidt. Acoustic simulations of combined electric and acoustic hearing (eas). Ear and Hearing, 26(4):371-380, 2005.
[39] Mario A Svirsky, Nai Ding, Elad Sagi, Chin-TTuc Tan, Matthew Fitzgerald, Katelyn Glassman, Keena Seward, and Arlene C Neuman. Validation of acoustic models of auditory neural prostheses. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pages
IEEE, 2013.
[40] MF Dorman, PC Loizou, A Spahr, and CJ Dana. Simulations of combined acousticfelectric hearing. In Proceedings of the 25th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (IEEE Cat. No. 03CH37439), volume 3. pages 1999- 2001. IEEE, 2003.
[41] Michael F Dorman, Philipos C Loizou, and Dawne Rainey. Speech intelligibility as a function of the number of channels of stimulation for signal processors using sine-wave and noise-band outputs. The Journal of the Acoustical Society of America, 102(4):2403- 24111, 1997.
[42] William B Cooper, Emily Tohey, and Philipos C Loizou. Music perception by cochlear implant and normal hearing listeners measured by the montreal battery for evaluation of amusia. Ear and hearing, 29(4):618 81, 2008.
[43] Philipos C Loizou, Michael Dorman. Oguz Poroy, and Tony Spahr. Speech recognition by normal- hearing and cochlear implant listeners a function of intensity resolution. The Journal of the Acoustical Society of Americo, 10865)23377-2887 2000.
[44] Hannes P Saal, Benoit P Delhaye, Brandon C Rayhaun, and Sliman J Bensmaia. Simulating tactile signals from the whole hand with millisecond precision. Proceedings of the National Academy of Sciences. E1422) 2017.
[45] Elizaveta V Okorokova, Qinpu He, and Sliman J Bensmaia. Biomimetic encoding model for restoring touch in bionic hands through nerve interface. Journal of neural engineering, 15(6):066033, 2018.
[46] Douglas J Weber, Rebecca Friesen, and Lee E Miller, Interfacing the somatosensory system to restore touch and proprioception: essential considerations. Journal of motor behavior, 44(6):403-15 2012.
[47] Sung Soo Kim, Arun P Sripati, and Sliman J Bensmaia. Predicting the timing of spikes evoked by tactile stimulation of the hand. Journal of neurophysiology. 104())1484-1499) 2010.
[48] Milana Mileusnic, Ian E Brown, Ning Lan, and Gerald E Loeb. Mathematical models of proprioceptors. i. control and transduction in the muscle spindle. Journal of neurophysiology, 96(4): 1772-1887 2006.
[49] N. Jansonius, Nevalainen, B. Selig.
Zangwill, A. Sample, W. M. Budde, B. Jonas, W. A. Lagrèze, P. J. Airaksinen, R. Vonthein, L. A. Levin, J. Paetzold, and U. Schiefer. A mathematical description of nerve fiber bundle trajectories and their variability in the human retina. Vision Research, (1(17)2117-2163 August 2009.
13

---

[50] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp. Geoffrey Irving, Michael Isard, Yangging Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg. Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals. Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning heterogeneous systems, 2015. Software available from tensorflow.org.
[51] Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. Technical Report arXiv:1409.1555 arXiv, April 2015. arXiv:1409. [es] type: article.
[52] Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Ming-Hsuan Yang. Universal Style Transfer Feature Transforms. In Advances Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[53] Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, June 2009. ISSN: 1063-6919.
[54] Sylvain Paris, Samuel W Hasinoff, and Jan Kautz. Local Laplacian Filters: Edge-aware Image Processing with Laplacian Pyramid. Communications of the ACM, 58:11, 2015.
[55] Li Deng. The mnist datahase of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 1916:1141-12,, 2012.
[56] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft COCO: Common Objects in Context. Technical Report arXiv:1405.0312 arXiv, February 2015. arXiv:1405.0312 [es] type: article.
[57] Nicole Han, Sudhanshu Srivastava, Aiwen Xu, Devi Klein, and Michael Beyeler. Deep Learning- Based Scene Simplification for Bionic Vision. In Augmented Humans Conference 2021, Alls 21, pages 45-54, New York, NY, USA, February 2021. Association for Computing Machinery.
[58] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
[59] Eli Peli. Testing Vision Is Not Testing For Vision. Translational Vision Science & Technology, 913):32-32, December 2020. Publisher: The Association for Research in Vision and Ophthalmology.
[60] Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Bonch, and Patrick McDaniel. Ensemble adversarial training: Attacks and defenses. arXiv preprint arXiv:1705.0722 2017.
[61] Alim Louis Benabid. Deep brain stimulation for Parkinson's 's disease. Current Opinion in Neurobiology, 1366666,, December 2003.
[62] Bardia Bozorgzadch, Douglas R. Schuweiler, Martin J. Bobak, Paul A. Garris, and Pedram Mohseni. Neurochemostat: Neural Interface SoC With Integrated Chemometries for Closed-Loup Regulation of Brain Dopamine. IEEE Transactions on Biomedical Circuits and Systems, 10(3)6554-667, June 2016. Conference Name: IEEE Transactions on Biomedical Circuits and Systems.
[63] M. Beyeler, G. M. Boynton, Fine, and A. Rokem. pulse2percept: A Python-based simulation framework for bionic vision. In K. Huff, D. Lippa, D. Niederhut, and M. Pacer, editors, Proceedings of the 16th Science Python Conference, pages 81-88, 2017.
[64] Ilya Loshchilov and Frank Hutter. arXiv:111.051001.22 2017.
Decoupled weight decay regularization.
arXiv preprint
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] we highlight both the limitations
of our general framework and our specific proof of concept, see Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 6, last paragraph
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] We do not present any theoretical results
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] We include the trained models, and code to reproduce the main result figures in the supplementary materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] detailed instructions are given in Section 4, in the training/opumiization subsection, which describes hyperparameter selection, learning rates, and many other training details. Data splits were predetermined by the datasets (MNIST and COCO already have splits), with some more details given in Appendix
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [No] We do not train models multiple times. Once trained, the networks produce deterministic results.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] This is specified in Section training paragraph
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] We used models
from [10, 22, 24), which are all cited whenever used in the paper.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either the supplemental material or as URL? [Yes] we provide saved weights for our final trained models, along with code to display results shown on main results figures.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] We do not use any personal data
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] The data used does not contain any identifying information or offensive content.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "XcWXX2XA57k", "reviewer": "Reviewer_UeDa", "summary": "The manuscript presents a encoder to generate optimal stimulation parameters for a visual prosthesis. Specifically, the focus of the paper is on developing an end-to-end deep learning model that is trained to invert a known fixed-forward model.  ", "questions": "Please check the weakness and strengths. ", "limitations": "Check Weakness. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n1. The idea is quite interesting on using HNA for mapping input images to stimulation parameters. \n2. It can increase the accuracy of retinal prosthesis and in general other sensory feedback systems. \n\nWeakness\n There are several questions that I would invite the authors to address\n\n1.\tThe results and metrics are based on a forward model designed in 19. If the forward model changes, would you need to update the encoder?\n2.\tIf the forward model was not fixed how would the HNA perform? From a real retinal implant perspective due to plasticity in visual cortex would the stimulation parameters learnt using a fixed decoder change?\n3.\tAfter Figure 1C can there be a Figure 1D to show how would this be deployed since there will not real time perceptual loss in case of patients (maybe some qualitative metrics but a quantitative loss)?\n4.\tFigure 3 can the authors add how the stimulation parameters for electrodes looks like for a given MNIST image. Say for 0 input what is the output of the gride of an electrode while using HNA, Surrogate and Naïve. \n5.\tWhat is computational burden of these Encoder? Since they are for implantable applications this would be an important metric. Latency for such computing these models. Definitely cannot use NVIDIA GPU for the implanted application.  How would this scale. \nMinor issues:\nLine 100 is not formatted correctly.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "WO58tIsp7J", "reviewer": "Reviewer_kHVG", "summary": "The authors propose to use an autoencoder trained to invert a known feedforward model that approximates the biological network underlying the early human visual system.  They demonstrate the efficacy in application to visual prosthetics and show that there method leads electrode activation patterns that produce better and more intelligible decodings downstream in the biological system than comparable methods.  They claim their system to be generalizable to any type of sensory neuroprostheses (with obvious remodeling and retraining required).", "questions": "\nThe author's described the limitations of the surrogate model and how even surrogates that perform well at the task have weakness that are able to be exploited by the encoding-decoding paradigm.\nDo the author's think the failure of their surrogate model is possibly due to the low number of percept images used to constrain the deep neural network (50000 images may be enough to constrain for MNIST classification but as a downstream model of biological encoding does not seem to be nearly enough to tightly constrain filter the model)?\n\nI know this is not the contribution of this work, but a bit more discussion of where the patient-specific data comes from in the text would be helpful for the reader.", "limitations": "The authors are upfront about the potential limitations and societal impacts of the work.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "They use a nice approach of treating the pseudo inverse of a known biological function as the encoder of an encoder decoder pair and learn the encoder given patient specific parameters.\n\nThe model of the biology, and treatment of the problem of phosphenes is very elegant, and works well.  It does suggest that generalizability of the methodology requires equally detailed biological forward models in other neuroprosthetic applications.\n\nThe results presented (especially in figure 3) are very impressive, but there should be more clear caveats that this methodology requires a patient specific model of phosphenes and different patients have quite different observed phosphenes.  Even with the reduction of patient to patient variability on MNIST introduced by this method, I would not anticipate as much success reducing variability on the already harder task of COCO like images.  It should also be noted that all of the evaluation here is done on a model of individual patients, and at no time are patients actually asked to observe these stimulation patterns and decode them downstream.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "9sE2rELZre9", "reviewer": "Reviewer_cH4d", "summary": "This paper develops an autoencoder to solve the inverse problem of converting incoming visual stimuli into electrical stimulation patterns such that the responses evoked by the electrical stimulation mimics the target visual stimulus. \n\nBy using an accurate decoding model, perceptually accurate loss function and training across multiple participants, the encoder outperforms existing approach for second-sight retinal prosthesis. ", "questions": "1. Is it hard to directly solve Eq. 1? Since the number of parameters in $\\mathbf{s}$ is not large (15 x 15 x 3), it might be possible to solve Eq. 1 directly. Why do you need to learn an amortized encoder? Atleast this will give an upper bound on HNA performance. \n\n2. How accurately are the patient specific parameters ($\\phi$ ) known? What are the consequences of inaccuracies in their estimation? Like the failure of surrogate model, does HNA fail if encoder and decoder have different estimates of $\\phi$ ? If  $\\phi$ is estimated incorrectly / changes over time, then this question becomes relevant. \n\n", "limitations": "Limitations are described adequately.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. Well written.\n2. Advances state-of-the-art in the artificial retina application and shows how their approach is better using overall convincing analysis. \n\nWeakness: \n1. A lot of innovation (particular architecture of encoder/decoder) relies on specifics of crude retinal prosthesis. I am not sure how general purpose the approach presented here is. Does the model of phosphenes translate to other neural stimulation systems like somatosensory / spinal cord stimulation? \n\n2. The approach only works for static images as visual targets. How does the autoencoder framework extend to dynamics targets? \n\n3. The decoder description on page 5 is short. Might help to improve it for broader applicability. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept