Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper develops an autoencoder to solve the inverse problem of converting incoming visual stimuli into electrical stimulation patterns such that the responses evoked by the electrical stimulation mimics the target visual stimulus. 

By using an accurate decoding model, perceptually accurate loss function and training across multiple participants, the encoder outperforms existing approach for second-sight retinal prosthesis. 
strengths_and_weaknesses: Strengths:

1. Well written.
2. Advances state-of-the-art in the artificial retina application and shows how their approach is better using overall convincing analysis. 

Weakness: 
1. A lot of innovation (particular architecture of encoder/decoder) relies on specifics of crude retinal prosthesis. I am not sure how general purpose the approach presented here is. Does the model of phosphenes translate to other neural stimulation systems like somatosensory / spinal cord stimulation? 

2. The approach only works for static images as visual targets. How does the autoencoder framework extend to dynamics targets? 

3. The decoder description on page 5 is short. Might help to improve it for broader applicability. 
questions: 1. Is it hard to directly solve Eq. 1? Since the number of parameters in $\mathbf{s}$ is not large (15 x 15 x 3), it might be possible to solve Eq. 1 directly. Why do you need to learn an amortized encoder? Atleast this will give an upper bound on HNA performance. 

2. How accurately are the patient specific parameters ($\phi$ ) known? What are the consequences of inaccuracies in their estimation? Like the failure of surrogate model, does HNA fail if encoder and decoder have different estimates of $\phi$ ? If  $\phi$ is estimated incorrectly / changes over time, then this question becomes relevant. 


limitations: Limitations are described adequately.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The manuscript presents a encoder to generate optimal stimulation parameters for a visual prosthesis. Specifically, the focus of the paper is on developing an end-to-end deep learning model that is trained to invert a known fixed-forward model.  
strengths_and_weaknesses: Strength
1. The idea is quite interesting on using HNA for mapping input images to stimulation parameters. 
2. It can increase the accuracy of retinal prosthesis and in general other sensory feedback systems. 

Weakness
 There are several questions that I would invite the authors to address

1.	The results and metrics are based on a forward model designed in 19. If the forward model changes, would you need to update the encoder?
2.	If the forward model was not fixed how would the HNA perform? From a real retinal implant perspective due to plasticity in visual cortex would the stimulation parameters learnt using a fixed decoder change?
3.	After Figure 1C can there be a Figure 1D to show how would this be deployed since there will not real time perceptual loss in case of patients (maybe some qualitative metrics but a quantitative loss)?
4.	Figure 3 can the authors add how the stimulation parameters for electrodes looks like for a given MNIST image. Say for 0 input what is the output of the gride of an electrode while using HNA, Surrogate and Naïve. 
5.	What is computational burden of these Encoder? Since they are for implantable applications this would be an important metric. Latency for such computing these models. Definitely cannot use NVIDIA GPU for the implanted application.  How would this scale. 
Minor issues:
Line 100 is not formatted correctly.

questions: Please check the weakness and strengths. 
limitations: Check Weakness.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose to use an autoencoder trained to invert a known feedforward model that approximates the biological network underlying the early human visual system.  They demonstrate the efficacy in application to visual prosthetics and show that there method leads electrode activation patterns that produce better and more intelligible decodings downstream in the biological system than comparable methods.  They claim their system to be generalizable to any type of sensory neuroprostheses (with obvious remodeling and retraining required).
strengths_and_weaknesses: They use a nice approach of treating the pseudo inverse of a known biological function as the encoder of an encoder decoder pair and learn the encoder given patient specific parameters.

The model of the biology, and treatment of the problem of phosphenes is very elegant, and works well.  It does suggest that generalizability of the methodology requires equally detailed biological forward models in other neuroprosthetic applications.

The results presented (especially in figure 3) are very impressive, but there should be more clear caveats that this methodology requires a patient specific model of phosphenes and different patients have quite different observed phosphenes.  Even with the reduction of patient to patient variability on MNIST introduced by this method, I would not anticipate as much success reducing variability on the already harder task of COCO like images.  It should also be noted that all of the evaluation here is done on a model of individual patients, and at no time are patients actually asked to observe these stimulation patterns and decode them downstream.
questions: 
The author's described the limitations of the surrogate model and how even surrogates that perform well at the task have weakness that are able to be exploited by the encoding-decoding paradigm.
Do the author's think the failure of their surrogate model is possibly due to the low number of percept images used to constrain the deep neural network (50000 images may be enough to constrain for MNIST classification but as a downstream model of biological encoding does not seem to be nearly enough to tightly constrain filter the model)?

I know this is not the contribution of this work, but a bit more discussion of where the patient-specific data comes from in the text would be helpful for the reader.
limitations: The authors are upfront about the potential limitations and societal impacts of the work.
review_title: null

Meta-review: metareview: This paper formulates the problem of learning how to stimulate a visual neuroprosthesis as a hybrid autoencoder.  While the decoder can be taken as a known and fixed model that describes how stimuli produce percepts, the encoder needs to be learned.  Once learned the encoder maps target percepts into stimuli that can be passed into the device (decoder).

Motivation and formulation of the problem is especially clear / strong.  The paper is well written and the reviewers and I appreciated the nice solution strategy for a potentially impactful application area.  There were some concerns about how generally applicable the approach is.  However, the results presented likely do advance the state of the art in this setting.

Given my own reading of the paper and the consistently positive reviewer scores, I'm very comfortable endorsing this paper for acceptance.
confidence: Certain
Decision: Accept