Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Similarly to Parthasarathy/Batty 2017 and Warland 1997, this paper proposes a method to reconstruct visual stimuli from retinal ganglion cell electrophysiological measurements.

It proposes to combine a forward model and a prior in a MAP framework to obtain the stimulus image with maximum a posteriori probability.

For the forward model it proposes a baseline linear-nonlinear-Poisson model and the recurrent model from Pillow 2008. For the prior a frequency power law Gaussian prior and a denoising neural network prior (implicit through plug and play method) are used.
Combinations of forward model and and prior are evaluated.

It is shown that reconstructions using the recurrent model and the denoising neural network prior exceed other works in terms of SSIM similarity with the stimulus image.


strengths_and_weaknesses: This paper does solid work in combining existing methods into a new method that improves performance in the particular application of reconstructing stimuli from retinal ganglion cell activity.
In terms of numbers, the demonstrated improvement in terms of SSIM is marginal, but perceptually the reconstructions are more compelling than the alternative methods (my general experience with SSIM has been that there is no chance of it capturing such subtleties).

My main concern is about the interaction of injecting strong priors and the declared purpose of this method. If you are concerned about "how the brain might interpret the signal", is it a good idea to inject ever stronger naturalistic priors to fill in the missing information? Granted, the brain is good at such completion inference, too. But there is no reason to believe that the priors are in any way the same. Relatedly, I would be worried that the prior adds nonexistent details as briefly mentioned in the paper. While this does become visible in some of the reconstructions, it should also be noted that, reassuringly (or not?), it is not to an extent that the reconstruction metric against the ground truth is affected by it.

While the MAP framework is appealing because it allows designating a prior and a forward model, it also leaves an aftertaste of serving only the narration. E.g. it is a known pitfall to name L1-regularization a "sparsity prior", since only the MAP estimate will be sparse but no typical sample from the posterior distribution. All ideas presented here would also easily fall under the less semantically laden framework of regularized risk estimation, in which case the denoising step would replace the proximal operator of the regularizer.


questions: What about using the model in Batty, ICLR2017 "Multilayer recurrent network models of primate retinal ganglion cell responses" as a forward model?

In Parthasarathy/Batty 2017 it is shown for the convolutional autoencoder that its performance is strongly dependent on the data set used for training. My guess would be that this could be less the case for the denoising network used here. Could this be evaluated?

The abstract juxtaposes (non-)linear regression and MAP estimation, which are largely orthogonal concepts. Since even in this contribution a regression model is used for prediction, this should probably be rephrased.
limitations: Some limitations are discussed - I'd say this discussion is adequate. The scope of this work has no immediate negative societal impact, but may lead to some positive medical research outcomes down the line.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose an inference method for reconstructing images presented to retinas from the spiking activity of Retinal Ganglion Cells (RGCs). This is achieved using a Maximum A Posteriori (MAP) estimation where the posterior distribution is obtained from the combination of a likelihood modeling key aspects of RGCs (individually and as a network) and a prior for natural images.

The authors detail their models and optimization methods. Then they compare, both qualitatively and quantitatively, variation of their algorithms and also to previous state-of-the-art (SOTA) models. Their best model reaches SOTA performances on some scores. 

strengths_and_weaknesses: Strengths : 
- reconstructing images from neural activity is key to understand both natural images and neural activity, it is also a good task to evaluate models of biological neural networks,
- the methods used are recent or fairly recent and originally combined (FISTA, HQS, dCNN priors, ...),
- the results are good (reaching SOTA),
- the model could be used in combination with retinal implants to restore vision.

Weaknesses :
- low risk (dCNN priors have shown to be powerful in a variety of tasks)
questions: I am a bit surprised by the first sentence of the introduction. How do the authors know that only a small portion of visual information arriving at each eyes is transmitted to the brain ? I would have appreciated a reference here as "information" is a clearly defined concept in signal processing. Maybe the authors wanted to strike the reader by the existing contrast between the torrent of light or photons arriving at the retina and the somehow discrete nature of spiking activity ? Please add a ref or rephrase appropriately. The same remark hold for the first sentence of the abstract.

Apart from this minor remark, I think this is a well-written paper and a nice piece of work despite the results not being surprising.
limitations: The authors mention the negative impact of animal experimentation. 
It is not clear if the data are openly accessible. If not, the authors should make them openly accessible to maximize the value of the data in many aspects (science, animals involved, ...)
The proposed work involve neural network training. The authors have used a single GPU but how long the training lasted for each model ? Does the performance gain justify the probable increase in energy consumption compared to other models ?
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a novel method for reconstructing natural images from recorded spikes of a several hundred retinal ganglion cells (RGC) by combining a generalized linear model (an augmentation of a linear-nonlinear-Poisson model)  encoded with the spike responses of the RGCs, along with image priors embedded in a deep, denoising CNN. Their combined model outperforms current state of the art that applies a deep convolutional autoencoder to linearly-reconstructed images.
strengths_and_weaknesses: The primary strength appears to be the separation of concerns with regards to embedding of the image priors and the encoding of retinal spiking responses. Because the denoising convolutional neural networks (dCNN) used for the natural image priors are trained independently of the retinal spike training, the model becomes modular, thereby allowing a finer granularity for tuning parameters of either end of the model pipeline, or for testing wholly different models entirely. The results demonstrate clear progress on SOTA, but the larger contribution may be in improving the pipeline itself.

One weakness may be the use of minimizing the negative log likelihood of the posterior, ie max estimation likelihood(MLE), in the MAP reconstruction of the RGC spikes. Due to the mean-seeking behavior of MLE, it may be possible the model is learning to output false positives, ie attributing image prior reconstruction to RGC spikes when the spikes did not in fact contribute to that particular prior.

Another weakness may be the assumption of the application of the inverse square law with regards to the spike response, ie assuming a Gaussian distribution for the image priors. This combined with the fact that the images are not only static, but as the paper states, they do not account for natural eye movement. Possibly, natural vision inducing a response to RGC spikes does not follow a traditional Gaussian distribution (despite the mathematical simplicity it provides) due to the need for the spikes to account for such high variance in an incoming “image”, where an object or image may even be still, but the photons are being received into a highly dynamic, organic mechanism, namely the eyeball. And so the distribution of the photons may in fact have a fat tail distribution, rather than a normal one.

questions: Why was the DRUNet denoiser network used specifically?
limitations: The biggest limitation that I see is, as the paper states, achieving these sorts of results in the face of the “jitter” that is present in natural eye movements, and how to handle not only the jitter but also these sorts of complex image reconstructions in real time as an enhancement to biological vision, but these are all concerns that the authors do address, and satisfactorily considering the limitations of really just present computational power, especially in terms of enhancing biological vision presumably with a device capable of being worn.
review_title: null

Meta-review: metareview: The paper proposes a MAP method for natural scene reconstruction from population recordings of RGCs which combines a deep image denoiser with a simple generalized linear neural encoding model. The results are shown to be state of the art, and the method is seen as novel and a useful contribution to the community.

The reviewers unanimously recommend acceptance.
confidence: Certain
Decision: Accept