Maximum a posteriori natural scene reconstruction from retinal ganglion cells with deep denoiser priors
Authors: Eric Gene Wu
Nora Brackbill
Alexander Sher
Alan Litke
Eero P Simoncelli
EJ Chichilnisky
Venue: NeurIPS 2022 Accept (2022)
Keywords: retina
ganglion cell
natural scenes
image reconstruction
image prior
Plug and Play
encoding model
neural coding
neuroscience
neural decoding

Abstract: Visual information arriving at the retina is transmitted to the brain by signals in the optic nerve, and the brain must rely solely on these signals to make inferences about the visual world. Previous work has probed the content of these signals by directly reconstructing images from retinal activity using linear regression or nonlinear regression with neural networks. Maximum a posteriori (MAP) reconstruction using retinal encoding models and separately-trained natural image priors offers a more general and principled approach. We develop a novel method for approximate MAP reconstruction that combines a generalized linear model for retinal responses to light, including their dependence on spike history and spikes of neighboring cells, with the image prior implicitly embedded in a deep convolutional neural network trained for image denoising. We use this method to reconstruct natural images from ex vivo simultaneously-recorded spikes of hundreds of retinal ganglion cells uniformly sampling a region of the retina. The method produces reconstructions that match or exceed the state-of-the-art in perceptual similarity and exhibit additional fine detail, while using substantially fewer model parameters than previous approaches. The use of more rudimentary encoding models (a linear-nonlinear-Poisson cascade) or image priors (a 1/f spectral model) significantly reduces reconstruction performance, indicating the essential role of both components in achieving high-quality reconstructed images from the retinal signal.

Maximum a posteriori natural scene reconstruction from retinal ganglion cells with deep denoiser priors
Eric G. Wu Stanford University vueric@stanf ord. edu
Nora Brackbill Stanford University
Alexander Sher & Alan M. Litke University of California, Santa Cruz
Eero P. Simoncelli
E.J. Chichilnisky Stanford University e @j@stanford. edu
Flatiron Institute, Simons Foundation, and New York University eero www.smmlmmmeennnnu.uuu edu
Abstract
Visual information arriving at the retina is transmitted to the brain by signals in the optic nerve, and the brain must rely solely on these signals to make inferences about the visual world. Previous work has probed the content of these signals by directly reconstructing images from retinal activity using linear regression or nonlinear regression with neural networks. Maximum posteriori (MAP) reconstruction using retinal encoding models and separately-trained natural image priors offers more general and principled approach. We develop novel method for approximate MAP reconstruction that combines a generalized linear model for retinal responses to light, including their dependence on spike history and spikes neighboring cells, with the image prior implicitly embedded in a deep convolutional neural network trained for image denoising. We use this method reconstruct natural images from ex vivo simuitaneously-recrre
spikes of hundreds of retinal ganglion cells uniformly sampling a region of the retina. The method produces reconstructions that match or exceed the state-of-the-an in perceptual similarity and exhibit additional fine detail, while using substantially fewer model parameters than previous approaches. The use of more rudimentary encoding models (a Seeaa--nonnine--ooiiiii Poisson cascade) image priors (a 1/ʃ spectral model) significantly reduces reconstruction performance, indicating the essential role both components in achieving high-quality reconstructed images from the retinal signal.
Introduction
torrent of visual information arrives at each of our eyes where it is transduced by large population of photoreceptors and transformed by retinal circuitry. The resulting signals are transmitted to the brain in the spikes of a relatively small number retinal ganglion cells (RGCs), suggesting that large fraction of this visual information is lost. Elucidating the nature of these encoded signals, and the inference processes the brain could use to interpret them, is fundamental to understanding biological vision. Image reconstruction provides method of visualizing the content encoded by RGC signals, evaluating it using standard image quality metrics, and reasoning about how the brain The fidelity and quality of reconstructed images also provides useful
might I objective function for optimizing the design of electrical stimulation patterns delivered by devices implanted restore vision to people blinded by retinal degeneration
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

The simplest and most well-studied image reconstruction method is linear regression [1 3 7 Optimal reconstruction kemelss are learned for each RGC using ceastesquares regression of recorded responses to many visual images, and the reconstruction of new incident image is computed with the sum of the filters weighted by the measured response of each cell. The quality of linearly reconstructed images can be enhanced by applying an autoencoder neural network to impose natural image priors [. or by using deep neural networks to non-linearly recover additional high spatial frequency image components E. Neural networks can also be directly trained (supervised) for reconstruction from raw data, but this is data-intensive, and has limited their use to simulated data, or low-dimensional stimuli and small numbers of cells [9] These direct approaches leave substantial room for improvement and interpretation. A Bayesian formulation, in which encoding model and image priors are made explicit and are separately designed and optimized, offers more flexible and interpretable solution.
Here, we present a method for approximate maximum a posteriori (MAP) image reconstruction from recorded RGC spikes, combining a retinal encoding model that accurately captures retinal responses [TC with state-of of-te-af- image priors that are implicitly embedded in deep denoising networks
15 16. Separating the contributions of image prior and RGC spiking response likelihood allows independent training of both elements, and offers flexible exploration and interpretation. Specifically, (1) any pre-trained or closed-form natural image prior can be used, and the effects of different priors can be compared; and (2) any model of RGC encoding that provides an explicit likelihood can be used, allowing examination of the relative importance of different model components in representing the visual signal, such as spike train history, cell-to-cell correlations and output nonlincarities. We apply our method to reconstruct flashed natural images from responses of several hundred macaque RGCs of identified types recorded with large-scale electrode arrays. We compare these results to published state-of-the-art linear and neural network regression methods. The new method matches or outperforms previous methods, producing sharper, more naturalistic reconstructions, and similar or greater perceptual similarity to ground truth. However, our method also produces some reconstructions with distinctive hallucinated image structure, as would be expected when RGC signals are noisy and image priors dominate the reconstruction process. Finally, comparisons to reconstructions using more conventional encoding models and image priors reveal that both aspects of the approach are important for the most accurate reconstructions. Source code, fitted models, and subset of the test dataset used to produce example reconstructions are available at https: gi thub. com http///eric/RRRCC _MPP__MPP____ httpa//CCCAAee.erutttrtttttoa
2 Retinal data and stimuli
Extracellular recordings from RGCs in the peripheral macaque retina were performed ex vivo using a 512-electrode system 11 (see G for details). Retinas were obtained from terminally anesthetized macaque monkeys used by other laboratories, in accordance with Institutional Animal Care and Use Committee requirements. Spikes from individual RGCs were identified with the YASS [[]] spike-sorter. A 30-minute spatio-temporal white noise stimulus 19 was used to compute spatio- temporal receptive fields, and to identify cells of distinct types. Analysis focused on the four major RGC types of the primate retina (ON parasol, OFF parasol, ON midget, and OFF midget) [20 21],
totaling roughly 700 cells per recording. The receptive fields (2] of all four cell types formed regular mosaics, with each providing nearly-complete coverage of a region of visual space (Figure 33).
Natural images were presented to the retina described previously B. Images from the ImageNet database (3] were converted to grayscale and cropped to 256x160, Each pixel measured approxi- mately 11 x 11 jum at the retina. Each stimulus image was displayed statically for 100 ms, followed by a 400 ms uniform gray display, allowing each image presentation to be treated as an independent trial (Figure A). Recordings were gathered from retinas of two animals, in response to 19,000 and 10,000 images, respectively. Details are summarized in Tables 4 and 5 in the Appendix.
3 MAP image reconstruction from RGC spikes
The MAP estimate of the stimulus image (a vector of pixel luminances) from observed RGC spike trains s corresponds to the minimum of the negative log of the posterior distribution, - p(g x s). which can be expressed using Bayes' Rule as:
2

---

A
B
ON parasol
parasol
ON midget
OFF midget
9 beR 4o0pee
50
100
150
200
250
300
350
400
450
Time [ms]
Figure (A) Trial structure for a single stimulus image presentation. Static stimulus images are displayed for 100 ms, separated by 400 ms uniform gray display. Spikes occurring within 150 ms of the onset of the stimulus image, corresponding to the shaded blue region, were used to compute encoding negative ligeliksisoods. (B) Receptive field mosaics from retina for the four major RGC types (ON parasol, OFF parasol, ON midget, OFF midget) used in the image reconstructions. Image quality metrics were computed over the shaded blue region, to exclude areas of the image that were insufficiently covered by receptive fields of recorded RGCs.
(ss) = arg min { - log x) - log p(x)).
(I)
The first term is the negative log likelihood of the stimulus, obtained from encoding model describing the probabilistic spiking of RGCs given a stimulus. In principle, any model can be used, and its parameters learned from experimentally measured RGC responses. The second term the negative log prior of the stimulus which can be learned from natural images, independent of retinal responses. Because encoding models with varying levels of fidelity and detail be mixed with priors of varying sophistication, the MAP approach allows us to probe the distinct roles of these two
components in image reconstruction 2 25
3.1 RGC encoding models
Encoding models for each RGC must be fitted to the experimental data before performing MAP reconstruction. Two types of encoding models were used: (1) linear-nonlincar- Poissso (LNP) cascade model with an exponential nonlinearity, the most commonly used model of RGC responses to visual stimuli 77; and (2) generalized linear model (GLM) that augments the LNP model with feedback loop and cross-connections between neighboring cells, and which can accurately capture fine spike timing structure and cell-to-cell correlations [IO]. Encoding models were fitted training partition of the data by minimizing the negative log-likelihood and regularization hyperparameters tuned by evaluating the negative likellhoooo test partition for small subset of RGCs of each type.
Aimear-nonlinearPPisse (LNP) encoding model The LNP model is the de facto standard model for describing the probabilistic spiking of RGCs in response to visual stimuli 77. The model parameters for each RGC consist of a linear spatial filter m, and a scalar bias b. In the model, scalar generator signal m + b passed through an exponential nonlincarity to yield a spike rate A(x). Responses, are defined as the RGC spike count in 150 ms interval, which are assumed to be boisson-distribute with rate A(x). The resulting negative log-likelihood for each RGC
log x) exp(m' x b} s(m" + b),
(2)
3

---

which is convex in m and b. In practice, to ensure that the spatial filters were spatially compact and corresponded approximately with receptive fields obtained using reverse correlation with white noise, the negative log-likelihood objective was augmented with two regularization terms: an 1.1 penalty to induce sparsity in the filter weights, and an 1.2 penalty to minimize the difference between the filter and the receptive field. The complete LNP-fitting objective function with both regularization terms is described in A.4 The parameters each cell were optimized separately using FISTA 26
MAP reconstruction requires the joint negative log-likelihodd of the image based on the observed spikes from every RGC. Since the LNP model assumes that the responses for each cell are statistically independent, this is simply the sum of the single-cell negative reg-likeiihoods which is convex in the image x:
- p(s xx)
E iECells
exp(m: + b;} (m; Tx b;).
(3)
Generalized linear encoding models The generalized linear model (GLM) is an augmentation of the LNP model that incorporates the effects of spiking history and cell-to-cell correlations on neural response [1O]. In the GLM, the ith RGC is parameterized by a spatio-temporal stimulus filter & [ar,,, y, f), feedback (spike history) filter fi(t). set of neighboring cell coupling filters (t) and a bias b;- The GLM was fitted to spike counts measured within ms time bins, a duration approximately matched the spike refractory period 27 28 yielding binary spike counts. To limit the number of parameters and improve computational efficiency, the stimulus filter was assumed to be space-time separable [a[ y. m. [z, y)4,[4]. Letting denote time-domain convolution, w[[] denote the time course of the stimulus (boxcar), and {i} denote the set of coupled neighboring cells, the generator signal g(t] is
g (m? (h, wist 1 + (8t fo)lt - 1]+
Σ(s (0) 22)((= 1] 1]+ b;- jE(iJC
(4)
Assuming sigmoidal nonlinearity oxx) (1 ez)
and a Bernoulli spiking distribution, the
negative log-likelihood used to fit single cell (see A.5.2 for complete derivation)
T
- Logg((s([N,........ 77 |1,xxx, s, N 11, s1pe (0, T)) =
Σ
log(1 + cxp(gx(tt)) 8; (t(t)g/tt)).
(5)
To simplify the GLM, the filters hus fo. and C were parameterized as weighted sums over a set of cosine-shaped "bump" functions (10) (discussed further in A.5.4.. As with the LNP model, L1 and I.2 regularization terms were added to constrain the spatial filters, and an additional I,,2 group sparsity penalty was imposed on the coupling filters to eliminate spurious cell-to-cell correlations. The complete objective function is described detail in A.5.3 Model parameters were found by alternating between convex minimization steps for the spatial and temporal filters of each RGC, using for each step.
FISTA
2
For MAP image reconstruction, we compute the joint negative log-likelihood of the image based on responses of all cells over time bins N, N,N N + 1..... T. This can be constructed from the single cell negative aog-likelihoode using the chain rule (see A.5.5 for derivation), and is again convex in the image
- Llog((s(,,..,.. x,s N-1))
Σ
Σ
log x, s; t- (0,,,
t= N FEcells
3.2 MAP with Gaussian 1/f priors
stationary Gaussian prior with 1/ʃ spectral energy the simplest and most commonly used image prior (30], and is the basis for most classical image processing algorithms. The 1/f prior

---

(1) Ground truth
(2)
(3) L-CAE.
(4) Kim et al.
(5)
(6)
(7)
Linear
MAP-GLM-dCMI MAP-LNP-dCNN
MAP-GLM/1F
A
B
F
K
M
N
Figure Selected stimulus images and reconstructions for the retina Figure Images are masked to only include regions covered by recorded RGCs. Colors the left correspond to the colored markers the seatter plots in Figure Column contents: (1) Ground truth stimulus image; (2) Linear reconstruction; (3) Linear reconstruction followed by CNN autoencoder (L-CAE, [8): (4) Linear reconstruction followed by neural network regression 44; (5) Our method, approximate MAP with GLM and denoiser prior MMA-GLMM-CCNN) (6) Approximate MAP with LNP encoder and denoiser prior (MAP-LNP-dCNN); (7) Exact MAP with GLM encoder and 1/ʃ prior (MAP-GLM-I1
5

---

assumes that pixels of the image are drawn from a stationary jointly Gaussian distribution (and thus that the spatial covariance matrix is diagonalized by the 2D Fourier basis) and that spectral power (variance of each spatial frequency component) falls off in inverse proportion to the square of the frequency f². Discarding terms that do not depend on the image x, the negative log prior can be written as - logop((( p(x) EE js |ay (zx))² where 0g (a:) is the amplitude of the hthh Fourier coefficient at frequency fi. MAP image reconstruction using this prior can be performed using standard unconstrained convex minimization methods because both the negative log prior and the RGC encoding negative ooglliielhooos described in and 6 are convex.
3.3 Approximate MAP with denoising convolutional neural network (dCNN) priors
Modern denoising convolutional neural networks can represent powerful image priors, but these priors are implicit [22 BT: : they are not expressed in closed form, and their values cannot be computed directly, making exact MAP inference difficult. The "plug-and-play" methodology provides an approximate iterative procedure for using such denoisers in MAP estimation problems (Л]. by incorporating them into variable-splittigg optimization methods such as half-quadratic splitting (HQS) [2 or alternating direction method of multipliers (ADMM) B33
34 351. Here, we use
method based on the HQS method presented in [15] to perform MAP reconstruction from RGC spikes. Following [.). we introduce an auxiliary variable z, split the original problem in equation ①1 into two complementary aub-probsems, incorporate a regularization parameter Xn to control the strength of the prior term, and iteratively alternate between solving the two sub-problems:
(Ak)
x ( = arg min x
{ logp(s x) + x 2())
2
((A)
(b-1) arg min
Xp log p(z)
lz (3)
(8)
Since the sub-problem in equation (8 has the same objective function as MAP Gaussian denoising
with known noise variance Xp/ p(4) we compute its solution using single forward pass of a pre- trained DRUNet denoiser network 15 This is an approximation, because the denoiser was trained with MSE objective and hence targets the mean of the posterior rather than its maximum. The approximation is correct in the limit of small noise, and thus can be used within an iterative procedure that gradually decreases the objective function [|] The overall iterative MAP reconstruction procedure is given in Algorithm
Algorithm 1 Approximate MAP reconstruction from RGC spikes (MAP-encoder-dCNN)
1: Hyperparameters: Ap. schedule f(2), p()) ... p(N)
2: Inputs: observed spike count vector E RC
3: Initialize (1)) to linear solution
4: for k 1, 2, do
5: ((++1) <- arg minx { log x) + 2 1--1-2%%0 (1)
6:
z (l+1) < SaablindDDUUNet-Denniiesse 12+ = prk)
7:
end for
Unlike most applications of HQS, the encoding term in equation is non-quadratis in x and hence was solved iteratively (gradient descent with momentum and backtracking line search) rather than in closed form. (1)) was initialized the linear solution; using random Gaussian initialization does not significantly affect the results (see Appendix A.6). Because convergence in the mathematical sense is not necessary for most imaging applications 36 K = 25 iterations were used in Algorithm
1 As in [155] (*)) was increased per-iteration on a log-spaced schedule. The hyperparameters Xp and [,("), ( were determined by performing a grid search and evaluating reconstruction quality on an 80-image subset of the test partition. Variations in hyperparameters over a reasonable range (p(1) E [10-2, 10,1001).. (225) E [30. 500), Ap 0.1) produced similar reconstruction quality, and optimal hyperparameters were similar across the two retinas and across LNP and GLM encoding models. Approximate MAP reconstructions using this algorithm are termed MAP-GLM-dCNN and MAP-LNP- for the GLM and LNP encoding models, respectively.

---

A
MS-SSIM
PSNR [dB]
B
MS-SSIM
PSNR [dB]
MAP- MAP-GLM-dCO (GLM-dCNN
0.8
MAP-GLM- M-
30-
WAP-GLM-dCNIP
0.8-
MAP-GLM- MAP-GLM-dCNN
COLM-dCNNN
20-
0.6
0.6
20-
20-
15-
0.2
0.4
0.6
0.8
1.0
10
15
20
25
20
35
0.2
0.4
0.6
0.8
1.0
10
15
20
25
30
35
Kim et al.
Kim al.
L-CAE
L-CAE
C
MS MIS-SSIM
PSNR [dB]
MS-SSIM
PSNR [dB]
MAP-GI MAP-GLM-dCO DI-GMM--ONN
0.8
TIP-GLM--CNNN
30-
MAP-GLM-DCNN
MAP-GLM-dCNN
20-
25-
25-
0.6
0.6
20
0.4
35
15-
0.2
0.4
0.6
3.8
1.0
10
15
20
25
30
35
0.2
0.4
0.5
0.8
1.0
10
15
20
25
30
35
MAP-GLM-1F
MAP-GLM-1F
NC
MAP-LNP-dCNN
Figure 3: Comparisons of reconstruction quality of test data from spikes the retina shown in Figure BB. Each panel shows values of two metrics multi-scale structural similarity (MS-SSIM) [37], and PSNR for all test images (individual points). Colored markers correspond to the example images in Figure Vertical axis of all graphs corresponds to our method (MAP-GLM-CG GL--GNNN and horizontal axis corresponds to other methods: (A) MAP-GLM-dCNN vs. Kim al. linear with neural network achieves comparable perceptual similarity but worse PSNR (B) MAP-GLM-dCNN vs. L-CAE B. MAP-GLM-dCNY achieves improved perceptual similarity with slightly worse PSNR. (C) MAP-GLM-JCNN vs. MAP-GLM-IF. Reconstructions using the weaker // prior have worse perceptual similarity and PSNR. (D) MAP-GLM-GCNN vs. MAP-LNP-dCNN. Reconstructions using the simpler LNP encoding model have worse perceptual similarity and PSNR.
regression A. MAP-GLM-dCCN
3.4 Benchmark: nonlinear regression with artificial neural networks
Current kate-of-the--a methods for reconstruction of natural images from RGC spikes rely initial linear reconstruction step [ followed by application of nonlinear neural networks to further improve the image. In approach, Parthasarathy et al. [ use deep convolutional autoencoder (L-CAE) trained with MSE loss to incorporate image priors in the reconstruction. The authors trained and tested the model on simulated RGC spikes. For comparisons, used the published architecture and hyperparameters. but trained the model using backpropagation on experimentally measured retinal spike counts. Further details and characterization of the L-CAE on experimental data are provided in Appendix A.7 Another method was developed by Kim el al. 4 who partition the target image into high and low spatial frequency components. Linear regression from RGC spike counts is used to recover the low-frequency component, and a fully-connected neural network is used non-linearly reconstruct the high-frequency component. The two components are summed and then passed through final deblurring CNN. Both the high-frequen are trained with backpropagation on RGC responses natural images. This method has achieved the most accurate reconstructions in the literature to date. Details of our implementation of the Kim er al. method are provided in Appendix A.8
network and the deblurring CNN
Results
4.1 Approximate MAP with GLM/dCNN matches or exceeds state-of-the-art results
To compare our MAP-GLM-dCNN method to current state-of-the-- art approaches, image reconstruc- tions were generated from the test partitions of the datasets. Example reconstructions are shown in columns 3-5 of Figure 2 for the L-CAE S for Kim et al. [4, and for our method, respectively. Qualitatively, the
reconstructions are seen to sharper than those of L--AE, and contain additional image details (especially extended contours, as in rows C, E, G. H. I, L). When
7

---

Table 1: Average test and heldout MS-SSIM for each reconstruction method and retina.
L-CAE
Kim et al.
MAP-GLM-dCNN
MAP-LNP-JCNN
MAP-GLM- IF
test
held
test
held
test
held
test
held
test
held
Retina
0.665
0.661
0.687
0.681
0.689
0.688
0.635
0.636
0.557
0.552
Retina 2
0.643
0.645
0.675
0.675
0.668
0.673
0.601
0.609
0.584
0.577
Table 2: Average test and heldout PSNR for each reconstruction method and retina.
L-CAE
Kim et al.
MAP-GLM-dCNN
MAP-LNP-ACNN
MAP-GLM-IF
test
held
test
held
test
held
test
held
test
held
Retina
19.9
20.1
20.2
20.5
19.5
19.5
18.3
18.4
16.8
16.8
Retina
19.3
19.6
19.8
20.1
18.5
18.5
17.0
17.1
17.3
17.2
compared to Kim et al., MAP-GLM-dCNI tended to recover more content, particularly straight edges (rows E, G, H, I, L), but sometimes exaggerated contrast (rows H. I, N). MAP-GIMM-CCNN produced qualitatively different artifacts than the other methods. In particular, it sometimes hallu- cinated naturalistic structure not present the stimulus images (rows J, K, N), including striking irregularities in contours (rows D. K, I,, M).
Reconstruction quality for each method was quantified over the test and heldout partitions using MS-SSIM and PSNR. Scatter plots comparing MS-SSIM and PSNR on the test partition of one retina are shown in Figure 3 and for Kim er al. and the L-CAE, respectively. and summary statistics over the test and held out partitions for both retinas are presented in Tables [ [and On a per-image basis, the perceptual similarity (measured with MS- MS-SSIM) of SAAP-GLM-CCN ground truth is higher than L-CAE B). The perceptual similarity of MAP-GLM-dCNT results is comparable to that of the much more complicated Kim al. method (A). But note that the PSNR of
reconstructions to
reconstructions was systematically lower than either benchmark. This is not surprising, as the MAP optimization procedure does not necessarily minimize MSE These results held for both retinas (Tables I and
4.2 Deep denoiser prior substantially improves image quality over
f/f prior
To test the importance of the image prior, MAP-GLM-dCNT results were compared against recon- struction using the GLM encoding model with the classical 1/f Gaussian prior (MAP-GLM-IF),
Example reconstructed images using the denoiser prior and 1/f prior are shown in columns 5 and 7, respectively, Figure P Images reconstructed with the denoiser prior are less "grainy", and tend to have both crisper edges and smoother surfaces. The artifacts seen in the 1/f examples are expected, since this simple prior does not constrain phase, whose alignment is essential for generating sharp spatially-localized features. Scatter plots of image quality on the test partition using MS-SSIM and PSNR are shown for one retina Figure BC, and mean values for both retinas are summarized in Tables
D and g Consistent with the visual appearance, PSNR and MS-SSIM were systematically higher when using the denoiser prior, both retinas. Thus, using the more sophisticated denoiser image prior substantially increased the perceptual similarity of the reconstructions to ground truth.
4.3
GLM encoding model recovers additional image structure over LNP encoding model
To test the importance of the encoding model, we compared images reconstructed using the GL.M and LNP encoding models, both in combination with the the denoiser prior. Example images are shown in columns 5 and of Figure 2 Images reconstructed using both models exhibit natural image structure, including smooth surfaces and well-defined edges, but the GLM-reconstructed images tended have more cealistic-looking textures, whereas the LNP-reconstructed images tended to be overly simplified. Moreover, the GLM method recovered more fine details (e.g., the legs of the insect in row C, the horizontal stripes on the tape cassette in row D, and the details on the hammock in row E, and the structure on the file cabinets in row G). The quality of image reconstructions for each image-response pair in the test partition for one retina were compared using MS-SSIM and PSNR in Figure 3D, and their mean values over the test and held out partitions for both retinas are summarized Tables T and 2 In both retinas, images reconstructed using the GLM encoding model had systematically higher
8

---

MS-SSIM scores, indicating greater perceptual similarity to ground truth, than those reconstructed using the LNP encoding model. This demonstrates that the choice of encoding model significantly affects reconstruction quality, and that the inclusion of temporal spike dependencies and cell-to-cell correlations in the more sophisticated GL.M encoding model provides important constraints on the visual signal encoded by the RGC spikes. This finding is consistent with previous work showing that decoding using the GLM (without priors) can access more information than simplified models lacking the cell-to-cell correlations or spiking history [TO].
Table 3: Number of parameters trained on retinal data for each method (for retina 1). MAP-GLM- dCNN and L-CAE have comparable numbers of parameters. Because of sparsity regularization in the GLM spatial encoding filters, more than half (55%) of the parameters in the GLM model are zero. The Kim et al. model contains nearly an order-of-magnitude more parameters than either MAP-GLM-dCNN or L-CAE.
L-CAE
Kim et al.
MAP-GLM-ICNN
MAP-LNP-dCNN
MAP-GLM-IF
Trained params.
3.07 10
2.44 10
2.88 10
2.85 10
2.88 188
5
Discussion
We have presented a novel upproximate MAP method for reconstructing natural images from the si- multaneously recorded spikes of several hundred retinal ganglion cells, using an accurate probabilistic model of retinal encoding and natural image prior implicit in pre-trained denoising neural network. The method matches or outperforms the current tatte-of-the-ane art in terms of recovering naturalistic image structure and/or the perceptual similarity of reconstructions ground truth, while also being more principled and interpretable due to the explicit Bayesian separation of the encoding model and prior. The new approach uses substantially fewer parameters than previous state-of-the-ar methods based on neural networks, and does not require training neural networks on retinal data (the prior is obtained from network trained exclusively on image denoising). We showed that both encoding model and image prior contributed to the high-quality image reconstructions: simplification of either substantially degraded performance. Thus, we expect that cell-to-cell correlations and temporal structure of spike trains, as well as sophisticated image priors, will prove important in understanding how the retinal signal is used by the brain.
Several previous studies have used GLM encoding models for stimulus reconstruction from apperimentaall-recordde retinal signals, revealing the significance of cell-to-cell correlations for decoding temporal structure in white noise stimuli [10] [3]. and the significance of the temporal structure of spike trains in tracking moving features (2. By including a complex natural image prior into a Bayesian reconstruction method, the present work more efficiently leverages both the GLM model and the experimental data produce state-of-the-art natural image reconstructions.
The enhanced reconstructions and interpretability obtained with our method could lead to improved function of retinal implants for restoring vision. Previous work [5] has suggested that electrical stimulation with a retinal implant can be guided by minimizing the expected MSE of linearly reconstructed images. This method ignored fine temporal structure in RGC spike trains, and assumed that image priors captured by linear regression are sufficient for high performance. The method presented here offers an alternative approach to choosing simulation patterns produce higher- fidelity artificial vision, while potentially being more robust than neural network methods. However, achieving this in real time with minimal latency presents a substantial technical challenge.
Our Baycsian reconstruction methodology relied on the convexity of the GLM encoding model, which allows for robust convergence. is worthwhile to consider generalizations that might be used to evaluate more complex encoding models that are non-convex in the stimulus, such as subunit We are currently working on to answer this question.
models [39 40 All or artificial neural network retinal models 42 1
methods based on score-based sampling methods
Though the present work is limited to reconstruction of flashed static natural images from RGC spikes, extensions of the method could used to probe how neurons encode visual information under more natural conditions. For example, a central problem is understanding how the visual system achieves high-acuity perception in the presence of "jitter" eye position, even during fixation 45.. Previous computational efforts have probed this question, but have been largely limited to simulated data with

---

simple encoding models and stimuli НБ 47 48 algorithms for image deblurring and pottin--correction 449 15 could yield more powerful methods to decode images from jittered retinal inputs. A related problem is understanding how the retina encodes the information contained in complex naturalistic movies 2 including movement of objects within scene and other non-rigid transformations over time. The dimensionality of such stimuli and the consequent data requirements are high, so the use of machine learning methods to capture spatio-temporal priors [50 [5]] separately from the retinal data will be important for understanding reconstruction in these contexts.
Combining the methods put forth here with modern
Funding
National Defense Science and Engineering Graduate (NDSEG) Fellowship (EGW), NSF IGERT 0801700 (NB), Wu Tsai Neurosciences Institute Big Ideas award, NSF CRCNS grant 1430348 (EJC/EPS), and NIH grants EY017992 and EY029247 (EJC).
10

---

References
[1] D. K. Warland, P. Reinagel, and M. Meister, "Decoding Visual Information From a Population of Retinal Ganglion Cells," Journal of Neurophysiology, vol. 78, pp.
Nov. 1997.
[2] V. Botella-Soter,
S. Deny, G. Martius, O. Marre, and G. Tkačik, "Nonlinear decoding of a com- plex movie from the mammalian retina," PLOS Computational Biology, vol. 14, p. e1006057, May 2018.
[3] N. Brackbill, C. Rhoades, A. Kling, N. P. Shah, A. Sher, A. M. Litke, and E. Chichilnisky, "Reconstruction of natural images from responses of primate retinal ganglion cells," elife, vol. e58516, Nov. 2020.
[4] Y.J. Kim, N. Brackbill, F. Batty, J. Lee, C. Mitelut, W. Tong, E. J. Chichilnisky, and L. Paninski, "Nonlinear Decoding of Natural Images From Large-Scale Primate Retinal Ganglion Recordings." Neural Computation, vol. 33, pp. 1719-1750, June 2021.
[5] N. P. Shah, S. Madugula, L. Grosberg, G. Mena, P. Tandon, P. Hottowy, A. Sher, A. Litke, S. Mitra, and E. Chichilnisky. "Optimization of Electrical Stimulation for High-Fidelity Artificial Retina," in 20/9 9th International IEEE/EMBS Conference on Neural Engineering (NER), (San Francisco, CA, USA), pp. 714-718, IEEE, Mar. 2019.
[6] N. P. Shah and E. J. Chichilnisky, "Computational challenges and opportunities for bi- directional artificial retina," Journal of Neural Engineering, vol. 17, 055002, Oct. 2020.
[7] Ricke, D. Warland, R. de Ruyter van Steveninck, and W. Bialek, Spikes: Exploring the Neural Code. Cambridge, MA, USA: MIT Press, 1999.
[8] N. Parthasarathy, E. Batty, W. Falcon, T. Rutten, M. Rajpal, E. Chichilnisky. and L. Paninski, "Neural Networks for Efficient Bayesian Decoding of Natural Images from Retinal Neurons," conference, Neuroscience, June 2017.
[9] Y. Zhang, S. Jia, Y. Zheng, Z. Yu, Y. Tian, S. Ma, T. Huang, and J. K. Liu, "Reconstruction of natural visual scenes from neural spikes with deep neural networks," Neural Networks, vol. 125, pp. 19-30, May 2020.
[10] J. W. Pillow, J. Shlens, L. Paninski, A. Sher, A. M. Litke, E. J. Chichilnisky, and E. P. Simoncelli, "Spaaio-temporal correlations and visual signalling in a complete neuronal population." Nature, vol. 454, pp. 995-999, Aug. 2008.
[11] S. V. Venkatakrishnan, C. A. Bouman, and B. Wohlberg, "Plug-and and--laddyyyyyy priors for model based reconstruction," 2013 IEEE Global Conference on Signal and Information Processing. (Austin, TX, USA). pp. 945-948, IEEE, Dec. 2013.
[12] Y. Romano, M. Elad, and P. Milanfar, "The Little Engine That Could: Regularization by Denoising (RED)." SIAM Journal on Imaging Sciences, vol. 10, pp. 1804-1844, Jan. 2017.
[13] Y. Song and S. Ermon, "Generative modeling by estimating gradients of the data distribution," in Advances in Neural Information Processing Systems 32, pp. 11918-1193, Curran Associates, Inc., 2019.
[14] Z. Kadkhodaie and E. P. Simoncelli, "Stochastic Solutions for Linear Inverse Problems using the Prior Implicit in a Denoiser," in Advances in Neural Information Processing Systems 34 (NeurIPS 202/), p. 13, 2021.
[15] K. Zhang, Y. Li, W. Zuo, L. Zhang. L. Van Gool, and R. Timofte, "Plug-and-Pio Image Restoration with Deep Denoiser Prior," " IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-1, 2021.
[16] B. Kawar, G. Vaksman, and M. Elad, "Stochastic image denoising by sampling from the posterior distribution," tech. rep., aug 2021.
[17] A. Litke, N. Bezayiff, E. Chichilnisky, W. Cunningham, W. Dabrowski, A. Grillo, M. Grivich, P. Grybos, P. Hottowy, S. Kachiguine, R. Kalmar, K. Mathieson, D. Petrusca, M. Rahman, and A. Sher, "What does the eye tell the brain?: Development of system for the large-scale recording of retinal output activity," IEEE Transactions Nuclear Science, vol. 51, pp. 1434- 1440, Aug. 2004.
[18] Lee, C. Mitelut, H. Shokri, Kinsella, Dethe, S. Wu, K. Li, E. B. Reyes, D. Turcu, E. Batty, J. Kim, N. Brackbill, A. Kling, G. Goetz, E, Chichilnisky, D. Carlson, and L. Paminski, "YASS:

---

Yet Another Spike Sorter applied to large-scale multi-electrode array recordings in primate retina," " preprint, Neuroscience, Mar. 2020.
[19] E. Chichilnisky, "A simple white noise analysis of neuronal light responses," Network: Computation in Neural Systems, vol. 12, pp. 199-213, 2001.
[20] E. J. Chichilnisky and R. S. Kalmar, "Functional Asymmetries in ON and OFF Ganglion Cells of Primate Retina," The Journal of Neuroscience, vol. 22, pp. 2737-2747, Apr. 2002.
[21] D. Dacey, "The mosaic of midget ganglion cells in the human retina," " The Journal of Neuro- science, vol. 13, pp. 3334-5355, Dec. 1993.
[22] J. L.. Gauthier, G. D. Field, A. Sher, M. Greschner, J. Shlens, A. M. Litke, and E. J. Chichilnisky, "Receptive Fields in Primate Retina Are Coordinated Sample Visual Space More Uniformly." PLoS Biology, vol. 7. e c1000063, Apr. 2009.
[23] L. Fei-Fei, J. Deng, and K. Li, "ImageNet: Constructing large-scale image database," " Journal of Vision, vol. 9, pp. 1037-1037, Aug. 2009,
[24] T. Naselaris, R. J. Prenger, K. N. Kay, M. Oliver, and J. L. Gallant, "Bayesian Reconstruction of Natural Images from Human Brain Activity," Neuron, vol. 63, pp. 902-915, Sept. 2009,
[25] L.-Q. Zhang, N. P. Cottaris, and D. H. Brainard, "An image reconstruction framework for characterizing initial visual encoding," eLife, vol. 11, p. e711 Jan. 2022.
[26] A. Beck and M. Teboulle, "A Fast Iterative Shrinkage- Thrnge----reehhiingg Algorithm for Linear Inverse Problems," SIAM Journal on Imaging Sciences, vol. 2. pp. 183-202, Jan. 2009.
[27] M. Berry and M. Meister, "Refractoriness and Neural Precision," The Journal of Neuroscience, vol. 18, pp. 2200- 2211, Mar. 1998.
[28] J. Uzzell and E. J. Chichilnisky, "Precision of Spike Trains in Primate Retinal Ganglion Cells," Journal of Neurophystology, vol. 92, pp. 780-789, Aug. 2004.
[29] J. Liu, S. Ji, and Ye, "Multi-TTar Task Feature Learning Via Efficient 2.1-Norm Minimization," " in Proceedings of the Twenty-Fitte Conference on Uncertainty in Artificial Intelligence, p. 10, 2009.
[30] E. P. Simoncelli, "Statistical Modeling of Photographic Images," in Handbook of Image and Video Processing, pp. 431-441, Elsevier, 2005.
[31] Z. Kadkhodaic and E. P. Simoncelli, "Solving linear inverse problems using the prior implicit in a denoiser," arXiv, July 2020.
[32] D. Geman and Chengda Yang. "Nonlinear image recovery with half-quadratic regularization," IEEE Transactions on Image Processing, vol. 4, pp. 932-946, July 1995.
[33] S. H. Chan, X. Wang, and O. A. Elgendy. "Plug-and-Play ADMM for Image Restoration: Fixed-Point Convergence and Applications, " IEEE Transactions on Computational Imaging, vol. 3. pp. 84-98, Mar. 2017.
[34] A. M. Teodoro, J. M. Bioucas- DDas, and M. A. T. Figueiredo, "Image Restoration and Recon- struction using Targeted Plug-and-Play Priors," " IEEE Transactions on Computational Imaging, vol. 5, pp. 675-686, Dec. 2019.
[35] Y. Sun, Z. Wu, X. Xu, B. Wohlberg, and U. Kamilov, "Scalable Plug-and-Play ADMM With Convergence Guarantees," IEEE Transactions on Computational Imaging, vol. 7, pp. 849-863, 2021.
[36] S. Diamond, V. Sitzmann, F. Heide, and G. Wetzstein, "Unrolled Optimization with Deep Priors," arXiv: hDN::/705.0044 [cs], Dec. 2018 arXiv: 1705.08041.
[37] Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli, "Image Quality Assessment: From Error Visibility to Structural Similarity," IEEE Transactions on Image Processing, vol. 13, pp. 600- 612, Apr. 2004.
[38] K. Ruda, Zylberberg, and G. D. Field, "Ignoring correlated activity causes a failure of retinal population codes," Nature Communications, vol. 11, p. 4605, Dec. 2020.
[39] J. Freeman, G. D. Field, P. H. Li, M. Greschner, D. H. Gunning, K. Mathieson, A. Sher, A. M. Litke, I.. Paninski, E. Simoncelli, and F. J. Chichilnisky, "Mapping nonlinear receptive field subunits in primate retina at single cone resolution," eLife, vol. 4:c05241, Oct 2015.
12

---

[40] J. K. Liu, H. M. Schreyer, A. Onken, F. Rozenblit, M. H. Khani, V. Krishnamoorthy, S. Panzeri, and T. Gollisch, "Inference of neuronal functional circuitry with matrix factorization," Nature Communicatiions vol. 8, p. 149, Dec. 2017.
non-negative
[41] N. P. Shah, N. Brackbill, C. Rhoades, A. Kling, G. Goetz, A. M. Litke, A. Sher, E. P. Simoncelli, and E. Chichilnisky. "Inference of nonlinear receptive field subunits with spike-triggered clustering," eLife, vol. 9, p. e45743, Mar. 2020.
[42] E. Batty, J. Merel, N. Brackbill, A. Heitman, A. Sher, A. Litke, E. J. Chichilnisky, and L. Panin- ski, "Multilayer recurrent network models of primate retinal ganglion cell responses," in International Conference on Learning Representations 20/7, p. 12, 2017.
[43] I.. McIntosh, N. Maheswaranathan, A. Nayebi, S. Ganguli, and Baccus, "Deep Learning Mod- els of the Retinal Response to Natural Scenes," in Advances in Neural Information Processing Systems 29 (NeurIPS 2016), P. 9, 2016.
[44] Y. Song, Shen, L. Xing, and S. Ermon, "Solving inverse problems in medical imaging with cosre-based generative models," in International Conference on Learning Representations, 2022.
[45] M. Rucei and M. Poletti, "Control and Functions of Fixational Eye Movements," Annual Review of Vision Science, vol. 1, pp. 499-518, Nov. 2015.
[46] X. Pitkow, H. Sompolinsky, and M. Meister, "A Neural Computation for Visual Acuity in the Presence of Eye Movements," PLoS Biology, vol. 5, p. e331, Dec. 2007.
[47] Y. Burak, U. Rokni, M. Meister, and H. Sompolinsky. "Bayesian model of dynamic image stabilization in the visual system," Proceedings of the National Academy of Sciences, vol. 107, pp. 19525-19530, Nov. 2010.
[48] A. G. Anderson, K. Ratnam, A. Roorda, and B. A. Olshausen, "High-acuity vision from retinal image motion," Journal of Vision, vol. 20, p. 34, July 2020.
[49] Y. Li, M. Tofighi, J. Geng, V. Monga, and Y. C. Eldar, "Efficient and Interpretable Deep Blind Image Deblurring Via Algorithm Unrolling," " IEEE Transactions on Computational Imaging, vol. 6. pp. 666 0066681,22 2020.
[50] A. Brifman, Y. Romano, and M. Elad, "Unifid Single-Image and Video Super-Resolution via Denoising Algorithms," IEEE Transactions on Image Processing, vol. 28, pp. 6063-6076 Dec. 2019.
[51] D. Y. Sheth*, S. Mohan*, J. L. Vincent, R. Manzorro, P. A. Crozier, M. M. Khapra, E. P. Simoncelli, and C. Fernandez- Computer Vision (ICCV), Oct 2021.
"Unsupervised deep video denoising," in Int'l Conf.
[52] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. Vetio,, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang. J. Bai, and S. Chintala, "Pytorch: An imperative style, high-performance deep learning library," in Advances in Neural Information Processing Systems 32 (H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, eds.),
pp. 8024-8035, Curran Associates, Inc., 2019.
[53] D. P. Kingma and J. Ba, "Adam: A Method for Stochastic Optimization," in 3rd International Conference on Learning Representations, 2015. arXiv: 1412.6980.
[54] O. Kupyn, T. Martyniuk, J. Wu, and Z.. Wang, "DeblurGAN-vv: Deblurring (Orders-of- Magnitude) Faster and Better," in 20/9 IEEF/CVF International Conference on Computer Vision (ICCV), (Seoul, Korea (South)), pp. 8877-8886, IEEE, Oct. 2019.
[55] K. Simonyan and A. Zisserman, "Very Deep Convolutional Networks for Large-Scale Image Recognition," " in International Conference on Learning Representations 2015, Apr. 2015. arXiv: 1409.1556.
13


Reviews:
[{"review_id": "maMUaWbXJXE", "reviewer": "Reviewer_Etny", "summary": "The authors propose an inference method for reconstructing images presented to retinas from the spiking activity of Retinal Ganglion Cells (RGCs). This is achieved using a Maximum A Posteriori (MAP) estimation where the posterior distribution is obtained from the combination of a likelihood modeling key aspects of RGCs (individually and as a network) and a prior for natural images.\n\nThe authors detail their models and optimization methods. Then they compare, both qualitatively and quantitatively, variation of their algorithms and also to previous state-of-the-art (SOTA) models. Their best model reaches SOTA performances on some scores. \n", "questions": "I am a bit surprised by the first sentence of the introduction. How do the authors know that only a small portion of visual information arriving at each eyes is transmitted to the brain ? I would have appreciated a reference here as \"information\" is a clearly defined concept in signal processing. Maybe the authors wanted to strike the reader by the existing contrast between the torrent of light or photons arriving at the retina and the somehow discrete nature of spiking activity ? Please add a ref or rephrase appropriately. The same remark hold for the first sentence of the abstract.\n\nApart from this minor remark, I think this is a well-written paper and a nice piece of work despite the results not being surprising.", "limitations": "The authors mention the negative impact of animal experimentation. \nIt is not clear if the data are openly accessible. If not, the authors should make them openly accessible to maximize the value of the data in many aspects (science, animals involved, ...)\nThe proposed work involve neural network training. The authors have used a single GPU but how long the training lasted for each model ? Does the performance gain justify the probable increase in energy consumption compared to other models ?", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths : \n- reconstructing images from neural activity is key to understand both natural images and neural activity, it is also a good task to evaluate models of biological neural networks,\n- the methods used are recent or fairly recent and originally combined (FISTA, HQS, dCNN priors, ...),\n- the results are good (reaching SOTA),\n- the model could be used in combination with retinal implants to restore vision.\n\nWeaknesses :\n- low risk (dCNN priors have shown to be powerful in a variety of tasks)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "coqzsf5Ib8G", "reviewer": "Reviewer_wKPR", "summary": "The paper proposes a novel method for reconstructing natural images from recorded spikes of a several hundred retinal ganglion cells (RGC) by combining a generalized linear model (an augmentation of a linear-nonlinear-Poisson model)  encoded with the spike responses of the RGCs, along with image priors embedded in a deep, denoising CNN. Their combined model outperforms current state of the art that applies a deep convolutional autoencoder to linearly-reconstructed images.", "questions": "Why was the DRUNet denoiser network used specifically?", "limitations": "The biggest limitation that I see is, as the paper states, achieving these sorts of results in the face of the “jitter” that is present in natural eye movements, and how to handle not only the jitter but also these sorts of complex image reconstructions in real time as an enhancement to biological vision, but these are all concerns that the authors do address, and satisfactorily considering the limitations of really just present computational power, especially in terms of enhancing biological vision presumably with a device capable of being worn.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The primary strength appears to be the separation of concerns with regards to embedding of the image priors and the encoding of retinal spiking responses. Because the denoising convolutional neural networks (dCNN) used for the natural image priors are trained independently of the retinal spike training, the model becomes modular, thereby allowing a finer granularity for tuning parameters of either end of the model pipeline, or for testing wholly different models entirely. The results demonstrate clear progress on SOTA, but the larger contribution may be in improving the pipeline itself.\n\nOne weakness may be the use of minimizing the negative log likelihood of the posterior, ie max estimation likelihood(MLE), in the MAP reconstruction of the RGC spikes. Due to the mean-seeking behavior of MLE, it may be possible the model is learning to output false positives, ie attributing image prior reconstruction to RGC spikes when the spikes did not in fact contribute to that particular prior.\n\nAnother weakness may be the assumption of the application of the inverse square law with regards to the spike response, ie assuming a Gaussian distribution for the image priors. This combined with the fact that the images are not only static, but as the paper states, they do not account for natural eye movement. Possibly, natural vision inducing a response to RGC spikes does not follow a traditional Gaussian distribution (despite the mathematical simplicity it provides) due to the need for the spikes to account for such high variance in an incoming “image”, where an object or image may even be still, but the photons are being received into a highly dynamic, organic mechanism, namely the eyeball. And so the distribution of the photons may in fact have a fat tail distribution, rather than a normal one.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "Fge1FOS1HwP", "reviewer": "Reviewer_mbYL", "summary": "Similarly to Parthasarathy/Batty 2017 and Warland 1997, this paper proposes a method to reconstruct visual stimuli from retinal ganglion cell electrophysiological measurements.\n\nIt proposes to combine a forward model and a prior in a MAP framework to obtain the stimulus image with maximum a posteriori probability.\n\nFor the forward model it proposes a baseline linear-nonlinear-Poisson model and the recurrent model from Pillow 2008. For the prior a frequency power law Gaussian prior and a denoising neural network prior (implicit through plug and play method) are used.\nCombinations of forward model and and prior are evaluated.\n\nIt is shown that reconstructions using the recurrent model and the denoising neural network prior exceed other works in terms of SSIM similarity with the stimulus image.\n\n", "questions": "What about using the model in Batty, ICLR2017 \"Multilayer recurrent network models of primate retinal ganglion cell responses\" as a forward model?\n\nIn Parthasarathy/Batty 2017 it is shown for the convolutional autoencoder that its performance is strongly dependent on the data set used for training. My guess would be that this could be less the case for the denoising network used here. Could this be evaluated?\n\nThe abstract juxtaposes (non-)linear regression and MAP estimation, which are largely orthogonal concepts. Since even in this contribution a regression model is used for prediction, this should probably be rephrased.", "limitations": "Some limitations are discussed - I'd say this discussion is adequate. The scope of this work has no immediate negative societal impact, but may lead to some positive medical research outcomes down the line.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper does solid work in combining existing methods into a new method that improves performance in the particular application of reconstructing stimuli from retinal ganglion cell activity.\nIn terms of numbers, the demonstrated improvement in terms of SSIM is marginal, but perceptually the reconstructions are more compelling than the alternative methods (my general experience with SSIM has been that there is no chance of it capturing such subtleties).\n\nMy main concern is about the interaction of injecting strong priors and the declared purpose of this method. If you are concerned about \"how the brain might interpret the signal\", is it a good idea to inject ever stronger naturalistic priors to fill in the missing information? Granted, the brain is good at such completion inference, too. But there is no reason to believe that the priors are in any way the same. Relatedly, I would be worried that the prior adds nonexistent details as briefly mentioned in the paper. While this does become visible in some of the reconstructions, it should also be noted that, reassuringly (or not?), it is not to an extent that the reconstruction metric against the ground truth is affected by it.\n\nWhile the MAP framework is appealing because it allows designating a prior and a forward model, it also leaves an aftertaste of serving only the narration. E.g. it is a known pitfall to name L1-regularization a \"sparsity prior\", since only the MAP estimate will be sparse but no typical sample from the posterior distribution. All ideas presented here would also easily fall under the less semantically laden framework of regularized risk estimation, in which case the denoising step would replace the proximal operator of the regularizer.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept