A computational framework to unify representation similarity and function in biological and artificial neural networks
Authors: Xuming Ran
Jie Zhang
Ziyuan Ye
Haiyan Wu
Qi Xu
Huihui Zhou
Quanying Liu
Venue: Submitted to ICLR 2023 (2023)

Abstract: Artificial neural network (ANN) is a versatile tool to study the neural representation in the ventral visual stream, and the knowledge in neuroscience in return inspires ANN models to improve performance in the task. However, it is still unclear how to merge these two directions into a unified framework. In this study, we propose an integrated framework called Deep Autoencoder with Neural Response (DAE-NR), which incorporates information from ANN and the visual cortex to achieve better image reconstruction performance and higher neural representation similarity between biological and artificial neurons. The same visual stimuli (i.e., natural images) are input to both the mice brain and DAE-NR. The encoder of DAE-NR jointly learns the dependencies from neural spike encoding and image reconstruction. For the neural spike encoding task, the features derived from a specific hidden layer of the encoder are transformed by a mapping function to predict the ground-truth neural response under the constraint of image reconstruction. Simultaneously, for the image reconstruction task, the latent representation obtained by the encoder is assigned to a decoder to restore the original image under the guidance of neural information. In DAE-NR, the learning process of encoder, mapping function and decoder are all implicitly constrained by these two tasks. Our experiments demonstrate that if and only if with the joint learning, DAE-NRs can improve the performance of visual image reconstruction and increase the representation similarity between biological neurons and artificial neurons. The DAE-NR offers a new perspective on the integration of computer vision and neuroscience.

Under review conference paper at ICLR 2023
A COMPUTATIONAL FRAMEWORK TO UNIFY REPRESEN- TATION SIMILARITY AND FUNCTION IN BIOLOGICAL AND ARTIFICIAL NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Artificial neural networks (ANNs) are powerful tools for studying neural repre- sentations in ventral visual stream the brain, which, in turn, have inspired new designs of ANN models to improve task performance. However, a unified framework for merging these two directions has heen lacking so far. this study, we propose an integrated framework called Deep Autocncoder with Neural Re- sponse (DAE-NR), which incorporates information from the visual cortex into ANN models to achieve better image reconstruction performance and higher neural representation similarity between hiological and artificial neurons. The same visual stimuli (i.e,, natural images) are input to both the mice brain and DAE-NR. The encoder of DAE-NR jointly learns the dependencies from neural spike encoding and image reconstruction. For the neural spike encoding task, the features derived from specific hidden layer of the encoder are transformed by mapping function to predict the ground-truth neural response under the constraint of image reconstruc- tion. Simultancously, for the image reconstruction task, the latent representation obtained by the encoder assigned to decoder to restore the original image under the guidance of neural information. In DAE-NR, the learning process of encoder, mapping function and decoder are all implicitly constrained by these tasks. Our experiments demonstrate that if and only with the joint learning, DAE-NRs improve the performance visual image reconstruction and increase the represen- tation similarity between biological neurons artificial neurons. The DAE-NR offers new perspective on integration of computer vision and neuroscience.
INTRODUCTION
Computer vision has achieved almost comparable performance the human visual system some tasks, mainly thanks recent advances in deep learning. Image reconstruction
of the essential tasks in computer vision Hinton Salakhutdinov 2006))[[Kngmm & Welling (2014); Raaiihankur al (2020)). As solution, the auto-encoder (AH) framework embeds the high-dimensional input to low-dimnsioonal latent space by the encoder and then reconstructs the image by the decoder (Hinoon Salakhutdinov (2006); (2006), GGodfeelow et al.] (216)). Despite the popularity and the practical successes of Al models, the setting of prior would largely influence the image reconstruction performance on DAHs Tomezak & Welling (2018). Morcover, there usually needs to be more biological interpretability in the model architecture.
Inspired by neuroseience, computer vision rescarchers have been interested how use information from biological neurons to achieve brain-like performance (such robustness and ability to learn from small samples). The biology-inspired AE models may help improve performance in image reconstruction tasks and bring biological interpretahility SraprrtthiiiyFFedeeee Federer et al] (2020) /Schrimpf et al. (2018); Safarani et al. 202T), To this end, the key question is how to integrate biological information into AEs.
the other hand, computational neuroscience interested building models that map stimuli to neural responses. Traditional models have difficulty expressing nonlinear characteristics between stimulus and neural response. Deep learning empowers computational neuroscience models and reveals the relationship between stimuli and neural spikes Klindi et al. (2017) Although biological and

---

to Under review conference paper at ICLR 2023
1100
Veriables
X stmmul
(b)
R: roco rstructd stimuli
artificaal neurt response
h, reprosertiaaon of artiicaal
neurant in the tth comvoisionall layer
(c)
0i00
Iatent
Figure 1: Scheme of the models: (a) the standard deep autoencoder (DAE) for image reconstruc- tion, (h) the convolutional neural network with factorized readout (CNN-FR) for predicting neural responses, (c) the DAE with neuron response (DAE-NR) for both image reconstruction and neural response prediction. s denotes the biological neural response; denotes the prediction of biological neural response; h, (i E (1, 2,
3, 4)) is the feature map of the ith convolutional layer.
artificial neural networks may have fundamental differences in computation and learning Macpherson al. (202T) both realized by interconnected neurons: the former biological neurons: the latter by artificial neurons. Some previous work focused investigating the similarities between information representation biological neurons and artificial neurons using end-to-end ANNs, suggesting that artificial neurons different layers of ANNs share similar representations biological neurons brain regions along the ventral visual pathway DiCarlo (2012); Yamins (2019) How build artificial neural networks most simllar biological neural representations remain an open question.
DiCarlo (2016); Walker et al. [2019): Bashivan
Some studies have leveraged real neural responses speed up the training process and improve (2020); [Schrimpl al. (218); [Safarani al. (2021). However, model yet utilized neural responses constraints improve image reconstruction performance. More importantly, our knowledge, there is no unified framework that can leverage both neural responses to improve model performance and image reconstruction improve representational similarity between biological and artificial neurons.
network performance in object detection tasks Federer
In this paper, aim tackle these questions biologically inspired framework, which jointly learns i) project features of visual input specific layer of the encoder biological neural responses hy mapping functton and ii) reconstruct result, encoder has higher representational similarity to
piece. Specifically, propose united
visual input the decoder. real neural responses, and decoder achieves better image reconstruction performance. Our contributions can be summarized as follows.
present biologically inspired framework called Deep Autoencoder with Neural sponse (DAE-NR). The framework can simultaneously learn predict neural responses and to reconstruct visual stimuli (See
The deep autoencoders embedded in DAE-NR can improve the image reconstruction quality with the help of a Poisson loss the predicted neural activity, compared to the baseline auto-enender models 4.e (.4).
The computational neuroscience model (CNM) DAE-NR offers higher resemblance between artificial neurons and biological neurons compared to the competing end-to-end CNM models (Sec 4.4).
2

---

Under review conference paper at ICLR 2023
Figure Reconstructed images of neural activity in mouse Region From top to bottom, each row displays 10 examples of the original stimuli images (a), the reconstructed images by CAE (b), and by CAE-FR, (o-f) for i {1, 2, 3, 4), respectively.
RELATED WORK
Image reconstruction via auto-emcoders: A big breakthrough in image reconstruction is in Hinton & Salakhutdinov (2006), which equips autoencoder with a stack of restricted Boltzmann ma- chines, The denoising autoencoder Viineet et al. (2008) and convolutional autoencoder (CAE) Masci et al. (2011) further improve the stability of autoencoders by adding noise and taking advantage of convolutional layers in feature extraction, respectively. Variational Autoencoder (VAH) enhances model robustness of generating insightful representations through a latent distribution learning mecha- nism (Kingma & Welling (2014). Vector quantisation VAE (VQ- VAE) employs the vector quantisation obtain discrete latent representation that improve the quality of image reconstruction and gen- 2017). Recent advances follow similar trend latent space exploration and generalize variants DAE many domains| IaannsLLarree Larsen et al (2016); Khattarr al. (2019): |Park al. (2020); Wang a (202T); Cai et al. [20T); Ran al. (2021). However, DAE and variants in image reconstruction suffer from the same problem; the parameters have high degree of freedom. In other words. parameters can only be learned through error backpropagation guided by gradients. Therefore, some meaningful constraints on the parameter space will be beneficial for learning.
cration van den Oord
Neural similarity in computational neuroscience: Many models in computational neuroscience have been proposed build relationship between stimuli and the corresponding neural spike responses, which are
ncural spike encoding. The goal of these models is increase similarity between predicted and true neural responses. Historically, much effort has been devoted to finding tuning curves for specific features of visual stimuli, such as the orientation of bars, to predict neural responses (Carandimi 1968) Niell al. al. 2008). This is feasible for some neurons in the primary visual cortex, but not all neurons. Nonlinear methods provide a more general approach to predicting neural responses, including energy models models/[ubel. [[bel & Wiesel (1922). the linear-nonlinear (LN) model and the LN-LN model Meyer et al. (2017). )) Traditional machine learning methods, such as generalized lincar models (GI (G) (Willmore et al. 2008). the multi-layer pereeptron (MIP) and the support vector regression (SVR) Das et al. (2019). have heen widely applicd in computational neuroscience to predict the neural responses to stimuli. More recently, hicrarchical structures have been found in both ventral visual pathway (DiCarlo et al. 2012 Vintch et al. 2015) Rowekamp & Sharpee 2017) and in
all, 2005 Dräger 1975 Hubel Wiesel, 1962
deep convolutional neural networks (Fukushima al. 1983 LeCinn et al. 1989 Krizhevsky et al. 2012). Underlying the similar hierarchy, the brain and CNN are believed to share similar neural representations for extracting features from stimuli layer by layer, from simple to complex. To test this hypothesis, Yamins el al Yamins & DiCarlo (2016) proposed computational neuroscience models investigate neural similarity. They report neural representation similarities hetween biological neurons along the ventral visual stream and artificial ncurons in different convolutional layers Yamins & DiCarlo (2016). Furthermore, CNN with cully-connected readout layer (CNN-FC) can directly map the convolutional features to neural responses Meintoshh el al. (2016). However, such fully-connected readout layer typically contains large number of parameters, To reduce numher of parumeters in the readout layer, CNN with a factorized readout
hierarchical CNNs
3

---

Under review conference paper at ICLR 2023
layer (CNN-FR) and CNN with a fixed mask (CNN-FM) are proposed which factorize convolutional features into spatial musk Klindt et all (2017). So far, CNN models with readout layers have become an important tool for studying neural similarity hetween the hrain and CNN.
METHOD
Here we first introduce the notation of data spaces and variables. We then present the framework of deep autoencoders with neural response (DAE-NR) in Sec. 3.1 Finally, we describe realizations of the DAE-NR framework using CAE and CNN-FR in Sec. 3.2
The data space of visual stimuli, the neural responses, and the features of stimuli in the i-th convo- lutional layer are denoted by X, S, and HG, The visual stimuli (Le., natural images) and the corre- sponding neural responses (i.e, neural spikes in the VI region) are represented as a E RRN xxxxxPXX PxC and E I XM respectively. The features of stimuli in the i-th convolutional layer are denoted as h E R x xXF,' with the sample size (N), the image resolution (P), the image channel (C), the kemel size of feature of stimuli (K), the number of kerclls (F), and the number of V1 neurons (M), respectively.
3.1 THE DAE-NR FRAMEWORK
The DAE-NR combines the function of DAE for image reconstruction and the role CNM for predicting neural responses. The DAE-NR framework consists of three parts, including an encoder
S1: X Hi, decoder f2: H. X. and mapping function J's Hi - S.
DAE: The standard DAEs and their variants (e.g., Masci et al. (011). VAE (Kingma & Welling 2014) and VQ- VA-VAE (van den Oord et al. 2017)) consist of an encoder, the latent space, and a decoder, which have become mainstream models for image reconstruction. In our work, instead of separating the encoder and decoder by the latent layer, split the encoder and decoder at the i-th layer of DAF. In this way, the architecture of our DAE turns to be: (i) encoder fi X - Hi to embed the input to neural representation in i-th layer; (ii) a decoder fa Hi + X, to reconstruct the input based on neural representations in i-th layer. We formally describe them as follows:
the encoder: h4 f((1). m)).
(I)
the decoder:
fa(02, h:),
where the é is reconstructed from the original image æ. Both the encoder and decoder are realized by neural networks with parameters Φ1 and (b2, respectively. The goal of DAE is to reconstruct the image by optimizing the the loss function L(Ø1 62).
CNM: The computational neuroscience models (e.g.. CNN-FR (Klindt et al. 2017 Bashivan et ul. 2019, CNN-FC |Mcintosh et al. 2016], and CNN-FM (Klindt et al. 2017) usually consist of the encoder and the readout layer. The encoder can be used to extract the image's features. The readout
layer is used to map the feature space H; to the neural responses in the space S. The mapping function fa H, S is
fh.
(2)
employ the (A) loss to optimize the representation similarity between artificial neurons (i.e., 8) and biological neurons neurons (i.e. s) in the mapping function.
The loss function of DAF-NR explicitly considers both the image reconstruction task in computer vision and the neural representation similarity task in computational neuroscience, as defined in Eq. equation g
DAE-NR:
02, L(d,, (22) + B C()),
(3)
where and 3 are the hyperparameters to trade off the image reconstruction task and the neural representation similarity task.
3.2 REALIZATIONS OF DAE-NR
Here, we realize DAE-NR by a toy model (i.e. CAE-FR) combining convolutional autoencoder (CAB) and a CNN with factorized readout (CNN-FR). important to note that the DAE-NR framework is compatible with different instantiations, We have full implementation in Table |
4

---

Under review conference paper at ICLR 2023
The variants of DAE-NR. The implementation details of CAE-FR are presented in Sec 3.2 The implementation of other variants is the same as CAF-FR. CAE, VAF, and VQ-V are the short name for convolutional autoencoder, variational autoencoder, and vector quantisation VAE, respectively. i E (1,2,3,4 representing the DAE-NR; extracted features from the different convolutional layers h,,
Table 1:
DAES
CAE CAE-FR; CAE-FC, CAE-FM,
VAE
VQ-VAR
CNMs
CNN with factorized readout (CNN-FR)
VAE-FR, VAE-FR- VAE-FC; VQ-VFE- VAR-FM, VQ-VAE-FM,
CNN with fully-connected readout (CNN-FC)
CNN with fixed mask (CNN-FM)
CA Both encoder and the decoder are reulized by convolutional neural networks with parameters Φ and dag respectively. The loss function of CAF is formulated as a fig norm:
C( 62) = || =|1 = æ f2( fi(ó1, x))1
(4)
where the d reconstructed from the original image ac.
The CNN-FR consists of two parts, the convolutional layers as the encoder and the factorized readout layer (Klindt et al. 2017: Bashivan et al. (201 Cadena et al. 2019 /Zhuang
CNN-FR:
et al. 202T). The convolutional layers convolve the image with number of kernels followed by batch normalization, resulting multiple feature maps. The readout layer pools the output of the convolutional layer (i.e., hs) by applying a spurse mask each neuron. Applying a sparse mask, the readout layer pools the output the convolutional layer (i.e., ht)) each neuron. Let denote that h2 lies in the feature space Hi and the neural responses in the space S. The mapping function
á fas 0., 0.)
Σ (0. h.)) 0. b.
(5)
where 02 is the spatial mask, 0a is the weights sum of all features ht, and is the bias. We use Poisson loss to optimize representation similarity between artificial neurons (i.c. 8) and biological neurons neurons (i.c. 8) in the mapping function as (6),
[(,, 0a)
Σ ($ s sllgg 8).
(6)
Previous studies have shown that the responses of V1 neurons to natural stimuli are sparse, and the activity of neural populations with higher sparseness exhibits greater discrimination against natural stimuli. (Vinje & Gallant 2000 WWeliky et al., [2033] Froudarakis et al. 2014:] Yoshida & Ohki, (020). Likewise, (Zhuang et al. 2017) has reported that the resemblance between the representation of biological neurons and artificaal neurons in higher convolutional layers exists only under the sparsity constraint on the CNN, regardless of any other factors (e.g., model structure, training algorithm, receptive field size, and property of training stimuli.. In our study, the representational similarity of VI neurons brought constraint for artificial neurons in this layer.
specific layer the CAE encuder (h,, (1,2,3, 1) with the sparsity
CAE-FR: FR: The CAE-FR combines the function of CAE and CNN-FR. The loss of CAE-FR explicitly considers both the image reconstruction task and the neural representation similarity task, as defined in Eq. equation 7
C(dr, dz, 0,, 0a) a+ | fa(02, file x) 2) + B Σ (Js(h,, 0.) s log Sh 8., 0.)).
(7)
Intuitively, the larger favors the reconstruction task, while the larger / biases toward the neural representation similarity task.
4 EXPERIMENTS AND RESULTS
4.1 EXPERIMENTAL SETTINGS
Datasets: We conduct experiments on a publicly available dataset with the gray-color images as visual stimuli and the corresponding neural responses. The neural response dataset obtained
5

---

Under review conference paper at ICLR 2023
from Antolik et al. 2016)) The neural responses in the dataset are recorded in three regions in the
primary visual cortex (VI) of sedated mice visually stimulated with the natural images (See Appendix
Fig.Fi The number of neurons over the three hrain regions is shown in the Appendix Table 7
Models: We realized DAE-NRs with nine different combinations of DAE and CNM (See Table D.
DAF-NRs are compared with DAF baselines for the image reconstructions task and compared with
CNM baselines for the neural representation similarity task. We choose CAE, VAE and VQ- VAE
as the DAE baselines and CNN-FR, CNN-FC and CNN-FM as the CNM baselines, respectively. Especially, we have the CNM with h, (CNM,, i E 3, AD) The CNM in DAE-NR can be readout from four different convolutional layers (ie., h, of the DAE encoder), resulting in four variants of
DAE-NRs (i.e. DAE-NR, i & 2,, 4)) which represent the CNM extracted from the different convolutional layers h,
The architecture of the DAE in DAE-NR is in the Appendix Table [4] The dimension of latent variable in CAE and VAE is set as 100; the prior of VAE and VQ- VÄE is the Gaussian distribution (0, 1) and uniform distribution, respectively. The dimension of latent embedding space in VQ-VAI is 32 x 256. Each convolutional layer is followed by a batch normalization with an ELU activation function, while the activation function in the final layer for image stronstruction is tanh. The CNM in DAE-NR shares the four different convolutional layers (h,, h2. hs. and h2). The CNM in DAE-NR can be readout from four different convolutional layers (i.e. h, of the encoder of DAE). so there are four variants of DAE-NR, i.e. DAE-NR,, i (1 4) representing the CNM extracted from the different convolutional layers h,
Network architectures:
Training procedures: We preprocess the images (i.e., reshape the size of the natural image to 32 x 32 x and normalize the intensity of the image to [-1.1 1) and then input them to the model. The model is trained with an initial learning rate of 0,001, and the early stopping strategy is applied based on a separated validation set. If the error in the validation set does not improve by 1000 steps. we return the best parameter set, reduce the learning rate by two, and train in the second time.Thh VAE is optimized by the evidence lower hound (EI.OO), while the VAR is optimized following the settings in (van den Oord al. 2017). The settings of hyperparameters for tasks in See Sec an2 Sec 5.3
detailed in Appendix Table S and 3 respectively. The Appendix Table 10 and Table Table 12
the settings the hyperparameters of different DAE-NR variants for image reconstruction Region
and neural similarity experiments on Region 1.2 and in Sec 4.4 respectively.
Table 2: The quantitative results of image reconstruction with all neurons in the region 1, 2, and respectively. The best result in each region under different metries is highlighted with boldface.
Region
Region
Region 2
Region 3
Metrie
MSEI
PSNR
SSIM
MSEL
PSNR1
SSIM
MSE.
PSNR1
SSIM
Model
CAB
0.022
23.709
0.771
0.024
23.338
0.754
0.081
17.039
0.561
CAE-FR,
0.021
23.829
0.776
0.023
23.392
0.753
0.044
19.751
0.763
CAE-FR2
0.021
23.779
0.775
0.023
23.440
0.759
0.043
19.819
0.764
CAE-FR, 0.021
23.778
0.775
0.024
23.330
0.755
0.043
19.789
0.761
CAE-FR, 0.022
23.721
0.773
0.023
23.491
0.760
0.059
18.462
0.668
Tasks: There are two tasks the experiment: 1) the image reconstruction (IR) task and the neural representation similarity (NRS) task. apply CAF-FR analyse the effects of neural responses on IR and effects of image reconstruction on NRS tasks in DAE-NR. We compare other DAE-NR variants with DAE and CNM for IR and NRS tasks to explore the generalisation cupability of DAE-NR. In the IR task, we use the mean squared error (MSE)), structural similarity (SSIM)) Wang et al. (2004), and peak hignal-to-noise ratio (PSNR 1) [Wag ct al. (2004) as metries to quantify the image reconstruction performance] We compare the DAE-NR, DAF-NR2, DAE-NR; and DAE-NR, with the standard DAE. In the NRS task, we use Pearson correlation cocfficient (PCCt) as a metric to quantitatively evaluate models. We implement the traditional end-to-end CNM, (i E [1, 2, 3, 4)) as baseline models.
'The up arrow i indicates that the higher the value, the better, and the down arrow I indicates that the lower, the better.
6

---

Under review conference paper at ICLR 2023
(1)
Region 1
logg doort
404-
20%-
Figure 3: The number of significant neurons (deep blue and deep red) and insignificant neurons (light blue and light red) in region I. 2, 3 in the image reconstruction task. The threshold for significance is & 0.05. The results show that our models (CAE-FRs) provide more significant neurons (red) than the baselinc models (hlue).
Table 3: The quantitative results of image reconstruction with constraints of significant neurons and insignificant neurons in the region 3. The best result under different metries is highlighted with boldface.
Metric
MSEL
SSIM
PSNRT
Significant?
Model
V
X
V
X
X
CAE-FR
0.043
0.125
0.761
0.332
19,784
15.168
CAE-FR;
0.047
0.082
0.743
0.547
19.467
16.970
CAE-FR;
0.049
0.116
0.724
0.362
19.245
15.463
CAE-FR, 0.047
0.045
0,740
0.752
19.497
19.628
4.2 BIOLOGICAL NEURAL RESPONSES IN CAE-FR IMPROVES IMAGE RECONSTRUCTION
Ten examples of reconstructed images by CAE-FR with neural responses in brain region 3 are presented in Fig. 2 It obvious that FR models, no matter which layer the neural response is mapped to, can better reconstruct the original images compared with the standard CAE The results of the brain region and 2 are illustrated in Appendix Fig. and Fig. 7 The quantitative comparisons of image reconstruction performance are listed in Table 2 It shows that CAE-FR,, CAE-FR, and CAE-FR, achieve the best performance when the network gets information from the neurons in the regions 1, 2, and 3, respectively. Although our model works in region 3 with signiticant improvement, there slight improvement in regions & 2. We deem that the neurons carrying more information about essential features of the visual stimulus results in Fig4 and Appendix Fig. 8 show that region has more significant neurons compared to Regions and 2. These results indicate that neurons in regional 3 carry more information than those in regions and 2 about essential features of the visual stimulus.
reconstruct better. The
Fig. 2 and Table suggests that the information from biological neurons could help CAF-FR for image reconstruction. Further questions are under what circumstances and to what extent CAF-FR is beneficial from the neural response. Our hypothesis is that only the biological neurons with high representation similarity artificial neurons can provide information and contribute to CAE-FR.
Iroigniiaai
Skgr Foar
100%
40.15
20%-
To test our hypothesis. we identify the biological neurons that significuntly correlate with the artificial neurons in the CAE-FR model on the region neurons in each brain region. Fig. 4 shows the number of significant neurons and insignificant neurons in region in the IR task. Then, from these two groups of neurons region separately. The quantitative results of the three metrics are shown in Table 3] confirming that insignificant neurons carry very little information about the images, and they can not
cAR-FR. CAR-FR; FRR FRR
well the insignificant
Figure 4: The number of significant neurons and insignificant neurons of the Region in the image recon- struction task. The threshold for sig- nificance is p S 0.05.
train the CAE-FRs with the information
the IR task
7

---

Under review conference paper at ICLR 2023
Table 4: Pearson correlation hetween the representations of artificial and biological neurons in region 1, 2, and 3, respectively. The best result in cach region under different layers of the model is highlighted with boldface.
Model
CNN-FR;
CAE-FR,
CNN-FRa
CAE-FR;
CNN-FR;
CAE-FR;
CNN-FR,
CAE-FR,
Region
Region
0.341
0.346
0.467
0.476
0.454
0.455
0.441
0.46.
Region 2
0.257
0.281
0.384
0.400
0.333
0.338
0.301
0.345
Repion
0.246
0.260
0.361
0.388
0.372
0.404
0.385
0.393
contribute improving image reconstruction performance. These experiments verify our hypothesis, indicating that information from significant neurons can better guide CAF-FR to reconstruct images.
4.3 IMAGE RECONSTRUCTION IN CAE-FR IMPROVES NEURAL REPRESENTATION SIMILARITY
Here, test image reconstruction's 's effects neural representation similarity in CAE-FR. The results of the Pearson correlation coefficient between the neural representation of artificial neurons and of biological neurons region region 2 and region are shown in Table 4 Our models (i.e., CAE-FR.) obtain larger PCC in all three regions compared to baseline models without image reconstruction loss (i.e., CNN-FR,). The results show that image reconstruction our model favors neural representation similarity. This phenomenon seems counter-intuitivy as there is tradeoff between the IR loss and NSR loss Eq. equation We hypothesize that the benefit may stem from feature learning for image reconstruction by CAE: more biological neurons would share representation similarity with artificial neurons when adding image reconstruction loss. In fact, our experimental results verify the hypothesis (Hig. F). suggesting that image reconstruction loss can help CAE-FR to improve neural representation similarity between artificial neurons and biological neurons.
4.4 GENERALIZABILITY ACROSS VARIANTS OF CNM AND DAE
To investigate the generalizability of DAE-NR variants, we compare different DAE-NR variants with baseline CNMs and DAES on IR and NRS tasks, respectively.
Comparisons in IR task: Table provides the quantitative comparisons among different models. Specifically, we compare different CNMs variants (i.e., CAF-FR, CAF-FC,, and CAF-FM,) with the standard CAE the IR task. The results show that the best performance is achieved when the network leverages information from biological neurons in Region 3. Moreover, examine generalizability in other DAE variants, we implement VAE-FM;) and VQ-VAE variants (VQ- VAE-ARR VQ- VAE-FC, and VQ-VAE-),, and compare their performance with the standard VAE and VQ-VAE on the IR task, respectively. Our results show that DAR-NR, VAR variants, and VQ-VAE variants that use information from biological neurons have better IR performances than baseline models.
VAE variants (VAE-FR,, VAE-FC, and
Comparisons in NSR task: Pearson correlation coefficient (PCCT) between the neural representation of artificial and biological neurons in region 1, region 2, and region 3 are shown in Table 6 To verify the effect of different DAEs for CNN-FR. compare the variants of CAE (CAE-FR,, CAE-FR,,RCACCC.C. CAE-FC, and CAE-FM,) with CNN-FR the NSR task. Our models obtain higher PCC in all three regions compared to the bascline models. Furthermore, to validate the function of other CNM variants with different constraints VQ-VAE-FC, and the variants of CNN-FM (CAE-FM, VAE-FM, and VA-VAE-FM,) to compare the performance on the NRS task, respectively. Our results show that the hiologically-inspire. variants of both CNN-FC and CNN-FM via our DAE-NR framework have better performance than the original models.
DAES. we apply the variants of CNN-FC (CAE-FC, VAE-FC, and
Together, demonstrate that the DAE-NR good generalization variants of CNM and DAE. Adding neural information to guide DAE can help reconstruct visual stimuli: in turn, integrating image reconstruction loss into CNM can improve the representation similarity between artificial and biological neurons.
8

---

Under review conference paper at ICLR 2023
Table 5: Quantitative results of image reconstruction with all neurons in the region 3. The best results under different metries for each region are highlighted in bold.
b:
h;
Model
Metric
MSEL
PSNR
SSIM
MSEI
PSNR1
SSIMI
MSEI
PSNR1
SSIM:
MSEL
[SNRT
SSIM1
CAE:
0.074
17.285
0,593
0,074
17:289
0.395
0.074
17:289
0.593
0.074
17:299
0.592
CAE-FR. (Ours)
0.043
19,832
0,761
0.043
19,852
0.767
0.015
19.698
0.758
0.051
19.107
0.718
PC (Ours)
0,043
19,876
0.768
0.044
19.811
0.763
0.061
13.290
0.044
19,502
0.760
CAEFM.
B.445
19,692
0.761
19.843
0.144
19,818
0.944
15.933
0.402
15.983
(Ours)
0.414
16.055
0.101
0.102
VAE-FC. (Ours)
0.103
16,000
0:112
0,101
16.094
0:112
0.101
16.09
0.116
0.801
16,089
0.418
VAE.FM. (Oors)
0.102
16,946
0.415
0.101
16.073
0.414
0.102
16,025
0.413
0.103
15.017
0.412
VO-VAE,
D.02
16.39)
0:407
12082
16.059
0.401
0082
0.27
0182
16329)
17497
VQ-VAE-FR, (Ours)
0.071
17,636
0.569
0.069
17.741
0.568
0.073
17 48-
0.557
0.011
17.581
0.560
VQ VAB FC, (Ours)
0,076
17.264
0.525
0.075
17.324
0.525
0.072
17.539
0.547
0.067
17.839
0.551
VQ-VAE-FM, (Ours)
0,068
17,782
0,572
0.068
17.791
0.571
0.091
196
0.546
0.072
17. 77.50
0.542
Table 6: Results of PCC1 between representations of artificial and biological neurons three brain regions. The best results are highlighted bold.
Region
Region
Region 2
Region
Model
h
bg
hg
h2
h;
h2
hg
h,
h;
bg
hg
h,
CNN-FR,
0.335
0.455
0.432
0.431
0.254
03:775
0.300
0.305
0.244
0.348
0.350
0.359
CAE-FR; (Ours)
0.338
0.471
0.449
0.452
0.277
0.396
0.319
0.348
0.254
0.377
0.397
0.394
VAE-FR, (Ours)
0.338
0.479
0.455
0.450
0.277
0.386
0.317
0.342
0.269
0.375
0.382
0.404
VO-VAE-FR, (Ours)
0.349
0.295
0.382
CNN-PC,
0.343
0.349
0.397
0.435
0.247
0.268
0.292
0.328
0.325
0.316
0.344
0.374
CAE-PC; (Ours)
0.349
0.382
0,415
0.438
0,266
0.281
0.319
0.341
0.335
0.341
0.384
0,399
VAE.FC. (Ours)
0.418
0.438
0:379
VQ-VAF-FC, (Ours)
0.347
0.377
0,415
0.434
0,264
0.289
0.313
0.361
0.338
0.341
0.388
0,427
CNN-FM,
0.150
0.222
0.110
0.220
0.123
0.186
0.162
D0064
0.102
CAB-EM, (Ours)
0.154
0.235
0.115
0.227
0.134
0.223
0.144
0.182
0.033
0.171
0.109
0,203
VAF-FM, (Ours)
0.149
0.238
0,125
0.240
0.136
0.224
0.131
0.176
0.074
0.172
0.111
0,201
VQ-VAE-FM, (Ours)
0.153
0.115
0.221
0.221
0.162
0.111
CONCLUSION
In this study, we proposed the DAE-NRs. hybrid framework that integrates the neural response into deep autoencoder models. Inspired by CNN-based neural activity prediction models in computational ncuroscience Klindt et al. (2077); Bashivan et al. (2019), we leveraged Poisson loss to bring neural information to specific layer of DAE, resulting in better image reconstruction performance. In turn, the IR task facilitates feature learning in DAE and leads to higher neural representation similarity between biological and artificial neurons. Our work bridges the gap between DAE for image reconstruction and CNM for neural representation similarity.
Broader impact Besides the image reconstruction task and neural representation similarity task, DAE-NR could enable many other potential applications in the future. For instance, DAE-NR provides natural way synthesize images that maximize control neural activity, compared with the method proposed by [Bashivan al. (2019). Also, DAE-NR can serve as data engine evirtual experiments, such as synthcsizing neural responses visual stimuli. Morcover, important to investigate the generalizability (2016): Khattar et al. (211); Park al. (2020) Wang et all (202T): Cai et all (202T). on various types of stimuli (e.g.. sound and face) and other tasks (e.g.. classification, generation and detection Ran ct all (2021)). Although we only tested DAF-NR with mice neural data, the DAF-NR framework can be easily extended to the neurons in primates (e.g., monkey ZZhang et all (202 or humans). DAE-NR opens a new window for combining artificial intelligence and brain intelligence.
DAE-NR on variants of auto-encoder Larsen et all
REFERENCES
Ján Antolik, S. Hofer, J. Bednar, and T. Mrsic-Flogel. Model construined by visual hicrarchy improves prediction of neural responses to natural scenes. PLaS Compatational Biology, 12, 2016.
9

---

Under review conference paper at ICLR 2023
P. Bashivan, Kohitij Kur, and J. DiCarlo. Neural population control via doep imuge synthesis. Science, 364, 2019.
Santiago A. Cadena, George H Denfield, Edgar Y. Walker, Leon A. Gatys, A. Tolias, M. Bethge, and Alexander S. Ecker. Deep convolutional models improve predictions macaque vi responses to natural images. PLoS Computational Biology. 2019.
Yujun Cai, Yiwei Wang. Yiheng Zhu. Tat-Jen Cham. Jianfei Cai, Junsong Yuan, Jun Liu, Chuanxia Zheng. Sijie Yan, Henghui Ding, al. unified human motion synthesis model via conditional variational auto-encoder. In Procendings of the IEEF/CV International Conference Computer Vision, pp. 11645-1 1655, 2021.
Matteo Carandini, Jonathan Demh, Valerio Mante, David J. Tolhurst, Yang Dan, Bruno Olshausen, Gallant, and Nicole C. Rust. Do we know what the early visual system does? The Journal of Neuroscience, 25:10577 10597, 2005.
Gautham P Das, Philip J Vunce, Dermot Kem,, Sonyu Colemun, Thomas McGinnity, and Jian Liu. Compu- tational modelling of salamander retinal ganglion cells using machine learning approaches. Neurocompading, 325:101-112, 12, 2019.
DiCarlo, D. Zoccolan, and N. Rust. How does the brain solve visual object recognition? Neuron, 73:415-434. 2012.
Ursula C. Dräger. Receptive fields of single cells and topography mouse visual cortex. Journal of Compurative Neurology, 160, 1975.
Callie Federer, Haoyan Xu, Alona Fyshe, and Joel Zyeherherg. Improved object recognition using neural networks trained to mimic the brain's 's statistical properties. Neural networks: the official journal of the International Neural Network Society, 131:103-1 14, 2020.
Emmanouil Froudarakis, Philipp Berens, Alexunder S. Beker, R. Jumes Cotton, Fabian H Sinz, Dimitri Yatsenko, Peter Saggau, Matthias Bethge, and Andreas Savas Tolias. Population code in mouse v1 facilitates read-out natural scenes through increased sparseness. Nature neuroscience, 17:851 857, 2014.
K. Fukushima, S. Miyake, and T. Ito, Noocognitron: A neural network model for a mochanism of visual pattern recognition. JEEE Transactions on Systems, Man, and Cybernetics. BMCC-3:826-333 834, 1983.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville, Deep learning. MIT press, 2016.
Geoffrey Hinton and Ruslan Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 313(5786):004-500 2006.
David H Hubel and Torsten N Wiesel. Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. The Journal of physiology, 160(1):106-154 1962.
David H. Hubel and Torsten N. Wiesel, Receptive fields and functional urchitecture of monkey striate cortex. The Journal of Physiology, 195, 1968.
Dhruv Khattar, Jaipal Singh Goud, Manish Gupta, and Vasudeva Varma. Mvae: Multimodal variational autoencoder for fake news detection. In The world wide web conference, pp. 2915-2921, 2019.
Diederik P. Kingma and Max Welling. Auto-cncoding variational baycs. International Conference Learning Representationss/CI.. ICLR), 2014.
David Klindt, Alexander S Ecker, Thomas Ealer, and Matthias Bethge. Neural system identification for large populations separating what and where. In Advances in Newral Information Processing Systemns (NeurlPS), pp. 15509-3519, 2017.
A. Krizhevsky, Ilya Sutskever, and Genffrey E. Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60:84- - 90, 2012.
Anders Boesen Lindbo Larsen, Sdren Kuae Sonderby, Hugo Larochelle, and Ole Winther. Autoencoding beyond pixels using a learned similarity metric. In International conference machine learning. pp. 1566-1566. PMLR. 2016.
Y. LeCun, B. Boser, J. Denker, D. Henderson, R. Howard, W. Hubbard, and L. Jackel. Handwritten digit recognition with a back-prooagtttoon network. In Advances in Newral Information Processing Systems (NewriPS), 1989.
10

---

Under review as conference paper at ICLR 2023
Tom Macpherson, Anne Churchland, Terry Sejnowski, James DiCarlo, Yukiyasu Kumitani. Hidchiko Takahushi, and Takutoshi Hikida. Natural and artificial intelligence: A brief introduction to the interplay between ai and neuroscience research. Neural Networks, 2021.
Jonathan Masci, Ucli Meier, Dun Cireçan, und Jurgen Schmidhuber. Stacked convolutional suto-encoders for hierarchical feature extraction. In International conference artificial neural networks, pp. 52-59. Springer, 2011.
Lane McIntosh, Niru Maheswaranathan, Aran Naycbi, Surya Ganguli, and Stephen Buccus. Deep leurning models of the retinal response to natural scenes. Advances in newral information pricessing xystenes NNuurPPS). 19:1369-1377. 2016.
Arne Meyer, Ross Williamson, Jennifer Linden, and Maneesh Sahani, Models neuronal stimulus- response functions: elaboration, estimation, and evaluation. Frentiers in systems neuroscience, 10:109, 2017.
Cristopher M. Niell, Michael P. Stryker, and Wendell M. Keck. Highly selective receptive fields in mouse visual cortex. The Journal of Neuroscience, 28:7520- 7536, 2008.
Thesung Park, Jun- Yan Zhu, Oliver Wang, Jingwan La, Eli Shechrman, Alexei Efros, and Richard Zhang. Swapping autoenender for deep image manipulation. Advances in Neural Information Processing Systems, 33:7198-7211 2020.
Xuning Ran, Mingkun Xu, Lingrui Mei, Qi Xu, and Quanying Lin. Detecting out-of distriibbttoo samples via variational auto- eiteder with reliahle uncertainty estimation, Newral Networks, 2021.
Saiprasdd Ravishankar, J. C. Ye, and Jeffrey A. Fessler. Image reconstruction: From sparsity to data- dala-adaptive methods and machine learning. Proceedings of the IEEE, 108:86-109, 2020.
Ryan J. Rowekamp and T. Sharpee. Cross-orienaation suppression in visual area v2, Nature Communications, 8, 2017.
Shahd Safarami, Arne Nix, Konstantin Friedrich Willeke, Santiago Cadena, Kelli Restivo, George Denfield, Andreas S Tolias, and Fabian H Sinz. Towards robust vision by multi-task learning on monkey visual cortex. In Advances in nearal informatiio processing systems (NewrlPS), 2021.
Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J. Majaj, Rishi Rajalingham, Blias B, Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy Kailyn Schmidt, Daniel Yamins, and James J. DiCarlo. Brain-score: Which artificial neural network for object recognition is most bruin-like? bioRxiv, 2018.
Jakub Tomezak and Max Welling, with vampprior. International Conference on Artificial Intelligence and Statistics, pp. 1214-1223, PMLR, 2018.
Aäron den Oord, Oriol Vinyals, and K. Kavukcuoglu. Neural discrete representation learning. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoiee Manzagol, Extractin und composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096-1103, 2008.
William Vinje and Jack L. Gallant. Spurse coding and decorrelation in primary visual cortex during natural vision. Science, 287 5456:1273-6. 2000.
Brett Vintch, Movshon, and Ecro Simoncelli. A convolutional subunit model for neuronal responses in macaque vl. The Journal of Neuroscience, 35:14829 14841, 2015.
Edgur Y Walker, Fabian H Sinz, Erick Cobos. Taliah Muhummad, Emmanouil Froudurakis. Paul G Fabey, Alexander S Eoker, Jucob Reimer. Xaq Pitkow, and Andreas S Tolius. Inception loops discover what excites neurons most using deep predictive models. Nature neuroscience, 22(12):2000220
2019.
Guojun Wang, Weijun Li. Liping Zhang, Linjun Sun, Peng Chen, Lina Yu, and Xin Ning. Encoder-x: Solving unknown coefficients automatically polynomial fitting by using un autoencoder. IEEE Transactions on Neural Networks and Learning Systems, 2021.
Zhou Wang. Alan Conrud Bovik, Hamid R. Sheikh, and Ecro P. Simoncelli. Image quality ussessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13:600-612, 2004.
Michael Weliky, József Fiscr, Ruskin H Hunt, and David N Wagner. Coding of natural scenes in primary visual cortex. Neuron, 37:703-718, 2003.
11

---

Under review conference paper at ICLR 2023
B. Willmoree, R. Prenger, M. C. Wu, und J. Gallant. The berkeley wavelet transform: A biologically inspired orthogonal wavelet transform. Neural Computation. 20:1537-1564, 2008.
Duniel Yamins and DiCarlo. Using goal-driven deep leaminn models to understand sensory cortex. Nature Neuroscience, 19:356-365, 2016.
Takashi Yoshida and Kenichi Ohki. Natural images reliably represented by sparse variable populations of neurons in visual cortex. Nature Communicution, 11, 2020.
Jie Zhang, Xisocang Zhu, Shanshan Wang, Hosscin Esteky, Yonghong Tian, Robert Desimone, and Huihui Zhou. Visual attention in the fovea and the periphery during visual search. bioRxiv, 2021.
Chengxu Zhuang. Yulong Wang. Daniel Yamins, and Xiaolin Hu. Deep learning predicts correlation between a functional signature of higher visual areas and sparse firing of neurons. Frontiers in Computational Neuroscience, 11, 2017.
Chengxu Zhuang, Siming Yan, Aran Nayebi, Unsupervised neural network models of the ventral visual stream, Proceedings the National Academy of Sciences of the United States of America, 118, 2021.
Schrimpf, Michael Frank, J. DiCarlo, Yamins.
12


Reviews:
[{"review_id": "fyxb02JbH7", "reviewer": "Reviewer_bubd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper builds a NN to match (a) image reconstruction and (b) neural similarity of the inner layers of the CNN to neural representations (NR) in a mouse visual cortex. The loss function minimizes image error and NR error.\n \nThe results assert that the NN does much better than a convolutional autoencoder (CAE) on image reconstruction, and also (as expected by construction) on NR.\n\nThe basic notion (if I got it right) is that if you fit the encoding layers of a CAE to neural representations from a mouse visual cortex, the CAE will have better image reconstructions. It's unclear to me is whether this effect (reported in the paper) is due to the extra parameters provided by the attached CNM, or by the sparsity prior on model weights, rather than to the NR matching. If it is due strictly to the NR matching, that would be, I think, a big thing.", "summary_of_the_review": "I am perhaps the wrong person to review this work, since I am not confident that the suggested parallels between CNNs and BNNs,  which motivate this work, are more than superficial (eg, the main layers can be made to somewhat correspond). Also, I am not up on this literature. This poor fit is reflected in my Confidence score and other scores.\n\nThe accumulation of non-clarities (see \"miscellaneous\" items above), combined with my skepticism about the extent of the CNN-BNN mapping, make me less favorable to the work. But I emphasize my lack of confidence in this assessment.", "strength_and_weaknesses": "(strengths)\n\nIt is an interesting idea to merge a CAE and a Convolutional neural model (CNM).\n\n(weaknesses)\nI am troubled by the basic assumption that the neurons in mouse V1, V2, etc can be mapped to layers in a CNN. \nIt is not clear if the gains in image reconstruction are due to the inner layer - Vi fittings, or to extra parameters or the imposed sparsity.\n\nMiscellaneous:\n\nbottom pg 1: Are the structural and functional relationships between ANNs and BNNs sufficiently deep to build on, or are they superficial?\n\nPage 2 \"however, hot to ... question\": This sets up an expectation. Perhaps remove it, or say that you address this question.\n\nFig 1: I believe that layers in a CNN (CAE) do not map 1-to-1 to visual layers in mice. Rather, previous studies showed that given a DNN with M layers, you can map a subset of those layers to Vi\n\nFig 2: why are the CAE images so fuzzy? Doesn't it give good image reconstruction? This is counter-intuitive, unless it is way too small a network, which would suggest that the improved results from an added CNM + fitting are due to extra parameters.\n\nFig 3: Are you able to add error bars to see if these differences are big? Also, is a higher number of significant neurons better? I believe that in Vi, given a constrained set of images, several neurons might be relatively uninvolved, so higher might be inaccurate. (I don't know).\n\nTypos: there are various repeated words (eg \"however however\") , and also a repeated comma.\n\nPage 4: \"turns to be\" -> \"turns out to be\"\n\nBottom of page 4: add sub-heading after description of CNM to clarify that the topic is now the hybrid DAE-NR.\n\neqn 3: beta = 1- alpha?\n\nAbbreviations could be made clearer, noted at time of first introduction. An example is CNM, which I thought was a typo for most of the paper. Also, what is VQ?\n\nPlease note when figures and tables (eg fig 5) are in the Appendix.\n\n\"Network architectures\": Maybe move the architecture description to the appendix where it can be more complete. The 8C88-9CD77 notation is confusing and leaves out detail.\n\nAlso: what does the \"32 x 256\" dimension for VQ-VAE relate to?\n\npage 5\" by applying a sparse mask\". the explanation of why this is done comes much later. Could you move that explanation so that it follows directly (or precedes)?\n\nTable 2 and 5: How do you restrict the CAE image reconstruction to use only a single layer? This is not described in 3.2.\n\nTable 5: do you have error bars (std devs)? Related: what is the appropriate number of decimal places?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_FIOL299uH", "reviewer": "Reviewer_YdJC", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper shows that optimizing a CAE model to fit neural data and reconstruct input images at the same time, actually improves the performance of both tasks.  The finding that integrating neural response into deep autoencoder models can improve its performance is quite interesting and novel. ", "summary_of_the_review": "This is a relatively simple paper that shows optimizing a CAE model to fit neural data and reconstruct input images simultaneously can actually improves the performance of both tasks.  The approach of integrating neural fitting with CAE for image reconstruction is novel and that fact that it actually shows improvement in performance is reasonable but still a bit surprising, and hence interesting. ", "strength_and_weaknesses": "Strength: Transfer learning based on latent features of a CAE for neural data fitting is not new.  The finding that introducing a loss to maximize the representational similarity of the latent feature representations in a CAE with the real neural activities can improve the image reconstruction performance of an autoencoder is novel and interesting.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "ZAnrB1986q", "reviewer": "Reviewer_vnH9", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, an image autoencoder is trained with two loss functions: 1. Image reconstruction loss and 2. Poisson loss to optimize representation similarity between artificial and biological neurons. \n\nThe authors show that by training in the above way:\n1. Image reconstruction is better than the case without Poisson loss\n2. The model shows high resemblance between artificial/biological neurons as compared to standard end to end models", "summary_of_the_review": "The idea although not new is applied in a new way to improve both the performance on image reconstruction task and neural response similarity task. \n\nDue to limited testing on brain dataset even though there are lots of publicly available benchmarks I am inclined towards giving it a borderline reject. \n\nHowever, if authors provide a valid reason to not use those benchmarks or provide new results I will be happy to update my rating.", "strength_and_weaknesses": "Strengths:\n1. The idea although simple improves the performance on both the tasks.\n2. Experiments seem sufficient enough for validating the proposed idea\n3. The idea is evaluated using different autoencoders suggesting generalizability of the claims.\n\nWeakness:\n1. Minor: the idea presented here is simple multitask network. I am not sure if this is a \"framework\"\n2. In Table 2: if CAE is trained without any brain data why is there any variation in result of region 1, 2 and 3?\n3. What is the difference between NR and FR.? there are lots of acronyms in the paper making it difficult to follow. I would suggest unifying the acronyms as in Tables only CAE is used but in the text there are multiple mentions of DAE-NR. It was only possible to figure out after multiple reads. \n4. There are lots of publicly available datasets from Brainscore (Schrimpf et al. 2018) and others. Why was this particular dataset chosen? \n5. Generalizability is only shown in V1, using other public datasets maybe it might have made sense to show that approach is generalizable to different brain regions as well.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AQEZ8cZXuc", "reviewer": "Reviewer_n7th", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a deep neural network that jointly learns 1) to\nencode images as a variational autoencoder, and 2) to use the images\nto predict neuronal activities recorded in mouse visual cortex (in\nresponse to the same set of images, presented as visual stimuli). By\nsharing the representations used to perform both tasks, the goal of is\nto improve the neural prediction performance by using the autoencoder\nas a regularizer, and, reciprocally, to improve the autoencoder's\nreconstruction quality by making its internals more similar to\nputative processes of biological vision. ", "summary_of_the_review": "In my opinion, this paper fails to back up some of its core claims and\nto justify the (claimed) relevance for the proposed architecture from\na computer vision standpoint. Therefore, my recommendation is not to\naccept this work for publication.", "strength_and_weaknesses": "## Strengths\n1. The idea of using a VAE as a regularizer for a neural activity\n   prediction task is interesting and (to my knowledge) novel, and\n   executed in a clean way. The methods and results of the research\n   are reasonably well described in the paper (with a few minor\n   exception, noted below).\n2. The claim that inclusion of the VAE improves representational\n   similarity is backed up by the data, if it is interpreted in the\n   narrow sense of only being related to the specific architecture at\n   hand. However, this comparison is made in complete isolation from\n   the existing literature on representational similarity (see point 3\n   below).\n\n## Weaknesses\n1. It is not clear what is meant by the \"image reconstruction\n   problem\", and how can the proposed architecture bring progress on\n   that. In the introduction, it is stated that the image\n   reconstruction problem is an important one in computer vision, and\n   a couple of references are given to papers by J Fessler on medical\n   applications. Is there any way in which DAE-NR could contribute to\n   this type of applications?\n2. In the abstract and the introduction, the \"guidance of neural\n   information\" is offered as a general principle that should somehow\n   improve image reconstruction performance. However, in the present\n   work, this is only shown to be true for the neural data from one of\n   the three regions recorded in  Antonlik et al\n   2016 (region 3). Including\n   data from the other recorded regions (regions 1 and 2) did not\n   improve appreciably the performance of the autoencoder (table 2),\n   invalidating the generality of the approach, but no comment or\n   justification is proposed for this fact. What makes neurons in\n   region 3 special? Is there a theoretical or practical reason why\n   inclusion of the data for these neurons, but not the others, should\n   help the autoencoder? Or is it simply that these neurons happen to\n   carry a bit more information than those in regions 1 and 2 about\n   important features of the visual stimulus, and therefore their\n   inclusion helps learning better representations in the autoencoder?\n3. For neural activity prediction, no comparison is made with the\n   state of the art (only comparisons with variants of the proposed\n   architecture are done). This is particularly surprising given that\n   the neural data used in this paper comes from another work where\n   neural prediction was performed. In Figure 5A of Antonlik et al\n   2016, the correlation coefficient between model prediction and\n   neural activity is shown to be about 0.5 for all three regions. In\n   the present work (Table 4), this metric is somewhat lower (a range\n   of about 0.3-0.5, considering 8 different models).\n4. The analysis of \"significant neurons\" in table 3 seem\n   overinterpreted. If I've understood correctly the procedure for\n   this analysis, it is as follows: after training the network,\n   neurons (only those for region 3, in the main text) are divided\n   among those for which the model prediction are better than chance\n   (the \"significant\" ones), and those for which they aren't. Then the\n   network is re-trained, from scratch, using only data from either\n   significant or insignificant neurons. It is observed that the\n   significant neurons contribute to improving the VAE, while the\n   insignificant ones don't. It is claimed that \"These experiments\n   verify our hypothesis, indicating that information from significant\n   neurons can guide CAE-FR to better reconstruct images\". I am not\n   sure what is added by this analysis: the simplest explanation is\n   that the insignificant neurons don't carry any (or carry very\n   little) information about the images, so they are effectively\n   random noise generators for the purpose of image reconstruction,\n   and therefore can't possibly contribute to improving it. In other\n   words, in my opinion this analysis is just showing in a different\n   way that some neurons carry some information about the images, and\n   some don't. In this sense, saying that information from informative\n   neurons can contribute to a better reconstruction is somewhat of a\n   vacuous claim; for instance, one could do even better by including\n   skip connections directly from the image input layer.\n5. Additionally, the numbers in the \"significant\" column in Table 3\n   seem uniformly worse than the numbers in Table 2. In other words,\n   it seems that training with the significant neurons alone is better\n   than training with the insignificant neurons alone, but worse than\n   training with all neurons. This seems relevant and should be\n   discussed. How do the authors square this fact with the statement\n   that \"insignificant neurons jeopardize CAE-FR\"?\n6. Why are the images in Figure 2 so small and featureless? The\n   abstract states that the input images are \"natural images\", and the\n   paper says that the data is taken from Antonlik et al 2016. But the\n   images that paper (see for instance Figure 1 in Antonlik et al)\n   look completely different, and the methods section of that paper\n   states \"The stimulus set was composed of static scenes from David\n   Attenborough’s BBC documentary Life of Mammals, depicting natural\n   scenes such as landscapes, animals or humans. Images were scaled to\n   have 256 equally spaced luminance steps, and were composed of\n   384×208 pixels, and expanded to fill the screen\". I must be missing\n   something, because the images in Figure 2 look like random blobs to\n   me.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject