Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, an image autoencoder is trained with two loss functions: 1. Image reconstruction loss and 2. Poisson loss to optimize representation similarity between artificial and biological neurons. 

The authors show that by training in the above way:
1. Image reconstruction is better than the case without Poisson loss
2. The model shows high resemblance between artificial/biological neurons as compared to standard end to end models
strength_and_weaknesses: Strengths:
1. The idea although simple improves the performance on both the tasks.
2. Experiments seem sufficient enough for validating the proposed idea
3. The idea is evaluated using different autoencoders suggesting generalizability of the claims.

Weakness:
1. Minor: the idea presented here is simple multitask network. I am not sure if this is a "framework"
2. In Table 2: if CAE is trained without any brain data why is there any variation in result of region 1, 2 and 3?
3. What is the difference between NR and FR.? there are lots of acronyms in the paper making it difficult to follow. I would suggest unifying the acronyms as in Tables only CAE is used but in the text there are multiple mentions of DAE-NR. It was only possible to figure out after multiple reads. 
4. There are lots of publicly available datasets from Brainscore (Schrimpf et al. 2018) and others. Why was this particular dataset chosen? 
5. Generalizability is only shown in V1, using other public datasets maybe it might have made sense to show that approach is generalizable to different brain regions as well.
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is readable but can be improved signficantly with a rewrite. I would encourage authors to reduce the number of acronyms used in the paper to help readability

Quality: The quality of experiments is good on the DNN side. The experiment however is done only on one dataset of brain recordings and how it compares to state of the art models in neural activity prediction on Brainscore. 

Originality: The idea to use neural data as an auxillary loss along with an image related task is not new. However, here it is applied to a new problem and shows improvement in both reconstruction and representational similarity with biological neurons. The results therefore are new.


summary_of_the_review: The idea although not new is applied in a new way to improve both the performance on image reconstruction task and neural response similarity task. 

Due to limited testing on brain dataset even though there are lots of publicly available benchmarks I am inclined towards giving it a borderline reject. 

However, if authors provide a valid reason to not use those benchmarks or provide new results I will be happy to update my rating.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper builds a NN to match (a) image reconstruction and (b) neural similarity of the inner layers of the CNN to neural representations (NR) in a mouse visual cortex. The loss function minimizes image error and NR error.
 
The results assert that the NN does much better than a convolutional autoencoder (CAE) on image reconstruction, and also (as expected by construction) on NR.

The basic notion (if I got it right) is that if you fit the encoding layers of a CAE to neural representations from a mouse visual cortex, the CAE will have better image reconstructions. It's unclear to me is whether this effect (reported in the paper) is due to the extra parameters provided by the attached CNM, or by the sparsity prior on model weights, rather than to the NR matching. If it is due strictly to the NR matching, that would be, I think, a big thing.
strength_and_weaknesses: (strengths)

It is an interesting idea to merge a CAE and a Convolutional neural model (CNM).

(weaknesses)
I am troubled by the basic assumption that the neurons in mouse V1, V2, etc can be mapped to layers in a CNN. 
It is not clear if the gains in image reconstruction are due to the inner layer - Vi fittings, or to extra parameters or the imposed sparsity.

Miscellaneous:

bottom pg 1: Are the structural and functional relationships between ANNs and BNNs sufficiently deep to build on, or are they superficial?

Page 2 "however, hot to ... question": This sets up an expectation. Perhaps remove it, or say that you address this question.

Fig 1: I believe that layers in a CNN (CAE) do not map 1-to-1 to visual layers in mice. Rather, previous studies showed that given a DNN with M layers, you can map a subset of those layers to Vi

Fig 2: why are the CAE images so fuzzy? Doesn't it give good image reconstruction? This is counter-intuitive, unless it is way too small a network, which would suggest that the improved results from an added CNM + fitting are due to extra parameters.

Fig 3: Are you able to add error bars to see if these differences are big? Also, is a higher number of significant neurons better? I believe that in Vi, given a constrained set of images, several neurons might be relatively uninvolved, so higher might be inaccurate. (I don't know).

Typos: there are various repeated words (eg "however however") , and also a repeated comma.

Page 4: "turns to be" -> "turns out to be"

Bottom of page 4: add sub-heading after description of CNM to clarify that the topic is now the hybrid DAE-NR.

eqn 3: beta = 1- alpha?

Abbreviations could be made clearer, noted at time of first introduction. An example is CNM, which I thought was a typo for most of the paper. Also, what is VQ?

Please note when figures and tables (eg fig 5) are in the Appendix.

"Network architectures": Maybe move the architecture description to the appendix where it can be more complete. The 8C88-9CD77 notation is confusing and leaves out detail.

Also: what does the "32 x 256" dimension for VQ-VAE relate to?

page 5" by applying a sparse mask". the explanation of why this is done comes much later. Could you move that explanation so that it follows directly (or precedes)?

Table 2 and 5: How do you restrict the CAE image reconstruction to use only a single layer? This is not described in 3.2.

Table 5: do you have error bars (std devs)? Related: what is the appropriate number of decimal places?


clarity,_quality,_novelty_and_reproducibility: I found the paper a bit difficult to follow (see miscellaneous comments above). 

I do not see indication of a public codebase.
summary_of_the_review: I am perhaps the wrong person to review this work, since I am not confident that the suggested parallels between CNNs and BNNs,  which motivate this work, are more than superficial (eg, the main layers can be made to somewhat correspond). Also, I am not up on this literature. This poor fit is reflected in my Confidence score and other scores.

The accumulation of non-clarities (see "miscellaneous" items above), combined with my skepticism about the extent of the CNN-BNN mapping, make me less favorable to the work. But I emphasize my lack of confidence in this assessment.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper shows that optimizing a CAE model to fit neural data and reconstruct input images at the same time, actually improves the performance of both tasks.  The finding that integrating neural response into deep autoencoder models can improve its performance is quite interesting and novel. 
strength_and_weaknesses: Strength: Transfer learning based on latent features of a CAE for neural data fitting is not new.  The finding that introducing a loss to maximize the representational similarity of the latent feature representations in a CAE with the real neural activities can improve the image reconstruction performance of an autoencoder is novel and interesting.

clarity,_quality,_novelty_and_reproducibility: clear, novel and the quality seems good.
summary_of_the_review: This is a relatively simple paper that shows optimizing a CAE model to fit neural data and reconstruct input images simultaneously can actually improves the performance of both tasks.  The approach of integrating neural fitting with CAE for image reconstruction is novel and that fact that it actually shows improvement in performance is reasonable but still a bit surprising, and hence interesting.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a deep neural network that jointly learns 1) to
encode images as a variational autoencoder, and 2) to use the images
to predict neuronal activities recorded in mouse visual cortex (in
response to the same set of images, presented as visual stimuli). By
sharing the representations used to perform both tasks, the goal of is
to improve the neural prediction performance by using the autoencoder
as a regularizer, and, reciprocally, to improve the autoencoder's
reconstruction quality by making its internals more similar to
putative processes of biological vision. 
strength_and_weaknesses: ## Strengths
1. The idea of using a VAE as a regularizer for a neural activity
   prediction task is interesting and (to my knowledge) novel, and
   executed in a clean way. The methods and results of the research
   are reasonably well described in the paper (with a few minor
   exception, noted below).
2. The claim that inclusion of the VAE improves representational
   similarity is backed up by the data, if it is interpreted in the
   narrow sense of only being related to the specific architecture at
   hand. However, this comparison is made in complete isolation from
   the existing literature on representational similarity (see point 3
   below).

## Weaknesses
1. It is not clear what is meant by the "image reconstruction
   problem", and how can the proposed architecture bring progress on
   that. In the introduction, it is stated that the image
   reconstruction problem is an important one in computer vision, and
   a couple of references are given to papers by J Fessler on medical
   applications. Is there any way in which DAE-NR could contribute to
   this type of applications?
2. In the abstract and the introduction, the "guidance of neural
   information" is offered as a general principle that should somehow
   improve image reconstruction performance. However, in the present
   work, this is only shown to be true for the neural data from one of
   the three regions recorded in  Antonlik et al
   2016 (region 3). Including
   data from the other recorded regions (regions 1 and 2) did not
   improve appreciably the performance of the autoencoder (table 2),
   invalidating the generality of the approach, but no comment or
   justification is proposed for this fact. What makes neurons in
   region 3 special? Is there a theoretical or practical reason why
   inclusion of the data for these neurons, but not the others, should
   help the autoencoder? Or is it simply that these neurons happen to
   carry a bit more information than those in regions 1 and 2 about
   important features of the visual stimulus, and therefore their
   inclusion helps learning better representations in the autoencoder?
3. For neural activity prediction, no comparison is made with the
   state of the art (only comparisons with variants of the proposed
   architecture are done). This is particularly surprising given that
   the neural data used in this paper comes from another work where
   neural prediction was performed. In Figure 5A of Antonlik et al
   2016, the correlation coefficient between model prediction and
   neural activity is shown to be about 0.5 for all three regions. In
   the present work (Table 4), this metric is somewhat lower (a range
   of about 0.3-0.5, considering 8 different models).
4. The analysis of "significant neurons" in table 3 seem
   overinterpreted. If I've understood correctly the procedure for
   this analysis, it is as follows: after training the network,
   neurons (only those for region 3, in the main text) are divided
   among those for which the model prediction are better than chance
   (the "significant" ones), and those for which they aren't. Then the
   network is re-trained, from scratch, using only data from either
   significant or insignificant neurons. It is observed that the
   significant neurons contribute to improving the VAE, while the
   insignificant ones don't. It is claimed that "These experiments
   verify our hypothesis, indicating that information from significant
   neurons can guide CAE-FR to better reconstruct images". I am not
   sure what is added by this analysis: the simplest explanation is
   that the insignificant neurons don't carry any (or carry very
   little) information about the images, so they are effectively
   random noise generators for the purpose of image reconstruction,
   and therefore can't possibly contribute to improving it. In other
   words, in my opinion this analysis is just showing in a different
   way that some neurons carry some information about the images, and
   some don't. In this sense, saying that information from informative
   neurons can contribute to a better reconstruction is somewhat of a
   vacuous claim; for instance, one could do even better by including
   skip connections directly from the image input layer.
5. Additionally, the numbers in the "significant" column in Table 3
   seem uniformly worse than the numbers in Table 2. In other words,
   it seems that training with the significant neurons alone is better
   than training with the insignificant neurons alone, but worse than
   training with all neurons. This seems relevant and should be
   discussed. How do the authors square this fact with the statement
   that "insignificant neurons jeopardize CAE-FR"?
6. Why are the images in Figure 2 so small and featureless? The
   abstract states that the input images are "natural images", and the
   paper says that the data is taken from Antonlik et al 2016. But the
   images that paper (see for instance Figure 1 in Antonlik et al)
   look completely different, and the methods section of that paper
   states "The stimulus set was composed of static scenes from David
   Attenborough’s BBC documentary Life of Mammals, depicting natural
   scenes such as landscapes, animals or humans. Images were scaled to
   have 256 equally spaced luminance steps, and were composed of
   384×208 pixels, and expanded to fill the screen". I must be missing
   something, because the images in Figure 2 look like random blobs to
   me.
clarity,_quality,_novelty_and_reproducibility: The paper is written in a reasonably clear way, and the idea of
regularizing a predictor of neural encoding with a VAE is
interesting. However, the quality of the work is not satisfactory in
my view, because - as detailed above - several of the claims made by
this work are not supported by the evidence. Regarding
reproducibility, the methods are described reasonably well, but I
could not find a code link.
summary_of_the_review: In my opinion, this paper fails to back up some of its core claims and
to justify the (claimed) relevance for the proposed architecture from
a computer vision standpoint. Therefore, my recommendation is not to
accept this work for publication.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper shows that optimizing a CAE model to fit neural data and reconstruct input images at the same time improves the performance on both tasks. There is general consensus that the finding that integrating neural response into deep autoencoder models can improve its performance is interesting. But there is also general consensus (with the exception of one review)  that the significance of the work remains limited as previous work has already shown the benefits of jointly training/regularizing vision tasks with neural data. Some of it is cited (and additional work that could be cited include: https://arxiv.org/abs/2211.07885).

The contribution of this work is really only to show how neural data help an auto-encoder which appears limited in terms of significance.
justification_for_why_not_higher_score: The significance of the work is limited.
justification_for_why_not_lower_score: NA
Decision: Reject