Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper challenges the interpretability of common neural recording analysis methods with deep artificial networks, by simulating such methods on artificial networks themselves. First, it shows that several architectures with totally different components are very similar in commonly used prediction measures when tested on a neural recording benchmark data set. Second, it shows that these common measures can not distinguish the ground truth even when it is an artificial neural network. Importantly, this simulation is done with a sample size of target neurons analogous to neuroscience experiments. Moreover, the authors show high variability in results when the stimulus type (family of the input images) changes. The authors briefly suggested two ways to overcome the interpretability issue, i.e. using more natural stimuli, and recording from more neurons. 
strength_and_weaknesses: Strength: This paper tackles an important problem in computational neuroscience and raises a very concerning issue regarding using deep networks for (neuro)science. It is clearly written and gives a nice overview of the literature on using neural networks for neuroscience. 

Weakness: 
My main concern is that the authors did not really provide a solution (or proof that there is none) for the raised issue. It is also not clear that the issues arise because of the measurements themselves or the authors' specific statistical choices such as reporting the median for correlations. I think a more in-depth analysis is needed to flesh out issues of using artificial architectures in neural recording analysis. For example, I think it is necessary to show how the measures and conclusions change with the increase in neural sample size, especially because increasing neural recording size is suggested as one of the solutions. In addition, as mentioned in the paper there exist several measures in these types of analysis. Given that the experiments are all simulations on very common networks, I think trying other measures such as SVCCA and SVD is necessary (see "SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability" by Raghu et al, NeurIPS 2017 and "insights on representational similarity in neural networks with canonical correlation" by Morcos et al, NeurIPS 2018). 
Finally, recurrent networks should be examined too especially because of feedback connections in the visual cortex. 
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and the architecture parameters are fully reported. While the paper has some novel aspects, this type of work has been already done, and new contributions need more extensive examinations (e.g. see the references above). 


summary_of_the_review: The paper is clear and tackles an interesting problem but needs further experiments given the history of this type of work in the DL community.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work takes a closer look at the prevalent tools in computational cognitive neuroscience today that are used to relate representations of deep learning models to representations in the brain. These tools are encoding models and a form of representational similarity analysis (CKA). The main question in this work goes along one of the several directions in this increasingly popular research area, which is to compare different deep learning architectures in terms of their alignment with brain recordings and conclude that the best performing one must be closer to the actual brain system. This specific work examines whether these comp neuro tools are indeed able to identify the correct target system, in the case that the target system is known. For this purpose, the work investigates the alignment (both in terms of regression scores and representational similarity) among various deep learning models. It reports that even though both encoding models and CKA are able to identify the correct system in most cases, the differences between the correct and incorrect source systems is small. Furthermore, the work also examines the effect of different types of images used for evaluating the alignment and finds that more naturalistic images lead to higher identifiability.
strength_and_weaknesses: Disclaimer: I reviewed a very similar version of this work at NeurIPS 2022. The authors have addressed some of my concerns in the current work but several of my major concerns remain and I will summarize these below, along with some new ones in this version.

Strengths:
- Investigates a problem that is becoming increasingly important as the popularity of the research area of relating representations in machines and brains grows
- The experiment that examines the effect of different types of images used for evaluating the alignment is novel as far as I know, and important for this subfield.

Weaknesses:
Major:
1. The ridge regularization parameter is fixed at 1. This parameter needs to be optimized with nested cross-validation in order to get a fairer estimate of the generalization performance. This is important since one of the points the authors are trying to make is that CKA is (slightly) better than ridge regression at system identification, and also that the difference between the performances for the correct and incorrect systems is small.
2. The takeaways from the work are not clear enough. The results suggest that system identification is possible in most cases with the current approaches in an ideal setting. How generalizable are these results to neural recordings? The authors can do more to investigate this question since it is the main motivation behind the work. In addition, the work tries to interpret the results pessimistically (though this is toned-down a lot from the previous version), but this further muddles the message. It is furthermore not clear whether any significance testing was done for the results (e.g. "However, for some target layers in AlexNet and ResNet18, although the layer with the highest score may be the matching layer in the same architecture, linear regression scores for other source models do not show a significant decrease in predictivity" is this statement based on a significance test?)
3. All of the figures are illegible due to size, small fonts, and blur.

Minor:
- "Following the procedure developed by previous works (Schrimpf et al 2020, Yasmins et al 2014, Conwell et al 2021).." The procedure for training encoding models for predicting brain recordings dates back much further than the cited works. See Kay et al. 2008 Nature (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3556484/), and Mitchell et al. 2008 Science (https://pubmed.ncbi.nlm.nih.gov/18511683/).

Suggestion:
- "One potential interpretation of the result would be that different neural network architectures are equally good (or bad) models of the visual cortex. An alternative explanation would be that the method we use to compare models with the brain has limitations in identifying precise computational operations. To test this hypothesis,.." -- Yet another alternative explanation (which I believe is actually more likely) is that even though different source models explain similar percent variance in the target model, this variance may be different from model to model (especially when comparing different layers of models). It is not uncommon nowadays to do a variance partitioning analysis (both for encoding models and for RSA/CKA) to better understand the contributions to predictive performance. One suggestion is to partition the explained variance in the target system by the different candidate source systems, which may provide more insight into why different source systems achieve good prediction performance for the same target system.
clarity,_quality,_novelty_and_reproducibility: The clarity, quality, and reproducibility are solid. 
The approach is not inherently novel (it is close to Kornblith, Norouzi, Lee & Hinton (2019)), but the significance can be improved by performing additional analyses that better reveal the reasons for the limitations of the current methods.
summary_of_the_review: While this work investigates an important and timely problem, it does not yet go deep enough to warrant a top-tier ML publication in my opinion. There are several good directions that this work has started (e.g. investigate the effect of training data distribution of the source and possibly target domain) and I encourage the authors to further pursue them.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper considers two approaches typically used in comparing deep
nets and brain recordings, linear regression and CKA, and explores
their behavior when used to measure the similarity of different neural
networks. This is used to highlight under which conditions (and in
what sense) these methods allow to identify common patterns between
information processing systems.
strength_and_weaknesses: ## Strengths
1. The idea of this paper will be of interest to the iclr
   community. Regardless of the issues I have with some of the
   conceptual framing of this paper (see below, point 1 in weaknesses), I
   believe it is useful to get a rough sense of what one can expect even in a best case scenario, when using these similarity/alignment metrics.
2. The range of networks analyzed seems appropriate and the results
   are reported in a clear way.
3. The paper conclusions are given in a balanced way, presenting both
   a "glass half full" and a "glass half empty" perspective.

## Weaknesses
1. Several passages in the text seem to imply that comparing deep net
   and brain representations of certain stimuli must be the same as
   testing the hypothesis that information processing in the brain
   follows the same patterns/motifs as the deep net. While this is
   certainly true of some existing works in the literature, it is not
   true of all of them. Indeed, one could appreciate having a good
   model of neural activations even from a purely phenomenological
   perspective. Just to make one example, in the Bashivan paper cited
   among the references, the deep net allows the experimenters to
   design stimuli that can drive target neurons very strongly. This
   would still have value even if the model brought no insight
   whatsoever about information processing in visual cortex, as it
   would enable new kinds of experiments. Now, I understand that the
   present paper focuses on system identification (so exactly the
   "hypothesis testing" scenario), but it would be good to mention
   somewhere that system identification is not the only possible
   motivation behind modeling the ventral stream/brain circuits using
   deep networks.
2. The value of λ (the regularization parameter for the linear
   regression) is fixed at a hand-picked value, rather than selected
   in a systematic way, for instance by cross-validation. Other values
   of λ are explored in the supplement. This seems like a suboptimal
   way of performing a systems identification analysis. The results of
   the paper would be strengthened if λ was not picked by hand.
3. I found it interesting to learn how identification reliability
   increases with increasing population size. The main analyses were
   done with populations of 3000 neurons, an arbitrary number. It is
   possible that using a larger number (still compatible with the
   upper limits of what is possible experimentally) would have yielded
   somewhat different results. This is especially true for some of the
   experiments that are already borderline, like the results in figure
   2, where a lot of emphasis is given to the fact that 2 layers out
   of 5 in one network out of five tested do not have the ground-truth
   model as the best match. In this light, it would be good to warn
   the reader up front, when specifying the size of the populations
   used (page 3, top) that the results are dependent on this choice.
clarity,_quality,_novelty_and_reproducibility: The paper is clear and easy to follow. The research question is
reasonably novel, and I couldn't find any inconsistency or
technical/correctness issue. The experimental procedures are
adequately described for reproducibility.

One minor issue I have found with the presentation quality is that the
labels and text on the plots is way too small. Please make sure that
the text is readable even when the document is visualized at its
"intended" dimension (as if printed on regular paper).

*Typos:*
- page 3, paragraph below equation 2: "principle components" (should
  be "principal").
summary_of_the_review: This paper provides some data about the behavior of common
similarity/alignment measures used to compare deep nets and neural
recordings. Although descriptive in nature, it will provide a useful
reference for a sizeable subcommunity at iclr.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper attempts to answer an interesting question: if we had a NN that matched a biological system, would we be able to tell?  Through a series of well controlled studies, the answer seems to be yes, some of the time, in close-to-ideal scenarios.  

The reviewers pointed out strengths in the paper: the authors agreed that the paper was clear and that the issue handled by the paper was important and timely.  

The reviewers also flagged some weaknesses: the results depend pretty strongly on the design choices the authors made.  Those design choices need to be fully explained, or verified to be ideal empirically. The authors have emabarked on this mission, but the results are not yet in.  The reviewers also wanted the authors to more clearly articulate the "so what" of this paper -- what should we do differently in light of these results.
justification_for_why_not_higher_score: Some of the experiments required to accept this paper are underway - I do think this is an important question and parts of it are answered well, so I could see this being bumped up (though I think that's unlikely given the two 3s)
justification_for_why_not_lower_score: n/a
Decision: Reject