System identification of neural systems: If we got it right, would we know?
Authors: Yena Han
Tomaso Poggio
Brian Cheung
Venue: Submitted to ICLR 2023 (2023)
Keywords: Computational Neuroscience
Neural Networks

Abstract: Various artificial neural networks developed by engineers are now proposed as models of parts of the brain, such as the ventral stream in the primate visual cortex. The network activations are compared to recordings of biological neurons, and good performance in reproducing neural responses is considered to support the model's validity. This system identification approach, however, is only part of the traditional ways to develop and test models in the natural sciences. A key question is how much the ability to predict neural responses tells us. In particular, do these functional tests about neuron activation allow us to distinguish between different model architectures? We benchmark existing techniques to correctly identify a model by replacing brain recordings with known ground truth models. We evaluate the most commonly used identification approaches, such as a linear encoding model and centered kernel alignment. Even in the setting where the correct model is among the candidates, system identification performance is quite variable; it also depends significantly on factors independent of the ground truth architecture, such as stimuli images. In addition, we show the limitations of using functional similarity scores in identifying higher-level architectural motifs.


Under review as a conference paper at ICLR 2023
SYSTEM IDENTIFICATION OF NEURAL SYSTEMS: IF WE GOT IT RIGHT, WOULD WE KNOW?
Anonymous authors Paper under double-blind review
ABSTRACT
Various artificial neural networks developed by engineers now proposed models of parts of the brain, such as the ventral stream in the primate visual cor- tex. The network activations are compared to recordings of biological neurons, and good performance in reproducing neural responses is considered to support the model's validity. This system identification approach, however, is only part of the traditional ways to develop and test models in the natural sciences. A key question is how much the ability to predict neural responses tells In par- ticular, do these functional tests about neuron activation allow us to distinguish between different model architectures? We benchmark existing techniques cor- rectly identify model by replacing brain recordings with known ground truth models. We evaluate the most commonly used identification approaches, such as a linear encoding model and centered kernel alignment. Even in the setting where the correct model is among the candidates, system identification performance is quite variable; it also depends significantly on factors independent of the ground truth architecture, such as stimuli images. In addition, we show the limitations using functional similarity scores in identifying higher-level architectural motifs.
INTRODUCTION
Over the last two decades, the dominant approach for machine learning engineers in search of better performance has been to use standard benchmarks to rank networks from most relevant to least relevant. This practice has driven much of the progress in the machine learning community. A standard comparison benchmark enables the broad validation of successful ideas. Recently such benchmarks have found their way into neuroscience with the advent of experimental frameworks like Brain-Score (Schrimpf et al. 2020) and Algonauts (Cichy et al., 2021). where artificial models compete predict recordings from real neurons in animal brains. Can engineering approaches like this be helpful in the natural sciences?
The answer is clearly yes: the "engineering approach" described above ranks models that predict neural responses better as better models of animal brains. While such rankings may be a good mea- sure of absolute performance in approximating the neural responses, which on its own valuable for various applications (Bashivan et al.) 2019). it is an open question whether they are sufficient. In neuroscience, understanding natural intelligence
the level of the underlying neural circuits requires developing model systems that reproduce the abilities of their biological analogs while re- specting the constraints provided by biology, including anatomy and biophysics (Marr & Poggio) 1976 [Schaeffer et al. 2022). A model that reproduces neural responses well but turns out to re- quire connectivity or biophysical mechanisms that are different from the biological ones is thereby falsified.
Consider the conjecture that the similarity of responses between model units and brain neurons allows us to conclude that brain activity fits better, for instance, a convolutional motif rather than a dense architecture. effectively constrains architecture. Then the need for separate test of the model at the level of anatomy would become, at least in part, less critical for model validation. Therefore, we ask the question: could functional similarity be reliable predictor of architectural similarity?
this were true, it would mean that functional similarity over large data sets

---

Under review as a conference paper at ICLR 2023
We describe attempt to benchmark the most popular similarity techniques by replacing the brain recordings with data generated by various known networks with drastically different architectural motifs, such as convolution attention. Such setting provides valuable upper bound to the identifiability of anatomical differences.
1.1 SYSTEM IDENTIFICATION FROM LEADERBOARDS
When artificial models are compared against common biological benchmarks for predictivity (Ymiins DiCarlo 2016). models with the top score are deemed better models for neuroscience. As improvements to scores are made over time, ideally, more relevant candidates emerge. Neverthe- less, if two artificial models with distinctly different architectures, trained on the same data, happen to be similar in reproducing neural activities (target model), then it would be impossible to conclude what accounts for the similarity. can be biologically relevant motifs from each architecture, the properties of the stimulus input, similarity metric. Such ambiguity is due the many-to-one mapping of a model onto a leaderboard score. Our work shows that multiple factors play a role in representational similarities.
An interesting example is offered by Chang et al. 2021 p. which compares many different models with respect to their ability reproduce neural responses IT to face images and concludes that the 2D morphable model is best. Operations required in the specific model, such as correspondece and vectorization do not have apparent biological implementation in terms of neurons and synapses. Nonetheless, it is perhaps not too surprising that the model can predict IT responses well, as there are multiple confounds besides the biological constraints, which affect the neural predictivity.
2 RELATED WORK
While the analogy between neural network models and the brain has been well validated (Bashivan al. 2019), the extent of this correspondence across multiple levels (Marr & Poggio 1976) has been taken for granted. This assumed correspondence could be attributed to methodological limitations of evaluating such models simultaneously across all levels. Jonas & Kording (2017) investigated the robustness of standard analysis techniques in neuroscience with a microprocessor as ground-truth model to determine the boundaries of what conclusions could be drawn about known system. The presumption of correspondence could also be attributed to underappreciated variability from model hyperparameters (Schaeffer et al. 2022). In a similar spirit to Jonas & Kording (017)); Lazebnik 2002), evaluate system identification on a known ground-truth model to establish the boundaries of what architectural motifs can be reliably uncovered. We perform our analysis under favorable experimental conditions establish an upper bound.
As modern neural network models have grown more prominent in unison with the corresponding resources to train these models, pre-trained reference models have become more widely available in research (Wightman] 2019). Consequently. the need to compare these references along different metrics has followed suit. Kornblith et al. (2019); Morcos al. (2018) explored using different similarity measures between the layers of artificial neural network models. Kornblith et al. (2019) propose various properties similarity measure should be invariant such as orthogonal transforma- tions and isotropic scaling while not invariant to invertible linear transformations. Kornblith et al. (2019) found centered kernel alignment (CKA), a a method very similar Representation Similarity Analysis (Kriegeskorte et al. 2008). to best satisfy these requirements. Ding et al. (2021) explored the sensitivity of methods like canonical correlation analysis, CKA, and orthogonal procrustes dis- tance to changes in factors that do not impact the functional behavior of neural network models.
BACKGROUND AND METHODS
The two predominant approaches to evaluating computational models of the brain are using metrics based on linear encoding analysis for neural predictivity and population-level representation simi- larity. The first measures how well a model can predict the activations of individual units, whereas the second metric measures how correlated the variance of internal representations is. We study the following neural predictivity scores consistent with the typical approaches: Linear Regression and Centered Kernel Alignment (CKA).
2

---

Under review as a conference paper at ICLR 2023
In computational neuroscience, usually have neural system (brain) that are interested in modeling. We call this network a target and the proposed candidate model a source. Formally, for a layer with Pr units in source model, let X E R"xp1 be the matrix of representations with pi features over n stimulus images. Similarly, let Y E Rexp= be a matrix of representations with P2 features of the target model (or layer) on the same stimulus images. Unless otherwise noted, we subsample 3000 target units to test an analogous condition as in biology, where recordings are far from exhaustive. Our analyses are partially dependent upon the target coverage, and we later examine the effect of increasing the number of target units.
3.1 ENCODING MODEL: LINEAR REGRESSION
Following the procedure developed by previous works (Schrimpf ct al. 2020 Yamins et al. 2014
Conwell et al. 202T Kar et al. 2019 Mitchell al. 2008), we linearly project the feature space of a single layer in a source model to map onto a single unit in a target model (a column of Y). The linear regression score is the Pearson's 's correlation r(t, coefficient between the predicted responses of a source model and the ground-truth target responses to a set of stimulus images.
argmin XS 1|BBI
(1)
LR(X, = r(X$3,
(2)
We first extract activations the same set of stimulus images for source and target models. To reduce computational costs without sacrificing predictivity, we apply sparse random projection S E RP: for QI << Pt. on the activations of the source model (Conwell et al. 2021). This projection reduces the dimensionality of the features to qi while still preserving relative distances between points (Li et al., 2006). Unlike principal component analysis, sparse random projection is a dataset- independent dimensionality reduction method. This removes any data-dependent confounds from our processing pipeline for linear regression and isolates dataset dependence into our variables of linear regression and candidate model.
interest:
We apply ridge regression on every layer of source model to predict target unit using these fea- tures. We use nested cross-validations in which the regularization parameter À is chosen in the inner loop and linear model fitted in the outer loop. The list of tested We use 5-fold cross-validation for both inner and outer loops. As there are multiple target units, the median of Pearson's 's correlation coefficients between predicted and true responses is the aggregate score for layer-wise comparison between source and target models. Note that layer of target model is usually assumed to correspond to a visual area, e.g. V1 or IT, in the visual cortex. For layer-mapped model, we report maximum linear regression scores across source layers for target layers.
[0.01, 0.1, 1.0, 10.0, 100].
3.2 CENTERED KERNEL ALIGNMENT
Another widely used type metric builds upon the idea of measuring the representational similarity between the activations of two neural networks for each pair of images. While variants of this metric abound, including RSA or re- weighted RSA (Kriegeskorte et al. 2008 Shtaligh-Razare et al., 2017), we use CKA (Cortes et al. 2012) as (Kornblith al. (2019) showed strong correspondence between layers of models trained with different initializations, which we will further discuss as a validity test perform. We consider linear CKA in this work:
CKA(X,Y) = | X
(3)
(Krnblith et al.] 2019) showed that the variance explained by (unregularized) linear regression accounts for the singular values of the source representation. In contrast, linear CKA depends on the singular values both target and source representations. Recent work (Diedrichsen et al. 2020) notes that linear CKA is equivalent RSA under certain conditions. We also call CKA a neural predictivity score because a target network is observable, whereas source network gives predicted responses.
whitened representational dissimilarity matrix (RDM) in
3

---

Under review as a conference paper at ICLR 2023
Target: Brain
0.8
0.7
Regression
Alech and VGGs
ResNets
Transformers
0.4
Visual Area
Figure Linear regression scores of deep neural networks for brain activations the macaque visual cortex. For V1, the top performing three models are in the VOneNet family (Dapello et al.) 2020), which are explicitly designed to mimic known properties of VI.
3.3 IDENTIRIABILTYY INDEX
To quantify how selective predictivity scores are when source matches the target architecture com- pared when the architecture differs between source and target networks, define an identifiabil- ity index as:
Score (source = target) Mean Score(source 전 target)
Identifiability Index
(4)
Score(source target) Mean Score(source target)
In brief, is a normalized difference between the score for the true positive and the mean score for the true negatives. Previous works (Dobs et al., 2022) Freiwald & Tsao 2010)) defined selectivity
indices the same way in similar contexts, such as the selectivity neuron to specific tasks.
3.4 SIMULATED ENVIRONMENT
target network is a brain, it is essentially a black box, making it challenging to understand the properties limitations of the comparison metries. Therefore, instead use artificial neural networks of our choice as targets for our experiments.
We investigate the reliability of metric compare models, mainly discriminate the underlying computations specified by the model's architecture. We intentionally create favorable conditions for identifiability in a simulated environment where the ground truth model is candidate among the source models. Taking these ideal conditions further, target and source models are deterministic and not include adverse conditions typically encountered in biological recordings, such as noise and temporal processing. We consider the following architectures:
Convolutional Networks: AlexNet (Krizhevsky et al. 2012), VGGII (Simonyan & Zisserman
2014), ResNet18 (He et al. 2016)
Recurrent Networks: CORnet-S Kubilius et al. 2019)
Transformer Networks: ViT-B/32 (Dosovitskiy al. 2020)
Mixer Networks: MI.P-Mixer-B/ Tolsstikhin al. 2021
These architectures are emblematic of the vision-based models used today. Each architecture has a distinct motif, making it unique from other models. For example, transformer networks use the soft-attention operation
core motif, whereas convolutional networks use convolution. Recurrent networks implement feedback connections which may be critical in the visual cortex for object recognition (Kubilius al.) 2019; Kar et al.] 2019). Moreover, mixer networks (Tolstikhin et al.) 2021 Touvron et al.] 2021] Mclas Kyriazi (202)) uniquely perform fully-connected operations over image patches, alternating between the feature and patch dimensions.
4

---

Under review as a conference paper at ICLR 2023
Target: AlexNet
Target: VGG11
Target: ResNet18
1.0
1.0
1.0
Regrecsten Spore
0.6
0.6
0.6
0.4
0.2
0.2
0.2
Target: CORNet-S
Target: VIT-B/32
Target: MLP-Mixer-B/16-
1.0
1.0
1.0
Ropressin Score
Peaton's
0.6
0.6
0.6
p. 0.01 Incorrect
0.0
0.0
10
10
Target Layer Number
Target Layer Number
Targst Layer Number
AlexNet
VGG11
ResRett1
CORNet-S
VIT-8/32
MLP-Mixer-BBII
Figure 2: Linear regression scores for artificial neural networks shown boxplots. We use different initialization seeds for source networks of the same architecture type as the target, except for MLP- Mixer-B/ (marked with an asterisk), for which we test identical weights. Source networks marked with black dots indicate the correct architecture, identical to the target, does not outperform those networks with statistically significant difference (p > 0.01). Source networks marked with red dots indicate that median scores corresponding those networks are higher than the median for the correct architecture. We compare the median as the ranking of source models is typically decided based the aggregate median scores (Schrimpf al. 2020
RESULTS
4.1 DIFFERENT MODELS TRAINED ON A LARGE-SCALE DATASET REACH EQUIVALENT NEURAL PREDICTIVITY
We compare various artificial neural networks with publicly shared neural recordings in primates MMjaj et al. 2015 Freeman et al. 2013) via the Brain-Score framework (Schrimpf et al. 2020). Our experiments show that the differences between markedly different neural network architectures are minimal after training (Figure T). consistent with the previous work (Schrimpf et al. 2020 Kubilius ct al. 2019 Conwell et al.] 202T). Previous works focused on the relative ranking of models and investigated which model yields the highest score. However, if take closer look the result, the performance difference is minimal, with the range of scores having a standard deviation < 0.03 (for V2=0.021, V4=0.023, IT=0.016) except for V1. For V1, VOneNets (Dapello al.) 2020). which explicitly build in properties observed from experimental works in neuroscience, significantly outperform other models. Notably, the models we consider have quite different architectures based on combinations of various components, such as convolutional layers, attention layers, and skip connections. This suggests that architectures with different computational operations reach almost equivalent performance after training on the same large-scale dataset, i.e., ImageNet.
4.2 IDENTIFICATION OF ARCHITECTRRES IN AN IDEAL SETTING
One potential interpretation of the result would be that different neural network architectures are equally good (or bad) models of the visual cortex. An alternative explanation would be that the method we use to compare models with the brain has limitations identifying precise computational operations. To test the hypothesis, we focus where underlying target neural networks are known instead of being a black box, as with biological brains. Specifically, we replace the target neural recordings with artificial neural network activations. By examining whether the candidate source model with the highest predictivity is identical to the target model, we can evaluate to what extent identify architectures with the current approach.
5

---

increase the coverage for future Under review as a conference paper at ICLR 2023
Target: AlexNet
Target: VGG11
Target: ResNet18
1.0
1.0
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.0
0.0
6
2
6
Target: CORNet-S
Target: VIT-B/32
Target: MLP-Mixer-B/16*
CKA
0.4
0.2
0.2
0.0
0.0
2
4
10
2
8
10
4
8
12
16
20
Target Layer Number
Target Layer Number
Target Layer Number
AlexNet
VGG11
ResNet18
CORNet-S
VIT-8/32
MLP-MIxer-8/16
Figure 3: CKA for different source and target networks. The experimental setup is identical to Figure 2 besides using CKA instead of linear regression the metric. As in Figure 2 when we test MLP-Mixer-Bie B/16 as target and the source network type matches the target, weights are identical. We show the results for MLP-Mixer-B/ bar plots with a pattern indicate the difference from other targets.
4.2. LINEAR REGRESSION
We first compare various source models (AlexNet, VGGII, ResNet18, CORnet-S, ViT-B/32, MLP- Mixer-B/16) with a target network, the same architecture as one of the source models and trained on the same dataset but initialized with different seed. We test dataset composed of 3200 images of synthetic objects studied in (Majaj et al.] 2013) to be consistent with the evaluation pipeline of Brain-Score. The ground-truth source model will yield higher score than other models if the model comparison pipeline reliable. For most target layers, source networks with the highest median score the correct network (Figure 2. However, strikingly, for several layers in VGG11, 1,, ResNet18, and CORnet-S the best-matched layers belong to source model that is not the correct architecture. In other words, given the activations of VGGII, for instance, and based linear regression scores, we would make an incorrect prediction that the system's 's underlying architecture closest to a ResNet18.
In addition, because of our ideal setting, where an identical network is one of the source models, we expect to see significant difference between matching and non-matching models. However, for some target layers in VGG11 and ResNet18, 8, linear regression scores for the non-identical ar- chitectures, when compared with those for the identical one, do show a significant decrease in predictivity based on Welch's is t-test with 0.01 applied as threshold (Figure This result sug- gests that the identification of the underlying architectures of unknown neural systems far from perfect.
4.2.2 CKA
Next, examine another widely used metric, CKA, for comparing representations. Again, compare different source models to a target model, also an artificial neural network. For the target models tested, the ground-truth source models achieved the highest score (Figure 3. Still, some unmatched source networks lead to scores close to the matched networks, even for the target MLP- Mixer-B/16, where the source network of the same architecture type also has identical weights.
When applying CKA to compare representations, we subsample a set (3000) of target units to mimic the limited coverage of single-unit recordings. Assuming

---

Under review as a conference paper at ICLR 2023
ImageNet
Synthetic Object
Target AlexNet
Target: VGG11
Target: ResNet18
Identifaban Micniffbblltty Index
Imageliet
Synthelic Chjjects
Texture-Mi Noise
CKA
0.1
0.1
0.1
0.0
0.0
0.0
Target: CORNet-S
Target: VIT-B/32
Target: MLP-Mixer-B/16
idonerfabillty Indox
0.3
0.5
CGA
0.4
0.1
0.1
0.0
0.0
Target Layer Number
Target Layer Number
Target Layer Number
(a) CKA Identifiability Index
Target:
Target: VGG11
Target: ResNet18
0.35
0.15
0.15
dcntifiabllity Index
0.30
0.30
Regression
Imagebatt
Synthetic
-0.10
Texture-Nois
-0.10
-0.10
Target: CORNet-S
Target: VIT-8/32
Target: MLP-Mixer-B/16
0.35
0.15
inentifabilty Innex
0.32
Regression
0:30
0.40
0.08
0.20
0.0
0.00
10
10
15
20
Target Layer Number
Targel Layer Number
Target Layer Number
(b) Regression Identifiability Index
Figure 4: Top Sample images of each stimulus image type. (a) Identifiability index using CKA and (b) linear regression for different types of stimulus images and target networks.
experiments with more advanced measurement techniques, we test whether the identifiability im- proves if include all target units. Additionally, methods similar CKA, such as RSA, are often applied to interpret neural recordings, including fMRI, MEG, and EEG (Cichy & Oliva 2020 Cichy & Pantazis 2017). which can have full coverage of the entire brain or a specific region of interest. Therefore, we simulate such analyses by having all units in the targets. Overall, the ground-truth source models outperform the other source models with significant margin (Figure More exper- imental results are in the Appendix). This suggests system identification can be more reliable with more recording units and complete target coverage.
4.3 EFFECTS OF THE STIMULUS DISTRIBUTION ON IDENTIFIABILITY
A potentially significant variable overlooked in comparing computational models of the brain the type of stimulus images. What types of stimulus images are suited for evaluating competing models?

---

Under review as a conference paper at ICLR 2023
Target: VGG13, 3000 Units/Layer
Target: Restiet34, 3000 unitsllayer
Target: VIT-U16, 3000 Unitss/
1.0
Mocel Tamily
CNN
95 0.03
0.6
0.0
Target Layer Number
Target Layer Number
Target Layer Number
Target VGG13, 3000 Unitalayeer
Target: ReshNet34, 3000 unet
Target: VIT-/16,, 3000 Unitsllayer
3.0
0.4
0.6
0.2
5.0
10 12
14
12
16
23
Target Layer Number
Target Layer Number
Target Layer Number
Figure 5: CNNs and ViTs of different architectural variants are compared with two CNNs and ViT target networks. Each data point is the maximum score of an architecture for corresponding target layers. Markers with darker shades indicate the mean score of the corresponding model class, and error bars are standard deviations. Frowning faces indicate that corresponding layers do not show a statistically significant difference between model classes.
In Brain-Score, stimulus images for comparing models of the high-level visual areas, V4 and IT, are images of synthetic objects (Majaj al. 2015). In contrast, those for the lower visual areas, VI and V2, are images of texture and noise (Freeman et al. 2013). To examine the effect using different stimulus images, test images of synthetic objects (3200 images), texture and noise (135 images), and ImageNet (3000 images), which are more natural images than the first two datasets.
In Figure 4 analyze Identifiability Index for different stimulus images. More realistic stimulus images (i.e., synthetic objects and ImageNet) show higher identifiability than texture and noise im- ages for all target models. For CKA, observe identifiability increases with layer depth. Notably, even for early layers in target models, which would correspond to VI and V2 in the visual cortex, texture and noise images fail to give higher identifiability. Also, between images of synthetic objects and ImageNet, ImageNet shows higher identifiability. As target models are trained on ImageNet, our results suggest that using stimuli images closer to the images that targets see will help identify architectures better.
It is important to note that the images of texture and noise we use in the experiment help character- ize certain aspects of V1 and V2 in the previous work (Freeman et al. 2013). More specifically, the original work investigated the functional role of V2 in comparison with Vi by showing that natu- ralistic structure modulates V2. Although the image set plays influential variable in carefully designed experiment for more targeted hypothesis, does not translate as a sufficient test set for any hypothesis, such evaluating different neural networks.
4.4 CHALLENGES OF IDENTIFYING KEY ARCHITECTURAL MOTIFS
Interesting hypotheses for more biologically plausible design principle of brain-like models often involve key high-level architectural motifs. For instance, potential questions are whether recurrent connections are crucial in visual processing or, with the recent success of transformer models in deep learning, whether the brain similarly implements computations like attention layers in transformers. The details beyond the key motif, such as the number of layers or exact type of activation functions, may vary and be underdetermined within the scope of such research questions. Likewise, it is unlikely that candidate models proposed by scientists align with the brain at every level, from low- level specifics to high-level computation. Therefore, an ideal methodology for comparing models should help separate the key properties of interest while being invariant other confounds.
8

---

Under review as a conference paper at ICLR 2023
Considering it is a timely question, with the increased interest in transformers models of the brain in different domains (Schrimpf et al. 2021; Berrios & Deza 2022 Whittington et al., 2021). we focus on the problem of identifying convolution
attention. We test 12 Convolutional Networks and 14 Vision Transformers of different architectures, and to maximize identifiability, we use Ima- geNet stimulus images. Note that an identical architecture with the target network is not included as a source network. Figure 5 shows that for most layers, mean CKA and regression scores are higher when there are correspondences between the target and source model classes than when there are not. However, three layers in ViT-L/16 (layers 9, 15, 21) show a higher mean for the CNN model family than for the VÝT model family. Moreover, the architecture with the highest CKA score belongs to the CNN model family for three layers (layers 9-15).
Overall, we observe high inter-class variance for many target layers for both methods. For CKA, one layer in VGGI3, 7 layers in ResNet34, and 7 layers in ViT-L/16, and for regression, three layers in VGG13, layers ResNet34, and one layer in ViT-L/16 do not show a statistically significant difference between the two model classes based Welch's t-test with p 0.01 used as thresh- old. The significant variance among source models suggests that model class identification can be incorrect depending the precise variation choose, especially if we rely limited set of models.
5 DISCUSSION
Under idealized settings, we tested the identifiability of various artificial neural networks with dif- fering architectures. We present two contrasting interpretations of model identifiability based on our results, one optimistic (Glass half full) and one pessimistic (Glass half empty).
Glass half full: Despite the many factors that can lead to variable scores, linear regression and CKA give reasonable identification capability under unrealistically ideal conditions. Across all the architectures tested, identifiability improves as a function of depth.
Glass half empty: However, system identification is highly variable and dependent on the properties of the target architecture and the stimulus data used to probe the candidate models. For architecture- wide motifs, like convolution vs. attention, scores overlap significantly across almost all layers. This indicates that such distinct motifs do not play significant role in the score.
Our results suggest two future directions for improving system identification with current ap- proaches: 1) Using stimuli images that are more natural, i.e., closer to the inputs to the target network (brain) in a natural setting. 2) With more neurons recorded in the brain, neural predictivity scores can be more reliable in finding the underlying architecture.
On the other hand, it is worthwhile to note that we may have reached close to the ceiling using neural predictivity scores for system identification. As example, when our source network is AlexNet, its regression scores against the brain (Figure [ are on par with, or slightly higher than, the scores against another AlexNet (Figure 2). In other words, based on the current methods, AlexNet predicts the brain as well as, if not better than, predicting itself. This observation is not limited to AlexNet but applies other target networks. This fundamental limitation of present evaluation techniques, such as the linear encoding analysis used in isolation, emphasizes the need to develop new approaches beyond comparing functional similarities.
As we argued earlier, ranking models in terms of their agreement with neural recordings is the first step in verifying or falsifying neuroscience model. Since several different models are very close in ranking, the next step architectural validation - is the key. Furthermore, it may have to be done independently of functional validation and with little guidance from it, using standard experimental tools in neuroscience, A parallel direction is, however, to try to develop specially designed, critical stimuli to distinguish between different architectures instead of measuring the overall fit to data. As simple example, it may be possible to discriminate between dense and local (e.g.. CNN) network architectures by measuring the presence or absence of interactions between parts of a visual stimulus that are spatially separated.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Pouya Bashivan, Kohitij Kar, and James J DiCarlo. Neural population control via deep image synthesis. Science, 36+(6433))eaav44336 2019.
William Berrios and Arturo Deza. Joint rotational invariance and adversarial training of dual-stream transformer yields state the art brain-score for area v4. arXiv preprint arXiv:2203.06649, 2022.
Le Chang, Bernhard Egger, Thomas Vetter, and Doris Y. Tsao. Explaining face representation in the primate brain using different computational models. Current Biology, 31(13):2755-2795.c 2021. ISSN 0960-9822. doi: http:///bbiorrg/o.100ff.....bbb 2022220111..411000 04.014, URL sciencedirect. http://ccw.c.c....cc.c..ocooo///
/www.
Radoslaw M Cichy and Aude Oliva. Am/eeg-fmri fusion primer: resolving human brain responses in space and time. Neuron, 107(5):72--71, 2020.
Radoslaw Martin Cichy and Dimitrios Pantazis. Multivariate pattern analysis of meg and eeg: A comparison of representational structure in time and space. Neurofmage, 158:441-454, 2017.
Radoslaw Martin Cichy, Kshitij Dwivedi, Benjamin Lahner, Alex Lascelles, Polina lamshchinina, M Graumann, A Andonian, NAR Murty, K Kay, Gemma Roig, al. The algonauts project 2021 challenge: How the human brain makes sense of world in motion. arXiv preprint arXiv:2104.13: 137144 2021.
Colin Conwell, David Mayo, Andrei Barbu, Michael Buice, George Alvarez, and Boris Katz. Neu- ral regression, representational similarity, model zoology & neural taskonomy at scale in rodent visual cortex. Advances in Neural Information Processing Systems, 34, 2021.
Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. Algorithms for learning kernels based centered alignment. The Journal of Machine Learning Research, 13:795-828 828, 2012.
Joel Dapello, Tiago Marques, Martin Schrimpf, Franziska Geiger, David Cox, and James DiCarlo. Simulating primary visual cortex at the front of cnns improves robustness to image perturbations.
Advances in Neural Information Processing Systems, 33: 13073-130877 2020.
Jörn Diedrichsen, Eva Berlot, Maricke Mur, Heiko H Schütt, Mahdiyar Shahbazi, and Nikolaus Kriegeskorte. Comparing representational geometries using whitened unbiased-diisance-matii similarity. arXiv preprint arXiv:2007.02789) 2020.
Frances Ding, Jean-Stanislas Denain, and Jacob Steinhardt. Grounding representation similarity with statistical testing, arXiv preprint 2Xiv:2008.0166/ 2021.
Katharina Dobs, Julio Martinez, Alexander JE Kell, and Nancy Kanwisher. Brain-like func- tional specialization emerges spontaneously in deep neural networks. Science advances, 8(11): cab18913, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.1 19299, 2020.
Jeremy Freeman, Corey M Ziemba, David Heeger, Eero P Simoncelli, and Anthony Movshon. A functional and perceptual signature of the second visual area in primates. Nature neuroscience, 16(7):974-981. 2013.
Winrich A Freiwald and Doris Y Tsao. Functional compartmenntalizattoo and viewpoint generaliza- tion within the macaque facc-psrcs system. Science, 330(6005):845-851) -511 2010.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
10

---

Under review as a conference paper at ICLR 2023
Eric Jonas and Konrad Paul Kording. Could neuroscientist understand microprocessor? PLoS computational biology, 13(1)=e0052268, 2017.
Kohitij Kar, Jonas Kubilius, Kailyn Schmidt, Elias B Issa, and James J DiCarlo. Evidence that recurrent circuits are critical to the ventral stream's 's execution of core object recognition behavior. Nature neuroscience, 22(6))974-995 983, 2019.
Seyed-Mahdi Saaligh-Razavi Linda Henriksson, Kendrick Kay, and Nikolaus Kriegeskorte. Fixed versus mixed Explaining visual representations by fixed and mixed feature from shallow and deep computational models. Journal of Mathematical Psychology, 76:184-191 2017.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pp. 3519- 3529. PMLR, 2019.
Nikolaus Kriegeskorte, Marieke Mur, and Peter A Bandettini. Representational similarity analysis- connecting the branches of systems neuroscience. Fronfiers in systems neuroscience, 2:4, 2008.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, 25, 2012.
Jonas Kubilius, Martin Schrimpf, Kohitij Kar, Rishi Rajalingham, Ha Hong. Najib Majaj. Elias Issa, Pouya Bashivan, Jonathan Prescott-Roy, Roy, Kailyn Schmidt, et al. Brain-like object recognition with nigh-performing shallow recurrent anns. Advances in neural information processing systems, 32, 2019.
Yuri Lazebnik. Can a biologist fix a radio?-or, what learned while studying apoptosis. Cancer cell. 2(3):199-182, 2002.
Ping Li, Trevor J Hastie, and Kenneth W Church. Very sparse random projections. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 287-296, 2006.
Najib Majaj, Ha Hong, Ethan Solomon, and James J DiCarlo. Simple learned weighted sums of inferior temporal neuronal firing rates accurately predict human core object recognition perfor- mance. Journal of Neuroscience, 35(39):13402-1344 2015.
David Marr and Tomaso Poggio. From understanding computation to understanding neural circuitry. 1976.
Luke Meras-Kyriazi. Do you even need attention? stack of feed-forward layers does surprisingly well on imagenet. arXiv preprint 1iXXi::2005.0222:: 2021.
Tom M Mitchell, Svetlana V Shinkareva, Andrew Carlson, Kai-Min Chang, Vicente L Malave, Robert A Mason, and Marcel Adam Just. Predicting human brain activity associated with the meanings of nouns. science, 320(5880): (1::800):119111 1191-1195, 2008.
Ari Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canonical correlation. Advances in Neural Information Processing Systems, 31, 2018.
Rylan Schaeffer, Mikail Khona, and lla Fiete. No free lunch from deep learning in neuroscience: A case study through models of the entorhinal-hippooaapaal circuit. bioRxiv, 2022.
Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J Majaj. Rishi Rajalingham, Elias B Issa, Ko- hitij Kar, Pouya Bashivan, Jonathan Prescott-Roy, Royy Franziska Geiger, et al. Brain-score: Which artificial neural network for object recognition is most brain-liked BioRxiv, pp. 407007, 2020.
Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A Hosseini, Nancy Kan- wisher, Joshua B Tenenbaum, and Evelina Fedorenko. The neural architecture of language: In- tegrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences, 118(45), 2021.
11

---

Under review as a conference paper at ICLR 2023
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 2014.
Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mip architecture for vision. Advances in Neural Information Processing Systems, 34, 2021.
Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard
Grave, Gautier Izacard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, al. Resmlp: Feedforward networks for image classification with data-efficient training. arXiv preprint arXiv:2105.03404, 2021.
James CR Whittington, Joseph Warren, and Timothy EJ Behrens. Relating transformers to models and neural representations of the hippocampal formation. arXiv prpp 2021.
Ross Wightman. Pytorch image models.
attps:///gtthub..omm/iii...
pytorch- ima-e---e----- models Is 2019.
Daniel LK Yamins and James J DiCarlo. Using goal-driven deep learning models to understand sensory cortex. Nature neuroscience, 19(3)3356-365, 2016.
Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan Solomon, Darren Seibert, and James
DiCarlo. Performance-opiien optimized hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences, 111(23):811--6624, 2014.
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 558-567, 2021.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 MODEL DETAILS FOR SECTION 4.1: BRAIN-SCORE
Below the full list of models tested on the benchmarks of Brain-Score as reported in Section 4.1. In addition to testing vision models pre-trained on ImageNet available from PyTorch's torchvision model package version 0.12, we test VOneNets that are pre-trained on ImageNet and made publicly available by the authors (Dapello et 2020). VOneNets are also family of CNNs.
Convolutional Networks: AlexNet, VGG11, VGGI3, VGG19, ResNet18, ResNet34, ResNet50, ResNer101, VOneAlexNer, VOneResNet50, VOnetCORnet-S
Transformer Networks: ViT-B/16, ViT-B/32, ViT-I/16,, ViT-L/32
A.2 MODEL DETAILS FOR SECTION 4.5: FINDING THE KEY ARCHITECTURAL MOTIF
For each target network reported in Section 4.5, namely VGG13, ResNet34, and ViT-L/16, below the full list of source models tested to compare two model classes, CNN and transformer. For Tokens-to-token VITs (T2T) (Yuan al., 2021), we use models released by the authors. We use Twins Vision Transformers and a Visformer from timm library (Wightman 2019). All other models available from PyTorch's torchvision model package version 0.12. All models are pre-trained on ImageNet.
Convolutional Networks: AlexNet, VGGIL, VGGI3, VGGI6. VGGI3.bn, ResNet18, ResNet34, ResNet50, Wide-ResNet50 2, SqueezeNetl 0, Densenet121, MobileNet v2
Transfomer Networks: ViT-B/16, ViT-B/32, ViT-L/16, ViT-L/32. T2T-ViT-t-14, T2T-ViT 19, TiT-V T2T-ViT-10, Swin-B, Swin-S, Swin-T, Twins-PCPPT--mmlll, Twins-SVT-Small, Visformer-Small
13

---

Under review as a conference paper at ICLR 2023
A.3 MODEL DETAILS: NUMBER LAYERS INCLUDED FOR EACH MODEL
Table
Model
Number of Layers
AlexNet
10
CORnet-S
12
Densenet121
30
MIP--MixerBB1--224
24
Mobilenet.v2
14
ResNet18
10
ResNet34
18
ResNet50
18
Squeezenet1_0
13
Swin-B
24
Swin-S
24
Swin-T
12
ViT-V
13
T2T-ViT-7
10
T2T-ViT_t-14
17
T2T-ViT--19
22
Twins-PCPVT-Small
16
Twins-SVT-Small
18
VGG11
10
VGG13
VGG13-BN
12
VGG16
Visformer-Small
16
ViT-B-16
12
ViT-B-32
12
ViT-L-16
24
ViT-L-32
24
Wide-ResNet502
18
14

---

Under review as a conference paper at ICLR 2023
A.4 CKA RESULTS WHEN ALL TARGET UNITS ARE TESTED
... -e--EE 40086
.. fgr p4gga
16-08
C 1p
4
Do
(a) CKA scores for every pair of layers
204
Lugan
Lugen
Layon
Laye
Alackt
Karverca
VI-401
(b) CKA scores for top per-perfognnng layer in cach source network
Traalt 4
Temeri Wist1
Terpeh
Target inp Mar Ku18
Tadura
6.21
Leen
Identifiability index for different types of stimuli images
Thrget VDG13, AllUmts
Target
Target VITL26 AlUNIs
Made:
Harbs
Torae Hamaer
75
(d) Comparing CKA scores for the model families CNN and VIT
Figure 6: We include all target units when computing CKA. Experimental setups identical to Figures and 5 otherwise. See Section 4.2.2 for discussion. For the target MLP-Mixer-B/ in (b), the source MLP-Mixer-B/ie has identical weights with the target; thus CKA is trivially 1.
15


Reviews:
[{"review_id": "sQdnJa9lY9", "reviewer": "Reviewer_RWEG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper challenges the interpretability of common neural recording analysis methods with deep artificial networks, by simulating such methods on artificial networks themselves. First, it shows that several architectures with totally different components are very similar in commonly used prediction measures when tested on a neural recording benchmark data set. Second, it shows that these common measures can not distinguish the ground truth even when it is an artificial neural network. Importantly, this simulation is done with a sample size of target neurons analogous to neuroscience experiments. Moreover, the authors show high variability in results when the stimulus type (family of the input images) changes. The authors briefly suggested two ways to overcome the interpretability issue, i.e. using more natural stimuli, and recording from more neurons. ", "summary_of_the_review": "The paper is clear and tackles an interesting problem but needs further experiments given the history of this type of work in the DL community. ", "strength_and_weaknesses": "Strength: This paper tackles an important problem in computational neuroscience and raises a very concerning issue regarding using deep networks for (neuro)science. It is clearly written and gives a nice overview of the literature on using neural networks for neuroscience. \n\nWeakness: \nMy main concern is that the authors did not really provide a solution (or proof that there is none) for the raised issue. It is also not clear that the issues arise because of the measurements themselves or the authors' specific statistical choices such as reporting the median for correlations. I think a more in-depth analysis is needed to flesh out issues of using artificial architectures in neural recording analysis. For example, I think it is necessary to show how the measures and conclusions change with the increase in neural sample size, especially because increasing neural recording size is suggested as one of the solutions. In addition, as mentioned in the paper there exist several measures in these types of analysis. Given that the experiments are all simulations on very common networks, I think trying other measures such as SVCCA and SVD is necessary (see \"SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability\" by Raghu et al, NeurIPS 2017 and \"insights on representational similarity in neural networks with canonical correlation\" by Morcos et al, NeurIPS 2018). \nFinally, recurrent networks should be examined too especially because of feedback connections in the visual cortex. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "eXrs14AxUD", "reviewer": "Reviewer_eR5e", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers two approaches typically used in comparing deep\nnets and brain recordings, linear regression and CKA, and explores\ntheir behavior when used to measure the similarity of different neural\nnetworks. This is used to highlight under which conditions (and in\nwhat sense) these methods allow to identify common patterns between\ninformation processing systems.", "summary_of_the_review": "This paper provides some data about the behavior of common\nsimilarity/alignment measures used to compare deep nets and neural\nrecordings. Although descriptive in nature, it will provide a useful\nreference for a sizeable subcommunity at iclr.", "strength_and_weaknesses": "## Strengths\n1. The idea of this paper will be of interest to the iclr\n   community. Regardless of the issues I have with some of the\n   conceptual framing of this paper (see below, point 1 in weaknesses), I\n   believe it is useful to get a rough sense of what one can expect even in a best case scenario, when using these similarity/alignment metrics.\n2. The range of networks analyzed seems appropriate and the results\n   are reported in a clear way.\n3. The paper conclusions are given in a balanced way, presenting both\n   a \"glass half full\" and a \"glass half empty\" perspective.\n\n## Weaknesses\n1. Several passages in the text seem to imply that comparing deep net\n   and brain representations of certain stimuli must be the same as\n   testing the hypothesis that information processing in the brain\n   follows the same patterns/motifs as the deep net. While this is\n   certainly true of some existing works in the literature, it is not\n   true of all of them. Indeed, one could appreciate having a good\n   model of neural activations even from a purely phenomenological\n   perspective. Just to make one example, in the Bashivan paper cited\n   among the references, the deep net allows the experimenters to\n   design stimuli that can drive target neurons very strongly. This\n   would still have value even if the model brought no insight\n   whatsoever about information processing in visual cortex, as it\n   would enable new kinds of experiments. Now, I understand that the\n   present paper focuses on system identification (so exactly the\n   \"hypothesis testing\" scenario), but it would be good to mention\n   somewhere that system identification is not the only possible\n   motivation behind modeling the ventral stream/brain circuits using\n   deep networks.\n2. The value of λ (the regularization parameter for the linear\n   regression) is fixed at a hand-picked value, rather than selected\n   in a systematic way, for instance by cross-validation. Other values\n   of λ are explored in the supplement. This seems like a suboptimal\n   way of performing a systems identification analysis. The results of\n   the paper would be strengthened if λ was not picked by hand.\n3. I found it interesting to learn how identification reliability\n   increases with increasing population size. The main analyses were\n   done with populations of 3000 neurons, an arbitrary number. It is\n   possible that using a larger number (still compatible with the\n   upper limits of what is possible experimentally) would have yielded\n   somewhat different results. This is especially true for some of the\n   experiments that are already borderline, like the results in figure\n   2, where a lot of emphasis is given to the fact that 2 layers out\n   of 5 in one network out of five tested do not have the ground-truth\n   model as the best match. In this light, it would be good to warn\n   the reader up front, when specifying the size of the populations\n   used (page 3, top) that the results are dependent on this choice.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8GnyzTVx-7J", "reviewer": "Reviewer_ngXd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work takes a closer look at the prevalent tools in computational cognitive neuroscience today that are used to relate representations of deep learning models to representations in the brain. These tools are encoding models and a form of representational similarity analysis (CKA). The main question in this work goes along one of the several directions in this increasingly popular research area, which is to compare different deep learning architectures in terms of their alignment with brain recordings and conclude that the best performing one must be closer to the actual brain system. This specific work examines whether these comp neuro tools are indeed able to identify the correct target system, in the case that the target system is known. For this purpose, the work investigates the alignment (both in terms of regression scores and representational similarity) among various deep learning models. It reports that even though both encoding models and CKA are able to identify the correct system in most cases, the differences between the correct and incorrect source systems is small. Furthermore, the work also examines the effect of different types of images used for evaluating the alignment and finds that more naturalistic images lead to higher identifiability.", "summary_of_the_review": "While this work investigates an important and timely problem, it does not yet go deep enough to warrant a top-tier ML publication in my opinion. There are several good directions that this work has started (e.g. investigate the effect of training data distribution of the source and possibly target domain) and I encourage the authors to further pursue them.", "strength_and_weaknesses": "Disclaimer: I reviewed a very similar version of this work at NeurIPS 2022. The authors have addressed some of my concerns in the current work but several of my major concerns remain and I will summarize these below, along with some new ones in this version.\n\nStrengths:\n- Investigates a problem that is becoming increasingly important as the popularity of the research area of relating representations in machines and brains grows\n- The experiment that examines the effect of different types of images used for evaluating the alignment is novel as far as I know, and important for this subfield.\n\nWeaknesses:\nMajor:\n1. The ridge regularization parameter is fixed at 1. This parameter needs to be optimized with nested cross-validation in order to get a fairer estimate of the generalization performance. This is important since one of the points the authors are trying to make is that CKA is (slightly) better than ridge regression at system identification, and also that the difference between the performances for the correct and incorrect systems is small.\n2. The takeaways from the work are not clear enough. The results suggest that system identification is possible in most cases with the current approaches in an ideal setting. How generalizable are these results to neural recordings? The authors can do more to investigate this question since it is the main motivation behind the work. In addition, the work tries to interpret the results pessimistically (though this is toned-down a lot from the previous version), but this further muddles the message. It is furthermore not clear whether any significance testing was done for the results (e.g. \"However, for some target layers in AlexNet and ResNet18, although the layer with the highest score may be the matching layer in the same architecture, linear regression scores for other source models do not show a significant decrease in predictivity\" is this statement based on a significance test?)\n3. All of the figures are illegible due to size, small fonts, and blur.\n\nMinor:\n- \"Following the procedure developed by previous works (Schrimpf et al 2020, Yasmins et al 2014, Conwell et al 2021)..\" The procedure for training encoding models for predicting brain recordings dates back much further than the cited works. See Kay et al. 2008 Nature (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3556484/), and Mitchell et al. 2008 Science (https://pubmed.ncbi.nlm.nih.gov/18511683/).\n\nSuggestion:\n- \"One potential interpretation of the result would be that different neural network architectures are equally good (or bad) models of the visual cortex. An alternative explanation would be that the method we use to compare models with the brain has limitations in identifying precise computational operations. To test this hypothesis,..\" -- Yet another alternative explanation (which I believe is actually more likely) is that even though different source models explain similar percent variance in the target model, this variance may be different from model to model (especially when comparing different layers of models). It is not uncommon nowadays to do a variance partitioning analysis (both for encoding models and for RSA/CKA) to better understand the contributions to predictive performance. One suggestion is to partition the explained variance in the target system by the different candidate source systems, which may provide more insight into why different source systems achieve good prediction performance for the same target system.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject