Position: Maximizing Neural Regression Scores May Not Identify Good Models of the Brain

Authors: rylan  schaeffer
mikail  khona
sarthak  chandra
mitchell  ostrow
brando  miranda
sanmi  koyejo
Venue: NeurIPS_cc_2024_Workshop_UniReps
Published: 2024-09-23 14:17:57

Abstract: A prominent methodology in computational neuroscience posits that the brain can be understood by identifying which artificial neural network models most accurately predict biological neural activations, measured according to regression test error or other similar metrics. In this opinion piece, we argue that the field lacks a canonical definition of model goodness, and rather than engaging with this difficult question, the neural regressions methodology simply asserted a proxy -- neural predictivity -- then overfit to this proxy. We begin with a notable failure of the neural regressions methodology in which the most predictive models disagreed with key properties of the neural circuit. Next, we highlight converging empirical and mathematical evidence that explains the disconnect: (linear) neural regressions are simply discovering the implicit biases of (linear) regression, which may not appropriately identify models that are actually brain-like. This is an instance of Goodhart's law: by selecting neural network models that optimize (linear) neural predictivity, the field's results have devolved into re-discovering general properties of (linear) regression, rather than furthering our understanding of the brain. These insights suggest that the neural regressions methodology may be insufficient for understanding the brain, and we call for a critical reevaluation of this methodology in computational neuroscience.

Main paper: 
 introduction: 
 An influential methodology in neuroscience-inspired artificial intelligence argues that task-optimized deep artificial neural networks (ANNs) should be considered good models of the brain if they capture a large fraction of variance in neural population recordings assessed via regressions of ANN unit activity onto biological neural responses [89] . The claim is that the ANN(s) with better performing neural regressions are more similar to the brain than alternative models [71] . This approach has been widely used in vision [90, 24, 43, 50, 72, 92, 40, 88, 79, 64, 18, 45] , audition [46, 82, 55, 80] , language [62, 39, 73, 3, 61, 15, 16, 33, 5, 2, 38, 58, 17, 44, 4, 81, 56, 37] , and spatial navigation [57] , most often with (regularized) linear models, but occasionally with non-linear models.In this position piece, we argue that Neuro-AI lacks sufficiently rich definitions of neural similarity, and such notions are context-dependent and difficult to quantify. The neural regressions methodology sidesteps these challenges by defining a proxy -for instance, the test R 2 of linear regression between biological recordings and model activations -and then choosing models based on this proxy. The The neural regressions methodology posits a proxy -neural predictivity -of how similar model(s) are to a neural system of interest without verifying the extent to which the proxy agrees with neural similarity. Right: Overfitting to the proxy leads to mismatches with neural similarity. Although we do not define neural similarity here, we emphasize that it is task, neural-system, and question-dependent, and hence likely cannot always be neural predictivity. For a system-specific example of neural similarity, we offer criteria for grid cells in Appendix Sec. A. models that win a selection process (e.g., on BrainScore [71] ) may do so more because of inductive biases of the proxy, independent of any meaningful relationships with the brain (Fig. 1 ). This perspective explains why, for example, the neural regressions methodology was confidently incorrect when applied to models of grid cells: linear regression does not capture in key criteria of neural similarity for grid cells (e.g., periodic tuning curves [34] , multiple grid modules with specific period ratios [77] , toroidal continuous attractor dynamics [91, 32] ; see Appendix Sec. A for a detailed list). This perspective also explains a finding by four independent research groups in different modalities, data, architectures and recording technologies [66, 25, 80, 17] of a quantitatively consistent relationship between test R 2 and effective dimensionality, that was mathematically corrected and further empirically studied by Canatar et al. [14] : (linear) neural predictivity is (linear) regression, and (linear) regression has inductive biases, irrespective of the underlying neuroscience. We focus on linear regression because of its ubiquity in the literature, but other preference functions (e.g., RSA [49] , CKA [48] , SVCCA [63] , Procrustes [84] , etc.) would not escape this critique; rather, they would simply change the inductive biases of the chosen preference function.Together, these insights suggest that the neural regression methodology, and more broadly the idea that a uniform set of metrics can automate model selection, may be fundamentally flawed by overfitting to those metrics rather than advancing our understanding of the brain. We conclude by suggesting a re-evaluation of such methodologies. 
neural regressions can reach incorrect conclusions with high confidence: 
 In vision, Bowers et al. [9] documented how artificial networks preferred by the neural regressions methodology lack or contradict properties of primate vision, and others have identified additional flaws [54, 88, 20, 35, 26, 27, 23, 36, 52] . Here, we chose to focus on the clearest example of a failure of the neural regressions methodology: grid cells. Why focus on grid cells? Grid cells -a surprising and important Nobel Prize-winning discovery [34] -differ from vision, audition and language in that humanity possesses scientific models [29, 11, 10, 76] that have repeatedly proven predictive ( [77, 91, 32] ), not in the regressions sense but in the sense of exhibiting fundamental properties, e.g., localization of each module to a two-dimensional subspace, quantization of grid module periods, preserved low-dimensional dynamics across waking and sleep that were subsequently validated. In a domain we understand well, how did the regressions methodology fare?When applied to a specific neural circuit (grid cells) that humanity possesses near-normative models of, the neural regressions methodology preferred incorrect models with high confidence.As context, the key research questions about grid cells are modeling their dynamics and the evolutionary causes for their existence. Previous and now near-normative models showed how strong recurrent interactions leading to pattern formation, coupled with a way for movement inputs to shift the pattern phase and thus perform path integration, could generate grid cell dynamics [11, 47] ; and  
discussion: 
 To summarize, NeuroAI lacks canonical definitions of neural similarity, and such notions are likely task-, system-, and question-dependent, as well as difficult to quantify. Rather than facing these challenges, the neural regressions methodology sidesteps them by defining a proxy -for instance, the test R 2 of linear regressions fit between biological recordings and artificial activations -and then choosing networks based on this proxy. The networks that win a selection process do so because of the proxy's implicit biases, independent of any meaningful relationship with the brain.To explain with an analogy, in the field of language modeling, researchers want language models to generate responses preferred by humans. However, collecting human preferences is slow, costly and noisy, so researchers instead train modified language models called reward models to serve as proxies of human preferences. These reward models are a proxy for what we actually care about -human preferences -but the field is willing to use these proxies because the reward models are directly trained to emulate human preferences and are correlated with human preferences empirically [93, 78, 6, 60, 51] ; even so, overfitting to the reward models at the expense of human preferences is still a commonly encountered problem [78, 30, 1] .In comparison, in computational neuroscience, researchers want models that are most similar to brain system(s) of interest. However, interacting with neural systems and running experiments is slow, costly and noisy, so researchers instead fit neural regressions to serve as proxies of neural similarity. These regressions are a proxy for what we actually care about -neural similarity -but in contrast with reward models, neural regressions are not trained to emulate neural similarity and have no known relationship with neural similarity.To reiterate an earlier point, we focused on linear regression because of its ubiquity in the literature, but other proxies of neural similarity (e.g., RSA [49] , CKA [48] , SVCCA [63] , Procrustes [84] ) would not escape this critique; rather, other proxies would simply change the pertinent implicit biases.Altogether, these insights suggest that the neural regressions methodology may be flawed, teaching us about the preferences that we as researchers implicitly chose instead of advancing humanity's understanding of the brain. We conclude by calling for a critical and careful re-evaluation in computational neuroscience of how the neural regressions methodology is used and interpreted. For a Future Outlook, please see Appendix Sec. B. 
 
 Appendix: Entorhinal Cortex [66] , Vision in Macaque IT Cortex [25] , Audition in Human Cortex. [80] , Language in Human Cortex [17] . Later work [14] provided a spectral theory of the neural regressions methodology, which reveals results like these are attributable to general properties of linear regression, not the brain.that multiple grid modules played key roles in disambiguating position over large ranges and in error correction [29, 76] . Later, deep recurrent networks trained in a supervised manner to path integrate were shown to learn grid-like units [7, 19, 74] , and neural-regressions based work [57] showed that these supervised deep path integrators achieved the best performance possible at predicting recordings from mouse medial entorhinal cortex, leading the authors to call for better neural data.However, multiple independent lines of evidence demonstrated that these high R 2 deep learning models are worse models of grid cells: (1) The required supervised targets, putative place cells, contradict known biological properties of place cells at both the single cell and population levels [67] ;(2) The grid-like units lack key properties of real grid cells: periodic triangular tuning curves, multiple discrete grid modules, and specific ratios between grid modules [66, 68] ; (3) the artificial grid units in some works were statistically indistinguishable from low pass-filtered noise [74, 75] . (4) In terms of evolutionary origins, the path integration objective of high-R 2 networks is not a sufficient objective for grid cells, as demonstrated in empirical deep neural network work [42, 41, 68] , argued by prior theoretical work [28, 76, 53, 83] , and shown by newer deep learning models [31, 85, 22, 70, 86, 87] .To summarize, the neural regressions methodology strongly supported deep learning-based path integrators because the networks achieved high neural predictivity scores, despite their discrepancies with multiple key criteria of neural similarity (listed in Appendix Sec. A). Why?3 The Neural Regressions Methodology Reveals the Implicit Biases of Regression, Not Which Candidate Networks Are Similar to the Brain Schaeffer et al. [66] made a conjecture: "different [models] achieve different neural predictivity scores because they learn different intrinsic dimensionalities, that then provide richer/poorer bases for linear regressions." Larger models simply provide more basis features for regression, and thus can provide better predictions independent of the similarity with the brain. To test their conjecture, the authors trained the same networks studied by Nayebi et al. [57] and empirically discovered that reported test Pearson correlations exhibit an approximately linear-log relationship with a widely-used measure of effective dimensionality called participation ratio (PR) [21] (Fig 2a ).More precisely, consider P stimuli, and denote artificial activations with M units as X ∈ R P ×M and biological responses with N neurons as Y ∈ R P ×N . We fit linear models using p < P data:EQUATION)Letting XX T = P i=1 λ i v i v T i , Schaeffer et al. [66] empirically found that approximately:EQUATION)Participation ratio (PR) is a linear geometric measure of effective dimensionality: for uniform eigenvalues, the PR is the ambient dimensionality, whereas for a single non-zero eigenvalue, the PR is 1.Concurrent and subsequent work found quantitatively similar results across species, modalities, brain circuits and recording technologies: Elmoznino and Bonner [25] in deep convolutional networks trained on vision tasks to predict macaque IT cortex (Fig 2b ), Tuckute et al. [80] in deep auditory networks to predict human brain-wide fMRI responses (Fig 2c ), and Cheng and Antonello [17] in language models to predict human brain-wide fMRI responses (Fig. 2d ). This finding by four independent research groups across different data modalities, tasks, architectures, species and recording technologies is puzzling. Are these results indicative of some deeper scientific insight into the brain?In our view, no. This pattern is attributable to the neural regressions methodology, not the brain. Participation ratio (PR) was a reasonable first guess but an imprecise one that was subsequently refined into a more complete spectral theory of the regressions methodology. Canatar et al. [14] showed the normalized error E g (p) of any linear model Ŷ(p) def = X β(p) is given as:EQUATIONwhere γ = P i=1 pλ 2 i (pλi+κ) 2 and κ = α reg +κ P i=1 λi pλi+κ must be solved self-consistently. This result says that the focus on PR by previous work was incomplete: PR partially captures the dimensionality of the learnable subspace, but the error also depends on the terms {||Y T v i || 2 2 /||Y || 2 F } i , which express whether the target Y lies in that subspace. Thus, higher PR can be beneficial to express the task fully, but can also be harmful by being too expressive and then harming sample complexity. This partially explains why ZCA whitening to maximize participation ratio did not achieve exceptional neural predictivity, why increasing the number of covariates does not necessarily increase neural predictivity [25] , and how randomly initialized networks can achieve high neural predictivity [45, 2] .While it may be tempting to think that these empirical results and this spectral theory of neural predictivity have taught us about the brain, note that this theory makes no assumptions about a neural, behavioral, biological, ethological or otherwise meaningful relationship between X and Y. Rather, as its origin makes clear [8, 13, 12] , this theory is fundamentally a description of linear regression [69] . Consequently, this leads to the following realization:Taken to its extreme, the neural regressions methodology has taught us the implicit biases of our chosen proxy function (e.g., test R 2 of linear regression), not which candidate artificial neural networks are actually similar to the brain.In this paper, we intentionally do not provide a general definition of "neural similarity" (see Future Outlook -Appendix Sec. B), in part because we feel such a definition is likely highly context dependent. But we can offer a constructive example in the narrow context of grid and place cells viewed at the level of their circuit dynamics. When considering models, researchers often consider the following (non-exhaustive) list of relevant criteria for evaluating whether a model is similar to the circuit:• Individual neurons exhibit equilateral triangular periodic tuning curves • In the population of grid cells, multiple grid periodicities exist • The periodicities of the grid cells are quantized • The quantized periods of the modules exhibit precise ratios between adjacent periods • The population states of each grid module (subpopulation with common period) lie on the surface of a 2D torus. • The cell-cell relationships of co-modular grid cells (and their toroidal population states) are invariant across spatial environments and behavioral states. • In any environment, grid-like tuning is present from the first trajectory fragment.• While grid cells remain invariant in their relationships across environments, place cells remap or scramble their relationships.Despite our critiques of the neural regressions methodology, model-system comparison is a fundamental and necessary component of a modeling science. How, then, can we move beyond flaws arising as a consequence of emphasizing only a single metric?One possibility is to use a number of different comparisons that emphasize different aspects of model and system. This may include comparing behavior on top of neural activations, as is already a feature of the Brain-Score platform ( [89, 71] ), neural dynamics on top of neural geometry [59] , or using a variety of different metrics that have different biases ( [35] ). Beyond linear regression, computational neuroscience has introduced a number of other candidates into the literature, including RSA [49] , Procrustes [84] , CKA [48] , SVCCA [63] , and a number of variants of these metrics. All of the above metrics compare geometric features of neural activations. Recently proposed methods, such as Dynamical Similarity Analysis (DSA, [59] ) compare different features of the system, like dynamical structure. Older work sought to study similarity using combined perspectives of behavior, representations, dynamics and circuit mechanisms, e.g., [65] . Using more types of comparison, both in terms of metrics and data, should help mitigate the biases of individual comparisons, making Goodharting more challenging. However, it is important to note that even combinations of such metrics are liable to fall prey to Goodhart's law. Depending on the scientific question, the relevant quantity to be compared may change.More generally, beyond significantly increasing the number of types of comparisons being done, it is worth taking a step back and asking what we mean by a 'good model'. Although we do not define neural similarity here, in our view, neural similarity depends on the task, the neural system and the particular scientific question. Thus, in our opinion, neural similarity cannot always be neural predictivity. This perspective lays bare the difficulty and, in a sense incoherence, of seeking to globally define good models in terms of one or even a set of metrics. 


Decision: In light of the positive reviewers' feedback and relevancy of the submission, we are pleased to accept this paper for presentation at UniReps 2024. We kindly ask the authors to incorporate the reviewers' suggestions and feedback in the final camera-ready version of the manuscript.

Reviews: review: title: title: Review of Submission 31
main_review: main_review: **Quality:** While I do tend to agree with the overall assessment that neural regression scores alone do not necessarily indicate greater similarity to the brain (and that "similarity" to the brain is itself context-dependent and not well-defined), I think a good chunk of the criticism here comes from the debate surrounding theories of the emergence of grid cells -- a debate that I don't think is definitively resolved in favour of one side (there are several responses to the Schaeffer et al. works from Sorscher et al. with defences of the latter's theory).

Section 3 seems to provide more compelling arguments in my opinion, but it is worth noting that papers such as Elmoznino & Bonner [20 in the paper] do not claim that DNN models with high-dimensional representations are better models of the visual cortex, they explicitly state that they are just able to predict neural activity better. From that paper:

> ... we found that the relationship between latent dimensionality and encoding performance generalized across layer depth, meaning that even within a single layer of a DNN hierarchy, encoding performance can widely vary as a function of latent dimensionality.

> ... while our findings show that computational models of visual cortex benefit from high latent dimensionality, our method cannot speak to the dimensionality of visual cortex itself and was not designed to do so ... high-dimensional DNNs should generally better explain neural activity even if neural representations in the brain are low-dimensional.

which do not seem like claims that models with high-dimensional representations are more similar to visual cortex than others. In that sense, it seems to align well with the argument made in Section 3 that higher dimensionality should improve neural predictability.

Finally, the paper calls for greater nuance in defining neural similarity in a context-dependent manner, which is appreciated. Indeed, one may find great similarity in the behaviour of two systems but the neural dynamics underlying these could be entirely different -- so comparisons must be made at multiple levels to be comprehensive. Suppl. Section B discusses some metrics that could provide for more comprehensive comparisons than linear regressions (while acknowledging the potential for Goodharting in those cases as well), but some work that explicitly discusses the flaws in these metrics could be cited, e.g., Davari et al., Dujmovic et al., Helmer et al. among others.

Davari, M., Horoi, S., Natik, A., Lajoie, G., Wolf, G., & Belilovsky, E. Reliability of CKA as a Similarity Measure in Deep Learning. In The Eleventh International Conference on Learning Representations.

Dujmovic, M., Bowers, J., Adolfi, F., & Malhotra, G. Inferring DNN-Brain Alignment using Representational Similarity Analyses can be Problematic. In ICLR 2024 Workshop on Representational Alignment.

Helmer, M., Warrington, S., Mohammadi-Nejad, A. R., Ji, J. L., Howell, A., Rosand, B., ... & Murray, J. D. (2024). On the stability of canonical correlation analysis and partial least squares with application to brain-behavior associations. Communications Biology, 7(1), 217.

**Clarity:** The writing is clear, however, I did spot a few issues:
1. Figure 1 is not referenced in the text, and the caption for the right panel does not clearly indicate whether it is just a schematic or an actual plot. When the paper argues that "neural similarity" isn't well-defined, it's unclear to me what the neural similarity on the y-axis is. It's also unclear what "number of papers" is referring to here.
2. Figure 2 caption: Missing reference to equation.
3. Line 421: Missing reference/in-text citation.
4. Relatively minor, Goodhart's law finds mention in the abstract and appendix but never in the main text.

**Originality:** I'm not sure whether discussing the originality of this submission matters, or how best one would judge originality in the context of a perspective paper. Criticism of regression-based neural similarity metrics has existed for a while and several works have addressed this, but this work attempts to consolidate the evidence and provide a coherent viewpoint: that "neural similarity" is not well-defined.

**Significance:** This submission would definitely elicit discussion at the workshop on what "neural similarity" is and how best to judge this in different contexts. It could also motivate researchers to be cautious and comprehensive when claiming similarity of models to brains.
title: title: Neural predictivity does not imply similarity
main_review: main_review: This opinion piece claims a common misconception in NeuroAI is that better neural predictivity (from Artificial Neural Networks (ANNs) representations) implies higher representational similarity to the brain (measured in terms of linear predictivity, RSA, CKA, shape metrics, etc.). Taking the example of grid cells, the authors cite the ANN models that do not exhibit grid cell-like properties but still achieve good neural predictivity. Furthermore, the authors cite converging evidence from various fields that demonstrate a relationship between the neural predictivity ($R^{2}$) and intrinsic dimensionality of ANNs (measured in terms of participation ratio). Overall, the authors call for a critical re-evaluation of the neural regressions methodology.

I agree with the author's opinion that predictivity does not imply similarity, however, I believe clearer and more explicit evidence would improve the paper towards removing this misconception. Some of the things that can be improved are as follows:
  1. The authors state that the definition of neural similarity and model goodness is context-dependent (line 30). At the end of section 2, the authors claim that highly predictive models are worse models of grid cells, but fail to provide the context of the implied model goodness. Intuitively, these ANN models are worse in terms of implementing properties of grid cells but are better in terms of predicting their activity. Rather than ending this section with a question, explicitly stating that neural regressions methodology supported ANN models because of predictivity rather than similarity, would provide a better takeaway for the reader.
  2. The sub-heading of section 3 can be improved. The authors state at the start of the paper that the neural regressions methodology is about whether and which ANNs are more similar to the brain. This implies that any insights gained from this methodology would be about the models and not about the brain. Hence, it is intuitive that the methodology does not reveal insights into the brain, but is rather believed to provide insights about model goodness. I recommend rephrasing the heading (similar to the last line in section 3) to be more explicit about what the methodology can and can not do.
  3. In section 3, the authors provide evidence that neural predictivity is dependent on the intrinsic dimensionality of ANNs. However, the link to neural similarity here is not explicitly stated. The effect of intrinsic dimensionality on neural similarity is not intuitive, but I believe it is implied that they are independent. Explicitly showing the effect of effective dimensionality on neural predictivity and neural similarity would greatly improve this section by providing better evidence for the claim.
  4. Similar to point 3, I believe the title of the paper can also be improved to explicitly state that maximizing neural regression scores does not teach us about which model is better/model goodness (or a similar statement framed as a question).
  5. At the end of the discussion, the authors state that the neural regressions may be fundamentally flawed. I believe that the methodology is not flawed, but the takeaways that some researchers take from it are flawed. It might be useful to highlight the context in which the methodology is flawed (in terms of similarity), or better state the cases where researchers may use this methodology (neural response prediction) and the cases where this should be avoided (estimating similarity).

I believe these fixes are very important to provide a clear message to the readers and avoid further confusion in the line of research which is already riddled with misconceptions (eg., the one stated in the paper). In the absence of a rebuttal period to improve the paper, I, unfortunately, opt for a lower rating. However, when improved, I believe this can be a crucial and important paper in guiding the field.

Other questions/suggestions:
  1. It is unclear to me how Figure 1 Right was obtained. If it is a schematic figure, explicitly state it.
  2. Figure 2: Label subplots (a-e)
  3. Figure 2 caption, typo: Eqn ??
  4. Line 94, mention that PR is the participation ratio, as PR is used several times without explicitly defining the acronym
  5. Line 421, fix RSA citation
  6. The first paragraph of the discussion is very similar to the second paragraph in the introduction. I recommend rephrasing for better use of limited space.
title: title: An interesting and provocative paper that needs more detail
main_review: main_review: This is an interesting brief report arguing that the high alignment in neural predictivity between higher dimensionality models is due to linear regression rather than properties of the networks themselves. I think this paper makes an interesting point, but while some aspects were clear others were harder to follow and felt like they need more justification. For example, it is not clear that methods of neural alignment do not account for similarity in eigenvectors/values and regression targets. Further, most of the references cited arguing for higher effective dimensionality show that simply increasing the number of regressors does not increase linear predictivity, and I had a hard time reconciling these results with this paper. Further, while there is a brief discussion of other methods in the appendix, it was not entirely clear whether the arguments extended to other common types of neural similarity / mapping measures.

Overall I think this is an interesting paper that will be a good addition to the workshop.
title: title: Review
main_review: main_review: This work calls into question the framework of evaluating models of the brain based on their ability to predict neural activity in the brain using linear regression on the model representations. The authors first use grid cells as evidence that the regression framework fails. They argue that models that perform well at predicting grid cell response (in terms of linear regression) lack the well-studied characteristics of grid cells. The authors then argue that the goodness of fit of linear regressions are more related to implicit biases of linear regression rather than a model capturing the relevant properties of neural responses. Specifically, they cite theoretical work demonstrating that the participation ratio (PR) of the model responses is an important factor in determining the performance a linear regression task, as well as empirical work showing that performance of linear regression correlates with PR.

Strengths: Overall I enjoyed reading this paper. It raises important consideration and questions about using a linear regression framework to evaluate models of the brain.

Relevance: This work is a clear fit for this workshop, therefore I recommend that it be accepted.

Question: Based on the authors' second argument that much of the performance of linear regression can be explained by the PR of model responses, why not just control for the PR when using linear regression as a measure of performance? Given the theory supporting the authors' argument, it seems like there should be a principled approach to this.

Minor: there is a missing equation reference in Figure 2's caption.

Comments: title: Responding to "Many Unsubstantiated Claims" by Dr. Nayebi
Response to Reviewer 65GB's Review of Submission 31
Response to Reviewer ZVey's Review of Submission 31
Response to Reviewer bHu3's Review of Submission 31
comment: I appreciate the author's response and believe the paper's quality has improved. It has definitely started a conversation in the field that would be very helpful to future researchers.
While we appreciate the interest, these comments misunderstand or misrepresent both our work and prior work.

> In short, unlike what the authors claim, effective dimensionality is not the whole story for model-brain linear regression, for several reasons:

Contrary to the implication, the manuscript actually state that effective dimensionality is not the whole story: “[Effective dimensionality] was a reasonable first guess but an imprecise one that was subsequently refined into a more complete spectral theory of the regressions methodology.” and “Thus, higher [ED] can be beneficial to express the task fully, but can also be harmful by being too expressive and then harming sample complexity. This partially explains why ZCA whitening to maximize participation ratio did not achieve exceptional neural predictivity, why increasing the number of covariates does not necessarily increase neural predictivity [25]”

Section 3 should not be understood as endorsing conjectures by early work in this area; rather, we are offering a historical retrospective of the field to substantiate our claim that “the field’s results have devolved into re-discovering general properties of (linear) regression.”

> The spectral theory due to Canatar*, Feather* et al. 2023 that they cite is misrepresented in this paper as not making any assumptions about a meaningful relationship between the model ($X$) and the neural data ($Y$). However, this theory crucially relies on model alignment with brain data. This is already evident from their equations, as it quite literally includes an alignment term

This response either misunderstands the spectral theory or our manuscript. The task-alignment is not a property of the brain or candidate artificial networks. Rather, the task-alignment is a geometric property of arbitrary covariates and targets that describes how well linear regression will generalize; consequently, the task-alignment describes generalization of linear regression, independent of the brain.

That the spectral theory is a statement about linear regression is explicitly stated by the original authors in an [earlier paper](https://proceedings.neurips.cc/paper/2021/hash/691dcb1d65f31967a874d18383b9da75-Abstract.html): “Our result is general in that it applies to any kernel, data distribution and target function.” This is why we write, “this theory makes no assumptions about a neural, behavioral, biological, ethological or otherwise meaningful relationship between X and Y”.

Dr. Nayebi's misinterpretation of the spectral theory has been (in our view) [confirmed by the senior author of the spectral theory](https://twitter.com/CPehlevan/status/1847325545115406472).

To clarify, we are not criticizing inductive biases. Rather, we are pointing out that Goodharting has shifted the field from offering insights into the system of interest (the brain) to insights into the preference function (e.g., test R^2 of linear regression).

> Furthermore, the above models that this paper cites participation ratios for, were also compared by their original authors against several non-fitted metrics (like RSA, score distributions, simpler-than-linear mappings, etc.), which found similar conclusions that matched linear regression results

Our perspective critiques how neural comparisons are currently used and interpreted. Though the current opinion focuses on linear regression, we will extend a future version of this work to encompass neural comparisons generally (including RSA).

To add nuance, using multiple neural comparisons (e.g., linear regression + RSA) is not inherently good and cannot be relied upon as a universal sanity check. Rather, multiple comparisons should be used to complement and correct for one another’s inadequacies. For example, in the grid cell literature, a common comparison between neurons and units is how grid-like the tuning curves are; however, this grid score is imperfect because it scores rotational symmetry - not hexagonal lattices - so researchers use multiple complementary methods (e.g., topological data analysis, circuit identification).

**To briefly restate our overarching perspective, comparing a system and a model requires specifying what similarity to that system means. Once “neural similarity” has been defined, one can then identify proxy metrics for neural similarity (e.g., test R^2 of linear regression). However, as the field optimizes against a fixed proxy metric, the field begins Goodharting: overfitting to the proxy at the cost of true “neural similarity” (which we believe is a neural system-, setting- and task- dependent notion).**

Solutions include: (1) clearly identifying what similarity to a specific neural system means, (2) choosing bespoke metrics for the neural system of interest rather than a fixed generic metric, (3) identifying specific limitations of each metric in that specific context + ensembling different metrics to correct for other metrics’ shortcomings in that specific context.
UniReps has graciously given submitting authors the opportunity to respond to reviewers, so we wanted to thank you for reviewing our paper. We submitted the work to workshops to begin a conversation and to solicit feedback, and we want to highlight how we've incorporated your feedback into our work.

- "Rather than ending this section with a question, explicitly stating that neural regressions methodology supported ANN models because of predictivity rather than similarity, would provide a better takeaway for the reader." -> We rewrote the end of Section 2 to explicitly state this divergence.

- "I recommend rephrasing the heading (similar to the last line in section 3) to be more explicit about what the methodology can and can not do." -> We changed the text of Section 3 to adhere more closely to the ending of the section

- "I believe the title of the paper can also be improved to explicitly state that maximizing neural regression scores does not teach us about which model is better/model goodness (or a similar statement framed as a question)." We changed the title of the manuscript and also preappended "Position" to clarify that this manuscript is a position piece.

- "At the end of the discussion, the authors state that the neural regressions may be fundamentally flawed. I believe that the methodology is not flawed, but the takeaways that some researchers take from it are flawed." We think this is a valuable nuance that is much appreciated. We will work to integrate this into the manuscript as we work towards a full conference submission.

- "It is unclear to me how Figure 1 Right was obtained. If it is a schematic figure, explicitly state it." -> We explicitly labeled Figure 1 as a *Schematic*

- "Figure 2 caption, typo: Eqn ??" We fixed the incorrect equation reference

- "Line 94, mention that PR is the participation ratio, as PR is used several times without explicitly defining the acronym" -> Done

- "Line 421, fix RSA citation" -> Done. Thank you for your keen eye.
UniReps has graciously given submitting authors the opportunity to respond to reviewers, so we wanted to thank you for reviewing our paper. We submitted the work to workshops to begin a conversation and to solicit feedback, and we want to highlight how we've incorporated your feedback into our work.

> Further, most of the references cited arguing for higher effective dimensionality show that simply increasing the number of regressors does not increase linear predictivity, and I had a hard time reconciling these results with this paper.

We rewrote a subset of Section 3 to clarify that we are not arguing that higher effective dimensionality monotonically increases neural predictivity. The two points we wished to communicate in Section 3 is (1) that the community discovered an interesting but *imprecise* phenomenon that was later theoretically and empirically refined to clarify that higher ED can help _or hurt_ in specific contexts, but that (2) these ED discoveries and ED theory are properties of _linear regression_, independent of neuroscience.

If you can suggest how we might better express this perspective, we would be keen to hear your feedback!
UniReps has graciously given submitting authors the opportunity to respond to reviewers, so we wanted to thank you for reviewing our paper. We submitted the work to workshops to begin a conversation and to solicit feedback, and we want to highlight how we've incorporated your feedback into our work.

- We included citations for the relevant prior work we missed that you recommended (Davari, Dujmovic, Helmer)
- We added a reference to Figure 1 in the text, and we clearly labeled Figure 1 as a **Schematic**
- We fixed the missing reference to an equation in Figure 2's caption
- Regarding "Relatively minor, Goodhart's law finds mention in the abstract and appendix but never in the main text.", this is a good point that we agree with. The feedback will help us shape this workshop submission into a more full submission.

> I think a good chunk of the criticism here comes from the debate surrounding theories of the emergence of grid cells -- a debate that I don't think is definitively resolved in favour of one side (there are several responses to the Schaeffer et al. works from Sorscher et al. with defences of the latter's theory)

*If we can solicit your feedback*, would it be helpful if we were to walk through the history of this debate and why we believe the evidence resolves the debate in favour of one side? We felt like recapitulating that history and the evidence might distract from the focus on neural regressions, but perhaps readers might appreciate a comprehensive appendix with a deep dive into the grid cell debate.

> a debate that I don't think is definitively resolved in favour of one side (there are several responses to the Schaeffer et al. works from Sorscher et al. with defences of the latter's theory).

If you have time and interest, you might find the latest response informative: https://arxiv.org/abs/2312.03954