Representation with a capital 'R': measuring functional alignment with causal perturbation

Authors: jacob s prince
george a alvarez
talia  konkle
Venue: NeurIPS_cc_2024_Workshop_UniReps
Published: 2024-09-24 09:46:43

Abstract: Ambiguity surrounding the term 'representation' in biological and artificial neural systems hampers our ability to assess their alignment. In this paper, we draw a critical distinction between two notions of representation: the conventional 'representation' as a mere encoding, and 'Representation-with-a capital-R', which entails functional use within a system. We argue that while current methods in neuroscience and artificial intelligence often focus on the former, advancing our understanding requires a shift toward the latter. We critique existing linking methods such as representational similarity analysis and encoding models, highlighting key limitations in their ability to capture functional correspondence. We then propose an updated paradigm that involves specifying explicit models of information readout and testing their functional properties using causal perturbation. This framework, which treats neural networks as a new species of model organism, may help reveal the principles governing functional representations in both biological and artificial systems.

Main paper: 
 introduction: 
 The goal of understanding how the visual system represents and transforms information has driven widespread efforts to model brain responses using deep encoding models. Initiatives like Brain-Score (1, 2) , Algonauts (3) , and Sensorium (4) benchmark the predictive power of deep neural networks (DNNs) against large-scale datasets of neural activity and behavior, aiming to reveal core principles of information processing. These efforts and other related studies (e.g. 5) have sparked extensive debate about the most appropriate methods for encoding: linear vs. nonlinear regression models (6) , rotation-invariant vs. rotation-sensitive mappings (7, 8) , and, approaches that emphasize tuning properties vs. the overall geometry of the representations (9, 10, see 11 for review). Why, as a field, have we yet to reach consensus on which procedures provide the most valuable insights? This lack of consensus may stem from an over-reliance on representational measures that fail to capture the functional aspects of the underlying processes. At the heart of this debate is a deeper issue: viewing 'representation' merely as an encoding format fails to account for the functional role these representations play in downstream processes.Encoding models and representational similarity analysis (RSA, 12) index what we call representation with a lowercase 'r': how information is encoded in a system. This is a broad notion of representation, reflecting any static mapping of stimuli into different bases, such as neural measurement units or model activations, or even pixel space. The modeling aim is to capture the structure of the encoding, and understand its format. This approach is safe, pragmatic, leveraging measurable data from biological and artificial systems and allowing for ranking of DNN feature spaces based on predictivity. However, typical similarity metrics stop far short of addressing whether a given set of internal representations play a similar functional role within both systems. As a result, they provide an incomplete picture of how encoded information is actually used to drive behavior or cognition.In contrast, Representation with a capital 'R' refers to the functional use of encoded information within the system (Figure 1 ). It is not enough for a model to account for the brain's encoding format of stimuli; for information to qualify as capital 'R' Representation, the system must actively use it in ways that mirror the biological processes driving perception or behavior. This idea, rooted in philosophical discussions of mental representation (see 13 for review), emphasizes the functional role of different activity patterns with respect to behavior (see also 14, 15, 16, 17) . For example, patterns of activity across face-selective neurons can reliably enable experimenters to decode non-face object information such as houses, chairs, and other objects (18) . However, if this encoding does not contribute functionally to the brain's recognition of these items (19, 20) , it would fall short of capital 'R' Representation. The key is functional relevance: the system must read out the content meaningfully. Testing for these signatures requires causal perturbation of the system, for instance by silencing model units to simulate the effect of lesions on behavior.Lower-case 'r' representations capture a static, often disconnected, view of neural activity While vision models increasingly succeed in matching neural data from different stages of the visual hierarchy in isolation, they often fail to replicate the actual interactions governing information flow from V1 to high-level visual cortex. The best layer for V1 in one model might appear mid-way through, while for V4, it might come from an earlier layer in another model. Feature reweighting makes divergence from hierarchical correspondence even more likely, and there's often no expectation that one single model will successfully account for multiple stages of the hierarchy (though some have pursued these modes of comparison, e.g. 21, 22, 23) . This limitation is evident in the current implementation of Brain-Score, where any layer of any model can be deemed the best match for a given benchmark. The risk with this approach is that it may isolate models based on superficial pattern matching, without necessarily advancing our understanding of the causal principles driving visual function.We advocate for developing new alignment procedures that directly test whether the information measured in a system is functionally used. This is important, as current linking methods may convey that different models have similar alignment, while masking that they make very different functional predictions. To illustrate this point, we implement a simple case study involving two linear encoding models of brain responses from the fusiform face area (FFA) of human visual cortex (Figure 2A ). The two regression scenarios both take as input activations from the same DNN layer (AlexNet fc6), but map onto FFA data using different regularization constraints (see Appendix for complete details). One approach uses standard ridge regression, while the other employs sparse (lasso) regression with an added positivity constraint (20, 24) .Both regression methods yield comparable predictions on held-out data, appearing similar on the surface in their ability to account for lower-case 'r' representations. However, with free access to the models' internals, we can perform causal perturbations to test the underlying commitments of these different encoding models. After lesioning the 1% of layer units that received the highest absolute weight (importance) in the regression, we observe that the two models produce very distinct patterns of functional predictions about what impairments are likely. For example, across the 1000 ImageNet categories, we computed which categories would have the most impaired recognition, finding a correlation between the two models' lesion-induced deficits of only r=0.149. Moreover, several of the most-impaired categories for lesions derived from sparse-positive regression contained faces (e.g. trombone, balance beam), unlike those derived from ridge regression weights (e.g. container ship, pole).In sum, while encoding models may show similar performance in a lowercase 'r' sense of representation, they can obscure critical (and untested) differences in a capital 'R' sense. Causal perturbations, such as lesioning highly weighted units, provide a powerful way to distinguish models' functional alignment and reveal the different commitments they make about underlying processes. Models that predict equally well under normal conditions can behave very differently when the systems are perturbed, providing key insights about the relationship between activation patterns and functional properties. These signatures may support stronger metrics of alignment to human brain and behavior.Overcoming these limitations requires making explicit commitments about how information is used and propagated within both systems. Popular linking methods like RSA and encoding models make implicit assumptions about readout. For instance, RSA assumes that tuning directions in neural populations do not matter for readout, as arbitrary rotations of the feature space do not alter the overall geometry. Instead, RSA focuses on matching population-level geometry between model layers and brain areas, assuming that the downstream system has access to the entire population of neural signals. Similarly, encoding models allow for arbitrary transformations of model unit tuning relative to their brain counterparts, implying that tuning matters less in models than it does in the brain. To move beyond these limitations, it is essential to make explicit commitments about how information flows through the system (e.g. 26) and how representations are read out to drive behavior.In feedforward hierarchical ReLU models, tuning directions-the patterns of a unit's responses to different inputs-are critical for information propagation. In these models, only positive signals are passed on to subsequent layers, making tuning a key determinant of function. Neuropsychological evidence suggests a similar link between high neural activity (i.e., selectivity) and functional outcomes in the brain's visual system (27, 28, 29, 30) . By designing comparison methods that respect these known signatures, we may naturally gain deeper insight into functional roles of representations.In line with this, some have proposed incorporating positivity and sparsity constraints into linear encoding models of the ventral visual stream to pressure them toward better alignment with capital 'R' representations (24) . The positivity constraint minimizes tuning reorientation (how much the feature weights warp the original tuning directions of the model layer), which reflects the operating principles of networks that propagate only positive activations. The sparsity constraint reduces feature remixing, encouraging a one-to-one alignment between DNN and brain feature tuning (see also 10). We hypothesize that these constraints will lead to model-to-brain links that better capture the functional relevance of information flow in both systems, providing a more faithful index of capital 'R' representation.What techniques will promote better investigation of functional or mechanistic alignment? Lesion studies, transcranial magnetic stimulation (31, 32, 33) , electrical microstimulation (34), pharmacological inactivation (35) , and optogenetics (36) are powerful tools for understanding causal structurefunction relationships, but they are costly and difficult to conduct. This hinders our ability to directly observe how representations in one brain region contribute to downstream functions. DNNs enable us to simulate scenarios that are difficult or impossible to explore in biological systems. By systematically lesioning or pruning layers, or inactivating specific subcircuitry, we can study how representations in earlier layers contribute to those in downstream layers, and directly link each to functional outcomes (37, 38) . This provides a powerful framework for exploring capital 'R' representations. Complementary insight could be gained from applying recent techniques from the fields of mechanistic interpretability and AI safety, which explore how information routes and circuits operate in neural networks (e.g. 39, 40, 41, 42) .In this way, we stand to learn about information transmission and Representation with a capital 'R' by studying the models directly. Each DNN can be conceived of as a simplified model organism, with more constrained inputs and outputs relative to the human brain. Crucially, a model's ability to perform meaningful tasks, such as object categorization, makes it valuable to study in its own right, regardless of whether its solution is human-like. The key idea is that the more we understand the internal processes of these models, and how different computational units contribute to the behavior of interest, the more likely we are to uncover principles of information transmission that could shed light on the function of biological systems. 
 
 Appendix:  


Decision: In light of the positive reviewers' feedback and relevancy of the submission, we are pleased to accept this paper for presentation at UniReps 2024. We kindly ask the authors to incorporate the reviewers' suggestions and feedback in the final camera-ready version of the manuscript.

Reviews: review: title: title: An Interesting Position Paper
main_review: main_review: - Summary: 
   - This paper explores the idea that many (all?) methods for comparing representations suffer from a common flaw: they do not privilege information content in representations that is more *functionally relevant* in their comparisons. For example two methods may produce high similarity scores (as measured by linear regression, RSA variants, or global shape metrics) by being similar only in terms of features that are discarded in subsequent stages of computation. 
   - The author's suggest that in order to measure the functional similarity between different representations, we ideally need to conduct causal perturbations (i.e. simulating lesions) and observe the impact on downstream behavioral outputs. They conduct one such experiment to disambiguate between two candidate mapping methods between an artificial representation (AlexNet) and fMRI responses in the fusiform face area (FFA). They find that linear mappings that use (1) ridge regression and (2) Lasso regression with a positivity constraint on the weights produce similar levels of predictivity, but distinct predictions when units identified as important for neural prediction are lesioned. Namely, lesioning the Lasso-important units  seems to produce more behavioral impairment for image classes containing faces than lesioning Ridge-important units.

- Strengths:
   - Clarity: ideas experiments are clearly presented, and the analysis of results is both fair and reasonably supports the arguments made in the text.
  - Originality: while a slew of works have considered both novel methods for representational comparisons and the implications of choosing one method over another, the approach of using "artificial" lesioning as a simple way to attempt to measure the functional similarity of how information is encoded in distinct representations, is to my knowledge novel (and potentially very interesting). 
  - Fair accounting of weaknesses: the difficulty in conducting lesion studies in-vivo with current experimental methodologies prevents this method of "functional comparison" from closing the loop (i.e. determining whether lesioning FFA produced similar behavioral deficits as lesioning Lasso-important units in Alex-Net). This limitation is clearly addressed in the main text and highlighting this as a weakness is important and valuable to the community (as it helps to dream up solutions!). 

- Weaknesses/Suggestions:  
  - On the positivity constraint: this is a bit of a nit, but I am not sure that this is necessarily a very large issue within current common practices in terms of linear regression from model to brain. For example many models simply use the outputs of ReLU layers to map to brain responses, so there is no issue with inverting these activities from the functional perspective (i.e. negative weight connections to downstream layers could be functionally important). The point about inverting pre-nonlinearity activations (that will be thresholded to zero) is well taken, but a quick survey of models on the BrainScore platform shows that many models already use the outputs of the nonlinearity to perform the mapping (see for example the top-of-the-leaderboard model here: https://www.brain-score.org/model/vision/646)
  - On the sparsity of mappings (More of a question/thought than a weakness): Should how many model units should be mapped to a specific type of brain measurement depends on the supposed level of physiological correspondence between the representations? For example, if we believe model units are akin to single cell responses than striving for a 1-to-1 mapping between units and Ephys single unit measurements seems reasonable, but would also imply that a large number of model units should be weighted/pooled in order to explain, say, an fMRI voxel response. In general I think it would be interesting to consider or comment on how the issues outlined in the paper relate to/are impacted by the fact that neural responses are measured with a wide variety of resolutions. 
  - Measuring functional similarity between pairs of Artificial networks: I think repeating the artificial lesioning experiments but instead on model-to-model comparisons (i.e. where AlexNet is mapped to ResNet via Ridge/Lasso regression) would be both interesting in its own right and a step toward more directly demonstrating that sparse regressions better capture functional similarity, even though these kind of two-way lesioning experiments are currently mostly impractical in the brain-to-model setting.
title: title: Compelling case with interesting direction, but could better address representation analysis literature
main_review: main_review: **Summary**:
- This perspectives paper concretizes an often-discussed distinction between basic encodings (lowercase representations) and representations with functional roles (uppercase 'R' Representations). The authors argue that current neuroscience and AI methods too-often focus on the former, while advancing our understanding requires a shift toward the latter. They critique existing methods like representational similarity analysis and propose a new paradigm based on encoding methods that respect the functional properties of representations.

**Strengths**:
1. Thoughtfully discusses the notion of how representations can be more meaningfully understood in the context of the downstream behaviors they cause and functional relevance
2. Effectively points out gaps in what traditional representation analysis methods (e.g. RSA) offers the Neuroscience and AI communities in terms of understanding
3. The causal perturbation approach to illustrating how encoding methods differ is quite clever

**Weaknesses**:
1. There are existing ways to study representations in a functionally-relevant and task-relevant way. The paper correctly addresses RSA but does not discuss existing alternative methods to study representations in more functionally-relevant ways. For example, [Cohen et al (2020)](https://doi.org/10.1038/s41467-020-14578-5).
title: title: Review of Submission67
main_review: main_review: This paper raises some criticisms to the current approach to assessing similarity between representations, especially between artificial and biological neural networks. The authors argue that current representational similarity analysis, focused on population-level statistical properties of representations, falls short of detecting true functional similarity between data encodings. Instead, they propose to focus on a different concept of Representation, that takes into account the functional role that encodings have in subsequent read out layers.

Strengths:

- The paper is well-written and easy to follow.
- The idea that statistical similarity between representations is not enough to draw parallels between very different systems is reasonable and well justified.

Weaknesses:

- My main concern with this paper is that, while the authors raise relevant criticisms against the current approach to representational similarity, they do not really propose any actionable way to put their insight into practise.
title: title: Will spark useful discussions
main_review: main_review: This is a well-written and thought-provoking short abstract. The authors argue that we as a community should place greater emphasis on causal perturbations and understanding of deep networks and biological systems, rather than just characterizing representational geometry (devoid of any context). It serves as sort of a call-to-action and poses more questions than it provides answers. I think it would be a good conversation starter at the workshop.

The one tidbit of analysis shown in Figure 1C is interesting, but I was confused on a critical detail. Did the authors re-train the linear regression models after lesioning the units in the deep network in order of importance? Or was the linear regression model frozen as well? To me, it would make sense to re-fit the regression model after you lesion a unit because the LASSO regularization will tend to zero out units with redundant information. Thus, if you lesion the only remaining unit with a certain kind of tuning (assuming LASSO lesioned out units with correlated tuning) you'd see a giant drop in performance. However, in this scenario re-fitting the LASSO regression from scratch would result in a less significant drop in performance.

I think the author's result is probably interesting either way, but the details should be clarified.