Range, not Independence, Drives Modularity in Biologically Inspired Representations

Authors: Will Dorrell, Kyle Hsu, Luke Hollingsworth, Jin Hwa Lee, Jiajun Wu, Chelsea Finn, Peter E. Latham, Timothy Edward John Behrens, James C. R. Whittington
Primary area: applications to neuroscience & cognitive science
Keywords: neuroscience, representation learning, disentanglement, modularisation, neural networks, hippocampus, cortex

Abstract:
Why do biological and artificial neurons sometimes modularise, each encoding a single meaningful variable, and sometimes entangle their representation of many variables? In this work, we develop a theory of when biologically inspired networks---those that are nonnegative and energy efficient---modularise their representation of source variables (sources). We derive necessary and sufficient conditions on a sample of sources that determine whether the neurons in an optimal biologically-inspired linear autoencoder modularise. Our theory applies to any dataset, extending far beyond the case of statistical independence studied in previous work. Rather we show that sources modularise if their support is ``sufficiently spread''. From this theory, we extract and validate predictions in a variety of empirical studies on how data distribution affects modularisation in nonlinear feedforward and recurrent neural networks trained on supervised and unsupervised tasks. Furthermore, we apply these ideas to neuroscience data, showing that range independence can be used to understand the mixing or modularising of spatial and reward information in entorhinal recordings in seemingly conflicting experiments. Further, we use these results to suggest alternate origins of mixed-selectivity, beyond the predominant theory of flexible nonlinear classification. In sum, our theory prescribes precise conditions on when neural activities modularise, providing tools for inducing and elucidating modular representations in brains and machines.

Decision: accept (poster)
Reviewer scores: 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730595677851

thread_type:
official_review

<thread_object>
creator: reviewer_gn2p

<content>
summary:
This paper seeks to explain why and when a population of biological or artificial neurons sometimes modularise and sometimes entangle the representation of source variables. This is a fundamental question that is highly relevant to both neuroscience and AI. The authors propose and prove a new theory emphasising the importance of the shape of empirical data distribution in extreme regions in dictating whether neurons are mixed selective or modular. Specifically if the sources to be represented are supported in all extreme regions, the neurons modularise. The application of this theory outside of linear autoencoders is tested in feed forward and recurrent neural networks, including experiments that provide explanations for discrepancies in previous neuroscience literature.

strengths:
This work is original, of high quality and undoubtedly contributes to the community’s understanding of neural modularisation. The nonlinear verification of theory and additional application to neuroscience results are significant for the field and a strength of the paper. The submission is well written and clear throughout, although its clarity suffers somewhat due to the amount this submission seeks to cover.

weaknesses:
- In my opinion this submission contains too much, and would benefit from more focus and time spent on fewer experiments. The appendix is already large but some experiments could be moved there.  
- Figure text and panels are too small throughout.
- It is not clear how relevant encoding of the extreme points of source distributions are for computation / cognition. I.e. neurons do not just autoencode.
- The bio description of energy minimisation assumes l2 penalty is an appropriate penalisation function for modelling biology. This is a fair starting assumption, but no argument is presented about how this maps to the costs biological neurons will be seeking to minimise.

questions:
- Biologically inspired linear rnns are repeatedly mentioned. Linear rnns are less like biological circuits (which are nonlinear).  Is the biologically inspired term referring to the energy costs?
150: satisfied for all w is a little unclear. The proof could  
- Autoencoding seems limited as an objective for theory, in that brains perform computations over inputs and states and produce behaviour. Is it not the relevance of representing the extreme points for behaviour that is important? 
- Section 2.2 assumes positive weights?
- 120 "be better" is imprecise 
- Fig 1d, neuron’s angle isn’t described? 
- I don’t understand the what where regression justification. 
- How relevant are these results for more typical ANN experiments?  E.g classic image benchmarks or language modelling tasks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730745856560

thread_type:
official_review

<thread_object>
creator: reviewer_1fwm

<content>
summary:
This paper studies the conditions under which modularity should arise in optimal representations. The authors developed a mathematical theory of when a modular representation should be favored in a linear autoencoder. They found that modularity should appear when the support of the sources is “sufficiently spread”. The paper also presented simulation results to show that some of theoretical results may be generalizable to non-linear problems. The later sections of the paper applied these theoretical ideas to explain some experimental observations from neurophysiological experiments in cognitive tasks. 
The paper makes several interesting points about when modularity should be favored, and applies the theory to several examples.

strengths:
Originality: The theoretical part of the work builds up prior work by Whittington et al, 2023 and other studies. Previous work by Whittington et al, 2023 assumed mutual independence of the sources. In the current work, the authors show that, with several additional assumptions, “sufficient spread” of the factors of variation can also lead to modular representation. The theory has some new elements, although it is a bit incremental. The application to the several neuroscience problems seems to be new.

Quality: The paper considered both linear and nonlinear cases. This is a strength. For the former, analytical results were provided. For the latter, some preliminary numerical results were given. 
The paper also considered several neuroscience applications. This may also be seen as a strength.


Clarity: The overall structure of the paper is clear. Some intuitions behind the theory were provided.


Significance: The question of when modularity arises in optimal representation is an interesting one and we still lack a clear understanding. This work made a few interesting points on this problem.

weaknesses:
The writing needs improvements throughout the paper. In particular, the description of the theory can be substantially improved. For example, Theorem 2.1 should be made more accessible. 


While several applications are attempted, each application appears to be preliminary. If the model predictions and experimental tests can be made more rigorous, that would strengthen the paper.

In Section 5, there are some qualitative differences between the model predictions and the data. As the paper pointed out, Panichello and Buschman (2021) showed that a substantial fraction of the neurons were tuned to both colors, contradicting a key prediction of the model. This seems to be a more important feature of the data compared to the issue of orthogonality v.s. non-orthogonality.

Looking at the math, the theory appears to only work for scalar variables. Can it be applied to circular variables? If the answer is no, the applications to real data would be questionable. In section 5, color is sampled from a color-wheel. 

Based on the way the theory is written, the results seem to rely the assumption that the energy (or cost) is a quadratic function of neural activity. If the cost scales linearly with neural activity, would the theoretical results change fundamentally? Assuming a linear scaling could make sense biologically, as the metabolic cost may scale linearly with the number of spikes. 

Relevant earlier theoretical literature on grid cell modularity was not cited/discussed (e.g., by Fiete/Burak et al, and Wei/Prentice/Balasubramanian).


It is difficult to understand what is really going on in Fig. 1. Fig.1 may come out too early and the results need to be better unpacked. 


The theory seems to ignore biologically-relevant noise.

questions:
What are the assumptions about the noise in the system being studied? How does noise affect the theoretical results?

In section 5, the authors seems to be equating orthogonality with modularity. Am I understanding this correctly? If this is the case, can the authors unpack the idea?

Can they authors unpack the results in Fig. 5c? Was that an actual simulation or just a schematic?

The definition of the matrix in Eq. 10 is unclear. Please clarify. 


Can the authors explain what the first part of the title mean?

In the abstract, it is stated that “From this theory, we extract and validate predictions…” What does “extract” predictions mean? [minor point]
</content>
</thread_object>

</thread>
<thread>
creation date:
1730780888366

thread_type:
official_review

<thread_object>
creator: reviewer_a7qj

<content>
summary:
This paper investigates why neural representations in biologically inspired networks sometimes form modular structures, where each neuron encodes a single variable, and other times create mixed-selective representations. The authors develop a theory that predicts when modularization will occur in networks optimized for energy efficiency with nonnegative firing rates. They derive necessary and sufficient conditions for modularity based on the spread of source variables. The theory is validated in both linear and nonlinear networks. The theory provides a cohesive explanation for the conflicting findings in the prefrontal cortex and entorhinal cortex data from neuroscience studies.

strengths:
1. The paper introduces a novel theory that precisely predicts necessary and sufficient conditions for modular representations in biologically inspired networks, extending previous work beyond statistical dependencies.

2. The mathematical formulation is rigorously derived, and validated across various neural network architectures and experiments.

3. The theory provides explanations for conflicting neuroscience findings and has close links to biologically plausible architectures and brain representations.

4. The paper provides a cohesive theory for understanding modularity in neural representations, with implications for both interpreting biological neural data and guiding the design of artificial neural networks for better interpretability and efficiency.

5. The paper is well-written, presenting complex theoretical concepts with clarity and intuition.

weaknesses:
1. The experiments use nonnegative activities in neural networks, which aligns with biological plausibility, but it would be valuable to discuss inhibitory neurons in the brain and how inhibition might relate to the theory and findings.

2. While the L2 norm of firing rates and weights is a reasonable approximation for biological energy, other biological constraints (e.g., sparse connectivity, synaptic range, anatomical structure, and decoding flexibility) may also play a role.

3. It’s unclear how the theory would extend to more complex datasets. For example, what would the different conditions/variations in source variables mean in naturalistic data (e.g., natural images, audio, text)? How might we approximate "spread", and quantify modularity conditions in such stimuli?

4. The discrepancies in prefrontal working memory modeling and the brain data could have further explanations, particularly why some neurons tune to both colors despite orthogonal encoding and why exact subspace angles were not obtained.

questions:
1. Beyond prefrontal working memory and the entorhinal cortex, does the theory generalize to other modular representations in the brain?

2. If decoding flexibility were considered as a biological constraint, how might it impact the theory and its predictions?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732167726851

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Comment

comment:
We’d like to thank all the reviewers for their time and attention to our work, and for their enthusiasm. Their comments have helped us to improve and sharpen the paper.

In general the reviewers praised the rigorous analytical results (a7qJ, 1fwm), empirical tests in nonlinear networks (a7qJ, Gn2p), links to neuroscience (a7qJ, Gn2p), clarity of presentation (a7qJ, Gn2p), and choice of question (1wfm). In addition, the reviewers shared concerns that we have tried to address. To highlight the largest changes in the new submission, we have:
1. Moved the section on PFC to the appendix
2. Extended the entorhinal section to comment on potential sources of mixed-selectivity
3. Significantly clarified the main theorem presentation.
4. Added more theorems that show our core results generalise to other choices of activity norms and multidimensional latents.

We now run through the shared concerns in more detail.

**PFC DATA** First, there was concern that, of the two comparisons to neural data, the PFC comparison was weaker. The original submission contained both a numerical mismatch of the subspace alignment between the theoretical and empirical representations (a1qJ), and the neural level modularisation results do not at first blush agree with our theory (1fwm, a1qJ). Further, we previously elided concepts of orthogonality and modularity in confusing ways (1fwm). While we have lots of thoughts for how to improve and develop these results, we largely agree with the reviewers on these points and have concluded that, at this stage, the results are slightly too preliminary.

As a result we have moved the section on the PFC to the appendix, and have replaced it with a new section that expands how our theory contributes to the ongoing debate over the role of mixed selectivity in the brain. This highlights how our theory leads to multiple additional interpretations of mixed selectivity of neurons, beyond the prevalent view that mixed selectivity exists to enable flexible downstream categorization (Rigotti et al. 2013).

**Overly Simplistic Model** Second, many reviewers pointed to the simplicity of our model, asking about including inhibitory neurons (a1qJ), other biological constraints (sparse connectivity, synaptic range, anatomical structure)) (a1qJ), biological noise (1fwm), and our focus on an autoencoding objective (Gn2p). These are all reasonable details to include. However, in contrast, we see the simplicity of our setting as a good thing! We are interested in understanding why neurons sometimes choose to mix their representations, and other times not. To do this we build the simplest model that can capture the observed phenomena. In order to make the representation encode the variables we use autoencoding, which implies minimal additional structure. Then we find that just energy minimsation leads us to rich phenomenology capable of explaining neural data in ways that were not previously possible, making testable experimental predictions. Each of these other effects would definitely be interesting to study, and other works have used such effects to derive interesting modularity conclusions (such as Liu, Khona, Fiete, & Tegmark. 2023), but with just these simplest set of constraints we can go surprisingly far!

**L2 Activity Norm** Some particular concerns reviewers pointed to were the use of the L2 activity norm rather than some other norm (1fwm, Gn2p), and the limitation to scalar variables (1fwm). In our updated version we present new results that show that our core result (that support independence drives modularisation) generalises to other choices of activity norm (Appendix C in new submission) and to multidimensional variables (Appendix D). In particular, this means that the particular choice of activity norm is not vital, as long as the simple idea that spiking is energetically costly is used.

**More Complex Tasks** Finally, reviewers wondered whether our work generalised to image or language tasks (a7qJ, Gn2p). First, we point out that figure 2 included experiments performed on a standard image dataset, Isaac 3D, and we now highlight this in the paper. This gives a model for how our work might be applied to these settings. I.e. if you think a particular representation contains information about a set of variables, for example the latent factors in Isaac 3D, then our theory gives tools to think about what properties of those variables will determine whether they are represented in a modular or mixed fashion. We are actively trying to use these results to think about modularisation in more complex tasks, such as in LLMs, by making guesses about the encoded latent variables, and seeing whether we can predict the modularity of their representation from the support properties. This, however, will be for future work, as it involves significant extra work.

Overall, we would like to thank the reviewers for their time and effort, and look forward to hearing what they have to say in response.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work tackles a timely subject in distributed computing in the brain: modularity. The work aims to provide a theoretical framework that can identify when and why modularity occurs. The theory itself builds off of the analysis of optimal encoding of a set of inputs, with the general constraints that the encoding must be "simplest" by minimizing the norms of the weights, and that the encoding variables are non-negative. Both assumptions are used to identify the key result, formulate the result in a way where comparing a convex hull of the data to a theory-driven ellipse results in an understanding of when inputs should be mixed or not. The paper itself is very dense with a significant appendix detailing much of the theoretical ideas and derivations. The primary result of the work was that the concept of data spread could be considered as important as statistical independence between variables that are encoded. 

The reviewers mentioned a number of points where clarity could be improved, and the authors in response amended the manuscript and included additional analyses for extending the model and adding more experimental results. Some of the more central points might be more important to the assessment of this manuscript. One reviewer raised the question of why en encoder framework, independent of task requirements etc. is the right framework in which to view modularity. Similarly, the question from another reviewer that the model depends too much on the linear setting. I believe these concerns actually speak to a deeper assumption: what are the chosen inputs? There are multiple levels in which tasks and performance can be modularized or not. In some ways receptive fields are "non modular" in the space of localized point sources. The entire endeavor rests on definition of what is a "single" input and "single" processing unit (here chosen to be a single neuron). These are largely assumed and I think instantiated in a number of different questions the reviewers had in different ways. 

Beyond these concerns, another recurring theme was the length of the manuscript. It is true the authors have done much work, but there is also significant duplication of information. I agree that a more focused paper would likely fair much better for the shorter format here. 

In all, while this is a borderline case, the reviewers in the end felt the merits outweighed the shortcomings and recommend accepting this work.