Range, not Independence, Drives Modularity in Biologically Inspired Representations
Authors: Will Dorrell
Kyle Hsu
Luke Hollingsworth
Jin Hwa Lee
Jiajun Wu
Chelsea Finn
Peter E. Latham
Timothy Edward John Behrens
James C. R. Whittington
Venue: ICLR 2025 Poster (2025)
Keywords: neuroscience
representation learning
disentanglement
modularisation
neural networks
hippocampus
cortex
Primary area: applications to neuroscience & cognitive science
TL;DR: We derive necessary and sufficient conditions on task structure under which nonnegativity and energy efficiency lead to modularised representations, and use this to understand the behaviour of artificial neural networks and biological neurons.

Abstract: Why do biological and artificial neurons sometimes modularise, each encoding a single meaningful variable, and sometimes entangle their representation of many variables? In this work, we develop a theory of when biologically inspired networks---those that are nonnegative and energy efficient---modularise their representation of source variables (sources). We derive necessary and sufficient conditions on a sample of sources that determine whether the neurons in an optimal biologically-inspired linear autoencoder modularise. Our theory applies to any dataset, extending far beyond the case of statistical independence studied in previous work. Rather we show that sources modularise if their support is ``sufficiently spread''. From this theory, we extract and validate predictions in a variety of empirical studies on how data distribution affects modularisation in nonlinear feedforward and recurrent neural networks trained on supervised and unsupervised tasks. Furthermore, we apply these ideas to neuroscience data, showing that range independence can be used to understand the mixing or modularising of spatial and reward information in entorhinal recordings in seemingly conflicting experiments. Further, we use these results to suggest alternate origins of mixed-selectivity, beyond the predominant theory of flexible nonlinear classification. In sum, our theory prescribes precise conditions on when neural activities modularise, providing tools for inducing and elucidating modular representations in brains and machines.

Published as a conference paper at ICLR 2025
RANGE, NOT INDEPENDENCE, DRIVES MODULARITY IN BIOLOGICALLY INSPIRED REPRESENTATIONS
Will 1
Kyle Hsu
Luke Hollingsworth
Jin Hwa Lee
Jiajun Wu?
Chelsea Finn
Peter E. Latham
Tim Behrens
James C.R. Whittington 2.3
University College London
"Stanford University
Oxford University
Co-First
Co-Second
ABSTRACT
Why do biological and artificial neurons sometimes modularise, each encod- ing a single meaningful variable, and sometimes entangle their representation many variables? In this work, we develop a theory when biologically spired networks-thos that are nonnegative and energy efficient-moduulriise their representation of source variables (sources). We derive necessary and sufficient conditions sample of sources that determine whether the neurons in op- timal biologically-inspreed linear autoencoder modularise. Our theory applies to any dataset, extending far beyond the of statistical independence studied in previous work. Rather we show that sources modularise if their support is "sufli- variety
ciently spread". From this theory, of empirical studies on how data distribution affects modularisation in nonlinear feedforward and recurrent neural networks trained on supervised and unsupervised tasks. Furthermore, we apply these ideas to neuroscience data, showing that range independence can be used to understand the mixing or modularising of spatial and reward information in entorhinal recordings in seemingly conflicting experiments. Further, we use these results to suggest alternate origins of miixedselecttviig be- yond the predominant theory of flexible nonlinear classification. In sum, our theory prescribes precise conditions on when neural activities modularise, providing tools for inducing and elucidating modular representations in brains and machines.
extract and validate predictions
INTRODUCTION
Our brains are modular. At the macroscale, different regions, such as visual or language cortex, perform specialised roles; at the microscale, single neurons often precisely encode single variables such as self- self-position (Hafting et al.) 2005) or the orientation of visual edge (Hubel and Wiesel 1962). This mysterious alignment of meaningful concepts with single neuron activity has for decades fuelled hope for understanding neuron's 's function by finding its associated concept. Yet, neural recording technology has improved, it has become clear that many neurons behave in ways that elude such simple categorisation: they appear be mixed selective, responding a mixture of variables linear and nonlinear ways (Rigotti et al. 2013 Tye et al.) 2002).. The modules vs. mixtures debate has recently been reprised in the machine learning community. Both the disentangled representation learning and mechanistic interpretability subfields are interested in when neural network representations decompose into meaningful components. Findings have been similarly varied, with some studies showing meaningful single unit response properties and others clearly showing mixed tuning (for full discussion on related work see Appendix A This brings us to our main research question: Why neurons, biological and artificial, sometimes modular and sometimes mixed selective?
In this work, we develop a theory that precisely determines, for any dataset, whether the optimal learned representations will be modular or not. More precisely, in the linear autoencoder setting, we show that modularity in biologically constrained representations is governed by a sufficient spread condition that roughly be thought of as measuring the extent to which the range of the source variables (sources, aka factors of variation) is rectangular. This sufficient spread condition bears resemblance to identifiability conditions derived in the matrix factorisation literature (Tatli

---

Published as a conference paper at ICLR 2025
and Erdogan 2021a b). though both the precise form of the problem study and the condition we derive differ (Appendix [A]. This condition on the sources is much weaker than the case of mutual independence studied in previous work (Whittingtn
al. 2023), and commensurately broadens the settings we can understand using our theory. For example, the loosening from statistical independence to rectangular support enables (RNN) representations of dynamic variables modularise.
predict when linear recurrent neural network
Further, these results empirieally generalise nonlinear settings: we show that our source support conditions predict modularisation in nonlinear feedforward networks on supervised and autoencoding tasks as well as in nonlinear RNNs. We also fruitfully apply our theory neuroscience data. We provide an explanation for why grid cells only sometimes warp in the presence of rewards, on the basis of the support independence properties space and reward. Further, we use these results to suggest other settings in which mixed-selectivity might appear beyond the traditional nonlinear classification theory. summary, our work contributes to the growing understanding of neural modularisation by highlighting how source support determines modularisation and explaining puzzling observations from both the brain and neural networks in a cohesive normative framework.
2 MODULARISATION IN BIOLOGICALLY INSPIRED LINEAR AUTOENCODERS
We begin with our main technical result: necessary and sufficient conditions for the modularisation of biologically inspired linear autoencoders.
2.1 PRELIMINARIES
Let IRd. be vector of d, scalar source variables (sources, aka factors of variation). We interested how the empirical distribution, p(s), affects whether the sources' neural encoding (latents), z E RRd,, are modular with respect to the sources, i.e., whether each neuron (latent)
is functionally dependent on at most one source. Following Whittington et al. (2023), we build simplified model in which neural firing rates perfectly linearly autoencode the sources while maintaining nonnegativity (since firing rates are nonnegative):
Wmn s + b 8 = W bout: =2 O.
(1)
Subject to the above constraints, we study the representation that uses the least energy, as in the classic efficient coding hypothesis (Barlow al. 1961). We quantify this using the l² norm of the firing rates and weights (other activity norms considered later):
min W W Bau
(1= + > (I) + Win s.t.
(2)
We remark that there are common analogues of representational nonnegativity and weight energy efficiency in modern machine learning (ReLU activation functions and weight decay, respectively). When the sources are statistically independent, i.e. p(s) = -I p(s;), then the optima of ( have modular latents (Whittington et al. (023). We now improve on this result by showing necessary and sufficient conditions that guarantee modularisation for any dataset, not just those that have statistically independent sources.
2.2 INTUITION FOR SOURCE SUPPORT MODULARISATION CONDITIONS
To provide intuition, consider a hypothetical mixed selective neuron,
= Wi1 + Wy2882 + by,
(3)
is functionally dependent on both 81 and 82, i.e. it mixed selective. Perhaps, however, a modular representation, in which this neuron is broken into two separate modular encodings, would be better. We can create such a solution with two neurons each coding a single source:
Z Sy1$1 + by: Zy" = Wy282 + by".
For simplicity, we assume the two sources are linearly uncorrelated, have mean zero, and are supported on interval centered at zero (Figure a top row; relax these assumptions in our full theory). Then, for fixed W and Wys. the optimal (energy efficient) bias should be large enough to keep the representation nonnegative, but no larger:
by min [W5181 $1 + Wy282], by min Wy181, by" min Wy282, 81.22
(5)
22
2

---

Published as a conference paper at ICLR 2025
where the minimisations over the empirical distribution p($1, 82). Now that both representations are specified, we can compare their costs. In our problem modularity is driven only by the activity loss (Appendix(B). Further, in this simplified setting, most terms in the activity loss zero cancel, and one finds that the mixed selective 3 case uses less energy than the modular (4 when
03 < by. + b3...
(6)
The key takeaway lies in how by is determined by joint minimization over $1 and 82 (5)
Assume positive W and Wy2: then 81 and 82 take their minima simultancously, as in the middle row of Figure Th the mixed bias must be large:
bj min [1y1s1 + wy282] = min [1111] min| [1y25]] = by + byr
In this case, the energy the mixed solution will always be worse than the modular, since b3 (by + byr)2 > i + Dz.... Alternatively, mixing will be preferred when 81 and do not take on their most negative values at the same time, as in the bottom row of Figure Dh, since then by can be smaller while maintaining positivity, and the corresponding energy saving satisfies the key inequality (6)
Sources
Modular Encodings
Sources
Conoe
Optimal Welghts
Highlighted Tuning
$
Hoxt Mbad
Nouron
Data Point (i]
Data Point [1]
Point [I]
34
Range Independent- Costly Mixed Neurons 8s #2
Remove One Danapoint
[Mi]]
Most Mitddd
Naumon
$1
Point
5, No Red Point + Range Dependent Cheaper Mixed Neurons Bi
Neuran's Angle
Theory predicts mixedd
Enengy Baving
Theory predicts modular
0.1
Data Point
Theory Moxing &Energy
Normalised Mutsinformation
Figure 1: a) Top row: Values of two sources across dataset and their modular encoding with associated costs (shaded regions). Middle row: If the dataset includes the red datapoint (left) then the variables take their minimal value at same time, Bottom row: If the red point instead missing (left) then the mixed encoding can use a smaller bias while remaining positive and save energy (right). b) Our modularisation conditions (Theorem 2.1 are equivalent whether the convex hull of the data (orange regions) encloses data-derived ellipse (Theorem 2.2 The precision these conditions validated two experiments (rows) which the source support differs by single datapoint. Removing this datapoint stops the convex hull from enclosing the key ellipse, resulting in a
mixed encoding must include a large bias, using more energy (right).
representation. In the middle column, we plot the weight vectors of the neurons. The top representation is modular all weight vectors
axis aligned, whereas the bottom contains mixed selective neurons. This is reiterated by extracting the most mixed selective a function the sources (right column). c) Our conditions (left) predict modularisation better than measures of statistical independence such as source multinformation (right). Here the y axis measures how mixed the representation is by the largest angle between neuron's 's weight vector and one of the source axes, and the left axis measures how much the inequalities are broken rough heuristic for how mixed the optimal solution is.
(orange dot) and plotting its activity as
2.3 PRECISE CONDITIONS FOR MODULARISING BIOLOGICALLY INSPIRED LINEAR AUTOENCODERS
We now make precise the intuition developed above. All proofs are deferred to Appendix B
3

---

Published as a conference paper at ICLR 2025
Theorem 2.1. Let E E ERR, Wm Rdxxd, bm R Wout RRaxds and bout E R with dg > dg. Consider the constrained optimization problem
(1121) 1 + (I] 1 I| l1:))
min Win.bin, W bont
(8)
21 = Wins + biu: s W Woutz bout, z 220 0.
s.t.
where i indexes finite set of samples of s. At the minima of this problem, the representation modularises, Le. each row of Win has at most one non-zero entry, iff the following inequality is satisfied for all w E IR
d, Σ
de Σ j=1
minnww s(u1)
wywy (5)
(wy min min )
(9)
JJ'
where 8 = 8 {slil); and assuming that min; 5/ S max;
s Vj [d.] w.l.o.g.
Our theory prescribes set of inequalities (9) that determine whether an optimal representation modular. These inequalities come from the difference activity energy between a modu- lar and mixed solutions, just like the intuition we built up in Section 2.2, and in particular (6) a single incquality is broken, the optimal representation is mixed (at least in part); else, it is modular: optimally, each neuron's 's activity is function of single source. These inequal- ities depend on two key properties of the sources: the shape of the source distribution's 's sup- port in extremal regions, and the pairwise source correlations. Remarkably, they do not de- pend on the energy tradeoff parameter A. These inequalities can be visualised, as done in the (1 and wrapped figure below. For a given dataset, {sll]] (blue dots), we can calculate
each min; as they are simple functions of the dataset. For all unit w, we can draw the line s
d2
w x
Σ
uy min o
wywy
= Vw"" Fw
(10)
where we have defined the following matrix:
(min, s |) (s s 1 b
(s 0fg
8
) (min; )2
(82
Bly
F
1
(min; 2/2
(11)
Each w gives us line, and if there is at least line that bounds the source support (such the red one), then an inequality is broken and the optimal representation mixed. Else it will be modular. This exercise also motivates the following equivalent statement of our conditions.
Theorem 2.2. In the same setting as Theorem 2.1 define the set E = (y y F- 1}. Then an equivalent statement of Theorem 2.1 is the representation modularises iff E lies within the convex hull of the datapoints.
This therefore provides a simple test: create F and draw the set which, if F is positive definite (as often is), is an ellipse as shown in the wrapped figure above. The optimal representation modularises iff the convex hull of the datapoints encloses E, Figure Tp. F has some positive and some negative eigenvalues, then the set E is unbounded, and the optimal representation must mix.
2.4 RANGE INDEPENDENT VARIABLES MODULARISE
To clarify our result we present a particularly clean special case.
Corollary 2.3. In the same setting Theorem 2.7 the optimal representation modularises if all sources are pairwise extreme-point independent, i.e. for all j,j E [d,]²:
min
[ Ejr
E
[K' |maxx 8 min 8/ = min s
(12)
4

---

Published as a conference paper at ICLR 2025
In other words, if the joint distribution is supported on all extremal corners, the optimal representation modularises.
As presented, theory has some limitations. It focuses sources that are I-dimensional, and uses a specific choice of activity norm, the L2, when others might be more reasonable. In fact, however, the core result that it is the independence of the variables' range (i.e. how rectangular they are) that drives modularity is very broadly true. In Appendix B..4 we show that this core result generalises to other activity norms, and in Appendix [.5 we show that also applies the modularisation of multi-dimensional variables such as angles on a 2D circle. Thus, our key takeaway is that the independence of variables' range is what determines whether the optimal representation is modular.
2.5 EMPIRICAL TESTS
Validation of linear autoencoder theory. We show our inequalities correctly predict modularisation. In particular, as an illustration of the precision of our theory, we create a dataset which transitions from inducing modularising to mixing via the removal of single critical datapoint (Figure [p). Further, we generate many datasets, create the optimal representation, and measure the angle 0 between the most-mixed neuron's 's weight vector and its closest source direction, proxy for modularity. The left of Figure T shows that our theory correctly predicts which datasets are modular (0 = 0). Further, despite our theory being binary (will it modularise or not?), empirically we that the degree to which the inequalities in Theorem 2.1 are broken is a good proxy for how mixed the optimal representation is. Finally, on the right of Figure TP show that on the same datasets measure of source statistical interdependence, as used in previous work, does not predict modularisation.
Predictions beyond our theory. From our theory we extract qualitative trends to empirically test in more complex settings. Since extremal points play an outsized role in determining modularisation, we consider three trends that highlight these effects. (1) Datasets from which successively smaller corners have been removed should become successively less mixed, until at a critical threshold the representation modularises. (2) It vital that is not just any data but corner slices that are removed. Removing similar amounts of random centrally located data from the dataset should not cause as much mixing. (3) Introducing correlations into a dataset while preserving extreme- extreme-point or range independence should preserve modularity relatively well.
just
3 MODULARISATION IN BIOLOGICALLY INSPIRED NONLINEAR FEEDFORWARD NETWORKS
Motivated by our linear theoretical results, we explore how closely biologically constrained nonlinear networks match our predicted trends. We study nonlinear representations with linear and nonlinear decoding supervised and unsupervised settings. Surprisingly, coarse properties predicted by our linear theory generalise empirically to these nonlinear settings (Figure 2
Metries for representational modularity and inter-source statistical dependence. To quantify the modularity of a nonlinear representation, we design family of metrics called conditional information- theoretic modularity (CInfoM), an extension of the InfoM metric proposed by HHsu et al. (2023). Intuitively, a representation is modular if each neuron is informative about only a single source. We therefore calculate the conditional mutual information between a neuron's activity and each source variable given all other sources. The conditioning of sources leaking information about each other; prior works consider independent sources or do not account for this effect. CInfoM then measures the extent which neuron specialises its favourite source, relative to its informativeness about all sources. In order to compare multiple schemes that change p(s) in different ways, we report normalised source multinnoormation (NSI) as measure of source statistical interdependence. NSI involves estimating the source multinformation (aka total correlation) DxL (p(s) )) _ p(s:)) and normalizing by En= ((s)) max, (s:). We defer further exposition the CInfoM metric to Appendix
other sources is necessary to remove the effect
What-where regression. Inspired by the modularisation of what and where into the ventral and dorsal visual streams, we present nonlinear networks with simple binary images in which one pixel is The network is trained to concurrently report within which region of the image ("where"", "), and where within that region ("what"") the on pixel is found, producing two outputs which we take as our sources, each an integer between and nine. (More complex inputs or one-hot labels also work,
5

---

Published as a conference paper at ICLR 2025
What-Where Regression,
Noninearly Autoencocing Sources, d,
IsaadD Quantised Latent Autoenooding.
Linear conditionl InfoM
1.0
Condiional InfoM
Conditional InfoM
0.42
0.8
0.40
0.7
0.8
0.6
0.30
0.0
0.7
0.000
0.056
-00000
0.08
0.14
0.000
0.005
.0100
0.015
0.020
0.00
0.01
0.02
0.08
0.04
Normallzzd source multinoomraiionn
Normalized source intinoommation
Normalzed source multinfomation
b)
Conner dropout
Oggonall treengheeiing (24)
Diagenal dropout (3)
Comer dispout (10)
Diagonal strenghheiig (S6)
Digonal dropou. (11)
Camer diopout (15)
Daponal wrenghhening (500))
Diagonal diopost
13)
(17)
Figure 2: a) We study three tasks varying source dimensionality de, model complexity, supervision signal, and input data modality (Section 3). For each task, we plot conditional InfoM, measure modularity, against normalised source multinformation (NSI), a measure of the statistical dependencies in the sources. Across the three tasks, observe that cutting corners severely degrades modularity (orange). whereas equivalently changing the NSI ways preserve the rectangular support boundary (green, purple) affects modularity less. Shaded regions denote standard error of the mean. b) For What- Shat-Where Regression, we visualise distributions at various NSI values (top) as well as the tuning of an example neuron. The z (y) axes correspond different what (where) values, and the colour shows the neuron's response. All neurons begin modular (left three plots), dropping corners heavily degrades modularity (left plot of subsequent groups of three).
source
We regularise the activity and weight energies, and enforce nonnegativity using ReLU. what and where are independent from one another, e.g. both uniformly distributed, then under our biological constraints (but not without them, see Appendix the hidden layer modularises what from where. Breaking the independence of what and where leads to mixed representations in patterns that qualitatively agree with our theory (Figure 2h left): cutting corners from the source support causes increasing mixing. Conversely, making other more drastic changes to the support, such as removing the diagonal, does not induce mixing. Similarly, introducing source correlations while preserving the rectangular support introduces less mixing when compared to corner cutting that induces the same amount of mutual information between what and where. The various source distributions different NSI values are visualised in Figure
Appendix [F)
Nonlinear autoencoding of sources. Next we study nonlinear autoencoder trained to autoencode a set of source variables. Again we find (Figure middle) that under biological constraints independent source variables (NSI = 0) result in modular latents and that source corner cutting (orange) induces far more latent mixing compared introducing source correlations while preserving rectangular support (green), or removing central data (purple).
Disentangled representation learning of images. Finally, for experiment involving high- dimensional image data, we turn to a recently introduced state-of-the-arr disentangling method, quantised latent autoencoding (QLAE; Hsu al. (2023) 2024)). QLAE is the natural machine learning analogue to our biological constraints. It has two components: (1) the weights in QLAE are heavily regularised, like our weight energy loss, and (2) the latent space axis-wise quantised, introducing privileged basis with low channel capacity. In our biologically inspired networks, non- negativity and activity regularisation conspire similarly structure the representation: nonnegativity creates a preferred basis, and activity regularisation encourages the representation to use as small portion of the space as possible. We study the performance QLAE trained to autoeneode subset of the Isaac3D dataset (Nie 2019), naturalistic image dataset with well defined underlying latent dimensions. We find the same qualitative patterns: corner cutting is a more important determinant of mixing than range-independent correlations or the removal of centrally located data (Figure 2a right).

---

Published as a conference paper at ICLR 2025
4 MODULARISATION IN BIOLOGICALLY INSPIRED RECURRENT NETWORKS
Compared to feedforward networks, recurrent neural networks (RNNs) often much more natural setting for neuroscience. Excitingly, the core ideas of our analysis for linear autoencoders also apply to recurrent dynamical formulations, and similarly extend to experiments with nonlinear networks.
4.1
LINEAR RNNS
Linear sinusoidal regression. Linear dynamical systems can only autonomously implement expo- nentially growing, decaying. or stable sinusoidal functions. We therefore study linear RNNs with biological constraints trained to model stable sinusoidal signals at certain frequencies:
cos(wit t + σ1) cos(w2t + (2)
(t + 8t) = We 2 (1) + beses Wot xt) =
=(t) 2 0.
(13)
We study the optimal nonnegative, efficient, recurrent representations, 2t, and show that whether the representations of the two frequencies within modularises depends on their ratio. We prove and verify empirically that one frequency is an integer multiple of the other the encodings mix, whereas if their ratio is irrational they should modularise. Further, we show empirically, and prove in limited settings, that rational non-harmonic ratios should modularise (Appendix D). The intuition for this result is much the same the linear autoencoding setting: the natural notions of sources (win))))))s(())))) (in(( Using these sources, we must simply ask: does their support allow for reduction in activity energy via mixing? In Figure Ba. we visualise the source support for the three prototypical relationships between w1 and w2: irrational, rational (but not harmonic), and harmonic. Results of neural network verifications are in Figure (details Appendix Hp. In the irrational case, the source support is essentially rectangular, so the model modularises. In the harmonic case, large chunks of various corners are missing from the source support, so the model mixes. In the rational case, even though the source support is quite sparse, the corners are sufficiently present such that modularising is still optimal.
are the signals (cos(w1t), t), sin(wi t), cos(w2t),
Modularisation of grid cells. We now show that these spectral ideas can explain modules of grid cells. Grid cells are neurons in the mammalian entorhinal cortex that fire in a hexagonal lattice of positions (Hafting et al., 2005). They come in groups, called modules; grid cells within the same module have receptive fields (firing patterns) that are translated copies of the same lattice, and different modules are defined by their different lattice (Stensola et al. 2012) (for further literature review see Appendix A). Current theories suggest that the grid cell system can be modelled as a RNN with activations built from linear combinations of frequencies (Dorrell et al. 2023), just like the linear RNNs considered here. Importantly, these grid cell theories use the same biological constraints as in this work and show that modules form because the optimal code contains con-harmonccally related frequencies that are encoded in different neurons (Figure 3 top). This modularisation can now be theoretically justified in our framework, since non-harmonic frequencies are range-independent and so should be modularised (Figure 3e bottom).
4.2 NONLINEAR RNNS
Mixed sinusoidal regression for nonlinear RNNs. To test how our ideas generalise beyond linear networks, we train nonlinear ReLU RNNs with biologically inspired constraints to perform frequency mixing task (details Appendix H). We provide pulse input Pa (t) = I [mod,. = 0] at
two frequencies, and the network has to output the resulting "beats" and "carrier" signals:
Pa, Pay (t)
[oos([w) w2 w2]()
z(t + At) = ReLU (t) + W
+ Bree)
= W out 2, + b
coos(us + w2
(14)
Results are in Figure 3. Identical range-dependence properties but applied the frequencies W1 W2 and W1 + W2 determine whether or not the network modularises: irrational, range- tanee-nndeeendent frequencies modularise; harmonics, with their large missing corners, mix; and other rationally related frequencies are range-dependent but no sufficient corner is missing, so they modularise.
Modularisation in nonlinear teacher-student distillation. To test our predictions of when RNNs modularise, but in settings more realistic than pure frequencies, we generate training data trajectories from randomly initialised teacher RNNs with tanh activation function, and then train student RNNs
7

---

Published as a conference paper at ICLR 2025
Irrational: (2pl.e)
Rational: (2. 5)
Harmonics: (2. 4) d
RNN
Depandence
Dusl Corner
Outputa Religon
Modilaitty
0.5
Teacher hidden atalle
Deal Comeer Spanslying
b
Visualization the Most Mixedd Neuron
WWW OAL
WWMM
Bational
Instonal Betional
Across
Linear
Newrrn in Miodue One
Nouncn Modul
X
Relative FFT Power of Key Frequerccies
Teasher hidden states
Visualization of Most Mixidd Niuuron
Moduie Two Largect Frequency
Frequanncy
Lineer Owgethtooonn wiok
Nonlinear
Pree' Relative Power Key Frequencles
Pree
Dual Comee Sparse
Off Diaganal Sparse
Etequancy
Modure
Source Multinformation 0.05
Figure 3: a) Source support visualisations: cos(wrt) cos(waz(). b) The activity of the most mixed neuron in an optimal solution, the same neuron's Fourier spectrum, and the population's Fourier spectrum the two task frequencies show clear modularisation the irrational and rational cases, and clear mixing the in the harmonic case. c) Same (b), but with nonlinear RNNs. d) Modularity of RNNs trained on 10 frequency pairs, e) Top: Two optimal grid cells from Dorrell et al. (2023) in different modules. Bottom: The modularisation of these two lattices be understood from the range-nneeeendence of their constituent frequencies, shown in the joint distribution the most significant frequency each module. Plot of joint distribution of hidden state activity for two teacher RNNs neurons as we increase the dependence in three different ways. g) Trends of modularity scores of student RNN in three different cases qualitatively agree with our theory.
(with ReLU activation funetion) on these trajectories. The student's representation is constrained to be nonnegative (via its ReL.U) and has its activity and weights regularised (see Appendix H.2 for details). Using carefully chosen inputs at each timestep, are able to precisely control the teacher RNN hidden state activity (i.c.. the source distribution). This allows us to change corre- lations/statistical independence of the hidden states, while either maintaining or breaking range independence. We consider three settings (Figure 3.. First, when statistical and range indepen- dence are progressively broken (in orange). Second, where statistical, and to a lesser extent range, independence gets progressively broken (in green). Third, where only statistical independence gets progressively broken (in purple). We observe that, in line with our theory, preserving the corner points preserves modularity (Figure 33 purple), while removing them breaks it (orange). Further, spasifying the corners breaks modularity, but more slowly than removing corners (green).
5 MODULAR OR MIXED CODES FOR SPACE & REWARD
We now apply our results to neuroscience to understand a puzzling difference in two seemingly similar recordings from entorhinal cortex. This brain area has been thought to contain precisely modular neurons, such as the grid cell code for self-position (Hafting et al.) (005), object vector cells that fire a particular displacement from objects (Høydal al. 2019), and heading direction cells (Taube) 2007). Two recent papers examined the influence of rewarded locations on the grid cell code and find differing effects on the modularity of grid cells. |Buller al. (2019) find that rewards rotate the grid cells, but preserve their pure spatial coding, while Boccara al. (2019) find that the grid cells warp towards the rewards, becoming mixed-selective to reward and space.
Whittington et al. (023) study this discrepancy and point to the importance of the reward distribution in these two tasks: Boccara et al. (2019) fix the positions of the possible rewards during one day, whereas Butler et al. (2019) alternate the animals between periods of free-foraging for randomly placed rewards and periods of specific rewarded locations. However, the arguments of Whittington et al. (2023), which rely on souree independence, effects, as in neither case are reward and position independent; even in the experiments of Butler et al.) (2019) there are regions of space that are much more likely be rewarded.
insufficient explain these modularisation
8

---

Published as a conference paper at ICLR 2025
Reward & Position Tracking
FIXED REWARD
Position
Vector Reward
LOCATIONS MIxED
Data (Boccara et al.)
Linear Conditionsl InfoM
REWARD LOCATIONS MoDULAR
16 RANDOM is FOXE REWARD
Biological RNN
Dav1
RNNNG Enw 1
Env
Intiaallation:
Actions:
Reward Position
(N, S. W)
Spatial Cells
(Butler et
Spatial
b Reward Distributions
1) Random
Cells
Mixed Spatial/Obyect
Object Calla
2) Reward fixed locations
Data (Haylul et
Cells
RNN
Seritaaiytyt obachhewrddeee alls
0.6
Probebility that Rewards are at the Two Fixed Locations
Figure 4: a, b) We train linear RNNs to integrate a sequence of actions to report their current position and displacement from reward. In some proportion of the rooms, rewards are randomly scattered; in the rest, they are in one of two fixed locations. c) Matching the theory, if some of the rooms random then reward and position are range-independent, and their representation modularises, as in (Stensola al. (2012) Butler (2019); all rooms the rewards occur one of two fixed locations the representation mixes, with neurons encoding both reward and self position, as in Boccara et al. (2019)
However, with our improved understanding of modularisation, this makes sense. Despite Butler et al. (2019) correlating reward and position, critically, they leave them range independent combinations reward and position are possible. On the other hand, Boccara et al. (2019) make certain combinations of reward and position impossible, making them not just correlated, but range correlated. As such, our theory matches this modularisation pattern. To test this we train linear RNN with biological constraints (details Appendix H.3) report both its self- self-position and displacement from a reward as it moves around an environment (Figure We train RNNs on settings with different relationships between reward and position; for some RNNs the rewards are in fixed positions in every environment (range dependent and statistically dependent), for others reward and position are uniformly random in each room (range independent and statistically independent), and a final group experiences both settings in different proportions (range independent but statistically dependent). As we vary the proportion of fixed rooms, we find the optimal representation stays modular, containing separate spatial and Sewrrd-vecoo cells as in Butler et al. (2019). until all the sampled rooms have fixed reward positions, at which point neurons become mixed selective reward and self- positioos as in Boccara et al. (2019). As such, our theory now covers all known grid cell modularity results.
Missed sources and mixed selectivity. contrast to the beautifully modular neurons in entorhinal cortex, recent work has highlighted neurons with mixed tuning to multiple navigational variables, such as position, heading direction, or speed (Hardcastlc et al. a 2017). We use our theory to highlight a potential artifact of this type of analysis. A purportedly mixed selective neuron may (in part) be purely selective for another unanalysed variable that itself correlated with the measured spatial variables or behaviour. We highlight simple example of this effect in Figure Imagine mouse is keeping track of three objects as it moves in an environment (Figure 5p) we model this as a linear RNN with biological constraints that must report the displacement of all objects from itself. We make the object positions range independent but correlated. As predicted by our theory, the RNN modularises these range-indegg ingeeindependent variables such that each neuron only encodes one object (Figure 5p.). However, an experimenter were only aware of two of the three objects, they would instead analyse the neural tuning with respect to those two objects, without being able condition on the third. Due to the statistical dependencies between object positions, they would find mixed selective neurons that are in reality purely coding for the missing object (Figure 5t). We do not believe that all neurons in entorhinal cortex are modular; for example, there is clear evidence that neurons co-tuned to position and velocity are meaningfully mixed-selective. implementing the path-integration updates in the grid cell system (Vollan al.) 2025). Indeed, many of the neurons analysed in Hardcastle et al.) (2017) likely are mixed selective. However, is also seems likely that more exploratory analyses such as these, mixed selectivity might often arise from missing encoded variables in the analysis.

---

Published as a conference paper at ICLR 2025
Missing Correlated Variables + Mixed Selectivity
Object Distributions: Half Random
b Object Tracking
C
Modular Object Vector Code: Object-Nouron Nouron Linear NCMI
Vector each object from current position
Fonangs Pippsie Phoness
Half Structured
d
Looks miked selective ignoring object
Biologicai RNN
At inibalisation:
Actions: IN, W)
Object Positions
Figure 5: a, b) train linear RNNs to report displacement three objects as an agent moves within many romms. In each room, object instances are either completely random (a, top) or clustered in random line (a, bottom); object positions are therefore range independent but correlated. c) The neurons are modular: each neuron's activity is conditionally informative of only single object given the others. d) However, if an experimenter were only aware of two of the three objects, the neurons that purely encode the disregarded object would appear mixed selective due to the statistical dependencies between objects.
6 DISCUSSION
We present precise constraints source distributions that cause lincar feedforward and recurrent networks with biological constraints to modularise. These constraints depend on the co-range properties of variables, matching patterns of modularisation in both nonlinear networks and the brain.
Limitations future work. That said, there remain many aspects our theory that need improve- ment. Deriving necessary (rather than just sufficient) modularisation conditions for other norms and of multidimensional sources, extending to nonlinear settings, or to understand a less binarised notion of modularity (rather than perfectly modular not), developing theory of linearly mixed sources as in matrix factorsiation work, are all attractive directions. Further, we assume perfect reconstruction. This leads to some of our suprising conclusions, like how the support of the distribution that matters, even if some parts of the support have very low probability. In Appendix C we show that allowing the representation tradeoff reconstruction accuracy with the other losses leads it to ignore very low probability points. Studying this tradeoff further would be interesting. Biologically. beyond the clear need for experimental validation, there are many pertinent effects whose impact on modularity would be interesting to study, such as connection sparsity, anatomical constraints, noise, or requiring additional computational roles from the network.
Mixed selectivity. In neuroscience our findings add nuance to the ongoing debate over mixed vs. modular neural coding. Thcorics of nonlinear mixed selectivity have argued that, analogously to a nonlinear kernel, such schemes permit linear readouts to decode nonlinear functions of the sources (Rigotti et al.) 2013). This is likely key part of the mixed selectivity found some brain areas, like the cerebellum (Lanore et al. (02T). mushroom body (Aso al. (014)) and perhaps certain prefrontal or hippocampal representations (Bernardi er al. 2020) [oyle et al. 2024). However, our theory raises the possibility for other explanations of both nonlinear and linear mixed selectivity that do not require tasks with nonlinear functions of the sources. First, purely from energy efficiency, our work analytically shows when dependent variables should be encoded in linearly mixed- selective representations, and empirically shows the same ideas apply to nonlinear mixed-selective representations, Figure 2 Further, since in our theory even statistically dependent variables might be represented modularly, we show that this causes problems for analysis techniques that do not account for all the variables encoded in a population, leading to spurious mixed selectivity, Figure Future work could usefully develop principled approaches to distinguishing these options.
Disentangling what? We agree with Roth et (2023), who argue that range independence the more meaningful form of independence for modularising variables. Further, the similarity between our conditions and those from the matrix factorisation literature (c.g. (Tatli and Erdogan 2221aa b)), point repeated instances of similar ideas. It is intriguing that constraints inspired by biology, which hold promise for disentangling (Whittington et al. 2023), and have links to ofate-of-thh-ar disentangling methods (Hsu et al.
2024), sensitive to precisely this form of independence.
Reproducibility Statement Our theoretical work is fully detailed in the appendices, and code to reproduce empirical work can be found
10

---

Published as a conference paper at ICLR 2025
Acknowledgements The authors thank Basile Confavreux, Pierre Glaser, & Bariscan Bozkurt for conversations about proof techniques.
We thank the following funding sources: Gatsby Charitable Foundation to W.D., J.H.L. & P.E.L.; Sequoia Capital Stanford Graduate Fellowship and NSERC Postgraduate Scholarship Doctoral to K.H.; Wellcome Trust (1 101 14/2/15/2) P.E.L.;; Wellcome Trust (219627/2/19/2) J.H.L.; Sir Henry Wellcome Postdoctoral Fellowship (222817/Z/21/22 to JCRW; Wellcome Principal Research Fellowship (219525/2/1//2), Wellcome Collaborator award (214314/2/187Z), and Jean-François and Marie-Laure de Neermont-Tonnentt Foundation award (ISMF220020372) to TEJB.; the Well- come Centre for Integrative Neuroimaging is supported by core funding from the Wellcome Trust (203////
REFERENCES
Yoshinori Aso, Daisuke Hattori, Yang Yu, Rebecca M Johnston, Nirmala A Iyer, Teri-BB Ngo, Heather Dionne, LF Abbott, Richard Axel, Hiromu Tanimoto, al. The neuronal architecture of the mushroom body provides logic for associative learning. elife, 3:e04577, 2014.
Horace B Barlow et al. Possible principles underlying the transformation of sensory messages. Sensory communication, 1(01), 1961.
Anthony J Bell and Terrence J Sejnowski. An information-mmxximizzion approach to blind separation and blind deconvolution. Neural computation, 7(6): 129:1122-1 1995.
Silvia Bernardi, Mareus Benna, Mattia Rigotti, Jérôme Munuera, Stefano Fusi, and Daniel Salzman. The geometry of abstraction in the hippocampus and prefrontal cortex. Cell, 183(4): 954-967, 2020.
Charlotte Boccara, Michele Nardin, Federico Stella, Joseph O'Neill, and Jozsef Csicsvari. The entorhinal cognitive map is attracted goals. Science, 363(6434); 1447, 2019.
Lara M Boyle, Lorenzo Posani, Sarah Irfan, Steven A Siegelbaum, and Stefano Fusi. Tuned geometries of hippocampal representations meet the computational demands of social memory. Neuron, 2024.
Bariscan Bozkurt, Cengiz Pehlevan, and Alper Erdogan. Biologically-plausibbe determinant maxi- mization neural networks for blind separation of correlated sources. Advances in Neural Informa- fion Processing Systems, 35: 15:13---33777, 2022.
Yoram Burak and lla R Fiete. Accurate path integration in continuous attractor network models of grid cells. PLoS computational biology, 5(2)xe000029, 2009.
William N Butler, Kiah Hardcastle, and Lisa M Giocomo. Remembered reward locations restructure entorhinal spatial maps. Science, 3cience,363(6434//)) (6444444))44444441422 2019.
William Dorrell, Peter E Latham, Timothy EJ Behrens, and James CR Whittington. Actionable neural representations: Grid cells from minimal constraints. In The Eleventh International Conference on Learning Representations, 2023.
Laura N Driscoll, Krishna Shenoy, and David Sussillo. Flexible multitask computation in recurrent networks utilizes shared dynamical motifs. Nature Neuroscience, 27(7):1349-133) 2024.
Alexis Dubreuil, Adrian Valente, Manuel Beiran, Francesca Mastrogiuseppe, and Srdjan Ostojic. The role of population structure in computations through neural dynamics. Nature neuroscience, 25(6): 783-794, 2022.
Lea Duncker, Laura Driscoll, Krishna V Shenoy, Maneesh Sahani, and David Sussillo. Organizing recurrent network dynamics by task-computation to enable continual learning. Advances in neural information processing systems, 33:1 1387-143977 2020.
Mhairi Dunion, Trevor McInroe, Kevin Sebastian Luck, Josiah Hanna, and Stefano Albrecht. Condi- tional mutual information for disentangled representations in reinforcement learning. Advances in Neural Information Processing Systems, 36, 2023.
11

---

Published as a conference paper at ICLR 2025
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. arXiv preprint arXiv:2209.1063 2022.
Torkel Hafting, Marianne Fyhn, Sturla Molden, May-Britt Moser, and Edvard Moser. Microstructure of spatial map in the entorhinal cortex. Nature, 436(7052):801-886 2005.
Kiah Hardcastle, Niru Maheswaranathan, Surya Ganguli, and Lisa M Giocomo. A multiplexed, heterogeneous, and adaptive code for navigation in medial entorhinal cortex. Neuron, 94(2): 375-387, 2017.
Daniella Horan, Eitan Richardson, and Yair Weiss. When unsupervised disentanglement possible? Advances in Neural Information Processing Systems, 34:5150-5161, 2021.
Kyle Hsu, William Dorrell, James Whittington, Jiajun Wu, and Chelsea Finn. Disentanglement via latent quantization. Advances in Neural Information Processing Systems, 36, 2023.
Kyle Hsu, Jubayer Ibn Hamid, Kaylee Burns, Chelsea Finn, and Jiajun Wu. Tripod: Three comple-
mentary inductive biases for disentangled representation learning, In International Conference on Machine Learning, 2024.
David H Hubel and Torsten N Wiesel. Receptive fields, binocular interaction and functional architec- ture in the cat's visual cortex. The Journal of physiology, 160(1):106, 1962.
Øyvind Arne Høydal, Emilie Ranheim Skytøen, Sebastian Ola Andersson, May-Britt Moser, and Edvard I. Moser. Object-vector coding in the medial entorhinal cortex. Nature, 568(7755):400-40, April 2019. ISSN 1476-4687. doi: 01008/441886-01-1077779 URL http: doi org/ 10.
Huseyin Inan and Alper T Erdogan. An extended family of bounded component analysis algorithms. In 2014 48th Asilomar Conference on Signals, Systems and Computers, pages 442-445. IEEE, 2014.
Devon Jarvis, Richard Klein, Benjamin Rosman, and Andrew Saxe. On the specialization of neural modules. In The Eleventh International Conference on Learning Representations, 2023.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014. URL https:://rxxv.cer wwwp:xwwwwoorroo.ooooaooooo abs//412/1111 6980
David Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan Paiton. Towards nonlinear disentanglement in natural data with temporal sparse coding. arXiv preprint arXiv:2007.10330 2020.
Frederic Lanore, N Alex Cayco-Gajic, Harsha Gurnani, Diccon Coyle, and R Angus Silver. Cerebellar granule cell axons support nigh-dimensional representations. Nature neuroscience, 24(8):1142- 1150, 2021.
Jin Hwa Lee, Stefano Sarao Mannelli, and Andrew Saxe. Why do animals need shaping? theory of task composition and curriculum learning. arXiv preprint arXiv:2402.18361 2024.
Alexander Mathis, Andreas VM Herz, and Martin Stemmler. Optimal population codes for space: grid cells outperform place cells. Neural computation, 24(9))2280-23)) 231,, 2012.
Aran Nayebi, Alexander Attinger, Malcolm Campbell, Kiah Hardcastle, Isabel Low, Caitlin S Mallory, Gabriel Mel, Ben Sorscher, Alex H Williams, Surya Ganguli, et al. Explaining heterogeneity in medial entorhinal cortex with task-driven neural networks. Advances in Neural Information Processing Systems, 34: 14:12167-1217, 2021.
Weili Nie. High resolution disentanglement datasets, 2019. URL https://github.cm AVLabs/High-res disett-iiinttat anggeeeeeeeeeetttteeeeeeeeetteee datasets
Cengiz Pehlevan, Sreyas Mohan, and Dmitri Chklovskii. Blind nonnegative source separation using biological neural networks, Neural computation, 29(1 (1111)):2559999 2017.
12

---

Published as a conference paper at ICLR 2025
Mark D Plumbley. Algorithms for nonnegative independent component analysis. IEEE Transactions on Neural Networks, 14(3):534-33 54035333,,, 2003.
Mattia Rigotti, Omri Barak, Melissa R Warden, Xiao-Jing Wang, Nathaniel D Daw, Earl K Miller, and Stefano Fusi. The importance of mixed selectivity in complex cognitive tasks. Nature, 497 (7451):585-590, 2013.
Brian C Ross. Mutual information between discrete and continuous data sets. PloS one, 9()xe87357, 2014.
Karsten Roth, Mark Ibrahim, Zeynep Akata, Pascal Vincent, and Diane Bouchacourt. Disentan- glement of correlated factors via hausdorff factorized support. In The Eleventh International Conference on Learning Representations, 2023.
Axel Ruhe. Perturbation bounds for means eigenvalues and invariant subspaces. BIT Numerical Mathematics, 10(3):343-35,, 1970.
Andrew Saxe, Shagun Sodhani, and Sam Jay Lewallen. The neural race reduction: Dynamies of abstraction in gated networks. In International Conference on Machine Learning, pages 19287- 19309. PMLR, 2022.
Simon Schug, Seijin Kobayashi, Yassir Akram, Maciej Wolezyk, Alexandra Maria Proca, Johannes Von Oswald, Razvan Pascanu, Joao Sacramento, and Angelika Steger. Discovering modular solutions that generalize compositionally. In The Twelfth International Conference on Learning Representations, 2024.
Jianghong Shi, Eric Shea- Brown, and Michael Buice. Learning dynamics of deep linear networks with multiple pathways. Advances in neural information processing systems, 35:34064-344 34076, 2022.
Daniel Soudry and Dror Speiser. Maximal minimum for sum of two (or more) cosines, 2015,
URL hatpss:/mmsthovvrrllo..... htt.a//aaarrrrtoooooooooootttttr /ussstionnnn/////11 maximal ott.:uaw-..............oooooooo-
Hanne Stensola, Tor Stensola, Trygve Solstad, Kristian Frøland, May-Britt Moser, and Edvard I Moser. The entorhinal grid map is discretized. Nature, 492(7427):72-78, 2012.
Gokcan Tatli and Alper T Erdogan. Generalized polytopic matrix factorization. In ICASSP 2021- 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3235-3239. IEEE, 2021a.
Gokcan Tatli and Alper T Erdogan. Polytopic matrix factorization: Determinant maximization based criterion and identifiability. IEEE Transactions Signal Processing, 69:5431-544 2021b.
Jeffrey Taube. The head direction signal: origins and sensory-motor integration. Annu. Rev. Neurosci, 30(1):181-20) 2007.
Kay M Tye, Earl K Miller, Felix H Taschbach, Marcus K Benna, Mattia Rigotti, and Stefano Fusi. Mixed selectivity: Cellular computations for complexity. Neuron, 2024.
Abraham Z Vollan, Richard J Gardner, May-Britt Moser, and Edvard Moser. Left deft-right-altenne theta sweeps in entorhinal-hippooampaa maps of space. Nature, pages 1-11, 2025.
Xue-Xin Wei, Jason Prentice, and Vijay Balasubramanian. A principle of economy predicts the
functional architecture of grid cells. Elife, 4:e08362, 2015.
James CR Whittington, Timothy H Muller, Shirley Mark, Guifen Chen, Caswell Barry, Neil Burgess, and Timothy EJ Behrens. The tolman-eichenbaum machine: unifying space and relational memory through generalization in the hippocampal formation. Cell. 18495))1249-1263, 2020.
James CR Whittington, Will Dorrell, Surya Ganguli, and Timothy Behrens. Disentanglement with biological constraints: A theory of functional cell types. In The Eleventh International Conference Learning Representations, 2023.
13

---

Published as a conference paper at ICLR 2025
James CR Whittington, William Dorrell, Timothy EJ Behrens, Surya Ganguli, and Mohamady El- Gaby. A tale two algorithms: Structured slots explain prefrontal sequence memory and are unified with hippocampal cognitive maps. Neuron, 113(2):311-333, 2025.
Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. A theory of usable infor- mation under computational constraints. In International Conference on Learning Representations, 2020.
Guangyu Robert Yang, Madhura R Joglekar, Francis Song, William Newsome, and Xiao-Jing Wang. Task representations in neural networks trained to perform many cognitive tasks. Nature neuroscience, 22(2):297-306, 2019.
Yujia Zheng, Ignavier Ng, and Kun Zhang. On the identifiability of nonlinear ica: Sparsity and beyond. Advances in Neural Information Processing Systems, 11166441116222 2022.
14

---

Published as a conference paper at ICLR 2025
A RELATED WORKS
Machine learning. The disentangled representation learning community has found inductive biases that lead networks to prefer modular or disentangled representations of nonlinear mixtures of inde- pendent variables. Examples include sparse temporal changes (Klindt et al. 2020), smoothness or sparseness assumptions in the generative model (Zheng et al. 2022] Horan et al. (02T), and axis-wise latent quantisation (Hsu et al.) 2023; (024). Relatedly, studies of tractable network models have identified a variety of structural aspects, in either the task or architecture, that lead to modularisation, including learning dynamics in gated linear networks (Saxe et al. 2022). architectural constraints in linear networks Qarvis et al. 2023] [Shi et al. 2202), compositional tasks in linear hypernetworks and nonlinear teacher-student frameworks (Schug et al. 2024 Lee et al. 2024). or task-sparsity in linear
autoencoders (Elhage et al. 2022).
Neuroscience. Theories prompted by the recognition of mixed-selectivig in biological neurons have argued that nonlinearly mixed selective codes might exist to enable a linear readout to flexibly decode any required categorisation (Rigotti al. 2013). suggesting generalisabillty--ees flexiiiiiyy tradeoff between modular and nonlinear mixed encodings (Bernardi et al., 2020). Modelling work has studied task-optimised network models of neural circuits, some of which have recovered mixed encodings (Nayebi et al. 2021). However, other models trained on a wide variety of cognitive tasks have found that networks contain meaningfully modularised components (Yang et al.) 2019 Driscoll et al.) 2024] Duncker et al. 2020). Dubreuil et al. (2022) study recurrent networks trained on similar cognitive
tasks and find that, depending on the task structure, neurons in the trained networks can be grouped into populations defined by their connectivity patterns. To this developing body our work and its predecessor (Whittington et al. 2023) add understanding of when simple biological constraints would encourage encodings of many variables to be modularised into populations of single pure-tuned variables, or when mixed-selecteee selectivityy would be preferred. Compared to (Whittington et al.) 2023) which only works when source variables are statistically independent, our work determines whether the optimal representation is modular for any dataset even when source variables are not independent. This complete understanding of the boundary between mixed and modular codes is what allows to understand behaviour in both biological and artificial neurons that previously eluded explanation.
Independent components analysis and matrix factorization. Our linear autoencoding prob- lem (Theorem 2.1 bears some similarity to linear independent components analysis (ICA) and matrix factorization (MF). These assume data y; is linearly generated from a set of constrained sources, y. = Ast and build algorithms to recover from yi- The literature studies different combinations of (1) assumptions on the sources 8,, e.g. independence (Bell and Sejnowski [995)). nonnegativity(PP) (Plumit 2003), or boundedness Inan and Erdogan (2014); (2) assumptions on the mixing matrix A, e.g. nonnegative in nonnegative matrix factorisation and (3) algorithms, often deriving identifiability results: conditions that the data must satisfy for given algorithm to succeed. Some work even derives biological implementations of such factorisation algorithms (Pehlevan et al., Bozkurt et al. 2022). In our work we study the conditions under which a particular biologically inspired representation modularises a set of bounded sources. We provide conditions on the sources under which the optimal representation is modular, morally similar to identifiability results (indeed, other identifiability results, like ours, sometimes take the form of sufficient spread conditions (Tatli and Erdogan| 2021a ())).. We differ from previous work in the representation we study (which has only been studied before by Whittington et al. (2023)), the form of our identifiability result (which is both necessary and sufficient, and has not been derived before), and how we apply these ideas, for example to neural data.
Grid Cell Modules One of the many surprising features of grid cells is their modular structure: each grid cell is member of one module, of which there are a small number in rats. Grid cells from the same module have receptive fields that are translated versions of one another (Stensola et al. 2012). Previous work has built circuit models of such modules showing how they might path-integrate (Burak and Fiete, 2009). or has assumed the existence of multiple modules, then shown that they form good code for space, and that parameter choices such as the module lengthscale ratio can be extracted from optimal coding arguments (Mathis et al. 2012 Wei et al. 2015). However, neither of these directions shows why, of all the ways to code space, multi modular structure is best. Dorrell let al. (2023) study a normative problem that explains the emergence of multiple modules grid cells as the best of all possible codes, but their arguments for the emergence of multiple grid modules relatively intuitive. In this work we are able to formalise parts of their argument.
15

---

Published as a conference paper at ICLR 2025
B POSITIVE LINEAR AUTOENCODER THEORY
In this section we derive our main mathematical results. We prove the main theorem, Appendix B.1 the equivalent statement terms of the convex hull of the data, Appendix [B.2] and the corollary on range-independent variables Appendix B.3 We then generalise this range-independen result to all activity norms, Appendix B.4. and to multidimensional variables, Appendix B.5
B.1 FULL THEORETICAL TREATMENT
Our proof of the main result, Theorem 2.1 takes the following strategy. First, we show that for a fixed encoding size, the weight loss is always minimised by orthogonalising, example of which modularising. Then we study when the activity loss is also minimised by modularising, and use that tell us about the whole loss. Recall Theorem 2.
Theorem B.1 (Main Result). Let E TRd,, TRd., Wm RRd, bm E RRd, Wout Rd.xd, and bott E TRd,, with d₂ > dz. Consider the constrained optimization problem
(1120)11)) + > (1I out I|? + +1|Winll?F)
min W Win. W Wout
(15)
2 Win + bin: 8 = W 2l + bout; 2 2200 0.
s.t.
where i indexes samples of At the minima of this problem, each of Wm has at most one non-zero entry, iee. the representation modularises, iff the following inequality is satisfied for all w E Rd
[ijnww' T 2l01)² +
Σ
WjWy
Wi min =)
(16)
where 8 == {sli)); and assuming that min; s/0 max; ofJ Vj [d.] w.l.o.g.
B.1.1 WEIGHT LOSSES ARE MINIMISED MODULARISING
First, we show lemma: that, a fixed encoding size, the weight loss is minimised by orthogonalis- ing the encoding of each source. Writing the representation as affine function of the de-meanee sources:
d
U [1]
b
Σ
14 ug
(t b'
b'
(17)
then by fixed encoding size mean that each |uyl is fixed. We express things in terms of mean-zero variables 5/ since it is easier and we can freely shift by constant offset.
Theorem B.2 (Modularising Minimises Weight Loss). The weight loss, W| 1 + minimised, for fixed encoding magnitudes, |uz]. by modularising the representation.
W 11
Proof. First, we know that Win must of the following form:
Win
[u1
ud.]
(18)
And therefore for a fixed encoding size the input weight loss is constant:
W 1
Tr W
Σ
(19)
So let's study the output weights, these are defined by the way they map the representation back to the sources:
W out 2 bout
(20)
16

---

Published as a conference paper at ICLR 2025
The min-norm Weut with this property is the Moore-Penross, Pseudoinverse, i.c. the matrix:
W out
(21)
Where each pseudoinverse U; is defined by U u, 6ig and lying completely within the span of the encoding vectors {uz}. J. We can calculate the norm of this matrix:
d Σ U,
W outl F W W
(22)
Now, each of these capitalised pseudoinverse vectors must have some component along its corre- sponding lower case vector, and some component orthogonal to that:
U
U1 2 U + د
(23)
We' ve chosen the size of the component along u, such that UT uj = 1, and the u, chosen so that U ug = dzk- Now, for a fixed size of |uzl this sets a lower bound on the size of the weight matrix:
W out
2
(24)
U 2
dd And this lower bound is achieved whenever the {u, j==1 vectors are orthogonal to one another, since then u O. Therefore, we see that, for fixed size of encoding, the weight loss minimised when the encoding vectors are orthogonal, and that is achieved when the code is modular.
L
B.1.2 ACTIVITY Loss
Now we will turn studying the activity loss, and find conditions under which a modular representa- tion is better than mixed one. We compare two representations, mixed neuron:
Unn
3ng3)
min
Ung
(25)
And another representation in which we break this neuron apart into its modular form, preserving the encoding size of each source:
[
[un min, sf1
Un 181
(26)
72
Und,
Und, miny
The activity loss difference between the modular and mixed representations is:
Σ 2 (min 5// (min
1212
11 11 Uni Unk (8)
8k
(27)
j=1
So the modular loss is smaller if, for all un E I
(min
Uny
ung (min 12
Σ J.k
Uny Uni (s/)
(28)
Further, we can see that if this equation holds for one vector, u, it also holds for any sealed version of that vector, therefore equivalently require the inequality to be true only for unit norm vectors.
17

---

Published as a conference paper at ICLR 2025
B.1.3 COMBINATION OF WEIGHT AND ACTIVITY
So, if these inequalities are satisfied you can always decrease the loss by modularising mixed neuron. Therefore, both the weight and activity loss are minimised at fixed encoding size by modularising, and this holds for all encoding sizes, therefore the optimal solution must be modular. Hence these conditions are sufficient for modular solutions to be optimal.
Conversely, if one of the inequalities is broken then the activity loss is not minimised at the modular solution. We can move towards the optimal mixed solution by creating new mixed encoding neuron with infinitesimally low activity, and mixed in proportions given by the broken inequality. This neuron can be created by taking small pieces of the modular encoding of each source, in proportion to the mixing. Doing this will decrease the activity loss. Further, the weight loss is minimised the modular solution, so, to first order, moving away from modular solution doesn't It change it. Therefore, for each modular representation, we have found a mixed solution that is better, so the optimal solution not modular.
18

---

Published as a conference paper at ICLR 2025
B.2 EQUIVALENCE OF CONVEX HULL FORMULATION
In this section we prove Theorem 2.2 an equivalent formulation of the infinitely many inequalities in Theorem B. in terms of the convex hull, restated below:
Theorem B.3. In the setting as Theorem B. define the following matrix:
(min,or sj [6
{sq 4 -1 83
)
(min;
)²
- (82
F=
83
[il
(82 83
[4] 11
)
(min: 83
(29)
Define the set E {y yyF = 1}. Then an equivalent statement of the modularisation inequalities (16), is the representation modularises iff E lies within the convex hull of the datapoints.
Notice that the matrix ellipse. In these cases no convex hull can contain the unbounded quadric it forms, and the optimal solution is mixed regardless the particular distribution the datapoints.
not guaranteed to positive definite, meaning E is not necessarily an
We'll begin by rewriting the inequalities, then we'll show each direction of the equivalence.
B.2.1 REWRITE INEQUALITY CONSTRAINTS
The current form the inequality is that, Vi E R
d,
(min w" 1
my (1) 2 Σ (min (min 2/u
(30)
id'
If this equation holds for one vector, w. also holds for all positively scaled versions of the vector, Aw for À > 0, we can equivalently state that this condition must only hold for unit vectors.
Construct the matrix F above (29). then this condition can be rewritten:
(min w "gllll22 w Fw Vw E R", w 1
(31)
Further, notice that we can swap the minima in the inequality for a maxima. We can do this because the inequality is satisfied for particular value of then the following inequality is satisfied for ww:
(max wT 801)22 2 w" T Fwn Yee
IR."" |wl
(32)
Since (31 is satisfied for all w, and all w have their ww unit vector pairs, so is (32). Similarly, satisfied for all unit vectors w. (3 is satisfied but using ww. Hence the two are equivalent.
32
B.2.2 SUFFICIENCY OF CONDITION
First, let's show the convex hull enclosing E is sufficient condition. Choose an arbitrary unit vector w. Choose the point within E that maximises the dot product with w:
yw arg max w y
(33)
WEE
We can find y by lagrange optimisation under the constraint that y F-1
Fw
Yn
(34)
Vwo" Fw
By the definition of E and its enclosure in the convex hull of the points we can write:
Yw Σ Aisl 4 0, M AS
(35)
Now:
w y
V TH Fw
Σ NowT gl4l < max :T [(
Σ A: max w [
(36)
Therefore, get main result:
(max Th 8 1/ 2 nh Fw
(37)
19

---

Published as a conference paper at ICLR 2025
Ellipse E
Maximising Point on E, y
Polyhedron
X
Vector Bj
Convex Hull
Datapoints, z
Figure 6: If the convex hull does not enclose the ellipse then you can always find unit vector that breaks the inequality. In particular, this is true for the unit vector that defines the faces of the convex hull, Bj.
B.2.3 NECESSITY OF CONDITION
Now we will show the necessity of this condition. We will do this by showing that if one of the points in E is within the convex hull of the datapoints then we can find of the inequalities in (32 that is broken.
First, get all the datapoints that for some unit vector w maximise w" g over the dataset. Construct the convex hull of these points, the polyhedron P. You can equivalently characterise any polyhedra by set of linear inequalities: can always choose to write the inequalities in such a way that the rows of B are unit length, by appropriately rescaling each by. Assume we have done this.
(a R Br S b). Denote with B, the jth row of B. We
Assume the convex hull does not contain the entire ellipse. Therefore there is at least one point on the ellipse that breaks one of the inequalities defining the polyhedron. These points are charactersied by being both the ellipse, y² F---F- y but least one of the incqualities defining the polyhedron broken: B y bj. Let's call the set of points on the ellipse for which this incquality is broken B, = f E R" FFF= A.FFly
and BJ v bi}.
Now, choose the element of 3y that maximises BJ: y. call it y" This will also be the element of E that maximises B3 y, which by the previous lagrange optimisation, (34), has the following dot product: v = BT
But now we derive a contradiction. This point is outside the polyhedron, BJy > bj. whereas all points inside the polyhedron, including all the datapoints, satisfy all incqualities, hence, BT s < Vsl But, By y = B FB; > by. therefore:
B s bb < B 4s
(38)
And hence we've found unit vector B, such that:
max BJ 1 < FBB
-max Bfs gliy2 BJ FB;
(39)
Hence one of the inequalities in
32 broken, showing the necessity.
20

---

Published as a conference paper at ICLR 2025
B.3 RANGE-INDEPENDEN VARIABLES
We will now prove the corollary on range dependence, Theorem 2.3 recapped here: Corollary B.4. the same setting as Theorem B. the optimal representation modularises if all sources are pairwise extreme-point independent, i.e. if
min
[4]
[d] 3' E maxx [9] 3 min Km 3'
{
min 14
(40)
for all j. [d,]2.
Proof. If the variables are extreme-point independent then we can take the min inside the sum in the definition of the mixed neuron bias:
dg
2n
Unyss
An
Uey
min
Ung
Unjz]
min(uns)!!1 (41)
j=1
Now can compute the activity loss:
((=1)))))
Unj 5 min (nj(nn)s((1)
(unss]] min Jenn sft) (un
+
min Uny
)
(42)
The cross terms at the end are strictly positive, and each the squared terms on the left is the same or larger than its corresponding modular neuron, since either Uhyy is positive and it is the same, or Unj is negative and is greater than or equal to the equivalent modular neuron. Therefore the mixed energy cost is larger than the modular energy cost for satrem--poin independent variables, so the optimal representation should modularise.
21

---

Published as a conference paper at ICLR 2025
B.4 EXTENSION TO ALL ACTIVITY NORMS
We now extend our theoretical results to other activity norms beyond L2. As shown in theorem Theo- rem B.2 the weight loss is minimised by orthogonalising the coding of different variables, can focus on the activity loss. We will show that, for all norms, modular solutions are optimal if the variables are range independent.
Consider set of modular neurons:
Mnn [6]] min, o(())
2 0
(43)
Mndd [(3]]
min,
And let's see if we could usefully mix these encodings together for any set of mixing coefficients Up E IR
de Uny
d.
M
Uny
8
min
Uny
(44)
the variables are range-independent then we can move the min inside the sum:
An
Unj 1
min 33 [1]
(45)
==1
Then the Lp norm is:
(12
d min Uny 5/ lp)
P) = (1
Uns8 10
+ An
12)s 1p/=((
Σ
(46)
Now using the convexity of the Lp norm:
Σ
min uny 12); 2 Σ
(lunj5
s/f
min uny w
(47)
:
j=1
with equality only p is equal to I, most one of the entries of u are O.
Finally, since we orient our variables such that min; | < max; g(i without loss of generality::
(Mags s/ min Uny 5 (p) (Iumg 12 (15) min If) PP):
(48)
So get our final result that the mixed activity loss is greater than equal to the modular activity loss if the variables are range independent:
(1
|ung P(s) min s PPx (1=)1p))
(49)
1=1
So. long > 1, the activity loss is lowered by modularising the representation of range- independent variables. Further, since the weight loss is also minimised, the optimal solution will be modular.
Two caveats: first, in the particular case of the L1 norm all that we can guarantee that the activity loss of modular solution is equal to that of mixed solution. This means that, even in the case of range-independent variables, there might be orthogonal but mixed solutions that are equally good as the modular solution. For p > 1 this caveat is not needed.
Second, this is a set of sufficient conditions, range-independent variables are optimally encoded modularly. There will be other nearly range-independent variables that are also optimally encoded modularly, just as we found for the L2 case.
22

---

Published as a conference paper at ICLR 2025
B.5 EXTENSION TO MULTIDIMENSIONAL VARIABLES
Many variables in the real world, such as angles on a circle, are not 1-dimensional. Ideally our theory would tell when such multi-dimensional sources were modularised. Unfortunately, precise necessary and sufficient conditions are not an easy step from our current theory because our proof strategies make extensive use of the optimal modular encoding. For mieddmensional sources this is very casy to find, simply align each source correctly in neuron, and make the representation positive. Finding similar optimal modular solutions for multidimensional sources is much harder, potentially as hard as simply solving the optimisation problem.
However, we can make progress on simpler problem: showing that sune-independence is sufficient condition for set of multi-dimennional sources to modularise. Consider two multidimensional sources 2114 RR and y RRy, and an encoding which they are mixed in single neurons:
JA Ai
Unj
[ Unj
An
(50)
Whatever choice of coefficient vectors, and v, activity norm (as long as p 1), we consider breaking this mixed coding into two modular neurons. Define new variables alt Ej= Unyz)
[6] and b1 = دe= Vnzyl If the two multidimensional sources are range-independent, then so too are a and Define the modular representation:
[att] min, abl 6lt)
[
(51)
miny b
All of our previous arguments apply to argue that this modular encoding will be preferred the multidimensional variables are range- inggpeeddee...tt..
We just have show that the weight loss is also reduced by modularising multidimensional sources, small extension of Theorem B.2 which we shall now show. Write a modular code as:
Uxl min[U=ll]] + Vy min | [
(52)
5
With the encoding of æ and spanning two different spaces: UTV = O.
Then define the minimal L2 norm readout weight matrix using the pseudoinverse:
T
W out
(53)
V
[
up some constant offset that can be removed by the bias term in the readout. Now consider mixing the representation of the two multidimensional sources using an orthogonal matrix that preserves the encoding size of each variable, O:
Such that W Wourlli
U A VIOT + B
z Uz! + OVyl
W out W out
(54)
Then in the subspace spanned by the columns of U W must be the same as Ut. However, in order to offset the newly aligned matrix OV there must be an additional component orthogonal, A:
W U = U + AU =I AU =0 OV WV..V- utov + AOV 0 AOV = -UtOV
(55)
W
So A and U live orthogonal subspaces, and A + The same applies to V and B, therefore:
W out ||22 || 2/2
||V 1| || + [B > || +
(56)
And hence orthogonalising the subspaces reduces the loss in this case well.
Combining these two, the optimal representation range-independent cilti-dimensional variables for activity losses using an Lp norm for p is modular.
23

---

Published as a conference paper at ICLR 2025
C IMPERFECT RECONSTRUCTION
In our results we assume perfect reconstruction. This is not the real situation in either biology or neural networks. If dataset is sange-independent, but only when considering a very unlikely point, then in reality that point might be dropped, incurring a small reconstruction cost, in order to save firing and weight energy.
We illustrate this in the following example. Our dataset comprises four points from the corners of square the dataset is range independent. However, we sample these points very nonuniformly, one with probability A 0.01, the rest with probability A - Figure 7 If the penalty on reconstruction error high enough during numerical optimisation, all four points are perfectly reconstructed, this dataset is range independent, and the representation modularises Figure 7B. If the reconstruction loss is made less important the representation changes dramatically, and mixes Figure?? 7
A) Data
Perfect Reconstruction + Modularity
C) Errors Mixing
p
3
Neurn
Nouron
s4imo
Ex/ee
1-4
Neuron Weights Ace Algined
Ayst 10?
Neuran Weights Hbad
Avea
Figure 7: A) We sample a dataset four points from the corners of a square, one with probability A, the rest with probability 1-2 B) If the penalty on reconstruction error is high enough, the data fits perfectly, and the representation modular. C) If the penalty lowered, it is preferred incur reconstruction cost, and mix the representation.
This is an interesting and very relevant effect, that future work could usefully characterise.
24

---

Published as a conference paper at ICLR 2025
D POSITIVE LINEAR RECURRENT NEURAL NETWORKS
One of the big advantages of changing the phrasing of the modularising constraints from statistical properties (as in Whittington et al. ) to range properties is that naturally generalises to recurrent dynamical formulations, and recurrent networks are often much more natural setting for neuroscience. illustrate this we'll study positive linear RNNs. Linear dynamical systems can only produce mixtures of decaying or growing sinusoids, so we therefore ask the RNN to produce different frequency outputs via an affine readout:
(os(w)))) cos(w22)
W (1) + bont =
(57)
Then we'll assume the internal dynamical structure is a standard linear RNN:
W - b = g(t + At)
(58)
We will study this two frequency setting and ask when the representation learns to modularise the two frequencies. These results could be generalised to multiple variables as in section B.5 Again, our representation must be non -eggatit,,,, g() 2 O, and we minimise the following energy loss:
L = (11((()) Xw W rec Pp + AR. W out
(59)
We will show that, if Aw is sufficiently large and the RNN solves the task for infinite time, the optimal representation contains activity cycling at only the two frequencies, wi and w2. Further we'll show that these two parts should modularise if the two frequencies are irrational ratios of one another, and should mix if one frequency is an integer multiple of the other, i.c. the frequencies are harmonics. Finally, we conjecture, show empircally, and present proof in limited cases, that non-harmonic rational frequency ratios should modularise. If the task runs for finite time then smudge factor is allowed, the frequencies can be near integer multiples, and how close you have to be to mix is governed by the length of the task. As long as during the task the frequencies are "effectively" integer multiples, the representation will mix.
To show this we will first study the weight losses, show that they are minimised if each frequency is modularised from the others. We will then argue that when dy is sufficiently large the representation will contain only two frequencies. Finally, we will study when the activity loss is minimised. Then, as in Appendix B when all losses agree on their minima, the representation modularises, else it mixes.
D.1 STRUCTURE OF REPRESENTATION ACT
Due to the autonomous linear system architecture, the representation can only contain a linear mixture of growing, decaying, stable frequencies, and constant. The growing and decaying modes cannot help you solve the task (since if they do at one point in time, they later won't 't), and will either cost infinite energy (the growing modes) will cost some energy without helping to reduce the bias (since, again, the bias has to be set large enough such that after the decaying mode has disappeared the representation is still positive). Hence.
(t)) = Σ a, cos(wit) + b, sin(wat) + by
(60)
Where F is smaller than half the number of neurons to ensure the autonomous linear system can propagate each frequency (each frequency "uses" 2 dimensions, as will be obvious from Appendix
D.2 READOUT LOss
The readout loss is relatively easy, again create some capitalised pseudoinverse vectors (A, B detined by being the min-norm vectors with the property that A aj dig and A b, 0. and the same for B, Then the min-norm readout matrix is:
AT
W out
(61)
A7'
25

---

Published as a conference paper at ICLR 2025
And the readout loss is:
W out
= Tr(W out out |A + |A2]2
(62)
+ dT where T: is orthogonal to and is included ensure the correct orthogonality properties hold. So, for a fixed encoding size, the readout loss is minimised if a: = 0. This occurs when the encodings are orthogonal (i.e. d1 and d2 are orthogonal from one another, all other a, vectors, and from each of the b, vectors). This happens the two frequencies are modularised from one another, and additionally the sine and cosine vectors for each frequency are orthogonal. I.e. a modularised solution with this property has the minimal readout weight loss for given encoding size.
Each vector has the following form, A,
D.3 RECURRENT LOSS WITHOUT BIAS
First we consider the slightly easier case where there is no bias in the recurrent dynamics:
Wg(() = g +
(63)
Then we will write down a convenient decomposition of the min-norm W. Call the matrix of stacked coefficient vectors, X:
an
b
by bo]
(64)
Similarly, call the matrix of stacked normalised psuedo-invers vectors Xt:
x
[A
B
AF
BF
Bu)
(65)
And finally create an ideal rotation matrix:
cos(wi At) sin(wi At)
sin(w, At)
cos(wiAt)
R
cos(w At)
sin(wr
0
sin(wpAt)
(t)(wFAt)
(66)
R
0
R
0
RF
0
0
Where each R; is 2 x 2 rotation matrix at frequency wr Now:
W = X
(67)
We can then calculate the recurrent weight loss:
W Tr XXX X
(68)
X X is symmetric 2F + 2F positive-defintt matrix, and its inverse is Xt.T Xt, another symmetric positive-definite matrix. To see that these matrices are inverses of one another perform the singular value decomposition, X UVVT, and X UE-IVT, Then X X 2 VT and
x x VE VV-²V" which are clearly inverses of one another.
Introducing a new variable, Y xx X:
W 2 TYY R²T
(69)
We then use the following trace inequality, from Ruhe (Ruhe, 1970). For two positive semi-definite symmetric matrices, E and F, with ordered eigenvalues,
en > 0 and fi 2 ... 2 fn >
Tr EF 2
ein-itt
(70)
26

---

Published as a conference paper at ICLR 2025
Now, since RY R' and Y-1 are similar matrices, they have the same eigenvalues, and Y-! is the inverse Y so its eigenvalues are the inverse those Y. Therefore:
2
A A
W - THYRY R²
+1
(71)
Then we can show that this lower bound on the weight loss is achieved when the coefficient vectors are orthogonal, hence making the modular solution optimal. If all the coefficient vectors orthogonal then Y and Y -1 are diagonal, so they commute with any matrix, and:
W TrYYR R TT R R Tr T1[22++1] 2F +
(72)
D.4 RECURRENT LOSS WITH BIAS
Now we return to the case of interest:
Wg b= g(t + 1)
(73)
Our energy loss, equation penalises the size of the weight matrix W and the bias. Therefore, if we make W smaller by assigning some of its job to b then should. We can do this by setting by = b (recall the definition of by from equation 60) and constructing the following, smaller, min-norm weight matrix:
A
R
0
0
B
R2
0
W
lai
b aF bF]
X
(74)
A E
R.
B
Using the definitions in the previous section. This slightly complicates our previous analysis because now X X ÿt is not the inverse of X X Y. If orthogonal to all the vectors (a;; b:) then it is the inverse, and the previous proof that modularity is optima goes through.
Fortunately, is easy to generalise to this setting. X not quite pseudoinverse of X because its vectors have to additionally be orthogonal to b. This means we can break down each of the vectors into two components, for example:
A A Ai.
(75)
The first of these vectors is the transpose of the equivalent row of the pseudoinverse of X, lives in the span of the vectors {a, bi) The second component is orthogonal this span and ensures that AT' b 0. Hence, the previous claim. If b is orthogonal to the span of {a, b, b this is the standard pseudoinverse and the previous result goes through.
We can express the entire X matrix in these two components:
x x
(76)
Then:
- x
x X
(77)
Both of these new matrices are positive semi-definite matrices, since they are formed by taking the dot product of of vectors. Then:
w = T (YR R
(78)
Now, the first term is greater than equal than 2F, as in the previous setting. And since Ý and Y are positive semi-definite, and so therefore is Rřý j²"" this second term is greater than or equal to 0. Hence. W 2 2F, and orthogonal encodings achieve this bound, therefore is an optimal solution according to the weight loss.
27

---

Published as a conference paper at ICLR 2025
D.5 STRUCTURE OF REPRESENTATION ACT II LARGE Aw
We saw that our representation takes the following form.
(())
Σ cos(w;t) + b, sin(wrt) + bo
(79)
Further, and we saw in the previous section that each added frequency increases the recurrent weight loss. To solve the task we only need two frequencies, why would have more than two? The only possibility is that by including additional frequencies we might save activity energy, at the cost of weight energy. We will simplify our life for now by saying Aw very large, so our representation only has two frequencies. We now turn to activity loss and ask which frequencies should mix, and which modularise.
D.6 FREQUENCIES
Consider mixed frequency neuron:
(t) (a1); cos(wit) + (bi): ): sin(wit) + cos((t)) + (b2)) sin(w2t) + bn
(80)
By rescaling and shifting time,
rewrite all such mixed neurons in simpler form:
gi(() a, cos(() + B cos(wrt + 6) + by
w >
(81)
Where w is the ratio of the larger to the smaller frequencies. We will show that the activity of this mixed neuron lower than its corresponding modular counterpart when an integer, and higher is irrational. Finally, we show empirically, and some theoretical evidence, that the same is true of rational, non-integer w. Hence, we find the activity energy minimised by modularising unless one of the frequencies is an integer multiple of the others. Further, since if the mixed neuron gi (t) preferred over its modular counterpart, then so too are positively scaled versions, (t.(t), we will consider responses of the form:
s(t) = cos(t) + 5 cos(wt + + + A
W W>I 1
(82)
Irrational: e2
Rational: (2, 5)
Harmonics: (2, 4)
1
1
0
-1
0
1
0
1
Figure 8:
Schematics showing irrational and rational ratio case the data are range dependent beyond the modular- mixed boundary while in do-modularising harmonics casc, the periodic waves are range independent.
Irrational Frequencies Modularise w is irrational then, by Kronecker's theorem, you can find a value of t. for which cos(t) and cos(wt Φ) take any pair of values. This makes the two frequencies, among other things, extreme point independent, and therefore no mixing is better than modularising.
Even Integer Multiples Mix According to Theorem to modularise, for all é and Ø values:
A
= 1 0(0))
(83)
28

---

Published as a conference paper at ICLR 2025
We will therefore try find a 8 and which breaks this inequality. Consider 6 < 0 and very small and @ = 0, then, writing w = 2m for integer m.
A min(cos(t) + cos(2mt))
(84)
(t) [6] (2mm) takes its extreme values when t is integer multiple of n, and the smallest it can be -1 + 6). Hence:
= < 1 0(8*)
(85)
This is smaller than the critical value, the representation should mix, since least one mixing inequality was broken.
Odd Integer Multiples Mix If is an odd integer then we can instead mix cosine with sine by choosing Φ
(t) = cos(t) sin(wi) +
(86)
Then the same argument goes through above.
Other Rational Multiples Modularise Now consider w = for two integers p and * 1. We
were inspired in this section by the mathoverflow post Souury and Speiser (2015). So we are considering the mixed encoding:
g(t) = cos(t)= + 5 cos( P t + )) +
(87)
And we have to show that for all 8 and Φ, A > + 82. For general 6 we have been unable to do this, but for & 0 present proof below in the hope this can be generalised.
We break down the problem into four cases based on the sign of 5, and whether and are both odd, or only one is. The easiest is if both are odd, and 5 0. Then take qut:
g()) cos(gT) + cos(pT) + A = - - 8 + A>0
(88)
Therefore A is at least 1 + 6, which larger than v1 + 82, hence modularising is better.
Conversely, if one of p or q even (let's say p w.l.o.g.) and then choose ( pm again:
g(() = - + 8 cco((p)) + A = -1 1 [ + > 0
(89)
And the same argument holds.
Now consider the case where 8 > 0 and one of p or is odd. Then there exists an odd integer & such that (Soudyy and Speiser (2015):
kp =p=q - mod 2g
(90)
Therefore take ler: g(t) cos( + 5 cos(T(
kp A = -1 ccos(T( (
1- ++ng )) A
-11
(91)
q
For some integer n.. Developing:
π + A s(1) = -1 -1 8 cos(
(92)
The key question is whether for any A below the critical value this representation is nonnegative. For this to be true:
-1 - ò cos
V 12 20
(93)
Developing this we get condition on the mixing coefficient 6:
2 cos( +
2 cos? (E)
(94)
We can apply a similar argument with
gkæ
to get:
π
cos
8
+
A
2
0
(95)
p
29

---

Published as a conference paper at ICLR 2025
This leads to:
1 -
1 cos2
2 cos( } 8
(96)
We squeeze inequalities eqn (4 and get
(1 cos (os2(T/ cos (
S
(97)
4 cos((T cos( ccos(T/p))
For any even, odd integer pair of p. g # and 5 0, cos (z /p) and cos( (1)) are bounded in (0. 1]. In this case, the maximum possible value of the righthand side is achieved with the smallest possible pair integers (3, 4), for which the inequality does not hold. Thus the inequalities eqn 96 94 cannot hold at the same time which let us to conclude that there is no A below the critical value satisfying nonnegativity constraint and leads to modular representation.
The last case is where å < 0 and both p. are odd. We can extend the relation shown in eqn 90 as following by simple substitution of oddity of the variables. With given odd integers p, q, there exists an even integer k satisfying eqn 900 Similar to above, take t gkz for the mixed encoding equ 87 and we develop:
9r = ==
+ + ( p
(98)
which leads to the following inequality:
1 - cos² (0e((T//) (T/P)
2
(99)
-2 cos (T) kpp)
Now take f = kt, and we get:
9i 8 cos(TT) (T/q) A
(100)
which leads to the in equality,
5 2 cos((T (T)2(T//))))
(101)
Again, the above two inequalities cannot hold together with the odd integers p,g > We also empirically show on Fig that in rational multiple frequencies, there is no modular-mixed boundary possible to break the modularisation.
30

---

Published as a conference paper at ICLR 2025
E METRICS FOR REPRESENTATIONAL MODULARITY AND SOURCE STATISTICAL INTERDEPENDENCE
We seek to design metric measure how much information latent contains about source. However, we would like to ignore information that could be explained through the latent's tuning different source. For example, perhaps two variables are correlated, if a latent encodes one of them, then it will also be slightly informative about the other despite not being functionally related. We therefore seek condition on all of the sources bar one, and measure how much information the latent contains about the remaining source.
Dunion al. (2023) leverage conditional mutual information in a similar way this in reinforcement learning context, but for training rather than evaluation, and therefore resort to a naive Monte Carlo estimation scheme that scales poorly. Instead, we leverage the identity
[ (zzi 8:(8_:) [(z; 8) 10---(aa (zz;
(102)
where s-; is shorthand for {s. + i}. Since this involves computing mutual information with multiple sources, wC restrict ourselves to considering discrete sources and use a continuous- KSG scheme (Ross) 2014) to estimate information with continuous neural activities. We normalise conditional mutual information by (s; to obtain a measure in [0. 1]. We then arrange the pairwise quantities into a matrix C Rd, and compute the normalised average "max-over-sum" in column as measure of sparsity, following Hsu et al. (2023)
de max; Ci 1 de
de (1-(1-1)
CInfoM(s, =
(103)
d2 1=1 Ci
CInfoM appropriate for detecting arbitrary functional relationships between a source and a neuron's 's activity. However, in some of our experiments, the sources are provided as supervision for linear readout the representation. In such cases, the network cannot use information that is nonlinearly encoded, so the appropriate meausure is the degree of linearly encoded information. Operationalising this we leverage the predictive y Y-information framework of Xu al. (2020). We specify the function class y linear and calculate
Ty(z, (1)8_)) [r( ++ Ty 8-i) = Hy Hy - Hy + Hy (s-1lz)),
-)
(104)
followed by a normalisation by (s, Each predictive conditional Y -entropp term estimated via standard maximum log-likelihood optimization over V/ which for us amounts to either logistic regression or linear regression, depending on the treatment of the source variables as discrete or continuous. The pairwise linear predictive conditional information quantities are reduced single linear conditional InfoM quantity by direct analogy to (103).
Finally, to facilitate comparisons across different source distributions, we report CInfoM against the normalised multinnformaionn of the sources:
H(s;) H(s)
NI(s)
(105)
H (8:) max; H (8))
This allows us test the following null hypothesis: that breaking statistical independence, rather than range independence, is more predictive of mixing. On the other hand, if range independence is more important, then source distributions that retain range independence while admitting nonzero multinformation will better induce modularity compared to those that break range independence.
31

---

Published as a conference paper at ICLR 2025
F WHAT-WHERE TASK
F.1 EXPERIMENTAL SETUP
F.1.1 DATA GENERATION
The network modularises when exposed to both simple and complex shapes. Simple shapes, used in all main paper figures, are 9-element one-hots, reflecting the active pixel's position in 3 3 grid. Formally, for a shape at position (i,j) in the grid, the corresponding vector E I29 is given by:
[1 if 3(i 1) +
Sk 0 otherwise
where i,j E 213
For complex shapes, each shape is binary vector e E IR", with exactly elements set to and elements set 0, representing the active and inactive pixels, respectively. Each complex shape takes the (approximate) shape of a letter (9) and can be shifted to any of the 9 positions in the 3 3 grid. The standard training and testing datasets include one every shape-position pair. The correlations introduced later include duplicates of a subset of these data points.
H
T
L
Different possible inputs the network. Both one-hot (left) and letter- (right) shapes can be outputted as either a concatenation of one-hots or as a 2D variable.
Figure 9:
F.1.2 NETWORK ARCHITECTURE
The network we here is designed in PyTorch and takes as input 81-element vector, flattened a x 9 image The network architecture is formally defined as follows:
Input: 81
Hidden layer: h = (W, b1).
W, W E
b E 25
Output layer: y W2h + b2, W2 2x225
by IR²
where o is the activation function, either ReLU or tanh, depending on the experiment. Weights W, and W. are initialised with a normal distribution NV 0.01), and biases by and by are initialised to zero.
F.1.3 TRAINING PROTOCOLS
The network uses the Adam optimiser (ingmma and Ba (2044), with learning rate and other hyper- parameters varying from experiment to experiment. The mean squared error (MSE) is calculated
32

---

Published as a conference paper at ICLR 2025
separately for 'what' and "where' tasks, and then combined along with the regularisation terms. The total loss function Leotal is a combination of the task specific losses and regularisation terms:
Liotal
Lwhhtt + Lwhere + >R([I 112 + W 2ell2 + ((bl2)
where Tras for 'what' and 'where' tasks defined as:
Ltast (v. N Σ yb 02)2
Where Ån is the regularisation hyperparameter, typically set to 0.01 unless specified otherwise. The network is trained using the Adam optimiser with learning rate ranging between 0.001 and 0.01, adjusted as needed. Experiments are run for order 10 epochs on 5 random seeds. Each experiment was executed using single consumer PC with 8GB of RAM and across all settings, the networks achieve negligible loss.
F.2
MODULARITY OF WHAT & WHERE
F.2.1 BIOLOGICAL CONSTRAINTS ARE NECESSARY FOR MODULARITY
To understand the effect of biological constraints modularisation, consider the optimisation problem under positivity and energy efficiency constraints. The activation function ((x) ensures non-negativity:
(xx) max(0, x)
The energy efficiency is enforced by adding L2 regularisation term to the loss function. By minimising the combined loss function Liotal the network encourages sparse and low-energy activations, leading separation of neurons responding different tasks ('what' and 'where').
To illustrate the necessity of biological constraints the modularisation of these networks, we show below the weights and activity responses of networks for networks where these constraints aren't 't present. As discussed above, positivity is introduced via the ReLU activation function, and energy efficiency is defined an L2 regularisation on the hidden weights and activities.
As shown in the neural tuning curves, the unconstrained networks do encode task features, but they do so such that each neuron responds to the specific values of both input features, they are mixed selective. Compare these results to the weights and activity responses when positivity and energy efficiency constraints are introduced. In this setting, there is clear separation of 'what' and 'where' tasks, into two distinct sub-population of neurons.
F.3 DROPOUT & CORRELATION
We use two different approaches to dropout in order to illustrate the importance of range independence.
The first approach removes example data points (i.e., chape-position pairs) from elements along the diagonals, starting in the middle and extending out to all four corners. This has the effect increasing the mutual information between data sources but does not significantly affect their range dependence. Formally, let D be the set of all shape-position pairs. In the diagonal dropout setting, we remove pairs (s., p.) where i lies along the diagonal the input grid:
Diagonal Dropout:
D' DD= ((;, diagonal positions}
In the most extreme case, only one data point from each corner is removed, and this is not sufficient force mixed-selecctvvtyy in the neurons.
The second approach removes data points from one corner of the distribution. This also increases the mutual information between sources but changes their extreme point dependence. Specifically, we remove pairs (s) where lies in the bottom-left corner of the input grid:
Corner Dropout:
D = D =D
((s1)14) | bottom-left corner positions]
33

---

Published as a conference paper at ICLR 2025
One-hot
Complex
2D
Hidden weights
Activity responses
Figure 10: Hidden weights and activity responses of FF networks without biologicallyinnpiie constraints. The
one-hot and letter-like binary cases shown, as well the 2D output setting with one-hot inputs.
One-hot
Complex
Hidden weights
Activity responves
Figure 11: Weights and activity response of networks under biological constraints. one-hot and letter-like binary cases are shown, as well the 2D output setting with one-hot inputs.
In this setting, a single data point removed from the corner is insufficient for breaking modularity; however, removing more than this causes mixed selective neurons appear.
34

---

Published as a conference paper at ICLR 2025
Correlation: To correlate sources, duplicate data points that appear along the diagonal. This increases the mutual information between sources without affecting their range independence:
D" DU ((s,. Pc) i diagonal positions)
The mutual information I(XY) between the shape X and position Y is calculated as follows:
P(x.v v)
((x Σ P(a, log P(x)
where P(x, y) is the joint probability distribution of X and Y.
XXX
Figure 12: Increasing dropout (left to right) for both the corner-cutting (top) and diagonal (middle) cases, as well as correlation distributions (bottom). Note: asterisk denotes the absence of training data for this pair of input features.
G NONLINEAR AUTOENCODERS
G.1 AUTOENCODING SOURCES
dineessiimesion source data is sampled from [o, 1|3 and discretised to 21 values per dimension.
The encoder and decoder are each two-layer MLP with hidden size 16 and ReLU activation. The latent bottleneck has dimensionality six. All models use Aveconstruct = 1. Aactivity energy = 0.01, Auctivity nomnegativity 1, and Aweight energy 0.0001. Models are initialised from He initialisation scaled by 0.3 and optimized with Adam using learning rate 0.001. Each experiment was executed using a single consumer GPU on HPC with CPUs and 4GB of RAM.
G.2 AUTOENCODING IMAGES
We subsample 6 out of the sources in the Isaac3D dataset, fixing single value for the other 3. This yields a dataset of size 12, 288. We use an expressive convolutional encoder and decoder taken from the generative modeling literature. We use 12 latents. each quantised to take on 10 possible values. We use a weight decay of 0.1 and a learning rate of 0.0002 with the AdamW optimizer. Each experiment was executed using single consumer GPU HPC with 8 CPUs and 8GB of RAM.
35

---

Published as a conference paper at ICLR 2025
H RECURRENT NEURAL NETWORKS
H.1 LINEAR AND NONLINEAR PERIODIC WAVE RNN EXPERIMENT DETAILS
For the linear RNN setting we provided a periodic pulse input (2D delta function of frequencies W1, ti2 input i.e. (t) = (wr))( - 1), k E (1,2)) and trained the network to generate two cosines of the same frequencies, yı: (t) cos(urtt)
For the nonlinear RNN we designed a two frequency mixing task. Given two source frequencies, the network must generate two cosine waves whose frequencies are the sum and difference of two input frequencies. The network requires non-linearities in order to approximate the multiplication: cas(a- + b) cos(a) cos(b) - sin(a) sin (b), cos(a - b) = cos(a) cos(b) + sin(a) sin(b). the task
provide the RNN with 2D periodic delta pulse of frequency wliu2 and w1 wl-w2 and learns to generate a trajectory of cos(w;) and cos(w2).
We use the following recurrent neural network,
g(t + 1) = /(Wrecg(() + Wi + bree)
(106)
y(1) Rg(() + bout
(107)
in our linear RNNs f(.) is identity whereas in the nonlinear ones we used ReLU activation to enforce positivity condition.
For irrational output frequency ratio case, we used
W1 = pi, W2 = / p ~ (0.5,4)) ~ 2 (1,
(108)
For rational case, sampled
11, W2 E [0] nZ.
(109)
Where Z is the set of integers. For harmonics:
w1 [0, n 02 2w/
(110)
We trained the RNN with the trajectory of length T = 200, bin size 0.1, and hidden dimension 16.
For the linear RNN, we used learning rate le-3, 30k training iterations and Aurget 1, Aactivity 0.5, Apositivity = 5, and Aweight = 0.02. For the nonlinear RNN, we used learning rate 7.5e-4, 40k training iterations and Atarget = 5, Aactivity = 0.5 and Weeight = 0.01. In both case, we initialised the weights to be orthogonal and the biases at zero, and used the Adam optimiser.
To assess the modularity of the trained RNNs, we performed a Fast Fourier Transform(FFT) on each neuron's 's activity and measured the relative power of the key frequencies W1, Wa with respect to the sum of total power spectrum
[FFT(g:)))) E, FFT f)
C neuronn,,
(111)
and used as a proxy of mutual information for modularity metric introduced in eqn. [03
H.2 NONLINEAR
RNNS
Network details. The Teacher network has input, hidden, and output dimensions of 2, and has orthogonal recurrent weights, input weights, and output weights. The Student RNN has input dimension 2, output dimension 2, and hidden dimension 64. It initialised as per PyTorch default settings. The Teacher network dynamics is vanilla RNN: h, = (anch + Winde), and each teacher predicts a target via O, W Wo The Student RNN has identical dynamics (but with a ReLU activation function, and different weight matrices etc).
Generating training data. The teacher RNN generates training data for the Student RNN. We want to tightly control the Teacher RNN hidden distribution (for corner cutting or correlation analyses), ie., tightly control the source distribution for the training data. To control the distribution hidden activities of the Teacher RNN, we use the following procedure. 1) We sample randomly initialised Teacher RNN. 2) ho is initialised as a vector of zeros. 3) With batch size of N, we take a single step of the Teacher RNN (starting from O hidden state) assuming 0. This produces network
36

---

Published as a conference paper at ICLR 2025
activations (for each batch), p. tonh(W (wreehu--.... 4) We then sample from idealised distribution of the teacher hidden states, h2. For example a uniform distribution, or a cornet cut distribution. At this point these are just i.i.d. random variables, and not recurrently connected. 5) To recurrently connect these points, we optimise the input to the RNN, é,, such that the RNN prediction, Pt. becomes, h4- We then repeat steps 3-5) for all subsequent time-steps, i.e., we find what the appropriate inputs are to produce hidden states as if they were sampled from an idealised distribution. To prevent the Teacher RNN from being input driven, on step 4), we solve a linear sum assignment problem across all batches (N batches), so the RNN (on average) gets connected to a sample, h2, that is close to its initial prediction, Pt- This means the input, é, will be as small as possible and thus the Teacher RNN dynamics are as unconstrained as possible.
We train the Student RNN on 10000 sequences generated by the Teacher RNN. The learning objective is a prediction loss joeachher oftaddent (2 plus regularisation of the squared activity of each neuron as well as each synapse (both regularisation values of 0.1). We train for 60000 gradient updates, with a batch size of 128. We use the Adam optimiser with learning rate 0.002.
Training.
H.3 RNN MODELS OF ENTORHINAL CORTEX
We now talk through the linear RNNs used in Section 5 Our model is inspired by the entorhinal literature and linear network models of Dorrell al. (2023); Whittintonn et al. (2020) 2025) (023). In each trial the agent navigates a 3x3 periodic environment. The RNN receives one special input only at timepoint 0 that tells the layout of the room (format described later). Otherwise at each timestep it is told which action (north, south, east, west), the agent took. It uses this action to linearly update its hidden state via an action-dependen affine transform:
9: Wa.9:-1 + boc
(112)
At each timestep it has to output, via an affine readout, particular target depending on the task.
In the spatial-reward task, Figure 4 at timepoint the agent starts at a consistent position and receives an input that is 9-dimensional 1-hot code telling the agent the relative position of the reward. At each timestep the agent then has to output two 9-dimensional 1-hot codes, one signalling its position in the room, the other the relative displacement of the reward from its position.
The agent experiences many rooms. Across rooms the rewards are either randomly sampled (random rooms), sampled from one of two fixed positions (fixed rooms). Each agent experiences different mixture of randomly sampled or fixed rooms. We then optimise the network to perform the task with non-negative neural activities while penalising the 1.2 norm of the activity and all weight matrices.
In the mixed-selectivity task, Figure 5] there are three objects in each room and the agent has to report the relative displacement of each of them. The input at the start of each trial is 27-dimensional code signalling where the three objects are relative to its own position. As such, as the agent moves around the room has keep track of the three objects. This task was harder train we moved from the mean squared error to the cross crosseentrppy loss and found it worked well, apart from that all details of loss and training are the same.
Between trials we randomise the position of the objects. Some portion (0.8) of the time these positions are drawn randomly (including objects landing in the same position). The rest of the time the objects were positioned so that the first object was one step north-east of the second, which itself was one step north-east of the third. This introduced correlations between the positions of the objects, while preserving their range independence all objects could occur in all combinations.
We measured the linear NCMI between the neural activity and the 27-dimensional output code and found that each neuron was informative about single source, as expected, it had modularised Figure 5C. However, pretend we did not know the third object existed. Instead we would calculate the linear NCMI between each neuron and those objects we know to exist. We would still find modular codes for these objects, but we would also find that the neurons that in reality code for third object, due to the correlations between object placements, second object, so they look mixed selective! The position of the objects is always informative about each other, but by conditioning on each object we are able to remove this effect with our metric and uncover the latent modularity. But without knowing which latents to condition on we cannot proceed, and instead get lost in correlations.
actually informative about the first and
37


Reviews:
[{"review_id": "ZTQXeWbzAH", "reviewer": "Reviewer_a7qJ", "summary": "This paper investigates why neural representations in biologically inspired networks sometimes form modular structures, where each neuron encodes a single variable, and other times create mixed-selective representations. The authors develop a theory that predicts when modularization will occur in networks optimized for energy efficiency with nonnegative firing rates. They derive necessary and sufficient conditions for modularity based on the spread of source variables. The theory is validated in both linear and nonlinear networks. The theory provides a cohesive explanation for the conflicting findings in the prefrontal cortex and entorhinal cortex data from neuroscience studies.", "questions": "1. Beyond prefrontal working memory and the entorhinal cortex, does the theory generalize to other modular representations in the brain?\n\n2. If decoding flexibility were considered as a biological constraint, how might it impact the theory and its predictions?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper introduces a novel theory that precisely predicts necessary and sufficient conditions for modular representations in biologically inspired networks, extending previous work beyond statistical dependencies.\n\n2. The mathematical formulation is rigorously derived, and validated across various neural network architectures and experiments.\n\n3. The theory provides explanations for conflicting neuroscience findings and has close links to biologically plausible architectures and brain representations.\n\n4. The paper provides a cohesive theory for understanding modularity in neural representations, with implications for both interpreting biological neural data and guiding the design of artificial neural networks for better interpretability and efficiency.\n\n5. The paper is well-written, presenting complex theoretical concepts with clarity and intuition.", "weaknesses": "1. The experiments use nonnegative activities in neural networks, which aligns with biological plausibility, but it would be valuable to discuss inhibitory neurons in the brain and how inhibition might relate to the theory and findings.\n\n2. While the L2 norm of firing rates and weights is a reasonable approximation for biological energy, other biological constraints (e.g., sparse connectivity, synaptic range, anatomical structure, and decoding flexibility) may also play a role.\n\n3. It’s unclear how the theory would extend to more complex datasets. For example, what would the different conditions/variations in source variables mean in naturalistic data (e.g., natural images, audio, text)? How might we approximate \"spread\", and quantify modularity conditions in such stimuli?\n\n4. The discrepancies in prefrontal working memory modeling and the brain data could have further explanations, particularly why some neurons tune to both colors despite orthogonal encoding and why exact subspace angles were not obtained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9SQ7tIVcwq", "reviewer": "Reviewer_1fwm", "summary": "This paper studies the conditions under which modularity should arise in optimal representations. The authors developed a mathematical theory of when a modular representation should be favored in a linear autoencoder. They found that modularity should appear when the support of the sources is “sufficiently spread”. The paper also presented simulation results to show that some of theoretical results may be generalizable to non-linear problems. The later sections of the paper applied these theoretical ideas to explain some experimental observations from neurophysiological experiments in cognitive tasks. \nThe paper makes several interesting points about when modularity should be favored, and applies the theory to several examples.", "questions": "What are the assumptions about the noise in the system being studied? How does noise affect the theoretical results?\n\nIn section 5, the authors seems to be equating orthogonality with modularity. Am I understanding this correctly? If this is the case, can the authors unpack the idea?\n\nCan they authors unpack the results in Fig. 5c? Was that an actual simulation or just a schematic?\n\nThe definition of the matrix in Eq. 10 is unclear. Please clarify. \n\n\nCan the authors explain what the first part of the title mean?\n\nIn the abstract, it is stated that “From this theory, we extract and validate predictions…” What does “extract” predictions mean? [minor point]", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Originality: The theoretical part of the work builds up prior work by Whittington et al, 2023 and other studies. Previous work by Whittington et al, 2023 assumed mutual independence of the sources. In the current work, the authors show that, with several additional assumptions, “sufficient spread” of the factors of variation can also lead to modular representation. The theory has some new elements, although it is a bit incremental. The application to the several neuroscience problems seems to be new.\n\nQuality: The paper considered both linear and nonlinear cases. This is a strength. For the former, analytical results were provided. For the latter, some preliminary numerical results were given. \nThe paper also considered several neuroscience applications. This may also be seen as a strength.\n\n\nClarity: The overall structure of the paper is clear. Some intuitions behind the theory were provided.\n\n\nSignificance: The question of when modularity arises in optimal representation is an interesting one and we still lack a clear understanding. This work made a few interesting points on this problem.", "weaknesses": "The writing needs improvements throughout the paper. In particular, the description of the theory can be substantially improved. For example, Theorem 2.1 should be made more accessible. \n\n\nWhile several applications are attempted, each application appears to be preliminary. If the model predictions and experimental tests can be made more rigorous, that would strengthen the paper.\n\nIn Section 5, there are some qualitative differences between the model predictions and the data. As the paper pointed out, Panichello and Buschman (2021) showed that a substantial fraction of the neurons were tuned to both colors, contradicting a key prediction of the model. This seems to be a more important feature of the data compared to the issue of orthogonality v.s. non-orthogonality.\n\nLooking at the math, the theory appears to only work for scalar variables. Can it be applied to circular variables? If the answer is no, the applications to real data would be questionable. In section 5, color is sampled from a color-wheel. \n\nBased on the way the theory is written, the results seem to rely the assumption that the energy (or cost) is a quadratic function of neural activity. If the cost scales linearly with neural activity, would the theoretical results change fundamentally? Assuming a linear scaling could make sense biologically, as the metabolic cost may scale linearly with the number of spikes. \n\nRelevant earlier theoretical literature on grid cell modularity was not cited/discussed (e.g., by Fiete/Burak et al, and Wei/Prentice/Balasubramanian).\n\n\nIt is difficult to understand what is really going on in Fig. 1. Fig.1 may come out too early and the results need to be better unpacked. \n\n\nThe theory seems to ignore biologically-relevant noise.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4UBzDt7SQC", "reviewer": "Reviewer_Gn2p", "summary": "This paper seeks to explain why and when a population of biological or artificial neurons sometimes modularise and sometimes entangle the representation of source variables. This is a fundamental question that is highly relevant to both neuroscience and AI. The authors propose and prove a new theory emphasising the importance of the shape of empirical data distribution in extreme regions in dictating whether neurons are mixed selective or modular. Specifically if the sources to be represented are supported in all extreme regions, the neurons modularise. The application of this theory outside of linear autoencoders is tested in feed forward and recurrent neural networks, including experiments that provide explanations for discrepancies in previous neuroscience literature.", "questions": "- Biologically inspired linear rnns are repeatedly mentioned. Linear rnns are less like biological circuits (which are nonlinear).  Is the biologically inspired term referring to the energy costs?\n150: satisfied for all w is a little unclear. The proof could  \n- Autoencoding seems limited as an objective for theory, in that brains perform computations over inputs and states and produce behaviour. Is it not the relevance of representing the extreme points for behaviour that is important? \n- Section 2.2 assumes positive weights?\n- 120 \"be better\" is imprecise \n- Fig 1d, neuron’s angle isn’t described? \n- I don’t understand the what where regression justification. \n- How relevant are these results for more typical ANN experiments?  E.g classic image benchmarks or language modelling tasks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This work is original, of high quality and undoubtedly contributes to the community’s understanding of neural modularisation. The nonlinear verification of theory and additional application to neuroscience results are significant for the field and a strength of the paper. The submission is well written and clear throughout, although its clarity suffers somewhat due to the amount this submission seeks to cover.", "weaknesses": "- In my opinion this submission contains too much, and would benefit from more focus and time spent on fewer experiments. The appendix is already large but some experiments could be moved there.  \n- Figure text and panels are too small throughout.\n- It is not clear how relevant encoding of the extreme points of source distributions are for computation / cognition. I.e. neurons do not just autoencode.\n- The bio description of energy minimisation assumes l2 penalty is an appropriate penalisation function for modelling biology. This is a fair starting assumption, but no argument is presented about how this maps to the costs biological neurons will be seeking to minimise.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)