Convex Efficient Coding
Authors: Will Dorrell
Peter E. Latham
James C. R. Whittington
Venue: ICLR 2026 Poster (2026)
Keywords: Neuroscience
Representation
Identifiability
Primary area: applications to neuroscience & cognitive science
TL;DR: We derive a family of convex representational optimisation problems and use them to derive identifiability results for matrix factorisation and single neuron tuning curves.

Abstract: Why do neurons encode information the way they do? Normative answers to this question model neural activity as the solution to an optimisation problem; for example, the celebrated efficient coding hypothesis frames neural activity as the optimal encoding of information under efficiency constraints. Successful normative theories have varied dramatically in complexity, from simple linear models (Atick & Redlich, 1990), to complex deep neural networks (Lindsay, 2021). What complex models gain in flexibility, they lose in tractability and often understandability. Here, we split the difference by constructing a set of tractable but flexible normative representational theories. Instead of optimising the neural activities directly, following (Sengupta et al. 2018), we instead optimise the representational similarity, a matrix formed from the dot products of each pair of neural responses. Using this, we show that a large family of interesting optimisation problems are convex. This includes problems corresponding to linear and some non-linear neural networks, and problems from the literature not previously recognised as convex such as modified versions of semi-nonnegative matrix factorisation or nonnegative sparse coding. We put these findings to work in two ways. First, we extend previous results on modularity and mixed selectivity in neural activity; in so doing we provide the first necessary and sufficient identifiability result for a form of semi-nonnegative matrix factorisations. Second, we seek to understand the meaningfulness of single neural tuning curves as compared to neural representations. In particular we derive an identifiability result stating that, for an optimal representational similarity matrix, if neural tunings are `different enough' then they are uniquely linked to the optimal representational similarity, partially justifying the use of single neuron tuning analysis in neuroscience. In sum, we identify an interesting space of convex problems, and use that to derive neural coding results.

CONVEX EFFICIENT CODING
William Dorrell, Peter F. Latham" Gatsby Computational Neuroscience Unit University College London
James Whittington
Department of Experimental Psychology
University of Oxford
dorrellwec@gme com
* Co-Senior Authors
ABSTRACT
Why do neurons encode information the way they do? Normative answers to this question model neural activity as the solution to an optimisation problem; for ex- ample, the celebrated efficient coding hypothesis frames neural activity as the op- timal encoding of information under efficiency constraints. Successful normative theories have varied dramatically in complexity, from simple linear models (Atick) & Redlich 1990), to complex deep neural networks (Lindsay] 2021). What com- plex models gain in flexibility, they lose in tractability and often understandability. Here, we split the difference by constructing a set of tractable but flexible norma- tive representational theories. Instead of optimising the neural activities directly, following (Sengupta et al. (2018), we optimise the representational similarity, a matrix formed from the dot products of each pair of neural responses. Using this, we show that a large family of interesting optimisation problems are convex. This family includes problems corresponding to linear and some non-linear neural networks, and problems from the literature not previously recognised as convex, such as modified versions of nonnegaivee matrix factorisation or nonnegative sparse coding. We put these findings to work in three ways. First, we provide the first necessary and sufficient identifiability result for form of semi-nonnegative matrix factorisation. Second, we show that if neural tunings are 'different enough' then they are uniquely linked to the optimal representational similarity, partially justifying the use of single neuron tuning analysis in neuroscience. Finally, we use the tractable nonlinearity of some of our problems to explain why dense retinal codes, but not sparse cortical codes, optimally split the coding of single variable into ON and OFF channels. In sum, we identify space of convex problems, and use them derive neural coding results.
INTRODUCTION
Neural activity forms the substrate of intelligence in both brains and artificial neural networks. To- wards understanding these systems, much work has therefore tried to frame and solve neural coding puzzles; for example, is neural activity temporal rate code (Gautrais & Thorpe 1998). why do we find Gabor filters in both visual cortex (Jones & Palmer 1987) and Convolutional Neural Networks trained to recognise objects (Yosinski al.] 2014). Normative approaches provide com- pelling models. They frame neural responses as the solutions to optimisation problems, the most classical of which the efficient coding hypothesis (Attneave) 1954 Barlow 1961); loosely, 'neu- rons use the representation that most efficiently encodes the required information Finding a match between optimal encodings and the brain provides evidence that these neurons can be thought of as solving the proposed optimisation problem, naturally leading to predictions and broader functional theories. These approaches have been successful, for example studying tuning curves (Laughhin 1981).
However, the tractability of these theories varies significantly. Some problems admit pleasing an- alytic solutions, outlining clearly how representations arise from the optimisation problem (Atick) & Redlich 1992). More often this is not the case. For example, neural activity is often modelled
using task- ootimipe neural networks (Hinton et al. (2006; Yamins et al. 2014 Lindsay 2021 Kell
ct al. 2018] Botvinick & Plaut] 2006 Wang et al. 2018 Zipser & Andersen, 1988 Tanaka 2016]
Goldstein et al.] 2022
Gauthier & Levy 2019), which largely analytically impregnable. Eieen

---

successful simpler models like sparse coding Olshausen & Field 1996) or nonnegative matrix fac- torisation (Lee & Seung 1999) are not fully understood (for example, it is not known under what conditions sparse coding generates Gabor filters). These complexities hinder potential insights.
A lot of work therefore attempts to 'open the black box', either by studying trained networks (e.g. (Sussillo & Barak] 2013)), using simpler tractable models. Here, we build a recent example of a tractable optimisation problem. Following a long line of related work (Pehlevan & Chklovskii 2019), Sengupta et al.] (2018) study a similarity matching objective-wiich loosely measures the alignment between the similarity of a pair of neural encodings and the similarities of their corre- sponding inputs-ane and show that the optimal nonnegative representations of compact variables are place cells if there are unlimited neurons. The technical novelty of their approach lies in showing that the optimisation problem can be written as convex optimisation problem over the set of rep- resentational dot-product similarity matrices (the matrices of dot-products of neural representations in each task condition). In so doing, [Sengupta et al. (2018) unlock the application of convexity to cutting-edge problems in neuroscience. However, this approach has not been extensively used, perhaps due to the bespoke nature of the similarity matching objective.
In this work, we broaden this approach. In section 2] we show that we can write large family of objectives and constraints as convex over the set of representational dot-product similarity matrices. Arbitrary sets of convex constraints and objectives can then be composed to create variety of interesting optimisation problems, each of which are convex. Many of these problems correspond to linear, even nonlinear, neural networks with different regularisation schemes. Indeed, using these components, some problems from the literature can be reframed as convex problems. In this paper, we study three of these problems of relevance AI and neuroscience.
First, in Section 3 we study an existing problem not previously recognised as convex: nonnegative- affine autoencoding of a set of sources. Previous works studied orthogonally embedded sources and derived set of necessary and sufficient conditions such that, optimally, each latent neuron encodes single source, and used this model patterns of modularity and mixee-selecttivtyy in the entorhinal cortex (Whittington et al., 2023 Dorrell et al. 2025). We generalise to the case of linearly (rather than orthogonally) mixed sources. This change seems small, but in so doing, we derive the first necessary and sufficient conditions for the identifiability of related matrix factorisation problems.
Second, in Section 4 we use the same theory to link population-level representational similarity to single neuron tuning curves. We derive conditions under which a given representational similarity matrix is created by a unique set of single-neuron tuning curves. Naturally one may expect many tuning curves to be associated with given representational similarity, since representations can often be rotated arbitrarily, scrambling the neural tuning curves, without changing representational similarity or task performance. If this were the case, looking at the activity of single neurons would not be a useful signal for inferring the function of a neural population. Here we show that this broadly not the case, since the convex constraint of neural activity being non-negative breaks rotational symmetry. For problems with this constraint (which applies to biological neurons), we derive a set of sufficient conditions that outline how, if the tuning curves are 'different enough', all optimal solutions will contain the same neural responses. In so doing, we provide justification for studying tuning curves, linking them precisely to the representation's 's optimality.
Finally, in section 5) we use our tractable convexity on a non-linear problem to answer neural coding puzzle. Retinal neurons display ON-OFF splitting, in which a single variable encoded in a pair of oppositely rectified neurons (Euler et al.) 2014). This splitting has been understood as an efficiency: splitting the stimulus reduces the range of each neuron, saving energy (Sterling & Laughlin 2015 Gjorgjieva et al., 2014). But not all variables are ON-OFF coded and existing theories cannot explain what governs this. We show that the transition between ON-OFF and pure coding is driven by the variable's sparsity, matching conjecture (Sterling & Laughlin 2015).
In sum, we greatly expand the set of problems amenable to convex analysis and apply these results to coding puzzles and identifiability results relevant both AI researchers and neuroscientists alike.
2 A FAMILY OF CONVEX REPRESENTATIONAL OPTIMISATIONS
In this section we establish the convexity of a series of representational optimisation problems, beginning with simple example.
2

---

2.1 A MOTIVATING EXAMPLE
We consider optimisation problems over -iprrseetaaiios-ddddddeeeennnn vectors of neural activity for each of N datapoints: =/1 E R 1, .... N. For example, we might require set of targets, y!! E R to be linearly decodable from our representation.
y Wo
There are many feasible [ additionally constrain the neural activity to be nonnegative (= 0), and penalise the energy use, either through firing rates or synaptic weights, the latter in line with evidence that the largest energy cost of spiking synaptic transmission (Harris et al.) 2012). We use the L2 norm for mathe- matical convenience, but consider modified Li activity loss m appendix A Hence our problem,
Ji=ı and W out. and for neurobiological realism and interpretability
min W Won:: (=0))
(1))=10 + Wotl
subject to
0, Wonz =y
(2)
Where we use the physicists notation () for average. This is a simple instantiation of the efficient y ensures 214 encodes information about y!,, and subject to this we maximise the efficiency by minimising the energy loss over both representation, =!0, and weights, W Yet, even simple models like this have widespread use in neuroscience (Dordek
coding hypothesis; W
al. 2016 Sorscher et al. [2019] Martin-Sáncher et al.) 2025 Huang et al. 2026). As written, this problem is not convex" I
We use classic approach to convexification (Shor 1987 Lasserre 2009 Anstreicher 2012, Pena et al.] (013)) following Sengupta al. (2018). and find that, if d; > N, problems like these can be framed as convex optimisations over the set of representational dot- product similarity matrix, defined as:
(Q), = (2 2bl
(3)
We can show this by showing each of the constraints and functions convex:
Firing Cost:
|||=||
1 N
E( T Tr[Q].
(4)
This is a linear, and hence convex, function of Q.
Weight Cost: Since the representation is linearly decodable, the min- norm choice of readout weight matrix is given by the pseudoinverse. Defining the data and representation matrices:
Y E R | )
Z E &R
[Z)..
Zi:
Wost
Zz.
(5)
where Z denotes the pseudoinverse. Then the weight cost becomes:
Wour out 3.- Tr [
oul Woa] Tr Y zt Zt yQt]
(6)
We show in appendix A that is a convex function of Q.
Nonnegativity: The set of dot- not-product matrices nonnegative vectors (Q = ZT
0) form a convex set called the set of completely positive matrices (Berman & Shaked-
Monderer 2003), as in (Sengupta et al. (2018).
Decodability: Requiring that the targets can be linearly decoded from the representation limits the set of allowed similarity matrices to those in which there is some subspace en-
coding the labels, a set we also show is convex in lappendix A
Each objective and constraint is convex, and combinations of convex functions and sets are con- vex (Boyd & Vandenberghe, 2004), hence the problem is convex. This ensures that all locally optimal representational dot-product similarity matrices, Q. are globally optimal. This will prove theoretically useful. However, as explain in section 6) optimising over the set of completely pos- itive matrices is NP-hard (Dickinson & Gijben 2014) stopping us from developing practical convex optimisation algorithms.
For example, consider two feasible solutions that differ by neuron swap:
W
[
2/
W out
W
out
(4]
20
(v
0
2
2
u
As shown by the last equation, despite both solutions satisfying cq. their convex combination does not
3

---

2.2 A FAMILY OF CONVEX PROBLEMS
In appendix A we present set of representational constraints and objectives and show that each is convex over the set of representational similarity matrices. Further, we combine them construct series of problems, few of which we now highlight.
Regularised Linear/Affine Networks Linear/Affine neural networks have proved to be popular tractable models in neuroscience and machine learning; for example in studying learning dynamics Saxe et al. (013)); (Braun al, (2022). With L2 weight regularisation and sufficient width, we show the optimisation problem posed by these networks is convex. Adding a nonnegativity constraint breaks rotational symmetry, making the neuron basis meaningful and allowing one to use these networks to ask neural tuning questions. For example. Whittington et al. ( Dorrell et al. (2025) use these models to reason about why neural recordings in the entorhinal cortex are sometimes modular, with disjoint sets of neurons encoding space and reward, and sometimes mixed-selectivv with the same neurons showing tuning to both space and reward. While nonnegativity makes the optimisation problem over Z non-convex, when reframed over the dot-product matrix Z" Z the problem becomes convex, which we use in /sctionn to derive a novel tight identifiability criterion.
Tractable Nonlinear Problems An appealing feature of [Sengupta et al. (2018) is its ability to tractably model nonlinear tuning curves, by optimising a nonnegative representation to minimise similarity matching loss (Tr(GQ] for positive-definite matrix G). We extend this, by show- ing that optimisations over nonlinear, but linear (affar(affine)>) -decodeed representations, as in section 2. are convex. These models have been used in ncuroscience model grid cells (Dordek et al. 2016 Sorscher et al.] 2019 | JSchoyec et al. 2023] Tang et al.) 2024) (via the nonnegative PCA). place
field remapping (Martin- Sänchezz et al. 2025), or zebrafish visual responses (Huang et al. 2026),
and we use them in section 5| model retinal coding. Further, they correspond to classic sparse
coding or matrix factorisation approaches that have found success in modelling Gabor filters (O1-
shausen & Field (996) and learning parts-based representations (Lee & Seung 1999). Future work could therefore use the uncovered convexity as analytic traction to understand these phenomena. In lappendix A, we also show that the arbitrary nonlinearity can be replaced by ReLU, allowing us to show that wide, regularised one-hidden layer ReI.U networks are convex, a result which seems similarly promising, and related recent work (Pilanci & Ergen 2020 Zeger & Pilanci, [025] Wang et al. (2025) (see srelssctionn& 6 for further comparison). Finally, lappendix Alalso shows that even the affine readout constraint can be dropped, by optimising nonlinear similarity matching loss- Tr for positive-definite S-which has been previously used as a model of multifield place cells (Dorrell et al. 2023). The flexibly nonlinear input and output mappings implicit this problem suggest an interesting model for studying the internal representations of deep networks, such as nonlinear disentangling autoencoders (Higgins al. 2017).
3 IDENTIFIABILITY OF SEMI-NONNEGATIVE MATRIX FACTORISATION
We now reframe a semi-nonnegative matrix factorisation algorithm as convex optimisation, permit- ting to derive necessary and sufficient conditions under which the "true" factors recovered.
Background Matrix factorisation problems seek to break matrix, such as the label matrix, Y E Rdy XN into two meaningful parts Y AS, E T xd. S E Rd,xN, For example, dictionary learning seeks to learn dictionary, A, and sources, S, that fit the data while using sparse S. In general, many choices of A and can fit Y. For example, given one feasible pair (A, S) such that Y AS, inserting any invertible d, x d, matrix, B, will generate another feasible pair, (AB, SS) since: ABB- S = AS = Y. Therefore, this problem is only usefully posed in settings with more structure, either via constraining the allowed factorisations or regularising their choice. For example, semi-nonnegative mairix factorisation problems constrain one of the matrices to be nonnegative, while sparse coding uses an L.1 regularisation on the source matrix.
Identifiability Identifiability results outline when a problem is well-posed. By well-posed we mean the following. First, the data- mata-generating assumptions are true, so in the case of matrix fac- torisation problems, that the data really is linearly generated from two sub-matrices (otherwise how do we know what the model 'should' do?). Second, the proposed algorithm will recover the true data-generating factors. Previous work has extensively studied the identifiability of matrix factori-
4

---

sation, as we'll review later. However, to the best of our knowledge, all results are either necessary or sufficient, but not both. Here, we use our convexity results, /section
results,secctio, 2 to derive necessary and suf- ficient identifiability conditions for neural network based factorisation algorithm with applications in theoretical neuroscience: nonnegative-affin autoencoders.
Nonnegative Affine Autoencoding We derive such identifiability result for particular matrix factorisation problem, nonnegative affine autoencoding, which has been previously used as a model of neural data |Whittington et al. [2023] Dorrell al. 2025). Data is generated via X AS and fed to the network: an autoencoder with two affine layers and non non-negativity constraint on the hidden layer activity. The network weights are optimised to perfectly reconstruct X while minimising the 1.2 weight and activity norms. We ask when, up to constant shift, the data-generating factors A and S are recovered in the optimal weights, W and activities, Z. of the autoencoder. In other words, we look for identifiability conditions under which Z 11 + b1, meaning the latents recover the sources up to an arbitrary offset b, and a 'rettangllar permutation" matrix, II, with at most one non- zero entry in each row (and at least d, non-zero entries). Previous work has derived conditions when A is an orthogonal matrix, and used this to understand patterns of modularity and mixed selectivity in the entorhinal cortex (Whittington et al. 2023 Dorrell et al. (025). Here, use the convexity of the optimisation over the set of representational similarity matrices, Q = z Z, section 2 to derive identifiability results for the case of arbitrary A. This problem is formalised below.
Problem 1 (Nonnegative Affine Autoencoding). We provided dataset, {x') xl E R generated linearly from dataset of sources { sbi E R.d,, dz dg. and a full-column-rank mixing matrix A gdd,xd,, x!!9 Ass We use weights. W.. E RR,,xd, and W Rd, and biases, bin E Rdd and bout E IRd,, to define a nonnegative affine-encoding of this dataset, {zll) J 211l Rdd, d₂ dy using the following constrained opfimisation problem:
(1 + (I/ + (/Woull/:..
min Win.bm. Wout Wout.Dou Dous
W Wm + bin: x!0 Woutz + bott, 211 0.
s.t.
Previous work using orthogonal A found that identifiability in this model is governed by the 'spread' of the sources. If the distribution of sources is "sufficiently rectangular' the optimal representation recovers the sources (Dorrell et al. 2025). Precisely, if the convex hull of the data engulfs partic- ular easily-calculabe set it is sufficiently rectangular, fig. 1A, and the optimal solution recovers the sources; otherwise it mixes them.
Returning to arbitrary A complicates things. If the encodings of two sources are aligned, i.e. A Aj is nonzero, where A, is the ith column of A, then weight regularisation encourages the encodings of those sources to align. This effect can cause the optimal solution to mix sources when it would otherwise modularise, or even modularise when it would otherwise mix lig. Our result improves the 'sufficient rectangularity' of previous work by adapting it appropriately to the mixing, A. In particular, we measure this adaptive rectangularity with tight scattering condition using a quadratic form calculated from the dataset covariance and minima, and the mixing gram matrix AT A.
Definition 1 (Tight Scattering). Given the same sources, [ and mixing matrix, A, as prob. lem 1 Generate the mean-zero sources, $/ =! ), and assume, without loss of general- ity that | min; =//l S max; 8d for each d (if not satisfied simply redefine 8a -8a). Stack the mean-zene zero sources into a matrix, 5 E RRdx with elements Sa = s and construct the diagonal D E de xds and the symmetric F E dRds xds matrices:
D3s = ((8)))))+ + (min; 5/ + A((ATA))))))
F= - XA7 A)-11
X(AT A)as
(8)
where E = SS is the covariance. Use these matrices to construct the following set:
E {xc x
F x 1}
2This arises because the 'range effects' explored in previous work, and these weight effects are both signed:
if AT A, is positive (negative) the weight loss encourages the source encodings to positively (negatively) align. If range and weight effects misalign the modular solution he optimal even if it isn't using orthogonal A.
5

---

The sources tightly scattered with respect A the following conditions hold:
Conv(S)
Conv(( bdE 1 bd the boundary. and e₁ the kthh canonical basis vector, (ex)1 òjk-
ds) where the * denotes the dual cone,
The first condition ensures that the convex hull of the sources, Conv(S) (S). engulfs the A-dependent ellipse. The second is a technical condition that ensures the boundary of the ellipse, bd and the convex hull of the sources only touch along basis directions.
A] CONDITIONS ENSURE
SOURCES SATISPY ORTHOGONAL NOT EXAMPLE ANTIALIGNED CONDITIONS
POPULATION TUNINGS
5 SUFFICIENT RECTANGULARITY
ANTIALIGNED MIXED
ORTHOGONAL MODULAR
Neurons
MAX
DATA CONVEX
UNDERLYING SOURCES
S1
ELLIPSE
NEURON'S
B) DIFFEENNT ELUPSE CONDITIONS DIFFERENT MOING MATRICES A EXAMPLE ANTIALIONED A[A, <.0
SOURCES EXAMPLE AUGNED NOT ORTHOGONAL CONDITIONS
POPULATION TUNINBS
ALIGNED MCDULAR
ORTHOGONAL MIXED
EXAMPLE ALINND:
A[A2 2
Neurons
ORTHOGONAL
MAX
52
51
HIGHLIGHTED NEURON'S ' RESPONSE
Figure 1: (A) We schematise the the identifiability conditions for two sources; the conditions spec- ify a set (e.g. the red ellipse that depends on in the observed data) that the convex hull of the underlying empirical source distribution has to engulf. If this condition is satisfied the empirical source data is 'rectangular enough', there is no better linear transformation, and the sources are recovered, else the optimal representation is mixed. (B) When the data consists of linearly mixed sources via A, then this equates to warping the identifiability conditions: either via aligning (bluc) or antialigning (green). (C) Source alignment can make the optimal solution mix when would have otherwise modularised. We show such a dataset in which the orthogonal but not exam- ple antialigned identifiability conditions are satisfied. Matching the theory, numerical solutions are modular for the orthogonally encoded sources (rightmost column), but not for the antialigned (mid- column). We plot linear conditional mutual information (Hsu et al., 2023 Dorrell et al.) 2025) between each neuron and source scaled by the neuron's 's peak activity. Below we display highlighted (purple) neuron's tuning to sources. (D) Similarly, source alignment can cause recovery of sources that would otherwise not be. We show example dataset where aligning the sources by a specific amount causes the warped identifiability conditions to be satisfied (blue). Matching this, the aligned sources recovered (middle column), but not the orthogonal ones (right column).
Our main theorem simply states that if and only if the sources tightly scattered the problem is identifiable, i.c., to scaling and permutation Z and Wout will recover S and A. Our theorem relates the empirical dataset to identifiability, our knowledge, this is the first identifiability condition for semi-nomee nonnegatneeee matrix factorisation that both necessary and sufficient, and we suspect similar ideas can generalise to dictionary learning.
is inherently finite-sample result. To the best of
Theorem (Identifiability of Nonnegative Affine Autoencoders). if and only if the matrix S is tightly scattered with respect to A then the optimal positive affine autoencoder recovers the sources, i.e. each neuron's activity is an affine function of at most one source, and every source is encoded by least one neuron.
the same setting as problem I

---

The proof is in appendix B. and in mappendix C) we derive similar imperfect reconstruction result in which fitting is enforced by a mean mean-square error term.
Previous Work The identifiability of various matrix factorisation problems has been extensively studied, and our condition is best seen as a generalisation of earlier sufficient scattering results. Donoho & Stodden (2003) studied nonnegative matrix factorisation and showed that if the data sat- isfied a scattering condition (that the convex cone of datapoints surrounded an 'ice-cream cone' in the postive orthant) the model was identifiable. HHu & Huang (2023) present state-of-the-ar suf- ficient identifiability condition for dictionary learning regarding the scattering of a sign-permuted dataset around a unit ball. This is similar both to our work, to work on polytope matrix factorisa- tion (Tatli& & Erdogan| 2021a by and to sufficient spread conditions for identifiabilty of nonnegative matrix factorisation (Huang et al. 2013). These works involve scattering relative to set defined independently of the mixing matrix A. In contrast, our work studies a different factorisation algo- rithm which lets us harness the convexity of the problem, and is the only work to adapt the scattering condition to both A and the dataset covariance. This allows us to derive the first necessary and suffi- cient criterion for semi- nonnegative matrix factorisation, as all the cited examples are only sufficient. The other works (Whittington et al. 2023: Dorrell et al. 2025 that study this form of neural net- work matrix factorisation, the affine nonnegative autoencoder, are far less general as they only only develop conditions for the simpler case where A is an orthogonal matrix.
4 WHEN DOES OPTIMALITY IMPLY UNIQUE SINGLE NEURON TUNING?
We have described conditions for recovery of the imnarly-mixed 'true' factors in semi noniennnneeative matrix factorisation via the construction of nonnegative affine autoencoders with hidden represen- tation, z. This result relied upon framing the problem in terms of the representational similarity matrix, Q z Z, which summarises the population structure. This link between population representational similarity and single neuron coding of sources raises the tantalising possibility of formally understanding how individual neural tuning curves relate to neural manifolds, and inferring the brain's 's underlying algorithm from population activity and single neurons alike.
Classical work in neuroscience involves correlating neural activity with task-relevant variables to infer function (e.g., evidence of whitening operations (Atick& & Redlich, 1990; 1992), or conceptual navigation strategies (Constantinescu et al.] (2166). This assumes that activity patterns are linked to function. However, results in machine learning have shown how fragile the representational- function link can be without further assumptions. The same function can be implemented by many neural networks (Baldi & Hornik 1989), and the internal representations of those networks might differ radically, confounding attempts to use correlations between neural activity and task-relevant variables to infer function. This point was reinforced by recent paper that analytically studied the solution space of linear neural networks, finding that without further assumptions the repre- sentational similarity matrices of networks performing the same function completely unrelated (Braun et al., 2025). Similar concerns arise when using single-neuron tuning curves. Most simply, rotating the neural population scrambles the tuning curves while leaving the representational simi- larity unchanged. potentially meaning the observed neural basis bears little relationship to function. Indeed, concern over the relevance of single neuron responses has led to debate over the correct level of neuroscientific enquiry (Barack & Krakauer 202T).
Here, we construct a defence of traditional neuroscience approaches. We begin the population level, defending the use of representational similarity matrices. Then we develop theory that outlines situations in which even single-neuron tuning curves are tightly coupled to optimality.
Unique Representational Similarity Matrices There is no unique neural network for given func- tion. This is simply illustrated by adding additional neurons to a network that are disconnected from the output, shaping representational similarity, but not affecting function. However, if the network is regularised, then irrelevant neurons will have low activity and so won't 't affect representational simi- larly. Indeed. Braun et al.) (0025) find that regularised linear networks have unique representational similarity matrices (those with either the objectives O1&04, or 03&04, in the language of [ap- pendix A). Our framework extends this argument, showing that large family of problems beyond regularised linear neural networks are convex over the set of representational similarity matrices, making the choice of representational similarity far from arbitrary.
7

---

Unique Single Neuron Responses We now go one step further: when should we expect every net- work that optimally implements the same function to exhibit the same sinle-neuron tuning curves? For this to happen, not only does function need unique representational similarity (as above), but also rotational symmetry must be broken. This is most naturally done by constraining neural activity to be nonnegative (Dordek al.] 2016 Sengupta et al.) [0188] Whittington et al.) [023) DDrrell al.) 2025). Thus, to answer the above question, we derive conditions under which nonnegativity ensures that all globally optimal representations are related only by the inherent permutation symmetry; in other words, all optimal solutions contain the same set of neurons, but shuffled.
Precise Problem and Sufficient Conditions We study a nonnegative representation, Z which has the globally optimal similarity structure: Z Z Q- For another representation to be globally optimal it must have the same dot-product similarity, z Z Q' and thus must be an orthogonal transform of the first representation: Z OZ Which Z are possible? This problem therefore directly asks how tightly neural tuning curves (Z) are linked to representational optimality (Q").
Following similar logic to Theorem 1, we now show that O is constrained to be a permutation matrix - meaning every optimal representation contains the same neural tuning curves - only when the neural response patterns, tuning curves, satisfy one of family of sufficient scattering condi- tions. Intuitively, if the neural tuning curves are 'different enough' from one another no rotation can preserve the optimality of the representation. This is described in the following theorem, roofs are presented in appendix D
Theorem (Tight Scattering Implics Unique Neurons). If a given neural dataset { JT=1 satisfies
the following scattering constraints with respect to a set E = {æ + (214)) F x = 1 for any positive definite matrix F with diagonal equal to ((z(0)))) then all orthogonal matrices such that Oz il 2 0 are permutation matrices:
(Cov((z1())) 2 E
.
(Cov((2!())) n bd(E") = (Aex/A R, 12 1....dd. ds}
For each choice of F you get an ellipse, E, circumscribed by the range of the data fig. 2/. The first term ensures that the convex hull of the neural responses (Conv((zli) (x) 'swallows' the ellipse, while the second is technical condition that ensures the intersections between the boundaries of the cllipse, (d))., and the convex hull are along the axes. Intuitively, as long as there is at least one F (and hence E) such that these are satisfied, the joint-distribution of neural activities is 'rectangu- lar enough' such that any rotation will necessarily push some activity negative, [īg. TA, and hence the optimal tunings are unique. This can be seen as an extension of the tight scattering results [sec-] tion 3) as long as you're re tightly scattered with respect to at least one A-dependent ellipse it must be impossible to rotate the code while preserving positivity, else our identifiability results would be void. In fig.. 2] we demonstrate (fig. 2C. D) satisfy these conditions. We have proved the sufficiency of this set of conditions, though conjecture that they might additionally be necessary.
example of pair of tuning curves that do fig. 2B) and don't 't
Grid Cell Modularisation As a neuroscientific example, we now apply these results to understand why the famous grid cell tuning curves appear in discrete modules. Grid cells are neurons with a lattice receptive field to position (Hafting et al., (005) and they come in groups called modules: grid cells within the same module share the same lattice receptive field (but translated) (Stensola et al. 2012), and each module has a different lattice frequency. Here we answer the question why each module has its own set of neurons (as opposed to mixing in the population). To do this, we consider two hypothetical grid cells from two different modules and test whether they satisfy our identifiability conditions. We find that, if the lattice frequency of one module is close to an integer multiple of the other, they don't satisfy our identifiability conditions, otherwise, they do. Thus, the fact that we observe grid cells in discrete modules is consequence of the non-integer frequency relationship between grid modules. If one module lattice was an integer multiple of the other the population would have done better by using a mixed encoding. This matches the finding that, in order to encode space effectively efficiently. modules are not integer multiples of one another (Sreenivasan & Ficte 2011//Wr et al. 2015 Dorrell et al. 2023) [Schafffer et al. 2023 Dorrell et al. 2025), and hence identifiable.
8

---

EXAMPLE ELUPSES:
B) DIS/OINT PLACE CELLS IDENTIFIABLE
E) NON-HARMONIC GRID HODULES IDENTIPABLE
NEURAL
Firing Rate
NEURAL
NEURAL
CONVE
RESPONSES
CONVEX HULL
HULL
SATISFIED
CONVER HULL
SATISPIED
ELLIPGE
ELUIPEE
1D Latent
C) OVERLAPPING PLACE CELLS NON-IDENNIABBLE
D) As OPTIMA ROTATED NEURONS F) HARMONIC GRID MODULES NON-IDENTIIABLE
R2
FR
22
DISSAI ELLIPSE
ELUPSE
Figure 2: A) Illustration of identifiability condition: the condition states that the convex hull of the neural responses must engulf one of a set of ellipses. Plotting the data for two neurons, and drawing two ellipses for different choices of F shows (blue) engulfed. Since there is at least one, the condition is satisfied. (B) We try this on two neurons, tuned like place cells single 1D latent. Plotting the neural responses and example ellipse shows us that these tunings are identifiable, since the convex hull of the data engulfs the ellipse. However, (C) moving the place cells to overlap leads to non-identifiability. Indeed, (D) we can rotate the responses to find two different tuning curves with the same optimal dot- dot-product structure. (E) We apply this to grid cells. Grid cells from two different modules are identifiable long as their wavevectors are not integer multiples of one another, otherwise (F) they become non-identifiable, and we can numerically find rotation of these neural responses that preserves nonnegativity.
example where the ellipse is not (purple) and is
5 TRACTABLE NONLINEAR THEORY OF ON-OFF CODING
Promisingly, many of our convex models are nonlinear (section 2, appendix A Here, use convex nonlinear model to confirm a neural coding conjecture relating the optimality of ON-OFF coding to sparsity. In an ON-OFF code, a single variable split into two oppositely rectified encod- ings (Euler et al. (014), and in some settings this has been shown to be energy efficient (Gjorgjieval ctt al.) 2014). However, not all variables are ON-OFF coded, and it has been conjectured that sparsity governs whether to use ON-OFF code (Sterling & Laughlin 2015). Existing theories that exhibit ON-OFF coding are either intractable neural networks (Ocko et al. 2018 Jun et al., 2022) or rely on direct model enumeration (Gjorgjieva al. 2014), stymieing efforts derive the parameters that govern ON-OFF optimality. Here, we use a tractable normative model to confirm the conjectured role of sparsity, and analytically derive simple threshold.
We study a nonnegative affine-decodable representation of a single variable, z(T), similar to eq. (2)
min(())((D)I)))) + Al w 12:),
subject to w" 21x(() bb 2(T) 2
(10)
In appendix E use the KKT conditions to find optimal solution, which convexity then guar- antees is unique. We find two regimes. In the first, all neurons linearly encode the stimulus; in the second, the neurons split into ON and OFF channels, fig. 3/. Sparsity governs this transition: lowers the firing rates. However, if the variable is sufficiently sparse (e.g. 2 but / often), is better to ensure that the encoding of / uses low firing rates, leading to a single channel code, fig. 33. We analytically derive this threshold sparsity, matching simulations, fig. C
I is dense, channel-splittin
(1)2
Prob(T = =0) (T2)
(11)
9

---

REPRESENTATION
B)
THRESHOLD
VARIABLE SINGLE CHANNEL
SINGLE-CHANNEL
ON & OFF CHANNELS
K
PREDICTED
THRESHOLD
0.07
SPARSENESS 0.4
SPARSENESS
Figure 3: A) We numerically find a solution to le. (10) and plot the firing rates of each neuron in the population function of I. We find that all tuning curves correspond either OFF ON channels. B). We add a point mass at = to the distribution over I, thus increasing the sparsity, For low sparsity the optimal representation contains both ON and OFF neurons. As sparsity increases the coding range of the ON neurons increase while the range of the OFF neurons decreases, until at the threshold (eq. 11p the OFF neuron disappears and are left with ON neurons only. We display only the unique tuning curves, the population is comprised of copies of these tuning curves. C) We quantify the degree of single channel coding, Jappendix E and find slowly increases to a ceiling the predicted sparsity threshold, corresponding to completely single channel representation.
6 DISCUSSION
In this work we showed that a variety of interesting representational optimisation problems could be written as convex optimisations over the set of representational similarity matrices. We used this to derive a tight catrix-factorisattin identifiability criterion, section 3 to understand the link between tuning curves and optimality, /section 4 and used the tractable nonlinearity of these theories to un- derstand ON-OFF coding codingsscction 5| These results sound an optimistic note for classic neuroscience: we can correctly frame the relevant neural computation and constraints, it seems that representa- tion and function are tightly coupled, including often the single neuron level, /section Given this optimism, we hope future work will use this paper's 's framework to develop analytic theories for nonlinear phenomena in both neuroscience and machine learning.
Convex Analysis & Neural Networks A lot of work has, as here, reframed the optimi- sation of neural networks as convex optimisation. Earlier approaches studied the optimal addition of single neurons to existing networks and found this was convex (Bengio et al. [2055] Bach 2017). Neural tangent kernel approaches showed that, in the infinite width, training neural network was convex, but this limit often removes interesting representational learning phenomenon (Jacot et al.)
Literature:
2018). More recent work has shown that the entire standard neural network problem can be framed as a convex optimisation for 1-hidden layer ReLU networks (Pilanci & Ergen] 2020) Ergen & Pi- lanci 202I). These results have been extensively developed and used to understand neural network phenomenology (Sahiner et al.) 2020; Zeger & Pilanci, 2025), and likely have much to offer neu- roscience. They differ technically from ours in the vanables used to construct the convex problem (we use representational similarity matrices, they
version of the weights). Further, they stick tightly to the neural network problem setting, while we follow the more neuroscience approach of optimising representation, allowing us flexibility the cost of less relevance for machine learning.
Limitation 1: Many Neurons First, If the number of neurons is larger than the number of data- points, the rank of Q unconstrained, and our optimisation problems are convex. Unfortunately, restricting the neuron number, and hence rank of Q, is a non-convex constraint. One general fix that future work could usefully explore constraint on the nuclear norm of Q (sum of singular values). Further, /section demmonstattes more bespoke workaround: the solution the unconstrained problem uses few neurons, it will also be constraint.apppediix
replace the rank constraint with its convex relaxation-a
the solution in the neuron-constraing setting, letting us relax the neuron constraint, (appendix B.5)
Limitation Computational Intractability possible dividend from a convex reformulation efficient optimisation algorithms. These, however, rely on efficiently testing whether the proposed solution is member of the feasible set. Unfortunately determining membership of the set com- pletely positive matrices is NP-hard (Dickinson & Gijben| 2014 One approach is to approximate the set with a hierarchy of increasingly precise enclosing sets (Nishijima & Nakata 2024). Alter- natively, could optimise directly over Z, and use the conditions in section 4 to try and prove global optimality. However, verifying sufficient scattering conditions, Tike those in section 4 is often hardard. though fortunately there is work tackling this problem (Gillis & Luce 2024).
10

---

Reproducibility Statement The appendices include all our mathematical proofs, and code for the small simulations we ran to test the theoretical results can be found at: https: /tts:://gtttbbb com/ WilburDoz/ Concex Efficient Coding ICLR 2026.
Acknowledgements The authors thank Pierre Glaser for discussions about convexity, Tim Behrens and Kris Jensen for careful readings of earlier drafts, James Fitzgerald for discussions about related optimisation problems, and especially Erin Grant for extensive reading and discussion, and Nicholas Gillis for extensive comments.
We thank the following funding sources: Gatsby Charitable Foundation (GAT3755; W.D. & P.E.L); Sir Henry Wellcome Post-doctoral Fellowship (222817/7/211/2; J.C.R.W); European Re- search Council Starting Grant (NARFB/101222868 J.C.R.W).
REFERENCES
Kurt M Anstreicher. On convex relaxations for quadratically constrained quadratic programming. Mathematical programming, 136(2):233-255) 2012.
Joseph Atick and Norman Redlich. Towards a theory of early visual processing. Neural compu- lution, 2(3):308-333 -2000 1990.
Joseph J Atick and A Norman Redlich. What does the retina know about natural scenes? Neural computation, 122):1962210 210, 1992.
Fred Attneave. Some informational aspects of visual perception. Psychological review, 61(3):183, 1954.
Francis Bach. Breaking the curse of dimensionality with convex neural networks. Journal of Ma- chine Learning Research, 18(19):1-53, 2017.
Pierre Baldi and Kurt Hornik. Neural networks and principal component analysis: Learning from examples without local minima. Neural networks, 2(1))53-58, 1989.
David L Barack and John W Krakauer. Two views on the cognitive brain. Nature Reviews Neuro- science, 2256)335-377, 2021.
Horace Barlow. Possible principles underlying the transformation of sensory messages. Sensory communication, 1(01), 1961.
Yoshua Bengio, Nicolas Roux, Pascal Vincent, Olivier Delalleau, and Patrice Marcotte. Convex neural networks. Advances in neural information processing systems, 18, 2005.
Abraham Berman and Naomi Shaked-Monderer. Completely positive matrices. World Scientific, 2003.
Matthew M Botvinick and David C Plaut. Short-term memory for serial order: a recurrent neural network model. Psychological review, 1 113(2):201, 2006.
Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge UP, 2004.
Lukas Braun, Clémentine Dominé, James Fitzgerald, and Andrew Saxe. Exact learning dynamics of deep linear networks with prior knowledge. Advances in Neural Information Processing Systems, 35:6615-666 6629, 2022.
Lukas Braun, Erin Grant, and Andrew M Saxe. Not all solutions are created equal: An analytical dissociation of functional and representational similarity in deep linear neural networks. In Forty- second International Conference on Machine Learning, 2025.
Alexandra O Constantinescu, Jill X O'Reilly, and Timothy EJ Behrens. Organizing conceptual knowledge in humans with a gridlike code. Science, 16446222):144441444 2016.
Peter JC Dickinson and Luuk Gijben. On the computational complexity of membership problems for the completely positive cone and its dual. Computational optimization and applications, 57 (2):403-415, 2014.
11

---

David Donoho and Victoria Stodden. When does non-negative matrix factorization give correct decomposition into parts? Advances in neural information processing systems, 16, 2003.
Yedidyah Dordek, Daniel Soudry, Ron Meir, and Dori Derdikman. Extracting grid cell character- istics from place cell inputs using non-negative principal component analysis. Elife, 5:e10094, 2016.
Will Dorrell, Peter Latham, Timothy Behrens, and James C. Whittington. Actionable neural representations: Grid cells from minimal constraints. In The Eleventh International Confer- ence on Learning Representations, 2023. URL httph::/ httt:::://pperevieew net forum?rdddd
xfqDe72zh4]
William Dorrell, Kyle Hsu, Luke Hollingsworth, Jin Hwa Lee, Jiajun Wu, Chelsea Finn, Peter Latham, Timothy Edward John Behrens, and James CR Whittington. Range, not independence, drives modularity in biologically inspired representations. The Thirteenth International Con- ference on Learning Representations, 2025.
Tolga Ergen and Mert Pilanci. Revealing the structure of deep neural networks via convex duality. In International Conference on Machine Learning, pp. 3004-3014, PMLR, 2021.
Thomas Euler, Silke Haverkamp. Timm Schubert, and Tom Baden. Retinal bipolar cells: elementary building blocks of vision. Nature Reviews Neuroscience, 15(8)5007-519 2014.
Xiao Fu, Kejun Huang, and Nicholas D Sidiropoulos. On identifiability of nonnegative matrix factorization. IEEE Signal Processing Letters, 25(3):328-332, 2018.
Jean Gallier. Notes on the schur complement. 2010.
Jean Gallier. Geometric methods and applications: for computer science and engineering, vol- ume 38. Springer Science & Business Media, 2011.
Jon Gauthier and Roger Levy. Linking artificial and human neural representations of language. arXiv preprint arXiv: 191002291, 2019.
Jacques Gautrais and Simon Thorpe. Rate coding versus temporal order coding: a theoretical ap- proach. Biosystems, 48:57-65, 1998.
Nicolas Gillis and Robert Luce. Checking the sufficiently scattered condition using global non- convex optimization software. IEEE Signal Processing Letters, 31:1610-1614, 2024.
Julijana Gjorgjieva, Haim Sompolinsky, and Markus Meister. Benefits of pathway splitting in sen- sory coding. Journal of Neuroscience, 34(36): 2014.
Ariel Goldstein, Zaid Zada, Eliav Buchnik, Mariano Schain, Amy Price, Bobbi Aubrey, Samuel A Nastase, Amir Feder, Dotan Emanuel, Alon Cohen, et al. Shared computational principles for language processing in humans and deep language models. Nature neuroscience, 25(3):369-380, 2022.
Gene H Golub and Victor Pereyra. The differentiation of pseudo-inverses and nonlinear least squares problems whose variables separate. SIAM Journal numerical analysis, (0(2):413-433 1973.
Torkel Hafting, Marianne Fyhn, Sturla Molden, May-Britt Moser, and Edvard Moser. Microstruc- ture of spatial map in the entorhinal cortex. Nature, 436(7052):801-8800 2005.
Julia J Harris, Renaud Jolivet, and David Attwell. Synaptic energy use and supply. Neuron, 75(5): 762-777, 2012.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In International conference on learning representations, 2017.
Geoffrey Hinton, Simon Osindero, Max Welling, and Yee- See-Whye Teh. Unsupervised discovery of nonlinear structure using contrastive backpropagation. Cognitive science, 30(4):725-731, 2006.
12

---

Kyle Hsu, William Dorrell, James Whittington, Jiajun Wu, and Chelsea Finn. Disentanglement via latent quantization. Advances in Neural Information Processing Systems, 36, 2023.
Jingzhou Hu and Kejun Huang. Global identifiability of 11-based dictionary learning via matrix
volume optimization. Advances in Neural Information Processing Systems, 36:36165-36186, 2023.
Kejun Huang, Nicholas D Sidiropoulos, and Ananthram Swami. Non-negative matrix factorization revisited: Uniqueness and algorithm for symmetric decomposition. IEEE Transactions on Signal Processing, (2(1):2112244 2013.
Shuhong Huang, Ruben Portugues, and James E Fitzgerald. Behavioral alignment an organizing principle in sensory coding. bioRxiv, pp. 2026-02, 2026.
Robert Israel. Is the trace of inverse matrix convex?, 2013.
URL
15 SsrTbe-ttaeee------eeeesettt--o.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kemnl: Convergence and gen-
eralization in neural networks. Advances in neural information processing systems, 31, 2018.
Judson P Jones and Larry A Palmer. An evaluation of the two-dimensional gabor filter model of simple receptive fields in cat striate cortex. Journal of neurophysiology, 58(6); 1233-11223, 1987.
Na Young Jun, Greg Field, and John Pearson. Efficient coding, channel capucity, and the emergence of retinal mosaics. Advances in neural information processing systems, 35:32311-32324 2022.
Alexander JE Kell, Daniel LK Yamins, Erica N Shook, Sam Norman-Haiinerr, and Josh H Mc- Dermott. A task-optimized neural network replicates human auditory behavior, predicts brain responses, and reveals a cortical processing hierarchy. Neuron, (8(3):630-6444 2018.
Jean Bernard Lasserre. Moments, positive polynomials and their applications, volume 1. World Scientific, 2009.
Simon Laughlin. A simple coding procedure enhances a neuron's information capacity. Zeitschrift füä Naturforschung 36:910-912, 1981.
Daniel D Lee and Sebastian Seung. Learning the parts of objects by non-negative matrix factor- ization. nature, 401(6755);788-791, 1999.
Grace W Lindsay. Convolutional neural networks model of the visual system: Past, present, and future. Journal of cognitive neuroscience, 33(10):2017-231, 2021.
Guillermo Martin-Sánchez, Christian K Machens, and William F Podlaski. Three types of remap- ping with linear decoders: a population-geometric perspective. bioRxiv, pp. 2025-03, 2025.
Mitsuhiro Nishijima and Kazuhide Nakata. Approximation hierarchies for copositive cone over symmetric cone and their comparison. Journal of Globul Optimizution, 88(4):831-87) 2024.
Kenneth Nordström. Convexity of the inverse and moore snore-penrose inverse. Linear algebra and its applications, 434(6):1499-1512, 2011.
Samuel A Ocko, Kiah Hardcastle, Lisa M Giocomo, and Surya Ganguli. Emergent elasticity in the neural code for space. Proceedings of the National Academy of Sciences, 115(50):E1199) 1806, 2018.
Bruno A Olshausen and David J Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381(6583):607-669, 1996.
Cengiz Pehlevan and Dmitri B Chklovskii. Neuroscience-inspired online unsupervised learning algorithms: Artificial neural networks. IEEE Signal Processing Magazine, 36(6):88-96 2019.
Javier Pena. Juan C Vera, and Luis Zuluaga. Completely positive reformulations for polynomial optimization. Mathematical Programming, 451,2)405-431, 2015.
13

---

Mert Pilanci and Tolga Ergen. Neural networks vex optimization formulations for two-layer networks. In International Conference on Machine Learning. pp. 7695-7705. PMLR, 2020.
regularizers: Exact polynomial con-
Arda Sahiner, Tolga Ergen, John Pauly, and Mert Pilanci. Vector-output relu neural network prob- lems are copositive programs: Convex analysis of two layer networks and polynomial-time algo- rithms. arXiv preprint arXiv:2012.13329 2020.
Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynam- ics learning in deep linear neural networks. arXiv preprint urXiv:1312.6/22 2013.
Andrew M Saxe, James L McClelland, and Surya Ganguli. A mathematical theory of semantic development deep neural networks. Proceedings of the National Academy of Sciences, 116 (23):11537-11546, 2019.
Rylan Schaeffer, Mikail Khona, Tzuhsuan Ma, Cristobal Eyzaguirre, Sanmi Koyejo, and Ila Fi- ete. Self-supervised learning of representations for space generates multi-modular grid cells. Advances in Neural Information Processing Systems, 36:23 0023140-23155 2023.
Isaac Schoenberg. Positive definite functions on spheres. Duke Math. J., 9:96-107, 1942.
Vemund Schøyen, Markus Borud Pettersen, Konstantin Holzhausen, Marianne Fyhn, Anders Malthe-Sorenssen, and Mikkel Elle Lepperød. Coherently remapping toroidal cells but not grid cells are responsible for path integration in virtual agents. Iscience, 26(11), 2023.
Anirvan Sengupta, Cengiz Pehlevan, Mariano Tepper, Alexander Genkin, and Dmitri Chklovskii. Manifold-tilinn tillfo localized receptive fields are optimal in similarity-presee preserving neural networks. Advances in neural information processing systems, 31, 2018.
Naum Z Shor. Class of global minimum bounds of polynomial functions. Cybernetics, 23(6): 731-734, 1987.
Samuel Silvey. Optimal design: an introduction to the theory for parameter estimation, volume 1. Springer Science & Business Media, 2013.
Ben Sorscher, Gabriel Mel, Surya Ganguli, and Samuel Ocko. A unified theory for the origin of grid cells through the lens of pattern formation. Advances in neural information processing systems, 32, 2019.
Ben Sorscher, Gabriel C Mel, Samuel A Ocko, Lisa M Giocomo, and Surya Ganguli. A unified theory for the computational and mechanistic origins of grid cells. Neuron, 111(1):211-377, 2023.
Sameet Sreenivasan and Ila Fiete. Grid cells generate an analog error-correcting code for singularly precise neural computation. Nature neuroscience, ((110):330--337) 2011.
Hanne Stensola, Tor Stensola, Trygve Solstad, Kristian Froland, May-Britt Moser, and Edvard I Moser. The entorhinal grid map is discretized. Nature, 492(7427):72-78, 2012.
Peter Sterling and Simon Laughlin. Principles neural design. MIT press, 2015.
David Sussillo and Omri Barak. Opening the black box: low-dimensional dynamics in high- dimensional recurrent neural networks. Neural computation, 25(3):626-649, 2013.
Hirokazu Tanaka. Modeling the motor cortex: Optimality, recurrent neural networks, and spatial dynamics. Neuroscience research, 104:64-71, 2016.
Mufeng Tang, Helen Barron, and Rafal Bogaez. Learning grid cells by predictive coding. arXiv preprint arXiv:2410.01022 2024.
Gokean Tatli and Alper T Erdogan. Generalized polytopic matrix factorization. In ICASSP 2021- 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp- 3235-3239. IEEE, 2021a.
14

---

Gokean Tatli and Alper T Erdogan. Polytopic matrix factorization: Determinant maximization based criterion and identifiability. IEEE Transactions on Signal Processing, 69:5431-5447, 2021b.
Bin Wang, W Jeffrey Johnston, and Stefano Fusi. A mathematical theory for understanding when abstract representations emerge in neural networks, arXiv preprint arXiv:2510.09816, 2025.
Jane X Wang. Zeb Kurth-Nelson, Dharshan Kumaran, Dhruva Tirumala, Hubert Soyer, Joel Leibo, Demis Hassabis, and Matthew Botvinick. Prefrontal cortex as meta-reinforcement learning system. Nature neuroscience, 21(())860-886) 2018.
Xue-Xin Wei, Jason Prentice, and Vijay Balasubramanian. principle of economy predicts the functional architecture of grid cells. Elife, 4:008362, 2015.
James CR Whittington, Will Dorrell, Surya Ganguli, and Timothy Behrens. Disentanglement with biological constraints: A theory of functional cell types. In The Eleventh International Conference on Learning Representations, 2023.
Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan Solomon, Darren Seibert, and James DiCarlo. Performance-optimizer hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences, 111((3):8619-86)) 2014.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances in neural information processing systems, 27, 2014.
Emi Zeger and Mert Pilanci. Unveiling hidden convexity in deep learning: sparse signal process- ing perspective. 2025.
David Zipser and Richard A Andersen. A back-propagation programmed network that simulates response properties of subset posterior parietal neurons, Nature, (355)))))679664 1988.
15

---

A A FAMILY OF CONVEX OBJECTIVES, CONSTRAINTS, AND PROBLEMS
In this section we demonstrate that a suite of interesting problems can be rewritten as convex opti- misation problems over the set of representational similarity matrices. In doing, are inspired by (Sengupta et al. 2018 which, to our knowledge, is the only other result of this sort. Their case considers nonnegative similarity matching. We broaden this to include (nonnegative) linear neural networks, and autoencoders using either a ReLU, unconstrained, or no nonlinearity.
We study family of optimisation problems; each of which we reframe as optimisations over the set of representational similarity. These problems are made by composing a set of objectives and constraints. Since both the sum of convex functions, and the intersection of convex sets, are convex (Boyd & Vandenberghe/ (2004) we show the convexity of each component objectives constraints, which we can then combine to show convexity of large set of problems. We begin by describing the family of problems, then study convexity of each constraint set, before finally studying the convexity of the objectives.
A.1 FAMILY OF PROBLEMS
We consider optimising representation, Z R22x where N is the number of datapoints and d, is the number of neurons. We will assume dg 2 N when needed. We will prove that the following constraints and objectives convex over the set of representational similarity matrices.
CONSTRAINTS
CI Nonnegativity: 2 0 (As in Sengupta et al. (2018)
C2 Bounded Firing Rates: Z S k
C3 Perfect affine decodability of dataset, Y RAXN; W Z + bour Y. (Can be relaxed to linear)
C4 Full rank affine relationship between representation and input dataset such that Y can be decoded, X E Rd.XN; W + 1 = Z, rank(W,,) = d (Can be relaxed to linear)
C5 A ReLU relationship between representation and input dataset, X E RdexN; ReLU(W,,, X + b 1²)
C6 A firing rate constraint: || S kVi. (As in Sengupta et 2018
OBJECTIVES
O1 L2 activity loss: <1|=\|113)
02 A modified LI activity loss: Σd. || 1. where zd E Rdd is neuron d's response vector, when the representation nonnegative (CI)
03 An L2 input weight loss when the relationship with input data is affine (C4): W l
04 An L2 output weight loss when the data is perfectly affine decodable (C3): || l
05 An affine L2 reconstruction error with weight regularisation on the output weights: Xue W ll || Z b 1 Y
06 Similarity matching for a dataset X, as in (Sengupta al. 2018 -Tr[(X'X a11 111))zzz Zz
Nonlinear similarity matching with input similarity G N and elementwise expo- nentiation: Tr(Ge
We list below some problems that can be constructed from this menu:
C3, C4, & 03, 05: A regularised, linear network as in Saxe et al. (2019).
C1, C3, C5, & 02, 05: A regularised one-layer ReLU neural network; such sparse autoencoder.
C1, C3, C4, & 01, 03, 04: A perfectly fitting positive affine autoencoder, as in Whittington et al. (003): Dorrlll al. (2055)
16

---

C1, C3, C4, & 03, 04: A regularised linear neural network with positivity constraint..
CI, C3, & O2, 04/05: A nonlinear but affine-decodable representation such as the last layer of unconstrained neural network.
.
C1, C6, & 06: Nonnegative similarity matching, as in Sengupta et al. (2018).
C1, C6, & 07: representation that nonlinearly matches the data similarity, such as used
to model multi-field place cells (Dorrell et al. 2023).
A.2 CONVEX FEASIBLE SETS
We show the convexity of the feasible set of representational similarity matrices for each constraint.
Cl: Positive Representations Following (Sengupta al. 2018 ) the set of dot product matrices
with unconstrained ranks formed from vectors that are themselves positive (Q - Z" Z for Z > 0) form convex set called the set of completely positive matrices (Berman & Shaaed--Modeee Monderer 2003 As long as we allow many neurons so there is no rank constraint on Q, we re good.
C2: Bounded Firing Rates We generalise slightly the normal proof of convexity of the set of completely positive matrices. Instead, we say that dot-product similarity matrix is made from bounded firing rates if it can be written as the product of representation where all elements are smaller than some constant k: Q z" Z. where Z < k. We can equivalently write this as
Q Et= zzzZ for some number L. Convexly combining such matrices preserves this property:
===1Q+ +1 1) = Σ zaza + (1-
Σ E
(12)
Therefore, since Qx can written as the sum of outer products of vectors that are all smaller than k. it is also member of the set, making the set convex.
C3: Perfect Affine Decodability To affine decodable the representation must contain a sub- space that encodes the demeaned labels. We find the relevant subspace using the reduced SVD: Y = USV The reduced SVD is the same as the SVD but you remove all singular vectors with singular value equal zero. Denoting the rank of Y with p. this gives us diagonal square positive definite matrix S E IRex,, and two rectangular orthogonal matrices U E Rd=Xp, V E RT'xp, Let's break down the representation into the part within the span of V and the part living in the orthogonal complement, Vi Txx2-d
AV + BV1
AdRdX B Rd2xx(d22-1
(13)
In order for the demeaned labels to be linearly decodable A must have full column rank and the columns of B must be linearly independent from the columns of A. These are constraints on the representation. What do they imply for the representational similarity matrices?
ATA
A
VT
Qc
V
Q -zz=1 V1
= V
V1
(14)
AT
B
VT
Q
Q
VT
We therefore relate the two constraints Z two constraints on Q:
1. A having full column rank is equivalent to A A being invertible, this is easy to see.
2. and A having linearly independent columns is equivalent to the generalised schur com- plement of Q = Q QcQ 2C being invertible, this less obvious.
Our proof will proceed in stages. First will demonstrate the equivalence stated in point 2. Then we will show that each of these two properties are preserved under convex combination of the representational similarity matrix. Finally, will relate the demeaned representational similarity back to Q.
Equivalence of Decodability Conditions First go in one direction and show that if QA QeQ8 Qe is singular that implies the column spaces of and B overlap. Singular means:
(Q^ (c)
for some
+
(15)
17

---

We can develop this into:
A- (LL )xx
B²) )x =0
(16)
Now using the fact that BB is the orthogonal projector onto the columnspace of B, and orthogonal projectors are hermitian and idempotent, this implies:
AT BB Az -0
(17)
(1 - BB") is the orthogonal projector onto the kernel of B. Since Ar is a linear combination of the columns A, it lives in the span A. Projecting this vector will only produce a vector with zero component in the span of A (i.e. A' 0) if it is entirely set to O. Hence the condition is that
(1 ) = O. In english, (1 BB Ax = 0 means that Ax lives entirely within the span of the columns of B implying the columns of A and B are not linearly independent.
Now in the reverse direction, if the columns of A and B are not linearly independent then some combination of columns of columns of A equals some combination of columns of B: Aa Bb. Using the previous logic, this implies that:
QQ. QcQcc) A² BB") Aa AT - BB
Bb BB
0
(18)
Since Bb lives in the span of the columns of B so its projection onto the orthogonal complement of B is zero. Hence Q^ Q Qc is non-invertible, demonstrating the stated equivalence.
Convex Combinations Condition 1 Now we show that these two properties are preserved un- der convex combination. Take the convex combination of two matrices, Q1 and Q2, that satisfy conditions and above:
XQ A. + +11 (1- A)Q Q^ XQ + (1-1) = IV V1
XQC,1 + (1 c)Qc22 AQ C,1 + + 1) AQ + (1 - A)Q
V
VT
(19)
where À E [0, 1]. First, the convex combination of two positive definite matrices is positive definite, therefore XQ + - A)Q is positive definite, satisfying the first condition.
Convex Combinations Condition 2 Now we
the convex combinations of nonsingular
generalised schur complements. First we show that the generalised schur complement is matrix concave function a very simple generalisation of an exercise from (Boyd & Vandenberghe 2004) using a proof technique from maths stack exchange user p.s. Then we use that to easily demonstrate the invertibility of the generalised schur complement.
Consider the map that takes positive semidefinite matrix to its generalised schur complement:
TQA
Qc Q
f S
Q =
+ QA - QQQQ
(20)
QT
A function is matrix concave if its the hypograph is convex set (Boyd & Vandenberghe 2004). The hypograph is:
hypof = [(Q QQ) T.Q E S4, ST}
(21)
We make use of the following characterisation of when matrix is positive semidefinite using the generalised schur complement (Gallier 2010 2011):
Q Q
Qc QB
Q =
> 0
=
QB > 0, (1 - Q2QU = Qa Q x 0
(22)
Since QB and (1 exchange:
Q=Q8 Qc 0, we can proceed with the proof as on maths stack
Qa QA-T QT
-T
Qc
T
0
f(Q) =
> T
0
Q
> 0
QB
0
(23)
Defining the linear map:
T
0
l(Q,T)=
= Q
(24)
0
0
18

---

Then:
hypof= = ( S Q S E 5
(25)
And because ST. a convex set, so the hypograph, hence the generalised schur complement matrix concave.
Given that the generalised schur complement matrix concave, convex combinations satisfy:
S(Q + +1 x)Q2) > A/(Q1) + (1)))
(26)
where the last line follows from the invertibility of both f(Q1) and f(Q2). Therefore f(AQ1 +
(1 - 2)Q2) is positive definite and hence convex.
Demeaned to Meaned Q We have therefore shown that the set of Q corresponding affine decod- able representations is convex. Q and Q are clearly related:
Q (z 22 --=====)-=====)))0--0-- - - Q- -
1'Q1 T 11
P(Q) (27)
And, since this is linear in Q. it is easy to check that, matching intuition, the demeaned representa- tional similarity of the convex combination of two representational similarity matrices is the convex combination of their demeaned partners:
P(Qx) = P(AQ (1- (1-))Q) XP(Q) +1 (1- X)P(()
(28)
Hence, a convex combination of affine decodable Q is also affine decodable.
C4: Affine Input
First let's rewrite the affine constraint more simply. Let's define:
x
X'
W = [i
bun such that Z W X'
(29)
Now, take dot product matrices formed from representations that affine functions of the ZT Z. Z W X'. Take their convex combination:
inputs: Z Z, Z W''', and
XQ + x) x (AW, W (1 A)W WWW) A E [0,
(30)
Both w W and W 'T Wi, are positive semi-definite matrices, therefore their convex combination will be as well. Further any positive semidefinite matrix can be decomposed into two parts, therefore
we can write XW W in + (1 A)W W w W for some matrix W Hence
in
XQ + (1 -) = x W = ( (")
2T
(31)
Hence the convex combination of two similarity matrices of affine functions of the input data will also be such similarity matrix.
C5: ReLU Input Let's say you have two representation that are ReLU and affine related to their inputs:
Z = W 1T) - ReLU bm11)
(32)
With their respective representational similarity matrices, Q z Z and Q žT Given enough neurons always create another representation Z, with representational similarity matrix XQ (1=
VAW in XW
1) v1 1b
v Abm 1T 1"
VAz VI AZ
ZA ReLU
x
(33)
C6: Constrained firing rate norm If the diagonals of both Q= 0- and Q -h 7 are below some constant k then the same will be true for convex combinations of Q and Q
19

---

A.3 CONVEX OBJECTIVES
Here we show the convexity of each proposed objective.
01: L2 Activity
The activity loss can be easily rewritten:
1 2 2 = T T+ z
Tr
(34)
T
T
which since is a linear function of Q is clearly convex function.
02: L1 Activity
We define particular form of the L1 activity loss:
|
Σ
18 )
Σ
(35)
d=1
d,i.j
However, if the representation is nonnegative a linear and therefore convex function of Q:
drop the absolute value. Then we see this
Σ d.y.j
b LI dd 2d
M.T
1eu==1 1
(36)
O3: L2 Input Weights (under the constrain that the relationship between input data and represen- tation is affine) We know that Z = Win + bu 1 The work of mapping the mean between these two representations can be least partly done by the bias, correcting any errors left by the linear map. Therefore the min-norm choice of linear map just maps the demeaned data to the demeaned representation:
W zx!
(37)
This makes its frobenius norm:
Tr Tr
ZT =
TT:
ZZ1T)T(Z - -Z11 = - Q]
(38)
since X1 0, and the columns of X has the same span as the rows of X, so 1 T x 0. This is a linear, and therefore convex, function of Q
04: L2 Readout Weights Similarly the previous result, the min-norm readout linear map has to map the demeaned representation to the demeaned data:
W out Z Y
(39)
The min-norm choice is given by the pseudoinverse:
W = 2
(40)
We have to use the pseudoinverse because W.. au is only constrained in subset of directions, in the others we can freely set the weights to zero. The best way to see these two spaces is, as in the previous derivation of the convexity C3, to construct the reduced SVD of the de-meaned data matrix, Y = USV. For a rank matrix this leaves a diagonal square positive definite matrix, S E px and two rectangular orthogonal matrices U E dd V E RT*p, Now let's again break down the representation into the part within the span of V and the part living in the orthogonal complement, V (x22-p))
- TV BVT
A A
B E Rdzx
(41)
In order to be linearly decodable A must be full column rank, and the columns of A must be linearly independent of the columns of B.
Now we can develop the weight loss:
Woud our 11 (x
(42)
Using the SVD of Z it is clear that Zt (Zt)T 2T Q giving:
W W |1: = Tr
(43)
20

---

To show this is a convex function the notion of matrix convexity, A function that maps the set of positive semidefinite matrices, S is matrix convex if:
f RR S
f(0m1 (1-0)m²) ক f (1-0) (m2))
Vm1, m2, 0 E [0, 1] (44)
where denotes the Loewner ordering on positive semidefinite matrices:
m1. m2 E S mi ক
m2
m2 mi E St
(45)
We quote the following result the matrix convexity of the projection pseudoinverse (Silvey) Nordström 2011):
2013
(TQQ + (1- &
AV
Q1.Q2 ST. V E
(46)
Under the condition that the range of the projection within the span of both matrices:
C(V) ((Q:))
(47)
where C denotes the column space or range of a matrix.
We can see that in our case this condition is satisfied due the linear decodability assumption. To see this, let's start with a vector, a E C(V) and show that it is also in the columnspace of any linearly decodable representational similarity matrix. Since a E C(V). a Vb. If a not in C(Q then the following would be zero:
Qa 20-z''z 2Vb = (VA*+V +V_
BV (VA' +V +VA'+V B Ab
(48)
This comprises two terms living in orthogonal spaces, so both must be zero independently, i.e. ATAb 0, and D" Ab. Since A has full column rank, A is invertible. A A = 0 implies that b = 0, which is also in the column space of V, meaning the inclusion is satisfied.
Finally, if a function, f. is matrix convex then x" f(x) x is convex for all æ Boydd & Vandenberghe 2004). We can see that this implies:
W 11 = Tr|S V'QQV = Σ f
(49)
1QQ)
where eд are canonical basis vectors and 5y are the singular values of X. Each eQ ex e is convex thanks to matrix convexity and since the sum of convex functions also convex, we find that the weight loss is convex.
Finally, this is a function on demeaned representational similarity. Changing the mean changes Q, but doesn't 't change the weight loss, so function of the full weight loss the readout weight are first a demeaning projection (it is easy to check this a projection):
1 QUI P. - Q- - -
1 - + Tre 111((1QQ1))
| 11 = T
(50)
Our previous convexity results then apply to the full similarity matrix.
05: Regularised Affine Reconstruction Given fixed Z, we consider the sum of an 1.2 affine reconstruction error and a weight regularisation, again using bar to denote demeaned:
W out Z - 8 + Aue W T
(51)
One can optimise this with respect to the readout weights to find:
out = YZ (Aw T1 I + - = Y(,,,, T1 + 2T 272)-112
(52)
Inserting this into the objective and writing terms of gives:
Tr. Don (A T1 + + - + -1 ((hr +
-1)
-1)
weight loss
(53)
21

---

Inserting = + TT in the place of of Qs in the reconstruction loss let's you cancel the weight loss, leaving:
TT
(() - A..
(54)
To prove convexity we modify the proof technique of maths stack exchage user Robert Israel (Irrael 2013). A function, L, of positive semi-definite matrix, Q is convex if for Q(t) Q + iS where S Is a symmetric matrix:
d (L)Q(() di2
20
(55)
t=
Using the fact that:
(A+1S) - 1 SA + -AA-
(56)
with the identification A Aw 11 Q we find:
d (t))
= > SA 54 SAASA--AAA-AAA' Y
(57)
dt2
Finally, writing = SA -1 " this is:
d' dC((((())
Am Tr/C" A A'C
(58)
d(2
t=
But we know - but positive. This means the loss is not just convex, but strictly convex, guaranteeing a single globally optimal matrix.
(A is positive definite, hence this quantity is not only nonnegative,
06: Similarity Matching Objective In terms of the representational similarity this is:
Tr((X - [M(('''xo11')0 al1"))
(59)
This is linear function of Q so is convex.
07: Nonlinear Similarity Matching We consider the nonlinear similarity matching objective:
Tr|Ge 2| =Tr[Ge
(60)
for -ositive-definit input similarity kernel G E TxT, We can take the second derivative of this with respect to Q:
0CC
Sage Qur
(61)
aQu
Schoenberg 1942) showed that elementwise functions of positive definite matrix are positive def- inite if the function is analytic with all positive coefficients. This tells us that the elementwise expo- nential of postive semidefinite matrix is positive semidefinite. Similarly, the elementwise product of two postive semi-definite matrices postive semi-definite. Hence this derivative is positive and the function is convex.
22

---

B NECESSARY AND SUFFICIENT IDENTIHABILITY OF NONNEGATIVE AFFINE AUTOENCODING
Let's recall optimisation problem:
Problem 1 (Nonnegative Affine Autoencoding), Let gftl E Rd,, x11 R 21 Rd, Win Idd xdv, bin & IRd,, W Wot E IR Xd,, bout E R2te, and A E Rd,xd, where d2 > d, and de 2 da and rank dz. We seek the solution to the following the constrained optimization problem.
min Win Dan W out Dous
(1121113)
+ (1I mnll |Wo.t
(62)
2 Wi + Din: a!
Wout Wo + bout, 2 2 20,
s.t.
where indexes a finite set of samples of s, and x!!! Ast
When will this optimisation problem produce modular solution? In other words, when will each latent neuron be a function of only one source: 2 d, (sk, min, sk )² We will present tight conditions the dataset {slt); and mixing matrix A such that, if this algorithm properly opti- mised, the neurons will recover the sources. This is known as an identifiability result: statement about what has be true about the data such that an algorithm will, principle, succeed extract- ing the groundtruth sources.
[+]
Tight Scattering Condition this work we show that necessary and sufficient condition for nonnegative affine autoencoder to recover the true sources is the following tight scattering condition: Definition (Tight Scattering). Generate the de de-meaned sources: S = S S11" t E IR XT with
elements Sa = s// Assume wlog that min, 2/"]| S max, 3 for each dimension d (if not satisfied simply redefine this source as sa)). Construct the diagonal matrix D E Id xds and the symmetric matrix Fdaxds,
A(A A)
Djj ((3)))2), + (min; , A((AT()))))) ((A*A)-)));
F DD - -
(63)
where E SSS is the covariance. Use these matrices to construct the following set:
E [x F = 1
(64)
is tightly scattered with respect to generating matrix if the following conditions hold:
Conv(S) 1)26 E
Conv(S) bdE {Xe,, + 0 E R. L.....d,) where the * denotes the dual cone, and bd the boundary.
These two conditions relate closely to the sufficient scattering conditions previously derived (Hu & Huang 2023: Tatli & Erdogan 2021 b Huang et al., 2013), but adaptive the structure of A. Finally, it is easy to check that diagonal elements of F are F5 = (min, [s] I)2 which will be useful later.
Main Result Our main theorem that tight scattering a necessary and sufficient condition for biological linear autoencoders recover linearly mixed sources.
Theorem 1 (Identifiability of Nonnegative Affine Autoencoders). Given dataset X AS, if the matrix S is tightly scattered with respect to A then the optimal biological linear autoencoder recovers the sources each neuron's 's activity is affine function of one source and every source has at least one neuron encoding it:
2/ dr (s
min
(65)
Our proof takes the following logic: we find the optimal modular solution (appendix B.I), then we locally perturb into the nearby mixed representations. We derive the above conditions those that
23

---

determine when the optimal modular solution is in fact also local minima of the full representa- tion. appendix B.2 Since the problem is convex, |Section 2 it is also then the global minima. We non-modular global minima. ie. the globally
then show that if these conditions hold there minimum representations are all modular. Finally, we show that in the special case where the sources mixed orthogonally recover the results found by Dorrell et al. (2025) (appendix B.4).
B.1 OPTIMAL MODULAR SOLUTION
An optimal modular representation can be implemented by da neurons, one for each source. Let's first demean the sources, which doesn't change the problem (since we can add or remove bias arbitrarily), but makes notation easier. Then:
de Σ 2=1
d min mins))) S
(66)
M
Where e, are the canonical basis vectors, dj represents the strength the encoding of each source, and the subscript M denotes the fact it is modular. Stacking the dj elements into diagonal matrix D we can derive each of the three loss terms. We know that:
Wine' i]
bm
Win 1 bi
Dz 11 bin
(67)
Hence the min L.2 norm weight matrices are:
Wn DA out w AD-
(68)
where i denotes the pseudoinverse. Hence:
will Tr Wil -
(69)
W our || Tr(D -0a]
(70)
Where we've defined O^ - A, and then A A'(At) O as be seen using the SVD.
Denote with az the diagonal elements of OA, positive numbers representing how strongly partic- ular source encoded the data. Further, denote with (a²)² the diagonal elements of O these represent the lengths of the 'pseudoinverse vectors'. If the encodings of each source are orthogo-
nal they are just the inverse of the encoding sizes, a,,, if not, they also tell you how 'entangled" source's 's representation is. In terms of these variables we can write the whole loss as:
da
C EICK + (min // + {(a3)
(71)
d3
Then the optimal choice of encoding sizes is:
Aa
(dj) (())))))) + (min, +aaa))²
(72)
This corresponds the elements of the matrix D from the definition of tight scattering, definition hence why we gave them the same name.
B.2 TIGHT SCATTERING IS A NECESSARY AND SUFFICIENT CONDITION FOR THE MODULAR REPRESENTATION BE GLOBAL MINIMA
Now let's just go balls to the wall and take the derivative of the whole loss with respect to the linear map that relates the ncurons to the sources, while being careful to preserve the positivity of the representation.
b W Il min W Wsl |
(73)
where the minima is taken nese.n-wise. Then the min-norm weight matrices are: W.. = WAt aut AW, so, defining the covariance matrix = g(0,)) .. we can write the loss as:
C(W)= =
TW W]+ + Σ [min Snin[w,,&e 8l (74)
24

---

where the min is now just the min of set of scalars, and we' ve denoted with w,, the nth row of W. Then the (sub)derivative is:
Oc 2 ow Wwow - -WW
WE + Σ min w s(t) a @mins(ues (w sld)
OW
(75)
Let's evaluate this at the optimal modular representation, which with slight abuse of notation is
W = D (previously D E Rdsxds, but now we'll pad it with zeros so it is D E g xds like W). The biggest immediate change is in the minima term; since:
for n ds Wp = end,
so
min (w و = d, min[s]]]
(76)
Further, the subderivative of the minima of a set of convex functions is within the convex hull of the derivatives of the functions that achieve that minima:
(x) min fx(
@f(x) Convex Hull((( f())))
(77)
So we know that:
a min a s/1]
5n On
an E Convex Hull([s|1|ss] min s
(78)
Owny
i.c the subdifferential vector an lives in the convex hull of the bounding datapoints in direction n. Now we want know if the derivative zero. Let's look the gradient with respect to one row of the matrix W,
Wn:
dC
-dn =, Ad, { ""=
+ d. min[s!] Ja, (79)
2 Ow, W=
Therefore, an equivalent question to establishing whether this gradient is zero is the following ques- tion, is this vector the convex hull of the axis- boiiddonn points?
((DD' D -
E
en E Convex Hull({s min }
min; 8t
J
(80)
We'll now show that, if the sources are tightly scattered, definition the answer is yes. First we do some work to identify the vector on the left-hand side as point on the ellipse is maximally displaced in the direction ee, We consider a vector on the ellipse, Ug. that is maximally displaced along some vector æ. To find we maximise v? x subject to the constraint that w F U2 1.
Performing the resulting lagrange optimisation tells us that:
Fx
(81)
V Fx
Hence using een and the definition of F, get:
Fen
( (w-e,,) min [s(e]
(82)
ming (8
and we see that this point touches the bounding box min along the nith axis. Therefore, if the ellipse is contained in the convex hull of the data, particular if the ellipse kisses the ellipse the minimising point along each axis, which it has to, then the gradient is zero.
So that's good, but we have to go to second order unfortunately. We can derive the horrendous second derivative in index notation, using Einstein summation convention, and 8,q the Kronecker delta:
J02C oW.
-X8 W) W
W W + +8 (Zig ((OA)TTT))
2AW WAS W
W
W) (OA)os W W)
W) Σ dsk i minn w s M
a min f s
(WT., Ww (
OW
(83)
25

---

Now, this is a godawful mess. Fortunately, derived it not you, and we don't have to conclude much from it. First notice that most terms have a osk this means they do not couple the perturbations of different neurons. There only two terms that do. But, notice another aspect of both these two terms: they contain terms of the type Wa and W&S- We are going to evaluate this second derivative W D, and recall that the first dg rows of D are diagonal matrix, and the rest are zeros. So, if both and k, the neuron indices, are below ds, the loss will couple, but for all extra neurons, i > ds, these two coupling terms will be zero, and they will contribute independently!
Therefore we split our evaluation of this quantity at the point W = D into an analysis of the indices k v dg. and the others. Let's begin with the more boring, i, k I dg. Now there's no summation convention being applied any more, every quantity is just scalar.
108 Oe OWW
6 (Eis ((OA)T)) (d (0 (4) (O))(((1)+
(84)
+ +2X51z (G))
+ Σ ds [slaaalaan
First notice that every term contains a kronecker delta over two of the indices and positive semi- definite matrix over the other twe This is all a bit of a pain to look at because we currently have tensor with four indices. To turn it clean matrices we have to vectorise over the two matrix indices. Then each term is just the Kronecker product of two positive semi-definite matrices, since the Kronecker delta is just the identity matrix in index land. And, since the Kronecker product of two positive definite matrices is itself positive definite, this quantity will be too! Finally, we see that the 2nd derivative of the loss with respect to this portion of the linear map is positive definite, therefore all perturbations in these direction increase the loss.
Alternatively, we can take the case of a new neuron,
n2:
2
Eig - M(O) +
+an.
(85)
OW OW nj
Fd
Again, an lives in the convex hull of the datapoints that minimise min, (w,, s()), but we evaluated the derivative the point W D, i.e where w, = 0. As such, all datapoints satisfy this minimia, and a, is just point that lives in the convex hull of the data.
To check whether this quantity will ever be zero we take the directional derivative along some direction wr
L02C
w" Ow, w = - FF + (w" an.)
(86)
So whether this quantity is negative, positive, 0, all depends the convex hull of the data, within which a, lies. If we are able to choose an as the maximising point on the ellipse in the direction w:
Fw
ag
(87)
V
Then the derivative is clearly zero. If a, is larger in the same direction the gradient is positive, and smaller it is negative.
Therefore the first sufficient scattering condition, theorem which guarantees that E C Convex Hull( { Jc), means that this quantity is nonnegative. Further, were this scattering condi- tion not satisfied we would have found a direction which would reduce the loss, hence this condition is a necessary condition for the modular solution to be optimal.
And it is the second scattering condition that circumscribes the set of cases in which this gradient can be O. The only allowed points of tangency between the convex hull and E are those that extremal along a basis direction. This is saying two things. First, all other directions the gradient is positive, and we cannot decrease the loss. Second, we can create new neuron with xx e.vk 1,.... ds and it won't 't change the loss, since the gradient is zero. Since this new neuron is a function of only
E is positive definite, each di is positive number, O A A so it and its inverse are positive definite too, and ag is elearly positive semi-definite
26

---

single source, this preserves the optimal representation's modularity. Were there points of tangency, i.c. the second scattering condition was broken, there would be other perturbations that would leave the loss unchanged but introduce non-modularity, find that this condition is also necessary.
In sum, find that, if both conditions hold, the modular representation is local optima, while each condition is necessary. Hence the pair of conditions are the necessary and sufficient conditions for the optimal modular representation to be local minima of the loss. Since the problem is convex optimisation problem over the set of representational similarity matrices, Q - 2 Z, and a locally optimal linear map implies locally optimal Q, we know that, when these conditions hold, the optimal modular representation is not just local minima but a global minima.
B.3 WHEN IS THE OPTIMAL MODULAR SOLUTION the GLOBAL MINIMA
In Section 2 we saw that out problem is convex optimisation problem. We have found a globally minimal solution, our question is now: are there any other globally minimal solutions and if so are they also modular? We will show that under the tight scattering conditions all global minima are just permutations of the optimal modular solution.
All our representations are affine function of the demeaned data: Z WX + b1 Further, we saw in appendix A that, using the reduced SVD of the demeaned data matrix, X USV? the readout weight loss could be written:
W = Q'VV
(88)
where, denoting the rank of X with p. U E Rdp S E Roxe, V E Rdaxx, Using the properties of the pseudoinverse of products:
Qt (x W WT (VSU WT WUSV VT "''-
(89)
Affine decodability ensures that U W WU is full rank, leading to a simplified form of the loss:
W | = TT((UTTTWT)
(90)
Further, let's break W into two parts, those aligning with U and those orthogonal:
W W + WLU1
(91)
The labels are zero along the U, directions, and so there is no value to a non-zero WI, hence, the loss becomes simply:
W 1|:- Tr[( W
(92)
But, thanks to the positive definiteness of W W, this is now not just convex function of WTW, W, but strictly convex one [Israel 20013). This means that there is a uniquely optimal W W: any other globally optimal representation Z' must therefore have an input linear map that is a(n) (semi- Jorthogonal transform of the optimal modular solution's s, W*
Z' = OW X + b'1" =2 0
(93)
Further, in order to be optimal Z must have the same loss ZM- We already know that they have the same readin and readout loss, since these are both fixed by the choice of readin transform OW*, therefore they must also have the same activity loss. The activity loss of this new representation is:
1 T Th z'| = T Tx W W ||b1|1
(94)
Therefore we conclude that in order to have the same activity loss the new optimal solution's bias vector must have the same length as the original: b' = O'b". Further, we know that in order for
Z' be an optimal representation 6' m (O demeaned optimal modular latent. Therefore, the puzzle becomes which rotations and reflections,
min; [ [ where z!f is the
O, can be apply =|i such that the length of the minimal bias vector doesn't change? We'll now show that if the sources are tightly scattered O must be a permutation.
Since the sources obey the tight scattering assumptions, the optimal modular representation is tightly scattered with respect to a different ellipse, Ez:
E. DF {E|z Dy, FFF= FF-ly== -
(95)
27

---

Let's call the nth row of the orthogonal matrix On, then need to find min, [o) (.. Since the convex hull of the demeaned latents supersets Eg. we can upper bound this quantity minyer, y). y). The element y that achieves this can be calculated through lagrange optimisation to be:
DF O DF
hence
yEB. min [o% - DF
(96)
Therefore calculate lower bound on ||b||, which can be simplified using the orthogonality of O:
||B'1|2 Σ
min WEE.
lo y))
Σ o DFDO, Tr(D²F) = E ( (5 1|6"11
(97)
where the penultimate equality follows from the fact that the diagonal elements of FF are (min;[s) 12) definition in fact this lower bound has to be achieved.
Further, we can now use the second tight scattering condition on the intersection of the boundary constrain this orthogonal matrix. The convex hull of the data only touches the ellipse at points whose tangents are orthogonal to basis direction. If o,, is not basis direction then the inequality is strict: (min; [o, 2011)2 > (minyeE, ()) the bias vectors cannot be the same length, and the solution cannot be optimal. Therefore we derive that each o,, must be along basis vector. The only way to achieve this is if the orthogonal matrix is a permutation.
B.4 ORTHOGONAL ENCODING SPECIAL CASE
Finally, as a corollary we show that the original orthogonally result recovered by Dorrell et al. 2025 fall out under the right conditions. To do this we assume x!!l] sbi This occurs when A 1, O O I; then:
(dz)*
(d)) =1 x (((s)]) )3 + (min s/1))))
(98)
A + ((0) 1): (min; 8, 31
Putting these into the definition of F:
(min; 8 5
15 82 ) (min: 1
AD - (
{8) 82
[1]
(99)
Exactly the F matrix from (Dorrell et al. 2025).
B.5 RELAXATION OF NUMBER OF NEURONS CONSTRAINT
To show the convexity of our problems we had to assume the number of neurons was larger than the number of datapoints: dz T. Restricting the number of neurons further will never improve the globally optimal solution. Therefore, if we can find a globally optimal solution to the many-neuron convex problem that uses a small number of neurons, n, will also be a globally optimal solution to any version of the problem with the neuron a neuron constraint dz 2 This is the case in this problem: under the tight scattering conditions a modular solution using only ds neurons is optimal. Further, if the tight scattering conditions are not satisfied all we need is one extra neuron whose activity we can perturb reduce the loss. This shows that tight scattering is still the necessary and sufficient condition for modularising when as long as dz > dg, as stated in thhoremm
28

---

C EXTENSION TO IMPERFECT RECONSTRUCTION
Here we extend our identifiability results to the imperfect reconstruction setting, i.c. fitting a non- negative affine autoencoder to data using combination of reconstruction and regularisation loss
Problem 2 (Imperfect Nonnegative Affine Autoencoding). Let s(i) E Rd, x R zl1 E R Win E Rd.xd,, dm E R Wout E Rd,xd, bout E R and A E R where d2 d, and dz 2 da and A is rank dg. We seek the solution to the following the constrained optimization problem.
Aa (112111B) Aw [[Wiill++ Wud (l2-) <Il Wo Dout x 20111
min Win bin W Wout bout
1 Win + bin: z 2 0,
S.t.
(100)
where: indexes a finite set of samples of s, and x! = As
We when the optimal solution modularises, i.e. under which conditions does the optimal solution use disjoint set of neurons to encode each source. We derive related tight scattering condition, adapts to the optimal imperfect reconstruction.
Definition 2 (Tight Scattering)- Generate the de-meaned sources: 5 - RRdxxT
with elements Sat =x Assume wlog that min; 8d S max, a14 for each dimension (if not satisfied simply redefine this source as - -84). Numerically find the diagonal positive definite matrix, xds that minimises the following loss:
D Xds
L = Cr(rDD((MMM + + AN A)) AN Tr(O + (ATA)
(101)
Use that construct the following symmetric matrix, Fdaxds:
Aw OA
Au A
F
A AT
-
A)
(102)
S is tightly scattered with respect to a generating matrix A if the following conditions hold with respect the set E fxx|=² F-1 1).
. Conv(S) 2
Conv(S)) n bdF (Xe,, \ F 1...,d where the denotes the dual cone, and bd the boundary.
Our theorem the natural extension of theorem If the data is tightly scattered with respect to this set, the optimal imperfect nonnegative affine autoencoder, as in problem 2 is modular:
Theorem 3 (Identifiability of Imperfect Nonnegative Affine Autoencoders). Given dataset X AS, if the matrix is tightly scattered with respect to A then the optimal biological linear au- toencoder recovers the sources each neuron's 's activity is affine function of one source and every source has at least one neuron encoding it:
2!! dn (s
min
(103)
The proof logic is as before, Jappendix encoded in the matrix D. Then we perturb the solution and show the above condition arises as the one that makes the optimal modular local and therefore global solution. Finally, we take the X and Ag small, but constant ratio, limit to recover the previous result.
first derive the optimal modular solution, which is
C.1 OPTIMAL MODULAR SOLUTION
We consider the set of modular representation:
N M
=
Σ d
min
(104)
1=1
29

---

Denote with D the diagonal matrix made from these weightings. We can write the loss in terms of D. First note that using Y = AS and Z DS, then, using the SVD or otherwise, we can rewrite the combination of output weight loss and reconstruction loss, 05 in appendix A.3) as:
Tr( 71 + Q) = Tr[(A + D2) (AT
(105)
Constructing diagonal matrix M with diagonal elements equal to (min, 5//)2 the full loss is:
L= (r, + AaΣ \ (A² -)] Au Tr( ( A)]
(106)
We optimise numerically, though finding simpler closed form solution does not seem impossible.
We find one further implicit definition of D, which we'll use later, by taking the derivative with respect to (the diagonal elements of) D We find:
dc 2 +
+ Aw A A) -
(107)
So the diagonal elements of this matrix equation equal zero at the optimal modular solution.
C.2 PERTURBING OPTIMAL MODULAR SOLUTION
In general our representations can be written using linear map from the demeaned sources, against which we can then optimise. Calling this map W:
W glil
min
W ]
(108)
Then the loss divided by Xa
Tr(WT + Σ (min w s/1) Aw
(A w AT
(109)
Taking the derivative with respect to one of the rows of W, wn, and dividing by two gives:
wn
w Σ + a min un s +
((A
EA
w W=
Aa
(110)
Where, in appendix B
i
min, @ 11 Ow
an
a, E Convex Hull((s|1))s) min 2
(111)
Evaluating this at the optimal modular solution and setting it to zero find that the condition that this is local minima is:
Dow (Nw D- -2) A' + D2)
A (ATA)) -1
E en E an (112)
min; 81s
[1] Na
Aa
Now identical logic the previous case follows through: if the sources are tightly scattered with respect to the matrix defined by F, this is minima, else it is not. We just need to show one final thing, that the diagonals of are the square minima, i.c. diag( = M This can be seen as follows, rearranging the definition of F we get:
F A, + Amp (A' A)
(A"
"2"'""""" D
(113)
pattern matching to eq. (107) we can see that the diagonals of this equation satisfied F M
Hence, tight scattering relative to this newly defined F again drives identifiability. We show some numerical verification of these results in fig.
Recovering Initial Results Finally, we can see that taking the limit that X2 and Aw are very small (but their ratio is not) let's us approximate (A + D²)-1 simply D in which case recover the previous definition of F. appendix B with
30

---

A) REGULARISATION DEPENDENT IDENTIFIABILITY CONDITIONS
B)
MODULARISATION
MIXING
A Aw 0.025
S1
$2
Neurons
MAX
S2
A Aw = 0.01
S1
0
S1
HIGHLIGHTED NEURON'S RESPONSE
Figure 4: Similarly to fig. in (A) We schematise identifiability conditions for two sources; which either satisfied (red) brocken (blue) by this dataset depending the regularisation hyperapa- rameter. (B) Matching the theory, numerical solutions are modular for the less regularised network (left), but not for the more regularised (right). We plot the linear conditional mutual information (Hsu et al. 2023; Dorrell et al. 2025) between each neuron and source scaled by the neuron's 's peak activity. Below display a highlighted (purple) neuron's tuning sources.
31

---

D IDENTIFIABILITY OF NEURAL TUNING CURVES
We consider strictly convex optimisations over the set of representational dot-product similarity matrices with the constraint of nonnegative neural activities. In this setting there a globally optimal similarity matrix, Q". Let's imagine you measure a nonnegative representation that is optimal, Z 2 z" Z Q'. What has be true about the single neuron properties of Z such that all optimal representations contain the same neurons?
In general, any other optimal representation must be orthogonal transform of your original repre- sentation, Z' = OZ. We consider constraints on Z such that, in order to preserve positivity, O must be a permutation matrix. In this case, the same single neuron responses appear in all representations, tieing them precisely to the optima.
We begin with looser but simpler condition that has occured repeatedly in the literature. Then we show that we can significantly loosen this condition.
D.1 SUFFICIENT SCATTERING
Following classic work on nonnegative matrix factorisation (Donoho & Stodden 2003 Fu et al.
2018 Tatli & Erdogan 2021a b), define the cone:
C [x==""
2|12]
(114)
Which has its corresponding dual cone
C
{ w
||a||2]
(115)
Then our sufficient scattering condition is two conditions. We will use the notion of cone of matrix, all conic combinations of the columns of the matrix:
cone X)= {a| Xa, Va 2 0}
(116)
1. First, sufficient scattering condition that says that the datapoints, i.e. the rows of the matrix Z', are spread around the positive orthant sufficiently:
C C cone [
(117)
2. Second, there few points at which the cone of the datapoints actually touches the edge of the cone C. This can be formalised as:
cone zo bdC {Xe |A 2 k .....N]
(118)
Where bd(C") is the boundary of C*.
Our claim is that, if these two sufficient scattering conditions are satisfied, then the single neuron response properties are unique.
Proof. It is classic result from convex optimisation stuff, for two cones K₁ and K2, if Kr S K2, then their duals satisfy the opposite relation: KCT 2 12z Since C C cone 2d' we therefore have that
C* 2 cone [
Now, imagine you have some other neural representation: Z = oz 2 0 for some orthogonal
matrix O. Now, our first claim is about what has be true about the rows of O such that Z 0. Call the nih row, O., then, since z o, 2 0:
On E cone )² CC
(119)
Now, membership in C* means that of 1 > | |12. Further, we know that ||o,|12 because O n
is an orthogonal matrix, therefore of 1 2 1. In fact, we can go further and show this has to be an equality:
N = 1 = 1 Σ 1)
(120)
*A dual cone for cone is defined C" (uly"" > 0Vn e C}
32

---

So, in order to satisfy the orthogonal matrix property, each 0 1. This is great, because it means that O, E bd(C*). But now the vectors are trapped, we can use the second part the scattering condition. Since each row o, is both in bd(C*) and cone )²') is in its intersection, and this is very preseribed by assumption:
cone Ddd = {dex X 2 k I..,,
(121)
Therefore, each row O is scaled version of the a basis element. But in order to be an orthogonal matrix, in fact this must just be a permutation matrix.
D.2 A FAMILY OF SUFFICIENT CONDITIONS
Theorem (Tight Scattering Implies Unique Neurons). If your neural data, 2 satisfies the fol-
lowing two tight scattering constraints with respect to set E {æ + (= = 13 for positive definite matrix with diagonals equal to ((2 then all orthogonal matrices such that Oz 4 2 are permutation matrices.
. Conv((z, [1] 2
CCov(( J:)" n bd( (Aex|) R k= 1,..,, ds}
Proof. The key constraint is that the transformed data must be positive: Z' OZ 2 0. Consider row of O, On- This has to satisfy:
2f2 0 =
(122)
necessary condition for this to be is the same statement but for members of the ellipse E:
N E
(123)
Given a putative transform vector On, the member of E that will most tax its ability to preserve this defining inequality is the member of the ellipse E. that has the largest projection along the On direction. Using lagrange optimisation we can find that this is the vector:
Fon
(2
(124)
Vo Fon
Inserting this into eq. (123)
Fon
of
(2(1))s = = of Fon + (2())) 20
(125)
Fo,
If this inequality is satisfied then all those in eq. (123) well. Let's rewrite this once more
(o (z ),)² Fo,
(126)
And this must hold concurrently for all the different rows of O. Since O is an orthogonal matrix we know a lot about these rows, i.e. of om 8am- Let's put this information by considering the sum of these inequalities over the population:
(2 O="")===((=0))
|| (M)1
Fo Tr FO Tr[F] = (1)
(127)
Where the last inequality follows from the fact that the diagonal elements of F are ((214)), ),, Fur- ther, since the sums are equal, each of the elements must be equal. Hence, we see that the first sufficient scattering condition satisfied, then this inequality constraint must be an equality:
o
Fo, Fon
+ (31 0
(128)
EE
Now we use the second scattering condition. The positivity constraint, of > OVi, is the defining property of membership of the dual cone of the data: on Conv((z)))) Further, since of 2
33

---

OVR E E, is also in the ellipse's dual cone o, Conv(E)' Finally, not only is on in the dual cone of the convex hull of E, it on the boundary, because, as shown the equation above, o x for point E. Therefore:
On Conv((z:):) n bd((") = (dea/) E 1...,ds
(129)
Where the last equality is the second scattering condition. Hence we have our final result. In order to have the same dot product structure and preserve positivity each on must align with basis direction, and since it is unit norm, it must be a basis direction. So each new neuron Zn o, = 2/ all single unit tuning properties preserved.
D.3 PARTIAL IDENTIFABLE POPULATIONS
In practice, it might be inconvenient to expect every neuron in population to be identifiable perhaps some set of neurons are together encoding one variable and can be easily rotated amongst themselves, but not mixed with another population. We study setting in which such statements can also be proven. We consider an optimisation problem that strictly convex over the de-meaned representational similarity matrices:
Z - Z11
(130)
In this setting, all optimal representations have to be rotations of the demeaned sources plus some bias. Assuming positivity and activity regularisation, you can further prove that to be optimal the minimal bias of the original and rotated sources have to have equal length (see previous section). We now show that, if two sets sources are umge-independent, no representation that mixes them will be optimal.
Let's consider a matrix, O, with orthogonal columns that transforms the de- de-meaned representation while preserving its dot product structure:
z Oz
= 2,
(131)
Let's say that:
x
2.
(132)
y
For two range independent variables x; and ys. Consider an orthogonal matrix that mixes the repre- sentation: O, Then consider breaking the mixed parts into separate populations of neurons:
T 0 O'
0 = O Og
(133)
0 O,
A representation made from O' still has the optimal dot-product similarity, because the columns of O' remain orthogonal. However, we will now show that the representation made from O' must have a shorter bias vector, and therefore the mixed representation cannot be optimal one. Let's calculate the two bias vectors:
||b||2 || 1ilmminn.,,,,t.... yy1l112 min min 1, +min 10y.. yy: > || +||||||+++|||] ,0y
(134)
And the final term is exactly the length of the bias required for the representation constructed from O'. The equality only occurs if min,|O,.a this only happens if O already kept the representation separate. Therefore, any time think find an optimal solution, it can be improved by keeping range-independent variables in different neurons.
min; (O,y. = Since the variables are mean-zero,
34

---

E A TRACTABLE NONLINEAR THEORY OF ON-OFF CODING
We consider a representation, - that is nonnegative and from which single stimulus, x²), can be decoded using an affine readout:
R2 + xb11
(135)
Subject to this encoding constraint and the nonnegativity of the representation we minimise an en- ergy loss:
L [||= 11); + iillk
(136)
In section 2 we showed that this problem is convex when reframed over the set of similarity matrices. We will use the KKT conditions to find necessary conditions on the optimal representations Z (Z denotes the stacked representation matrix, [Z],,
b and similarly [z>] r We will find that the representation either comprises both an ON and an OFF chamnel, or just single channel, and
we will show that sparsity delimits these two scenarios. For these simple representations, it's easy to use the results in section to show that ON-OFF coding cannot be rotated.
E.1 KKT CONDITIONS LEAD TO ON-OFF CODING
The min-norm readout vector, R. is the pseudoinverse that maps the de-meaned representation, Z, the de-meaned data, æ. Using this, we can write the loss as:
1 T
L - T Z a (2)) This has to be minimised subject to non-negativity. so we construct the following lagrangian:
Z
Triz"
Z + Xx= (21) Z
(137)
1 - 2) {
C L=
Tr z
(138)
Slightly non-rigorously, we take the derivative with respect to and, using the expression for the derivative of the pseudoinverse of constant rank (Golub & Pereyra 1973). and we find that:
7 Z 1z-xZ21- (2t) FE (2)) ;
XZž
1 2
(139)
And further, if Zi > then Psj = 0, or if P.j > 0 then Zij = 0. If the former case:
Zij 2 1 'z T Z'(Z")7Ts
(140)
Else it is zero. Since P.j 2 this can all be summarised by the following equation:
0 in these cases we know that Z ZZ (2t) Tax (21)) < 0. Hence,
Zij TZ ( a.." T Z 1 1
(141)
Where [] denotes the elementwise operation [x]+ max(x, 0). We can study the firing a single neuron, Zn IRT::
zn T'|Z ( ( 'x" Z [ yTT zn]+
(142)
Noticing that the same vector appears twice: a = f (Z z. we can rewrite much more simply:
T'aaa" 2,1++
(143)
We can break into two parts, its positive and negative components, a.+ = [a]+, a_ then this equation already tells that in the optimal population there are only two types of neural responses, governed by the sign of a" zn:
[-a] +]+
Tha an if a 2, > 0
Z
T Zn a if a" T nn <
(144)
Now we derive these responses using the definition of a: Z (Z a. Since & is in the span of Z:
Za
Za
(145)
35

---

Now, all neurons belong to these two firing patterns. Call the set of positive neuron indices S+ then define the sum of all positive weightings a. = Enes, |a nn and the same for negative. Then:
z" Za a+ a, a a_
(146)
Hence:
T 1 a'
(147)
Where we' we've ve used the de-meaned variables a+. Expanding these in terms of their means, P4i a= a= + p+l 1 and writing the mean of Ha. we find that, with some rearrangement:
2
1 - -Waaa
1 Tua)a_)) x T2x² (Ma Tuu +W-)+ TX
M
)1
2 + b1
TX
TX
(148)
concurrently non-zero. this tells that they each code for portion of T + b1, a+ the positive, a the negative. We could wade further through this mess, but instead we skip a neuron representation with exactly this form and optimise the remaining parameters directly:
Since a. and a_
a4 [r((]] b]+ a a (a.11) b)]+
1
(149)
Then calculate the loss:
L = (||
115 + la R| = až [1= 11 b]2)s + a² a2 (1-(a11) A
+
(150)
a
We will assume each lies within the range of x (i.e. we are using an ON-OFF code) and take the derivative with respect to each of the to get:
A
a a- (x b)
(151)
And with respect to the bias we end up with the implicit equation:
1 (x b)dp(()
(b-) x))p((r)
(152)
(x-
(b x)) 2
So, in general, solve this implicit equation for (even numerically), then calculate a., and a_ and you have your representation. A simpler solution is that if p(x) symmetric you can see that 6 is a solution.
E.2 DIRECT VS. ON-OFF CODING
The previous result assumed that we had an ON and an OFF neuron each with non-zero firing. The other alternative is single channel, for which we have two choices, ON or OFF:
either: 20 = = min or: 210 -
max !
(153)
We only have to consider the optimal one, which will be the lower energy option. This will be the ON channel if:
((x1 min 211)))) S ((-x()) max m
(154)
and the OFF channel otherwise.
For simplicity let's assume the ON channel is the preferred single neuron response (all the arguments can be switched to the OFF if needed). Let's calculate what has to be true for the one neuron solution to be stable to perturbations into neuron coding.
To do that we compare the one neuron loss for the optimal value of a4 = ((M min, al P))
44 2 - min x(d)))
(155)
To one where we shift the bias the tiny-iest amount positive to create an ON and OFF neuron. Since the dataset is finite, choose this tiny bias to sit between the minimal x value and the
36

---

next smallest. Then the OFF neuron will only activate for the minimal probability of the minimal æ, then the loss:
Let's denote with S the
L = [[ min r!t b12,); + o + X dt
+ =a²
(156)
We can study the second moment:
{[x:1) min 21 b) = ((xf)min min w -0)-)) ((x19 -min x ) 2b(x min +
(157)
And then, using this expression, calculate the optimal a+ and a_ and find, to first order in b:
L = 2v XBS + 2 /((alt min x()) 2b(x(11) min xll));
(158)
Comparing this to the previous loss and series expanding in b:
(x min, x 2/ C- L+ bvs 6 V((xc(t min, x: (. ))))
(159)
2VA
Hence, dual ON-OFF channel coding better if:
(x min, xe 11
$ < ((x.1) min; x: )2),
(160)
else, when the representation is sparse enough, single channel coding is better.
E.3 NUMERICAL DETAILS
To produce fig. 3 we numerically optimised (q.. (136) subject to decoding and nonnegativity con- straints for one-dimensional readout, and plotted the resulting representation. lig. 3C shows measure of linear coding. Specifically, for each neuron we calculate the set of datapoints on which has significantly non-zero activity. We then quantify what proportion of total firing those datapoints encapsulate by taking the norm of the representation matrix restricted to those points divided by the norm of the total representation matrix, Z. We then take the max of this quantity across neurons.
37


Reviews:
[{"review_id": "2FNdW3e4jW", "reviewer": "Reviewer_78Ze", "summary": "This submission develops multiple theoretical contributions in the domain of representation learning, which are made possible by the observation that a class of optimization problems, which may not be convex in the representation itself, are in fact convex in the representation similarity matrix (RSM). The contributions are as follows:\n- Demonstrating that non-negative similarity matching, in tandem with standard regularizers and a linear readout constraint, is a convex optimization problem in the RSM. In a similar spirit the paper discusses how this observation can be extended to describe a larger family of optimization problems.\n- Leveraging this convexity result to derive necessary and sufficient conditions under which non-negative affine autoencoding admits an identifiable (up to permutation) solution. In particular this result holds for more general conditions than previous similar examples in the literature. \n- Leveraging the convexity result to derive sufficient conditions under which single-unit's tuning properties are necessarily unique (i.e. only one set of tuning curves can produce an optimal RSM).", "questions": "- Can the author's expand on their thought in the discussion about the limitation of perfectly fitting the training data? What are the utilities and drawbacks of constraining your view of a system to the set of points that are perfectly fit?\n- Can the author's answer the question from the weaknesses section about the severity of the assumptions used to define problem 1 relative to prior work on this problem? \n- The author's content their \"results sound an optimistic note for classic neuroscience\" but I am not sure this is necessarily the case. For example if the \"true\" generative model of signals encoded by neural systems do not obey the tight scattering condition (or similarly if the conditions of theorem 2 are not met) wouldn't the interpretation be reversed? It is in general not obvious to me which scenario we are in! Either way it is good to know the lay of the land from a theoretical perspective.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The heart of the paper, that optimizations problems that may not be convex in a representation $\\bf{Z}$ can be convex in the RSM $\\bf{Q} = \\bf{Z}^T \\bf{Z}$, is both quite interesting, well placed in the literature of neural circuits (via the anchor of similarity matching objectives), and to my knowledge novel.\n- This key observation is used to substantial effect: it allows for the generalization of previous results in matrix factorization, and can be used to derive interpretable insights into neural tuning properties (at least in a theoretical setting). \n- The presentation of the work is for the most part very clear. The presentation of technical results is coherent, there is some effort to build intuition around these results (particularly in the example about when single-neuron tuning properties are identifiable), and there is a thoughtful effort to couch the contributions of this work in the broader literature (including acknowledging key limitations of the theory as it stands).", "weaknesses": "- Theorem 1 generalizes a previous results on problem 1 to the case of arbitrary (linear) source mixing (from orthogonal linear mixing). It is not obvious to me how significant this generalization is in practice. I.e. the value of this contribution could be made more clear if the author's outlined some real world cases where this contribution would be necessary to deliver theoretical predictions. \n- Regarding the limitations of $N_{neurons} > N_{samples}$ and the assumption of perfect fitting: it was not obvious to me whether the results for factorization with orthogonal source mixing relied on a similar set of assumptions, or if these were relatively stronger or weaker than the dependencies of previous theories?\n- As noted in the limitations section of the discussion, though these problems can be demonstrated to be convex in $\\bf{Q}$ the payoff of this observation in terms of computational approaches may be limited by the fact that the convex set of possible similarity matrices is not well suited for use in current numerical convex optimization solvers.\n- I don't have much of an intuition for the types of circumstances in which the tight scattering condition would hold. Clearly one requirement is that the level set of the relevant quadratic form is elliptical, but it would be nice to develop a clearer way of seeing when this condition holds in terms of the structure of A and S (perhaps even with some practical examples).\n- Nit: Typo in problem statement 1 (~ line 237) $\\bf{b}_{\\text{in}} \\in \\mathbb{R}^{d_x}$ --> $\\bf{b}_{\\text{in}} \\in \\mathbb{R}^{d_z}$", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9tK4GFFjle", "reviewer": "Reviewer_czPt", "summary": "This paper reframes efficient-coding objectives as convex programs over representational similarity matrices. Optimizing over similarity structure, rather than activities, allows the authors to identify broad convex families of efficient-coding problems, including linear, affine, and some nonlinear networks.\n\nTwo key theoretical results follow:\n\n1. Necessary and sufficient identifiability for semi-nonnegative factorization implemented by a nonnegative affine autoencoder with arbitrary mixing matrix A. This yields the first tight \"scattering condition\" guaranteeing recovery of sources up to scaling and permutation.\n2. Uniqueness of single-neuron tuning: sufficient conditions under which nonnegativity plus convex optimality break rotational symmetry, making neuron tuning curves unique up to permutation.\n\nThe paper also provides a catalog of convex constraints (firing cost, readout norm, nonnegativity as complete positivity) and acknowledges limits such as $d_z > N$ and the need for perfect fits.", "questions": "* Under identical datasets and regularization, how does your convex RS-matrix method compare to nonnegative sparse coding and semi-NMF in reconstruction error, sparsity, and tuning modularity? Are there cases where NSC fails but your method succeeds?\n* Does your notion of representational similarity reduce to RSA? If so, how do your claims about single-unit uniqueness hold under more robust geometry measures such as linear CKA?\n* Can you provide an algorithmic test or relaxation for the tight-scattering and dual-cone conditions, and quantify how finite-sample noise affects identifiability?\n* Could classic sparse-coding or convex ReLU formulations reproduce your observed modularity transitions? If not, what falsifiable predictions distinguish your theory?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Originality: Unifies a large class of efficient-coding models under a convex RS-matrix formulation. The \"tight-scattering\" condition provides an elegant, necessary-and-sufficient identifiability test not seen in prior work.\n* Quality: Derivations are correct and geometrically motivated. Proofs connect convex geometry to neural interpretability.\n* Clarity: Writing is clean, with an explicit \"menu\" of convex constraints and objectives.\n* Significance: Offers theoretical justification for single-neuron tuning in convexly optimal codes, which is relevant for both computational neuroscience and interpretable ML", "weaknesses": "* The proposed semi-nonnegative factorization closely mirrors nonnegative sparse coding and semi-NMF [1]. In both cases, only the code (not the dictionary) is nonnegative, which already produces parts-based modularity. The paper should explicitly benchmark against these baselines under matched sparsity or energy constraints, clarifying whether convex RS-matrix optimization yields new behavior or simply a reformulation.\n* The paper equates \"representational similarity\" with geometry preservation, which resembles RSA. However, high RSA correlation does not ensure good single-neuron fits. The authors should complement RSA with CKA [2] or other subspace-alignment measures and report per-unit, sparsity, or selectivity to substantiate claims about unique tuning.\n* It remains unclear whether the key advance is the convex reformulation itself, the new identifiability theorem, or an explanatory account of neural modularity. Competing frameworks (NSC, sparse coding, convex ReLU networks) can explain similar results [3, 4].\n* Demonstrations are illustrative but not quantitative; no finite-sample stress tests or noisy-data robustness.\n* Completely positive programming is computationally intractable in general; implementable relaxations or surrogate constraints are not provided.\n\nRefs:\n\n[1] Hoyer PO (2002). Non-negative sparse coding. In: Proceedings of the 2002 12th IEEE Workshop on Neural Networks for Signal Processing; 2002; Martigny, Switzerland. Piscataway, NJ: IEEE; 2002. p. 557–565.\n\n[2] Kornblith S, Norouzi M, Lee H, Hinton G (2019). Similarity of Neural Network Representations Revisited. ICML 2019\n\n[3] Hoyer PO (2003). Modeling receptive fields with non-negative sparse coding. Neurocomputing 52–54:547–552.\n\n[4] Beyeler M, Rounds EL, Carlson KD, Dutt N, Krichmar JL (2019). Neural correlates of sparse coding and dimensionality reduction. PLoS Comput Biol 15(6): e1006908", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tUyiDzT09E", "reviewer": "Reviewer_TVF6", "summary": "This paper studies a set of constrained optimization problems that linearly encodes the sources subject to resource constraints. Under certain conditions, the optimization problems were found to be convex. Some analytical results on the identifiability of nonnegative affine auto-encoders were presented.  Preliminary implications of the theoretical results to the neuroscience problems were mentioned.", "questions": "Would it be possible to show some real applications of the framework to the biological data?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "****The paper carefully studies a set of constrained optimization problems that are akin to semi-negative matrix factorization. Some analytical results are derived and presented.\n\n****Some discussions on the connections to neuroscience were presented.", "weaknesses": "****The work is quite incremental. The results, in particular the applications of the modeling framework, is rather preliminary. \n\n\n****Despite of the claim that the model framework analyzed is widely used in neuroscience, it is actually quite limited.  Maybe the framework and the results are indeed potentially useful to understanding some phenomena in neuroscience. If that is the case, it would be helpful to show these applications directly in the paper. Section 4 mentioned some potential applications but these are very pereininary. For example, it is mentioned that under certain conditions, one might be able to uniquely constrain the single neuron responses. But it is unclear whether theses conditions would be met in neural systems and whether the theoretical results are indeed capable of revealing new insights into biological systems. \n\n\n****The writing of the paper need substantial improvement. The paper has a lot of jargons, and   difficult to follow at the moment. The results need to be more carefully interpreted, and the implications and limitations need to be more explicitly described.  Overstates need to be toned down. \n\n****It was claimed in the paper that the paper serves to bridge the gap between deep networks and simple linear sparse coding models. But the framework is too simplistic when comparing to deep networks. It is in fact quite similar to the sparse coding models and matrix factorization.  The examples used in the papers are only toy examples and it remain unclear whether the results would generalize to more complex settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CtzE6ucmQI", "reviewer": "Reviewer_Vz3x", "summary": "This paper proposes an intriguing and original analysis of normative approaches to neural coding by casting optimization problems as problems of representational similarity. Using the representational similarity, which describes the population structure, authors show that some known optimization problems are convex, even when they appear not to be convex because of the constraint of non-negative firing rates or because of regularisation. They derive analytical conditions that a network has to obey to be convex. These conditions are crucially related to the scatter (spread) of neural activity vectors. An application of the theory is found in the implication that each neuron in an optimal network has a unique tuning curve, i.e., a tuning curve that does exceedingly overlap with the tuning curve of any other neuron.", "questions": "1) Typically, optimization problems are defined through minimization of a representational error (see for example Olshausen & Field, 1996), while authors here define the optimization problem in Eq. 2 only through the neural activity and decoding weights, requiring that the representation is linearly decodable from the neural activity. How does this ensure that the reconstruction is accurate and thus efficient?\n\n2) Authors first introduce a general optimization problem in Eq. 2, and then a more specific problem (Nonnegative Affine Autoencoding) in Eq. 5. Why is introducing the nonnegative affine autoencoding necessary? Does Tight scattering in definition 3 not solve the problem in Eq.2?\n\n3) In the simplified case where A is an identity matrix (no mixing), does the Tight Scattering condition hold? If so, does it resolve to an intuitive solution?\n\n4) Why does the number of neurons has to be large than the number of datapoints?\n\n5) Why is Frobenius norm used in the loss in Eq. 2? This could be justified when it is introduced, together with the intuition why it is used.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The analysis seems technically correct and carefully conducted. The paper builds on previous work, proposing a refreshing and rather general approach to characterising efficient coding models. There is enough of high-level description and intuition to follow the ideas and the main results. I find that the paper makes useful and insightful contribution on normative approaches to neural coding.", "weaknesses": "1) It remains unclear why would be the solution where each neuron only encodes a single source optimal? This claim also seems rather unrealistic from the point of view of biological neural networks.\n\n2) On some occasions, the definitions are not given much introduction and intuitive explanation. For example, the Definition 3.1 (Tight scattering) comes out of the blue and is not introduced with much justification, let alone intuition and is thus not well prepared. Definitions of the matrix D and F in Definition 3.1 are complex but not much effort is done to explain why they have to be as they are. \n\n3) On several occasions (line [228], for example) authors make a specific claim and promise to \"review\"or \"explain\" later. I find this is not particularly helpful for the reader. If a full explanation feels too lengthy, at least a short and intuitive explanation would fit better than saying it will be explained later. If it is about a commentary that will come later (as is the case of [228]), it would be better to cite the previous work that is of relevance.\n\n4) It remains unclear why do networks have to recover the representation exactly and why they cannot be approximators of some exact solution. \n\nMinor:\nline [152] there is a sum over t and then a normalization with 1/T - not clear.\nline [77-79] \"identifiability of related matrix factorization problems\" reads unclear. Related to what?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)