Adaptive whitening with fast gain modulation and slow synaptic plasticity
Authors: Lyndon Duong
Eero P Simoncelli
Dmitri Chklovskii
David Lipshutz
Venue: NeurIPS 2023 spotlight (2023)
Keywords: neuroscience
adaptation
whitening
efficient coding
recurrent neural network
gain modulation
synaptic plasticity
local learning rules
TL;DR: We propose a neural circuit for adaptive whitening with synapses that learn across statistical contexts and gains that adapt within statistical contexts.

Abstract: Neurons in early sensory areas rapidly adapt to changing sensory statistics, both by normalizing the variance of their individual responses and by reducing correlations between their responses. Together, these transformations may be viewed as an adaptive form of statistical whitening. Existing mechanistic models of adaptive whitening exclusively use either synaptic plasticity or gain modulation as the biological substrate for adaptation; however, on their own, each of these models has significant limitations. In this work, we unify these approaches in a normative multi-timescale mechanistic model that adaptively whitens its responses with complementary computational roles for synaptic plasticity and gain modulation. Gains are modified on a fast timescale to adapt to the current statistical context, whereas synapses are modified on a slow timescale to match structural properties of the input statistics that are invariant across contexts. Our model is derived from a novel multi-timescale whitening objective that factorizes the inverse whitening matrix into basis vectors, which correspond to synaptic weights, and a diagonal matrix, which corresponds to neuronal gains. We test our model on synthetic and natural datasets and find that the synapses learn optimal configurations over long timescales that enable adaptive whitening on short timescales using gain modulation.

Adaptive whitening with fast gain modulation and slow synaptic plasticity
Lyndon R. Duong .2
Eero Simoncelli 1:2 Dmitri B. Chklovskii 1.3
David Lipshutz! !
Center for Computational Neuroscience, Flatiron Institute
Center for Neural Science, New York University
3 Neuroscience Institute, NYU Langone Medical School
(lyndon.duss duong, eero w www.mncelllyoyyy..deee fdchklovskii, dl.pehutzz/ddflaaiironmmmtttttt
Abstract
Neurons in early sensory rapidly adapt to changing sensory statistics, both by normalizing the variance their individual responses and by reducing correlations between their responses. Together, these transformations may be viewed adaptive form of statistical whitening. Existing mechanistic models of adaptive whitening exclusively biological substrate for adaptation; however, on their own, each of these models has significant limitations. In this work, unify these approaches in normative multi-timescale mechanistic model that adaptively whitens its responses with complementary computational roles for synaptic plasticity and gain modulation. Gains are modified on a fast timescale to adapt to the current statistical context, whereas synapses are modified on slow timescale to match structural properties of the input statistics that are invariant across contexts. Our model is derived from whitening objective that factorizes the inverse whitening matrix into basis vectors, which correspond to synaptic weights, and a diagonal matrix, which corresponds to neuronal gains. We test our model on synthetic and natural datasets and find that the synapses learn optimal configurations over long timescales that enable adaptive whitening on short timescales using gain modulation.
either synaptic plasticity gain modulation
the
novel mulli-timeseeee
Introduction
Individual neurons in early sensory areas rapidly adapt to changing sensory statistics by normalizing the variance of their responses [1-3]. At the population level, neurons also adapt by reducing correlations between their responses [4; 5]. These adjustments enable the neurons maximize the information that they transmit by utilizing their entire dynamic range and reducing redundancies in their representations [6-9]. natural normative interpretation of these transformations is adaptive whitening, deeendtt linear transformation of the sensory inputs yielding responses that have unit variance and are uncorrelated.
Decorrelation of the neural responses requires coordination between neurons and the neural mech- anisms underlying such coordination are known. Since neurons communicate via synaptic connections, it is perhaps unsurprising that most existing mechanistic models of adaptive whitening decorrelate neural responses by modifying the strength of these connections [10-16]. However, long-term synaptic plasticity is generally associated with long-term learning and memory [17], and thus may not be suitable biological substrate for adaptive whitening (though short-term synaptic
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

$1
r1
context
context 2
context
context
21
r2
Adapted within contexts: <4 g
Inputs ($1, 82)
Adapted contexts: W o-W
Responses (r1, r2)
Figure I: Adaptive whitening circuit, illustrated with N primary neurons and K 2 internerrons. Left: Dashed ellipses representing the covariance matrices of 2D stimuli s drawn from different statistical contexts. Primary neurons (shaded blue circles) receive feedforward stimulus inputs (shaded purple circles), and recurrent weighted inputs, Wn, from the intereurron (teal circles), producing responses The interneurons receive weighted inputs, z W r, from the primary neurons, which are then multiplied elementwise by gains to generate their outputs, n = o z. The gains g are modulated at fast timescale to adaptively whiten within specific stimulus context. Concurrently, the synaptic weights are optimized at slower timescale to learn structural properties of the imputs across contexts. Right: Dashed unit circles representing the whitened circuit responses r each statistical context.
Center:
plasticity has been reported [18]). On the other hand, there is extensive neuroscience literature on rapid and reversible gain modulation [19-26]. Motivated by this, Duong et al. [27] proposed mechanistic model of adaptive whitening in neural circuit with fixed synaptic connections that adapts exclusively by modifying the gains interneurons that mediate communication between the primary neurons. They demonstrate that appropriate choice of the fixed synaptic weights can both accelerate adaptation and significantly reduce the number interneurons that circuit requires. However, it remains unclear how the circuit learns such an optimal synaptic configuration, which would seem to require synaptic plasticity.
In this study, we combine the learning and adaptation of synapses and gains, respectively, in a unified mechanistic neural circuit model that adaptively whitens its inputs over multiple timescales (Fig. 1). Our main contributions are as follows:
1. We introduce a novel adaptive whitening objective in which the (inverse) whitening matrix is factorized into synaptic weight matrix that is optimized across contexts and diagonal (gain) matrix that is optimized within each statistical context.
2. With this objective, we derive a multi-timescale online algorithm for adaptive whitening that can implemented
neural circuit comprised primary neurons and an auxiliary population of internerrons with slow synaptic plasticity and fast gain modulation (Fig. I).
3. We test our algorithm synthetic and natural datasets, and demonstrate that the synapses learn optimal configurations over long timescales that enable the circuit to adaptively whiten its responses on short timescales exclusively using gain modulation.
Beyond the biological setting, multi-timescale learning and adaptation may also prove important in machine learning systems. For example, Mohan al. [28] introduced "gain-tuning", in which the gains of channels in a deep denoising neural network (with pre-trained synaptic weights) are adjusted improve performance samples with nutoof-distribution noise corruption or signal properties. The normative multi-timescale framework developed here offers a new approach to continual learning and tist adaptation problems such as this.
2 Adaptive symmetric whitening
Consider a neural population with N primary neurons (Fig. 1). The stimulus inputs to the primary neurons are represented by a random N-dimensional vector whose distribution p(s|c) depends

---

on latent context variable The stimulus inputs s can be inputs to peripheral sensory neurons (e.g., the rates at which photons are absorbed by N cones) or inputs to neurons in an early sensory area (e.g., glomerulus inputs to N mitral cells in the olfactory bulb). Context variables can include location (e.g., forest or meadow) or time (e.g., season or time of day). For simplicity, we assume the context-dependent inputs are centered; that is, Paapplale) (s] = 0, where Be--((see(() denotes the expectation over the conditional distribution p(s]c) and O denotes the vector of zeros. See Appx. A for a consolidated list of notation used throughout this work.
The goal of adaptive whitening is to linearly transform the inputs s so that, conditioned on the context variable e, the N-dimensional neural responses have identity covariance matrix; that is,
r r= F such that Exsp(slee rr = IN:
where F, is a context-depenndet Nx N whitening matrix. Whitening is not a unique transformation- left multiplication of the whitening matrix F- by any N x N orthogonal matrix results in another whitening matrix. We focus on symmetric whitening, also referred to as Zero-phase Components Analysis (ZCA) whitening or Mahalanobis whitening, in which the whitening matrix for context is uniquely defined as
F (c),
C.. (c) == (espp(slc) [ss] [ss"],
(1)
where we assume (c) is positive definite for all contexts c. This is the unique whitening transfor- mation that minimizes the mean-squared difference between the inputs and the outputs [29].
To derive an algorithm that learns the symmetric whitening matrix F., we express F. as the solution to an appropriate optimization problem, which is similar to the optimization problem in [15, top of page 6]. For a context e, we can write the inverse symmetric whitening matrix M. == F-1 as the unique optimal solution to the minimization problem
M
arg min fc().
fc(M) == Tr CCe + MM,
(2)
MEa
where $N___ denotes the set of N x N positive definite matrices.' This follows from the fact that [M( strictly convex with its unique minimum achieved M.. where fc(M.) (M,))
(Appx. B.I). Existing recurrent neural circuit models of adaptive whitening solve the minimization problem in Eq. 2 by choosing a matrix factorization of M. and then optimizing the components
[12; 13; 15; 27].
3 Adaptive whitening in neural circuits: a matrix factorization perspective
Here, we review two adaptive whitening objectives, which we then unify into a single objective that adaptively whitens responses across multiple timescales.
3.1 Objective for adaptive whitening via synaptic plasticity
Pehlevan and Chklovskii [12] proposed recurrent neural circuit model that whitens neural responses by adjusting the synaptic weights between the N primary neurons and K 2 N auxiliary interneurons according to Hebbian update rule. Their circuit can be derived by factorizing the context-dependent matrix M. into symmetric product M, = W. W.W. for some context-dependent N x K matrix W. [15]. Substituting this factorization into Eq. results in the synaptic plasticity objective in Table 1. In the recurrent circuit implementation, W denotes the feedforward weight matrix of synapses connecting primary neurons to interneurons and the matrix -W. denotes the feedback weight matrix of synapses connecting interneurons to primary neurons. Importantly, under this formulation, the circuit may reconfigure both the synaptic connections and synaptic strengths each time the context e changes, which runs counter to the prevailing view that synaptic plasticity implements long-term term learning and memory [17].
3.2 Objective for adaptive whitening via gain modulation
Duong et al. [27] proposed a neural circuit model with with synapses that whitens the N primary responses by adjusting the multiplicative gains in set of K auxiliary interneurons. To derive neural
For technical purposes, we extend the definition of fe all @ by setting fc(M) if M o S.
3

---

Table 1: Factorizations of the inverse whitening matrix M. and objectives for adaptive whitening circuits.
Model
Matrix factorization
Objective
Synaptic plasticity [12]
W. W
minw fe (WWT)
Gain modulation [27]
In + (a.diag W
ming Se (IN + (ordiaggg)
Multi-timescale (ours)
oIN (diag(g.) W
minw Bespte) [ming fo (aIN + (d W]
circuit with gain modulation, they considered a novel diagonalization of the inverse whitening matrix, M. = Iy + Wixdiag(g. W fixt where Wix nix an arbitrary, but fixed N K matrix of synaptic weights (with K > KN == N (N + 1)/2) and g. is an adaptive, context-dependent real-valued K-dimensional vector of gains. Note that unlike the conventional eigen-decomposition, the number of elements along the diagonal matrix is significantly larger than the dimensionality of the input space. Substituting this factorization into Eq. 2 results in the gain modulation objective in Table 1. As in the synaptic plasticity model, W denotes the weight matrix of synapses connecting primary neurons to interneurons while Wix fix connects interneurons to primary neurons. In contrast to the synaptic plasticity model, the interneuron outputs are modulated by context-dependent multiplicative gains, ge, that are adaptively adjusted to whiten the circuit responses.
Duong et al. [27] demonstrate that an appropriate choice of the fixed synaptic weight matrix can both accelerate adaptation and significantly reduce the number of interneurons in the circuit. In particular, the gain modulation circuit can whiten any input distribution provided the gains vector ge has dimension K KN (the number of degrees of freedom in an N N symmetric covariance matrix). However, in practice, the circuit need only adapt to input distributions corresponding to natural input statistics [7; 30-32J. For example, the statistics of natural images are approximately translation- invariant, which significantly reduces the degrees of freedom in their covariance matrices, from O(N2) to (NN. Therefore, while the space of all possible correlation structures is Kx-dimensional, the set of natural statistics likely has far fewer degrees of freedom and an optimal selection of the weight matrix W ixx can potentially offer dramatic reductions in the number of interneurons K required to adapt. As example, Duong et al. [27] specify weight matrix for performing "local" whitening with O(N) interneurons when the input correlations are spatially-localized (e.g., as in natural images). However, they do not prescribe method for learning (synaptic) weight matrix that is optimal across the set of natural input statistics.
3.3 Unified objective for adaptive whitening via synaptic plasticity and gain modulation
We unify and generalize the two disparate adaptive whitening approaches [12; 27] in a single multi- timescale nested objective in which gains g are optimized within each context and synaptic weights W are optimized across contexts. In particular, we optimize, with respect to W, the expectation of the objective from [27] (for some fixed K > 1) over the distribution of contexts p(c):
min WERN
Besple) min fo (aIN + (ddag(g)WT))
KEEK
where we have also generalized the objective from [27] by including a fixed multiplicative factor a 2 front of the identity matrix Ly, and we have relaxed the requirement that 2 Ky-
What is an optimal solution of Eq. 3? Since the convex function fo is uniquely minimized at Mc- sufficient condition for the optimality of a synaptic weight matrix W that for each context c, there is a gain vector ge such that a + W diag(g. ) WT = M.. Importantly, under such synaptic configuration, the function fa can attain its minimum exclusively by adjusting the gains vector g. In the space of covariance matrices, we can express the statement as
C..(c) E F(W) == {[oIN Wdiag(g) w TJ² :g E (RK)
nsw for every context e,
where F(W) contains the set of covariance matrices that can be whitened with fixed synapses W and adaptive gains g. Fig. 2 provides an intuitive Venn diagram comparing a non-optimal synaptic configuration Wo and optimal synaptic configuration WT.

---

F(W7)
retual
natural
slow synaptic plasticity W, W
(e)
fest sain modilition
fast gain modallation
50D8 of postive detinate marricss
oode of positive defizite matrices
Illustration of multi cilti--tmmscale learning the space of covariance matrices. Orange and purple regions (identical the left and right) respectively represent the cone of all positive definite matrices S and the subset of matrices that can be whitened with adaptive gains for a particular synaptic weight matrix. On each side, the yellow circle denotes a naturally-occurring input covariance matrix (c) and the dotted white curve illustrates the trajectory of covariance matrices the circuit is adapted to whiten as the gains modulated (with fixed synapses, note the dotted white curve remains in the blue region). Left: With initial synaptic weights W. the circuit cannot whiten some natural input distributions exclusively via gain modulation. i.e., (C.,(c)) E F(Wo). Right: After learning optimal synaptic weights Wr, the circuit can match any naturally- occurring covariance matrix using gain modulation, i.e., (C) C F(Wr).
Figure 2:
covariance matrices (c)) Blue regions represent the set of covariance
4 Multi-timescale adaptive whitening algorithm and circuit implementation
In this section, we derive an online algorithm for optimizing the multi-timesseae objective in Eq. 3, then map the algorithm onto neural circuit with fast gain modulation and slow synaptic plasticity. Direct optimization of Eq. 3 results in an offline (or full batch) algorithm that requires the network to have access the covariance matrices (c), Appx. C. Therefore, derive online algorithm that includes neural dynamics, we first add neural responses r to the objective, which introduces third timescale the objective. We then derive multi-timescale gradient-based algorithm for optimizing the objective.
Adding neural responses to the objective. First, observe that we can write fc(M), for M e st+. in terms of the neural responses r:
(c( = Eese()
max Tr (2rsT - Mrr + M)]
(4)
FERN
To this, maximize
obtain M and then use the definition of Ccs(c) from Eq. 1 (Appx. B.2). Substituting this expression for fe. with M = aIy + Wdiag(g)W" into Eq. 3, dropping the constant term aIy term and using cyclic property of the trace operator results the following objective with 3 nested optimizations (Appx. B.2):
min WERN*X
E Bepple) min Bemp(ele) gEER
max {(W g, r,s) FERN
(5)
where (W.ggr,s)-a 2 Tr a||r||
Σ 9i [(w, |[
The inner-most optimization dynamics. The outer 2 optimizations correspond to the optimizations over the gains g and synaptic weights W from Eq. 3.
corresponds to neural responses and will lead to recurrent neural
To solve Eq. 5 in the online setting, we assume there a timescale separation between neural dynamics and the gain/weight updates. This allows to perform the optimization over before optimizing g and W concurrently. This is biologically sensible: neural responses (e-g., action potential firing) operate much faster timescale than gain modulation and synaptic plasticity [25; 33].
5

---

Recurrent neural dynamics. At each iteration, the circuit receives a stimulus s. We maximize &(W. r. with respect by iterating the following gradient- -sscentt-scent steps that correspond to repeated timesteps of the recurrent circuit (Fig. 1) until the responses equilibrate:
K
- 8
n. or
(6)
where 7/- > 0 is small constant, -w denotes the weighted input to the 7th interneuron, n, = 9.44 denotes the gain-modulated output of the 7th interneuron. For each i, synaptic weights, Wi, connect the primary neurons to the 7th interneuron and symmetric weights, -w,, connect the interneuron to the primary neurons. From Eq. 6, we see that the neural responses are driven by feedforward stimulus imputs s, recurrent weighted feedback from the interneurons -Wn, and leak term -ar.
gth
Fast gain modulation and slow synaptic plasticity. After the neural activities equilibrate, minimize &(W, s) by taking concurrent gradient-descent steps
Ag: = "7 (2? l/w:l²)
(7)
Aw; = The (r74 w, w.))).
(8)
where Mg and The are the respective learning rates for the gains and synaptic weights. By choosing fl >> Tu. we can ensure that the gains are updated at a faster timescale than the synaptic weights.
The update to the 7th interneuron's gain 9s depends on the difference between the online estimate of the variance of its input, =,, and the squared-norm of the 7th synaptic weight vector, ||w,ll², quantities that are both locally available to the 7th interneuron. Using the fact that =; = w, r, we can rewrite the gain update as Ag. = 77 w, (rr' T IN w..]. From this expression, we see that the gains equilibrate when the marginal variance of the responses along the direction w, is 1, for i I...., K.
The update the ( synaptic weight way proportional to the difference between r:n and Wzj9j= which depends only on variables that are available in the pre- and postsynaptic neurons. Since rym; is the product of the pre- and postynaptic activities, we refer this update as Hebbian. Appx. E.2, we decouple the feedforward weights w, and feedback weights ww; and provide conditions under which the symmetry asymptotically holds.
Multi-timescale online algorithm. Combining the neural dynamics, gain modulation and synaptic plasticity yields our online multi-timescall adaptive whitening algorithm, which we express in vector-matrix form with 'o' denoting the Hadamard (elementwise) product of two vectors:
Algorithm 1: Multi-timescale adaptive whitening via synaptic plasticity and gain modulation
1: Input: 81, 82, ... E V
2: Initialize: W E RRNK, g 1) O: ng Tho
3: for ft= =,,2... do
4: F, +-+0
5: while not converged do
6:
Zt W rr:
// interneuron inputs // gain-modul ated interneuron outputs // recurrent neural dynamics
7:
n, - mnggozz
8:
Fi g.trr, Ft + (s: Wn; ar)):
9:
end while
g f g + Ml (z, oz diag (WT 11: W - W + The (ren, Wdiag(g))
// gains update // synaptic weights update
12: end for
Alg. 1 is naturally viewed as a unification and generalization of previously proposed neural circuit
models for adaptation. When a = and the gains g are constant (e.g., ?/ = 0) and identically equal to the vector of ones (so that n, = 2,), we recover the synaptic plasticity algorithm from [15]. Similarly, when a and the synaptic weights W are fixed (e-g., Th = 0), we recover the gain modulation algorithm from [27].

---

10-
A)
Erfer isneothed
2.5
Wo
2.0
1.5
Error
1.0
0.5
0.0
$1
$1
10
10a
1000
2000
3000
5000
Context sample
Stimulus samole
D)
E)
Wr We
Wr
Demsity
0.0
10
10
10
10-1
$1
51
IW-
Figure 3: Adaptive whitening synthetic dataset with 2. 1/ 1E-5, ?o 5E-2. A) Covariance ellipses (orange) out 64 synthesized contexts. Black dashed lines are axes corresponding the column vectors of V. The unit circle is shown in green. Since the column vectors of V are not orthogonal, these covariance matrices do not share common set of eigenvectors (orange lines). B) Whitening error with moving average window spanning the last 10 contexts. C) Error at each stimulus presentation within five different contexts (gray panels), presented with Wo. or Wr. D) Column vectors Wo. Wr, (each axis corresponds the span of column vector in I E) Smoothed distributions of error (in Frobenius norm) between W and V across 250 random initializations of Wo-
5 Numerical experiments
We test Alg. on stimuli $1; ..... drawn from slowly fluctuating latent contexts C1 C2. ...; that is, St ~ p(s]c,) and a = C-1 with high probability." To measure performance, we evaluate the operator norm the difference between the expected response covariance and the identity matrix:
Error(t) = |M, (c. - IN olpp
M = aIN + WWdiagg W.
(9)
Geometrically, this "worst-case" error measures the maximal Euclidean distance between the ellipsoid and the (N 1)-sphere along possible axes. compare
corresponding to (c.(()) two synaptic weight matrices A, B E RN we evaluate || Á Bllr, where À = AP and is the permutation matrix (with possible sign flips) that minimizes the error.
5.1 Synthetic dataset
To validate our model, we first consider 2-dimensional synthetic dataset in which optimal solution is known. Suppose that each context-dependent inverse whitening matrix is of the form
M. = IN + VA(c)V where V is fixed 2 x 2 matrix and (c)) diag A (c), A²(c)) is context-dependent diagonal matrix. Then, in the case a and K = 2, an optimal solution of the objective in Eq. 3 is when the column vectors of W align with the column vectors of V.
To generate this dataset, we chose the column vectors of uniformly from the unit circle, so they are not generally orthogonal. For each context e = 1,... 64, we assume the diagonal entries of A(c) are sparse and i.i.d.: with probability 1/2, (<(c) is to zero and with probability 1/2, As(c) is chosen uniformly from the interval [0, 4). Example covariance matrices from different contexts are shown in Fig. 3A (note that they do not share a common decooposooiiiiinn) Finally, for each context, we generate 1E3 i...d samples with context-dependent distribution ~ (0,M).)
We test Alg. with 1, K 2, The IE-5, and п) 5E-2 on these sequences of synthetic inputs with the column vectors of W chosen uniformly from the unit circle. The model successfully learns whiten the different contexts, as indicated by the decreasing whitening error with the number of contexts presented (Fig. 3B). At the end of training, the synaptic weight matrix Wy. is optimized such that the circuit can adapt changing contexts exclusively by adjusting its gains. This is evidenced by the fact that when the context changes, there is brief spike in eиror as the gains adapt to the new context (Fig. 3C, red line). By contrast, the error remains high many of the contexts when using
Pyttho code accompanying this study be found https://gihubb.om//ynooooo
timescale whitening.
7

---

A)
E)
W:
mod slated Pest
104
Overcamplete
W:
fixed
Flast
modulateed
Enro
100
Irain
Error
10-1
D)
2N
# of Interneurons K
F)
B)
A/ i Q
Count
20
A 1 o
A [Qtc) +
10
Context sample
Basis error
Figure Adaptive whitening of natural images. A) Examples of 2 out of 56 high-resolution images (top) with cach image corresponding to a separate context. For each image, 5 x 5 pixel patches are randomly sampled generate context-dependent stimuli with covariance matrix C..(c) S (bottom). B) Mcan error during training (Eq. 9) with K N 25. Shaded region is standard deviation over 2E3 random initializations Wo E (25). C) Smoothed distributions of average adaptive whitening error over all 2E3 initializations. The red distribution corresponds to the error on the held-out images with fixed learned synapses Wr and modulated gains The blue (resp. green, purple) distribution corresponds to the same error, but tested on the training images (resp. with fixed gains equal to the average gains the final 100 iterations, with fixed random synapses Wo). D) The learned weights (re-shaped columns Wr) approximate orthogonal 2D sinusoids. E) Final error (after T = SE4 iterations) as a function of number interneurons K. Bars are standard deviations centered on the mean error at each K. The red horizontal line denotes average error when K = 0 (in which case = =). F) Frohenius norm between the eigenhasis of Eu--ple) (c,(c) (i.e. across all contexts), Q., with Wr, Wo. and the eigenbasis of each individual context covariance, Q(e), when K N 25. See Appx. for additional experiments.
the initial random synaptic weight matrix Wn (Fig. 3C, blue line). In particular, the synapses learn (across contexts) an optimal configuration in the sense that the column vectors of W learn to align with the column vectors of V over the course of training (Fig. 3DE).
5.2 Natural images dataset
By hand-crafting particular set of synaptic weights, Duong et al. [27] showed that their adaptive whitening network can approximately whiten a dataset of natural image patches with ((N) gain- modulating interneurons instead of O(N2). Here, we show that our model can exploit spatial structure across natural scenes to learn an optimal set of synaptic weights by testing our algorithm on 56 high- resolution natural images [34] (Fig. 4A, top). For each image, which corresponds to a separate context 5 x 5 pixel image patches are randomly sampled and vectorized to generate context- context-dependent samples s E IR 25 with covariance matrix (c) 25 (Fig. 4A, bottom). We train our algorithm in the offline setting where we have direct access to the context-dependent covariance matrices (Appx. C, 50, 74 =5E-1, 7/u -5E-2) with K = N 25 and random W, E ((25) on training set of 50 of the images, presented uniformly at random 1E3 total times. We find that the model successfully learns a basis that enables adaptive whitening across different visual contexts via gain modulation, shown by the decreasing training error (Eq. 9) in Fig. 4B.
Alg. 2, a 1,
How does the network learn leverage statistical structure that is consistent across contexts? We test the circuit with fixed synaptic weights Wr and modulated (adaptive) gains g on stimuli from the held-out images (Fig. 4C, red distribution shows the smoothed error over 2E3 random initializations Wo). The circuit performs as well on the held-out images as on the training images (Fig. 4C, red versus blue distributions). In addition, the circuit with learned synaptic weights WT and modulated gains g outperforms the circuit with learned synaptic weights WT and fixed gains (Fig. 4C, green distribution), and significantly outperforms the circuit with random synaptic weights Wo and modulated gains (Fig. 4C, purple distribution). Together, these results suggest that the circuit

---

learns features Wi that enable the circuit to adaptively whiten across statistical contexts exclusively using gain modulation, and that gain modulation crucial to the circuit's 's ability to adaptively whiten. In Fig. 4D, we visualize the learned filters (columns of Wr), and find that they are approximately equal to the 2D discrete cosine transform (DCT, Appx. D), an orthogonal basis that is known to approximate the eigenvectors of natural image patch covariances [35; 36].
To examine the dependence of performance on the number of internurrons K, we train the algorithm with K = 1,.... 2N and report the final error in Fig. 4E. There is a steady drop in error as K ranges from to N. at which point there is a (discontinuous) drop in error followed by a continued, but more gradual decay in both training and test images error as K ranges from N to 2N (the overcomplete regime). To understand this behavior, note that the covariance matrices of image patches approximately share an eigen- sesennddcooposttion [36]. To see this, let Q(c) denote the orthogonal matrix of eigenvectors corresponding to the context-dependent covariance matrix (c) As shown in Fig. 4F (green histogram), there is small, but non-negligible, difference between the eigenvectors Q(c) and the eigenvectors Q. of the average covariance matrix Tecppte) [c). When K N, the column vectors of WT learn to align with Q, (as shown in Fig. 4F, blue histogram), and the circuit approximately adaptively whitens the context-dependent stimulus inputs via gain modulation. As K ranges from 1 to N, Wr progressively learns the eigenvectors of Q. (Appx. D). Since WT achieves a full set of eigenvectors at K = N, this results in a large drop in error when measured using the operator norm. Finally, as mentioned, there is negligiee difference between the eigenvectors Q(c) and the eigenvectors Q. Therefore, increasing the number of interneurons from N to 2.V allows the circuit to learn an overcomplete set of basis vectors Wr to account for the small deviations between (cc) and Q., resulting in improved whitening error (Appx. D).
6 Discussion
We've derived adaptive statistical whitening circuit from a novel objective (Eq. 3) in which the (inverse) whitening matrix is factorized into components that are optimized at different timescales. This model draws inspiration from the extensive neuroscience literature on rapid gain modulation [25] and long-term synaptic plasticity [17], and concretely proposes complementary roles for these computations: synaptic plasticity facilitates learning features that are invariant across statistical contexts while gain modulation facilitates adaptation within a statistical context. Experimental support for this will come from detailed understanding of natural sensory statistics across statistical contexts and estimates of (changes in) synaptic connectivity from wiring diagrams (e.g., [37] or neural activities (e.g.. [38]).
Our circuit uses local learning rules for the gain and synaptic weight updates that could potentially be implemented in low-power neuromorphic hardware [39] and incorporated into existing mechanistic models of neural circuits with whitened or decorrelated responses [40-45]. However, there are aspects of our circuit that are not biologically realistic. For example, we do not sign-constrain the gains or synaptic weight matrices, so our circuit can violate Dale's law. In addition, the feedforward synaptic weights W and feedback weights W are constrained to be symmetric. In Appx. E, we describe modifications of our model that can make it more biologically realistic. Additionally, while we focus on the potential joint function of gain modulation and synaptic plasticity in adaptation, short-term synaptic plasticity, which operates on similar timescales as gain modulation, has also been reported [18]. Theoretical studies suggest that short-term synaptic plasticity is useful multi-timescale learning tasks [46-48] and it may also contribute to multi-timescale adaptive whitening. Ultimately, support for different adaptation mechanisms will adjudicated by experimental observations.
Our work may also be relevant beyond the biological setting. Decorrelation and whitening transfor- mations are common preprocessing steps in statistical and machine learning methods [49-53], and are useful for preventing representational collapse in recent self-supervised learning methods [54-57]. Therefore, our online multi-timescaa algorithm may be useful for developing adaptive self supervueee learning algorithms. In addition, our work is related to the general problem of online meta-learning [58; 59]: that is, learning methods that can rapidly adapt to new tasks. Our solution- which is closely related to mechanisms of test-time feature gain modulation developed for machine learning models for denoising [28], compression [60; 61], and classification [62]-suggests a general approach to meta-learning inspired by neuroscience: structural properties of the tasks (contexts) are encoded in synaptic weights and adaptation to the current task (context) is achieved by adjusting the gains of individual neurons.

---

References
[1] Naama Brenner, William Bialck, and Rob de Ruyter Van Steveninck. Adaptive rescaling maximizes information transmission. Neuron, 26(3):695-702, 2000.
[2] Adrienne I. Fairhall, Geoffrey D) Lewen, and William Bialek. Efficiency and ambiguity in adaptive neural code. Nature, 412:787-792, 2001.
[3] Katherine I Nagel and Allison J. Doupe. Temporal processing and adaptation in the songbird auditory forebrain. Neuron, 51(6):845-88 859, 2006.
[4] James Muller, Andrew Metha, John Krauskopf, and Peter Lennie. Rapid adaptation in visual cortex to the structure of images. Science, (4054442)::45-1440000 1999.
[5] Andrea Benucci, Aman B Saleem, and Matteo Carandini. Adaptation maintains population homeostasis in primary visual cortex. Nature Neuroscience, 16(6):724-729, 2013.
[6] Fred Attneave. Some informational aspects of visual perception. Psychological Review, 61(3):
183-193, 1954.
[7] H B Barlow. Possible Principles Underlying the Transformations of Sensory Messages. In Sensory Communication, pages 216-234 The MIT Press, 1961.
[8] Simon Laughlin. A Simple Coding Procedure Enhances a Neuron's Information Capacity. Zeitschrift fur Naturforschung C, Journal of Bioxciences, pages 910-2, 1981.
[9] HB B Barlow and P Foldiak. Adaptation and decorrelation in the cortex. The Computing Neuron, pages 54-72. Addison-Westt Wesley, 1989.
[10] Stuart D Wick, Martin T Wiechert, Rainer W Friedrich, and Hermann Riecke. Pattern thogonalization via channel decorrelation by adaptive networks. Journal of Computational Neuroscience, 28(1):29-45 2010.
[11] Paul D King, Joel Zylberberg, and Michael R DeWeese. Inhibitory interneurons decorrelate excitatory cells to drive sparse code formation in a spiking model of V1. Journal of Neuroscience, 33(13):5475-5485 2013.
[12] Cengiz Pehlevan and Dmitri B Chklovskii. normative theory of adaptive dimensionality reduction in neural networks. Advances in Neural Information Processing Systems, 28, 2015.
[13] Cengiz Pehlevan, Anirvan M Sengupta, and Dmitri B Chklovskii. Why do similarity matching objectives lead to Hebbian/anti- Hathr/iaantteeiiaannnna networks? Neural Computation, 30(1):84-122) 2018.
[14] Nikolai M Chapochnikov, Cengiz. Pehlevan, and Dmitri B Chklovskii. Normative and mecha- nistic model of an adaptive circuit for efficient encoding and feature extraction. Proceedings of the National Academy of Sciences, 120(29):e21 11222117444100 74881120, 2023.
[15] David Lipshutz, Cengiz Pehlevan, and Dmitri B Chklovskii. Interneurons accelerate learning dynamics in recurrent neural networks for statistical adaptation. The Eleventh International Conference on Learning Representations, 2023.
[16] Zachary M Westrick, David J Heeger, and Michael $ Landy. Pattern adaptation and normaliza- tion reweighting. Journal of Neuroscience, 36(38):9805-9816 2016.
[17] SP Martin, Paul D Grimwood, and Richard GM Morris. Synaptic plasticity and memory: an evaluation of the hypothesis. Annual Review of Neuroscience, (1())664--11, 2000.
[18] Robert Zucker and Wade G Regehr. Short-term synaptic plasticity. Annual Review of Physiology, 64(1):355-405, 2002.
[19] Larry F Abbott, JA Varela, Kamal Sen, and SB Nelson. Synaptic depression and cortical gain control. Science, (75(5297) 1997.
[20] Emilio Salinas and Peter Thier: Gain modulation: major computational principle of the central nervous system. Newron, 27(1):15-21, 2000.
10

---

[21] Odelia Schwartz and Eero P Simoncelli. Natural signal statistics and sensory gain control. Nature Neuroscience, 4(8):819-822 2001.
[22] Frances Chance, Larry Abbott, and Alex Reyes. Gain modulation from background synaptic input. Neuron, 35(4):773-782, 2002.
[23] Yousheng Shu, Andrea Hasenstaub, Mathilde Badoual, Thierry Bal, and David A McCormick. Barrages of synaptic activity control the gain and sensitivity of cortical neurons. Journal of Neuroscience,
2003.
[24] Pierre-Olivier Polack, Jonathan Friedman, and Peyman Golshani. Cellular mechanisms of brain dete-deepnddnt gain modulation visual cortex. Nature Neuroscience, 16(9):1331-1339
2013.
[25] Katic A Ferguson and Jessica A Cardin. Mechanisms underlying gain modulation in the cortex. Nature Reviews Neuroscience, 21(2):80-92, 2020.
[26] David Wyrick and Luca Mazzucato. State-dependent regulation of cortical processing speed via gain modulation. Journal of Neuroscience, 41(18):3988-4005. 2021.
[27] Lyndon R Duong, David Lipshutz, David Heeger, Dmitri B Chklovskii, and Eero P Simoncelli. Adaptive whitening in neural populations with gain-modulating interneurons. Proceedings of the 40th International Conference on Machine Learning, PMLR, 202:8902-8921, 2023.
[28] Sreyas Mohan, Joshua I. Vincent, Ramon Manzorro, Peter Crozier, Carlos Fernandez- -Gandaa, and Eero Simoncelli. Adaptive denoising via gaintuning. Advances in Neural Information Processing Systems, 34:23727-23740, 2021.
[29] Yonina C Eldar and Alan V Oppenheim. MMSE whitening and subspace whitening. IEEE Transactions on Information Theory, 49(7):1846-1851, 2003.
[30] Eero P Simoncelli and Bruno A Olshausen. Natural image statistics and neural representation. Annual Review of Neuroscience, 1933-1216, 2001.
[31] Deep Ganguli and Eero Simoncelli. Efficient sensory encoding and Bayesian inference with heterogeneous neural populations. Neural computation, 26(10))2133-2134 2014.
[32] Wiktor F Miynarski and Ann M. Hermundstad. Efficient and adaptive sensory codes. Nature Neuroscience, 24(7):998- 10099 2021.
[33] Xiao-Jing Wang, Yinghui Liu, Maria Sanchez-Vives, and David A McCormick. Adapta- tion and temporal decorrelation by single neurons in the primary visual cortex. Journal of Neurophysiology, 89(6):327933299 2003.
[34] JH van Hateren and A van der Schaaf. Independent component filters of natural images compared with simple cells in primary visual cortex. Proceedings: Biological Sciences, 265 (1394):3593366 1998.
[35] Nasir Ahmed, T Natarajan, and Kamisetty R Rao. Discrete cosine transform. IEEE Transactions on Computers, 100(1):90-99) 1974.
[36] David Bull and Fan Zhang. Intelligent Image and Video Compression: Communicating Pictures. Academic Press, London, 2021.
[37] Adrian Wanner and Rainer W Friedrich. Whitening of odor representations by the wiring diagram of the olfactory bulb. Nature Neuroscience, 23(3):433-442, 2020.
[38] Scott Linderman, Christopher H Stock, and Ryan P Adams. A framework for studying synaptic plasticity with neural spike train data. Advances Neural Information Processing Systems, 27, 2014.
11

---

[39] Mike Davies, Narayan Srinivasa, Tsung-Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, Yuyun Liao, Chit- Lin, Andrew Lines, Ruokun Liu, Deepak Mathaikutty, Steven McCoy, Arnab Paul, Jonathan Tse, Guruguhanathan Venkataramanan, Yi-Hsin Weng, Andreas Wild, Yoonseok Yang. and Hong Wang. Loihi: A neuromorphic manycore processor with on-chip learning. IEEE Micro, 38(1): 82-99, 2018.
[40] Cristina Savin, Prashant Joshi, and Jochen Triesch. Independent component analysis in spiking neurons, PLoS Computational Biology. 6(0):e10 2010.
[41] Siavash Golkar, David Lipshutz, Yanis Bahroun, Anirvan Sengupta, and Dmitri Chklovskii. A simple normative network approximates local non non-Heb learning in the cortex. Advances in Neural Information Processing Systems, 33:7283-7295, 2020.
[42] David Lipshutz, Yanis Bahroun, Siavash Golkar, Anirvan M Sengupta, and Dmitri B Chklovskii. biologically plausible neural network for multichannel canonical correlation analysis. Neural Computation, 33(9):2309-233) 2352, 2021.
[43] David Lipshutz, Cengiz Pehlevan, and Dmitri B Chklovskii. Biologically plausible single-layer networks for nonnegative independent component analysis. Biological Cybernetics, 1 116(5-6): 557-568, 2022.
[44] Manu Srinath Halvagal and Friedemann Zenke. The combination of Hebbian and predictive plasticity learns invariant object representations in deep sensory networks. Nature Neuroscience, 2023.
[45] Siavash Golkar, Tiberiu Tesileanu, Yanis Bahroun, Anirvan Sengupta, and Dmitri Chklovskii. Constrained predictive coding as biologically plausible model of the cortical hierarchy, Ad- vances in Neural Information Processing Systems,
2022.
[46] Misha Tsodyks, Klaus Pawelzik, and Henry Markram. Neural networks with dynamic synapses. Neural Computation, 10(4):821-835, 1998.
[47] Nicolas Y Masse, Guangyu R Yang. H Francis Song, Xiao-Jing Wang, and David J Freedman. Circuit mechanisms for the maintenance and manipulation of information in working memory. Nature Neuroscience,, 22((): 1990))119911777 2019.
[48] Kyle Aitken and Stefan Mihalas. Neural population dynamics of computing with synaptic modulations. Elife, 12:e83035, 2023.
[49] Bruno Olshausen and David Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381:607-609, 1996.
[50] Anthony J. Bell and Terrence J. Sejnowski. The "independent components" of natural scenes are edge filters. Vision Research, 37:3327-3338, 1996.
[51] Aapo Hyvärinen and Erkki Oja. Independent component analysis: algorithms and applications. Neural Networks, 13(4-5)4411-433 2000.
[52] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, 2009.
[53] Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsuper- vised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215-223. JMLR Workshop and Conference Proceedings. 2011.
[54] Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self-supervised representation learning. In International Conference on Machine Learning, pages 3015-3024. PMLR, 2021.
[55] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. In International Conference on Machine Learn- ing, pages
PMLR, 2021.
12

---

[56] Tianyu Hua, Wenxiao Wang, Zihui Xue, Sucheng Ren, Yue Wang, and Hang Zhao. On feature decorrelation self-supervised learning. In Proceedings of the IEEE/CVE International Conference Computer Vision, pages 9598-9608, 2021.
[57] Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: ienianceinvarrianc--e covariance regu- larization for self-suuee learning. International Conference on Learning Representations. 2022.
[58] Sebastian Thrun and Lorien Pratt. Learning to Learn. Springer Science & Business Media, 2012.
[59] Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online meta-learning. In International Conference on Machine Learning, pages 1720-1930. PMLR, 2019.
[60] Johannes Ballé, Philip A Chou, David Minnen, Saurabh Singh, Nick Johnston, Eirikur Agusts- son, Sung Jin Hwang, and George Toderici. Nonlinear transform coding. IEEE Journal of Selected Topics in Signal Processing, 15(():339-3533 2020.
[61] Lyndon Duong, Bohan Li, Cheng Chen, and Jingning Han. Multi-rate adaptive transform coding for video compression. In ICASSP 2023-2023 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
[62] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006. 10726, 2020.
[63] John F Kolen and Jordan B Pollack. Backpropagation without weight transport. In Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94), volume pages 1375-1380, IEEE, 1994.
A Notation
For N, 2 let N denote N-dimensional Euclidean space equipped with the usual Euclidean
norm || and let R. denote the non-negative orthant. Let 0 = [0..... O] and = [1.... 1,] respectively denote the vectors of zeros and ones, whose dimensions should be clear from context.
Let N denote the set of N x real-valued matrices. Let || || denote the Frobenius norm and || olop denote the operator norm. Let ((N) denote the of N V orthogonal matrices. Let g (resp. S.Y. ) denote the set of N N symmetric (resp. positive definite) matrices. Let Iy denote the N x N identity matrix.
Given vectors u. E RRW, let uov [uiv1,.... uyeN] E R denote the Hadamard (elementwise) product of and v. Let dia denote the N x N diagonal matrix whose (i, 2)th entry is uy. Given matrix M I N also let diag(M) denote the N v-dimensional vector whose :h entry is M.
B Calculation details
B.1 Minimum of the objective
Here we show that the minimum of for defined in equation 2, is achieved at M. 1/2 2 Diffcrentiating fe(M) with respect to M yields
(M) = C. IN-
(10)
Setting the gradient to zero and solving for yields M C3/2 (c). Substituting into fo yields f(.M)
M.) = (T.M)...
13

---

B.2 Adding neural responses
Here show that equation 4 holds. First, note that the trace term in equation is strictly concave with respect to r (assuming M is positive definite) and setting the derivative equal to zero yields
2s 2Mr = 0.
Therefore, the maximum in equation 4 is achieved at M-1s. Substituting into equation 4 with this form for r, we get
(ssp())) PERN
max Tr (2rs"- Mrr + M)
= p(sp((le) (1.(M-ss sss + M)]
= =Tr Css + MM)
= fc(M),
where the second equality uses the linearity of the expectation and trace operators as well as the formula (c) == Bx-p(s))) (ss This completes the proof that equation 4 holds. Next, using this expression for for we have
fe (aIN Wdiag(g) W Exap(s/c)
max Tr (2rs orr Wdiag(g) W rr) )r
PERN
+ a&
Tr (I((Wiag())))))
= E sss((ee
max (W,ggr,s, + +
FERN
Substituting into equation and dropping the constant oN term results in equation 5.
C Offline multi-timescale adaptive whitening algorithm
We consider an algorithm where directly optimize the objective in equation 3. In this case, assume that the input to the algorithm is sequence of covariance matrices (1), (2) Within each context 2...... we take J > concurrent gradient descent steps with respect g and W:
Ag dydiag [ Vmfc(aIN + Wdiag(g) W W].
AW The (a +
where we assume 17g >> The > 0 as in Algorithm and the gradient of fc(M) with respect to M is given in equation 10. These updates for and W can also be obtained by averaging the corresponding updates in equations 7 and 8 over the conditional distribution p(s|c). This results in Algorithm 2.
Algorithm 2: Offline multi timescale adaptive whitening
1: Input: Covariance matrices C.s (1), C.s (2),... ...
2: Initialize: W E R NxK, gg R 2 0: J 2 1; >> The > 0
3: for c 1, do
4: for j = J.. do
5: M aIN + Wdiag(g) W
6:
g < g -- diag (M))
7: W - w The (M) W diag(g)
8:
end for
9:
end for
D Adaptive whitening of natural images
In this section, we elaborate on the converged structure of WT using natural image patches. To better visualize the relationship between the learned columns of W and sinusoidal basis functions (c.g.
14

---

A)
B)
C)
W
MA
W
WM
WW
MMM
MON
MAM
MAM
Figure 5: Control experiment accompanying Sec. 5.2. A) Wr learned from natural image patches. B) Basis vectors from A displayed as line plots, compared to the ID DCT, and principal components of Ee-p(e) (c)). C) Control condition. Wr learned from spectrally-matched image patches with random eigenvectors.
DCT), focus 1-dimensional image patches (rows of pixels). The results are similar with 2D image patches.
It is well known that eigenvectors of natural images are well-approximated by sinusoidal basis functions [e.g. the DCT: 35: 36). Using the same images from the main text [34], we generated 56 contexts by sampling 16 x 1 pixel patches from separate images, with 2E4 samples each. We train Algorithm with the images, presented uniformly at random T 1E5 times. Fig 5A.B shows that WT approximates the principal components of the aggregated context-dependent covariance, Ee-p(e) [ (c), which closely aligned with the DCT. To show that this structure is inherent in the spatial statistics of natural images, we generated control contexts, (c), by forming covariance matrices with matching eigenspectra, but each with random and distinct eigenvectors. This destroys the structure induced by natural image statistics. Consequently, the learned vectors in WT are no longer sinusoidal (Fig 5C). As result, whitening error with WT is much higher on the training set, with 0.3 + 0.02 error (mean + standard error over 10 random initializations; Eq. 9) on natural image contexts and 2.7 + 0.1 on the control contexts. While for the natural images, a basis approximating the DCT was sufficient to adaptively whiten all contexts in the ensemble, this is not the case for the generated control contexts.
N = 16, Thu = 5E-2, and random Wo (116) on a training set of of
Finally, we find that K increases from K to K 16, the basis vectors in WT progressively learn higher frequency components of the DCT (Fig. 6). This is a sensible solution, due to the l2 reconstruction error of objective, and the 1/f spectral content of natural image statistics. With more flexibility, as K increases past (i.e. the overcomplete regime), the network continues improve its whitening error (Fig. 7A) by learning a basis, WT, that account for within- context information that is insufficiently captured by the DCT (Fig. 7B). Taken together, our model successfully learns basis WT T that exploits the spatial structure present in natural images.
E Modifications for increased biological realism
In this section, we modify Algorithm to be more biologically realistic.
E.1 Enforcing unit norm basis vectors
In our algorithm, there is no constraint on the magnitude of the column vectors of W. We can enforce a unit norm (here measured using the Euclidean norm) constraint by adding Lagrange multipliers to the objective in equation 3:
max Eepp(e) WERNXX & MERK
min
min F RERK
Ea-p(sle [g (W,
(11)
15

---

K=1
K=2
K=3
K=4
K=5
K=6
K=7
K=8
16
16
16
16
K-9
K=10
K=11
K-12
K-13
K=14
K-15
K-16
16
16
16
16
Figure 6: As increases, columns of W progressively learn higher frequency components of the DCT.
A) 101
B)
K=17
K=18
K=19
K=
K=21
K=22
K=
K=24
Error
100
16
32
32
K-25
K-26
K-27
K-28
error
error
16
1.6
K=N
10
20
K
Figure 7: A) Error on training and test function of K. B) In the overcomplete regime, the network learns basis vectors Wr that further improve the error compared with the K S N regime.
where
K
g W, (WW,gr,s)= = ((,,,,,,) + Σ m. ( 1).
Taking partial derivatives with respect to w, and m; results in the updates:
Aw; = Thu (n,r- (gs + m.)w,)
Am; = w; 1|
Furthermore, since the weights are constrained to have unit norm, we can replace ||w;|| with 1 in the gain update:
Ag: (g(2 1).
E.2 Decoupling the feedforward and feedback weights
We replace the primary neuron-to-intermeuro weight matrix W (resp. interneuron-to-prinary neuron weight matrix -W) with W,, (resp. -W.r). In this case, the update rules are
W f- Win + The (n.ry diag(g + m) W..)
W W + De (ran. Wi.diag(g + m))
16

---

A)
B)
Erour (snoothed)
0.8
Frtor
0.4
10P
1000
2000
3000 4000
5000
Stimulue sample
Stimulus sample
C)
D)
Wr
IW.- W..
9.04
51
104
101
Figure 8: With stricter constraints biological realism (Algorithm 3), the model succeeds to whiten synthetic data setup from Section 5.1 when the column vectors V are chosen to be nonnegative. A) Eιror decreases as training progresses. B) Online algorithm performance before and after training. C) The learned basis (row vectors of W.r after learning, red) is well-aligned with the generative basis (dashed black) compared to initialization (row vectors of W nr at initialization, blue). D) Asymmetric feedforward weights, Wr.. and feedback weights, W nr. converge to being symmetric.
Let W rn,t and W nn,r denote the values of the weights W rn and W rewpretiveelll..... after = 0, 0,1, l.... iterates. Then for all t = 0, 1, l....,
W rn, W nn. (W rn.0 W Wnr.0) (IN Tha + m))"
Thus, if + m. (0, 20,2 for all i (e.g., by enforcing non-negative gi, m, and choosing Yh sufficiently small), then the difference decays exponentially in and the feedforward and feedback weights are asymptotically symmetric. This result is similar that of Kolen and Pollack [63], where the authors trained network with weight decay to show that weight symmetry naturally arises to induce bidirectional function synchronization during learning.
E.3
Sign-constraining the synaptic weights and gains
The synaptic weight matrix W and gains vector are not sign-constrained in Algorithm 1, which is not consistent with biological evidence. We can modify the algorithm enforce the sign constraints by rectifying the weights and gains at each step. Here [-]+ denote the elementwise rectification operation. This results in the updates
g ig + 11g (2 1)]+
W + [ rm + The (n.r, diag(g + m) ww) wm)]
W W + The (r.n, W diag(g + m)) m))|4.
E,4 Online algorithm with improved biological realism
Combining these modifications yields our more biologically realistie multi multi-timescale online algorithm, Algorithm 3.
17

---

Algorithm 3: Biologically realistic multi-timescale adaptive whitening
1: Input: $1, $2, ... E I
2: Initialize: Wnr E RNN Wm E RKN, m. E Rf; Tr Tm > 0: fl > ne > 0
3: for t = ....... do
4: I₁ +-0
5: while not converged do
6:
Zt f- Wr
// interneuron inputs // maan-modulated interneuron outputs // recurrent neural dynamics
7:
n, g
8:
rg < Fr + (s,
W ny - or,);
9:
end while
10: m - [m Tm (diag( W Wnr) - 1)]+
// weight normalization update // gains update // synaptic weights update
11:
g + gg + (Ze o 1)]
W rn
W rn + The nnx diag(g mn)]r))+ W + Twe (r.n. (i + m))
13:
W
14:
end for
We test Algorithm 3 on similar synthetic data setup to what was used in section 5.1, except that we sample the column vectors of from the intersection of the unit eircle with the nonnegative quadrant, Figure 8.
18


Reviews:
[{"review_id": "Qen4lBJgiz", "reviewer": "Reviewer_yTCH", "summary": "This paper proposes a neural circuit model that combines fast gain modulation and slow synaptic plasticity to adaptively white sensory inputs. It appears that this paper is a combination of the studies of ref. 11 and 18.", "questions": "- Line 91: ref. 11 requires $K \\geq N$, i.e., the number of I neurons is larger than the E neurons. Does this requirement also need in this paper? Is there a way to make the number of I neurons less than the E neurons? This is also the case in the cortex.\n- It seems that the $f_c(M)$ function was used before but the writing still looks abrupt to me about why to propose an objective function with such a form.\n- The paragraph starting at line 90. It is better to first explain the decomposition $M_c = W_c W_c^T$ regards to the weights between E and I neurons. Otherwise, people can easily regard $W_c$ as the feedforward weights.", "limitations": "I realize the framework assumes the synaptic weights between excitatory (E) and inhibitory (I) neurons are symmetric with each other by just differing a sign. Is there a way to break this symmetry? Will the symmetry breaking of E-I weights facilitate optimization?", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "### Originality \nThe strength of this paper is that it combines the fast gain modulation and slow synaptic plasticity in a whitening neural circuit, and addresses the shortcomings of earlier models with either gain modulation or synaptic plasticity. This is the most conceptual and technical advance of this study. \n\n### Clarity\nOverall, the paper is well written. But see my comments and questions on some parts I am unclear about.", "weaknesses": "I don't have major concerns about this study from the math point of view. Nonetheless, I have a significant concern from the neuroscience point of view.\n\nThe derived updating rule of gain (Eq. after line 174) depends on the current weight. The neural \"gain\" is typically adjusted by inhibitory interneurons, and thus I am concerned about how interneurons \"sense\" the synaptic weights. If there are neurobiological studies supporting this updating rule, please cite the reference in the paper, otherwise, it is better to discuss it at the end.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review and for your thoughtful questions.\n\nWeaknesses:\n\nThank you for voicing this concern. Although it's not so apparent in the main text,  we've resolved this issue in Appdx D.1 . Specifically, we scale the column vectors of ${\\bf W}$ to have constant norm 1. This effectively replaces the $\\text{diag}({\\bf W}^\\top {\\bf W})$ term with a constant vector of ones, so gain updates do not need to \"sense\" the synaptic weights. This is reminiscent of the synaptic scaling/synaptic redistribution mechanisms discussed in Abbott \\& Nelson [2]. \nAdditionally, when the weights are near-optimal, then they are approximately constant across contexts and the target term is effectively constant. \n\nQuestions: \n\n1. Yes, for cases such as in the cortex when $K<N$, it's still possible to produce whitened responses provided the set of (inverse) whitening matrices lie within a $K$-dimensional subspace. Further, some experiments in cortex have shown that primary neurons reduce redundancy after adaptation, but are not perfectly whitened [3]. The partially whitened solution found when $K<N$ may provide an explanation for this effect.\n\n2. Thanks for pointing this out, we will amend the text as you suggest to improve clarity.\n\n3. Thanks, we will make this change in conjunction with the previous one.\n\nLimitations:\n\nWe discuss weight decoupling in Appendix D.2. But it is worth mentioning that reciprocal connections between excitatory and inhibitory neurons have been observed in structures such as the olfactory bulb [1], where dendrodendritic synapses give rise to symmetric connectivity matrices (although, not necessarily symmetric weight matrices). An exploration of circuit function with asymmetric weights (e.g. for non-symmetric whitening transforms) is an interesting direction to pursue. We have some preliminary findings suggest that non-symmetric weights can lead to non-symmetric whitening transformations.\n\n[1] Shepherd, G. M. (Ed.). (2003). *The Synaptic Organization of the Brain*. Oxford University Press.\n\n[2] Abbott, L. F., & Nelson, S. B. (2000). Synaptic plasticity: taming the beast. *Nature Neuroscience*, 3(11), 1178-1183.\n\n[3] Benucci, A., Saleem, A. B., & Carandini, M. (2013). Adaptation maintains population homeostasis in primary visual cortex. *Nature Neuroscience*, 16(6), 724-729."}, {"review_id": "zeG38GKIp9", "reviewer": "Reviewer_3ayL", "summary": "The authors produce a mechanistic model that combines synaptic plasticity and gain modulation to adaptively whiten responses. This model is constructed from an objective for learning a whitening transformation and then considering matrix factorization.  Simple factorization introduces interneurons and optimization via gradient descent can be mapped to synaptic plasticity which adapts the weights to a particular context.  Another decomposition that enforces a particular diagonalization leads whitening via gain modulation when the weights of the interneurons are fixed.  The contribution of this paper is to combine these two mechanisms into a single cost function and network instantiation in which the gain optimization produces a whitening for a given context, and the synaptic plasticity adapts to properties useful over multiple contexts.  The describe an implementation of this optimization in a recurrent neural network and consider different time scales, in which gain adaptation happens quickly within a context and synaptic plasticity happens over a longer timescale.  The authors test this framework on simulated data that matches the decomposition assumptions and show that the method is successful.  They further test it on natural images and show that the learned connectivity successfully adapts to new contexts.  ", "questions": "Can you make any statements about how general the matrix decomposition in Section 3.3 is?  It seems very specific.  You demonstrate numerically that this decomposition works for your experiment.  When do you expect it to fail?  Can you characterize or quantify the differences in context necessary before this decomposition no longer works?  \n\nRelated to this, it looks like you’re using images from the van Hateren database.  Can you try the held out image experiments with images from very different contexts and origins, since those images all have clear similarities in large scale properties?  This might help find the limits of this algorithm/decomposition.  \n\nI was going to ask about breaking the symmetry on the weights, which you answered in an appendix.  They will asymptotically align, but can you say something about this timescale?  Is there a restriction on when the alignment has to happen with respect to the gain modulation and synaptic plasticity that is meaningful or prohibitive?\n\nYou provide the error as a function of number of interneurons for your natural image experiment.  Can you compute some normalization so that there is an idea of what errors are meaningful (maybe the Frobenius norm of C)?  Your interneurons are not necessarily the “interneurons” people talk about in real circuits, but it makes one imagine nonetheless that you will have far fewer interneurons than primary neurons.  Is this a functional problem given the amount of error you see?  Maybe it’s “good enough” even with few interneurons?  ", "limitations": "The authors do not discuss societal impact.\n", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This is a very nice paper that connects the computational problem of whitening with multiple neural mechanisms into a single normative framework.  While it still has issues with full biological plausibility (as noted by the authors) this is an interesting step in thinking about this computation and the roles of these neural mechanisms.  The presentation is very clear (Figure 2 is particularly nice).  \n", "weaknesses": "There are no significant weaknesses in this paper.  To the authors’ credit, every question I had that I thought was in the category of “necessary for publication” they addressed.  I’ll leave everything else to “Questions”, below.\n\nThe biggest weakness I see regards the generality of the matrix decomposition.  The extent to which the authors test or address this appears on par with other approaches in the field, though.  I have some questions below on this topic.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive review and thoughtful questions about the work!\n\nQuestions:\n\n1. We can be precise in describing the set of covariance matrices that can be whitened with this decomposition. In particular, when the inverse whitening matrices lie in a $K$-dimensional linear subspace, then this representation can match the whitening matrices with $K$ interneurons. Since the covariance matrices of natural images approximately share an eigen-basis, this representation can approximately match the corresponding whitening matrices with $K=N$. When the inverse whitening matrices do not lie in a low-dimensional linear subspace, we expect that the representation will require $K$ to be much larger, which can perhaps explain why some circuits have many more interneurons than primary neurons (e.g., olfactory bulb).\n\n2. This is an interesting question which can be answered by appealing to well-known results from the image/video coding literature: a sinusoidal basis (e.g. the DCT) approximates the Principal components (Karhunen Loeve Transform) for natural images [4,5]. Our algorithm learns an approximately orthogonal sinusoidal basis as interneuron weights ${\\bf W}$ without supervision (Fig 4D, 4F, C.1B, C.2); therefore, this trained model is well-suited to any held-out test images (e.g., outside the van Hateren database) whose statistics do not deviate too far from those of natural images.\n\n3. Yes, as we show in Appendix D.2 that the convergence to symmetric weights is exponential. If the weights are order 1, then the exponential convergence rate is on the order $-\\log(1-\\eta_w)\\approx\\eta_w$ when $\\eta_w$ is small. That is, the convergence rate is determined by the learning rate for the synaptic weight updates.\n\n4. Thanks for pointing out this issue of interpretability of the error plots. To provide a better sense of error improvement, we've added a horizontal reference line to show what the error would be with no interneurons (i.e. without whitening).\n\n5.  For olfactory bulb, it seems that this proportion of interneurons is in fact a relevant scenario: interneurons can outnumber excitatory neurons by up to 100:1 [1]. \nWe do show in Fig 4E and Appendix C that our network transitions gradually to the regime where $K\\ll N$. In this regime when $K<N$, the error is indeed significantly improved (relative to no whitening --- horizontal line in PDF Fig E.1).\n\n[1] Shepherd, G. M. (Ed.). (2003). *The Synaptic Organization of the Brain*. Oxford University Press.\n\n[4] Ahmed, N., Natarajan, T., & Rao, K. R. (1974). Discrete cosine transform. *IEEE Transactions on Computers*, 100(1), 90-93.\n\n[5] Bull, D., & Zhang, F. (2021). *Intelligent image and video compression: communicating pictures.* Academic Press."}, {"review_id": "UtgGbLMpYR", "reviewer": "Reviewer_NSc1", "summary": "This paper gives an algorithm for learning weights of a neural network over long time scales, which allow interneurons to decorrelate the responses of excitatory neurons by modulating their gains over short time scales. On both synthetic and natural image datasets, this algorithm is shown to be effective and to generalize to new contexts.", "questions": "* Line 83: Missing a trace on the RHS of the equation?\n* I would appreciate a more in-depth evaluation, perhaps on a synthetic dataset, of what happens when there are fewer interneurons than required.\n* Did you consider adding an activation function, at least to the interneurons? I could imagine that there is a similar learning algorithm which allows for fewer nonlinear interneurons.", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "* This paper makes progress on a fundamental question in neuroscience: How a population of neurons can coordinate their responses to stimuli to encode it. This work should also be of interest in machine learning, as it has obvious applications to transfer learning.\n* Overall, the paper is clear and easy to understand.\n* The learning algorithm is online and local.\n* The technical evaluations are fairly thorough and support the paper's claims.", "weaknesses": "Some weaknesses are inherited from the Duong et al. model which this paper is building on, specifically related to its biological plausibility:\n* All neurons are linear.\n* Dale's law is not observed.\n* Forward and backward weights are mirrored.\n* In general, more interneurons than excitatory neurons are required (in the worst case, as many as $\\Omega(n^2)$ interneurons for $n$ excitatory neurons), which does not match the ratios observed in the brain. Experimentally, the algorithm fails gracefully when fewer interneurons than required are available.\nA more realistic version of the model which addresses the first two issues is provided in the appendix. However, given that the scope of this work is primarily modeling a phenomenon in neuroscience, I think the paper would be stronger if this biologically plausible version were featured and evaluated more prominently.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review and for your useful comments. We appreciate your comment that our model has broader implications for ML and applications to transfer learning.\n\nWeaknesses: We readily acknowledge there are aspects of our model that are not biologically realistic. This is in part due to the fact that the model is derived from an objective that can whiten any input distribution, which is perhaps unrealistic for a biological system. While we did not have space to fully address this in the main paper, in Algorithm 3 of Appendix D, we considered modifications of our circuit to improve its biological realism. \n\n1. Adding a nonlinearity is certainly worth pursuing in a future study. We will update an included reference on related work by Chapochnikov et al. [6] containing analyses on rectifying interneuron responses. \n\n2. In Appendix D.2 we decouple the feedforward and feedback weight updates and in Appendix D.3 we consider a modified synaptic update rule that enforces Dale's law. In Figure E.2 of the attached PDF, we show that the modified algorithm indeed works on synthetic datasets.\n\n3. See #2 above.\n\n4. First, interneurons outnumbering excitatory neurons is in fact a relevant phenomenon observed in neuroscience, and exists in structures such as the olfactory bulb, where interneurons outnumber primary neurons by up to 100:1 [1]. Second, in Figure E.1 of the attached PDF, we show that even when $K\\ll N$, the circuit responses which be much more decorrelated than the circuit inputs. This approximate decorrelation using fewer interneurons can perhaps explain why responses in V1 are only approximately decorrelated [3].\n\nQuestions: \n\n1. Thanks for pointing this out. We have corrected this equation.\n\n2. In Fig 4E (natural images example) and Appendix C, we carefully analyze the effect of having fewer interneurons than required. In Fig 4D, we show that the \"whitening error\" gradually decreases with the number of interneurons $K$ until $K=N$, at which point there is a discontinuous decrease that is due to the fact that we are measuring the difference using the operator norm. We see that even for $K\\approx N/3$, which corresponds to the ratio of interneurons to primary neurons in the cortex, the whitening error is much smaller than that for $K=1$. We have now also included the case with no interneurons ($K=0$) and therefore no whitening - see Figure E.1 of the attached pdf.\n\n3. This is great suggestion for an extension to the model! Adding a nonlinear activation to the interneuron activations may indeed allow for a smaller number of interneurons $K$. Recent related work by Chapochnikov et al. [6] analyzes the role of rectifying interneurons in the whitening olfactory circuit in the context of similarity-matching-based adaptation. \nHere, we were focused on introducing the novel concept of multi-timescale whitening, and wish to keep the computation as analytically tractable as possible with this added layer of complexity. Any definitive claims on the whitening capability of our network with nonlinearities warrants a separate, follow-up study with extensive numerical/empirical investigation.\n\n[1] Shepherd, G. M. (Ed.). (2003). *The Synaptic Organization of the Brain*. Oxford University Press.\n\n[3] Benucci, A., Saleem, A. B., & Carandini, M. (2013). Adaptation maintains population homeostasis in primary visual cortex. *Nature Neuroscience*, 16(6), 724-729.\n\n[6] Chapochnikov, N. M., Pehlevan, C., & Chklovskii, D. B. (2023). Normative and mechanistic model of an adaptive circuit for efficient encoding and feature extraction. *Proceedings of the National Academy of Sciences*, 120(29), e2117484120."}, {"review_id": "XuoGA5m7ty", "reviewer": "Reviewer_yS92", "summary": "This paper proposes a normative principle for the symmetric whitening problem, i.e., a batch optimization problem which is provable to attain symmetric whitening is used to derive an online adaptive algorithm. The proposed framework unifies the previous works, and the resulting algorithm maps into single layer network with interneurons and local learning rules. The mathematical findings of this normative approach is also supported with the evidences from neuroscience. The proposed approached is tested with synthetic data and natural images to demonstrate the performance. Potential improvements for more biologically realistic method is also discussed.", "questions": "* The numerical experiments presented in Appendix C uses Algorithm 2, but the biological plausibility of it is questionable due to matrix inversion and batch learning. Did you experiment with this setup using Algorithm 1?\n\n* The authors propose Algorithm 3 as more biologically plausible algorithm compared to the Algorithm 1, by complying to the Dale's law and decoupling the feedforward and feedback weights for asymmetry (weight transport problem). However, no numerical experiment has been demonstrated for this proposal. Did you test this algorithm?\n\n* The value of $\\alpha$ seems to be also an hyperparameter for the proposed framework. Did you analyze the effect of this choice?", "limitations": "I think that the limitations are adequately discussed in the paper. \n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper provides a neural network solution to the whitening problem which satisfies two important constraints for biological plausibility: i) the network operates in an online manner, and ii) the weight and gain updates are local. The proposed online algorithm is illustrated to obtain whitening with both synthetic and natural datasets.", "weaknesses": "The writing is lacking in some parts, and some important details are missing in the paper. More numerical experiments should be presented to assess the introduced framework. For exact details on the weaknesses, please see my comments below and questions section.\n\n   * Line 83, \"... where $f_c(\\mathbf{M}_c) = 2 \\mathbf{M}_c$\" should be changed to \"... where $f_c(\\mathbf{M}_c) = \\text{Tr}(2 \\mathbf{M}_c)$\" since $f_c$ is a scalar function of matrices. Moreover, even though it is a simple proof, it would be valuable how this unique minimum is achieved. \n    \n   * The intermediate step (stated between line 150 and line 153) to obtain Equation (4) should be written \texplicitly in the paper (can be in the appendix).\n    \n   * The decoupling the feedforward and feedback weights (in Appendix D.2) is very similar to the idea presented in [i] since the assumption $g_i + m_i \\in (0, 2 \\eta_w^{-1})$ corresponds to weight decaying. The paper should cite this reference.\n\n    [i]J.F. Kolen and J.B. Pollack. Backpropagation without weight transport. In Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN’94), volume 3, pages 1375–1380 vol.3, 1994. doi: 10.1109/ICNN.1994.374486.\n    \n   * For the experiment with synthetic data, the experiment only considers the whitening matrix of the form $\\mathbf{M}_c = \\mathbf{I}_N + \\mathbf{V} \\mathbf{\\Lambda}(c)\\mathbf{V}^T$. An experiment with more general form of whitening matrix is required to assess the performance of the proposed algorithm.\n\n   * Hyperparameters (i.e., $\\eta_w, \\eta_g, \\eta_r$) are not provided in the paper (except that the authors provide $\\eta_w$ for Algorithm 2 in Appendix C). It would be beneficial for the paper to include a table that provides the hyperparameters used in each experiment\n\n   * The differences and novelties compared to previous works by Pehlevan and Chklovskii [11] and Duong et al. [18] are discussed quite well, but the comparison in the numerical experiments is missing.\n\n   * A single Python script is shared for the proposed method, although it does not include the experiment code. The authors mentioned in the paper that the full code would be shared upon publication. However, the current code does not provide a comprehensive understanding of the conducted experiments.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful reading of our work and for your suggestions. We regret that you found the writing lacking in some parts. We have revised our paper in accordance with your suggestions, which we believe has improved the overall clarity of the paper.\n\nWe are concerned that a central contribution of our work may not have been appreciated. Our primary motivation is to develop a biologically realistic circuit for adaptive, _context-dependent_ whitening with complementary computational roles for synaptic plasticity and gain modulation. We formalize existing mechanisms of gain modulation and synaptic plasticity into a joint computation operating across multiple timescales, yielding a new mechanistic theory and solution to context-dependent adaptation in neural circuits. We believe this is a novel and significant contribution, which we will emphasize in our revision.\n\nWeaknesses:\n\n1. We corrected the equation and added a short proof that explains how the minimum is achieved.\n\n2. We added a short section to the appendix with the intermediate steps written explicitly.\n\n3. Thank you for pointing us to reference [i]. It is indeed relevant and we've added a citation.\n\n4. A strength of our normative approach is that its performance can be predicted by analyzing the starting objective. In particular, given ${\\bf W}$, we can analytically describe the set of covariance matrices that can be whitened by the circuit (see the display after eq 3). Our numerical simulations verify this analytical prediction. Given the analytical nature of our results, we believe that the current set of experiments adequately validates the performance of the circuit.\n\n5. We amended the text to include a table of these hyperparameters.\n\n6. The Pehlevan and Chklovskii [11] and Duong et al. [18] models, which respectively use either synaptic plasticity or gain modulation, correspond to our network in the regimes $\\eta_g=0$ and $\\eta_w=0$, respectively. A comparison to these regimes is shown in Fig. 4C, where we quantify the test error for our model (red histogram), for fixed gains [11] (green histogram) and for fixed synapses [18] (purple histogram). We amended the text to emphasize this comparison to previous work.\n\n7. We have submitted full code (to the AC via an anonymous link, as instructed) for the synthetic data experiments.\n\nQuestions:\n\n1. Yes, using the Algorithm 1 for e.g. Appendix Fig C.1 still learns an approximately orthogonal sinusoidal basis for the data. The main difference is that the online algorithm requires far more iterations than the offline algorithm.\n\n2. Yes, see Fig E.2 in the rebuttal PDF. \n\n3. Yes, we've analyzed the $\\alpha$ parameter, which corresponds to the leak term in the neural dynamics. In the context of the synthetic experiment when the generative basis, ${\\bf V}$, is not orthogonal, a non-zero $\\alpha$ parameter is necessary (effectively due to the matrix inversion); when ${\\bf V}$ is orthogonal, $\\alpha$ is less important and can be set to zero. See Fig E.3 in rebuttal PDF."}]

Author rebuttal:
Thank you for your careful reading of our work and for your helpful comments. We have revised our paper in accordance with your suggestions and provide individual responses below. Here we list general changes and additions to the manuscript.

1. **Adaptation with fewer interneurons than primary neurons:** As the reviewers correctly pointed out, many circuits in the brain have far fewer interneurons than primary neurons (e.g., in the cortex the ratio is approximately 1:3). In Figure E.1 of the attached PDF (which is an updated version of Figure 4E from our original submission to include the case $K=0$), we show the performance of our algorithm on the images dataset when $K<N$. We see that even when there are relatively few interneurons (e.g., $K=4$), the whitening error is much less than when $K=0$. Perhaps neural circuits balance the benefits of reducing response correlations with the increased metabolic costs of having more interneurons. Finally, we mention that there are neural circuits in the brain where interneurons greatly outnumber primary neurons; e.g., in the olfactory bulb where the ratio of granule cells to mitral cells is upwards of 100:1 [1].
2. **Biological realism:** There are aspects of our model that are not biologically realistic. In our original submission, we briefly listed these aspects in the Discussion and in more depth in Appendix D, where we presented a more biologically realistic online algorithm (Algorithm 3).  We have now tested Algorithm 3 on a synthetic dataset and found that it successfully learned optimal filters and was able to adaptively whiten the data using gain modulation, see Figure E.2 of the attached PDF.
3. **Effect of hyperparameter $\alpha$:** We tested the effect of varying the $\alpha$ parameter on the synthetic dataset, Figure E.3 of the attached PDF. We find that when the column vectors of ${\bf V}$ are orthogonal, varying $\alpha$ does not affect the performance of the algorithm; however, when ${\bf V}$ are not orthogonal, there is a *slight* degradation of performance when $\alpha$ is not exactly 1. This is due to the fact that if ${\bf V}$ is orthogonal (and full rank), then $\alpha{\bf I}+{\bf V}\Lambda {\bf V}^\top$ can be decomposed as ${\bf V}(\alpha{\bf I}+\Lambda){\bf V}^\top$, so the basis vectors for the (inverse) whitening matrix do not depend on $\alpha$.

[1] Shepherd, G. M. (Ed.). (2003). *The Synaptic Organization of the Brain*. Oxford University Press.

Meta-review: This paper examines the question of how neural circuits engage in whitening, i.e. normalizing the variance of responses and reducing correlations between responses. In this work, the authors build a model of neural whitening that uses fast gain adaptation and slow synaptic mechanisms. They test the algorithm on synthetic and natural datasets, and demonstrate that the synapses learn optimal solutions over long timescales that enable the circuit to adaptively whiten responses on short timescales using gain modulation.

The reviewers all agreed that this paper make an interesting, novel contribution and is overall well-written. The reviewers had some concerns related to the biological links and the tests run to compare the model, but the concerns were largely addressed by the rebuttals. Thus, this is a clear accept.
Decision: Accept (spotlight)