Interneurons accelerate learning dynamics in recurrent neural networks for statistical adaptation
Authors: David Lipshutz
Cengiz Pehlevan
Dmitri Chklovskii
Venue: ICLR 2023 poster (2023)
Keywords: Interneurons
recurrent neural networks
gradient flows
implicit acceleration
statistical whitening

Abstract: Early sensory systems in the brain rapidly adapt to fluctuating input statistics, which requires recurrent communication between neurons. Mechanistically, such recurrent communication is often indirect and mediated by local interneurons. In this work, we explore the computational benefits of mediating recurrent communication via interneurons compared with direct recurrent connections. To this end, we consider two mathematically tractable recurrent neural networks that statistically whiten their inputs --- one with direct recurrent connections and the other with interneurons that mediate recurrent communication. By analyzing the corresponding continuous synaptic dynamics and numerically simulating the networks, we show that the network with interneurons is more robust to initialization than the network with direct recurrent connections in the sense that the convergence time for the synaptic dynamics in the network with interneurons (resp. direct recurrent connections) scales logarithmically (resp. linearly) with the spectrum of their initialization. Our results suggest that interneurons are computationally useful for rapid adaptation to changing input statistics. Interestingly, the network with interneurons is an overparameterized solution of the whitening objective for the network with direct recurrent connections, so our results can be viewed as a recurrent neural network analogue of the implicit acceleration phenomenon observed in overparameterized feedforward linear networks.

Published as a conference paper at ICLR 2023
INTERNEURONS ACCELERATE LEARNING DYNAMICS IN RECURRENT NEURAL NETWORKS FOR STATISTICAL ADAPTATION
David Lipshutz', Cengiz Pehlevan', Dmitri B. Chklovskij
Center for Computational Neuroscience, Flatiron Institute
"John A. Paulson School of Engineering and Applied Sciences, Harvard University
"Neuroscience Institute, New York University School Medicine
weriiiifflaiiieee www.iaaniinnttttteeet...o.eeeeee org epehlevan@seas harvard. edu
(dlipshutz,
ABSTRACT
Early sensory systems in the brain rapidly adapt fluctuating input statisties. which requires recurrent communication between neurons. Mechanistically, such recurrent communication often indirect and mediated by local interneurons. this work, explore the computational benefits of mediating recurrent commu- nication via interneurons compared with direct recurrent connections. To this end, we consider two mathematically tractable recurrent linear neural networks that sta- tistically whiten their inputs- one with direct recurrent connections and the other with interneurons that mediate recurrent communication. By analyzing the corre- sponding continuous synaptic dynamics and numerically simulating the networks, we show that the network with interneurons is more robust to initialization than the network with direct recurrent connections in the sense that the convergence time for the synaptic dynamics in the network with interneurons (resp. direct re- current connections) scales logarithmically (resp. linearly) with the spectrum of their initialization. Our results suggest that interneurons are computationally use- ful for rapid adaptation to changing input statistics. Interestingly, the network with interneurons an overparameterized solution of the whitening objective for the network with direct recurrent connections, so our results can be viewed as a recurrent linear neural network analogue of the implicit accelcration phenomenon observed in overparameterized feedforward linear neural networks.
INTRODUCTION
Efficient coding and redundancy reduction theories of neural coding hypothesize that early sory systems decorrelate and normalize neural responses to sensory inputs (Barlow) 1961; Laughlin
1989 Barlow & Földiák 1989 Simoncelli & Olshausen] 2001| Carandini & Heeger [012 Westrick
et al.
2016 Chapocchihnn et 2001)) operations elosely related to statistical whitening of inputs. Since the input statisties are often in flux due dynamic environments, this calls for early sen-
sory systems that can rapidly adapt (Wark et al. 20077] Whitmire & Stanley 2016). Decorrelating neural activities requires recurrent communication between neurons, which is typically indirect and mediated by local interneurons (Christensen et al. 1993 Shepherd et al., 2004). Why do neuronal circuits for statistical adaptation mediate recurrent communication using interneurons, which take up valuable space and metabolic resources, rather than using direct recurrent connections?
common explanation for why communication between neurons mediated by interneurons is Dale's 's principle, which states that each neuron has exclusively inhibitory or excitatory effects on all of its targets (Strata & Harvey 1999). While Dale's principle provides a physiological constraint that explains why recurrent interactions are mediated by interneurons, we seek computational principle that can account for using interneurons rather than direct recurrent connections. This perspective is useful for couple of reasons. First, perhaps Dale's principle is not hard constraint; (Saunders jet al. 2013; Granger et al.] (020)p for results along these lines. In this case, a computational benefit of interneurons would provide normative explanation for the existence of interneurons mediate

---

Published as a conference paper at ICLR 2023
-W
Y1
Yn
Yn
MM
W
Figure 1: Recurrent neural networks for ZCA whitening with direct recurrent connections (left, Algorithm T and with interneurons (right, Algorithm
recurrent communication. Second, decorrelation and whitening useful in statistical and machine learning methods (Hyvärinen Oja 2000 Krizhevsky. 2009), especially recent self-supervised methods (Ermolov et al.; 2021 Zhontar al.] [021] Bardes et al. 2022, Hua et al., 2021),
analysis is potentially relevant to the design of artificial neural networks, which are not bound by Dale's principle.
In this work, to better understand the computational benefits of interneurons for statistical adapta- tion, analyze the learning dynamics of two mathematically tractable recurrent neural networks that statistically whiten their inputs using Hebbian/anti Hebbian learning rules one with direct recurrent connections and the other with indirect recurrent interactions mediated by interneurons, Figure [ We show that the learning dynamics of the network with interneurons are more robust than the learning dynamics of the network with direct recurrent connections. In particular, we prove that the convergence time of the continuum limit of the network with direct lateral connections scales linearly with the spectrum of the initialization, whereas the convergence time of the continuum limit of the network with interneurons scales logarithmically with the spectrum of the initialization. We also numerically test the networks and, consistent with our theoretical results, find that the network with interneurons is more robust to initialization. Our results suggest that interneurons compu- tationally important for rapid adaptation to fluctuating input statistics.
Our analysis is closely related analyses of learning dynamics in feedforward linear networks trained using backpropagation (Saxe et al., 20144) Arora et al. 2018: Saxe et al.) 2019 Gidel et al.]
2019 Tarmoun al. 202I). The optimization problems for deep linear networks overparame- terizations of linear problems and this overparameterization can accelerate convergence of gradient descent or gradient flow optimization
a phenomenon referred as implicit acceleration (Arora et al. 2018). Our results can be viewed as an analogous phenomenon for gradient flows correspond- ing to recurrent linear networks trained using Hebbian/anti heatiin learning rules. In our setting, the network with interneurons is naturally viewed as overparameterized solution of the whitening objective for the network with direct recurrent connections. analogy with the feedforward setting, the interneurons can be viewed hidden layer that overparameterizes the optimization problem.
In summary, our main contribution a theoretical and numerical analysis of the synaptic dynamics of two linear recurrent neural networks for statistical whitening one with direct lateral connec- tions and one with indirect lateral connections mediated by interneurons. Our analysis shows that the synaptic dynamies converge significantly faster in the network with internerrons than the network with direct lateral connections (logarithmic versus linear convergence times). Our results have po- tential broader implications: (i) they suggest biological interneurons may facilitate rapid statistical adaptation, see also (Duong et al. (023):; (ii) including interneurons in recurrent neural networks for solving other learning tasks may also accelerate learning; (iii) overparameterized whitening objec- tives may be useful for developing online self-supervised learning algorithms in machine learning.
2 STATISTICAL WHITENING
Let > 2 and XI,...,X7 XT be sequence of n--dimensional centered inputs with positive definite
(empirical) covariance matrix C.z == XXX, where X := [xi...., Xrr is the n x data matrix of concatenated inputs. The goal of statistical whitening to linearly transform the inputs that the n-dimensional outputs yı. Yi...., Yr have identity covariance; that is, Cyy := YY I,,. where
Y = y yr) is the T data matrix of concatenated outputs.
2

---

Published as a conference paper at ICLR 2023
Statistical whitening is not unique transformation. For example, if Cyy = =I,, then left multipli- cation of Y by any 71 X n orthogonal matrix results in another data matrix with identity covariance (Kessy al. 2018). We focus Zero-phase Components Analysis (ZCA) whiteming (Bell & Se- jnowski, 1997), also referred to as Mahalanobis whitening, given by C which is the whitening transformation that minimizes the mean-squared error between the inputs X and the out- puts Y (Eldar& & Oppenheim 2003). In other words, it is the unique solution to the objective
YEREXT 7
arg min
Y
X Plob
subject to
In-
(I)
The goal of this work to analyze the learning dynamics recurrent neural networks that learn perform ZCA whitening in the online, streaming, setting using Hebbian/anti-Hehbian learning rules. The derivations of the 2 networks, which we include here for completeness, are closely related
to derivations of PCA networks carried out in (Pehlevan & Chklovskil 2015 Pehlevan al. (018).
Our main theoretical and numerical results are presented in sections 6 and respectively.
OBJECTIVES FOR DERIVING ZCA WHITENING NETWORKS
In this section, we rewrite the ZCA whitening objective to obtain 2 objectives that will be the start- ing points for deriving our recurrent whitening networks. We first expand the square in equation [ substitute in with the constraint YYT TIm, drop the terms that do not depend on Y and finally flip the sign to obtain the following objective in terms of the neural activities data matrix Y:
max YEREXT T
Tr(Y
subject to
YY = =In-
3.1 OBJECTIVE FOR THE NETWORK WITH DIRECT RECURRENT CONNECTIONS
To derive network with direct recurrent connections, introduce the positive definite matrix M as Lagrange multiplier to enforce the upper bound YY TT,
min MEST YEREXT
max
J(M, Y);
where 4 denotes the set of positive definite matrices and
2 f(M, Y) = = IrXx - TY(M - -T,)).
Here, we have interchanged the order of optimization, which is justified because f(M, Y) is linear in M and strongly concave in Y, so it satisfies the saddle point property (see Appendix AA. The maximization over Y ensures that the upper bound YY < 7I, is in fact saturated, so the whiten- ing constraint holds. The minimax objective equation 2 will be the starting point for our derivation of a statistical whitening network with direct recurrent connections section
3.2 OVERPARAMETERIIED OBJECTIVE FOR THE NETWORK WITH INTERNEURONS
To derive a network with & interneurons, we replace the matrix M in equation with the overparameterized product WWT where W is an n x matrix, which yields the minimax objective
min max (W, Y). W YEREX?
(3)
where
2 (W, (W,Y) f(:w] Y) x - - 1
IrY -TI)).
The overparameterized minimax objective equation will be the starting point for our derivation of statistical whitening network with interneurons in section 5

---

Published as a conference paper at ICLR 2023
4 DERIVATION OF A NETWORK WITH DIRECT RECURRENT CONNECTIONS
Starting from the ZCA whitening objective in equation 2 we derive offline and online algorithms and map the online algorithm onto network with direct recurrent connections. We assume that the neural dynamics operate at faster timescale than the synaptic updates, so we first optimize over the neural activities and then take gradient-descent steps with respect the synaptic weight matrix.
In the offline setting, at each iteration, we first maximize f(M, Y) with respect to the neural activi- ties Y by taking gradient ascent steps until convergence:
Y Y xX - MY)
Y
where 7 > 0 is a small constant. After the neural activities converge, we minimize fMM, Y) with respect M by taking gradient descent step with respect M:
M M++ 1 YY I.)
(4)
In the online setting, at each time step 4, we only have access to the input x, rather than the whole dataset X. In this case, we first optimize f(M, Y) with respect to the neural activities vector yı by taking gradient steps until convergence:
Yr + Yt + 7(x - My;)
Yr
MM X2.
After the neural activities converge, the synaptic weight matrix M is updated by taking a stochastic gradient descent step:
M M+ I.).
This results in Algorithm |
Algorithm 1: A whitening network with direct recurrent connections
input centered inputs {x+}; parameters initialize positive definite matrix M for /. = 1...., T do
y:
repeat
Y: 1 Yz + 7xx, My:)
until convergence
M +- M nyvyy. I,)
end for
Algorithm 1 can be implemented in network with principal neurons with direct recurrent nections - M, Figure (left). At each time step f, the external input to the principal neurons X,, the output of the principal neurons is y, and the recurrent input to the principal neurons is -My.- Therefore, the total input to the principal neurons is encoded in the r-dimensional vector x; - My The neural outputs are updated according the neural dynamics in Algorithm T until they converge at M- 'x, After the neural activities converge, the synaptic weight matrix M is updated according to Algorithm
DERIVATION OF A NETWORK WITH INTERNEURONS
Starting from the overparameterized ZCA whitening objective in equation we derive offline and online algorithms and we map the online algorithm onto a network with k interneurons. As in the last section, we assume that the neural dynamics operate at faster timescale than the synaptic updates.
In the offline setting, at each iteration, we first maximize g( (, Y) with respect to the neural activi- ties Y by taking gradient steps until convergence:
Y +-Y+ -WWTY
Y= (WWT)-XX
4

---

Published as a conference paper at ICLR 2023
After the neural activities converge, minimize g( (,, Y) with respect to W by taking gradient descent step with respect to W:
W W+-Wi (-
(5)
In the online setting, each time step first optimize g(W, Y) with respect to the neural activities vectors Yt by running the gradient ascent-descent neural dynamics:
Yt - Yr + ((x: - ww yr)
Yr = (WW
After convergence of the neural activities, the matrix W is updated by taking stochastic gradient descent step:
W w + (y.y, W W).
To implement the online algorithm in recurrent neural network with interneurons, we let z, W y, denote the k-dimensional vector of interneuron activities at time t. After substituting into the neural dynamics and the synaptic update rules, we obtain Algorithm
Algorithm 2: A whitening network with interneurons
input centered inputs {xe]: k 2 parameters % initialize full rank n x l matrix W fort 1...., T do
Y: < 0 repeat
2, + W y:
Yt f Y: + 7xx Wzz)
until convergence
W< W + nyyie W)
end for
Algorithm can be implemented in network with principal neurons and interneurons, Figure (right). The principal neurons (resp. interneurons) are connected to the interneurons (resp. principal neurons) via the synaptic weight matrix W (resp. WW). At each time step t. the external input to the principal neurons is x,, the activity of the principal neurons is ye. the activity the interneurons 2, W y, and the recurrent input the principal neurons is Wz, The neural activities are updated according to the neural dynamics Algorithm 2 until they converge to Yt = (WWT)- )- and 2, W y:. After the neural activities converge, the synaptic weight matrix W is updated according to Algorithm 2
Here, the principal neuron-to-interneuure weight matrix WT is the negative transpose of the interneuron-to-prriippall neuron weight matrix - W. Figure T (right). In general, enforcing such symmetry biologically plausible and commonly referred to the weight transport problem. In addition, we do not sign-constrain the weights, so the network can violate Dale's 's principle. In Appendix B we modify the algorithm to be more biologically realistic and we map the modified algorithm onto the vertebrate olfactory bulb and show that the algorithm is consistent with several experimental observations.
ANALYSES OF CONTINUOUS SYNAPTIC DYNAMICS
We now present our main theoretical results on the convergence the corresponding continuous synaptic dynamics for M and W. We first show that the synaptic updates are naturally viewed as (stochastic) gradient descent algorithms. We then analyze the corresponding continuous gradient flows. Detailed proofs of our results are provided in Appendix
6.1 GRADIENT DESCENT ALGORITHMS
We first show that the offline and online synaptic dynamics are naturally viewed as gradient descent and stochastic gradient descent algorithms for minimizing whitening objectives. Let ((M) be the
5

---

Published as a conference paper at ICLR 2023
convex function defined by
U(M) = max f(M, Y) = Tr (M- a2
-M)
YERUXT"
Substituting the optimal neural activities Y = M into the offline update in equation we see that the offline algorithm is a gradient descent algorithm for minimizing U (M):
M + M M++ C I.) M (M)
Similarly, Algorithm is stochastic gradient descent algorithm for minimizing (M). Next, let V(W) the nonconvex function defined by
V(W) == max YER"X7
g W. Y) = Tr ((WW) (WW.T)-CC.. WWT).
Again, substituting the optimal neural activities Y (WWT)-1 - into the offline update in equa- tion 5 see the offline algorithm is gradient descent algorithm for minimizing V(W):
W - W + ((WWT) - - w) w VVWWW.
2
Similarly, Algorithm 2 is a stochastic gradient descent algorithm for minimizing V(W).
common approach for studying stochastic gradient descent algorithms is analyze the corre- sponding continuous gradient flows (Saxe et al., 2014: 2019 Tarmoun et al. (02T), which are more mathematically tractable and are useful approximations of the average behavior of the stochastic gradient descent dynamics when the step size is small. In the remainder of this section, we analyze and compare the continuous gradient flows associated with Algorithms Daathmmmmann and
To further facilitate the analysis, we consider so-called 'spectral initializations" that commute with Can Saxx et al. 2014 2019, Gidel et al. 2019 Tarmoun et al., 2021). Specificially, we say that T where U, is the n x orthogonal matrix of eigenvectors of C.u and dI...., On > 0. To characterize the convergence
Ao E 8 is spectral initialization if Ao
(.dia((D,,.... .n) U
rates of the gradient flows, define the Lyapunov function
((A)= == |Czz - A² F|Fob,
A E S++
6.2 GRADIENT FLOW ANALYSIS OF ALGORITHM 1
The gradient flow U (M) is given by
dM(()
(U((())) M(.) I.
(6)
di
To analyze solutions of M(t), we focus spectral intializations.
Lemma 1. Suppose Mo a spectral initialization. Then the solution M(t) of the ODE 6 is of the
form M(I) Undiag(a (1), ...,, on ( where (I on (1), are the solutions the ODE
(t.
X?
1,
i= I....,
dt (x(1)2
Consequently,
d dt (2))
4 -ax(1)=-X3)
x²)=
x2)2, i I.....n. (1
-
(8)
From Lemma 1 we see that for a spectral initialization with o,( < As the dynamics of ax(() satisfy
(0,
dt
It follows that (t)² converges to X² exponentially with convergence rate greater than 2/ On the other hand, suppose (0) Aç. From equation 7 at approximately unit rate; that is,
we see that while a,(t) > A, at( decays
da
-1
dt
Therefore, the time for (t) to converge to A; grows linearly with (0). We make these statements precise in the following proposition.

---

Published as a conference paper at ICLR 2023
Proposition 1. Suppose My is spectral initialization and let M(t) denote the solution of the ODE 6 starting from Mo. If o; I X all L..... n., then for e < ((Mo).
Vdmax log (e(M))e
min (t 2 ((M Sc
(10)
where Amax max; Aç. On the other hand, > for some 1,..., then for of - X2.
minft 2 0: :::0::00P ((M()) { 2 /x2 te.
(11)
6.3 GRADIENT FLOW ANALYSIS ALGORITHM 2
The gradient flow of the overparameteriized cost V(W) is given by
(t VV(W) (W()W(7)) C.z (W(W)(T) -1 W(I W(I).
(12)
dt 2
Next, we show that (()) converges to zero exponentially for any initialization Wo. Proposition (L) denote the solution of the ODE starting from any Wo E Raxk Let < E(Wo W For spectral initializations W.
min 2 0: ((((()(())) S() 2 log (e(Wo (W)
(13)
For general initializations Wo W
min(t 2 0: (W())())) log (o((eW (W
(14)
7 NUMERICAL EXPERIMENTS
In this section, we numerically test the offine and online algorithms synthetic datasets. Let
Whitening error(t) = CCzAA I, LrloFr
A, E (M,, W,W.),
where A, is the value of matrix after the 4h iterate. To quantify the convergence time, define
Convergence time == min ( 2 1 Whitening error(1) < 0.1).
In plots with multiple runs, lines and shaded regions respectively denote the means and 95% confi- dence intervals over 10 runs.
7.1 OFFLINE ALGORITHMS
Let n = 5, k = 10 and T = 105. We generate a data matrix with i.i.d. entries कτ, chosen uniformly from the interval (0, 12). The eigenvalues of Czr (24.01, 16.42, 10.45, 6.59, 3.28). We initialize W, QT where > 0, Σ diag(5. (5, 4, 3, 2, 1). Q an matrix and P is random & x n matrix with orthonormal column vectors. We set M, W, W QQ'º We consider spectral initializations (i.e.. Q = =@,,)) and nonspectral initializations (i.e..
orthogonal
Q is random orthogonal matrix). We use step size 1 10-3
The results of running the offline algorithms for a 0.1, 1, with Propositions I and 2 the network with direct recurrent connections convergences slowly for 'large' a, whereas the network with interneurons converges exponentially for all a. Further, as predicted by equation 9 when the eigenvalues of M are 'large', i.e., a; X; they decay linearly.
1, 10 are shown in Figure 2 Consistent
In Figure 3 we plot the convergence times of the offlin algorithms with spectral and nonspectral initializations, for vergence times for network with direct lateral connections (resp. interneurons) grows linearly (resp. logarithmically) with
20. Consistent with analysis of the gradient flows, the
7

---

Published as a conference paper at ICLR 2023
#pphaw.33
#phh-m3
appaam2 apre-20
lourons
Fipetala
(a) Spectral initializations
(b) Nonspectral initializations
Figure 2: Comparison of whitening error and eigenvalue evolution for the offline algorithms with (a) spectral initializations and (b) nonspectral initializaitons.
Convergancc 5,70
Intorouuteo Epectrel
Diooc tne ineropetech
alpha
alpho (ing-orale)
Figure 3: Comparison of convergence times for the offline algorithms with both spectral and non- spectral initializations, as functions of a ........ 20.
7.2 ONLINE ALGORITHMS
We our online algorithms on synthetic dataset with samples from 2 distributions. Let
= 2, k = 4 and T = 10a. Fix covariance matrices CĄ = U d4a 25) U and Cв = U Ugdiag(9 (9, 1, B. where UA. UB are random 2 x 2 rotation matrices. We generate a 2 x 47' dataset X [i...... ... X4r] with independent samples X1...., X7, X2r-1..... XyT N/ CA) and Xf+1..... Xyr, +37+1..... X47 ~ (O,, Cg). We evaluated our online algorithms with step size 1 = 10-4 and initializations Mo = W W. where W, Qdiag(ar 02 PT Q a random 2 x 2 rotation matrix, P is random 4 2 matrix with orthonormal column vectors and 01. 02 are independent random variables chosen uniformly from the interval (1, 1.5). The results are shown in Figure 4 Consistent with our theoretical analyses,
that the network with interneurons adapts to changing distributions faster than the network with direct recurrent connections.
7.3 APPLICATION PRINCIPAL SUBSPACE LEARNING
A challenge in unsupervised and self- self-supervised learning is preventing collapse (i.e., degenerate solutions). recent approach in self-supervised learning is decorrelate or whiten the feature
Direct
intor nourons
Egenvalees
Figure 4: Comparison of whitening error and eigenvalue evolution for the online algorithms on dataset with switching distributions (white light orange backgrounds).
8

---

Published as a conference paper at ICLR 2023
+
V
32
V
X
-W
MM
W!
Figure 5: Left: networks for principal subspace projection with output whitening using direct lateral connections or interneurons. Right: comparison of subspace error for the two networks with relative learning rates 17 or 10ç.
representation (Ermolov et al. 2021 Zbontar al. 2021 Bardes et al. 2022 Hua et al. 202
Here, using Oja's online principal component algorithm Oja 1982) as tractable example, we demonstrate that the speed of whitening can affect the accuracy of the learned representation.
Consider a neuron whose input and output at time are respectively s; E IR and ye := V where IRd represents the synaptic weights connecting the inputs to the neuron. Oja's 's algorithm learns the top principal component of the inputs by updating the vector v as follows:
V f V + Cyyis v2.
(15)
where Ç > 0 is the step size.
Next, consider a population of of 2 S d neurons with outputs y: E R" and feedforward synaptic weight vectors Vi...., Vn E R2d connecting the inputs S, to the neurons. The goal is to project the imputs onto their n-dimensional principal subspace. Rumning n instances Oja's 's algorithm in parallel without lateral connections results in collapse each synaptic weight vector v, converges to the top principal component. One way to avoid collapse is to whiten the output y: using recurrent connections, Figure 5 (lett). Here show that when the subspace projection and output whitening are learned simultancously, it is critical that the whitening transformation is learned sufficiently fast.
2 and generate i.i.d. inputs S, (O, diag(5, 2, 1)). We initialize two random vectors V1, V2 E Rs with independent (0, 1) entries. At each time step i, we use the projection x, = (v] St, V.J (s)) as the input to either Algorithm [ or 2 and we let yr be the output; that is, according to equation 15 with < = 10 and with v, (resp. y/..) in place of v (resp. yr). We update the recurrent weights M or W according to Algorithm 1 To measure the performance, we define
We set d
y: = M X2 or Yr = -(WWT) ) Xt. For i = 1, 2. we update
Subspace enror
|| V) 'VT -
V [v v2] 3x2
which is equal to zero when V1, V2 span the 2-dimensional principal subspace of the inputs. We plot the subspace error for Algorithms and 2 using learning rates 7) = < and r 10ç, (righ)) The results suggest that in order to learn the correct subspace, the whitening transformation must be learned sufficiently fast relative to the feedforward vectors v;: for additional results along these lines, see (Pehlevan et al. [2018) section 6). Therefore, since interneurons accelerate learning of the whitening transform, they are useful for accurate or optimal representation learning.
DISCUSSION
We analyzed the gradient flow dynamies of 2 recurrent neural networks for ZCA whitening one with direct recurrent connections and one with interneurons. For spectral initializations we can analytically estimate the convergence time for both gradient flows. For nonspectral initializations, show numerically that the convergence times are close to the spectral initializations with the same initial spectrum. Our results show that the recurrent neural network with interneurons is more robust initialization.
An interesting question is whether including interneurons in other classes of recurrent neural net- works also accelerates the learning dynamics of those networks. We hypothesize that interneurons accelerate learning dynamics when the objective for the network with interneurons can viewed as overparameterization of the objective for the recurrent neural network with direct connections.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We thank Yanis Bahroun, Nikolai Chapochnikov, Lyndon Duong, Johannes Friedrich, Siavash Golkar, Jason Moore and Tiberiu Teşileanu for helpful feedback on an earlier draft of this work. C. Pehlevan acknowledges support from the Intel Corporation.
REFERENCES
Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparameteriration. In International Conference on Machine Learning, pp- 244-253, PMLR, 2018.
Adrien Bardes, Jean Ponce, and Yann Lecun. VICReg: carianeeinvarrance-cvviii regular- ization for self-supervised learning. In International Conference Learning Representations (ICLR), 2022.
Horace Barlow and Peter Földiák. Adaptation and Decorrelation in the Cortex, pp. 54-72. Addison- Wesley Longman Publishing Co., Inc., USA, 1989. ISBN 020118348X.
Horace Barlow. Possible principles underlying the transformation of sensory messages. Sensory Communication, 1(01), 1961.
Anthony J Bell and Terrence J Sejnowski. The "independent components" of natural scenes are edge filters. Vision Research, 37(23):3327-3338, 1997.
Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, 2004.
Matteo Carandini and David Heeger. Normalization as a canonical neural computation. Nature Reviews Neuroscience, 13(1):51-62, 2012.
Nikolai M Chapochnikov, Cengiz Pehlevan, and Dmitri Chklovskii. Normative and mechanistic model of adaptive circuit for efficient encoding and feature extraction. bioRxiv, 2021.
TA Christensen, BR Waldrop, ID Harrow, and JG Hildebrand. Local interneurons and informa- tion processing in the olfactory glomeruli of the moth manduca sexta. Journal of Comparative Physiology A, 173(4):385-35) 1993.
Lyndon Duong, David Lipshutz, David Heeger, Dmitri B Chklovskii, and Eero Simoncelli. Statistical whitening of neural populations with gain-modulating interneurons. arXiv preprint arXiv:2301.11955 2023.
Yonina C Eldar and Alan V Oppenheim. MMSE whitening and subspace whitening. IEEE Trans- actions on Information Theory, 49(7))1846-188 2003.
Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self- supervised representation learning. In International Conference on Machine Learning, pp. 3015- 3024. PMLR, 2021.
Rainer W Friedrich and Gilles Laurent. Dynamic optimization of odor representations by slow temporal patterning of mitral cell activity. Science, (91(5505) 894, 2001.
Rainer W Friedrich and Gilles Laurent. Dynamics of olfactory bulb input and output activity during odor stimulation in zebrafish. Journal of Neurophysiology, (lcelloyyyy19000 (66))22658-)66, 2004.
Gauthier Gidel, Francis Bach, and Simon Lacoste-Julien Implicit regularization of discrete gradient dynamics in linear neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Sonya Giridhar, Brent Doiron, and Nathaniel N Urban. depeedel shaping of correlation by olfactory bulb lateral inhibition. Proceedings of the National Academy of Sciences, 108(14); 5843-5848, 2011.
10

---

Published as a conference paper at ICLR 2023
Siavash Golkar, David Lipshutz, Yanis Bahroun, Anirvan Sengupta, and Dmitri Chklovskii. A simple normative network approximates local non-Hebbian learning in the cortex. Advances in Neural Information Processing Systems, 33:7283- -295, 2020.
Adam J Granger, Wengang Wang, Keiramarie Robertson, Mahmoud El-Rifai, Andrea F Zanello, Karina Bistrong, Arpiar Saunders, Brian W Chow, Vicente Nuñez, Miguel Turrero Gareía, et al. Cortical chat+ neurons co-transmit acetylcholine and gaba in a target-and www---reeioosieiiiiee manner. Elife, 9:e57749, 2020.
Olivier Gschwend, Nixon M Abraham, Samuel Lagier, Frédéric Begnaud, Ivan Rodriguez, and Alan Carleton. Neuronal pattern separation the olfactory bulb improves odor discrimination learning. Nature Neuroscience, 18( 3(100)1474-1453 2015.
Tianyu Hua, Wenxiao Wang, Zihui Xue, Sucheng Ren, Yue Wang, and Hang Zhao. On feature decorrelation in self-supervised learning. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pp. 9598-9608, 2021.
Aapo Hyvärinen and Erkki Oja. Independent component analysis: algorithms and applications. Neural networks, ((4-5):411-430 2000.
Agnan Kessy, Alex Lewin, and Korbinian Strimmer. Optimal whitening and decorrelation. The American Statistician, 72(4):309-311 2018.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Toronto, 2009.
Simon B Laughlin. The role of sensory adaptation in the retina. Journal of Experimental Biology, 146(1):39-62, 1989.
Antoine Nissant, Cedric Bardy, Hiroyuki Katagiri, Kerren Murray, and Pierre-Marie Lledo. Adult neurogenesis promotes synaptic plasticity in the olfactory bulb, Nature Neuroscience, 12(6):728- 730, 2009.
Erkki Oja. Simplified neuron model as principal component analyzer. Journal of Mathematical Biology, 153))2677-273, 1982.
Cengiz Pehlevan and Dmitri B Chklovskii. A normative theory of adaptive dimensionality reduction in neural networks. Advances in Neural Information Processing Systems, 28, 2015.
Cengiz Pehlevan, Anirvan M Sengupta, and Dmitri B Chklovskii. Why do similarity matching objectives lead to Hebbiaan/ani--lebbian networks? Neural Computation, 30(1):44-124 2018.
Kurt A Sailor, Matthew T Valley, Martin T Wiechert, Hermann Riecke, Gerald J Sun, Wayne Adams, James Dennis, Shirin Sharafi, Guo-li Ming, Hongjun Song, et al. Persistent structural plasticity optimizes sensory information processing in the olfactory bulb. Neuron, 91 396, 2016.
Arpiar Saunders, Ian A Oldenburg, Vladimir K Berezovskii, Caroline A Johnson, Nathan D Kingery, Hunter Elliott, Tiao Xie, Charles R Gerfen, and Bernardo Sabatini. A direct gabaergic output from the basal ganglia to frontal cortex. Nature, 521(7550):85-- 09, 2015.
Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions the nonlinear dy- namics of learning in deep linear neural networks. In International Conference on Learning Representations (ICLR), 2014.
Andrew M Saxe, James L McClelland, and Surya Ganguli. A mathematical theory of semantic development in deep neural networks. Proceedings of the National Academy of Sciences, 116 153):11537-11566, 2019.
Gordon Shepherd. Dendrodendritic synapses: past, present and future. Annals of the New York Academy of Sciences, 1 1170, 2009.
Gordon M Shepherd, Chen Wei R, and Charles A Greer. Olfactory bulb. In Gordon M Shepherd (ed.), The Synaptic Organization of the Brain, chapter 5, pp. 165 165-216. Oxford University Press, 2004.
11

---

Published as a conference paper at ICLR 2023
Eero P Simoncelli and Bruno A Olshausen. Natural image statistics and neural representation. Anmual Review of Neuroscience, 24(1):1193-1216 2001.
Piergiorgio Strata and Robin Harvey. Dale's principle. Brain Research Bulletin, 50(5):349-350, 1999.
Salma Tarmoun, Guilherme Franca, Benjamin D Haeffele, and Rene Vidal. Understanding the dynamics of gradient flow in overparameterized linear models. In International Conference on Machine Learning, pp. 10153-10161. PMLR, 2021.
Adrian A Wanner and Rainer W Friedrich. Whitening of odor representations by the wiring diagram of the olfactory bulb. Nature Neuroscience, 23(3):433 442, 2020.
Barry Wark, Brian Nils Lundstrom, and Adrienne Fairhall. Sensory adaptation. Current Opinion in Neurobiology, 17(4):423-429, 2007.
Zachary M Westrick, David J Heeger, and Michael Landy. Pattern adaptation and normalization reweighting. Journal of Neuroscience, 36(38):9805-95) 9816, 2016.
Clarissa J Whitmire and Garrett B Stanley. Rapid sensory adaptation redux: circuit perspective.
Neuron, 92(2):298-315, 2016.
Stwart D Wick, Martin T Wiechert, Rainer W Friedrich, and Hermann Riecke. Pattern orthogonal- ization via channel decorrelation by adaptive networks. Journal of Computational Neuroscience, 28(1):29 45, 2010.
Thomas B Woolf, Gordon M Shepherd, and Charles A Greer. Serial reconstructions of granule cell spines in the mammalian olfactory bulb. Synapse, 7(3):181-192, 1991.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, pp. 12310- 12320. PMLR, 2021.
A SADDLE POINT PROPERTY
Here recall the following minimax property for function that satisfies the saddle point property Boyd Vandenberghe 2004 section 5.4).
Theorem 1. Let V C R", W C Rm and f V x W + R. Suppose f satisfies the saddle point property; that is, there exists (a", b*) V x W such that
f(a', f(a" b*) f(a,b"),
for (a, EV
Then
min max f(a,b) b) max min f(a. b) = f(a", aeV beW
beW aEV
B RELATION TO NEURONAL CIRCUITS
In this section, modify Algorithm to satisfy additional biological constraints and we map the modified algorithm onto the vertebrate olfactory bulb.
B. DECOUPLING THE INTERNEURON SYNAPSES
The neural circuit implementation of Algorithm 2 requires that the principal synaptic weight matrix W is the negative transpose of the mnterneuron-to-princee neuron synaptic weight matrix - W. In general, enforcing this symmetry not biologically plausible, and is com- monly referred to the weight transport problem. Here, following (Golkar al. 2020 appendix D), we decouple the synapses and show that the (asymptotic) symmetry of the synaptic weight ma- trices follows from the symmetry of the local Aaaiiiannani-Heebiiae updates.
12

---

Published as a conference paper at ICLR 2023
We begin by replacing W and WT in Algorithm 2 with Wa, and W yes respectively. Then the neural activities are given by
Yt Yt 7x, W zy)2e);
24 24 y=y; zz);
(16)
which converge to (W W y= 'x, and W y: After the neural activities converge. the synaptic weights are updated according to the update rules
W zy f Way zy (y:z W wxy)
W yz W y= (z.y, W y=).
Let W z..t and W yz denote the values of the synaptic weight matrices after updates. By iterating the above updates, we see that the difference between the weight matrices after updates is given by
W na,t
W zz. - n)²² (W vz.0 W sy.0).
Therefore, provided < < 1, the difference decays exponentially.
B.2 SIGN-CONSTRAINING THE INTERNEURON WEIGHTS
In equation [6] the matrix W yz is preceded by a positive sign and the matrix W sy preceded by negative sign, which consistent with the fact that the principal nouron-to-interneuron synapses are excitatory whereas the interneuron-ot torpriiconn---piiippalll neuron synapses are inhibitory. However, this inter- pretation is superficial because the matrices are not constrained to be non-negative, so the network can violate Dale's principle.
One approach to enforce Dale's principle is to take projected gradient descent steps when updating the synaptic weight matrices, which results in the offline updates
1 -
W sy
W zy
+
Yz
W
W ys
+ +
ZY Wys yz
yz
and online updates
W ay -- W ay + nly
W Wey]l+>
W 92 yz + n2,y!
W Wyz)l+
where [++ denotes the element-wise rectification operation. This results in Algorithm
Algorithm 3: A whitening network with interneurons and sign-constrained weights
input centered inputs {x parameters % initialize non-negative matrices W y2s W... for / 1,..., T do
repeat
Ye - Y: + 7(2, W 2:)
Z, 2, + (W wayt zi)
until convergence
W y= + Wy yy + n
W Wg=)|= W W.y)l+
W sy + Way + nlyzz
end for
In general, this modification results in network output that does not correspond to ZCA whitening of the inputs. That being said, it still worth comparing the performance of the network to the net- work with interneurons derived in infsectionn In Figure 6) we compare the performance of the offline versions of Algorithms 2 and We see that Algorithm 3 equilibrates at a rate that is comparable to Algorithm (and therefore much faster than Algorithm for large a). In particular, this modified algorithm appears to also exhibit the accelerated dynamics due to the overparamterization of the objective.
13

---

Published as a conference paper at ICLR 2023
0.p
alpho-1
alpha-5
alpha-10
40
Interneurons Non-negative
13
Whitening error
20
15
35
10
-527
4031
1031
1830
276
A46
1881
1250
230
64.0
1462
1827
1037
+3.01
0..00
Eigenvalues
.003
JE33
4032
3330
200C
JMCC
1000
2000
2033 20:0
Iterations
Iterations
lterations
Iterations
Figure 6: Comparison of whitening error and eigenvalue evolution for the offline algorithms (with non-spectral initializations) corresponding to the network with interneurons with unconstrained or non-negative weights. The lines and shaded regions respectively denote the means and 95% confi- dence intervals over 10 runs.
As expected, the outputs of the projected gradient descent algorithm not fully whitened after the synaptic dynamics of Algorithm equilibrate, the eigenvalues of the output covariance Cgy are {1.39, 1.01, 1.00, 0.98, 0.61). J. This be compared with the eigenvalues of the input covariance matrix Cez: (24.01, 16.42, 10.45, 6.59, 3.28), which suggests that the network significantly nor- malizes the eigenvalues of the output covariance without exactly performing ZCA whitening. In general, understanding the exact statistical transformation that results from the projected gradient descent algorithm is more mathematically challenging. For instance, in the case of Algorithm ?
the weights W adapt so that WW = C1/2, that is, the algorithm solves a symmetric matrix factorization problem whose solutions can be written in closed form in terms of the SVD of the covariance matrix Crr. In the case of Algorithm 3 the algorithm is essentially solving for the op-
timal (approximate) non-negative symmetric matrix factorization of C for which there do exist general closed form solutions. That being said, the empirical results suggest that the more biologically realistic network still performs rapid statistical adaptation.
B.3 MAPPING ALGORITHM| ONTO THE OLFACTORY BULB
In vertebrates, early stage of olfactory processing bulb receives direct olfactory inputs, which it processes and transmits to higher order brain regions (Shepherd et al. 2004), Figure Olfactory receptor neurons project to the olfactory bulb, where their axon terminals cluster by receptor type into spherical structures called glomeruli. Mitral cells, which are the main projection neurons of the olfactory bulb, receive direct inputs from the olfactory receptor neuron axons and output to the rest of the brain. Each mitral cell extends its apical dendrite into a single glomerulus, where forms synapses with the axons of the olfactory receptor neurons expressing a common receptor type. The mitral cell activities are modulated by lateral inhibition from interneurons called granule cells, which are axonless neurons that form reciprocal dendroden- dritic synapses with the basal dendrites of mitral cells (Shepherd 2009). Experimental evidence indicates that the olfactory bulb transforms its inputs so that mitral cell responses to distinct odors approximately orthogonal (Friedrich & Laurent 20011 2004 Giridhar et al.) 2011 Gsccwwend et al. 2013| Wanner & Friedrich, 2020), transformation referred to as pattern separation, which is closely related to statistical whitening (Wick et al. 2010)).
in the olfactory bulb. The olfactory
We explore the possibility that the mitral gratule cell microcircuit implements Algorithm 3 Figure In this case, the vectors x, and y, represent the inputs to and outputs from the mitral cells,
14

---

Published as a conference paper at ICLR 2023
olfactory
receptor neurons
glomeruli
x glomeri inputs to mitral cells
y mitral cell outputs
granule cell outputs
mitral cell granule cell synapses granule cell mitral cell synapses
mitral cells
granule cells
yy
Figure 7: A simplified schematic of the olfactory bulb.
respectively. The vector 2, represents the granule cell outputs. We let W ye (resp. - - W2y) denote the mitral cell-to-granule cell (resp. granule cell-to-mitral cell) synaptic weights.
3 learns by adapting the matrices Wy and W zy- This is in line with experimental observations that gran- ule cells in the olfactory bulb are highly plastic suggesting that the dendrodendritic synapses are synaptic substrate for experience dependent modulation of the olfactory bulb output (Nissant et al. 2009, [Sailor et al. 20166) Second, a consequence of Algorithm 3 is that the mitral cell-to- granule cell synapses and granule cell-to-mitral cell synapses are (asymptotically) symmetrie; that is, W xy = W There is experimental evidence in support of this symmetry - the dendrodendritic synapses are mainly present reciprocal pairs (Woolf et al. 199T). In other words, the connectivity matrix of mitral cell-to-granule cell synapses (the matrix obtained by setting the non-zero entries in Wy to !) is approximately equal to the transpose of the connectivity matrix of granule cell-to- mitral cell synapses. Finally, Algorithm 3 requires that k than mitral cells. This consistent with measurements the olfactory bulb indicating that there are approximately 50-100 times granule cells than mitral cells (Shepherd al. 2004).
We compare our model with additional experimental observations. First, Algorithm
that is, there more granule cells
C PROOFS OF OUR MAIN RESULTS
C.I PROOF OF LEMMA 1
Proof. Let 01 (t), ...., (1) be solutions of the ODE
7 and define M(!) = U, E.) where
E(t) == diag( 01 (t), ..., o, (t).. Then
dM()) U,,iiag
X
X2
UT
I.
01 (1)2))
Fn (1)2
-1-1)-111 I,..
In particular, that M(t) must be the unique solution of the ODE 6 where uniqueness of solutions follows because the hight-hand-aa of equation is locally Lipschitz continuous on its domain of definition. Equation then follows from the ODE and the chain rule.
[]
15

---

Published as a conference paper at ICLR 2023
C.2 PROOF OF PROPOSITION 1
Proof. Suppose (0) - A for i = ,,..., n.
By equation 8
0:(()
S X for all 2 0 and so
(o.(t)² < (a:(0)² - x3)a exp (- 4t
(-vX. Vx
1 n.
Therefore,
41
{((())) (a:(1)²- x2)² v {(M))² exp
Vimax
==1
It follows that equation 10 holds. Now suppose o.(0) 2 x for some i = .I...,n By equation]
(t(2)² X² 2 (a,(0) t)² x2 E (0) -x,]-
It follows that equation 11 holds.
C.3 PROOF OF PROPOSITION
Proof. Define (t)) == ((t) (t) By the product rule,
(I (tt) CCz + A1)- (t()).
(17)
dt
Then, by the chain rule and equation
17 for all 2 0,
(A))))) dt
4Th (AA()) C.z) A(t) dA(t)
dt
---))(((-----CCC... -4TTT C..) 4Tr ((A(t)) C.. -4)(A(1)) - -11 Czz)A(()!?I Poo
-ATT(A(1)2 - C..) 4TH
Suppose Wo W. is spectral initialization. Then A(t) commutes with C.z and so
((()())) SE(A()))2
((A(t) ((A exp(-4t).
Thus, equation 13 holds. For general initializations.
dl(A(1))2 S -
((A(t)) < ((A(0)) (x000))xppp222)))) cxp(-2t).
Therefore, inequality equation 14 holds.
16


Reviews:
[{"review_id": "Y-qcf3-j_", "reviewer": "Reviewer_JKUS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a theoretical analysis of the computational benefits of interneuron-mediated recurrent connections by characterizing the synaptic change dynamics while learning to whiten an input. Specifically, the authors provide analytical solutions to the dynamics of a recurrent neural network without any pointwise non-linearity when the recurrent connections are direct and when they are mediated by a layer of interneurons. They motivate the problem from a computational neuroscience and neuromorphic setting and demonstrate that interneurons enable faster synaptic plasticity dynamics, thus suggesting that it is beneficial to have an intermediate layer of interneurons in biological recurrent neural networks for faster learning. In this work, the cost function is assumed to be the whitening objective, i.e. the activity in the recurrently connected neurons are pushed towards having an Identity covariance matrix.\nAlthough the results and insights are interesting, the work builds on certain strong assumptions that limit its utility to current studies in computational neuroscience or machine learning.", "summary_of_the_review": "Thir work presents an interesting perspective about interneuron-mediated recurrent connections, but it uses certain biologically implausible assumptions. Moreover, the definition of recurrent network used here is slightly different from the canonical definition used by the machine learning community and given the audience of ICLR, I feel this work might not be pertinent to a broad audience. Finally, despite the interesting analytical results and corresponding empirical validation, it is unclear how the benefits in dynamics translates to computational benefits with respect to task performance or learning ability. Therefore, this work would need significant changes for it to be presentable to the ICLR community.", "strength_and_weaknesses": "Strengths:\n1. The paper presents a strong theoretical analysis of synaptic change dynamics under optimization of the whitening objective. This analysis would be useful and is pertinent to current advances in self-supervised deep learning, specifically the BarlowTwins or VIC-Reg objectives (Zbontar _et al._ 2021, Bardes _et al._ 2021).\n2. Through numerical simulations, the authors demonstrate that some of the assumptions in their analytical results, albeit necessary for deriving the closed form expressions are not overbearing in practice. These results suggest that the inferences hold even if commonly used initialization strategies in machine learning are adopted. \n\nWeakness:\n\nI have some concerns about the presentation, but I believe they can be fixed and I have described them in detail under \"Clarity, Quality, Novelty and Reproducibility\". \n\n1. My main concern is from the computatinal neuroscience perspective. The authors present Dale's law as a motivating factor to understand the interneuron-mediated recurrent connections. However, it is not clear if the sign constraint is applied on the recurrent connectivity matrix or if the simulations used projected gradient descent to ensure that the weights were always all +ve. With the negative sign incorporated in the network formulation, the interneurons are supposed to be inhibitory but if the weights are allowed to be both +ve or -ve, it doesn't respect the Dale's law anymore. \n2. The overparameterization narrative assumes that there are more interneurons than pyramidal neurons ($ k \\geq n$), which is generally not observed in biological neural circuits. Instead it is the other way around. Could the authors comment on this and what it means for their results and inferences? Additonally, the constraint on weights being tied to and from the interneuron population, i.e. the requirement for $W$ and $W^T$, is biologically unrealistic. Could the authors comment on whether this is necessary?\n3. The whitening objective provides a nice analytical setup wherein the authors converge to a closed form solution. Although the difference in dynamics for direct vs interneuron-mediated connections is presented in the paper, it is unclear how these learned representations are different, if at all. This understanding will be important for both computational neuroscience and machine learning, wherein an important question is do interneurons provide a computation benefit in realistic tasks. In other words, are interneuron-mediated recurrent networks better able to learn certain tasks because they can whiten the inputs quicker? Demonstrating this in a toy task would also provide useful evidence in favour of such an architecture. If not, it raises a question whether whitening is a reasonable objective to consider when learning representations from data. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "On-YtWUad2a", "reviewer": "Reviewer_hapb", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper compares two bioplausible implementations of Mahalanobis whitening based on (anti)hebbian plasticity rules, in either a network with direct connections or a network with interneurons. The core finding is that without interneurons, the convergence times scales linearly, but with interneurons it scales only logarithmic.", "summary_of_the_review": "The submission is a nice addition to a series of paper on bioplausible implementations of basic algorithms (PCA, CCA,etc) with a plausible biological interpretation.", "strength_and_weaknesses": "Strength:\n* Nice analytical result with biological interpretation and numerical confirmation. Very clearly written, transparent derivation. \n\nWeaknesses:\n* Not clear how that would be done by a more biologically plausible neuron using spikes.\n\n\n\nMinor issues:\n\"overeparameterized\" should be \"overparameterized\"\n\"Interestingly, contrast\" should be \"Interestingly, in contrast to\"\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GofrHXvUyya", "reviewer": "Reviewer_qLnU", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the learning dynamics of a linear recurrent network with and without interneurons. Theoretical analysis and numerical simulation both suggest the network with interneurons converges faster in learning.", "summary_of_the_review": "I have gone through nearly all the math and I am certain that the math derivations are correct.", "strength_and_weaknesses": "The present paper is mathematically sound by theoretically analyzing the learning dynamics. I have no problem with the techniques and the results of the paper. \n\n\n### Major questions\n(PS: below are some of my questions regarding the results which are not a criticism of the present study.)\n\n- The interneurons in neuroscience studies are inhibitory neurons. For the interneurons in the present model, do authors constrain the polarity of interneurons' weights? I don't find this constraint and I guess no such constraint in the study. Moreover, for the network model with interneurons, there are no direct recurrent connections between y neurons in the network (Fig. 1, right). I am curious how the result will be changed if we include direct recurrent connections.\n\n- The recurrent connections $W$ in the network with interneurons correspond to decompose the recurrent connections $M$ in the network without interneurons, i.e., $M=WW^\\top$. This implicitly assumes the connections between primary neurons y and the interneurons z are symmetric. I am happy to see some discussions in releasing this constraint in that such symmetry probably doesn't exist in the brain.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "Gnf2ywf3v0", "reviewer": "Reviewer_Meay", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors investigate the role of inhibitory interneurons in the statistical adaptation of the brain. More specifically the investigate their role in neural networks that perform input whitening. The authors demonstrate that networks with inhibitory interneurons are more resilient to initializations than networks with just direct recurrent connectivity that operate in violation of Dale’s principle.", "summary_of_the_review": "A solid and interesting paper, however the impact of the findings seems a bit too limited for a broad venue such as the ICLR conference.  Maybe a more specialized venue could be a better fit for this paper. However, I am aware that I may be overlooking some aspects of the potential impact of this paper, and I will be happy to be corrected and will adjust my rating accordingly. ", "strength_and_weaknesses": "Strengths \nAnalytical derivations and numerical simulations convincingly support the claims of the paper. \n\nWeaknesses \nDale’s principle is a fundamental dogma in Neuroscience and thus structures that violate it cannot be deemed biologically plausible by our current standards. Comparing the advantages of networks that follow Dale’s principle against ones that don’t could be arguably deemed as little more but an interesting mathematical exercise. The main result of the paper, namely that networks obeying Dale’s principle are more robust to initializations is a bit underwhelming and likely to have very low impact in the broader field of Neuroscience and RNNs.    \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster