On the Stability and Scalability of Node Perturbation Learning
Authors: Naoki Hiratani
Yash Mehta
Timothy P Lillicrap
Peter E. Latham
Venue: NeurIPS 2022 Accept (2022)
Keywords: Biologically Plausible Deep Networks
Credit Assignment
Neuroscience
Dynamical Systems

Abstract: To survive, animals must adapt synaptic weights based on external stimuli and rewards. And they must do so using local, biologically plausible, learning rules -- a highly nontrivial constraint. One possible approach is to perturb neural activity (or use intrinsic, ongoing noise to perturb it), determine whether performance increases or decreases, and use that information to adjust the weights. This algorithm -- known as node perturbation -- has been shown to work on simple problems, but little is known about either its stability or its scalability with respect to network size. We investigate these issues both analytically, in deep linear networks, and numerically, in deep nonlinear ones.
We show analytically that in deep linear networks with one hidden layer, both learning time and performance depend very weakly on hidden layer size. However, unlike stochastic gradient descent, when there is model mismatch between the student and teacher networks, node perturbation is always unstable. The instability is triggered by weight diffusion, which eventually leads to very large weights. This instability can be suppressed by weight normalization, at the cost of bias in the learning rule. We confirm numerically that a similar instability, and to a lesser extent scalability, exist in deep nonlinear networks trained on both a motor control task and image classification tasks. Our study highlights the limitations and potential of node perturbation as a biologically plausible learning rule in the brain.

On the Stability and Scalability of Node Perturbation Learning
Naoki Hiratani
Yash Mehta
Harvard; Gatsby Unit, UCL Cambridge, MA, USA
Janelia; Gatsby Unit, UCL Ashburn, VA, USA mehtay@jamelia hhmi org
w.hharttaii @gmail .omail.com com
Timothy P. Lillicrap Deepmind London, UK
Peter E. Latham
Gatsby Unit, UCL
London, UK pel0gatsby. ucl.ac....... acc uk
countzero@googee.com com
Abstract
To survive, animals must adapt synaptic weights based on external stimuli and rewards. And they must do using local, biologically plausible, learning rules a highly nontrivial constraint. One possible approach is to perturb neural activity (or use intrinsic, ongoing noise to perturb it), determine whether performance increases or decreases, and use that information to adjust the weights. This algorithm known node perturbation has been shown to work simple problems, but little is known about either its stability or its scalability with respect to network size. We investigate these issues both analytically, in deep linear networks, and numerically, deep nonlinear ones. We show analytically that deep linear networks with one hidden layer, both learning time and performance depend very weakly on hidden layer size. However, unlike stochastic gradient descent, when there is model mismatch between the student and teacher networks, node perturbation is always unstable. The instability triggered by weight diffusion, which eventually leads to very large weights. This instability can be suppressed by weight normalization, the cost bias the learning rule. We confirm numerically that a similar instability, and lesser extent scalability, exist in deep nonlinear networks trained both a motor control task and image classification tasks. Our study highlights the limitations and potential of node perturbation as biologically plausible learning rule in the brain.
Introduction
The immense success of deep learning in recent years has revived interest the backpropagation algorithm (known simply "backprop") learning mechanism in the brain [[2]. Although in its pure form this algorithm incompatible with biological constraints (most notably, the fact that synapses do not know the weights of other synapses in the brain), many biologically plausible approximations have been proposed 3 approximate learning rules are, or how they perform on large neural networks [TO,, partially because we do not have clear analytical understanding of them.
7 & 9. However, it remains unclear how robust these
In the node perturbation (NP) algorithm, synaptic weights updated according to how the error changes when a small perturbation added to each node a neural network [I]] 12]. If a perturbation improves performance, the weights modified that the perturbation is consolidated, and vice versa (see $3). This algorithm has two advantages over more standard biologically plausible
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

supporting the learning rules. First, synaptic updates rely only on a global error signal and pre- and postsynaptic activity. Such global error signal, mediated by neuromodulators, is commonly observed across the brain [13], and is known to modulate synaptic plasticity [14]. This is in contrast to many biologically plausible credit assignment algorithms, which require vector-valued supervising signals (what the true label was), not just scalar error signals (how accurate the network output was). Second, neural activity in the brain is inherently stochastic [5 [6]. which NP can make use of making noise a feature rather than bug.
Because of its biological plausibility and algorithmic simplicity, NP has been investigated as a potential learning mechanism of several brain regions, including the vocal production circuits of songbirds [1 [8, motor cortex [T]. and the cerebellum 1201. In particular, experimental work on birdsong learning suggest that perturbation of neural activity in the song production circuit via a projection from a region called LMAN is crucial for song acquisition (21 22
presence of perturbation-basee learning.
Here we analyze the learning dynamics of node perturbation. We develop a mean-field theory of NP learning in one hidden layer linear networks, and we show analytically, and confirm empirically, that the minimum training time of NP depends linearly on the number of output nodes, as suggested previously [12] Perhaps surprisingly, we show that the training time is independent of the hidden layer width in the over-paramette over-parameterieed regime. Finally, we show that NP is always unstable when the supervised signal contains noise. This instability can be mitigated by regularizing the weight norms, but that induces bias in the update.
To verify that these results are not specific to deep linear networks, we investigate the scalability and stability of NP in nonlinear deep neural networks applied to the SARCOS 2 and MNIST tasks (24] In both tasks, the minimum learning time required to reach high performance scales sub-linearly with the hidden layer size. These observations support the relative robustness of NP against hidden layer width expansion, a prediction from linear networks. However, NP dynamics is unstable at high learning rate, with the instability preceded by an expansion of the weight norm, as predicted. Weight normalization prevents this instability, but impairs performance, especially for SARCOS. For MNIST, stabilization by weight normalization improves the convergence time of NP against the depth expansion. We also confirmed the stabilization of learning dynamics by weight regularization in convolutional neural networks solving the CIFAR-10 10 task 25 but it also impairs performance.
The minimum learning time of NP is about one hundred times longer than that of stochastic gradient descent (SGD) in both tasks. However, a portion of this discrepancy is explained by the fact that NP is a reinforcement learning algorithm while SGD uses supervising signals. Our study thus suggests that, depending on the task and the architecture, NP may play role as a credit assignment mechanism in the brain. However, our results also indicate that NP alone is not enough to account for the learning ability of the brain. In particular, NP too slow to be practical in supervised image recognition tasks.
2 Related work
NP has been previously proposed as a learning mechanism of birdsong 17 18
motor cortex 19).
and cerebellum (20]. Nonetheless, the practicality of NP as a credit assignment mechanism in the brain remains unclear, all of these studies considered simplified tasks, and scarcely provided analytical insight. Werfel and colleagues did provide analytical insight into the scalability of NP in linear regression setting: they showed that the convergence of NP becomes progressively slower than SGD as the output layer size increases 12]. However, they could not tell whether the slower convergence was due to an increase in the number of perturbed neurons or the number of output neurons, as they were the same. In this work, we distinguish these two possibilities by introducing a hidden layer (see $ for the details). Ficte and colleagues numerically demonstrated that when the loss function is defined on a low dimensional projection of output neurons, the training time doesn't 't depend on the output layer size [[7]. However, it remains unclear how general this result is, as their analysis is limited to a linear regression setting. Lansdell and colleagues used NP to train feedback weights of a deep neural network that hack-propagates the error 26). The performance of their algorithm was comparable to SGD on CIFAR-10. 10. However, it required supervising signals, unlike the vanilla NP, which only needs reinforcement signals. In addition, it remains unelear how much performance gain in their algorithm was brought by the weight alignment rather than from NP.

---

If perturbations are added to synaptic weights instead of neural activity, the algorithm is called weight perturbation [3]. This algorithm is noiser than node perturbation the vanilla setting [12.. but recent result suggests that the weight perturbation algorithm is competitive with, or better than, NP when the feedback signal sparse [27]. should also be noted that weight perturbation is a variation of zeroth order optimization, which is known to scale badly to high-dimensional optimization problems However, recent works found that when there is low-dimensional latent structure in the
28 [
parameter space, this algorithm works efficiently 30
Scalability of noisy weight update rules has also been investigated in the bandit literature h2 In bandit problems, the gradient needs to be estimated from insufficient feedback, thus its estimation inevitably becomes either biased or noisy. This limitation raises the question of whether one should choose a biased update or a noisy one, for which the following property is known [33]. Consider an optimization of strongly convex loss function I(w) with biased noisy gradient. Suppose there is parameter 5 such that, the update rule is noisy but unbiased in the limit 5 - 0, and biased but noise free the limit 8 oo. Then, under some mild conditions, the value of 5 that achieves the
minimax regret under fixed number of iterations n, denoted 8", is given by 8 & V n where d is the number of parameters. Consequently, when the number of iterations n is small while the number of parameters d is large, as in the brain, the system should choose biased update, not noisy one. However, the results from convex optimization are often not directly applicable to deep neural networks, because the effective number of degrees of freedom is typically much smaller than the number of parameters in these networks unclear if the brain should always choose a biased learning rule over a noisy one, such NP.
R and the loss function is non-convex. Thus, it remains
3 Node perturbation is unbiased, but noisy
We formulate node perturbation in the context of deep learning. Let us consider a vanilla deep feedforward network
24 fxx (w == 1,2, K
where No and æ the input and the output respectively, and f*c()) is element-wist activation function. Throughout the paper, we use bold italic lower case letters to denote column vectors and bold italic upper case letters to denote matrices. Adding small Gaussian perturbation o each layer except the input layer gives us a perturbed network,
T fu (W + ofs), k= 1,,....,
(2)
where (ExE?) Jg with I the identity matrix in the appropriate dimension, and a < 1. Under a loss function {( zo), the node perturbation update is
Sw - " (f(0K, 2o) - ex 20)) E
In words, if the perturbation decreases the error (i.e., 8- 0), the weights are shifted towards the direction of the perturbation (Ex), and vice versa. Importantly, in this update rule, the network only needs to know how much the loss changes when perturbation is added to the network. This is in contrast to SGD and most of its biologically plausible variants, which require a supervised signal telling what the correct answer It is straightforward show that, the small perturbation limit (ie. a -> 0; see Appendix I.
K
gk = de
SW
Σ ENET g where
and h; WATE-1-
(4)
Ohk
Taking the expectation over &, we recover the SGD update rule,
(SWT") = -f = SWegd
(5)
Consequently, NP is unbiased relative to SGD. However, because the weight update is driven by random perturbation to the nodes, NP is much more noisy than SGD. For instance, the cosine similarity between SGD and NP update at k-th layer is given by
(cos (4 lowad SW mp
1
(6)
VL
3

---

where Lx is the number of neurons in the k-th layer (see Appendix A.2 Thus, for wide networks, the NP and SGD updates becomes nearly perpendicular (see Figs. 4A and S4A).
Another way to characterize how noisy NP is to look at the noise covariance. Because both SGD and NP are unbiased against the true gradient, both update rules can be written
sw -1) (19. 2
where Q up or sgd, and 2 is (not necessarily Gaussian) random matrix. We show in Appendix
A.3 that the covariance of Z under NP, denoted Cil is written as,
Cil 20 + Ebg Σ
9 11
I @ Db 23-1100
where, I is a size-La identity matrix, Coadd is the noise-covariance under SGD, and "8" denotes tensor product between two matrices (i.e., [A & Blaskt A, B&i). Because the second term is zero unless two weights share the same postsynaptic neuron, the noise correlation of NP updates across neurons in different layers, or across different neurons in the same layer, is only factor of two larger than that of SGD. However, the NP update at each synapse contains additional, mostly independent, noise from the second term that scales with the total gradient norm > |l9m.l This term is typically much larger than the SGD variance, which only depends on the gradient amplitude at each weight (see Eq.
4 NP in one hidden layer linear networks: scalable but unstable
We have so far observed that NP is unbiased but noisy, meaning that a one-step update by NP is just noisy version of SGD. However, it is not clear what that implies for the global stability and the scalability of the algorithm. Because it is very difficult to address these issues in general, here we first focus deep linear networks in student-teacher setting 335 36).
In the absence of hidden layers (i.e., in a linear regression setting), it is known that the minimum number of training steps required to reach a target error level Etg from initial error e, is
Tagalco + <ggl= = Lz log(<o/ Taplco - Etgl = Ly (og((c//egg))
where Lg and Ly are input and output layer sizes, respectively (see [2] and Appendix B.6). Thus, NP is linearly slower than SGD as the number of output units increases. Note that at small learning rate m, the learning trajectories of NP and SGD behave similarly (left edge of Fig. S1, in the Appendix). However, the optimal learning rate- the learning rate that minimizes the training time- - is smaller under NP than SGD, so the minimum training time becomes longer (Fig. S1; here, the optimal learning rates are the points at which the curves touch the horizontal dotted lines). Nevertheless, in the linear regression setting, NP learning dynamics is simple, and monotonic: the error either decreases or increases monotonically throughout learning. As we will see, however, this is not the case in deep neural network; they display non- non-trivial, and non-monotonic, dynamics.
Equation D tells us that NP is slower than SGD by a factor of Ly, the number of output units. However, doesn't t tell in general whether training time scales with the number of output units the number perturbed units, because the two are the same in linear regression. To disambiguate between these two possibilities, we analyze linear network with one hidden layer,
= W2Ww
(10)
where the matrices W₂ and W1 are L, x L, and Ln, L., respectively. We consider a student-teacher setting, in which the target output y" is generated from a teacher network,
y" Ax + o.C.
(11)
where 3 is zero mean Gaussian noise that creates a mismatch between the student and the teacher networks. We implement NP this network by applying small perturbations to both the hidden and output layers simultaneously (see Eq. 3 and Appendix B 1). Although the exact learning dynamies is not tractable for this network, the limit of large hidden layer size, and under the assumption that weight updates are dominated by noise, we can use mean-field approximation to

---

B
10"2
Errot
Errr
Error
10-2
10
0=1.44"
20"7
n=1
10"2
=3
g*1.21
1.2n"
10-3
g=1.0/)
10
1=
1.0nn
10"4
0=0.8n"
7=0.8
0.5a,
10-1
10
10-3
Iterations
Input weight norm
Input weight norm
Figure 1: Learning dynamics of NP in one hidden layer linear networks. A) Learning dynamics of node perturbation under various learning rates. Solid lines are empirical results and dotted lines are theory (Eq. 12p rf" is the analytically esttmmmtd critical learning rate. B) The same A, but plotted in the phase space spanned by the imput weight norm and the error. Lines represent trajectories with different random seeds. Dotted vertical lines are the threshold 2/c,. C) Learning dynamics under different initial weight norms. In all three panels, we set Lg = 100, L, = 10000, Ly = 10, and of = O. See Appendix C.2 for further details. Simulation codes for the figures are made available at https:/ thub. com/nhiratani htw.epeeeturattiio..
gain considerable insight into the learning dynamics as described below. Our derivation proceeds in two steps. First, we introduce four order parameters capturing the weight norms || Wiills Wl and [[ [ error IW2 W All and their inner product tr[( W W A)" Wil. I. and approximately write down the learning dynamics terms these order parameters (B.2). Secondly, we reparameterize the learning dynamics, which leads us to two-variable description of the dynamics (B.3). The derivation of the mean-field approximation is detailed in Appendix
Scalability of NP at the zero teacher noise limit. We first consider zero teacher noise (o, 0 in Eq. [T]. In network satisfying Ln > Lg >> and Ln >> Ly > 1, and using the Kavier-Glord initialization (see [37 and Eq. 69), for the first ((L, iterations, the learning dynamics of NP is described by the following coupled dynamics (see Appendix 3).
à = CoC, é = - (2 c,a) ac,
(12)
W1 1| is the normalized input weight norm, and 6o is a positive coefficient proportional to the learning rate (see Eq. 76). This simple dynamical system captures the learning dynamics of NP well (Fig. 1A; solid lines are simulations, and dotted lines are Eq. [2]. Because 2 0 by definition, the norm a increases monotonically with time. By contrast, the loss either decreases 2/co or not. When the learning rate 77 is fixed to a small value, co is also small. Therefore, the weight norm increases slowly (purple and blue lines in Fig. 1B), and the weight norm threshold 2/c, is relatively large (purple and blue vertical dotted lines). Thus, the error converges to zero before the weight norm crosses the threshold. In contrast, when the learning rate is larger than the critical value r"" (defined by Eq. 83), the weight norm surpasses the threshold 2/c, before the error converges zero; that point the error starts go up again (green lines in Fig. 1B). Similarly, if the initial weight norm is set to a large value, the error rises before converging to zero (green vs. blue lines in Fig. IC). This non mon-oontooicc learning dynamics in contrast to the linear regression setting in which the error either decreases or increases monotonically throughout learning.
where c = 1 W2W, A is the loss, T. P.
increases depending
whether is smaller than
When optimize the learning rate, find that the minimum number of training steps required to reach a target error level, Ctg, from an initial error, fo, under SGD and NP are given by
Tageleo + + La log(e Tapleo + eegl = Log Ly 1+ 30
rag
a Coletnt a 22/3) a²
(13)
where Bor Cos Etot are (1) coefficients, and and atg are the weight norms under the initial error 6, and the target error cgg, respectively (see Appendix B.3). Notably, the training time Top doesn't depend on the hidden layer size Li, except for small-size effects a, and Bo (see Eq. 71. indicating
5

---

165
Under parameterzze
Over-paramelerized
Miniuu training J
10f
Minimum training time
107
10*
10*
SGD
SGD
200
103
109
109
10a
20tt
102
Hidden layer size 4.
Output layar by
W,
Figure 2: Scalability of NP in one hidden layer linear networks. A, B) Minimum training time of SGD (blue) and NP (orange) under range of hidden (A) and output (B) layer sizes, linear network with one hidden layer. Lines are theory (Eq. 13 points are empirical estimations. In panel A, by = 10; in panel B, LA 10000. We used L 100 and o; = 0.0 in both. C) Schematic figures of a weight update under NP. Blue shade represents the loss landscape (blue: low loss, white: high loss), and green and red arrows represent good and bad directions, respectively.
that NP doesn't it slow down linearly with the number of perturbed nodes. Indeed, simulations show that the minimum training time almost flat when plotted against the hidden layer size (Figs. 2A and S2A). Moreover, the optimal learning rate in the simulations scales as 1 x Ln 1/3 as predicted (Fig. S2B). In contrast, the training time increases linearly with the output layer size Ly (Fig. 2B).
may appear puzzling that training time saturates in the large hidden layer size limit (Fig. 2A and S2A), considering that weight updates under NP almost random in that limit (Eqs. 6 and However, as illustrated in Fig. 2C, while the noise in the weight updates (gray circles) increases the loss for some directions (red arrows), it decreases it for others (green arrows). When the loss function is strongly convex, there are always more bad directions (red arrows) than good directions (green arrows), as illustrated in the left panel of Fig. 2C. Therefore, and also because the loss associated with red arrows is larger than the gain with green arrows in a convex loss, noise is detrimental to learning on average. Nevertheless, in overparameterized networks, there are large number of directions that don't affect the loss (because the Hessian of an over-parameterized neural network is typically low rank [ ); such directions are shown in the right panel of Fig. 2C. Since NP noise mostly homogeneous, the more overparameterized the network is, the lower the volume of weights space in which the loss increases. Thus, in an over-paramenn wver-parameterized network, NP noise is not as detrimental as anticipated from convex optimization theory.
Instability of NP in the presence of teacher noise. So far we have focused on the zero-noise limit (o, = 0). We now investigate the effect of zero noise (a, * 0) in the teacher network, which introduces a mismatch between the student and the teacher networks. Our goal is to understand how this mismatch affects the stability of the learning dynamics. In this setting, the error decreases only when (see Eq. 84a)
(14)
L2 - coa)
Thus, even at very small learning rate (proportional cu), eventually Eq. [4 will not be satisfied, and the error will begin to increase. This process is illustrated in Fig. 3A: the loss decreases until it crosses the dotted lines, which are the nullelines, given by equality Eq. 14
// (2 c,a)), Because weight norm is monotonically increasing function of time, the right-hand side of Eq. 14 is also monotonically increasing. Thus, this instability occurs faster under a larger learning rate (green vs blue lines in Fig. 3A).
The black line in Fig. 3B shows the number of iterations takes until the error starts to increase. At learning rate > '7",, the number of iterations scales roughly as 3 (right side of vertical line), but the scaling becomes more moderate (x 1) 1155) at a lower learning rate (left side). Moreover, the error eventually surpasses the initial error level, though that takes a long time when the learning rate is below the critical value (gray line). Under SGD, the teacher noise affects the learning dynamies in a qualitatively different manner. Up to a certain learning rate, SGD dynamics is always unstable from the initial update (filled circles in Fig. 3C), whereas below the critical learning rate, the error keeps decreasing almost monotonically at least for 10 iterations (open circles).

---

93%
124
1:9
ileratonn
Enrer
Error
An"
1e1 Dn"
32000 10000
(10 En"
Intial errrr
q-0 21g"
Miniuum error
32000
1011
10P
181
16-
12-11
12-4
1277 1077
Input weight norms
Learning rale
Leaminn rate
Denations
Figure 3: Instability of NP in one hidden layer linear networks with teacher noise. A) Learning dynamies of node perturbation in the presence of teacher noise. Lines are simulations, dotted lines are Eq. [4 B) The number of iterations until the error reaches minimum and starts to increase (black) and until the error goes back to its initial value (gray). Points are simulations and lines are theory calculated from Eq. 84a The vertical dashed lines is 17".. C) The number of iterations until the error reaches a minimum under SGD, The dashed horizontal line represents the iteration where simulation was terminated, meaning that on average the error decreased monotonically for at least 10t iterations. D) Learning dynamics under three different hidden layer sizes with (solid lines) and without (dotted lines) weight normalization. The learning rates were set to the critical rate 17" under o, = (Eq. 83). The dotted lines are almost identical to each other due to the scale invariance. In all panels we used L 100, Ly 10, and of 0.1. In panels A-C, we used Ln 10000.
Our analysis so far indicates that the instability of NP is triggered by the expansion of the weight norm. Thus, we expect the system to regain stability if we regularize the weight. Indeed, if we keep the norm of the incoming weights of each neuron constant, the error converges unlike for the un-regularized network (Fig. 3D; solid vs. dotted lines). However, because the weight normalization induces bias in the weight update, the error no longer goes to zero; instead, it saturates at a finite value. The effect of this bias is particularly detrimental when the hidden layer size is large, because strong weight normalization is necessary for a larger network (yellow vs purple lines Fig. 3D).
5
NP in nonlinear networks solving motor and visual tasks
Our analysis in deep linear networks revealed that NP scales well against vve-parameterizztionn but the learning dynamics is susceptible to an instability because noise expands the weight norm monotonically. Weight normalization suppresses this instability, but the normalization also introduces a bias that impairs scalability. To see if these results generalize to more complex tasks that better mimic typical credit assignment problems the brain faces, we analyze nonlinear networks solving the SARCOS [23] MNIST 24. and CIFAR-10 tasks 25
We first applied NP to the SARCOS dataset. kinematics dataset collected from a seven degree-of- freedom SARCOS robotic arm [23 38 39. The task is to predict the torques at the joints (seven- dimensional output) given their positions, velocities, and accelerations as inputs (21 -2lddimensional input). We chose this task because NP is previously proposed as learning algorithm for motor systems 17 19 20 As expected, NP is indeed noisy, even with mini-batch implementation, resulting a low cosine similarity with SGD (Fig. 4A; here mini-batch size is 100). In particular, in one-hidden layer networks the cosine similarity between the NP and SGD weight updates in the hidden layer fall off as 1/ VL., while the similarity in the output layer stays roughly constant, as predicted by the theory (Fig. 4A and Eq.
Moreover, learning under NP is indeed susceptible to an instability, as predicted. Under a small learning rate, the learning dynamics of NP is stable for more than 10 epochs (purple and blue lines in Fig. 4B), but as the learning rate increases, the error starts to diverge (green lines). Because the weight norm increases monotonically, we expect NP dynamics eventually become unstable even under a small learning rate, though it is difficult to confirm numerically due to the high computational cost. The learning- rate dependence of this instability is also similar to what we observed in deep linear networks (compare Fig. 4C with Fig. 3B). At relatively high learning rate, the error, measured by the test error, surpasses the initial error level immediately after the error starts to go up (right half of Fig. 4C: black and gray lines are close to each other). On the other hand, at lower learning rate, the error increases slowly after it reaches the minimum (left half of Fig. 4C). This instability absent from SGD learning; in that case, learning either stable unstable from the beginning (Fig, S3A).
7

---

02
10'
101
NP-SGD cosino similarity
0.1
Epochs
71 Dx10
201
2x100
307
Hidden layer
0x10
Initial error
Output layer
(x3.2x10
Minimum error
0=110x000
200
400
200
1600
100
101
107
10a
10
10-3
10"
Hidden layer width
#Epochs
Learning rate
Minimum training time m
F
10
304
Reinforce
Test error
10a
SGD
301
Test
/F1. 0xx10
L.= 1600
1=1.0x10"
L,= 800
173.2x10
4F 400
50
1.00
200
-00
800
1600
Input weight norm
Hidden layer width
#Epochs
Figure 4: Nonlinear networks solving SARCOS. A) Cosine similarity between NP and SGD at various hidden layer widths. Solid and dotted lines are the similarities around the beginning (c (c= 50.0) and the end (c = 5.0) of learning, respectively. B) Learning dynamics NP under various learning rates. Shades, which are barely visible at small learning rates, the standard deviations over five random seeds. C) The number of epochs until the error starts to increase (black) and the enror goes back to the initial error level (gray). D) Learning dynamics of NP in space spanned by the input weight norm and the test error measured MSE. E) Minimum training time of NP, Reinforce, and SGD in one hidden layered networks with various hidden layer widths. In dashed green line, we added perturbation to both the intermediate and output layers. F) NP learning with (solid) or without (dotted) weight normalization. See Appendix for the details.
Plotting the curves as in deep linear networks (Fig. 4D vs. Fig. 1B and 3A). This result suggests that expansion of the input weight norm underlies the observed NP instability, even in nonlinear networks.
against the squared weight norm of the input layer [| Il
U-shaped
We next estimated the scalability of NP against the hidden layer width. Under NP, the minimum training time required for reaching the target (here set the test error, 5.0, which corresponds to R² = 0.986, roughly one hundredth of the initial error) depends only weakly on the hidden layer width (Fig. 4E Fig. 2A; Fig. S3B for the choice of the optimal learning rates). However, we also observed that its learning efficiency is about one hundred times worse than SGD (orange vs. blue lines in Fig. 4E). One key difference between NP and SGD, which may partially explain this discrepancy, is that NP is a reinforcement learning algorithm that doesn't 't use a vector-valued supervised signal, unlike SGD.. To examine its effect, consider the following reinforcement learning algorithm 40) By adding perturbations only to the output layer, the gradient with respect to the last layer is estimated as
gK R 9K = (((ā,, [(OK, xo)) Ek/o.
(15)
By back-propagee the gradient, ŌK. the network can approximately perform SGD based on the reinforcement signals {xxx) and {(xg). The minimum training time under this reinforcement learning algorithm is about three times longer than SGD (solid green versus blue lines in Fig. 4E), though much better than NP. Moreover, assuming that the perturbation originated from intrinsic noise in the system, we added perturbations to the hidden layer as well. This additional perturbation impairs the performance of the reinforcement learning algorithm (dashed vs solid green lines). However, NP still five ten times slower than this noisy reinforcement learning (orange dashed green lines).
Our analysis of deep linear networks indicates that weight normalization prevents the instability in NP learning dynamics. Indeed, in the presence of neuron-wise weight normalization (see Appendix learning dynamics becomes stable even when the hidden layer size large (dotted vs solid lines Fig. 4F), but this prevents the error from converging to the target level (e 5.0).

---

100.0
NP Weight-noraat NP
97.5
1 uaining time
Minimum training time
sccuracy
95.0
NP
92.5
Reinforce
6000
90.0
g-1 8x10
SGD
87.5
n=3 2x10
4200
g 8x10
85.0
0 0=100
2000
82.5
0=1 11110
80.0
10000
#Epochs
Hidden layer width
Depth (#hiddon layors)
Figure 5: Nonlinear networks solving MNIST. A) Learning dynamics of NP under various learning rates. Shades represent the standard deviation over 5 random seeds. B) Minimum training time of NP, Reinforce, and SGD in one hidden layered networks at various hidden layer widths. Solid and dashed green lines are reinforcement learning without/with additional perturbation to the hidden layers, respectively. C) Minimum training time of NP with or without weight normalization in networks with various depths. Here, we all the hidden layer sizes to 300 regardless of the depth.
At a low learning rate, NP and SGD show similar error curves (Fig. S3C), but it doesn't 't mean that the two algorithms learn the same representation. We found that the linear dimensionality of the hidden layer representation, measured by the participation ratio of PCA eigenvalues (see Appendix C.T.. was lower under NP than SGD, even at low learning rate (Fig. S3D). This result implies that it may be possible to distinguish the two learning rules, just by observing activity in the hidden layer.
NP exhibits similar learning dynamics when applied to MNIST (Figs. and S4), Under a large learning rate, learning becomes unstable before reaching high performance (green lines in Fig. 5A). The minimum training time depends only sub-linearly on the hidden layer width (orange line in Fig. 5B). Compared to SGD, NP learning is hundred to thousand times slower, but the performance deficit against reinforcement learning is smaller (Fig. 5B). Although the analysis of the depth dependence is complicated even in the linear setting, we empirically observed that the minimum learning time of NP scales supra-linearly with respect to depth (orange line in Fig. 5C). However, this effect be ameliorated by introducing weight normalization (red line). Lastly, applied NP to a convolutional network learning CIFAR- 10, and found that weight regularization stabilizes NP learning dynamics (Fig. S5). However, only achieves a low accuracy 50%) after few thousand epochs of training due to its bias and slowness.
6 Discussion
We analyzed the learning dynamics of NP in deep neural networks and revealed its limitation and potential. This study provides several insights into biologically plausible learning mechanisms.
Firstly, considering biological implementation, it is often useful to measure the performance by the minimum number of iterations required to reach a target - which could, for instance, be human-level performance. This is because an algorithm is biologically implausible if it takes inordinately long time to reach good performance. For instance, under a sufficiently small learning rate, NP eventually achieves good performance, since it is unbiased ($3). However, convergence is significantly slower than SGD ($4 and $5), which limits its utility, especially in the presence of supervising signals. Though we found the minimum training time of NP relatively robust against over-parameetriizatoo. its scalability against complex supervised learning tasks is clearly limited due to this slowness.
In the brain, learning often needs to rely scalar- valued reward signals rather than supervised signals. Although NP is much slower than SGD, its performance deficit is smaller, when compared to reinforcement learning rule using the error backpropagation, though NP is still more than one order of magnitude slower (Figs. 4E and 5B). Thus, NP might be utilized in a noisy reinforcement learning problem, such as motor learning, as building block of learning mechanism.
Our work highlights the importance of theoretical investigations an idealized scenario, such as a deep linear network, not only for understanding how the proposed learning algorithm works, but also for obtaining insight into what kind of regularization needed for making a learning algorithm works efficiently. We revealed that neuron-wist weight normalization stabilizes the learning dynamics. By

---

contrast, a constant-rate weight decay doesn't t work robustly, because the change in the weight norm constant (black vs colored lines in Fig. S4F).
Finally, it is important to analyze the representation in the hidden layers in the learned circuit for obtaining experimentally testable prediction 41] 2
Although the representation in the output layer is almost invariant among any successful learning rules by definition, hidden layer representation may vary depending on the learning rule, particularly in over-parameterized neural networks. Here, we showed that the linear dimensionality of the hidden layer representation is different between NP and SGD, even when they achieve similar performance (Figs. S3CD and S4E).
Acknowledgement
This work was supported by the Swartz Foundation (NH), Wellcome Trust (110114/2/15/2: PEL), and the Gatsby Charitable Foundation (PEL).
References
[1] James CR Whittington and Rafal Bogacz. Theories of error back-propagation in the brain. Trends in cognitive sciences, 23(3):235-25 250, 2019.
Timothy P Lillicrap, Adam Santoro, Luke Marris, Colin J Akerman, and Geoffrey Hinton. Backpropagation and the brain. Nature Reviews Neuroxcience, 21(6):335-346 2020.
Marwan Jabri and Barry Flower. Weight perturbation: An optimal architecture learning technique for analog vlsi feedforward and recurrent multilayer networks. IEEE Transactions on Neural Networks, 3(1):154-) 157, 1992.
[4] Barry Flower and Marwan Jabri. Summed weight neuron perturbation: An (n) improvement over weight perturbation. In Advances in neural information processing systems, pages 212-219, 1993.
[5] Xiaohui Xie and H Sebastian Seung. Equivalence of backpropagation and contrastive hebbian learning in layered network. Neural computration, 15(2):441-454, 2003.
Dong-Hyun Lee, Saizheng Zhang, Asja Fischer, and Yoshua Bengio. Difference target propagation. In
Joint european conference on machine learning and knowledge discovery in databases, pages 498-515. Springer, 2015.
[7] Timothy P Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin Akerman. Random synaptic feedback weights support error backpropagation for deep learning. Nature communications, 7(1):1-10, 2016.
[8] James CR Whittington and Rafal Bogacz. An approximation of the error backpropagation algorithm in predictive coding network with local hebbian synaptic plasticity. Neural computation, 2965):222-1122 2017.
[9] João Sacramento, Rui Ponte Costa, Yoshua Bengio, and Walter Senn. Dendritic cortical microcircuits approximate hackpropagation algorithm. Advances in neural information processing systems, 31, 2018.
[10] Sergey Bartunov, Adam Santoro, Blake Richards, Luke Marris, Geoffrey E Hinton, and Timothy Lillicrap. Assessing the scalability of mooogicalll-mottvatee deep learning algorithms and architectures. In Advances in Neural Information Processing Systems, pages 9368-9378, 2018
[11] Beraard Widrow and Michael A Lehr. 30 years of adaptive neural networks: perceptron, madaline, and backpropagation. Proceedings of the IEEF, 1415511155 1990.
[12] Justin Werfel, Xiaohui Xie, and H Sebastian Seung. Learning curves for stochastic gradient descent in linear feedforward networks. Advances neural information processing systems, pages 1197-1204, 2004.
[13] Eve Marder. Neuromodulation of neuronal circuits: back to the future. Neuron, 76(1):1-11, 2012.
[14] Zuzanna Brzosko, Susanna B Mierau, and Ole Paulsen. Neuromodulation of spike-timing-deeendeel plasticity: past, present, and future. Newron, 103(4):563-58)) 581, 2019.
[15] Michael N Shadlen and William T Newsome. The variable discharge of cortical neurons: implications for connectivity, computation, and information coding. Journal of neuroscience, (0100)3870-3880 1998.
10

---

[16] Michael London, Arnd Roth, Lisa Beeren, Michael Häusser, and Peter E Latham. Sensitivity to per- turbations in vivo implies high noise and suggests rate coding in cortex. Nature, 466(7302):123-127, 2010.
[17] Ila R Fiete, Michale Fee, and H Sebastian Seung. Model of birdsong learning based on gradient estimation by dynamic perturbation of neural conductances. Journal of neurophysiology. 98(4):2038-2055 2007.
[18] Jorgen Kornfeld, Michal Januszewski, P Schubert, Viren Jain, Winfried Denk, and Michale S Fee, An anatomical substrate of credit assignment in reinforcement learning. BioRxiv, 2020.
[19] Thomas Miconi. Biologically plausible learning in recurrent neural networks reproduces neural dynamics observed during cognitive tasks. Elife, 6xc20899, 2017.
[20] Guy Bouvier, Johnatan Aljadeff, Claudia Clopath, Célian Bimbard, Jonas Ranft, Antonin Blot, Jean-Piert Pin-rieere Nadal, Nicolas Brunel, Vincent Hakim, and Boris Barbour. Cerchellar learning using perturhations. Elife, 7:e31599, 2018.
[21] Dmitriy Aronov, Aaron S Andalman, and Michale S Fee. A specialized forebrain circuit for vocal babbling in the juvenile songbird. Science, 32065876):60066666 634, 2008.
[22] Farhan Ali, Timothy M Otchy, Cengiz Pehlevan, Antoniu L Fantana, Yoram Burak, and Bence P Ölveczky. The basal ganglia is necessary for learning spectral, but not temporal, features of birdsong. Neuron, 80(2):494-506, 2013.
[23] Sethu Vijayakumar and Stefan Schaal. Locally weighted projection regression: An o (n) algorithm for incremental real time learning high dimensional space. In Proceedings of the seventeenth international conference on machine learning (ICML 2000), volume 1, pages 288 293. Morgan Kaufmann, 2000.
[24] Yann LeCun. Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2278-2324, 1998.
[25] Alex Krizhevsky, Geoffrey Hinton, et Learning multiple layers features from tiny images. 2009.
[26] Benjamin James Lansdell, Prashanth Ravi Prakash, and Konrad Paul Kording, Learning to solve the credit assignment problem. In International Conference on Learning Representations, 2020.
[27] Paul Manfred Züge. Christian Klos. and Raoul-Martin Memmesheimer. Weight perturbation learning outperforms node perturbation on broad classes of temporally extended tasks. bioRxiv, 2021.
[28] Andrew R Conn, Katya Scheinberg, and Luis Vicente. Introduction to derivative-free optimization. SIAM, 2009.
[29] Yurii Nesterov and Vladimir Spokoiny. Random gradient-free minimization of convex functions. Founda- tions of Computational Mathematics, 17(2):527-566, 2017.
[30] Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and llya Sutskever. Evolution strategies as a scalable alternative to reinforcement learning. arXiv preprint arXiv: 1703.0380..4 2017.
[31] Daniel Golovin, John Karro, Greg Kochanski, Chansoo Lec, Xingyou Song, and Qiuyi Zhang. Gradientless descent: High-dimensional zeroth-order optimization. arXiv preprint arXiv: 0ttt:99111.6617 100317, 2019.
[32] Tor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
[33] Xiaowei Hu, LA Prashanth, András György, and Csaba Szepesvári. (bandit) convex optimization with biased noisy gradient oracles. In Artificial Intelligence and Statistics, pages 819-828, 2016.
[34] Levent Sagun, Utku Evei, Ugur Guney, Yann Dauphin, and Lcon Bottou. Empiriaal analysis of the hessian of over-parametrizee neural networks. arXiv preprint 1706.01706 2017.
[35] Pierre Baldi and Kurt Homik. Neural networks and principal component analysis: Learning from examples without local minima. Neural networks, 2(1):53-58, 1989.
[36] Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of lcarning in deep linear neural networks. arXiv preprint arXiv: /3/2.6/20, 2013.
[37] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics. pages 256-256. JMLR Workshop and Conference Proceedings, 2010.

---

[38]
Christopher K Williams and Carl Edward Rasmussen. Gaussiun processes for machine learning, volume 2. MIT press Cambridge. MA, 2006.
[39] Eren Sezener, Agnieszka Trabska-Barwiiskee Dimitar Kostadinov, Maxime Beau, Sanjukta Krishnagopal, David Budden, Marcus Hutter, Joel Veness, Matthew Botvinick, Claudia Clopath, al. A rapid and efficient learning rule for biological neural eircuits. BioRxiv, 2021.
[40] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning. 8(3-4):229-256, 1992.
[41] Timo Flesch, Keno Juechems, Tsvetomira Dumbalska, Andrew Saxe, and Christopher Summerfield. Orthogonal representations for robust context-dependent task performance in brains and neural networks. Newron, 2022.
[42] Francesca Mastrogiuseppe, Naoki Hiratani, and Peter Latham. Evolution neural activity circuits bridging sensory and abstract knowledge. bioRxiv, 2022.
[43] Karthik Abinav Sankararaman, Soham De, Zheng Xu, Ronny Huang, and Tom Goldstein. The impact neural network overparameetrizztion on gradient confusion and stochastic gradient descent. In International conference machine learning, pages 8469-8479. PMLR, 2020.
[44] Nicholas Cheney, Martin Schrimpf, and Gabriel Kreiman. On the rohustness of convolutional neural networks to internal architecture and weight perturbations. arXiv preprint 7rXiv:1703.08245, 2017.
[45] M Copelli, R Eichhorn, Osame Kinouchi, M Biehl, Simonetti, P Riegler, and Nestor Caticha. Noise robustness in multilayer neural networks. EPL. (Europhysics Letters), 37(6),22, 1997.
[46] Gina G Turrigiano. The self-tuning neuron: synaptic scaling of excitatory synapses. Cell, 135(3):422-435, 2008.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A] This is
neuroscience-oriented theoretical work with limited direct societal impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Appendix
B.
(b) Did you include complete proofs of all theoretical results? [Yes] We included the detailed derivation of the theoretical results in Appendices A and B.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed reproduce the main ex- perimental results (either in the supplemental material or as URL.)? [Yes] See Ap- pendix C and anonymized repository: https: //github com/nhiratani/node. perturbation.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix C.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] We depicted variability over random seeds in Figs. IBC, 4BF, and 5A. For the rest of figures, we used the average trajectory over ten (linear networks) or five (nonlinear networks) random seeds, to capture the average behavior. (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Appendix C.T
4. If you using existing assets (c.g., code, data, models) or curating/releasing assets...
12

---

(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals. if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "rJpSaPMlEsW", "reviewer": "Reviewer_bpjP", "summary": "The authors study a stochastic gradient-free learning rule for deep feed forward neural networks, node perturbation. The learning rule in brief is to perturb the activity of each neuron by a Gaussian and then update the corresponding weights in proportion to the sampled difference in the loss function.\n\nThe authors develop a mean field theory for a two-layered linear network whose target function is given by a linear transformation with additive Gaussian noise. This allows them to solve for a critical learning rate in which the learning process is as fast as possible. They find that even with the critical learning rate, node perturbation is slower than stochastic gradient descent. In particular, it is asymptotically slower by approximately a factor of the output size. In addition, their theoretical results reveal an instability in the error dynamics once a certain threshold is crossed, and the authors introduce weight regularization to remedy this.\n\nFinally, the authors numerically test the performance of the regularized node perturbation algorithm on the supervised learning tasks MNIST and SARCOS, and show that the algorithm is indeed capable of learning these tasks.", "questions": "My most central and concrete question is whether NP is slower or faster than the reinforcement algorithm. I got mixed messages from the figures and from the discussion section.\n\nIn the abstract of Fiete et al, 2007, the authors state that their version of node perturbation \"learns in a reasonable amount of time\" and \"Varying the number of neurons in HVC and RA has little effect on learning time.\" Do these 2007 results agree or disagree with the present results and why?", "limitations": "Societal impact is not very relevant; the results of the paper are far too theoretical for negative societal impact to be speculated upon. Performance limitations were adequately reflected upon.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The authors provide original theoretical and numerical results about a particular learning algorithm which are clear and straightforward to understand. From a machine learning perspective, the main drawback of the paper is that they are mostly showing weaknesses of an existing algorithm instead of introducing something new and performant. That said, the weight regularization does successfully stabilize the learning process and it is somewhat interesting that the algorithm is able to learn MNIST and SARCOS with the regularization in place.\n\nOne problem I have with the SARCOS figures is that there is no benchmark shown; everyone knows that an accuracy of 98% on MNIST is acceptable, but since I'm not intimately familiar with the SARCOS task I would like to see either a SoTA MSE taken from the literature, or at least the MSE of the SGD-trained models alongside the NP-trained models for comparison. It is explained in the text that the target error was 5.0, but it is still not clear to me whether or not this is a good value in the grand scheme of algorithms which solve this task. Another way to make the error values easier to interpret would be to report $R^2$ instead of MSE, but either way some kind of benchmark should be shown.\n\nThe central justification for the lacking performance of the algorithm is its biological plausibility. This makes sense, but I did not get a very clear message from the authors as to whether or not they really believe this algorithm is used somewhere in the brain. I agree with the author's statement \"an algorithm is biologically implausible if it takes an inordinately long time to reach good performance.\" However, I did not understand the citation of Figures 4E and 5B when they said \"its [NP's] performance deficit is smaller when compared to\na reinforcement learning rule using the error backpropagation.\" From what I understood, these figures show that it takes longer to train NP than reinforcement learning or SGD, especially as depth increases. If I understood correctly, it seems this would be in contradiction to what is said in the discussion, and then it is difficult to rationalize the use of NP in the brain. There should be some clarification either in the figures or in the discussion.\n\nI also think its worth noting that inverse problems (including SARCOS) are often solved innately by the brain. When a calf (or any quadruped) is born, they don't spend any number of epochs learning what torques are required to move their hooves from one place to the other, walking occurs immediately, suggesting that the capability was genetically encoded into the architecture, not learned. The authors could, of course, defend the claim that NP is used for learning more complex motor tasks, such as those performed by humans, but then it would also have to be clear whether or not the algorithm is really fast enough for this.\n\nI understand the motivation of using SARCOS to test the capability of the algorithm to solve a motor task. However, since the paper's story has biology at the center, I think it would be much more convincing if the authors also took one of the tasks from one of their biological citations and replicated it with their algorithms. Even including the purely model-based task from Fiete et al, 2007 would make the argument that NP aids in songbird learning more than a passing remark.\n\nOverall, I think a more clear story would be greatly beneficial to the paper and better determine its significance. If a central argument is that the brain *probably* does use regularized NP for motor learning, I would be skeptical and would be interested in further discussion of the biological literature; the citations Kornfeld et al, 2020 and possibly Bouvier et al, 2018 might support this line of argument if the results were better reflected upon in the paper. On the other hand, the story could be that the slowness of the algorithm, even in an idealized setting, demonstrates that the brain cannot be using it except as an auxiliary tool to a more powerful learning method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "S76nPIiF27f", "reviewer": "Reviewer_pkGH", "summary": "The paper considers the dynamics of training deep networks with node perturbation. The authors provide a detailed theoretical analysis of learning dynamics of node perturbation vs. SGD in linear networks with one hidden layer in the limit of infinite width. The analysis reveals that the input weight norm determines whether the loss increases or decreases during training with node perturbation, with large weight norm corresponding to unstable training. This motivates using weight normalization for node perturbation. Empircally, on linear and nonlinear networks, weight normalization stabilizes training with node perturbation.", "questions": "How does node perturbation with weight normalization perform in more complex datasets and architectures?\n\nHow might neuron-wise weight normalization be implemented biologically? Is there biological evidence for such a normalization?\n\nWhy is node perturbation a relevant model of biological learning to consider compared to other proposed biologically-plausible learning rules?", "limitations": "The authors address the limitations of their work in the discussion section. As they note, the utility of node perturbation is limited in the supervised setting, although its utility is more clear in a reinforcement learning setting. The authors may also want to comment on the potential applicability of their linear analysis to other settings; as they empirically find, the qualitative observations of the linear network extend to certain nonlinear networks. The authors may wish to specify in which settings these observations may not apply.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": " **Originality**\nThe work is original. The analysis on deep linear networks is novel and the insights from the experimental and theoretical results are not previously explored. To my knowledge, this paper is the first to provide a theoretical explanation for why weight normalization is useful for node perturbation.\n\n\n**Quality**\nThe contributions are high quality. The theoretical analysis appears sound and the experiments are comprehensive. One potential drawback is that the experiments mainly consider simple datasets (SARCOS and MNIST) and mostly consider only one hidden layer networks (although a few experiments consider multilayer nonlinear networks). Investigating the performance of node perturbation in more challenging settings and more complex architectures could help justify the usefulness of the linear network analysis. However, this is not strictly necessary given that many of the main contributions of the paper are theoretical.\n\n\n**Clarity**\nThe paper is well written. The mathematical notation is clear and the figures are well illustrated. However, many of the key and interesting theoretical results of the paper are in the supplementary material (particularly the mean-field dynamics of NP). It may help to provide a brief sketch of these results in the main paper.\n\n\n**Significance**\nThe paper may have significance to researchers specifically studying node perturbation as a learning rule. However, its applicability to the field of biologically plausible learning appears more limited especially given that node perturbation does not appear to be empirically as effective as other biologically-plausible alternative learning rules. Moreover, the experimental results are limited to simple datasets and architectures. The significance of the paper could be significantly enhanced by exploring more complex settings and showing, for example, significantly improved performance of node perturbation when it is combined with weight normalization.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "QXhp0UthPnT", "reviewer": "Reviewer_3nni", "summary": "The present paper analyzes the learning dynamics of Node Perturbation (NP), a biologically plausible learning algorithm used to train feedforward networks. Overall, the paper states a negative result about the unstability of NP due to weights diverging through learning, grounded in analytically tractable results obtained on linear models in the student-teacher setting carefully checked against numerical experiments. This analysis leads the authors to prescribe weight normalization to prevent this phenomenon. The predicted behavior in the linear regime is empirically observed in non-linear models on two training tasks, which validates the soundness of the aforementioned analysis.\n\nMore precisely:\n\n- Section 3 introduces the NP algorithm, recalls that the resulting weight update provides an unbiased estimate of the SGD weight update and shows that the cosine similarity between these two weight updates for a given output layer scales as the inverse of the square-root of the size of this output layer (Eq. 8), suggesting that NP and SGD updates become nearly orthogonal for wide networks. Also, by comparing the covariance matrix of the weight updates for NP and SGD (Eq. 10), they want to highlight that NP is much noisier than SGD.\n\n- Beginning of Section 4 recalls the minimum number of training steps needed to reach an error level for NP in the linear regression setting, and introduces a “deep linear model”, consisting of two linear layers, namely $y=W_2 \\cdot W_1 \\cdot x$. The rationale behind this choice is to analyze separately the impact of the number of perturbed units and that of the size of the output layer. This deep linear model is studied in the student-teacher setting, whereby the target is given by a teacher network consisting of a linear transformation with some additive Gaussian noise (if this noise is non-zero then the authors say there is a “mismatch” between the student and teacher networks).\n\n- First paragraph of Section 4 shows the analytical NP learning dynamics in terms of the error $\\epsilon$ (defined as the squared distance between student and teacher model) and the input weight norm $a=||W_1||$ through time (where time unit is a batch iteration), obtained in the large hidden layer, noiseless (i.e. no mismatch) limit. All the calculations leading to these results are provided in the Appendix. This analysis unveils two main results, summarized by Eq. 14. First, the weight norm grows monotonically increases through time. Second, there are two working regimes for NP, depending on the value of the learning rate used. If the learning rate is smaller than a critical threshold, the error converges to zero, if not the error decreases until the input weight norm a reaches a threshold (corresponding to the sign inversion of $\\dot{\\epsilon}$) wherefrom the error rises again. Hence the “instability”, which is caused by the input weight norm crossing a threshold. This theory is successfully checked against numerical simulations (Fig 1). Also, the minimal number of training steps required to reach an error level is analytically computed (Eq. 15) and numerically verified (Fig. 2), and highlights its weak dependency with the hidden layer size. \n\n- Second paragraph extends the previous study to the case of mismatch (e.g. the labels given by the teacher network are noisy, $\\sigma_t \\neq 0$). It shows that the instability previously always happens, regardless of the learning rate value, the frontier at which it occurs is analytically derived (Eq. 16), and these results are checked numerically (Fig. 3A). There is still a “critical” learning rate threshold delimiting different scalings of the number of updates required to reach minimal error as a function of the learning rate (Fig 3B), a behavior which is in stark contrast with that of SGD (Fig 3C). Fig 4D shows that by normalizing the NP weight update, NP learning is stabilized, at the cost of a bias in the weight update which grows with the hidden layer size.\n\n- In Section 5, the learning dynamics of NP are numerically analyzed on non-linear models on the MNIST and SARCOS training tasks. Again, the instability is observed and vanishes when normalizing the NP weight updates, suggesting that the previous analysis also holds in the non-linear setting. It is also shown that NP is up to 1000 times slower than SGD, which is partially explained by the fact that NP relies on a scalar rather than vector-valued error signal.", "questions": "- L.12: you haven’t yet defined “model mismatch” at this stage, I would remove it. \n- L.24: “However, it remains unclear how robust these approximate learning rules are, or how they perform on large neural networks and complex tasks”. I would downweight this a little bit since you do not go beyond the MNIST task in this paper.\n- L.90-106: I’m not sure the biasedness-noise tradeoff brings much added value here. It would free some space to add some more details about the teacher-student theory in the paper.\n- L.493, Eq. 27b (Appendix): please detail the computation leading to $L_k + 2\\delta_{kl}$. I could check by myself going on the internet to find the moments of a multivariate Gaussian, but I wouldn’t take for granted that any reader knows them by heart.\n- L.506, Eq. 33 (Appendix): why does the tensor product between $\\left(\\sum_l \\xi_k\\xi_l^Tg_l – g_k\\right)x_{k-1}^T$ and $(g_lx_{l-1}^\\top - \\langle g_l x_{l-1}^\\top \\rangle)$ disappear?\n- L. 510, Eq. 36a (Appendix): I am really confused here. After you swap the second and third dimension, it seems to me, from Eq. 31, that $C_{kl}^{sgd}= \\langle g_k g_l^\\top \\otimes x_{k-1}x_{l-1}^\\top\\rangle$, which is the second term of Eq. (36a). However, not only you distinguish $ C_{kl}^{sgd}$ from $\\langle g_k g_l^\\top \\otimes x_{k-1}x_{l-1}^\\top\\rangle$ here, but also a factor 2 appears for no reason in front of $ C_{kl}^{sgd}$. Isn’t there a typo there? My take is that by adding up $ C_{kl}^{sgd}$ and $\\langle g_k g_l^\\top \\otimes x_{k-1}x_{l-1}^\\top\\rangle$ you may obtain $ 2C_{kl}^{sgd}$.\n- L.143: why do you call “deep” your linear model? It’s two layers deep. If you want to refer to the *width* of your linear model (since I acknowledge you study your model in the limit of a large number of hidden units), then use “width” instead of “depth”.\n- L.149, Eq. 11: for completeness, it would have been useful to include the derivation of this result in the Appendix.\n- **General important remark**: please do not abusively refer to equations or Figures in the Appendix, it hinders the fluid reading of the main.\n- L.152: we do not understand here what you mean by “optimal learning rate”, i.e. how $\\eta^*$ is precisely defined.\n- L.549, Eq. 46 (Appendix): since the quantity $E\\cdot x + \\sigma_t z$ appears, it goes to suggest that $E$ is defined as $A – W_2\\cdot W_1$, which is opposite in sign to Eq. 43. \n- L.557 (Appendix): when you write that you “drop the higher-order terms”, I understand that you approximate the mean of a product of two quantities as the product of the means of these two quantities (at least this is how I can recover your result), is this correct? Please be more precise about it.\n- L.561-567: very difficult to understand.\n- While I can understand all the computations up until Eq. 57, from Eq. 58 onwards, the computations are no longer detailed. It’s very hard to follow because when you write $\\approx$, you may perform two approximations at once: 1/ the mean of a product is the product of the means (eliminating “higher order terms”) 2/ the trace of a product is the product of the traces (Eq. 52).\n- L.596, Eq. 71 (Appendix): please detail the computation.\n- L.626, Eq. 82: with $0 < a_{init}<2/c_0$, you write $-a_{init}^3 < -(2/c_0)^3$, which is wrong ($x\\to x^3$ increases on $\\mathbb{R}^+$), so I am skeptical about the $\\eta^*$ you obtain in Eq. 83.\n- B.4 (appendix): these computations are not sufficiently detailed.\n- L. 197: which saturation are you talking about? I can’t see the orange curve saturating on the Fig you refer to (Fig 2A).\n- L.199-209: it’s very handwavy, and I do not understand the added value of this paragraph.\n- Figure 5: this figure displays the accuracy while the others show the error. It would be easier to compare results across different figures if the same convention (all error curves, or all accuracy curves) is taken. ", "limitations": "My main concern is that this work,  titled \"stability and **scalability** of NP learning\", does not go beyond the MNIST task.\n\nI think the most important contribution of this work is theoretical: there is an excellent match between theory and experiments as per the Figures shown in the main on linear models, which extend to non-linear models and gives a very simple insight as to how to use NP properly -- that is: by normalizing the NP weight updates. I think this simple result would have been even more compelling if for instance demonstrated on a 4-5 layers convolutional architecture trained by NP on CIFAR-10, the ideal situation being: without weight update normalization the model doesn't train, with weight update normalization it trains (even very slowly, but it does). Then, it could be concluded that the theoretical model proposed totally accounts for the unscalability of NP when normalization is not applied. As it stands, with MNIST being the most difficult task tested, it's harder to conclude the same.\n\nIn the bioplausible deep learning literature, the MNIST task alone as a benchmark to claim scalability is unsufficient to my eyes. However, it might be that the sole theoretical contribution of this work and MNIST benchmark abide by the standards of another community (e.g. statistical physics x ML) I may not be aware of, so my judgement might be biased by the fact I come from the bioplausible deep learning literature. Also, I think that the theoretical contributions should be better highlighted in the main. \n\nI would recommend accept if:\n\n1 - The authors ran a CIFAR-10 experiment with a 5 layers-deep convolutional architecture, observed the same kind of behavior as on MNIST and SARCOS.\n\n2 - Computations in the appendix are more detailed than they are now -- see the questions section above. ", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The structure and the writing of the paper are clear. The figures are neat. \n- The mean-field model derived in the linear student-teacher setting matches very well the experiments.\n- The theoretical analysis seems to lead to a simple trick to unlock NP scalability.\n\nWeaknesses:\n- The derivations in the Appendix are not easy to follow, especially because multiple approximations are sometimes used without detailed steps (I will come back to this in the questions).\n- The title raises big expectations that are not met: the term “scalability” calls for complex tasks beyond MNIST. This is a major limitation of the paper.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "Ca04Ke8kJ2w", "reviewer": "Reviewer_up3q", "summary": "The authors present a neat analysis of the scalability and stability of the node perturbation algorithm, which is one of the popular bio-plausible credit assignment algorithms, for deep networks. Based on their analysis and inferences, they introduce a weight normalization trick that seems to alleviate the issues with the algorithm, albeit at the cost of adding a bias to the gradient estimates. In the first part of the paper, the authors use analytical tools to dissociate the effect of number of output nodes of the network from the number of perturbed nodes of the network to claim that node perturbation is scalable for deep linear networks. However, they demonstrate that the dynamics entails an instability in the weight norm. They validate these analytical results empirically in deep nonlinear networks and therefore establish a key result in training networks using node perturbation. They also show that the instablity is worse when there is noise in the labels (i.e. teacher noise). Finally, they demonstrate that the weight normalization trick can help stabilize the algorithm, albeit hurt performance by introducing a bias in the algorithm -- as evidenced by lower performance on the SARCOS and MNIST tasks.", "questions": "1. What is the dimensionality of $g_l$? I initially assumed that you used the numerator notation while writing vector calculus and therefore $g_l \\in \\mathcal{R}^{1 \\times L_l}$, where $L_l$ is the dimensionality of $h_l$ or the number of neurons in $l^{th}$ layer. But from my understanding after reading the appendix, you assumed $g_l$ to be a column vector and therefore derived: $\\frac{1}{\\sigma}l(\\tilde{x_K},x_0) - l(x_K,x_0) = \\frac{1}{\\sigma} \\frac{\\partial l}{\\partial x_K} (\\tilde{x_K} - x_K) = \\sum_l \\frac{\\partial l}{\\partial x_K} \\frac{\\partial x_K}{\\partial h_l} \\xi_l = \\sum_l g_l^T \\xi_l$. Is that correct? Perhaps add the dimensionality of the quantities in the appendix for clarity?\n2. What does it mean to have lower linear dimensionality in the representations? Does the lower dimensionality underlie poor performance? This link was not clear from your description.\n3. Re the weigh normalization scheme: Is it applied such that the weight normalization is the same at initialization? Can it be applied intermittently or as a regularizer? You mentioned why weight decay is not a solution to the instability problem but it wasn't clear if explicit weight normalization where the weights are allowed to grow from the initialization case could be a potential solution to the instability problem.", "limitations": "I think the authors do a commendable job in stating the limitations of their proposal but I feel it could be elaborated in the discussions. Specifically, how does the weight normalization introduce bias and its effect on the performance. ", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. The paper is well motivated and tackles an important problem in the field. The analytical methods used in the paper are an important contribution towards understanding how node perturbation algorithms can be used to develop bio-plausible learning rules in deep networks.\n2. The paper is theoretically strong and demonstrates via simulations how the analytical results hold in practice, as well as when nonlinearity is introduced in the network.\n3. The weight normalization solution is a smart solution and a good integration of the inferences from the analytical framework. Furthermore, it also fits well with the homeostasis viewpoint wherein weights of a neuron are thought to conserve some quantity over time.\n\nWeaknesses:\n1. The presentation of the paper can be improved, particularly how the analytical results are presented. I felt a lot of the details of how the results were derived were buried in the appendix with limited reference to these considerations/assumptions in the main text. The authors could consider rewriting certain sections of the paper to better reflect the derivations of their theorems and in doing so, allow the reader to appreciate the contribution that this paper makes. \n2. The weight normalization strategy could be better introduced, particularly how in adding a weight normalization step introduces bias in the weight update. It would be great if the authors could add elaborate on this bias and how a bigger hidden layer size contributes to a higher bias. \n3. The metrics used in the paper make sense while analyzing the instablity of the node perturbation algorithm. However, it would be nice to report the accuracy/performance metrics on SARCOS and MNIST for SGD and weight-normalized NP. I feel this would be more complete and lay the foundation for further researchers (who could use weight-normalized NP as a baseline method).\n4. The discussion about representation similarity is a bit sudden and lack compact presentation. Given that the merits of the paper lie in the stability analysis of the NP algorithm and proposing a possible workaround to that, I would suggest either cutting down on analyzing learned representations (although it is very important and an interesting direction) or fleshing it out more to convey the inferences better.\n\nOverall, I feel that the paper presents a very thorough analysis of the NP algorithm and demonstrates a key feature in the dynamics. If the authors could improve the writing a bit, this work would appeal to a larger audience and could make a significant impact in the field.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept