Tensor decompositions of higher-order correlations by nonlinear Hebbian plasticity
Authors: Gabriel Koch Ocker
Michael A Buice
Venue: NeurIPS 2021 Poster (2021)
Keywords: Hebbian learning
tensor decompositions
nonlinear synaptic plasticity

Abstract: Biological synaptic plasticity exhibits nonlinearities that are not accounted for by classic Hebbian learning rules. Here, we introduce a simple family of generalized nonlinear Hebbian learning rules. We study the computations implemented by their dynamics in the simple setting of a neuron receiving feedforward inputs. These nonlinear Hebbian rules allow a neuron to learn tensor decompositions of its higher- order input correlations. The particular input correlation decomposed and the form of the decomposition depend on the location of nonlinearities in the plasticity rule. For simple, biologically motivated parameters, the neuron learns eigenvectors of higher-order input correlation tensors. We prove that tensor eigenvectors are attractors and determine their basins of attraction. We calculate the volume of those basins, showing that the dominant eigenvector has the largest basin of attraction. We then study arbitrary learning rules and find that any learning rule that admits a finite Taylor expansion into the neural input and output also has stable equilibria at generalized eigenvectors of higher-order input correlation tensors. Nonlinearities in synaptic plasticity thus allow a neuron to encode higher-order input correlations in a simple fashion.

Tensor decomposition of higher-order correlations by nonlinear Hebbian plasticity
Gabriel Koch Ocker
Michael A. Buice MindScope Program Allen Institute Seattle, WA 98109
Department of Mathematics and Statistics
Boston University
Boston, MA 02136
gkocker@bu.c edu
michaellufalleennnstttee....rr
Abstract
Biological synaptic plasticity exhibits nonlinearities that not accounted for by classic Hebbian learning rules, Here, we introduce simple family of generalized nonlinear Hebbian learning rules. We study the computations implemented by their dynamics in the simple setting of neuron receiving feedforward inputs. These nonlinear Hebbian rules allow neuron learn tensor decompositions of its higher- order input correlations, The particular input correlation decomposed and the form of the decomposition depend on the location of nonlinearities in the plasticity rule. For simple, biologically motivated parameters, the neuron learns eigenvectors of higher-order input correlation tensors. We prove that tensor eigenvectors are attractors and determine their basins of attraction. We calculate the volume of those basins, showing that the dominant eigenvector has the largest basin of attraction. We then study arbitrary learning rules and find that any learning rule that admits a finite Taylor expansion into the neural input and output also has stable equilibria at generalized eigenvectors of higher-ou order input correlation tensors. Nonlinearities in synaptic plasticity thus allow a neuron to encode higher-order input correlations in a simple fashion.
Introduction
In Hebbian learning, potentiation of the net synaptic weight between two neurons is driven by the correlation between pre- and postsynaptic activity [[] That postulate is cornerstone of the theory of synaptic plasticity and learning 22] R] In its basic form, the Hebbian model leads to runaway potentiation or depression of synapses, since the pre-post correlation increases with increasing synaptic weight [ That runaway potentiation can be stabilized by supplemental homeostatic plasticity dynamics (5) by weight dependence in the learning rule 6 7 or by synaptic scaling regulating a neuron's 's total synaptic weight [ T. In 1982, Erkki Oja observed that linear neuron with Hebbian plasticity and synaptic scaling learns the first principal component of its inputs 100 In 1985, Oja and Karhunen proved that this is global attractor of the Hebbian dynamics to fountain of research on unsupervised feature learning in neural networks 12 1
This led
Principal component analysis (PCA) describes second-order features of random variable. Both naturalistic stimuli and neural activity can, however, exhibit higher-order correlations [44 [5]
Canonical models of retinal and thalamic processing whiten inputs, removing pairwise features 11 Beyond-pairwise features, encoded in tensors, can provide a powerful substrate for learning from data 23 27
The basic Hebbian postulate does not take into account fundamental nonlinear aspects biological synaptic plasticity in cortical pyramidal neurons. First, synaptic plasticity depends on seyond-pairwisg activity correlations
X 33
Second, spatially clustered and temporally coactive synapses exhibit
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

correlated and cooperative plasticity 1411 There is a rich literature on computationally motivated forms of nonlinear Hebbian learning (section B.2). Here, we will prove that these biologically motivated nonlinearities allow a neuron to learn higher-order features of its inputs.
We study the dynamics of simple family of generalized Hebbian learning rules, combined with synaptic scaling (eq. D. Equilibria of these learning rules are invariants of higher-order input correlation tensors. The order of input correlation (pair, triplet, etc.) depends on the pre- and postsynaptic nonlinearities of the learning rule. When the only nonlinearity in the plasticity rule is postsynaptic, the steady states are eigenvectors of higher-order input correlation tensors [ 1 We prove that these eigenvectors are attractors of the generalized Hebbian plasticity dynamics and characterize their basins of attraction.
Then, we study further generalizations of these learning rules. We show that any plasticity model (with finite Taylor expansion in the synaptic input, neural output, and synaptic weight) has steady states that generalize those tensor decompositions to multiple input correlations, including generalized tensor eigenvectors. We show that these generalized tensor eigenvectors are stable equilibria of the learning dynamics. Due to the complexity of the arbitrary learning rules, we are unable to fully determine their basins of attraction. We do find that these generalized tensor eigenvectors are in an attracting set for the dynamics, and characterize its basin of attraction. Finally, we conclude by discussing extensions of these results spiking models and weight-dependent plasticity.
2 Results
Take a neuron receiving K inputs (x))). i E [K], each filtered through a connection with synaptic weight (t) to produce activity n(t). We consider synaptic plasticity where the evolution of Jr can depend nonlinearly on the postsynaptic activity n(t), the local input (t), and the current synaptic weight (t). We model these dependencies in learning rule f:
J,(1) + 4/- fi((t),,,, x, (1), [t + dt) (((()) + (n()) x(I),
where f:(t) n" (t) (E()).
(1)
The parameter a sets the output-dependent nonlinearity of the learning rule, b sets the input-
nonlinearity, and sets its dependence on the current synaptic weight. Eq. D assumes simple form for these nonlincarities; we discuss arbitrary nonlinear learning rules in section 2.3 We assume that and b are positive integers, as in higher-order voltage or spike timing-dependent plasticity (STDP) models [4444] The scaling by the norm of the synaptic weight vector, || models homeostatic synaptic scaling S [C. Bold type indicates vector, matrix, or tensor (depending on the variable) and regular font with lower indices indicates elements thereof. Roman type denotes random variable (x).
We assume that x(t) is drawn from a stationary distribution with finite moments of order a + b. Combined with a linear neuron, following dynamics for J (appendix A.
J x, and a slow learning rate, t > dt, this implies the
JJ. J Σ (j J: JJ Uj.a (J (JSa)...
j.a
In eq. 2. J. dJs/dt, a = (f1...., Ja) is a
and @ is the vector outer product; T is the a-fold outer product of the synaptic weight vector J. μe a higher-order moment (correlation) tensor of the inputs:
(3) where ()x denotes the expectation with respect to the distribution of the inputs. f is an (a + 1) oordee tensor containing an (a + b)-order joint moment of x. The order of the tensor refers to its number of
Pi.a = (xf )a)x
indices, vector is first-order tensor and matrix second-order tensor. fz is cubical; each mode of μ has the same dimension K. In the first term of eq. 2 for example, Σa Mi,a (i,aaaa dot product of J along modes 2 through a + of μ..
(J80), )a))a takes the
2.1 Steady states of nonlinear Hebbian learning
If we take = b = 1, then μ is the second-order correlation of x and a is just the index j. With 0 also, eq. 2 reduces to Oja's 's rule and J is guaranteed to converge to the dominant eigenvector

---

of J 10 M..
We next investigate the steady states of eq. for arbitrary (a, Z3, CE R. J, =0
a trivial steady state. At steady states of eq. D where J 7 0,
Σ Emia(Jon) = A where {(H = E jaa
1 (jaa(J
(4)
so that J is invariant under the multilinear map of la except for a scaling by À and element-wise exponentiation by C. For two parameter families (a, b, c). eq. reduces different types of tensor We next briefly describe these and some of their properties.
eigenequation 4
4
XJ Qi called J the tensor eigenpair 2 and Lim called them the (o-morm eigenpair + There are Ka such eigenpairs
First, af we have the tensor eigenequation En Mi,a (J
42] If p 2 0 clement-wise, then it has a unique largest eigenvalue with a real, non-negative eigenvector J. analogous the Perron-Frobenic theorem for matrices 44] 4
If µt is weakly irreducible, that eigenvector is strictly positive [9]. In contrast to matrix eigenvectors, however, for these tensor eigenvectors are not necessarily invariant under orthogonal transformations
12
c 0, we have another variant of tensor eigenvalue/vector equation:
Σ Pi,a (JSa) =J=
(5)
Qi called these X, J an E-eigenpair 42 and Lim called them the -igenppair 43 In general, tensor may have infinitely many such eigenpairs. If the spectrum of K-dimensional tensor of order a + is finite, however, there are (a -1)((a-)) eigenvalues counted with multiplicity, and the spectrum of a symmetric tensor is finite [50 5111 (If b = 1, is symmetric.) Unlike the steady states when + c 1, these eigenpairs are invariant under orthogonal transformations 44.. For non-negative μe, there exists a positive eigenpair [52] It may not be unique, however, unlike the largest eigenpair for a + 1 (an
Frobon-Frobenius result) [5]] In the remainder of this paper we will usually focus on parameter sets with = O and use "tensor eigenvector" to refer to those of eq.
2.2 Dynamics of nonlinear Hebbian learning
For the linear Hebbian rule, (a, c) (1, 1, 0). Oja and Karhunen proved that the first principal component of the inputs a global attractor of eq. I.I We thus asked whether the first tensor eigenvector is global attractor of eq. 2 when c = but (a, b) o (1,1). We first simulated the nonlinear Hebbian dynamics. For the inputs x. we whitened 35 x5 pixel image patches sampled from the Berkeley segmentation dataset (fig. Ia 53 ). For b of 1, the correlation of these image patches was not symmetric (fig- [D). The mean squared error of the canonical polyadic (CP) approximation of these tensors was higher for b 1 than b (fig. Id). It decreased slowly past rank 10, and the rank of the input correlation tensors was at least 30 (fig. D
The nonlinear Hebbian learning dynamics converged to an equilibrium from random initial conditions (e.g., fig. D. f), around which the weights fluctuated due to the finite learning timescale т. Any equilibrium is guaranteed to be some eigenvector of the input correlation tensor μu (section 2.T. For individual realizations of the weight dynamics, we computed the overlap between the final synaptic weight vector and each of the first 10 eigenvectors (components of the Tucker decomposition) of the corresponding input correlation fs 47 54 The dynamics most frequently converged to the first eigenvector. For non-negligible fraction of initial conditions, however, the nonlinear Hebbian rule converged to subdominant cigenvectors (fig. [g.h). The input correlations μ did have a unique dominant cigenvector (fig. 3h, bluc), but the dynamics of eq, did not always converge to it. This finding stands in contrast to the standard Hebbian rule, which must converge to the first eigenvector it is unique ΑТ
While the top eigenvector of matrix can be computed efficiently, computing the top eigenvector of tensor is, in general, NP-hard [55 To understand the learning dynamics further, we examined them analytically. Our main finding is that with (b, c) = (1, 0) in the generalized Hebbian rule, eigenvectors of ju are attractors of eq. 2 Contrary to the case when
(Oja's 's rule), the dynamics are thus not guaranteed to converge to the first eigenvector of the input correlation tensor when > 1. The first cigenvector of μ does, however, have the largest basin of attraction.
3

---

x102
(a,b)=
Synaptic weight
5
(2,1)
Input
Overlap
-5
Pixel P
Input
10000
25000
Input
Time
Time
Pixel
Final weights closest factor: 0
Mean sq. error
0.6
(1,b)=(1,2)
Frequency
0
(2,))=(2,1)
0.4
Pixel
Input
(a,b)=(2,2)
0.2
0.0
p2
Input
p2
20 Rank
Pixel
0 Eigenvector
10
Figure I: Convergence of higher-order Hebbian plasticity tensor eigenvectors of natural image correlations. a) Example /' x /' image patches (P = 35 pixels). b) Third-order input correlation μa that drives plasticity for 1, 2. e) Third-order input correlation μu that drives plasticity for 2, 1. d) Mean squared error of the CP approximation of the input correlations. (Curves: mean across 4 initializations for the alternating least-squares computation of the CP, with standard deviation within the line thickness.) e) Learning dynamics on natural inputs: five randomly selected synaptic weights with (a, b, c) (2, 1). f) Example final synaptic weight configuration with (a, b, c) = (2, 1, 0). Overlap of the synaptic weights with their final closest singular vector, Uf J. Solid line: mean over 10 samples natural image patches and realizations of the weight dynamics for each input patch. Shaded region: standard deviation. Initial conditions for J are chosen uniformly on the unit sphere. h) Histogram the closest singular vector to the final synaptic weights for (a, b, = 1,).
Theorem 1. In eq. 2 take (b,c) (1, Let be cubical, symmetric tensor of order + 1 and odeco with R components:
R X (U,)
S0+1
(6)
where U is matrix of unit-norm orthogonal eigenvectors: U I. Let X₂ > for each E and da F Xj V (i,5) [R] x R with i Then for each [R]:
[R]
1. With any odd 1, +Uk are attracting fixed points of eq. and their basin of attrac- Within that region, the separatrix of
tion is Niepaa { J:
| J/UT] < (Ax/2.) Via
+UB and -Uk is the hyperplane orthogonal to Ug: {J: J
2. With any even positive J =U is attracting fixed point of eq. attraction is {J: U. J > 0} Nietray {J: UF < (Ne/x.) 1/e-11
and its basin of
3. With any even positive a, J 0 is neutrally stable fixed point of eq. 2 with basin of attraction { R EU= J)² < < < <0 k E [R]
Remark. Each component of the orthogonal decomposition of an odeco tensor μ (eq. 5 is an eigenvector of μ. R < (d--1)/(--1), there are additional eigenvectors. The components of the orthogonal decomposition are the robust eigenvectors of μ.: the attractors of its multilinear map 126 561. The non-robust eigenvectors of an odeco tensor are fixed by its robust eigenvectors and their eigenvalues (57).

---

The proof of theorem is given in appendix A.2 To prove theorem we project J onto the U J. This leads to the discovery
eigenvectors of μ., and study the dynamics of the loadings of collection of unstable manifolds: each pair of axes (i, k) has an associated unstable hyperplane 2/ = by (>*/A,) Vi--)) (and if a odd, also the corresponding hyperplane with negative slope). These partition the phase space into the basins of attraction of the eigenvectors of μ.
For example, consider fourth-order input correlation (corresponding eigenvectors. The phase portrait of the loadings is in fig. 2h. with the nullelines in black and attracting and unstable manifolds in blue. The attracting manifold is the unit sphere. There are unstable hyperplanes that partition the phase space into the basins of attraction of (0, 1 and (=1,0) 0), where the synaptic weights J are an eigenvector of μ..
in eq. D with two
For even a, only the unstable hyperplanes with positive slope survive (fig. 2p. blue line). The unit sphere (fig. D blue) is attracting from any region where at least one loading positive. Its vertices [i] x [1] are equilibria; (v1, U22) = (1, 0) and (0, 1) are attractors and the unstable hyperplane separates their basins of attraction. For the region with all loadings negative that is the basin of attraction of the origin, noise will drive the system away from zero towards one of the eigenvector equilibria.
a
a=3
a=2
a=
V1
V/2
V/2
Figure 2: Example dynamics of the loadings, U J, for rank-two input correlations. Black curves: nullclines of vn, 12- Blue curves: stable sets and separatrices of the basins of attraction of the sparse attractors for a) Odd (a 3). The stable set is the unit sphere, and the separatrices
are 1/2 = 1u (A1/x2) Vo-1 b) Even a (a = 2). The unstable set is U/ = 81 (A1/>a) V--1 (solid blue line). In (a,b), (A1 A2) (3, 1). e) Phase portrait for two-term learning rule (eq. 9 All parameters of the input correlation tensors (Amr: Ad) are equal to one. Dashed blue curve: = {v: Em. 2 0}.
By partitioning the phase space J into basins of attraction for eigenvectors of μ., theorem I also allows determine the volumes of those basins of attraction. The basins of attraction are open sections of R&K so we measure their volume relative to that of a large hypercube.
Corollary 1.1. Let Vy the relative volume of the basin of attraction for J = Up. For odd a > 1,
R
1/4
V = R II
Au
A
Corollary 1.2. Let V& be the relative volume of the basin of attraction for = Uk- For even positive
1(a-1)
Aa
1/a-1)
V& -22-
-R--1)
(R
A
(8)
A
1/(-11
+R 2)
Σ
J.14k
The calculations for corollaries are given in appendix basins of attraction depend on the spectrum of μ. its rank R, and its order The result for odd also
see that the volumes of the
5

---

provides lower bound on the volume for even The result is simpler for odd a so we focus our discussion here on that case.
While eigenvectors of la with small eigenvalues contribute little to values of the input correlation 1e, they can have large impact the basins of attraction. The volume of the basin of attraction of eigenvector is proportional to A R/(a-1) An eigenvector with eigenvalue e scales the basins of attraction for the other cigenvectors by E -'//-11 The relative volume of two eigenvectors basins of attraction is, however, unaffected by the other eigenvalues whatever their amplitude. With odd, the ratio of the volumes of the basins attraction of cigenvectors l and j is Vi/v, = (>+/2,) R/--1.
We see in theorem I that attractors of eq. are points on the unit hypersphere, S. For odd a, S is an attracting set for eq. 2 For even a, the section of with least one positive coordinate is an attracting set (fig. 2 b, blue circle; see proof of theorem [ in appendix A We thus next computed the surface area of the section of in the basin of attraction for eigenvector k. Ag (corollary [.3 in appendix A.2). The result requires knowledge of all non-negligible eigenvalues of μa, and the ratio Ax/A, does not exhibit the cancellation that V does for odd a. We saw in simulations with natural image patch inputs and initial conditions for J chosen uniformly at random on S, the basin of attraction for U was ~ 3x larger than that for U and the higher eigenvectors had negligible basins of attraction (fig. mT
In Oja's model, (a. b, c) = 1,0) eq. 1 the largest eigenvalue has multiplicity d >> then the d-sphere spanned by those codominant eigenvectors is a globally attracting equilibrium manifold for the synaptic weights. The corresponding result for a > (for the formal statement and proof, see corollary
L.4 in appendix A.2):
Corollary 1.3, (Informal) If any d robust eigenvalues of μ are equal, the d-sphere spanned by their robust eigenvectors is an attracting equilibrium manifold and its basin of astraction is defined by each of those d eigenvectors' basins of attraction boundaries with the other R d robust eigenvectors.
2.3 Arbitrary learning rules
So far we have studied phenomenologicaa plasticity rules in the particular form of eq. [ The neural output n, imput x, and synaptic weight J4 were each raised power and then multiplied together. Changes in the strength of actual synapses are governed by complex biochemical, transcriptional, and regulatory pathways (58). We view these as specifying some unknown function of the neural output, input, and synaptic weight, f(n, x, J). That function might not have the form of eq. [ So, we next investigate the dynamics induced by arbitrary learning rules f. We see here that under a mild condition, any equilibrium of the plasticity dynamics will have similar form as the steady states of eq. 2 If f does not depend on except through n, equilibria will be generalized eigenvectors of higher-order input correlations.
The Taylor expansion of f around zero is:
06
fo (n(t). (t). = Σ Am n Rm (t) x, (t) J.m (t)
(9)
where the coefficients Am are partial derivates of f. We assume that there exists a finite integer N such that derivatives of order N + and higher are negligible compared the lower-order derivatives. We then approximate f. truncating its expansion after those N terms. With a linear neuron, synaptic scaling, and slow learning, this implies the plasticity dynamics
TJ. Σ
m miHiiam (J00m))
Jr EΣ
Jgm
m.tljam (J00 aa
(10)
J. Ckn
where m m/liaam
Am (x Dm (x (0 )amix. x At steady states where J; + 0.
am (Joem) )am = Adis where A(J, (m+))
Σ Σ Jem mijja (J (Sa))a. )ar
192
m
3.0m
(11)
If each Cm. = 0, this is kind of generalized tensor eigenequation:
Σ mfii (J@a) )a AJ,
(12)

---

so that J is invariant under the combined action of the multilinear maps mfd (which are potentially of different orders). If a1 02
am, then this can be simplified equation by summing the input correlations mns If different terms of the expansion of generate different-order input correlations, however, the steady states are no longer necessarily equivalent to tensor eigenvectors. If there exists synaptic weight vector J that is an eigenvector of each of those input correlation tensors, En mfHi.om steady state of the plasticity dynamics with each Cm 0.
tensor eigenvector
(J (J@am Jm XJ; for each m, then that configuration J is a
We next investigated whether these steady states were attractors in simulations of learning rule with a contribution from two-point and three-point correlations (a = (1, 2). b = 1. 0. A (1, 1/2) eq. 99. As before, we used whitened natural image patches for the inputs x (fig. TT] The two- and laree-point correlations of those image patches had similar first eigenvalues, but the spectrum of the two-point correlation decreased more quickly than the three-point correlation (fig. Bb. The first three eigenvectors of the different correlations overlapped strongly (fig. PD. This was not due to trivial constant offset since the inputs were whitened. With this parameter set, the synaptic weights usually converged to the (shared) first eigenvector of the input correlations (fig. Dedd e (blue)).
f
x10
Eigenva lue
3-point 2-point
Synaptic weight
5
A=
015
2- 2-point
e'vec 1
4
A (1, 0.5) A= (0.5,1)
2-point
2
e e'vec
Final alignment
Frequency
10
2-point
e'vec
0
Eigenvector
d
Alignment
3-point
1
0.8
2-point e'vec
Alignment
e'vec
3-point
5
3-point
e e'vec 1
e'vec 2
Time 15000
Final alignment to e'vec 1
3-point e'vec
2-point weight, A
Figure 3: Multi-term Hebbian plasticity rule converges to input correlation tensors' shared eigenvec-
tors. We study a learning rule with two terms, driven by the second- and third-order input correlations (a = (a=1),2), b b= 1, c = 0). a) Tucker spectra of the natural image patches' correlation tensors. Solid lines: mean over 10 samples of 200 image patches. Shaded regions: standard deviation. b) Overlap of the eigenvectors
the image patches' two- and three-point correlations (mean over 10 samples of 200 image patches). c, d) Example dynamics for the two-term learning rule. Transparency increases with the eigenvector number in d. e) Distribution of the final alignment to the first eigenvectors over 10 random initial conditions for each of 10 samples of image patches. f) Final alignment to the two- and three-point correlations' eigenvectors as a function of the weight on the two-point correlation, A A2 1.
We next asked how the weights of the different input correlations in the learning rule (the parameters Ag) affected the plasticity dynamics. When the learning rule weighted the inputs' three-point correlation more strongly than the two-point correlation (A = (2/2, 1)), the dynamics converged almost always to the first eigenvector of the three-point correlation (fig. 3.. blue vs orange). Without loss of generality, we then fixed A₂ 1 and varied the amplitude of A₁. As A₁ increased. the learning dynamics converged to equilibria increasingly aligned with the top eigenvectors of the input correlations (fig. B). For sufficiently negative A1, the dynamics converged to steady states that were neither eigenvectors of the two-point input correlation nor any of the top 20 eigenvectors of the three-point input correlation (fig. 3
Farlier, we saw that in single-term learning rules, the only attractors were robust eigenvectors of the input correlation (thcorem I. The dynamics of eq. 2 usually converged to the first eigenvector
7

---

because it had the largest basin of attraction (corollaries I.1 Here, we saw that at least for some
parameter sets, the dynamies of multi-term generalized Hebbian rule may not converge to an imput eigenvector. This suggested the existence of other attractors for the dynamics of eq. TO
We next investigated the steady states of multi-term nonlinear Hebbian rules analytically. We focused on the case when the different input correlations generated by the learning rule all have a shared set of cigenvectors. In this case, those shared eigenvectors are all stable equilibria of eq. T0 They are not, however, the only stable equilibria. We see in a simple example that there can be equilibria that are linear combinations of those eigenvectors with all negative weights (in fig. _ the fixed point in the lower left quadrant on the unit circle). In fact, any stable equilibrium that is not a shared eigenvector must be such a negative combination. Our results are summarized in the following theorem:
Theorem 2. In eq.
ZN and consider N cubical, symmetric tensors, m/,, each of order Um for m E [N], that are mutually odeco into R components:
10 lake b 1,
mf U, U, ... UU,
(13)
with UT I. Let Amr 2 and Em Amr ffoor each m.r [M] x [R) Let
N
R.n
S(J) EVT and L(J) i=1
Ami (U, J)amt!
(14)
mrm
==
Then:
1. S [J= (() > 0} an attracting set for eq. includes (J LJJ) 0}.
and its basin of attraction
2. For each k E [R] J Us is a stable equilibrium of eq. 10
3. For each & E [R], J = -U is stable equilibrium of eq. 10 if Em m=(-1)aan < 0 (and
unstable if 3 < 0).
4. Any other stable equilibrium must have U. J S 0 each k E [R].
The claims of theorem 2 are proven in appendix A.2 Similar to theorem I we see that the robust eigenvectors of each input correlation generated by the learning rule are stable equilibria of the learning dynamics. The complexity of eq. TO has kept us from determining their basins of attraction. We can, however, make several guarantees. First, in large region, the unit sphere is attracting set for the dynamics of eq. [TO] Second, the only stable fixed points are either the eigenvectors or combinations of the eigenvectors of la with only nonpositive weights. This is in contrast the situation where the learning rule has only one term; then theorem guaranttess that the only attractors eigenvectors.
+u
3 Discussion
We have analyzed biologically motivated plasticity dynamics that generalize the Oja rule. One class of these compute tensor cigenvectors. We proved that without a multiplicative weight-dependence in the plasticity, those cigenvectors are attractors of the dynamics (theorem D figs. 2h. b). Contrary to Oja's rule, the first cigenvector of higher-order input correlations not a unique attractor. Rather,
each eigenvector k has finite basin of attraction, the size of which is proportional to X R/--1 If there are codominant eigenvectors (Aı = A/2 equilibrium manifold (corollary 1.4 in appendix A.2 Furthermore, steady states of any plasticity model with a finite Taylor polynomial in the neural output and inputs are generalized eigenvectors of
Xx),, the d-sphere they span is an attracting
multiple input correlations. These steady states are stable and attracting (theorem) figs. 2 3)
3.1 Spiking neurons and weight-dependencce
While biological synaptic plasticity is certainly more complex than the simple generalized Hebbian rule of eq. neural activity is also more complex than the linear model n - x. We examined the
8

---

simple aneear-nonlinea-ae (STDP) rule (45); appendix A.3)). Similar eqs. 2 and I0 we can write the dynamical equation for J as function of joint cumulant tensors of the input (eq. 56 in appendix A.3) 'hese dynamics have a different structure than eqs. and TO
spiking model and a generalized spike timing- dependent plasticity
We focused here mainly on learning rules with no direct dependence on the synaptic weight (c = 0 in eq. 1 c = 0 in eq. When e + the learning dynamics cannot be simply analyzed in terms of the loading onto the input correlations' eigenvectors. We studied the learning dynamics with weight-dependence for two simple families of input correlations: diagonal μ and seecewise-constenn rank one μ (appendix A.4. In both cases, we found that with eigenvectors of those simple input correlations were also attractors of the plasticity rule. With diagonal input correlations, sparse steady states with one nonzero synapse are always stable and attracting when a + 0, but if + c S 0 synaptic weights converge to solutions where all weights have the same magnitude (fig. [A.4.T). With rank one input correlations, multiplicative weight-dependence can interfere with synaptic scaling and lead to an instability for the neurons' total synaptic amplitude (fig. A.4.2)
3.2 Related work and applications
There is rich literature on generalized nonlinear forms of Hebbian learning. We briefly discuss the most closely related results, our knowledge. The family of Bienenstock, Cooper, and Munro (BCM) learning rules supplement the classic Hebbian model with a stabilizing sliding threshold for potentiation rather than synaptic scaling (59] BCM rules balance terms driven by third- and fourth-order joint moments of the pre- and postsynaptic activity G0. A triplet STDP model with rate-dependent depression and uncorrelated Poisson spiking has BCM dynamics
[45] and can develop
selective (sparse) connectivity in response to rate- or correlation-based input patterns [61] If the input is drawn from a mixture model then under BCM rule, the synaptic weights are guaranteed to converge to the class means of the mixture 62
Learning rules with suitable postsynaptic nonlinearities can allow a neuron to perform independent component analysis (ICA) 63 644 These learning rules optimize the kurtosis of the neural response. In contrast, we show that simple nonlinear Hebbian model learns tensor eigenvectors of higher-order input correlations. Those higher-order input correlations can determine which features are learned by gradient- based ICA algorithms (65) Taylor and Coombes showed that a generalization of the Oja rule to higher-order neurons can also learn higher-order correlations [66], which can allow learning independent components [67]. In that model, however, the synaptic weights J are higher-order tensor.
Computing the robust eigenvectors of an odeco tensor μa by power iteration has O(K*t1) space complexity: it requires first computing μ.. The discrete-time dynamics of eq. [ correspond to streaming power iteration, with ((K) space complexity (68 70). Eq. defines limiting continuous- time dynamics for tensor power iteration, exposing the basins of attraction.
Local learning rules for approximating higher-order correlation tensors may also prove useful, for example in neuromorphic devices YA 73
Oja's rule inspired generation of neural algorithms for PCA and subspace learning
12
13
Code availability
The code associated with figures one and three is available at TensorHebb
Acknowledgments
We thank the Allen Institute founder, Paul G. Allen, for his vision, encouragement and support.

---

References
[1] Hebb DO. The organization of behavior: a neuropsychological theory. Mahwah, N.J.:
Erlbaum Associates; 1949.
[2] Engel A, Van den Broeck C. Statistical Mechanics of Learning. Cambridge: Cambridge
University Press; 2001. Available from: htttps://ww..cmbrrdgg...oo cambridge htwp://dw...r..//reeeeeeee htwpbriiddd...rg/rr///oooooooo
[3] Caporale N. Dan Y. Spike Timing- Plasticity; A Hcbbian Learning Rule, Annual
Review of Neuroscience. 200::3((1):25466 doi:10. 1 http://urrr.....00011.11000.
[4] Turrigiano Gina G The dialectic of Hebb and homeostasis. Philosophical Transactions of the Royal Society B: Biological Sciences. 2017:372(1715)22216623. doi: wew.:/999//tt.........
[5] Zenke F, Gerstner W. Hebbian plasticity requires compensatory processes on multiple timescales. Phil Trans Soc 2017:322(775):221162299
[6] Van Rossum MC, Bi GQ. Turrigiano GG. Stable Hebbian learning from spike sming-dependent plasticity. The Journal Neuroscience. 2000:20023::812-8887
[7] Rubin J, Lee D, Sompolinsky H. Equilibrium Properties of Temporally Asymmetric Hebbian Plasticity. Physical Review Letters. 2001:86(2):364-367, doi:10. htt..011000/hyyseeeeet......66
[8] Bourne JN, Harris KM. Coordination of size and number of excitatory and inhibitory synapses results in balanced structural plasticity along mature hippocampal CA1 dendrites during LTP. Hippocampus. 201 1:21 (4):35433333 doi.htpss//doi.orr//... 1002/ioo.000
[9] Turrigiano G. Homeostatic Synaptic Plasticity: Local and Global Mechanisms for Stabi- lizing Neuronal Function. Cold Spring Harbor Perspectives Biology. 2012;4()):005733. doi: 0o0100.100//shhpepeec..00777777.
[10] Oja E. Simplified neuron model principal component analyzer. Journal of Mathematical Biology. 1982:15(3)2667-273 doi: 10. 10.1007/BF00275687
[11] Oja E, Karhunen the expectation of random matrix. Journal of Mathematical Analysis and Applications. 1985;106(1):69-888)
stochastic approximation of the eigenvectors and eigenvalues of
[12] Becker S, Plumbley M. Unsupervised neural network learning procedures for feature extraction and classification. Applied Intelligence. 1996;6(3):185-4 160:10.00077/FF0012222
[13] Diamantaras KI. Kung SY. Principal component neural networks: theory and applications. USA: John Wiley & Sons, Inc.; 1996.
[14] Montani F, Ince RAA, Senatore R, Arabzadeh E, Diamond ME, Panzeri S. The impact of high-order interactions the rate of synchronous discharge and information trans- mission somatosensory cortex. Philosophical Transactions of the Royal Society of London A: Mathematical, Physical and Engineering Sciences. 2009:367(9001:327-3310 doi: 10. 1098//tta.20009 2009.008..
[15] Montangie I., Montani F. Effect of interacting second- and third-order stimulus-dependent correlations on population-coding asymmetries. Physical Review E. 2016:94(4)0043303. 1oi:10. 1001001133PhhyyeE.9940022233
[16] Attneave F. Some informational aspects of visual perception. (3) doi:1001037/h00556665
Psychological Review.
[17] Barlow HB, Possible principles underlying the transformations of sensory messages. In:
Rosenblith A W, editor. Sensory Communication. MIT Press; 1961. p. 217-234.
[18] Atick JJ, Redlich AN. Towards a Theory of Early Visual Processing. Neural Computation. 1990:2(3):0883200 320.
10

---

[19] Atick JJ, Redlich AN. What Does the Retina Know about Natural Scenes? Neural Computation. 1992:4(2):196-2111 210. 1th:10.11622mec.1199222
[20] Atick JI, Redlich AN. Convergent Algorithm for Sensory Receptive Field Development. Neural Computation. 1993;5(1):45 60. doi:10.1 www.:....//////o/11 1993.5.1.45.
[21] Dong DW, Atick JJ. Temporal decorrelation: a theory of lagged and nonlagged responses in the lateral geniculate nucleus. Network: Computation in Neural Systems. 1995;6(():159-172 doi: 10i: /00////00-0888062222222
[22] Dan Y. Atick JJ. Reid RC. Efficient Coding of Natural Scenes in the Lateral Geniculate Nucleus: Experimental Test of Computational Theory. Journal of Neuroscience. 1(1))36(1)
3362. doi: 1odd:10.1123/INE1 10233//NEUURSST11110000030
[23] Kolda TG, Bader BW. Tensor Decompositions and Applications. SIAM Review. 2009;51 500. doi: 1137/0007011
[24] Cichocki A, Zdunek R, Phan AH, Amari Si. Multi Multi-Way Array (Tensor) Factorizations and Decompositions, In: Nonnegative Matrix and Tensor Factorizations. John Wiley & Sons,
Ltd: 2009, p. 337-432. Available from: httpp:://onlliiellbbrrrrr wiw.y..wwwwyccc................ wileeyccomdoi/aaa.. 10. 70011002/97804777777788
[25]
Cichocki A. Tensor Decompositions: A New Concept in Brain Data Analysis? arXiv: [cs, q-bio, stat]. 2013:.
Anandkumar A, Ge R, Hsu D, Kakade SM, Telgarsky Tensor Decompositions for Learning [26]
Latent Variable Models. Journal of Machine Learning Research. 2014;15(80)2273--2822.
[27] Williams AH, Kim TH, Wang F. Vyas S, Ryu SI, Shenoy KV, et al. Unsupervised Discovery of Demixed, Now-Dimensional Neural Dynamics across Multiple Timescales through Tensor Component Analysis. Neuron. 201 2018:98(6): 109-111)1119-911..ee doi: 10. (100..116]] 2018..0..1 8.0.0115.
[28] Sjöström PJ, Turrigiano GG, Nelson SB. Rate, timing, and cooperativity jointly determine cortical synaptic plasticity. Neuron. ;001;32(6): 114--1164.
[29] Bi GQ, Wang HX. Temporal asymmetry in spike -eeendd synaptic plasticity. Physiol- ogy & Behavior. 2002:77(4-5):551--55
[30] Froemke RC, Dan Y. Spile-timming-deeendenn synaptic modification induced by natural spike trains. Nature. 2002:41666657))4333-433 doi:10. 1038/416433a.
[31] Wang HX, Gerkin RC, Nauen DW, Bi GQ. Coactivation and timing-dependen. integra- tion of synaptic potentiation and depression. Nature Neuroscience. 2005:8(2)1887-193, 10. 1038/nn1387.
[32] Froemke RC, Poo Mm, Dan Y. Spike-timing synaptic plasticity depends on dendritic location. Nature. (303:4447703))221-22
[33] Froemke RC, Tsay IA, Raad M, Long JD, Dan Y. Contribution of individual spikes in burst- induced long-term synaptic modification. Journal of Neurophysiology. 2006:95(); doi: 1tt:10.112//...091000000
[34] Harvey CD, Svoboda K. Locally dynamic synaptic learning rules in pyramidal neuron dendrites. Nature. 2007:450(7173)) 1995-10000 doi: 10. 1038//aauure66411..
[35] Harvey CD, Yasuda R, Zhong H. Svoboda The Spread of Ras Activity Trig- Science. 2008:22155885))13314444
gered by Activation of Single Dendritic Spine. doi: foo::0..i/2//cceneee.ll 159675..
[36] Govindarajan A, Israely I, Huang SY, Tonegawa The dendritic branch is the pre- ferred integrative unit for protein synthesis-dependent LTP. Neuron. 201 2011:;901) 10. 100.....0....
[37] Makino H, Malinow R. Compartmentalized versus Global Synaptic Plasticity on Dendrites Controlled by Experience. Neuron. 2011:7266):00111011

---

[38] Kleindienst T, Winnubst J, Roth-Alpermamn C, Bonhoeffer T, Lohmann C. Srtivity-Dependen Clustering of Functional Synaptic Inputs on Developing Hippocampal Dendrites. Neuron. 201 2011:7266):0112-102
1ai:100016//..eurre 201..o00.011100015..
[39] Chen Villa K, Cha J. So PC, Kubota Y, Nedivi E. Clustered Dynamies of Inhibitory Synapses and Dendritic Spines in the Adult Neocortex. Neuron. 2012:74(2:361-373.
[40]
Takahashi N, Kitamura K. Matsuo N, Mayford M, Kano M, Matsuki N, et al. Locally Synchro- nized Synaptic Inputs. Science. 2012:33566066):333333 doi: 10i:10.11 h2tp///iecc..2/ll.ee. 1210362.
[41] Lee KFH, Soares C, Thivierge JP, Béique JC. Correlated Synaptic Inputs Drive Dendritic Calcium Amplification and Cooperative Plasticity during Clustered Synapse Development. Neuron. 2016:99(4)774-779, 10i:10.1011// www..neuroo.206..0000
[42] Qi L. Eigenvalues of real supersymmetric tensor. 2005:40(6):1302 1324. 010.:0....066.ss.2000..........
Journal of Symbolic Computation.
[43] Lim LH. Singular values and eigenvalues of tensors: a variational approach. In: 1st IEEE International Workshop Computational Advances in Multi-Sensor Adaptive Processing, 2005.; 2005. 129-132.
[44] Gerstner W, Kistler WM. Mathematical formulations of Hebbian learning. Biological Cyber- netics. (002:887(566:4444 415. 160:100..077/644222000443335335
[45] Pfister JP, Gerstner W. Triplets of Spikes in Model of Spike Timing-Depended Dependent Plasticity. The Journal of Neuroscience. 2006:26(38):9673 9682.
[46] Clopath C, Gerstner W. Voltage and Spike Timing Interact in STDP - A Unified Model. Frontiers in Synaptic Neuroscience. 2010:2. 11--100338//hhmmm200000022.
[47] De Lathauwer L, De Moor B. Vandewalle A Multilinear Singular Value Decom- position. SIAM Journal Matrix Analysis and Applications. 2000:21(4):1233-2773
112:01111177088944788666666
[48] Yang Q Yang Y. Further Results for Perron-Frobenius Theorem for Nonnegative Ten- sors II. SIAM Journal on Matrix Analysis and Applications. 2011 (011:32(4)12333-122 10111...13711001133661
[49] Friedland S. Gaubert S, Han L. Perron-Frobenius theorem for nonnegative multilin- car forms and extensions. Linear Algebra and its Applications. 2013:438(2)7338-74) no0:10.1016/..aa.20011
[50] Cartwright D, Sturmfels B. The number of eigenvalues a tensor. Linear Algebra and its Applications. 2013:438(2)942 952.. 1o1:1001016/)..aa20011 1.05.040.
[51] Chang KC, Pearson KJ, Zhang Some variational principles for Z-cigenvalues of nonnegative tensors. Linear Algebra and its Applications. 2013:438(11):41664411 doi: 10. 100.1001......213.222000
[52] Chang KC, Pearson K, Zhang T. Perron-Frobenius theorem for nonnegative tensors. Communi- cations Mathematical Sciences. 2008:6(2):507-52..
[53] Martin D, Fowlkes C, Tal D, Malik J. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, vol. 2; 2001. p. 416-423 vol.2.
[54] Tucker LR. Some mathematical notes on three-mode factor analysis. 1966:31 1 (3):279-311. doi:100.1077/FF22288444
Psychometrika.
[55] Hillar CJ. Lim LH.. Most Tensor Problems Are NP-Hard. Journal of the ACM. 2013 45:39. doi: 10..2.1.////2229.
12

---

[56] Kolda TG, Mayo JR. Shifted Power Method for Computing Tensor Eigenpairs. SIAM Journal on Matrix Analysis and Applications. 201 2011:3224):199551122 doi:10. 111:101117/1000014882
[57] Robeva E. Orthogonal Decomposition of Symmetric Tensors. SIAM Journal on Matrix Analysis and Applications. 201 (): 16::100.1133/1449999444
[58] Yap EL., Greenberg ME. Activity-Regulatee Transcription: Bridging the Gap between Neural Activity and Behavior. Neuron. 2018: 2018:100()):330 100(2):300-3444
101.100.0066..euuoo.200000 2018. 10.011..
[59] Bienenstock EL. Cooper LN, Munro PW. Theory for the development of neuron selectivity: orientation specificity and binocular interaction in visual cortex. The Journal of Neuroscience. 1982:2(
[60] Intrator N, Cooper LN. Objective function formulation of the BCM theory of visual cortical plasticity: Statistical connections, stability conditions. Neural Networks. 1992:5(1):3-17. doi: 1oi:10. 00010010066/00995400000000000000
[61] Gjorgjieva J, Clopath C, Audet J, Pfister JP. A triplet spike-timing hikke-iiinng-deeeddeettt plasticity model generalizes the Bienenstock Bicnenstock-Cooper-Mee Cooper-Munro relations. Proceedings of the National Academy of Sciences. 201 1011::00044) doi: 10.7://a.a// 105933108.
rule to higher-order spatiotemporal cor-
[62] Lawlor M, Zucker SW. Feedforward Learning of Mixture Models. Advances in Neural Information Processing Systems. 2014:27.
[63] Hyvärinen A, Oja E. Simple neuron models for independent component analysis. International Journal of Neural Systems. 1996:7 60010000011222220222200700006660
[64] Bell AJ, Sejnowski TJ. An information-maxxiiiaaii deconvolution. Neural Computation. 1995:7/6):1229-11555 doi:10. 162/neco. (ttp///co.1995..66... 1995.7.6.11122..
approach to blind separation and blind
[65] Lee D, Rokni U, Sompolinsky H. Algorithms for Independent Components Analysis and Higher Order Statistics. In: Advances in Neural Information Processing Systems. vol. 12. MIT Press; 2000.Availabll from: https: /tttps://ppprrss nips cctpppaap/////9///ashhll (eieebbb677666811554477776666363 www.sleeeAssttttt..oo
[66] Taylor JG, Coombes S. Learning higher order correlations. Neural Networks. 1993;6(3):423-
427. 1oi: 10i:10..116//0 001006608993-6000099)900000
[67] Ziegaus C, Lang EW. A neural implementation of the JADE algorithm (nJADE) using higher- order neurons. Neurocomputing. 2004;56:79-100, doi: 460::0..006630022222122000133113
[68] Wang Y, Anandkumar A. Online and differentially-priinn tensor decomposition. In: Proceed- ings of the 30th International Conference Neural Information Processing Systems. NIPS' 16.
Red Hook, NY, USA: Curran Associates Inc.; 2016. p. 3539-3547.
[69] Wang PA, Lu CJ. Tensor Decomposition via Simultaneous Power Iteration. In: International Conference on Machine Learning. PMLR; 2017. p. 3665-3673 Available from: http:// proceedings mlr, ppwss//0//waa..oooooppprrpso
html
[70] Ge R. Huang F, Jin C, Yuan Y. Escaping From Saddle Points Online Stochastic Gradient for Tensor Decomposition. In: Conference on Learning Theory. PMLR; 2015. p. 797-842. Available from:
httpreeeddnnnnmm...meessssseeeee www.pprees/eeee Seteess/440//e5555 htmmm
mlr.
[71] Merolla PA, Arthur JV, Alvarez-Icaza R, Cassidy AS, Sawada Akopyan F, al. A million spiking-neuron integrated circuit with scalable communication network and interface. Science. 2014:35(6119):668 663. doi: 10. 126//cience. 22f/sieencee.255642....
[72] Davies M. Srinivasa N. Lin TH, Chinya G. Cao Y. Choday SH. Loihi: A Neu- romorphic Manycore Processor with On-Chip Learning. IEEE Micro. 2018:38(1::22-99. 1oi: 2018..12130359
[73] Zenke F, Neftei EO. Brain-Inspired Learning on Neuromorphic Substrates. Proceedings of the IEEE. 2021; 109(5))999 doi: 110.0100000PPP002000000000221
13

---

[74] Brito CSN, Gerstner W. Nonlinear Hebbian Learning
Unifying Principle
in Receptive Field Formation. 1o1:10.1371/joumall pebi. 1005070.
PLOS Computational Biology. 2016:120))e1000070
[75] Miller KD, Troyer TW. Neural noise can explaine expansive, power-law nonlinearities in neural
response functions. J Neurophysiol. 2002:87(2):653-6666 659.
[76] Priebe NJ, Ferster D. Inhibition, Spike Threshold, and Stimulus Selectivity in Primary Visual Cortex. Neuron. 2008;57(4)4422-999 doi:10. 1ad.10000166..muroooi m0016/..eumon.2008.22.0000 00.000..00........
[77] Kempter R. Gerstner W, Van Hemmen JL. Hebbian learning and spiking neurons. Physical Review E. 19995
[78] Gerstner W, Kempter R, van Hemmen JL. Wagner H. A neuronal learning rule for sub- millisecond temporal coding. Nature. 19966333(5595)77-77777 doi:10. 1038/383076a0.
[79] Markram H. Regulation of Synaptic Efficacy by Coincidence of Postsynaptic APs and EPSPs. Science. 1997:275(5297)2213-22 doi:10. 126/science222555297..11
[80] Bi Gq, Poo Mm. Synaptic modifications in cultured hippocampal neurons: dependence spike timing, synaptic strength, and postsynaptic cell type. The Journal of Neuroscience.
1994:
[81] Hansel D, Vreeswijk Cv. How Noise Contributes Contrast Invariance of Orienta- tion Tuning in Cat Visual Cortex. Journal of Neuroscience. 2002:22(12;:111-5512
[82] Priebe NJ, Mechler F, Carandini M, Ferster The contribution of spike threshold to the dichotomy of cortical simple and complex cells. Nature Neuroscience. (004:7(10):111331122 doi: 10.100 1308/
[83] Priebe NJ, Ferster D. Mechanisms underlying cross-orientation suppression in cat visual cortex. Nature Neuroscience. 2006:9(4):552-561. doi: 103./000666660
[84] Güitig R, Aharonov R, Rotter S, Sompolinsky H. Learning input correlations through nonlinear temporally asymmetric Hebbian plasticity. The Journal of Neuroscience: The Official Journal of the Society for Neuroscience. 2003:230)
14


Reviews:
[{"review_id": "sLwArE4eYSt", "reviewer": "Reviewer_2bLY", "summary": "The authors analyze a broad class of nonlinear Hebbian learning rules. These rules generalize the well-studied Oja's learning rule to higher-order input correlations, which can be represented in symmetric higher-order tensors of synaptic input statistics. There is a broad modeling literature on synaptic plasticity, and it is not my primary area of research. Nonetheless, this generalization of Oja's rule appears novel to me, and the extensions outlined in this manuscript feel very natural and interesting.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors analyze a broad class of nonlinear Hebbian learning rules. These rules generalize the well-studied Oja's learning rule to higher-order input correlations, which can be represented in symmetric higher-order tensors of synaptic input statistics. There is a broad modeling literature on synaptic plasticity, and it is not my primary area of research. Nonetheless, this generalization of Oja's rule appears novel to me, and the extensions outlined in this manuscript feel very natural and interesting.\n\nThe authors point out that \"any plasticity model with a finite Taylor polynomial\" can be seen as a special case of their framework. However, there are few insights into this general case, and most of their analysis is confined to simpler cases that are analytically tractable. These special cases are still interesting.\n\nIn the classic case of Oja's rule, the largest eigenvector is the unique/global attractor of the system. However, this is not the case for the generalized model considered here. Thus, the authors' results are somewhat pessimistic because the learning dynamics will reach different steady-states depending on the initialization. This makes it more difficult to interpret the functional purpose or outcome of the learning rule, unless one is able to make additional assumptions about the initial state of the system.\n\nOn the other hand, it appears that the dynamics do converge, in practice, to the top few eigenvectors when initialized randomly (see fig. 1h). This at least holds in the special cases the authors examined empirically. One of the more useful results in the paper is a proof on the relative sizes of the basins of attraction, which provides good intuition.\n\nOne unanswered question I have is whether there are reasons to think the top eigenvectors of higher-order input statistics would be useful features to learn? In Oja's rule, the synaptic weights converge to the top principal component, and thus the neuron becomes tuned to detect salient (i.e. high-variance) features. The interpretation of feature-detectors for higher-order statistics is less intuitive to me, though I know there has been some work done on this -- e.g., https://doi.org/10.1167/13.9.974\n\nOverall, I think these results are very thorough and interesting, if still a bit abstract and general. If the authors could provide readers with more intuition about how higher-order features may be useful features for downstream neural computation, I think it would broaden the appeal and impact of the paper. Either way, I hope to see this paper accepted.\n\nOther comments:\n\n- I attempted to verify all derivations and statements in the supplemental material and I could not find any errors.\n\n- Do the dynamics in fig. 1e actually converge? The trajectories look very volatile.\n\n- The fact that the learning rule does not always converge to the largest eigenvector is not suprising, since recovering the top eigenvector is NP-hard even for symmetric tensors (Hillar & Lim, 2013).\n\n- Notation is somewhat confusing and inconsistent in places. There are three different typefaces for the variable \"x\" in equations 1 and 2. The variable \\mu_{i, \\alpha} is boldfaced in equation 3, but is not boldfaced in equation 2. It would be helpful if the explanation that \\alpha is a multi-index could be moved a few sentences earlier, right after equation 2.\n\n- On lines 90-91, it is claimed that the first k E-eigenvectors define the best rank-k approximation approximation of \\mu in terms of a least-squares criterion. Typically a rank-k approximation is understood to mean a CP-decomposition of the tensor (see Kolda & Bader, 2009), but it seems like here it is meant to be the best rank-k Tucker decomposition to approximate the tensor? This is the sense I got from reading the citation to De Lathauwer (2000), which distriguishes between a rank-R and a rank-(R_1, R_2, ..., R_n) approximation to a tensor. Can the authors clarify these points more explicitly?\n\n- In the caption of Figure 2, a small typo appears: a \"v2\" should be \"v_2\"\n\n- On line 553 of the supplement, a reference / citation is broken.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are adequately discussed and I foresee no negative societal impacts.", "review_text": "", "extra_scores": {"ethical_concerns": "n/a", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "i5C-gQ1cVX4", "reviewer": "Reviewer_aWnj", "summary": "The authors study generalized, nonlinear Hebbian learning rules, and find that neurons with such plasticity and feedforward inputs can learn tensor decompositions of higher-order correlations.  They further demonstrate that for simple, biologically plausible scenarios, the tensor eigenvectors are attractors. After determining the basins of attraction, the authors show that the basin of the dominant eigenvector has the largest volume.  Finally, they show that any plasticity rule with a finite Taylor expansion in synaptic input, neural output, and weight variables has equilibrium states  that are similar to those already found and describe higher-order input correlations.", "questions": "", "limitations": "", "rating": 9, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors have extended a large body of prior work on local linear learning rules. These Hebbian (and generalized Hebbian) rules have been shown to allow neurons to model input statistics; for example, Oja's rule has been shown to be stable and instantiate an online algorithm for PCA.  The thorough analysis encapsulated in this manuscript represents a welcome advance in this classical unsupervised learning framework.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "hmezRj5rDd2", "reviewer": "Reviewer_gbB7", "summary": "This work proposes to determine the solutions of \"generalized\" Hebbian rules, i.e., applied to tensor matrices, and analyze the stability of these solutions. The model starts by postulating a biologically sensible form of nonlinear Hebbian rule as polynomials of local input activity, postsynaptic activity, and current synaptic weight. From this generalized form of Hebbian rule, the authors can determine the nature of the steady-states of the dynamic and relate it to the spectrum and eigenspace of the higher-order covariance matrix. Furthermore, the size of the basins of attraction is determined. The model is then used on toy experiments proving the convergence of the learning rules for specifically chosen parameters of the learning rule. ", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "First is addressed the organization and clarity of the paper. \n\nThe paper is well-written, and results and theorems are nicely explained using informal explanations and leaving the full proof to the appendix. Unfortunately, the paper starts with a \"Result\" section without introducing the necessary background and existing rules and contextualizing the problem. Related work can certainly appear at the end, but I believe various things should have been introduced early on. \n\nFor example:\n1) What are the limitations of existing Hebbian rules, and why do we need higher-order power? And what is the difference with nonlinear inputs/outputs?\n\n2) The point above would have naturally lead to the introduction of higher-order covariance matrices, which are naturally formulated as tensors. As a result, one can regret the mention of prevalent tasks that is tensor PCA or independent component analysis (very briefly mentioned by the authors). For example, the work of Cichocki, most recently in [i], presents examples where tensor decomposition is essential. The author of [i] is one of many interested in biological implementation for third and fourth-order tensors and has been omitted to be cited. For example, the work of [ii] proposes similar learning rules in the case of ICA and discusses the possible biological implementation using higher-order neurons. \n\n3) Tensor decomposition is also a very active field of research in the machine learning community, and one can regret that \"gradient-based\" methods (which Oja is one of) have not been discussed. \n\nThe experiments are following each other as they happened chronologically, following the authors' discoveries as they happened rather than proposing an overarching methodology. It can be seen when using \"We next investigated\" \"We next asked\" followed by another \"We next investigated\" in section 2.3. Having a separate methodology section could help the reader understand what the paper aims to achieve. But it still important to point out that paragraphs taken separately are nicely written and clear. \n\nNow is addressed the originality of the paper. \n\nThe main originality of the paper is the analysis of the solution of the ad-hoc learning rule. It cannot be said that such learning has never been proposed nor analyzed in other contexts cf reference given above. Hebbian/Oja's rules have long been known to be gradient-based methods of \"Extended covariance\" matrices, so this result is not very surprising, although important to be theoretically proved. Unfortunately, it is also true that how these rules can be biologically implemented is heavily discussed and shown to require recurrent connections or to operate on various timescales when considering more than one output neuron. These facts are largely overlooked, which greatly weakens the paper as its relevance can be questioned from the start if the main assumptions of the paper are not properly cleared out. \nThe authors claim to address phenomenological rules, but no results seem to validate or invalidate known plasticity rules. One could have preferred fewer but clearer examples that would relate this work to neuroscientific facts, rather than speculations of what is and what is not biologically plausible. In the case of the spiking neuron (Sec 3.1), this work should have related to the paper from [iv] which would have strengthened the paper as it offers a nice theoretical framework for it.  \n\nNow is addressed the significance of the paper. \n\nThe paper appears to propose an interesting analysis of the solution of an ad-hoc learning rule. Unfortunately, this work does not make predictions on how (implementation-wise) and where (structurally) they could occur in the brain. There have also been various higher-order neuron models that have been used c.f. [ii] in concrete tasks, when the experiments are not really relatable and do not add much to the paper. One could have expected examples where tensor decomposition has been used either in neuroscience or machine learning. \n\n\n\n[i] Cichocki, Andrzej. \"Tensor decompositions: a new concept in brain data analysis?.\" arXiv preprint arXiv:1305.0395 (2013).\n\n[ii] Ziegaus, Ch, and Elmar Wolfgang Lang. \"A neural implementation of the JADE algorithm (nJADE) using higher-order neurons.\" Neurocomputing 56 (2004): 79-100.\n\n[iii] Ge, Rong, et al. \"Escaping from saddle points—online stochastic gradient for tensor decomposition.\" Conference on learning theory. PMLR, 2015.\n\n[iv] Gjorgjieva, Julijana, et al. \"A triplet spike-timing–dependent plasticity model generalizes the Bienenstock–Cooper–Munro rule to higher-order spatiotemporal correlations.\" Proceedings of the National Academy of Sciences 108.48 (2011): 19383-19388.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations of the paper are hard to find as the paper does not clearly contextualize in which fields it operates, as mentioned in the main review. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "7B3a45TOzU8", "reviewer": "Reviewer_SKSQ", "summary": "This paper shows that nonlinear Hebbian rules allow a neuron to learn tensor decompositions of the higher-order input correlation. There have been theoretical works on nonlinear Hebbian learning rules as well as triplet STDP BCM learning rules, but I believe this might be the first that showed systematically and mathematically it enables the learning of tensor eigenvectors and characterize their basins of attractions.  ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While I understand that higher-order bit,  I found the paper too dense and hard to follow.  I found the math notations in the main text complex, perhaps unnecessarily complex,  and hard to follow.  I looked into the Appendix for clarification but found Eq. 1 in the main text and Eqn. 2 in the Appendix to be inconsistent -- was one of them wrong?  I also could not derive Eq. 3 from Eq. 2 in the Appendix. My puzzle is that even if expanding the norm of J of dt, the 1st order expansion still depends on the norm of vector J.  In contrast,  Eq. 3 in Appendix doesn't depend on the norm of  vector J, but only depend on each element J_j.  The definition of  \\dot{J}_j(t) right after Eq. 4 in Appendix is also unclear -- what is the difference between J_i and the bold J_i on the right-hand side of the definition?   While they might be minor issues,  these issues in the first few steps made it sufficiently frustrating for me to press on to evaluate the details. In any case, it is unreasonable for the reviewers to plow through the 20 pages of math in the Appendix.\n\nWhile I believe the work is probably solid and the contribution fundamental, I wonder if NeurIPS is the appropriate forum. Maybe Neural Computation or Plos Computational Neuroscience would be a better forum. It would be helpful to provide more explanations (e.g. explaining  \\mu in Eq. 3 is the input correlation tensor ) and perhaps focus on one or two simple concrete examples to illustrate the results and show what higher-order correlations the neurons actually learn in those examples.  Perhaps something like the long contours in Lawlor and Zucker's mixture model, or perhaps corners or curvatures as in Olshausen's overcomplete sparse coding?    Perhaps they are trying to do this with Figure 1c? But more explanations are necessary.  I don't enjoy the presentation of the paper,  but I am willing to give it a high score because I do think the contribution is potentially important and am willing to give it the benefit of the doubt for now. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Broader impact section missing.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper shows how nonlinear Hebbian rules learn tensor decompositions of higher-order input correlations. This is an important contribution to the literature on Hebbian synaptic plasticity. Original submission missed important citations and related work, however the revised introduction proposed by the authors better frame the work and its contributions.
Decision: Accept (Poster)