Tensor decompositions of higher-order correlations by nonlinear Hebbian plasticity

1 Introduction . In Hebbian learning , potentiation of the net synaptic weight between two neurons is driven by the correlation between pre- and postsynaptic activity [ 1 ] . That postulate is a cornerstone of the theory of synaptic plasticity and learning [ 2 , 3 ] . In its basic form , the Hebbian model leads to runaway potentiation or depression of synapses , since the pre-post correlation increases with increasing synaptic weight [ 4 ] . That runaway potentiation can be stabilized by supplemental homeostatic plasticity dynamics [ 5 ] , by weight dependence in the learning rule [ 6 , 7 ] , or by synaptic scaling regulating a neuron ’ s total synaptic weight [ 8 , 9 ] . In 1982 , Erkki Oja observed that a linear neuron with Hebbian plasticity and synaptic scaling learns the first principal component of its inputs [ 10 ] . In 1985 , Oja and Karhunen proved that this is a global attractor of the Hebbian dynamics [ 11 ] . This led to a fountain of research on unsupervised feature learning in neural networks [ 12 , 13 ] . Principal component analysis ( PCA ) describes second-order features of a random variable . Both naturalistic stimuli and neural activity can , however , exhibit higher-order correlations [ 14 , 15 ] . Canonical models of retinal and thalamic processing whiten inputs , removing pairwise features [ 16–22 ] . Beyond-pairwise features , encoded in tensors , can provide a powerful substrate for learning from data [ 23–27 ] . The basic Hebbian postulate does not take into account fundamental nonlinear aspects of biological synaptic plasticity in cortical pyramidal neurons . First , synaptic plasticity depends on beyond-pairwise activity correlations [ 28–33 ] . Second , spatially clustered and temporally coactive synapses exhibit 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . correlated and cooperative plasticity [ 34–41 ] . There is a rich literature on computationally motivated forms of nonlinear Hebbian learning ( section 3.2 ) . Here , we will prove that these biologically motivated nonlinearities allow a neuron to learn higher-order features of its inputs . We study the dynamics of a simple family of generalized Hebbian learning rules , combined with synaptic scaling ( eq . 1 ) . Equilibria of these learning rules are invariants of higher-order input correlation tensors . The order of input correlation ( pair , triplet , etc . ) depends on the pre- and postsynaptic nonlinearities of the learning rule . When the only nonlinearity in the plasticity rule is postsynaptic , the steady states are eigenvectors of higher-order input correlation tensors [ 42 , 43 ] . We prove that these eigenvectors are attractors of the generalized Hebbian plasticity dynamics and characterize their basins of attraction . Then , we study further generalizations of these learning rules . We show that any plasticity model ( with a finite Taylor expansion in the synaptic input , neural output , and synaptic weight ) has steady states that generalize those tensor decompositions to multiple input correlations , including generalized tensor eigenvectors . We show that these generalized tensor eigenvectors are stable equilibria of the learning dynamics . Due to the complexity of the arbitrary learning rules , we are unable to fully determine their basins of attraction . We do find that these generalized tensor eigenvectors are in an attracting set for the dynamics , and characterize its basin of attraction . Finally , we conclude by discussing extensions of these results to spiking models and weight-dependent plasticity . 2 Results . Take a neuron receiving K inputs xi ( t ) , i 2 [ K ] , each filtered through a connection with synaptic weight Ji ( t ) to produce activity n ( t ) . We consider synaptic plasticity where the evolution of Ji can depend nonlinearly on the postsynaptic activity n ( t ) , the local input xi ( t ) , and the current synaptic weight Ji ( t ) . We model these dependencies in a learning rule f : Ji ( t+ dt ) = Ji ( t ) + dt/⌧ fi ( n ( t ) , xi ( t ) , Ji ( t ) ) J ( t ) + dt/⌧f n ( t ) , x ( t ) , J ( t ) , where fi ( t ) = n a ( t ) xbi ( t ) J c i ( t ) . ( 1 ) The parameter a sets the output-dependent nonlinearity of the learning rule , b sets the input-dependent nonlinearity , and c sets its dependence on the current synaptic weight . Eq . 1 assumes a simple form for these nonlinearities ; we discuss arbitrary nonlinear learning rules in section 2.3 . We assume that a and b are positive integers , as in higher-order voltage or spike timing–dependent plasticity ( STDP ) models [ 44–46 ] . The scaling by the norm of the synaptic weight vector , ||J || , models homeostatic synaptic scaling [ 8–10 ] . Bold type indicates a vector , matrix , or tensor ( depending on the variable ) and regular font with lower indices indicates elements thereof . Roman type denotes a random variable ( x ) . We assume that x ( t ) is drawn from a stationary distribution with finite moments of order a + b . Combined with a linear neuron , n = JTx , and a slow learning rate , ⌧ dt , this implies the following dynamics for J ( appendix A.1 ) : ⌧ J̇i = J c i X ↵ µi , ↵ ( J ⌦a ) ↵ Ji X j , ↵ Jc+1j µj , ↵ ( J ⌦a ) ↵ . ( 2 ) In eq . 2 , J̇i = dJi/dt , ↵ = ( j1 , . . . , ja ) is a multi-index , and ⌦ is the vector outer product ; J⌦a is the a-fold outer product of the synaptic weight vector J . µ is a higher-order moment ( correlation ) tensor of the inputs : µi , ↵ = hxbi ( x ⌦a ) ↵ix ( 3 ) where hix denotes the expectation with respect to the distribution of the inputs . µ is an ( a+ 1 ) -order tensor containing an ( a+ b ) -order joint moment of x . The order of the tensor refers to its number of indices , so a vector is a first-order tensor and a matrix a second-order tensor . µ is cubical ; each mode of µ has the same dimension K. In the first term of eq . 2 , for example , P ↵ µi , ↵ ( J ⌦a ) ↵ takes the dot product of J along modes 2 through a+ 1 of µ . 2.1 Steady states of nonlinear Hebbian learning . If we take a = b = 1 , then µ is the second-order correlation of x and ↵ is just the index j . With c = 0 also , eq . 2 reduces to Oja ’ s rule and J is guaranteed to converge to the dominant eigenvector of µ [ 10 , 11 ] . We next investigate the steady states of eq . 2 for arbitrary ( a , b ) 2 Z2+ , c 2 R. Ji = 0 is a trivial steady state . At steady states of eq . 2 where Ji 6= 0 , X ↵ µi , ↵ ( J ⌦a ) ↵ = J 1 c i , where ( µ , J ) = X j , ↵ Jc+1j µj , ↵ ( J ⌦a ) ↵ , ( 4 ) so that J is invariant under the multilinear map of µ except for a scaling by and element-wise exponentiation by 1 c. For two parameter families ( a , b , c ) , eq . 4 reduces to different types of tensor eigenequation [ 47 , 42 , 43 ] . We next briefly describe these and some of their properties . First , if a+c = 1 , we have the tensor eigenequation P ↵ µi , ↵ ( J ⌦a ) ↵ = Jai . Qi called , J the tensor eigenpair [ 42 ] and Lim called them the ` a-norm eigenpair [ 43 ] . There are KaK 1 such eigenpairs [ 42 ] . If µ 0 element-wise , then it has a unique largest eigenvalue with a real , non-negative eigenvector J , analogous to the Perron-Frobenius theorem for matrices [ 43 , 48 ] . If µ is weakly irreducible , that eigenvector is strictly positive [ 49 ] . In contrast to matrix eigenvectors , however , for a > 1 these tensor eigenvectors are not necessarily invariant under orthogonal transformations [ 42 ] . If c = 0 , we have another variant of tensor eigenvalue/vector equation : X ↵ µi , ↵ ( J ⌦a ) ↵ = Ji ( 5 ) Qi called these , J an E-eigenpair [ 42 ] and Lim called them the ` 2-eigenpair [ 43 ] . In general , a tensor may have infinitely many such eigenpairs . If the spectrum of a K-dimensional tensor of order a+ 1 is finite , however , there are ( aK 1 ) / ( a 1 ) eigenvalues counted with multiplicity , and the spectrum of a symmetric tensor is finite [ 50 , 51 ] . ( If b = 1 , µ is symmetric . ) Unlike the steady states when a + c = 1 , these eigenpairs are invariant under orthogonal transformations [ 42 ] . For non-negative µ , there exists a positive eigenpair [ 52 ] . It may not be unique , however , unlike the largest eigenpair for a+ c = 1 ( an anti-Perron-Frobenius result ) [ 51 ] . In the remainder of this paper we will usually focus on parameter sets with c = 0 and use “ tensor eigenvector ” to refer to those of eq . 5 . 2.2 Dynamics of nonlinear Hebbian learning . For the linear Hebbian rule , ( a , b , c ) = ( 1 , 1 , 0 ) , Oja and Karhunen proved that the first principal component of the inputs is a global attractor of eq . 2 [ 11 ] . We thus asked whether the first tensor eigenvector is a global attractor of eq . 2 when c = 0 but ( a , b ) 6= ( 1 , 1 ) . We first simulated the nonlinear Hebbian dynamics . For the inputs x , we whitened 35⇥35 pixel image patches sampled from the Berkeley segmentation dataset ( fig . 1a ; [ 53 ] ) . For b 6= 1 , the correlation of these image patches was not symmetric ( fig . 1b ) . The mean squared error of the canonical polyadic ( CP ) approximation of these tensors was higher for b = 1 than b = 2 ( fig . 1d ) . It decreased slowly past rank ⇠ 10 , and the rank of the input correlation tensors was at least 30 ( fig . 1d ) . The nonlinear Hebbian learning dynamics converged to an equilibrium from random initial conditions ( e.g. , fig . 1e , f ) , around which the weights fluctuated due to the finite learning timescale ⌧ . Any equilibrium is guaranteed to be some eigenvector of the input correlation tensor µ ( section 2.1 ) . For individual realizations of the weight dynamics , we computed the overlap between the final synaptic weight vector and each of the first 10 eigenvectors ( components of the Tucker decomposition ) of the corresponding input correlation µ [ 47 , 54 ] . The dynamics most frequently converged to the first eigenvector . For a non-negligible fraction of initial conditions , however , the nonlinear Hebbian rule converged to subdominant eigenvectors ( fig . 1g , h ) . The input correlations µ did have a unique dominant eigenvector ( fig . 3a , blue ) , but the dynamics of eq . 2 did not always converge to it . This finding stands in contrast to the standard Hebbian rule , which must converge to the first eigenvector if it is unique [ 11 ] . While the top eigenvector of a matrix can be computed efficiently , computing the top eigenvector of a tensor is , in general , NP-hard [ 55 ] . To understand the learning dynamics further , we examined them analytically . Our main finding is that with ( b , c ) = ( 1 , 0 ) in the generalized Hebbian rule , eigenvectors of µ are attractors of eq . 2 . Contrary to the case when a = 1 ( Oja ’ s rule ) , the dynamics are thus not guaranteed to converge to the first eigenvector of the input correlation tensor when a > 1 . The first eigenvector of µ does , however , have the largest basin of attraction . Theorem 1 . In eq . 2 , take ( b , c ) = ( 1 , 0 ) . Let µ be a cubical , symmetric tensor of order a+ 1 and odeco with R components : µ = RX r=1 r ( Ur ) ⌦a+1 ( 6 ) where U is a matrix of unit-norm orthogonal eigenvectors : UTU = I . Let i > 0 for each i 2 [ R ] and i 6= j 8 ( i , j ) 2 [ R ] ⇥ [ R ] with i 6= j . Then for each k 2 [ R ] : 1 . With any odd a > 1 , J = ±Uk are attracting fixed points of eq . 2 and their basin of attrac- tion is T i2 [ R ] \k n J : UTi J/UTk J < ( k/ i ) 1/ ( a 1 ) o . Within that region , the separatrix of +Uk and Uk is the hyperplane orthogonal to UTk : { J : U T k J = 0 } . 2 . With any even positive a , J = Uk is an attracting fixed point of eq . 2 and its basin of attraction is J : UTk J > 0 T i2 [ R ] \k n J : UTi J/UTk J < ( k/ i ) 1/ ( a 1 ) o . 3 . With any even positive a , J = 0 is a neutrally stable fixed point of eq . 2 with basin of attraction n J : PR j=1 ( U T j J ) 2 < 1 ^UTk J < 0 8 k 2 [ R ] o . Remark . Each component of the orthogonal decomposition of an odeco tensor µ ( eq . 5 ) is an eigenvector of µ . If R < ( aK 1 ) / ( a 1 ) , there are additional eigenvectors . The components of the orthogonal decomposition are the robust eigenvectors of µ : the attractors of its multilinear map [ 26 , 56 ] . The non-robust eigenvectors of an odeco tensor are fixed by its robust eigenvectors and their eigenvalues [ 57 ] . The proof of theorem 1 is given in appendix A.2 . To prove theorem 1 , we project J onto the eigenvectors of µ , and study the dynamics of the loadings v = UTJ . This leads to the discovery of a collection of unstable manifolds : each pair of axes ( i , k ) has an associated unstable hyperplane vi = vk ( k/ i ) 1/ ( a 1 ) ( and if a is odd , also the corresponding hyperplane with negative slope ) . These partition the phase space into the basins of attraction of the eigenvectors of µ . For example , consider a fourth-order input correlation ( corresponding to a = 3 in eq . 1 ) with two eigenvectors . The phase portrait of the loadings is in fig . 2a , with the nullclines in black and attracting and unstable manifolds in blue . The attracting manifold is the unit sphere . There are two unstable hyperplanes that partition the phase space into the basins of attraction of ( 0 , ±1 ) and ( ±1 , 0 ) , where the synaptic weights J are an eigenvector of µ . For even a , only the unstable hyperplanes with positive slope survive ( fig . 2b , blue line ) . The unit sphere ( fig . 2b , blue ) is attracting from any region where at least one loading is positive . Its vertices [ 1 ] ⇥ [ 1 ] are equilibria ; ( v1 , v2 ) = ( 1 , 0 ) and ( 0 , 1 ) are attractors and the unstable hyperplane separates their basins of attraction . For the region with all loadings negative that is the basin of attraction of the origin , noise will drive the system away from zero towards one of the eigenvector equilibria . are v2 = ±v1 ( 1/ 2 ) 1/a 1. b ) Even a ( a = 2 ) . The unstable set is v2 = v1 ( 1/ 2 ) 1/a 1 ( solid blue line ) . In ( a , b ) , ( 1 , 2 ) = ( 3 , 1 ) . c ) Phase portrait for a two-term learning rule ( eq . 9 ) . All parameters of the input correlation tensors ( mr , Ai ) are equal to one . Dashed blue curve : L = { v : P m , j mjv am+1 j = 0 } . By partitioning the phase space of J into basins of attraction for eigenvectors of µ , theorem 1 also allows us to determine the volumes of those basins of attraction . The basins of attraction are open sections of RK so we measure their volume relative to that of a large hypercube . Corollary 1.1 . Let Vk be the relative volume of the basin of attraction for J = Uk . For odd a > 1 , Vk = R 1 RY i=1 ✓ k i ◆1/ ( a 1 ) ( 7 ) Corollary 1.2 . Let Vk be the relative volume of the basin of attraction for J = Uk . For even positive a , Vk =2 1 R R 1 Y i ✓ k i ◆1/ ( a 1 ) + ( R 1 ) 1 X j 6=k Y i 6=j ✓ k i ◆1/ ( a 1 ) + ( R 2 ) 1 X j , l 6=k Y i 6=j , l ✓ k i ◆1/ ( a 1 ) + . . .+ 1 ! ( 8 ) The calculations for corollaries 1.1 , 1.2 are given in appendix A.2 . We see that the volumes of the basins of attraction depend on the spectrum of µ , its rank R , and its order a . The result for odd a also provides a lower bound on the volume for even a . The result is simpler for odd a so we focus our discussion here on that case . While eigenvectors of µ with small eigenvalues contribute little to values of the input correlation µ , they can have a large impact on the basins of attraction . The volume of the basin of attraction of eigenvector k is proportional to R/ ( a 1 ) k . An eigenvector with eigenvalue ✏ scales the basins of attraction for the other eigenvectors by ✏ 1/a 1 . The relative volume of two eigenvectors ’ basins of attraction is , however , unaffected by the other eigenvalues whatever their amplitude . With a odd , the ratio of the volumes of the basins of attraction of eigenvectors k and j is Vk/Vj = ( k/ j ) R/a 1 . We see in theorem 1 that attractors of eq . 2 are points on the unit hypersphere , S. For odd a , S is an attracting set for eq . 2 . For even a , the section of S with at least one positive coordinate is an attracting set ( fig . 2a , b , blue circle ; see proof of theorem 1 in appendix A.2 ) . We thus next computed the surface area of the section of S in the basin of attraction for eigenvector k , Ak ( corollary 1.3 in appendix A.2 ) . The result requires knowledge of all non-negligible eigenvalues of µ , and the ratio Ak/Aj does not exhibit the cancellation that Vk/Vj does for odd a . We saw in simulations with natural image patch inputs and initial conditions for J chosen uniformly at random on S , the basin of attraction for UT1 was ⇠ 3⇥ larger than that for UT2 and the higher eigenvectors had negligible basins of attraction ( fig . 1h ) . In Oja ’ s model , ( a , b , c ) = ( 1 , 1 , 0 ) in eq . 1 , if the largest eigenvalue has multiplicity d > 1 then the d-sphere spanned by those codominant eigenvectors is a globally attracting equilibrium manifold for the synaptic weights . The corresponding result for a > 1 is ( for the formal statement and proof , see corollary 1.4 in appendix A.2 ) : Corollary 1.3 . ( Informal ) If any d robust eigenvalues of µ are equal , the d-sphere spanned by their robust eigenvectors is an attracting equilibrium manifold and its basin of attraction is defined by each of those d eigenvectors ’ basins of attraction boundaries with the other R d robust eigenvectors .

Summary: This work proposes to determine the solutions of "generalized" Hebbian rules, i.e., applied to tensor matrices, and analyze the stability of these solutions. The model starts by postulating a biologically sensible form of nonlinear Hebbian rule as polynomials of local input activity, postsynaptic activity, and current synaptic weight. From this generalized form of Hebbian rule, the authors can determine the nature of the steady-states of the dynamic and relate it to the spectrum and eigenspace of the higher-order covariance matrix. Furthermore, the size of the basins of attraction is determined. The model is then used on toy experiments proving the convergence of the learning rules for specifically chosen parameters of the learning rule.