Online approximate factorization of a kernel matrix by a Hebbian neural network

1 INTRODUCTION . Brain inspired learning algorithms have a long history in the field of neural networks and machine learning ( Rosenblatt , 1958 ; Olshausen & Field , 1996 ; Lee & Seung , 1999 ; Riesenhuber & Poggio , 1999 ; Hinton , 2007 ; Lillicrap et al. , 2016 ) . While many algorithms have diverged from their biological roots , the motivation to study biology remains clear : the human brain is such a powerful learning agent , there must be insights to be gained by making our algorithms look “ brain-like ” . This paper is focused on merging biological constraints with the well-established field of kernel-based machine learning . A common assumption in brain-inspired models of learning is that synaptic update rules should be a ) online , meaning the algorithm only has access to a single input pattern at a time and b ) local , meaning synapses should only be modified using information immediately available to the synapse , often just the pre- and post-firing rates of the neurons to which it is connected . Learning rules with these properties are commonly referred to as Hebbian learning rules ( Chklovskii , 2016 ) . Recent works have devised neural networks with Hebbian learning rules that perform linear similarity matching . These networks map every input xt to a representation yt such that linear output similarities match linear input similarities ys ·yt ≈ xs ·xt . These networks are interesting as models for real brains because they display a number of interesting biological properties : they are recurrent networks with correlation-based learning rules ( Pehlevan et al. , 2018 ) and can be modified to include non-negativity ( Pehlevan & Chklovskii , 2014 ) , sparsity , and convolutional structure ( Obeid et al. , 2019 ) . However there is a problem if one believes these networks should ultimately generate representations which are useful for downstream tasks . If similarities are actually matched , that is if ys ·yt = xs ·xt , then the outputs are simply an orthogonal transformation of the inputs , yt = Qxt , which is unlikely to have significant impact on downstream tasks . Bahroun et al . ( 2017 ) identified this problem and proposed a solution : instead of matching linear input similarities , one can match nonlinear input similarities : ys · yt ≈ f ( xs , xt ) . The authors provided a method that can be applied to any shiftinvariant kernel . They applied the random Fourier feature method of Rahimi et al . ( 2007 ) to map inputs to nonlinear feature vectors x→ ψ and then applied the linear similarity matching framework of Pehlevan et al . ( 2018 ) to these nonlinear features . In this paper , tackle the same neural kernel similarity matching problem with a different approach . Instead of using random nonlinear features , we directly optimize for the features with Hebbian learning rules that resemble the learning rules derived in the original works on linear similarity matching . To derive our learning rules , we show that for any kernel we can upper bound the sum of squared errors |ys · yt − f ( xs , xt ) |2 with a correlation-based energy . Gradient-based optimization of our upper bound with will lead to a neural network with correlation-based learning rules . 2 CORRELATION-BASED BOUND FOR KERNEL SIMILARITY MATCHING . Roadmap for this section We first define the kernel similarity matching problem ( Eq . 1 ) . We then derive a correlation-based optimization ( Eq . 6 ) which is an upper bound for to Eq . 1 ( up to a constant that does not depend on the representations ) . We then use a Legendre transform to derive an equivalent ( except for the numerical stability parameter λ ) optimization problem in Eq . 9 that will lend itself towards online updates . Kernel similarity matching Assume we are given a set of input vectors { xt ∈ RM } Tt=1 and a positive semi-definite kernel function f : RM × RM → R which defines the similarity between input vectors . The goal is to find a corresponding set of representations { yt ∈ RN } Tt=1 such that for all pairs ( s , t ) we have ys ·yt ≈ f ( xs , xt ) . We will assume that T N > M : the dimensionalities of x , y are much lower than the number of samples T , but y are still higher dimensional than the inputs . This is formalized by minimizing the sum of squared errors : min { yt } 1 T 2 T∑ s , t [ f ( xs , xt ) − ys · yt ] 2 ( 1 ) This is known as the classical multidimensional scaling objective ( Borg & Groenen , 2005 ) . For arbitrary nonlinearity f this can be solved exactly by finding the top N eigenvectors of the T × T input similarity matrix ( Borg & Groenen , 2005 ) , and is therefore closely related to kernel PCA ( Schölkopf et al. , 1997 ) . However , this requires computing and storing similarities for all pairs of input vectors which breaks the online constraint that we require for biological realism . The purpose of this paper is to find an online algorithm , with correlation-based computations , that can at least approximately minimize Eq . 1 . Correlation based upper bound In this section we provide an upper bound to Eq . 1 which does not require computing f ( xs , xt ) for any ( s , t ) . The first step is to expand the square in Eq ( 1 ) to yield : 1 T 2 T∑ s , t [ f ( xs , xt ) − ys · yt ] 2 = − 2 T 2 ∑ s , t f ( xs , xt ) ys · yt + 1 T 2 ∑ s , t ( ys · yt ) 2 + const ( 2 ) We will now show how to bound the first term on the right hand side . Theorem 1 . If f is a positive semidefinite kernel function , then 1 2T 2 ∑ s , t ysytf ( xs , xt ) ≥ 1 T ∑ t qytf ( xt , w ) − 1 2 q2f ( w , w ) ( 3 ) for all q and w. Proof . Because f is a positive semi-definite kernel , we can assign to any set of M -dimensional vectors { w , x1 , . . . , xT } , a corresponding set of ( at most ) T+1-dimensional vectors { φw , φ1 , . . . , φT } whose inner products yield the similarity defined by f : φt · φt′ = f ( xt , xt′ ) φt · φw = f ( xt , w ) φw · φw = f ( w , w ) ( 4 ) Now consider the vector difference 1T ∑ t ytφt − qφw . The squared norm of this difference is of course non-negative . Additionally we can expand out this square : 0 ≤ 1 2 ∥∥∥∥∥ 1T ∑ t ytφt − qφw ∥∥∥∥∥ 2 = 1 2T 2 ∑ s , t ysytφs · φt − 1 T ∑ t qytφt · φw + 1 2 q2φw · φw ( 5 ) At this point we can simply replace all dot products with the equivalent nonlinear similarities f ( · , · ) in Equation 20 and rearrange the terms to yield our key inequality ( Eq . 3 ) . Our inequality still holds if we maximize the right hand side with respect to q and w. For every index i of y , we find the optimal wi , qi , and then replace the first pairwise sum in Eq . ( 2 ) with our upper bounds . Additionally we rearrange the order of the summations in second term on the right hand side of Eq . ( 2 ) to yield the following upper bound for the y-dependent terms in Eq . ( 2 ) : min yt min qi , wi − 1 T T∑ t=1 N∑ i=1 [ qiy t if ( wi , x t ) − 1 2 q2i f ( wi , wi ) ] + 1 4 N∑ i , j=1 ( 1 T T∑ t=1 ytiy t j ) 2 ( 6 ) Online focused reformulation We can further remove the square of the correlation matrix 1 T ∑ t y t iy t j ( another impediment to online learning ) by introducing a Legendre transformation : 1 2C 2 ij → maxLij CijLij − 12L 2 ij : min W , Y , q max L 1 T T∑ t=1 − N∑ i=1 [ qiy t if ( wi , x t ) − 1 2 q2i f ( wi , wi ) ] + 1 2 N∑ i , j=1 [ Lijy t iy t j − 1 2 L2ij ] ( 7 ) We can swap the order of the y and L optimizations , because the objective obeys the strong minmax property with W , q fixed ( Appendix Section A of Pehlevan et al . ( 2018 ) ) . We add one final term λNT ∑T t=1 ∑N i=1 ( y t i ) 2 to the objective , which can be important for numerical stability of our resulting algorithm . In our experiments λ = 0.001 . Finally , to better motivate our online algorithm , we define the “ per-sample-energy ” : et : = N∑ i=1 − [ qiy t if ( wi , x t ) − 1 2 q2i f ( wi , wi ) ] + 1 2 N∑ i , j=1 [ Lijy t iy t j − 1 2 L2ij ] + λ 2 N∑ i=1 ( yti ) 2 ( 8 ) where et : = e ( yt , xt ; W , q , L ) . The final optimization we will perform , which is equivalent to the optimization in Eq . 6 , and is derived as an upper bound to Eq . 1 , is thus : min W , q max L min Y 1 T T∑ t=1 e ( yt , xt ; W , q , L ) ( 9 ) 3 NEURAL NETWORK OPTIMIZATION . Applying a stochastic gradient descent-ascent algorithm to Eq . ( 9 ) yields a neural network ( Fig . 1 ) in which yti is the response of neuron i to input pattern t , wi is the vector of incoming connections to neuron i from the input layer , qi is a term which modulates the strength of these incoming connections , and Lij is a matrix of lateral recurrent connections between outputs . Specifically the neural algorithm procedes as follows . We initialize Wia ← N ( 0 , 1 ) , qi ← 1 and Lij ← Iij . At each iteration sample a minibatch of inputs { xb } . Using Eq.12 we compute the { yb } which minimize Eq . 9 for fixed synapses . Using these optimal { yb } , we compute the minibatch energy e = 1B ∑ b e ( x b , yb ; W , q , L ) and take a gradient descent step for q , a rescaled gradient descent step for w and a gradient ascent step for L : wi ← wi − ηw q2i ∂e ∂wi qi ← qi − ηq ∂e ∂qi Lij ← Lij + ηl ∂e ∂Lij ( 10 ) Convergence of the neural algorithm We treat convergence of this gradient descent ascent algorithm as an empirical issue . We adopt the ” two time scale ” strategy that has shown empirical successes for training generative adversarial networks ( Heusel et al. , 2017 ) . We choose the learning rates such that ηq , ηw ηl . Intuitively when choosing ηl to be large , the Lij can approximately maximize Eq . 9 for any particular q , W so that the min-max ordering is roughly preserved . In practice this ratio is important for convergence . We do not observe convergence when the ratios ηw/ηl or ηq/ηl are large . Unfortunately it is an empirical question of what is “ too large ” . If we could show that the objective were concave in L , it can be gradient descent ascent with smaller learning rates for W , q would indeed converge to a saddle point ( Lin et al. , 2020 ; Seung , 2019 ) . However , this question will have to be left for future work . Empirically it is sometimes observed that qi quickly shrinks to a small value early in training , which subsequently leads to small gradients for w. The rescaling of the wi updates provides an adaptive learning rate that appeared to improve training times in practice . We have attached the main portion of the training code , written using PyTorch , in the appendix .

Summary: This paper considers the problem of finding a low-rank approximation to a kernel matrix. In the low-rank approximation problem, one would want to find a low-rank matrix such that the Frobenius distance between the low-rank matrix and the original matrix is minimized. The paper shows that this optimization problem can be upper bounded by the cost function of a Hebbian neural network with a recurrent layer. Using some experiments the paper shows that this neural network can produce accurate low-rank approximations to some synthetic datasets as well as the MNIST dataset for the Gaussian and Arc-cosine kernels.