Online approximate factorization of a kernel matrix by a Hebbian neural network

1 INTRODUCTION . Brain inspired learning algorithms have a long history in the field of neural networks and machine learning ( Rosenblatt , 1958 ; Olshausen & Field , 1996 ; Lee & Seung , 1999 ; Riesenhuber & Poggio , 1999 ; Hinton , 2007 ; Lillicrap et al. , 2016 ) . While many algorithms have diverged from their biological roots , the motivation to study biology remains clear : the human brain is such a powerful learning agent , there must be insights to be gained by making our algorithms look “ brain-like ” . This paper is focused on merging biological constraints with the well-established field of kernel-based machine learning . A common assumption in brain-inspired models of learning is that synaptic update rules should be a ) online , meaning the algorithm only has access to a single input pattern at a time and b ) local , meaning synapses should only be modified using information immediately available to the synapse , often just the pre- and post-firing rates of the neurons to which it is connected . Learning rules with these properties are commonly referred to as Hebbian learning rules ( Chklovskii , 2016 ) . Recent works have devised neural networks with Hebbian learning rules that perform linear similarity matching . These networks map every input xt to a representation yt such that linear output similarities match linear input similarities ys ·yt ≈ xs ·xt . These networks are interesting as models for real brains because they display a number of interesting biological properties : they are recurrent networks with correlation-based learning rules ( Pehlevan et al. , 2018 ) and can be modified to include non-negativity ( Pehlevan & Chklovskii , 2014 ) , sparsity , and convolutional structure ( Obeid et al. , 2019 ) . However there is a problem if one believes these networks should ultimately generate representations which are useful for downstream tasks . If similarities are actually matched , that is if ys ·yt = xs ·xt , then the outputs are simply an orthogonal transformation of the inputs , yt = Qxt , which is unlikely to have significant impact on downstream tasks . Bahroun et al . ( 2017 ) identified this problem and proposed a solution : instead of matching linear input similarities , one can match nonlinear input similarities : ys · yt ≈ f ( xs , xt ) . The authors provided a method that can be applied to any shiftinvariant kernel . They applied the random Fourier feature method of Rahimi et al . ( 2007 ) to map inputs to nonlinear feature vectors x→ ψ and then applied the linear similarity matching framework of Pehlevan et al . ( 2018 ) to these nonlinear features . In this paper , tackle the same neural kernel similarity matching problem with a different approach . Instead of using random nonlinear features , we directly optimize for the features with Hebbian learning rules that resemble the learning rules derived in the original works on linear similarity matching . To derive our learning rules , we show that for any kernel we can upper bound the sum of squared errors |ys · yt − f ( xs , xt ) |2 with a correlation-based energy . Gradient-based optimization of our upper bound with will lead to a neural network with correlation-based learning rules . 2 CORRELATION-BASED BOUND FOR KERNEL SIMILARITY MATCHING . Roadmap for this section We first define the kernel similarity matching problem ( Eq . 1 ) . We then derive a correlation-based optimization ( Eq . 6 ) which is an upper bound for to Eq . 1 ( up to a constant that does not depend on the representations ) . We then use a Legendre transform to derive an equivalent ( except for the numerical stability parameter λ ) optimization problem in Eq . 9 that will lend itself towards online updates . Kernel similarity matching Assume we are given a set of input vectors { xt ∈ RM } Tt=1 and a positive semi-definite kernel function f : RM × RM → R which defines the similarity between input vectors . The goal is to find a corresponding set of representations { yt ∈ RN } Tt=1 such that for all pairs ( s , t ) we have ys ·yt ≈ f ( xs , xt ) . We will assume that T N > M : the dimensionalities of x , y are much lower than the number of samples T , but y are still higher dimensional than the inputs . This is formalized by minimizing the sum of squared errors : min { yt } 1 T 2 T∑ s , t [ f ( xs , xt ) − ys · yt ] 2 ( 1 ) This is known as the classical multidimensional scaling objective ( Borg & Groenen , 2005 ) . For arbitrary nonlinearity f this can be solved exactly by finding the top N eigenvectors of the T × T input similarity matrix ( Borg & Groenen , 2005 ) , and is therefore closely related to kernel PCA ( Schölkopf et al. , 1997 ) . However , this requires computing and storing similarities for all pairs of input vectors which breaks the online constraint that we require for biological realism . The purpose of this paper is to find an online algorithm , with correlation-based computations , that can at least approximately minimize Eq . 1 . Correlation based upper bound In this section we provide an upper bound to Eq . 1 which does not require computing f ( xs , xt ) for any ( s , t ) . The first step is to expand the square in Eq ( 1 ) to yield : 1 T 2 T∑ s , t [ f ( xs , xt ) − ys · yt ] 2 = − 2 T 2 ∑ s , t f ( xs , xt ) ys · yt + 1 T 2 ∑ s , t ( ys · yt ) 2 + const ( 2 ) We will now show how to bound the first term on the right hand side . Theorem 1 . If f is a positive semidefinite kernel function , then 1 2T 2 ∑ s , t ysytf ( xs , xt ) ≥ 1 T ∑ t qytf ( xt , w ) − 1 2 q2f ( w , w ) ( 3 ) for all q and w. Proof . Because f is a positive semi-definite kernel , we can assign to any set of M -dimensional vectors { w , x1 , . . . , xT } , a corresponding set of ( at most ) T+1-dimensional vectors { φw , φ1 , . . . , φT } whose inner products yield the similarity defined by f : φt · φt′ = f ( xt , xt′ ) φt · φw = f ( xt , w ) φw · φw = f ( w , w ) ( 4 ) Now consider the vector difference 1T ∑ t ytφt − qφw . The squared norm of this difference is of course non-negative . Additionally we can expand out this square : 0 ≤ 1 2 ∥∥∥∥∥ 1T ∑ t ytφt − qφw ∥∥∥∥∥ 2 = 1 2T 2 ∑ s , t ysytφs · φt − 1 T ∑ t qytφt · φw + 1 2 q2φw · φw ( 5 ) At this point we can simply replace all dot products with the equivalent nonlinear similarities f ( · , · ) in Equation 20 and rearrange the terms to yield our key inequality ( Eq . 3 ) . Our inequality still holds if we maximize the right hand side with respect to q and w. For every index i of y , we find the optimal wi , qi , and then replace the first pairwise sum in Eq . ( 2 ) with our upper bounds . Additionally we rearrange the order of the summations in second term on the right hand side of Eq . ( 2 ) to yield the following upper bound for the y-dependent terms in Eq . ( 2 ) : min yt min qi , wi − 1 T T∑ t=1 N∑ i=1 [ qiy t if ( wi , x t ) − 1 2 q2i f ( wi , wi ) ] + 1 4 N∑ i , j=1 ( 1 T T∑ t=1 ytiy t j ) 2 ( 6 ) Online focused reformulation We can further remove the square of the correlation matrix 1 T ∑ t y t iy t j ( another impediment to online learning ) by introducing a Legendre transformation : 1 2C 2 ij → maxLij CijLij − 12L 2 ij : min W , Y , q max L 1 T T∑ t=1 − N∑ i=1 [ qiy t if ( wi , x t ) − 1 2 q2i f ( wi , wi ) ] + 1 2 N∑ i , j=1 [ Lijy t iy t j − 1 2 L2ij ] ( 7 ) We can swap the order of the y and L optimizations , because the objective obeys the strong minmax property with W , q fixed ( Appendix Section A of Pehlevan et al . ( 2018 ) ) . We add one final term λNT ∑T t=1 ∑N i=1 ( y t i ) 2 to the objective , which can be important for numerical stability of our resulting algorithm . In our experiments λ = 0.001 . Finally , to better motivate our online algorithm , we define the “ per-sample-energy ” : et : = N∑ i=1 − [ qiy t if ( wi , x t ) − 1 2 q2i f ( wi , wi ) ] + 1 2 N∑ i , j=1 [ Lijy t iy t j − 1 2 L2ij ] + λ 2 N∑ i=1 ( yti ) 2 ( 8 ) where et : = e ( yt , xt ; W , q , L ) . The final optimization we will perform , which is equivalent to the optimization in Eq . 6 , and is derived as an upper bound to Eq . 1 , is thus : min W , q max L min Y 1 T T∑ t=1 e ( yt , xt ; W , q , L ) ( 9 ) 3 NEURAL NETWORK OPTIMIZATION . Applying a stochastic gradient descent-ascent algorithm to Eq . ( 9 ) yields a neural network ( Fig . 1 ) in which yti is the response of neuron i to input pattern t , wi is the vector of incoming connections to neuron i from the input layer , qi is a term which modulates the strength of these incoming connections , and Lij is a matrix of lateral recurrent connections between outputs . Specifically the neural algorithm procedes as follows . We initialize Wia ← N ( 0 , 1 ) , qi ← 1 and Lij ← Iij . At each iteration sample a minibatch of inputs { xb } . Using Eq.12 we compute the { yb } which minimize Eq . 9 for fixed synapses . Using these optimal { yb } , we compute the minibatch energy e = 1B ∑ b e ( x b , yb ; W , q , L ) and take a gradient descent step for q , a rescaled gradient descent step for w and a gradient ascent step for L : wi ← wi − ηw q2i ∂e ∂wi qi ← qi − ηq ∂e ∂qi Lij ← Lij + ηl ∂e ∂Lij ( 10 ) Convergence of the neural algorithm We treat convergence of this gradient descent ascent algorithm as an empirical issue . We adopt the ” two time scale ” strategy that has shown empirical successes for training generative adversarial networks ( Heusel et al. , 2017 ) . We choose the learning rates such that ηq , ηw ηl . Intuitively when choosing ηl to be large , the Lij can approximately maximize Eq . 9 for any particular q , W so that the min-max ordering is roughly preserved . In practice this ratio is important for convergence . We do not observe convergence when the ratios ηw/ηl or ηq/ηl are large . Unfortunately it is an empirical question of what is “ too large ” . If we could show that the objective were concave in L , it can be gradient descent ascent with smaller learning rates for W , q would indeed converge to a saddle point ( Lin et al. , 2020 ; Seung , 2019 ) . However , this question will have to be left for future work . Empirically it is sometimes observed that qi quickly shrinks to a small value early in training , which subsequently leads to small gradients for w. The rescaling of the wi updates provides an adaptive learning rate that appeared to improve training times in practice . We have attached the main portion of the training code , written using PyTorch , in the appendix .

Summary: The paper describes an online stochastic optimization of a neural network to learn an embedding such that the inner-product between embedded points approximates a non-linear kernel based similarity. This can be a radial basis function network that approximates the embedding of a Gaussian kernel. It can also use a homogeneous kernel to approximate cosine-similarities. The network approach can be seen as a sort of Nyström approximation where each output dimension is associated with a different Nyström support point (landmark). The approximation cost is in terms of minimizing an upper bound on the squared error of the similarities. Experimental results show that the method is able to learn meaningful representations.