Convergence Analysis and Implicit Regularization of Feedback Alignment for Deep Linear Networks

1 INTRODUCTION . In order to train a Machine Learning ( ML ) architecture in the context of supervised learning , it is a consolidated practice to resort to the Gradient Descent ( GD ) algorithm and its variants ( e.g . stochastic GD ) and to calculate the gradient of the loss function via backpropagation ( Rumelhart et al. , 1986 ) . However , one of its limitations is the so-called weight transport problem ( Grossberg , 1987 ) : the update of each neuron during the learning phase relies on the downstream weights ( which are therefore `` transported '' along the backward pass ) . This implies that 1 ) the synaptic weights are the same , up to transposition , for both the forward ( inference ) and backward ( learning ) pass and 2 ) each neuron has a precise knowledge of all of its forward synapses . It is known that such a model does not accurately reflect the behaviour of a human brain , thus it is biologically implausible ( Crick , 1989 ) . Feedback Alignment ( FA ) , originally proposed by Lillicrap ( Lillicrap et al. , 2016 ) , has been an attempt to mitigate this discrepancy between the neuroscience side and the engineering side of AI and to tackle the weight transport problem in a systematic way . The simple , yet powerful idea is to replace the transposed weight matrices in the backpropagation algorithm with arbitrary , fixed matrices : via this strategy , each neuron during training receives a random projection of the error vector . In order to illustrate the algorithm and for the purpose of our study , let us consider a fully-connected L-layer neural network ( NN ) : given the input , x = h0 ∈ Rd , the predicted label ŷ ∈ Ro is computed as a ` = W ` h ` −1 , h ` = σ ( a ` ) , ` ∈ { 1 , . . . , L } : = [ L ] , ( 1 ) σ being a ( potentially ) nonlinear function applied entry-wise to the vector a ` , and finally ŷ = f ( aL ) for some output nonlinear function f . We consider a regression task with loss function L being the standard Mean Square Error ( MSE ) . The backpropagation strategy for GD prescribes the following updates for each layer in the network : given a step size η > 0 and the error vector e = δaL = ∂L∂ŷ = ŷ − y where y is the true label , we have δWL = −ηeh > L−1 , δW ` = −ηδa ` h > ` −1 and δa ` = ( W > ` +1δa ` +1 ) σ′ ( a ` ) , ` ∈ [ L−1 ] ( GD ) where denotes the Hadamard product . The FA recipe prescribes the following : Feedback Alignment : in ( GD ) use δa ` = ( M ` δa ` +1 ) σ′ ( a ` ) , ` ∈ [ L− 1 ] . ( 2 ) where { M ` } are a collection of arbitrarily chosen matrices , which do not evolve during training . Related and previous work . Inspired by Lillicrap et al . ( 2016 ) and shortly after its publication , Nøkland ( 2016 ) proposed two new algorithms called Direct and Inverse FA . In particular , Direct FA reduces to the FA algorithm in the case of a 2-layer NN or multi-layer linear NN ; the difference between these two algorithms in the general multi-layer nonlinear NN resides in injecting the error vector directly into the update of each hidden layer of the network : δa ` = ( M ` e ) σ′ ( a ` ) , ` ∈ [ 1 , L − 1 ] . Such a modification of the FA algorithm can be interpreted as a noisy version of layer-wise training ( Gilmer et al. , 2017 ) . Since its first formulation , there has been an extensive numerical analysis on testing whether FA and Direct FA algorithms can be successfully applied to ML problems , in particular in the presence of complex datasets and deep architectures ( Bartunov et al. , 2018 ; Moskovitz et al. , 2018 ; Launay et al. , 2019 ) . We mention the recent survey paper Launay et al . ( 2020 ) where the focus is on Direct FA and it is shown that the algorithm performs well in modern deep architectures ( Transformers and Graph NN , among others ) applied to problems like neural view synthesis or natural language processing . Furthermore , more on the applied ( biologically-inspired ) side , there have been recent pushes in furthering the idea of FA : spike-train level direct feedback alignment ( Lee et al. , 2020 ) , memory-efficient direct feedback alignment ( Chu et al. , 2020 ) and direct random target projection ( Frenkel et al. , 2021 ) to mention a few examples . On the other hand , rigorous theoretical results are scarce . Preliminary asymptotic results can be found in the original papers by Lillicrap et al . ( 2016 ) and Nøkland ( 2016 ) . More recently , Refinetti et al . ( 2020 ) presented the derivation of a system of differential equations that governs the Direct FA dynamics for a 2-layer non-linear NN in the regime where the input dimension goes to infinity . There is no explicit formula for the solution of the system , but via a careful local analysis it is possible to identify two separate phases of learning ( alignment and memorization ) and a `` degeneracy breaking '' effect : at the end of the FA training , the selected solution maximizes the overlap between the second weight matrix W2 and the FA matrix M . Such an `` alignment '' dynamics is also evident in our analysis ( Section 2 ) . Convergence guarantees for deep networks are still missing from the literature and one of the goals of this paper is indeed to provide rigorous proofs of convergence , together with convergence rates . Our analysis is in a similar vein as some recent works on implicit regularization in linear neural networks Saxe et al . ( 2018 ) ; Gidel et al . ( 2019 ) ; Arora et al . ( 2018 ) . Even though our setting is very similar , our analysis is different : because of the feedback alignment matrices , the autonomous differential equation obtained significantly differ and lead to different dynamics . Moreover our focus is slightly different from Saxe et al . ( 2018 ) ; Gidel et al . ( 2019 ) ; Arora et al . ( 2018 ) which are fully dedicated to implicit regularization of GD while we also aim at proving the convergence of FA . Our contributions . We aim to provide a theoretical understanding of how and when FA works . We focus on two aspects : 1 ) proving convergence of the algorithm and 2 ) analyzing implicit regularization phenomena . We will extensively study the case of deep linear NNs . Despite being sometimes regarded as too simplistic , the study of linear networks is a crucial starting point for a systematic analysis of the FA algorithm : it will allow us to rigorously analyze a tractable model and it will give us insights and intuition on the general nonlinear setting ( Section 7 ) . Our main results are the following . • We prove convergence of the FA continuous dynamics , with rates , for L-layer NNs with L ≥ 2 ( Sections 2 and 3 ) . Such results hold for any input dimension and number of neurons : the network is not necessarily overparametrized . • We prove that , for certain initialization schemes , the continuous FA dynamics sequentially learn the solutions of a reduced-rank regression problem , but in a backward fashion : smaller components first , dominant components later ( Section 4 ) . Additionally , we provide guidelines for an initialization scheme that avoids such phenomena . • Finally ( Section 5 ) , we analyze the corresponding discrete FA dynamics and we prove convergence to the true labels with linear convergence rates . We assumed some conditions on the data and on the model in order to render the exposition and the reading more clear . However , some of the `` strong '' assumptions that will appear in the paper can be easily relaxed ( see Section 2 ) . 2 WARM-UP : SHALLOW NETWORKS . In this section , we consider a 2-layer linear NN with vector output ŷ = W2W1x ∈ Ro and input vector x ∈ Rd such that ( x , y ) ∼ D for some distribution D. Define Σxx : = E [ xx > ] ∈ Rd×d , that we assume to be positive definite , and Σxy : = E [ yx > ] ∈ Ro×d , then the following result holds : Proposition 1 . For any distribution D , the FA dynamics that dictates the learning process is Ẇ1 = M ( Σxy −W2W1Σxx ) , Ẇ2 = ( Σxy −W2W1Σxx ) W > 1 , ( 3 ) where the dot refers to the time derivative . Note that this result particularly holds for empirical distributions corresponding to finite datasets . In this work , we will assume that the two matrices Σxx , Σxy can be simultaneously decomposed as Σxx = V ΛxxV > and Σxy = UΛxyV > , where the latter corresponds to the singular value decomposition ( SVD ) of Σxy . This assumption holds in many situations , as discussed in details by Gidel et al . ( 2019 ) . Moreover , a perturbation analysis similar as the one done in Gidel et al . ( 2019 , Theorem 1 ) could be performed in the general case in order to handle the non-commutative case . Next , we introduce the change of variablesW1 = RW̃1V > andW2 = UW̃2R > , for some arbitrary left-orthogonal matrixR , and we choose the FA matrixM to be decomposable in a similar fashion : M = RDU > , for some matrixD with positive diagonal entriesDii > 0 and zero entries otherwise . In particular , this implies that M is full rank , therefore guaranteeing convergence to a global minimum of the dynamics . The equations for the FA flow will then become ˙̃W1 = D ( Λxy − W̃2W̃1Λxx ) ˙̃W2 = ( Λxy − W̃2W̃1Λxx ) W̃ > 1 . ( 4 ) Moreover , by the change of variableW ′1 : = W̃1Λ −1/2 xx , W ′2 : = Λ −1/2 xx W̃2 , D′ : = DΛ −1/2 xx where Λ 1/2 xx is the coordinate-wise square root of Λxx,1 we get after some simple calculations that Ẇ ′1 = D ′ ( Λ1/2xx ΛxyΛ −1/2 xx −W ′2W ′1 ) , Ẇ ′2 = ( Λ1/2xx ΛxyΛ −1/2 xx −W ′2W ′1 ) W ′1 > . ( 5 ) Therefore , by considering Λ′xy : = Λ 1/2 xx ΛxyΛ −1/2 xx , we may equivalently assume Σxx = Id ( isotropic features ) , without any loss of generality . It is now crucial to observe that ifW ′1 ∈ Rh×d , W ′2 ∈ Ro×h are diagonal matrices , then the dynamics decouples into k independent equations with k = min { d , h , o } . In particular , let θi1 = ( W ′1 ) ii and θi2 = ( W ′ 2 ) ii and assume that at time t = 0 the matricesW ′ 1 ( 0 ) andW ′ 2 ( 0 ) are diagonal . Then , for each i = 1 , . . . , k we obtain the following scalar system : θ̇i1 = d i ( λi − θi2θi1 ) θ̇i2 = ( λi − θi2θi1 ) θi1 , ( 6 ) where λi = ( Λxy ) ii and di = Dii . Note that for i > k one could define diagonal coefficients for the matricesW ′1 orW ′ 2 , but their derivative will be 0 , thus non-trivial dynamics only occur for i ∈ [ k ] .

Summary: The authors study feedback alignment (FA, Lillicrap '14), an alternative algorithm to the standard backpropagation algorithm to train neural networks. The key idea is to replace the weight matrices of the network with fixed, random "feedback matrices" when computing the weight updates while training the network. A precise understanding how such a learning rule can lead to learning remains an open question, and has recently attracted some interest (see related works)  Here, the authors study the dynamics of learning with FA in (deep) linear neural networks. The dynamics of linear neural networks trained with backpropagation has been studied extensively, thus providing an ideal test bed for this study. The authors derive a set of continuous-time equations governing the dynamics of FA for linear networks, discussing in detail their assumptions etc. They use these equations to discover an interesting implicit bias of FA: for certain initialisations, features of the data are learnt in the *inverse* order of importance, quantified by the singular value associated with each mode of the data. For other initialisations, features are learnt in the order of importance, as expected. The sensitivity of this bias to the initialisation doesn't appear in backprop. Finally, the authors also study linear auto-encoders trained with FA, and find that not FA does not only recover (a rotation of) the eigenvectors, but that it also speeds up training significantly. They also study the discrete-step dynamics of FA.