Convergence Analysis and Implicit Regularization of Feedback Alignment for Deep Linear Networks

1 INTRODUCTION . In order to train a Machine Learning ( ML ) architecture in the context of supervised learning , it is a consolidated practice to resort to the Gradient Descent ( GD ) algorithm and its variants ( e.g . stochastic GD ) and to calculate the gradient of the loss function via backpropagation ( Rumelhart et al. , 1986 ) . However , one of its limitations is the so-called weight transport problem ( Grossberg , 1987 ) : the update of each neuron during the learning phase relies on the downstream weights ( which are therefore `` transported '' along the backward pass ) . This implies that 1 ) the synaptic weights are the same , up to transposition , for both the forward ( inference ) and backward ( learning ) pass and 2 ) each neuron has a precise knowledge of all of its forward synapses . It is known that such a model does not accurately reflect the behaviour of a human brain , thus it is biologically implausible ( Crick , 1989 ) . Feedback Alignment ( FA ) , originally proposed by Lillicrap ( Lillicrap et al. , 2016 ) , has been an attempt to mitigate this discrepancy between the neuroscience side and the engineering side of AI and to tackle the weight transport problem in a systematic way . The simple , yet powerful idea is to replace the transposed weight matrices in the backpropagation algorithm with arbitrary , fixed matrices : via this strategy , each neuron during training receives a random projection of the error vector . In order to illustrate the algorithm and for the purpose of our study , let us consider a fully-connected L-layer neural network ( NN ) : given the input , x = h0 ∈ Rd , the predicted label ŷ ∈ Ro is computed as a ` = W ` h ` −1 , h ` = σ ( a ` ) , ` ∈ { 1 , . . . , L } : = [ L ] , ( 1 ) σ being a ( potentially ) nonlinear function applied entry-wise to the vector a ` , and finally ŷ = f ( aL ) for some output nonlinear function f . We consider a regression task with loss function L being the standard Mean Square Error ( MSE ) . The backpropagation strategy for GD prescribes the following updates for each layer in the network : given a step size η > 0 and the error vector e = δaL = ∂L∂ŷ = ŷ − y where y is the true label , we have δWL = −ηeh > L−1 , δW ` = −ηδa ` h > ` −1 and δa ` = ( W > ` +1δa ` +1 ) σ′ ( a ` ) , ` ∈ [ L−1 ] ( GD ) where denotes the Hadamard product . The FA recipe prescribes the following : Feedback Alignment : in ( GD ) use δa ` = ( M ` δa ` +1 ) σ′ ( a ` ) , ` ∈ [ L− 1 ] . ( 2 ) where { M ` } are a collection of arbitrarily chosen matrices , which do not evolve during training . Related and previous work . Inspired by Lillicrap et al . ( 2016 ) and shortly after its publication , Nøkland ( 2016 ) proposed two new algorithms called Direct and Inverse FA . In particular , Direct FA reduces to the FA algorithm in the case of a 2-layer NN or multi-layer linear NN ; the difference between these two algorithms in the general multi-layer nonlinear NN resides in injecting the error vector directly into the update of each hidden layer of the network : δa ` = ( M ` e ) σ′ ( a ` ) , ` ∈ [ 1 , L − 1 ] . Such a modification of the FA algorithm can be interpreted as a noisy version of layer-wise training ( Gilmer et al. , 2017 ) . Since its first formulation , there has been an extensive numerical analysis on testing whether FA and Direct FA algorithms can be successfully applied to ML problems , in particular in the presence of complex datasets and deep architectures ( Bartunov et al. , 2018 ; Moskovitz et al. , 2018 ; Launay et al. , 2019 ) . We mention the recent survey paper Launay et al . ( 2020 ) where the focus is on Direct FA and it is shown that the algorithm performs well in modern deep architectures ( Transformers and Graph NN , among others ) applied to problems like neural view synthesis or natural language processing . Furthermore , more on the applied ( biologically-inspired ) side , there have been recent pushes in furthering the idea of FA : spike-train level direct feedback alignment ( Lee et al. , 2020 ) , memory-efficient direct feedback alignment ( Chu et al. , 2020 ) and direct random target projection ( Frenkel et al. , 2021 ) to mention a few examples . On the other hand , rigorous theoretical results are scarce . Preliminary asymptotic results can be found in the original papers by Lillicrap et al . ( 2016 ) and Nøkland ( 2016 ) . More recently , Refinetti et al . ( 2020 ) presented the derivation of a system of differential equations that governs the Direct FA dynamics for a 2-layer non-linear NN in the regime where the input dimension goes to infinity . There is no explicit formula for the solution of the system , but via a careful local analysis it is possible to identify two separate phases of learning ( alignment and memorization ) and a `` degeneracy breaking '' effect : at the end of the FA training , the selected solution maximizes the overlap between the second weight matrix W2 and the FA matrix M . Such an `` alignment '' dynamics is also evident in our analysis ( Section 2 ) . Convergence guarantees for deep networks are still missing from the literature and one of the goals of this paper is indeed to provide rigorous proofs of convergence , together with convergence rates . Our analysis is in a similar vein as some recent works on implicit regularization in linear neural networks Saxe et al . ( 2018 ) ; Gidel et al . ( 2019 ) ; Arora et al . ( 2018 ) . Even though our setting is very similar , our analysis is different : because of the feedback alignment matrices , the autonomous differential equation obtained significantly differ and lead to different dynamics . Moreover our focus is slightly different from Saxe et al . ( 2018 ) ; Gidel et al . ( 2019 ) ; Arora et al . ( 2018 ) which are fully dedicated to implicit regularization of GD while we also aim at proving the convergence of FA . Our contributions . We aim to provide a theoretical understanding of how and when FA works . We focus on two aspects : 1 ) proving convergence of the algorithm and 2 ) analyzing implicit regularization phenomena . We will extensively study the case of deep linear NNs . Despite being sometimes regarded as too simplistic , the study of linear networks is a crucial starting point for a systematic analysis of the FA algorithm : it will allow us to rigorously analyze a tractable model and it will give us insights and intuition on the general nonlinear setting ( Section 7 ) . Our main results are the following . • We prove convergence of the FA continuous dynamics , with rates , for L-layer NNs with L ≥ 2 ( Sections 2 and 3 ) . Such results hold for any input dimension and number of neurons : the network is not necessarily overparametrized . • We prove that , for certain initialization schemes , the continuous FA dynamics sequentially learn the solutions of a reduced-rank regression problem , but in a backward fashion : smaller components first , dominant components later ( Section 4 ) . Additionally , we provide guidelines for an initialization scheme that avoids such phenomena . • Finally ( Section 5 ) , we analyze the corresponding discrete FA dynamics and we prove convergence to the true labels with linear convergence rates . We assumed some conditions on the data and on the model in order to render the exposition and the reading more clear . However , some of the `` strong '' assumptions that will appear in the paper can be easily relaxed ( see Section 2 ) . 2 WARM-UP : SHALLOW NETWORKS . In this section , we consider a 2-layer linear NN with vector output ŷ = W2W1x ∈ Ro and input vector x ∈ Rd such that ( x , y ) ∼ D for some distribution D. Define Σxx : = E [ xx > ] ∈ Rd×d , that we assume to be positive definite , and Σxy : = E [ yx > ] ∈ Ro×d , then the following result holds : Proposition 1 . For any distribution D , the FA dynamics that dictates the learning process is Ẇ1 = M ( Σxy −W2W1Σxx ) , Ẇ2 = ( Σxy −W2W1Σxx ) W > 1 , ( 3 ) where the dot refers to the time derivative . Note that this result particularly holds for empirical distributions corresponding to finite datasets . In this work , we will assume that the two matrices Σxx , Σxy can be simultaneously decomposed as Σxx = V ΛxxV > and Σxy = UΛxyV > , where the latter corresponds to the singular value decomposition ( SVD ) of Σxy . This assumption holds in many situations , as discussed in details by Gidel et al . ( 2019 ) . Moreover , a perturbation analysis similar as the one done in Gidel et al . ( 2019 , Theorem 1 ) could be performed in the general case in order to handle the non-commutative case . Next , we introduce the change of variablesW1 = RW̃1V > andW2 = UW̃2R > , for some arbitrary left-orthogonal matrixR , and we choose the FA matrixM to be decomposable in a similar fashion : M = RDU > , for some matrixD with positive diagonal entriesDii > 0 and zero entries otherwise . In particular , this implies that M is full rank , therefore guaranteeing convergence to a global minimum of the dynamics . The equations for the FA flow will then become ˙̃W1 = D ( Λxy − W̃2W̃1Λxx ) ˙̃W2 = ( Λxy − W̃2W̃1Λxx ) W̃ > 1 . ( 4 ) Moreover , by the change of variableW ′1 : = W̃1Λ −1/2 xx , W ′2 : = Λ −1/2 xx W̃2 , D′ : = DΛ −1/2 xx where Λ 1/2 xx is the coordinate-wise square root of Λxx,1 we get after some simple calculations that Ẇ ′1 = D ′ ( Λ1/2xx ΛxyΛ −1/2 xx −W ′2W ′1 ) , Ẇ ′2 = ( Λ1/2xx ΛxyΛ −1/2 xx −W ′2W ′1 ) W ′1 > . ( 5 ) Therefore , by considering Λ′xy : = Λ 1/2 xx ΛxyΛ −1/2 xx , we may equivalently assume Σxx = Id ( isotropic features ) , without any loss of generality . It is now crucial to observe that ifW ′1 ∈ Rh×d , W ′2 ∈ Ro×h are diagonal matrices , then the dynamics decouples into k independent equations with k = min { d , h , o } . In particular , let θi1 = ( W ′1 ) ii and θi2 = ( W ′ 2 ) ii and assume that at time t = 0 the matricesW ′ 1 ( 0 ) andW ′ 2 ( 0 ) are diagonal . Then , for each i = 1 , . . . , k we obtain the following scalar system : θ̇i1 = d i ( λi − θi2θi1 ) θ̇i2 = ( λi − θi2θi1 ) θi1 , ( 6 ) where λi = ( Λxy ) ii and di = Dii . Note that for i > k one could define diagonal coefficients for the matricesW ′1 orW ′ 2 , but their derivative will be 0 , thus non-trivial dynamics only occur for i ∈ [ k ] .

Summary: This paper theoretically analyzes the Feedback Alignment (FA, Eqn 2) algorithm in the optimization of deep linear networks. * Under the assumption that the data matrices, FA matrices and initial weight matrices can be diagonalized simultaneously, the optimization can be divided into several one-dimensional problems, and the paper proves the convergence of both continuous and discrete dynamics. * After the diagonalization, each eigenvalue corresponds to a one-dimensional dynamics. The authors construct two types of initialization to show that the convergence for the larger eigenvalues can be either faster (implicit regularization) or slower (implicit anti-regularization) than the smaller eigenvalues. * Numerical comparison between FA and GD with 2- and 3-layer linear networks and random initialization. FA converges faster than GD and shares similar implicit regularization behavior.