Learning to solve the credit assignment problem

1 INTRODUCTION . It is unknown how the brain solves the credit assignment problem when learning : how does each neuron know its role in a positive ( or negative ) outcome , and thus know how to change its activity to perform better next time ? This is a challenge for models of learning in the brain . Biologically plausible solutions to credit assignment include those based on reinforcement learning ( RL ) algorithms and reward-modulated STDP ( Bouvier et al. , 2016 ; Fiete et al. , 2007 ; Fiete & Seung , 2006 ; Legenstein et al. , 2010 ; Miconi , 2017 ) . In these approaches a globally distributed reward signal provides feedback to all neurons in a network . Essentially , changes in rewards from a baseline , or expected , level are correlated with noise in neural activity , allowing a stochastic approximation of the gradient to be computed . However these methods have not been demonstrated to operate at scale . For instance , variance in the REINFORCE estimator ( Williams , 1992 ) scales with the number of units in the network ( Rezende et al. , 2014 ) . This drives the hypothesis that learning in the brain must rely on additional structures beyond a global reward signal . In artificial neural networks ( ANNs ) , credit assignment is performed with gradient-based methods computed through backpropagation ( Rumelhart et al. , 1986 ; Werbos , 1982 ; Linnainmaa , 1976 ) . This is significantly more efficient than RL-based algorithms , with ANNs now matching or surpassing human-level performance in a number of domains ( Mnih et al. , 2015 ; Silver et al. , 2017 ; LeCun et al. , 2015 ; He et al. , 2015 ; Haenssle et al. , 2018 ; Russakovsky et al. , 2015 ) . However there are well known problems with implementing backpropagation in biologically realistic neural networks . One problem is known as weight transport ( Grossberg , 1987 ) : an exact implementation of backpropagation requires a feedback structure with the same weights as the feedforward network to communicate gradients . Such a symmetric feedback structure has not been observed in biological neural circuits . Despite such issues , backpropagation is the only method known to solve supervised and reinforcement learning problems at scale . Thus modifications or approximations to backpropagation that are more plausible have been the focus of significant recent attention ( Scellier & Bengio , 2016 ; Lillicrap et al. , 2016 ; Lee et al. , 2015 ; Lansdell & Kording , 2018 ; Ororbia et al. , 2018 ) . These efforts do show some ways forward . Synthetic gradients demonstrate that learning can be based on approximate gradients , and need not be temporally locked ( Jaderberg et al. , 2016 ; Czarnecki et al. , 2017b ) . In small feedforward networks , somewhat surprisingly , fixed random feedback matrices in fact suffice for learning ( Lillicrap et al. , 2016 ) ( a phenomenon known as feedback alignment ) . But still issues remain : feedback alignment does not work in CNNs , very deep networks , or networks with tight bottleneck layers . Regardless , these results show that rough approximations of a gradient signal can be used to learn ; even relatively inefficient methods of approximating the gradient may be good enough . On this basis , here we propose an RL algorithm to train a feedback system to enable learning . Recent work has explored similar ideas , but not with the explicit goal of approximating backpropagation ( Miconi , 2017 ; Miconi et al. , 2018 ; Song et al. , 2017 ) . RL-based methods like REINFORCE may be inefficient when used as a base learner , but they may be sufficient when used to train a system that itself instructs a base learner . We propose to use REINFORCE-style perturbation approach to train feedback signals to approximate what would have been provided by backpropagation . This sort of two-learner system , where one network helps the other learn more efficiently , may in fact align well with cortical neuron physiology . For instance , the dendritic trees of pyramidal neurons consist of an apical and basal component . Such a setup has been shown to support supervised learning in feedforward networks ( Guergiuev et al. , 2017 ; Kording & Konig , 2001 ) . Similarly , climbing fibers and Purkinje cells may define a learner/teacher system in the cerebellum ( Marr , 1969 ) . These components allow for independent integration of two different signals , and may thus provide a realistic solution to the credit assignment problem . Thus we implement a network that learns to use feedback signals trained with reinforcement learning via a global reward signal . We mathematically analyze the model , and compare its capabilities to other methods for learning in ANNs . We prove consistency of the estimator in particular cases , extending the theory of synthetic gradient-like approaches ( Jaderberg et al. , 2016 ; Czarnecki et al. , 2017b ; Werbos , 1992 ; Schmidhuber , 1990 ) . We demonstrate that our model learns as well as regular backpropagation in small models , overcomes the limitations of feedback alignment on more complicated feedforward networks , and can be used in convolutional networks . Thus , by combining local and global feedback signals , this method points to more plausible ways the brain could solve the credit assignment problem . 2 LEARNING FEEDBACK WEIGHTS THROUGH PERTURBATIONS . We use the following notation . Let x ∈ Rm represent an input vector . Let an N hidden-layer network be given by ŷ = f ( x ) ∈ Rp . This is composed of a set of layer-wise summation and non-linear activations hi = f i ( hi−1 ) = σ ( W ihi−1 ) , for hidden layer states hi ∈ Rni , non-linearity σ , weight matrices W i ∈ Rni×ni−1 and denoting h0 = x and hN+1 = ŷ . Some loss function L is defined in terms of the network output : L ( y , ŷ ) . Let L denote the loss as a function of ( x , y ) : L ( x , y ) = L ( y , f ( x ) ) . Let data ( x , y ) ∈ D be drawn from a distribution ρ . We aim to minimize : Eρ [ L ( x , y ) ] . Backpropagation relies on the error signal ei , computed in a top-down fashion : ei = { ∂L/∂ŷ ◦ σ′ ( W ihi−1 ) , i = N + 1 ; ( ( W i+1 ) Tei+1 ) ◦ σ′ ( W ihi−1 ) , 1 ≤ i ≤ N , where ◦ denotes element-wise multiplication . 2.1 BASIC SETUP . Let the loss gradient term be denoted as λi = ∂L ∂hi = ( W i+1 ) Tei+1 . In this work we replace λi with an approximation with its own parameters to be learned ( known as a synthetic gradient , or conspiring network , ( Jaderberg et al. , 2016 ; Czarnecki et al. , 2017b ) , or error critic ( Werbos , 1992 ) ) : λi ≈ g ( hi , ẽi+1 ; θ ) , for parameters θ . Note that we must distinguish the true loss gradients from their synthetic estimates . Let ẽi be loss gradients computed by backpropagating the synthetic gradients ẽi = { ∂L/∂ŷ ◦ σ′ ( W ihi−1 ) , i = N + 1 ; g ( hi , ẽi+1 ; θ ) ◦ σ′ ( W ihi−1 ) , 1 ≤ i ≤ N . For the final layer the synthetic gradient matches the true gradient : eN+1 = ẽN+1 . This setup can accommodate both top-down and bottom-up information , and encompasses a number of published models ( Jaderberg et al. , 2016 ; Czarnecki et al. , 2017b ; Lillicrap et al. , 2016 ; Nøkland , 2016 ; Liao et al. , 2016 ; Xiao et al. , 2018 ) . 2.2 STOCHASTIC NETWORKS AND GRADIENT DESCENT . To learn a synthetic gradient we utilze the stochasticity inherent to biological neural networks . A number of biologically plausible learning rules exploit random perturbations in neural activity ( Xie & Seung , 2004 ; Seung , 2003 ; Fiete & Seung , 2006 ; Fiete et al. , 2007 ; Song et al. , 2017 ) . Here , at each time each unit produces a noisy response : hit = σ ( ∑ k W i·kh i−1 t ) + chξ i t , for independent Gaussian noise ξi ∼ ν = N ( 0 , I ) and standard deviation ch > 0 . This generates a noisy loss L̃ ( x , y , ξ ) and a baseline loss L ( x , y ) = L̃ ( x , y , 0 ) . We will use the noisy response to estimate gradients that then allow us to optimize the baseline L – the gradients used for weight updates are computed using the deterministic baseline . 2.3 SYNTHETIC GRADIENTS VIA PERTURBATION . For Gaussian white noise , the well-known REINFORCE algorithm ( Williams , 1992 ) coincides with the node perturbation method ( Fiete & Seung , 2006 ; Fiete et al. , 2007 ) . Node perturbation works by linearizing the loss : L̃ ≈ L+ ∂L ∂hij chξ i j , ( 1 ) such that E ( ( L̃ − L ) chξij |x , y ) ≈ c2h ∂L ∂hij ∣∣∣∣ x , y , with expectation taken over the noise distribution ν ( ξ ) . This provides an estimator of the loss gradient λ̂i : = ( L̃ ( x , y , ξ ) − L ( x , y ) ) ξ i ch . ( 2 ) This approximation is made more precise in Theorem 1 ( Supplementary material ) . 2.4 TRAINING A FEEDBACK NETWORK . There are many possible sensible choices of g ( · ) . For example , taking g as simply a function of each layer ’ s activations : λi = g ( hi ) is in fact sufficient parameterization to express the true gradient function ( Jaderberg et al. , 2016 ) . We may expect , however , that the gradient estimation problem be simpler if each layer is provided with some error information obtained from the loss function and propagated in a top-down fashion . Symmetric feedback weights may not be biologically plausible , and random fixed weights may only solve certain problems of limited size or complexity ( Lillicrap et al. , 2016 ) . However , a system that can learn to appropriate feedback weights B may be able to align the feedforward and feedback weights as much as is needed to successfully learn . We investigate various choices of g ( hi , ẽi+1 ; Bi+1 ) outlined in the applications below . Parameters Bi+1 are estimated by solving the least squares problem : B̂i+1 = arg min B E ∥∥∥g ( hi , ẽi+1 ; B ) − λ̂i∥∥∥2 2 . ( 3 ) Unless otherwise noted this was solved by gradient-descent , updating parameters once with each minibatch . Refer to the supplementary material for additional experimental descriptions and parameters . 3 THEORETICAL RESULTS . We can prove the estimator ( 3 ) is consistent as the noise variance ch → 0 , in some particular cases . We state the results informally here , and give the exact details in the supplementary materials . Consider first convergence of the final layer feedback matrix , BN+1 . Theorem 1 . ( Informal ) For gFA ( hi , ẽi+1 ; Bi+1 ) = Bi+1ẽi+1 , then the least squares estimator ( B̂N+1 ) T : = λ̂N ( eN+1 ) T ( eN+1 ( eN+1 ) T ) −1 , ( 4 ) solves ( 3 ) and converges to the true feedback matrix , in the sense that : limch→0 plimT→∞ B̂ N+1 = WN+1 , where plim indicates convergence in probability . Theorem 1 thus establishes convergence of B in a shallow ( 1 hidden layer ) non-linear network . In a deep , linear network we can also use Theorem 1 to establish convergence over the rest of the layers . Theorem 2 . ( Informal ) For gFA ( hi , ẽi+1 ; Bi+1 ) = Bi+1ẽi+1 and σ ( x ) = x , the least squares estimator ( B̂i ) T : = λ̂i−1 ( ẽi ) T ( ẽi ( ẽi ) T ) −1 1 ≤ i ≤ N + 1 , ( 5 ) solves ( 3 ) and converges to the true feedback matrix , in the sense that : limch→0 plimT→∞ B̂ i = W i , 1 ≤ i ≤ N + 1 . Given these results we can establish consistency for the ‘ direct feedback alignment ’ ( DFA ; Nøkland ( 2016 ) ) estimator : gDFA ( hi , ẽN+1 ; Bi+1 ) = ( Bi+1 ) TẽN+1 . Theorem 1 applies trivially since for the final layer , the two approximations have the same form : gFA ( hN , ẽN+1 ; θN ) = gDFA ( h N , ẽN+1 ; θN ) . Theorem 2 can be easily extended according to the following : Corollary 1 . ( Informal ) For gDFA ( hi , ẽN+1 ; Bi+1 ) = Bi+1ẽN+1 and σ ( x ) = x , the least squares estimator ( B̂i ) T : = λ̂i−1 ( ẽN+1 ) T ( ẽN+1 ( ẽN+1 ) T ) −1 1 ≤ n ≤ N + 1 , ( 6 ) solves ( 3 ) and converges to the true feedback matrix , in the sense that : limch→0 plimT→∞ B̂ i =∏i j=N+1W j , 1 ≤ i ≤ N + 1 . Thus for a non-linear shallow network or a deep linear network , for both gFA and gDFA , we have the result that , for sufficiently small ch , if we fix the network weights W and train B through node perturbation then we converge to W . Validation that the method learns to approximate W , for fixed W , is provided in the supplementary material . In practice , we update B and W simultaneously . Some convergence theory is established for this case in ( Jaderberg et al. , 2016 ; Czarnecki et al. , 2017b ) .

Summary: It's unclear how multi-layer biological neural networks could implement gradient-based learning, as they don't have the symmetric connections needed for backpropagation. This paper proposes a perturbation-based synthetic gradient estimator that does not rely on symmetric backward connections. Hidden unit perturbation is used to estimate the loss gradient, and backward connections are trained via gradient descent to predict the approximate gradients from the perturbation-based estimator.