Credit Assignment in Neural Networks through Deep Feedback Control

1 Introduction . The error backpropagation ( BP ) algorithm [ 1 , 2 , 3 ] is currently the gold standard to perform credit assignment ( CA ) in deep neural networks . Although deep learning was inspired by biological neural networks , an exact mapping of BP onto biology to explain learning in the brain leads to several inconsistencies with experimental results that are not yet fully addressed [ 4 , 5 , 6 ] . First , BP requires an exact symmetry between the weights of the forward and feedback pathways [ 5 , 6 ] , also called the weight transport problem . Another issue of relevance is that , in biological networks , feedback also changes each neuron ’ s activation and thus its immediate output [ 7 , 8 ] , which does not occur in BP . Lillicrap et al . [ 9 ] convincingly showed that the weight transport problem can be sidestepped in modest supervised learning problems by using random feedback connections . However , follow-up studies indicated that random feedback paths can not provide precise CA in more complex problems [ 10 , 11 , 12 , 13 ] , which can be mitigated by learning feedback weights that align with the forward pathway [ 14 , 15 , 16 , 17 , 18 ] or approximate its inverse [ 19 , 20 , 21 , 22 ] . However , this precise alignment imposes strict constraints on the feedback weights , whereas more flexible constraints could provide the freedom to use feedback also for other purposes besides learning , such as attention and prediction [ 8 ] . A complementary line of research proposes models of cortical microcircuits which propagate CA signals through the network using dynamic feedback [ 23 , 24 , 25 ] or multiplexed neural codes [ 26 ] , thereby directly influencing neural activations with feedback . However , these models introduce ∗Equal contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . highly specific connectivity motifs and tightly coordinated plasticity mechanisms . Whether these constraints can be fulfilled by cortical networks is an interesting experimental question . Another line of work uses adaptive control theory [ 27 ] to derive learning rules for non-hierarchical recurrent neural networks ( RNNs ) based on error feedback , which drives neural activity to track a reference output [ 28 , 29 , 30 , 31 ] . These methods have so far only been used to train single-layer RNNs with fixed output and feedback weights , making it unclear whether they can be extended to deep neural networks . Finally , two recent studies [ 32 , 33 ] use error feedback in a dynamical setting to invert the forward pathway , thereby enabling errors to flow backward . These approaches rely on a learning rule that is non-local in time and it remains unclear whether they approximate any known optimization method . Addressing the latter , two recent studies take a first step by relating learned ( non-dynamical ) inverses of the forward pathway [ 21 ] and iterative inverses restricted to invertible networks [ 22 ] to approximate Gauss-Newton optimization . Inspired by the Dynamic Inversion method [ 32 ] , we introduce Deep Feedback Control ( DFC ) , a new biologically-plausible CA method that addresses the above-mentioned limitations and extends the control theory approach to learning [ 28 , 29 , 30 , 31 ] to deep neural networks . DFC uses a feedback controller that drives a deep neural network to match a desired output target . For learning , DFC then simply uses the dynamic change in the neuron activations to update their synaptic weights , resulting in a learning rule fully local in space and time . We show that DFC approximates Gauss-Newton ( GN ) optimization and therefore provides a fundamentally different approach to CA compared to BP . Furthermore , DFC does not require precise alignment between forward and feedback weights , nor does it rely on highly specific connectivity motifs . Interestingly , the neuron model used by DFC can be closely connected to recent multi-compartment models of cortical pyramidal neurons . Finally , we provide detailed experimental results , corroborating our theoretical contributions and showing that DFC does principled CA on standard computer-vision benchmarks in a way that fundamentally differs from standard BP . 2 The Deep Feedback Control method . Here , we introduce the core parts of DFC . In contrast to conventional feedforward neural network models , DFC makes use of a dynamical neuron model ( Section 2.1 ) . We use a feedback controller to drive the neurons of the network to match a desired output target ( Section 2.2 ) , while simultaneously updating the synaptic weights using the change in neuronal activities ( Section 2.3 ) . This combination of dynamical neurons and controller leads to a simple but powerful learning method , that is linked to GN optimization and offers a flexible range of feedback connectivity ( see Section 3 ) . 2.1 Neuron and network dynamics . The first main component of DFC is a dynamical multilayer network , in which every neuron integrates its forward and feedback inputs according to the following dynamics : τv d dt vi ( t ) = −vi ( t ) +Wiϕ ( vi−1 ( t ) ) +Qiu ( t ) 1 ≤ i ≤ L , ( 1 ) with vi a vector containing the pre-nonlinearity activations of the neurons in layer i , Wi the forward weight matrix , ϕ a smooth nonlinearity , u a feedback input , Qi the feedback weight matrix , and τv a time constant . See Fig . 1B for a schematic representation of the network . To simplify notation , we define ri = ϕ ( vi ) as the post-nonlinearity activations of layer i . The input r0 remains fixed throughout the dynamics ( 1 ) . Note that in the absence of feedback , i.e. , u = 0 , the equilibrium state of the network dynamics ( 1 ) corresponds to a conventional multilayer feedforward network state , which we denote with superscript ‘ - ’ : r−i = ϕ ( v − i ) = ϕ ( Wir − i−1 ) , 1 ≤ i ≤ L , with r−0 = r0 . ( 2 ) 2.2 Feedback controller . The second core component of DFC is a feedback controller , which is only active during learning . Instead of a single backward pass for providing feedback , DFC uses a feedback controller to continuously drive the network to an output target r∗L ( see Fig . 1D ) . Following the Target Propagation framework [ 20 , 21 , 22 ] , we define r∗L as the feedforward output nudged towards lower loss : r∗L ≜ r−L − λ ∂L ( rL , y ) ∂rL ∣∣∣ rL=r − L = r−L + δL , ( 3 ) with L ( rL , y ) a supervised loss function defining the task , y the label of the training sample , λ a stepsize , and δL shorthand notation . Note that ( 3 ) only needs the easily obtained loss gradient w.r.t . the output , e.g. , for an L2 output loss , one obtains the convex combination r∗L = ( 1− 2λ ) r−L + 2λy . The feedback controller produces a feedback signal u ( t ) to drive the network output rL ( t ) towards its target r∗L , using the control error e ( t ) ≜ r∗L − rL ( t ) . A standard approach in designing a feedback controller is the Proportional-Integral-Derivative ( PID ) framework [ 34 ] . While DFC is compatible with various controller types , such as a full PID controller or a pure proportional controller ( see Appendix A.8 ) , we use a PI controller for a combination of simplicity and good performance , resulting in the following controller dynamics ( see also Fig . 1A ) : u ( t ) = KIu int ( t ) +KPe ( t ) , τu d dt uint ( t ) = e ( t ) − αuint ( t ) , ( 4 ) where a leakage term is added to constrain the magnitude of uint . For mathematical simplicity , we take the control matrices equal to KI = I and KP = kpI with kp ≥ 0 the proportional control constant . This PI controller adds a leaky integration of the error uint to a scaled version of the error kpe which could be implemented by a dedicated neural microcircuit ( for a discussion see App . I ) . Drawing inspiration from the Target Propagation framework [ 19 , 20 , 21 , 22 ] and the Dynamic Inversion framework [ 32 ] , one can think of the controller and network dynamics as performing a dynamic inversion of the output target r∗L towards the hidden layers , as the controller dynamically changes the activation of the hidden layers until the output target is reached . 2.3 Forward weight updates . The update rule for the feedforward weights has the form : τW d dt Wi ( t ) = ( ϕ ( vi ( t ) ) − ϕ ( Wiri−1 ( t ) ) ) ri−1 ( t ) T . ( 5 ) This learning rule simply compares the neuron ’ s controlled activation to its current feedforward input and is thus local in space and time . Furthermore , it can be interpreted most naturally by compartmentalizing the neuron into the central compartment vi from ( 1 ) and a feedforward compartment vffi ≜ Wiri−1 that integrates the feedforward input . Now , the forward weight dynamics ( 5 ) represents a delta rule using the difference between the actual firing rate of the neuron , ϕ ( vi ) , and its estimated firing rate , ϕ ( vffi ) , based on the feedforward inputs . Note that we assume τW to be a large time constant , such that the network ( 1 ) and controller dynamics ( 4 ) are not influenced by the weight dynamics , i.e. , the weights are considered fixed in the timescale of the controller and network dynamics . In Section 5 , we show how the feedback weights Qi can also be learned locally in time and space for supporting the stability of the network dynamics and the learning of Wi . This feedback learning rule needs a feedback compartment vfbi ≜ Qiu , leading to the three-compartment neuron schematized in Fig . 1C , inspired by recent multi-compartment models of the pyramidal neuron ( see Discussion ) . Now , that we introduced the DFC model , we will show that ( i ) the weight updates ( 5 ) can properly optimize a loss function ( Section 3 ) , ( ii ) the resulting dynamical system is stable under certain conditions ( Section 4 ) , and ( iii ) learning the feedback weights facilitates ( i ) and ( ii ) ( Section 5 ) . 3 Learning theory . To understand how DFC optimizes the feedforward mapping ( 2 ) on a given loss function , we link the weight updates ( 5 ) to mathematical optimization theory . We start by showing that DFC dynamically inverts the output error to the hidden layers ( Section 3.1 ) , which we link to GN optimization under flexible constraints on the feedback weights Qi and on layer activations ( Section 3.2 ) . In Section 3.3 , we relax some of these constraints , and show that DFC still does principled optimization by using minimum norm ( MN ) updates for Wi . During this learning theory section , we assume stable dynamics , which we investigate in more detail in Section 4 . All theoretical results of this section are tailored towards a PI controller , and they can be easily extended to pure proportional or integral control ( see App . A.8 ) .

Summary: The authors introduce Deep Feedback Control, an algorithm for training deep neural networks where credit assignment is done with the help of a proportional-integral-derivative (PID or rather PI in this case) controller.   Theoretically, the authors show that a learning rule similar to what they propose (but not the same), under certain assumptions, gives the Gauss-Newton update for an L2 loss, and under milder assumptions gives the minimum norm updates of W. The authors show that the PI controller helps with stability and perform some experiments to show that the assumptions remain more or less valid during training.  ### After rebuttal  Because of the increased quality of the work stemming from more theoretical arguments and more experiments, I have increased my score from 5 to 8.