Equilibrium Propagation with Continual Weight Updates

1 INTRODUCTION . A motivation for deep learning is that a few simple principles may explain animal intelligence and allow us to build intelligent machines , and learning paradigms must be at the heart of such principles , creating a synergy between neuroscience and Artificial Intelligence ( AI ) research . In the deep learning approach to AI ( LeCun et al. , 2015 ) , backpropagation thrives as the most powerful algorithm for training artificial neural networks . Unfortunately , its implementation on conventional computer or dedicated hardware consumes more energy than the brain by several orders of magnitude ( Strubell et al. , 2019 ) . One path towards reducing the gap between brains and machines in terms of power consumption is by investigating alternative learning paradigms relying on locally available information , which would allow radically different hardware implementations : such local learning rules could be used for the development of extremely energy efficient learning-capable hardware . Investigating such bioplausible learning schemes with real-world applicability is therefore of interest not only for neuroscience , but also for developing neuromorphic computing hardware that takes inspiration from information-encoding , dynamics and topology of the brain to reach fast and energy efficient AI ( Ambrogio et al. , 2018 ; Romera et al. , 2018 ) . In these regards , Equilibrium Propagation ( EP ) is an alternative style of computation for estimating error gradients that presents significant advantages ( Scellier and Bengio , 2017 ) . EP belongs to the family of contrastive Hebbian learning ( CHL ) algorithms ( Ackley et al. , 1985 ; Movellan , 1991 ; Hinton , 2002 ) and therefore benefits from an important feature of these algorithms : neural dynamics and synaptic updates depend solely on information that is locally available . As a CHL algorithm , EP applies to convergent RNNs , i.e . RNNs that are fed by a static input and converge to a steady state . Training such a convergent RNN consists in adjusting the weights so that the steady state corresponding to an input x produces output values close to associated targets y. CHL algorithms proceed in two phases : in the first phase , neurons evolve freely without external influence and settle to a ( first ) steady state ; in the second phase , the values of output neurons are influenced by the target y and the neurons settle to a second steady state . CHL weight updates consist in a Hebbian rule strengthening the connections between co-activated neurons at the first steady state , and an anti-Hebbian rule with opposite effect at the second steady state . A difference between Equilibrium Propagation and standard CHL algorithms is that output neurons are not clamped in the second phase but elastically pulled towards the target y . A second key property of EP is that , unlike CHL and other related algorithms , it is intimately linked to backpropagation . It has been shown that synaptic updates in EP follow gradients of recurrent backpropagation ( RBP ) ( Scellier and Bengio , 2019 ) and backpropagation through time ( BPTT ) ( Ernoult et al. , 2019 ) . This makes it especially attractive to bridge the gap between neural networks developed by neuroscientists , neuromorphic researchers and deep learning researchers . Nevertheless , the bioplausibility of EP still undergoes two major limitations . First , although EP is local in space , it is non-local in time . In all existing implementations of EP the weight update is performed after the dynamics of the second phase have converged , when the first steady state is no longer physically available . Thus the first steady state has to be artificially stored . Second , the network dynamics have to derive from a primitive function , which is equivalent to the requirement of symmetric weights in the Hopfield model . These two requirements are biologically unrealistic and also hinder the development of efficient EP computing hardware . In this work , we propose an alternative implementation of EP ( called C-EP ) which features temporal locality , by enabling synaptic dynamics to occur throughout the second phase , simultaneously with neural dynamics . We then address the second issue by adapting C-EP to systems having asymmetric synaptic connections , taking inspiration from Scellier et al . ( 2018 ) ; we call this modified version C-VF . More specifically , the contributions of the current paper are the following : • We introduce Continual Equilibrium Propagation ( C-EP , Section 3.1-3.2 ) , a new version of EP with continual weight updates : the weights of the network are adjusted continually in the second phase of training using local information in space and time . Neuron steady states do not need to be stored after the first phase , in contrast with standard EP where a global weight update is performed at the end of the second phase . Like standard EP , the C-EP algorithm applies to networks whose synaptic connections between neurons are assumed to be symmetric and tied . • We show mathematically that , provided that the changes in synaptic strengths are sufficiently slow ( i.e . the learning rates are sufficiently small ) , at each time step of the second phase the dynamics of neurons and synapses follow the gradients of the loss obtained with BPTT ( Theorem 1 and Fig . 2 , Section 3.3 ) . We call this property the Gradient Descending Dynamics ( GDD ) property , for consistency with the terminology used in Ernoult et al . ( 2019 ) . • We demonstrate training with C-EP on MNIST , with accuracy approaching the one obtained with standard EP ( Section 4.2 ) . • Finally , we adapt our C-EP algorithm to the more bio-realistic situation of a neural network with asymmetric connections between neurons . We call this modified version C-VF as it is inspired by the Vector Field method proposed in Scellier et al . ( 2018 ) . We demonstrate this approach on MNIST , and show numerically that the training performance is correlated with the satisfaction of Gradient Descending Dynamics ( Section 4.3 ) . For completeness , we also show how the Recurrent Backpropagation ( RBP ) algorithm of Almeida ( 1987 ) ; Pineda ( 1987 ) relates to C-EP , EP and BPTT . We illustrate the equivalence of these four algorithms on a simple analytical model ( Fig . 3 ) and we develop their relationship in Appendix A . 2 BACKGROUND : CONVERGENT RNNS AND EQUILIBRIUM PROPAGATION . Convergent RNNs With Static Input . We consider the supervised setting , where we want to predict a target y given an input x . The model is a recurrent neural network ( RNN ) parametrized by θ and evolving according to the dynamics : st+1 = F ( x , st , θ ) . ( 1 ) F is the transition function of the system . Assuming convergence of the dynamics before time step T , we have sT = s∗ where s∗ is the steady state of the network characterized by s∗ = F ( x , s∗ , θ ) . ( 2 ) The number of timesteps T is a hyperparameter that we choose large enough so that sT = s∗ for the current value of θ . The goal is to optimize the parameter θ in order to minimize a loss : L∗ = ` ( s∗ , y ) . ( 3 ) Algorithms that optimize the loss L∗ for RNNs include Backpropagation Through Time ( BPTT ) and the Recurrent Backpropagation ( RBP ) algorithm of Almeida ( 1987 ) ; Pineda ( 1987 ) , presented in Appendix B. Equilibrium Propagation ( EP ) . EP ( Scellier and Bengio , 2017 ) is a learning algorithm that computes the gradient of L∗ in the particular case where the transition function F derives from a scalar function Φ , i.e . with F of the form F ( x , s , θ ) = ∂Φ∂s ( x , s , θ ) . The algorithm consists in two phases ( see Alg . 1 of Fig . 1 ) . During the first phase , the network follows a sequence of states s1 , s2 , s3 . . . and converges to a steady state denoted s∗ . In the second phase , an extra term β ∂ ` ∂s pertubs the dynamics of the neurons ( where β > 0 is a scalar hyperparameter ) : starting from the steady state sβ0 = s∗ , the network follows a second sequence of states s β 1 , s β 2 , s β 3 . . . and converges to a new steady state denoted sβ∗ . Scellier and Bengio ( 2017 ) have shown that the gradient of the loss L∗ can be estimated based on the two steady states s∗ and sβ∗ . Specifically , in the limit β → 0 , 1 β ( ∂Φ ∂θ ( x , sβ∗ , θ ) − ∂Φ ∂θ ( x , s∗ , θ ) ) → −∂L ∗ ∂θ . ( 4 ) 3 EQUILIBRIUM PROPAGATION WITH CONTINUAL WEIGHT UPDATES ( C-EP ) . This section presents the main theoretical contributions of this paper . We introduce a new algorithm to optimize L∗ ( Eq . 3 ) : a new version of EP with continual parameter updates that we call C-EP . Unlike typical machine learning algorithms ( such as BPTT , RBP and EP ) in which the weight updates occur after all the other computations in the system are performed , our algorithm offers a mechanism in which the weights are updated continuously as the states of the neurons change . 3.1 FROM EP TO C-EP : AN INTUITION BEHIND CONTINUAL WEIGHT UPDATES . The key idea to understand how to go from EP to C-EP is that the gradient of EP appearing in Eq . ( 4 ) reads as the following telescopic sum : 1 β ( ∂Φ ∂θ ( x , sβ∗ , θ ) − ∂Φ ∂θ ( x , s∗ , θ ) ) ︸ ︷︷ ︸ global parameter gradient in EP = ∞∑ t=1 1 β ( ∂Φ ∂θ ( x , sβt , θ ) − ∂Φ ∂θ ( x , sβt−1 , θ ) ) ︸ ︷︷ ︸ parameter gradient at time t in C-EP . ( 5 ) In Eq . ( 5 ) we have used that sβ0 = s∗ and s β t → s β ∗ as t→∞ . Here lies the very intuition of continual updates motivating this work ; instead of keeping the weights fixed throughout the second phase and updating them at the end of the second phase based on the steady states s∗ and s β ∗ , as in EP ( Alg . 1 of Fig . 1 ) , the idea of the C-EP algorithm is to update the weights at each time t of the second phase between two consecutive states sβt−1 and s β t ( Alg . 2 of Fig . 1 ) . One key difference in C-EP compared to EP though , is that , in the second phase , the weight update at time step t influences the neural states at time step t+ 1 in a nontrivial way , as illustrated in the computational graph of Fig . 2 . In the next subsection we define C-EP using notations that explicitly show this dependency . Algorithm 1 EP Input : x , y , θ , β , η . Output : θ . 1 : s0 ← 0 . First Phase 2 : repeat 3 : st+1 ← ∂Φ∂s ( x , st , θ ) 4 : until st = s∗ 5 : Store s∗ 6 : sβ0 ← s∗ . Second Phase 7 : repeat 8 : sβt+1 ← ∂Φ∂s ( x , sβt , θ ) − β ∂ ` ∂s ( sβt , y ) 9 : until sβt = s β ∗ 10 : . Global Parameter Update 11 : θ ← θ + ηβ ( ∂Φ ∂θ ( sβ∗ , θ ) − ∂Φ∂θ ( s∗ , θ ) ) Algorithm 2 C-EP ( with simplified notations ) Input : x , y , θ , β , η . Output : θ . 1 : s0 ← 0 . First Phase 2 : repeat 3 : st+1 ← ∂Φ∂s ( x , st , θ ) 4 : until st = s∗ 5 : sβ0 ← s∗ . Second Phase 6 : repeat 7 : sβt+1 ← ∂Φ∂s ( x , sβt , θ ) − β ∂ ` ∂s ( sβt , y ) 8 : . Parameter Update at Time t 9 : θ ← θ + ηβ ( ∂Φ ∂θ ( sβt+1 ) − ∂Φ∂θ ( sβt ) ) 10 : until sβt and θ are converged . Figure 1 : Left . Pseudo-code of EP . This is the version of EP for discrete-time dynamics introduced in Ernoult et al . ( 2019 ) . Right . Pseudo-code of C-EP with simplified notations ( see section 3.2 for a formal definition of C-EP ) . Difference between EP and C-EP . In EP , one global parameter update is performed at the end of the second phase ; in C-EP , parameter updates are performed throughout the second phase . Eq . 5 shows that the continual updates of C-EP add up to the global update of EP .

Summary: The paper is concerned with biologically plausible models of learning. It takes equilibrium propagation -- where updates depend on local spatial information, in the sense that the information is available at each neuron -- and modifies the algorithm so updates are also local in time, thus obtaining C-EP. The key insight is that the updates in EP can be written as a telescoping sum over time points, eq (5).