Dyadic Learning in Recurrent and Feedforward Models

Venue: NeurIPS_cc_2024_Workshop_MLNCP
Published: 2024-09-12 13:20:41

Abstract: From electrical to biological circuits, feedback plays a critical role in amplifying,
dampening and stabilizing signals. In local activity difference based alternatives to
backpropagation, feedback connections are used to propagate learning signals in
deep neural networks. We propose a saddle-point based framework using dyadic
(two-state) neurons for training a family of parameterized models, which include
the symmetric Hopfield model, pure feedforward networks and a less explored
skew-symmetric Hopfield variant. The resulting learning method reduces to equi-
librium propagation (EP) for symmetric Hopfield models and to dual propagation
(DP) for feedforward networks, while the skew-symmetric Hopfield setting yields
a new method with desirable robustness properties. Experimentally we demon-
strate that the new skew-symmetric Hopfield model performs on par with EP and
DP in terms of the resulting model predictive performance, while exhibiting en-
hanced robustness to input changes and strong feedback and is less inclined to
neural saturation. We identify the fundamentally different types of feedback sig-
nals propagated in each model as the main cause of differences in robustness and
saturation.

Main paper: 
 introduction: 
 It has been proposed that the brain may perform some instance of approximate backpropagation, using neural activity differences to represent error signals [1, 2] . Activity difference based approaches come in two main flavours: (i) using temporal differences between activities at different times to represent error signals [3, 4, 5] , or (ii) using spatial differences by subtracting activities of distinct neurons or neuronal compartments [6, 7, 8] . Temporal activity difference methods require at least two relaxations to equilibrium, or infinitely many through slow adiabatic oscillations of the same circuit [9] , yet at the expense of latency. Conversely, spatial activity difference methods require neurons to multiplex by propagating activities and finite difference error signals simultaneously and some degree of weight sharing, which may be problematic from a biological perspective.In this manuscript, we take a closer look at the temporal activity difference based algorithm equilibrium propagation (EP) [4] for symmetric Hopfield models (HM), the spatial activity difference based algorithm dual propagation (DP) for feedforward models (FFM) [8] , and the fundamental role feedback plays in each of the algorithms. In the context of this work, feedback refers to both the error signal propagated in a supervised learning setting and to top-down connections in a layered network architecture (and therefore introducing cycles in the underlying computational graph). In DP, the feedback signals reduce to top-down (finite difference) errors due to the lack of recurrent connections in feedforward models. While this is an asset for inference speed in practice, recurrent connections are a key feature underpinning attention and robustness in biological systems. On the other hand, in EP (and Hopfield nets in general) positive feedback is always present: the sign of the signal send from neuron A to neuron B always matches the sign of the signal sent from B to A. This is a consequence of the symmetry of the underlying Hopfield model (W ij = W ji ). Positive feedback connections in symmetric Hopfield nets yield many desirable properties, but may also lead to excessive amplification of neural activities. This is not too surprising as high sensitivity to input changes and excessive saturation are known features of positive feedback [10] .As FFMs without an explicit error signal lack any kind of feedback from downstream layers and HMs suffer from the aforementioned limitations of persisting positive feedback, it is natural to ask what a negative feedback model would look and behave like? As a symmetric Hopfield model results in positive feedback, a skew-symmetric Hopfield model (SSHM), with with W ij = -W ji , would instead provide negative feedback. Negative feedback loops are typically used to decrease sensitivity to input perturbations and increase the linearity of a system [10] which could respectively endow the resulting model with a degree of biological plausibility and alleviate vanishing gradient issues during training.We establish a family of models and their associated learning dynamics, which recover as special cases symmetric Hopfield models (HM), feedforward models (FFM) and skew-symmetric Hopfield models (SSHM). In the HM setting, the neural state and synaptic weight dynamics correspond to EP, and the FFM setting results in DP. The SSHM scenario yields a two-phased algorithm which exhibits reduced sensitivity to input perturbations and also to decreased neural saturation.While the role of feedback in the brain is much more diverse and subject to constraints we are not modelling (such as weight transport and, Dale's principle) we hope this study helps bring attention to the importance of negative feedback in the context of learning. The importance of negative feedback is well known in both neuroscience and control theory, yet in the context of biologically inspired learning algorithms negative feedback has received little to no attention. 
related work: 
 Contrastive learning algorithms. In energy-based models, inference amounts to energy minimization. Training can in this setting be achieved by contrasting the states inferred when applying different perturbations to the output units. In variations of contrastive Hebbian learning (CHL) [3, 11, 5] , this is done by clamping output units. In EP, the output units are instead weakly nudged by a teaching signal derived from a loss function. EP and CHL typically assume a symmetric connectivity, which for instance naturally map to resistive networks where top-down currents inherently flow through the weights transpose [12, 13, 14, 15] .Credit assignment with dual compartments. DP introduces "dyadic" neurons, which permits representing errors and activities simultaneously as the difference and mean of each neurons compartments, which in terms permits turning the two distinct inference phases of CHL or EP into a single phase. The underlying motivation is conceptually similar, albeit different in terms of resulting algorithms, to the neuroscience inspired approximations of backpropagation [16, 7] which achieve single-phased learning by modelling pyramidal neurons with distinct compartments for integrating top-down and bottom-up signals.Lifted neural networks. Lifted neural networks have roots in the community of mathematical optimization and frame neural network training as bilevel optimization over weights and activations [17, 18, 19, 20, 21, 22] . Although predating the term lifted network [23] also fits in this category. As in variants of CHL, neurons in lifted neural networks rely only on local information and require less synchronization than backpropagation. In the context of lifted networks robustness to input perturbations has previously been explored in [24] . 
a saddle point objective for dyadic learning: 
 We consider a Hopfield-like model with N neurons (not counting input neurons). Each neuron is a dyad with two internal states, giving two state vectors denoted by s + ∈ R N and s -∈ R N respectively. Let θ ∈ R N ×N be a strictly lower triangular matrix -a lower triangular matrix with zeros on the main diagonal -and W λ = θ + (2λ -1)θ ⊤ ∈ R N ×N be the connectivity matrix induced by a λ ∈ [0, 1]. Further, let W λ ∈ R 2N ×2N be an extended connectivity matrix defined asEQUATIONwhere λ := 1 -λ. An additional matrix θ 0 ∈ R N ×M maps an input vector x ∈ R M to R N (though in layered networks θ 0 will only map to a subset of the elements of s + and s -). Training consists in minimizing the following objective with respect to θ (with s := (sEQUATIONEquation 2 resembles a contrastive learning loss using Hopfield-like energies. It can be derived in a principled manner by extending the methods introduced in [25] to recurrent networks where neural states satisfy s = f (W λ s + θ 0 x). The function G(u) determines the nonlinear activation function (which we assume to be element-wise for simplicity). For invertible activation functionsG is chosen to satisfy ∇G(u) = f -1 (u)as in [26, 3] . The non-invertible hard sigmoid activation function can be achieved by choosingG(u) = 1 2 ∥u∥ 2 + ı ∈[0,1] (u), where ı ∈[0,1] (u) = {0 if u ∈ [0, 1] and ∞ otherwise}.Dynamics. Training amounts to solving min θ L λ (θ). Let the accent * denote optimality, then the gradient of L λ (θ) with respect to θ isEQUATIONAs W λ is already symmetric, the gradient with respect to s ± is thereforeEQUATIONThe stationary conditions for * s ± are therefore equivalent to *EQUATIONWe determine * s ± using dampened fixed-point iterations (see appendix A for details) of the preactivations a ± (defined through the relation s ± = f (a ± )) of the formEQUATIONIn the remaining part of the paper we will further restrict θ to be strictly lower block triangular (rather than just strictly lower triangular). We write this in terms of layerwise weight matrices θ k .EQUATIONThis approach is commonly applied to obtain layered Hopfield networks. In this setting we get mirror descent dynamics for the hidden layers of the formEQUATIONwhere the subscript k denotes layer index. When η = 1 this reduces to layer-wise closed form updates typically employed in EP [27] .Recovering symmetric Hopfield models trained with EP. In the case λ = 1 and (i.e. λ = 0)then the saddle-point problem Eq 2 decouples into two separate minimization problems over s + and s -, respectively. This corresponds to the two phases of EP in a deep Hopfield network, yielding the hidden layer dynamics:EQUATIONIn a hardware realization, this means the two sub-problems can be either solved in parallel (which requires maintaining two identical sets of weights) or sequentially using the same hardware (which avoids weight doubling, but increases runtime). Figure 1a and 1b illustrates the connectivity matrix W λ=1 for a small network. Recovering feedforward models trained with DP. The choice λ = λ = 1/2 recovers the DP dynamics.EQUATIONIn the absence of a teaching signal (β → 0) s + k = s - k for all k, hence the feedback term1 2 θ ⊤ k (s ± k+1 -s∓ k+1) vanishes and the model behaves exactly like a feedforward model. For hardware implementations it is noteworthy that the two compartments s + k,i and s - k,i of a neuron i in layer k receive the same bottom up and the same absolute feedback signal (but opposite sign), which allows them to share weights in physical realizations.Skew-symmetric Hopfield models Fixing λ = 0 (i.e. λ = 1) implies that W λ=0 is skewsymmetric (see Fig. 1(c-d )), and only the blocks governing interactions between s + and s -remain in W λ (Eq 1). The dynamics in the layered setting we consider here areEQUATIONA critical observation here is that states with odd layer indices of a ± only interact directly with a ∓ states with even indices, and vice versa. This means that the global minmax problem over a + and a -decouples into two independent minmax problems: one minmax problem over odd layers of a + and even layers of a -, and another over even layers of a + and odd layers of a -. Consequently, as in the case of the symmetric Hopfield model this setting has two phases, but unlike the symmetric setting each phase here corresponds to a minmax rather than a pure minimization problem.One might expect that the lack of direct s + -to-s + and s --to-s -interactions makes inference less stable. In fact in early research on binary skew-symmetric Hopfield networks it was shown that skew-symmetric weights lead to cycles in binary Hopfield models [28, 29] . We do observe oscillating behaviour in our continuous SSHM when applying analogous closed form updates, but not when employing a small inference stepsize (we used η = 0.05) in the dampened fixed point iterations. Apart from requiring a small stepsize the algorithm is very robust. In fact it appears to be compatible with much larger feedback gain β than the symmetric variants of EP and DP we compare against here.The skew-symmetric Hopfield model gives rise to remarkably different Lipschitz properties. Since a fixed point satisfies s = f (W λ s + x), we have via implicit differentiationEQUATION)By assuming linear activation mappings for the moment, we obtain ds dx = (I -W λ ) -1 . In the skew-symmetric setting (λ = 0), W 0 has a decomposition of the form W 0 = QΣQ ⊤ where Q is orthogonal and Σ is a matrix with only zeros and 2 × 2 blocks of the form 0 λi -λi 0 on its diagonal.Consequently,EQUATIONand therefore W ⊤ 0 W 0 has only non-negative eigenvalues. Hence,EQUATIONand the mapping x → (I+W 0 ) -1 x is 1-Lipschitz. In Appendix B we show that this property carries over to element-wise non-expansive activation functions. This is exciting as it means the network has a degree of built-in robustness to adversarial perturbations [30] . 
experiments: 
 We train MLPs with 4 hidden layers with 500 neurons each, using the three special cases λ = 1(SM), λ = 1/2 (FFM) and λ = 0 (SSHM). The models were trained on MNIST and FashionM-NIST. EP (which our learning dynamics reduces to for λ = 1) typically works best with bounded activation functions so we trained all models with a tempered hard sigmoid activation functionEQUATION)). This activation function was introduced in [31] to reduce the degree of saturation (neurons with value 0 or value 1). For reduced runtime we used undampened fixed-point iterations when possible (FFM and HM) and dampened one otherwise (SSHM) in order to ensure stable inference.Weak nudging. We carry out experiments applying weak centered nudging (analogous to the centered nudging introduced in [31] ). As shown in table 1 each of the models manage to solve both the MNIST and Fashion MNIST tasks, although the HM model lacks a bit behind on MNIST.As predicted by the theory, the upper bound on the sensitivity (measured by the Lipschitz constant L hidden ) of the hidden units to changes in their input y = θ 0 x is smallest for the skew-symmetric Hopfield model. We also report the Lipschitz estimate L output for the output units as a function of the network input x, where L output is the spectral norm ofEQUATIONThe symmetric Hopfield model achieves a smaller L output than the feedforward model, which is in line with a recent study finding EP to be more adversarially robust than feedforward models [32] .Figure 2b and 2e show that the SSHM model consistently generalizes better (difference between validation and training performance is smaller). Figure 2c and 2f show that the fraction of saturated neurons is highest for the HM model and lowest for the SSHM model.Strong nudging. Although EP and DP to a large extent are motivated as algorithms for new nonvon Neumann compute platforms they require β to be small. This is problematic as a small nudging ( The use of negative feedback in the skew-symmetric Hopfield model permits stable learning with significantly stronger nudging strength. 
discussion: 
 The proposed saddle-point framework in Eq. 2 encompasses training of a spectrum of algorithms parameterized in terms of λ. From a physical implementation perspective, only the Hopfield model with symmetric, skew-symmetric, and strictly lower triangular (feedforward model) connectivity matrices are truly appealing as they avoid excessive weight copying. Conveniently, such models can be implemented in analog compute platforms with programmable weights. For example, spatial light modulators allow one to program weights with up to 10 bit precision [33] . With such a device, the matrix-vector multiplication could be efficiently computed in the optical domain using light, while the required nonlinearities and other operations could be further realized in the analog electronic domain, resulting in the efficient opto-electronic Hopfield networks. In practice, real physical systems suffer from various nonidealities, which may result in approximately symmetric or skew-symmetric matrices. The impact of such physical imperfections on the dynamical properties of Hopfield-like networks is an exciting direction to investigate.Due to the negative feedback connections the skew-symmetric Hopfield model exhibits a strong inherent degree of robustness to perturbations, and is able to learn in the presence of a strong teaching signal. These are aspects we expect to be important in noisy substrates such as the brain as well as neuromorphic hardware. We hope this work will encourage further research into the role of negative feedback in learning.Outlook. Due to hardware concerns we have not explored the general case λ / ∈ {0, 1/2, 1}. One use for λ / ∈ {0, 1/2, 1} would be to start from a trained model and slowly alter λ while fine-tuning the weights. This would allow a kind of translating between between feedforward (λ = 1/2), Hopfield models (λ = 1) and skew-symmetric Hopfield models (λ = 0). It is also possible to choose λ on a layer by layer basis. This would be an alternative way to approach the joint training of chained feedforward and Hopfield models proposed in [34] .Here we show that inference of states in the dyadic models can be viewed as mirror descent/ascent (a generalization of the projected gradient method) [35] . In projected gradient descent we aim to take a negative gradient step on an objective L with respect to a variable s subject to the constraint that the the updated value s t+1 is inside a chosen domain C [36] (note that we here use subscripts to denote timesteps rather than layers). This can be formulated as the following problemEQUATIONThis yields the updates s t+1 = π C (s t -η∇ s E), where π C is a projection onto the domain C.In mirror descent the squared Euclidean distance penalizer in Eq 16 is replaced with with a Bregman divergence. Bregman divergences are non-negative, but not necessarily symmetric. Familiar examples of Bregman divergences include the squared Euclidean distance, the KL divergence and the Mahalanobis distance. The Bregman divergence between s and y induced by a convex function G is equal to G(s) minus its first order Taylor expansion at y evaluated at s.EQUATIONMirror descent with an appropriately chosen Bregman divergence can yield better convergence rates and in some cases turn constrained optimization problems into unconstrained optimization problems [37] . This has recently been used to train quantized neural networks [38] .Applying mirror descent to the Hopfield setting We restrict our analysis to the setting where G is chosen such that it satisfies ∇G(u) = f -1 (u), where f is a (typically nonlinear) activation function. We use as objective βL λ (scaling eq. 2 simplifies things). Using a Bregman divergence B G induced by G we can replace Eq 16 with the following:EQUATIONs - t+1 = arg maxEQUATIONNext we solve for first order optimality with respect to s using the definition of B G and the fact that B G (s + , s + t ) is convex with respect to its first argument.EQUATION⇐⇒EQUATIONUsing that ∇ s ± t βL λ = ±G(s ± ) ∓ λ(θ + θ ⊤ )s ± ∓ λ(θ -θ ⊤ )s ∓ ∓ θ 0 x ± βℓ ′ , and expressing things in terms of the preactivations a ± := f -1 (s) yields the following dampened fixed point iterations.EQUATIONMirror descent allows us to avoid the derivative of the activation function and ensures that the updated states are always feasible (this is tricker to ensure when using projected gradient descent).Mirror descent allows us to avoid the derivative of the activation function and ensures that the updated states are always feasible (this is tricker to ensure when using projected gradient descent).Non-invertible activation functions such as the sign and relu functions can be seen as limit cases of invertible activation functions such as tanh and leaky-relu.We apply implicit differentiation on ∇G -1 (s) = W s + x =⇒ ∇ 2 G -1 (s) ds dx = W ds dx + I. 
 
 Appendix:  



Reviews: review: title: title: Interesting paper with both theoretical and practical implications
main_review: main_review: This paper develops a particular parametrisation of networks of two-state neurons with the weight-space (including feedback connections) ranging from symmetric to feedforward to skew-symmetric. This enables the authors to develop a generalised saddle-point based learning framework covering Equilibrium Propagation (EP) in symmetric Hopfield models, Dual Propagation in feedforward models, and a novel two-phase learning algorithm in skew-symmetric Hopfield models (SSHM). While the training for SSHM is two-phased just like in EP, there are important differences in what constitutes the two phases of learning. Furthermore, the negative feedback inherent to SSHM also provides them significantly more robustness to large nudging signals during training. They also, in principle, could provide enhanced robustness to adversarial perturbations.

**Strengths:**
- Well-developed novel theoretical framework that covers a range of important local learning approaches.
- SSHMs seem understudied, and this paper highlights potential advantages in exploring them further. I found the larger point about negative feedback having desirable properties in the context of temporal-difference approximations to backprop very interesting.
- Paper is well-written with the motivation and experimental results clearly laid out.

**Potential improvements:**
- The theory is quite dense, and would benefit from a more in-depth explanation of terms and motivations. Given the limited space, this is understandable, but it will be easier to digest in long-form in a full paper.
- More comments on the differences to EP would add to the strength of the paper. What are the implications of having minmax optimisation steps instead of two minimisation steps? What are the potential implications for neuroscience and biological learning? These would all be interesting questions to address or at least provide some comments in the discussion.
title: title: The authors investigated closer the role of negative feedback learning and did so using skew symmetric Hopfield networks
main_review: main_review: The paper is well written and follows a clear approach. While not being very familiar with EP and Hopfield networks, the approach sounds convincing to me and the derivation and analysis seems well done.