Learning representations for binary-classification without backpropagation

1 INTRODUCTION . A key factor enabling the successes of Deep Learning is the backpropagation of error ( BP ) algorithm ( Rumelhart et al. , 1986 ) . Since it has been introduced , BP has sparked several discussions on whether physical brains are realizing BP-like learning or not ( Grossberg , 1987 ; Crick , 1989 ) . Today , most researchers consent that two distinct characteristics of BP render the idea of a BP based learning in brains as implausible : 1 ) The usage of symmetric forward and backward connections and 2 ) the strict separation of activity and error propagation ( Bartunov et al. , 2018 ) . These two objections have lead researchers to search for more biologically motivated alternatives to BP . The three most influential families of BP alternatives distilled so far are Contrastive Hebbian Learing ( CHL ) ( Movellan , 1991 ) , target-propagation ( TP ) ( LeCun , 1986 ; Hinton , 2007 ; Bengio , 2014 ) and feedback Alignment ( FA ) ( Lillicrap et al. , 2016 ) . The idea of CHL is to propagate the target activities , instead of the errors , backward through the network . For this reason , a temporal dimension is added to the neuron activities . Each neuron then adapts its parameters based on the temporal differences of its ” forward ” and ” backward ” activity . The two significant critic points of CHL are the requirement for symmetric ” forward-backward ” connections and the use of alternating ” forward ” and ” backward ” phases ( Baldi & Pineda , 1991 ; Bartunov et al. , 2018 ) . TP shares the idea with Contrastive Hebbian Learning of propagating target activities instead of errors . However , rather than keeping symmetric forward and backward paths , the reciprocal propagation of the activities are realized through learned connections . Consequently , each layer has assigned two objectives : Learning the inverse of the layer ’ s forward function and minimizing the difference to the back-projected target activity . Variants of TP differ in how exactly the target activity is projected backward ( LeCun , 1986 ; Bengio , 2014 ; Bartunov et al. , 2018 ) . Theoretical guarantees of TP rely on the assumption that each reciprocal connection implements the perfect inverse of the corresponding forward function . This issue of an imperfect inverse was also found to be the ” bottleneck ” of TP in practice ( Bartunov et al. , 2018 ) . When the output of a layer has a significant lower dimension than its input , reconstructing the input from the output becomes challenging , resulting in poor learning performance . Feedback alignment algorithms eliminate the weight sharing implausibility of BP by replacing the symmetric weights in the error backpropagation path by random matrices . The second objection , i.e. , separate activity and error channels , is attenuated by Direct Feedback Alignment ( Nøkland , 2016 ) which drastically reduces the number of channels carrying an error signal . While feedback alignment algorithms work well on small and medium-sized benchmarks , a recent study identified that they are unable to provide learning on more challenging datasets like ImageNet ( Bartunov et al. , 2018 ) . Another criticism of FA algorithms is the lack of rigorous mathematical justification and convergence guarantees of the performed computations . In this work , we investigate feed-forward networks where the weights of all , expect the first , layers are constrained to positive values . We prove that this constraint does not invalidate the universal approximation capabilities of neural networks . Next , we show that , in combination with monotonic activation functions , all layers from the second layer on realize monotonically increasing functions . The backpropagation of a scalar error signal through these layers only affects the magnitude of the error signal but does not change its sign . Consequently , we prove that algorithms that bypass the error backpropagation steps , such as Direct Feedback Alignment , can compute the sign of the true gradient with respect to the weights of our constraint networks without the need for backpropagation . Finally , we show that our algorithm , which we call monotone Direct Feedback Alignment , can deliver its theoretical promises in practice , by surpassing the learning performance of existing feedback alignment algorithms in binary classification task , i.e. , when the error signal is scalar , and provide decent performance even when the error signal is not scalar . We make the following key contributions : • First FA algorithm that has provable learning capabilities for non-linear networks of arbitrary depth • Experimental evaluation showing that our FA algorithm outperforms the learning performance of existing FA algorithms and match backpropagation in binary classification tasks • We make an efficient TensorFlow implementation of all tested algorithms publicly available1 2 BACKPROPAGATION AND FEEDBACK ALIGNMENT . We consider the feed-forward neural network hl ( hl−1 ) : = { f ( Wlhl−1 + bl ) if l < L Wlhl−1 + bl if l = L h0 : = x Wl ∈ Rnl×nl−1 , bl ∈ Rnl ( 1 ) where f is the non-linear activation function , x the input and hL the output of the network . For classification tasks , hL is usually transformed into a probability distribution with discrete support by a sigmoid or softmax function . During training , the parametersWl , bl , l = 1 , . . . L are adjusted to minimize a loss functionL ( y , hL ) on samples of a giving training distribution p ( y , x ) . This is usually done by performing gradient descent θl ← θl − α dL dθ , α ∈ R+ ( 2 ) with respect to the parameters θl ∈ { Wl , bl } , 1 ≤ l ≤ n of the network . 1https : //github.com/mlech26l/iclr_paper_mdfa a ) BP b ) FA c ) DFA d ) mDFA y y y y 2.1 BACKPROPAGATION . Backpropagation ( Rumelhart et al. , 1986 ) is the primary method to compute the gradients needed by the updates in equation ( 2 ) by iteratively applying the chain-rule dL dθl = ( dhl dθl ) T dL dhl ( 3 ) dL dhl = ( dhl+1 dhl ) T dL dhl+1 ( 4 ) dhl+1 dhl = Wl+1diag ( f ′ ( Wl+1hl + bl+1 ) ) . ( 5 ) A graphical representation of how information first flow forward and then backward in BP through each layer is shown in Figure ( 1 ) a . Two major concerns argue against the idea that biological neural networks are implementing BPbased learning . I ) The weight matrix Wl of the forward path is reused in the backward path in the form of WTl ( weight sharing ) , and II ) the strict separation of activity carrying forward and error carrying backward connections ( reciprocal error transport ) . 2.2 FEEDBACK ALIGNMENT ALGORITHMS . Feedback alignment addresses the implausibility of reusing WTl in the backward path by replacing WTl by a fixed random matrix Bl . Lillicrap et al . ( 2016 ) showed that this somewhat counterintuitive approach works remarkably well in practice . The term ” feedback alignment ” originates from the observations made by Lillicrap et al . ( 2016 ) that the angle between the FA update vector and the true gradient starts to decrease , i.e. , align , after a few epochs of the training algorithm . Theoretical groundwork on this alignment principle of FA relies on strong assumptions such as a linearized network with one hidden layer ( Lillicrap et al. , 2016 ) . FA avoids any weight sharing but does not address the reciprocal error transport implausibility , due to its strict separation of forward and backward pathways , as shown in Figure ( 1 ) b . Direct Feedback Alignment ( DFA ) ( Nøkland , 2016 ) relaxes this issue by replacing all backward paths with a direct feed from the output layer error-gradient dLdhL . Consequently , there is only a single error signal that is distributed across the entire network , which is arguably more biologically plausible than reciprocal error connections . The resulting parameter updates of DFA are of the form δθl : = { dL dhL hL θl if l = L dL dhL Bl hl θl if l < L ( 6 ) , where Bl ∈ RnL×nl is a random matrix . A graphical schematic of DFA is shown in Figure ( 1 ) c. Similar to FA , DFA shows a decent learning performance in mid sized classification tasks ( Nøkland , 2016 ) , but fails on more complex datasets such as ImageNet ( Bartunov et al. , 2018 ) . Theory on adapting the alignment principle to DFA shows that under the strong assumptions of constant DFA update directions and a layer-wise criterion minimization , the DFA update vector will align with the true gradient ( Nøkland , 2016 ; Gilmer et al. , 2017 ) . Recently , Frenkel et al . ( 2019 ) proposed to combine ideas from feedback alignment and targetpropagation in their Direct Random Target Projection ( DRTP ) algorithm . While DRTP shows decent empirical performance , theoretical guarantees about DRTP rely on linearized networks . 2.3 SIGN-SYMMETRY ALGORITHMS . Liao et al . ( 2016 ) introduced the sign-symmetry algorithm , a hybrid of BP and FA . Sign-symmetry locks the signs of the feedback weight Bl to have the same signs as WTl , but random absolute value . The authors showed that this approach drastically improves learning performance compared to standard FA . Furthermore , Moskovitz et al . ( 2018 ) and Xiao et al . ( 2019 ) demonstrated that the sign-symmetry algorithm is even able to match backpropagation for training deep network architectures and large datasets such as ImageNet . While these empirical observations suggest that the polarity of the error feedback is more important than its magnitude , the mathematical justification of sign-symmetry remains absent . Similar to FA , sign-symmetry relaxes the strict weight sharing implausibility , but still relies on an unrealistic reciprocal error transport . 3 MONOTONE DIRECT FEEDBACK ALIGNMENT . In this section , we first introduce a new class of feed-forward networks , where all , except the first , layers are constrained to realize monotone functions . We call such networks mono-nets and show that they are as expressive as unconstrained feed-forward networks . Next , we prove that for our mono-nets with single output tasks , e.g. , binary-classification , feedback alignment algorithms provide the sign of the gradient . The sign of the gradient is interesting for learning , as it tells us if the value of a parameter should be increased or decreased in order to reduce the loss . At the end of this section , we will highlight similarities to algorithms from literature , which can provide resilient learning by only relying on the sign of the gradient . Neural networks with monotonic constraints have been already studied in literature ( You et al. , 2017 ) , however not in the context of learning algorithms . Definition 1 ( mono-net ) . A mono-net is a feed-forward neural network with L layers h1 , . . . hL , each layer l composted of nl units and the semantics hl ( hl−1 ) : = { f ( Wlhl−1 + bl ) if l < L Wlhl−1 + bl if l = L ( 7 ) h0 : = x ( 8 ) W1 ∈ Rn1×n0 , ( 9 ) Wl ∈ R nl×nl−1 + , for l > 1 ( 10 ) bl ∈ Rnl ( 11 ) where R+ are the positive reals , i.e . R+ = { x ∈ R|x ≥ 0 } , f is a non-linear monotonic increasing activation function , x the input and hL the output of the network . The major difference between mono-nets and general feed-forward neural networks is the restriction to only positive weight values in layers from the second layer on . Combined with the monotonic increasing activation function , this means that each layer hl ( hl−1 ) , l ≥ 2 realizes a monotone increasing function . Because functional composition preserves monotonicity , the complete network up to the first layer hl ◦ hl−1 ◦ · · · ◦ h2 ( h1 ) ( 12 ) implements a monotone increasing function . mono-nets are Universal Approximators At first glance , this restriction seems counterproductive , as it might interfere with the expressiveness of the networks . However , we proof in Theorem 1 that our mono-nets with tangent hyperbolic activation are universal approximators , meaning that we can approximate any continuous function arbitrarily close . A potential drawback of the monotonicity constraint is that we might need a larger number of units in the hidden layers to achieve the same expressiveness as a general feed-forward network , as illustrated in our proof of Theorem 1 . Theorem 1 ( mono-nets are Universal Approximators ) . Let In be the n-dimensional unit hypercube [ 0 , 1 ] n and C ( In ) denote the set of continuous functions f : In → R. We define ‖f‖∞ as the supremum norm of f ∈ C ( In ) over its domain In . For any given f ∈ C ( In ) and ε > 0 , there exist a function m : In → R of the form m ( x ) : = M∑ i=1 v̄i tanh ( w̄i Tx+ ŵi T ( −x ) + b̄i ) + c ( 13 ) with v̄ ∈ R+M , w̄i ∈ R+n , ŵi ∈ R+n , b̄ ∈ RM , c ∈ R and M < ∞ such that ‖m ( x ) − f ( x ) ‖∞ < ε . In essence , the set of functions m ( x ) of the form given in ( 19 ) is dense in C ( In ) . Proof . See supplementary materials 3. mDFA provides the sign of the gradient Here , we prove that for 1-dimensional outputs DFA applied on a mono-net , which we will call simply mDFA , provides the sign of the true gradient . Note that we focus our methods on DFA instead of ” vanilla ” FA , due to the superiority of DFA in terms of biological plausibility and empirical performance ( Nøkland , 2016 ; Bartunov et al. , 2018 ) . Theorem 2 ( For 1-dimensional outputs mDFA computes the sign of the gradient ) . Let L ( y , hL ) be a loss function , m ( x ) : = hL ◦ hL−1 ◦ · · · ◦ h2 ◦ h1 ◦ h0 ( x ) be a mono-net according to Definition 1 with parameters Θ : = { Wl , bl ∣∣l = 1 , . . . L } . We denote δθ the update value computed by mDFA and ∇θ as the gradient ∂L∂θ for any θ ∈ { Wl , bl } with 1 ≤ l ≤ L. If nL = 1 , it follows that ( δθ ) i , j · ( ∇θ ) i , j ≥ 0 , ( 14 ) for each coordinate ( i , j ) of θ . Proof . See supplementary materials . A graphical illustration of how activities and errors propagate in mDFA is shown in Figure ( 1 ) d. Literature on learning by relying only on the sign of the gradient Two learning concepts related to mDFA are RPROP ( Riedmiller & Braun , 1993 ; Riedmiller , 1994 ) and signSGD ( Bernstein et al. , 2018 ) . RPROP aims to build a more resilient alternative to gradient descent by decoupling the amplitude of the gradient from the step size of parameter updates . In essence , for each coordinate RPROP adapts the step size based on the sign of the most recent gradients computed . Riedmiller & Braun ( 1993 ) showed that their approach could stabilize the training of a neural network compared to standard gradient descent . Performing gradient descent with taking the sign of each gradient coordinate is on an algorithmic level equivalent to the well-known steepest descent method with L∞ norm ( Boyd & Vandenberghe , 2004 ; Bernstein et al. , 2018 ) . signSGD ( Bernstein et al. , 2018 ) studies convergence properties of the stochastic approximation of this algorithm . What about networks with more than one output neuron ? Theorem 2 applies only to networks with scalar output . As a natural consequence , one may ask whether such theoretical guarantees can be extended to more dimensional output variables . The simple answer is , unfortunately not . In the supplementary materials section A.3 we provide a counterexample showing that Theorem 2 naively extended to two output neurons does not hold anymore . We want to note that the requirement of a neural network to have only a single output neuron is biologically unjustified . It is known that sub-circuits of biological neuronal networks can feed to multiple motor neuron groups Cook et al . ( 2019 ) . How does mDFA relate to the non-negative matrix factorization ? A seemingly related concept to mDFA is the non-negative matrix factorization ( NMF ) algorithm . NMF decomposes an observation matrix V into a weight matrix W and a latent variable matrix H such that V ≈ WH . In contrast to other decomposition-based unsupervised learning methods , all three matrices V , W and H are restricted to non-negative entries . While NMF can model data that is inherently non-negative , such has semantic features of images and text , effectively Yuan & Oja ( 2005 ) ; Shahnaz et al . ( 2006 ) , the method is unable to learn subtractive and non-linear structures that are present in the data Lee & Seung ( 1999 ) . Semi-non-negative matrix factorization Ding et al . ( 2008 ) relaxes the original restriction to nonnegative observations of NMF , by only constraining the weight matrix W to be non-negative . Deep semi-NMF Trigeorgis et al . ( 2014 ) further enhances the expressiveness of NMF by adding multiple layers and non-linearities between them to the decomposition . Concerning this work , the semantics of mono-nets from the second layer on is equivalent to that of deep semi-NMF models . However , the unconstrained first layer of mono-nets provides universal approximation capabilities , enabling mono-nets to learn subtractive and non-monotonic input dependencies . Moreover , while deep NMF models are mostly trained via layer-wise learning in an unsupervised context Trigeorgis et al . ( 2014 ) ; Yu et al . ( 2018 ) , the sole purpose of mono-nets is to investigate alternatives to backpropagation for training multi-layer classifiers .

Summary: This paper examines the question of learning in neural networks with random, fixed feedback weights, a technique known as “feedback alignment”. Feedback alignment was originally discovered by Lillicrap et al. (2016; Nature Communications, 7, 13276) when they were exploring potential means of solving the “weight transport problem” for neural networks. Essentially, the weight transport problem refers to the fact that the backpropagation-of-error algorithm requires feedback pathways for communicating errors that have synaptic weights that are symmetric to the feedforward pathway, which is biologically questionable. Feedback alignment is one approach to solving the weight transport problem, which as stated above, relies on the use of random, fixed weights for communicating the error backwards. It has been shown that in some cases, feedback alignment converges to weight updates that are reasonably well-aligned to the true gradient. Though initially considered a good potential solution for biologically realistic learning, feedback alignment both has not scaled up to difficult datasets and has no theoretical guarantees that it converges to the true gradient. This paper addresses both these issues.