Activation Sharing with Asymmetric Paths Solves Weight Transport Problem without Bidirectional Connection

1 Introdution . Backpropagation ( BP ) [ 1 ] is the representative approach to training various deep neural networks . While BP exhibits excellent training performance , similar to or even better than that of humans , it has been long argued that the structure of biological neural networks does not support the backpropagation of errors [ 2 , 3 , 4 , 5 ] . To resolve this issue , a wide range of studies have been conducted to develop an algorithm that is feasible in biological neural networks . One important reason behind the biological implausibility of BP is the weight transport problem [ 6 ] . BP requires identical forward and feedback paths for reliable training ; i.e. , the two paths must have the same synaptic weights . While biological neural networks may also implement two separate processing paths ( forward and feedback paths ) , it is impossible to explicitly pass the weights between the two paths , as it requires a very fast transmission of information along the axon from each synapse output [ 7 ] . The Feedback Alignment ( FA ) algorithm [ 8 ] solves the weight transport problem by propagating errors through the feedback path with random fixed weights . It was shown that the feedback weights could be aligned with the forward weights in the course of training , and consequently , the network is trained in a similar way to BP . Similarly , Direct Feedback Alignment ( DFA ) [ 9 ] directly propagates errors from the top layer to lower layers using random fixed weights , and Direct Random Target Projection [ 10 ] locally creates errors using targets rather than using backpropagated errors . Although these algorithms demonstrate good training performance on simple networks , they exhibit large performance degradation when applied to complex networks , especially deep convolutional neural networks [ 11 , 12 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . More recent works suggest training the feedback weights as well as the forward weights instead of using random fixed weights in the feedback path . Training feedback weights through Reinforcement Learning [ 13 ] or Spiking Neural Network [ 14 ] provides better accuracy in classification tasks than the algorithms using random fixed weights . However , those approaches have been only evaluated in shallow neural networks , and it is questionable whether they would be capable of training deeper neural networks . The Sign-Symmetry algorithm [ 15 ] achieves good training performance in deep neural networks by transporting only the signs of the forward weights to the feedback path during training . However , since it still needs to transport the signs of the weights , it is difficult to say that the weight transport problem has been solved entirely . Recently , Weight Mirrors and modified Kolen-Pollack algorithms [ 16 , 17 ] , which align the feedback weights with the forward weights without explicit weight transport , succeeded in training deep convolutional neural networks on large datasets with outstanding performance close to BP . However , these algorithms require a bidirectional connection : forward and feedback neurons explicitly exchange information with each other through a pair of direct unidirectional paths . While this type of connection is found in some biological neurons [ 18 , 19 , 20 , 21 ] , it is difficult to generalize them to a broader range of biological neural networks that typically only have unidirectional paths between neurons , where feedback must route through multiple neurons in order to reach their destination [ 22 ] . In this paper , we propose an Activation Sharing algorithm , which performs weight update of a layer using the output ( activation ) of a lower layer , not the output of the immediately previous layer . Applying this learning method to both forward and feedback paths solves the weight transport problem without using bidirectional paths and also successfully trains deep convolutional networks such as ResNet-34 . Furthermore , since activation is shared across multiple layers during training , the algorithm could significantly reduce memory access and is suitable for efficient hardware implementation . 2 Related works . 2.1 Backpropagation and weight transport problem . We consider typical training process of a deep neural network . The forward path propagates input data through multiple layers , generating output activation of each layer : hl+1 = φ ( Wl+1 hl + bl+1 ) ( 1 ) where hl is the output activation of the hidden layer l , Wl is the weight matrix , and bl is the bias vector of layer l in the forward path . In the brain , h could be interpreted as neural firing rates , W as synaptic weights between neurons , b as bias currents , and φ as nonlinearities in neurons [ 16 ] . To train the model , synaptic weight changes ∆W are obtained by ∆Wl+1 = −η δl+1 hTl ( 2 ) where δ and η denote the error signal that flows through the feedback path and the learning rate , respectively . For calculating synaptic weight changes ∆W , the error δ should propagate from the output layer to lower layers in the reverse direction . In backpropagation [ 1 ] , errors that propagate through the feedback path are represented as δl = φ ′ ( hl ) W T l+1 δl+1 ( 3 ) where φ ′ denotes the derivative of the activation function φ in equation ( 1 ) . For this BP to occur in the brain , the error must be propagated in the reverse direction of the forward path , which is not feasible in biological neural networks that only have unidirectional paths . Even if there is a separate feedback path , as shown in Fig . 1a , the synaptic weights W in equation ( 1 ) should be also used to propagate the error in equation ( 3 ) . In other words , the forward and feedback paths must have identical synaptic weights . Neurons need to exchange their synaptic weights to realize this , but there is no proof that this weight transport actually occurs in the brain ( weight transport problem ) . 2.2 Training with random fixed feedback weights . The Feedback Alignment algorithm [ 8 ] was developed based on the idea that errors could be propagated using random fixed weightsB rather than the forward weightsW in equation ( 1 ) . As shown in Fig . 1b , errors propagate through the feedback path following the equation below . δl = φ ′ ( hl ) B T l+1 δl+1 ( 4 ) Moreover , the Direct Feedback Alignment algorithm [ 9 ] showed that the neural network could be trained if the error propagates directly from the output layer to each layer instead of flowing through all hidden layers sequentially ( Fig . 1c ) : δl = φ ′ ( hl ) B T l+1 δL ( 5 ) where L denotes the output layer of a deep neural network . These methods perfectly solve the weight transport problem by using random fixed weights in the feedback path and demonstrate good training performance in simple networks . However , they fail to train the model or incur significant performance drop when applied to deep convolutional neural networks [ 11 , 12 , 15 , 23 ] . 2.3 Training with feedback weight update . Since using random fixed weights for error propagation causes poor performance for training deep and complex neural networks , various methods that train the feedback weights simultaneously have been recently proposed [ 13 , 14 , 24 ] . For example , the Weight Mirror ( WM ) and modified Kolen-Pollack ( KP ) algorithms attain very competitive training performance on deep convolutional networks [ 16 ] . In WM , the feedback weights are updated by ∆Bl+1 = −η hl+1 hTl ( 6 ) using output activations of layers l and l + 1 while the error propagates in the same way as FA according to equation ( 4 ) . This update rule makes ∆Bl+1 proportional to Wl+1 , which aligns the feedback weights with the forward weights during training as discussed in [ 16 ] . On the other hand , the KP algorithm updates the feedback weights as below : ∆Bl+1 = ∆Wl+1 = −η δl+1 hTl ( 7 ) This method supports updating forward and feedback weights in the same direction , and those two weights could also be aligned during training . These algorithms succeed in training deep convolutional neural networks with similar performance to BP . Both WM and KP solve the weight transport problem by avoiding explicitly exchanging information related to synaptic weights such as weight changes ∆W and ∆B between the two paths ; instead , a forward neuron and its corresponding neuron in the feedback path exchange activation h and error δ . For example , δl+1 of the feedback path is needed to update the forward weightsWl+1 in equation ( 2 ) , while hl+1 of the forward path is required to update the feedback weightsBl+1 in equation ( 6 ) in WM . KP also needs hl+1 to update the feedback weightsBl+2 in equation ( 7 ) . However , exchanging information between two neurons constitutes a bidirectional connection , which is not supported in general biological neural network structures consisting of only unidirectional connections . 3 Activation Sharing to solve weight transport problem without bidirectional connections . Prior studies to find biologically plausible algorithms have mainly focused on approximating the forward path without modifying the forward path for training . However , the demand for correct activations to update weights inevitably incurs bidirectional connections when training feedback weights as depicted in Fig . 1d . Our study , on the other hand , develops a learning algorithm not relying on accurate activations . In this section , we first review the Feedback Alignment algorithm , which uses an approximate feedback path . Based on this , we show that the forward path could also be approximated , and we present an Activation Sharing algorithm . Finally , we show that the algorithm solves the weight transport problem without bidirectional connections in the network . 3.1 Approximate feedback path : Feedback Alignment . For the sake of simplicity , here we consider a linear network ( i.e. , we assume hl+1 = Wl+1 hl neglecting φ ) . As shown in equation ( 2 ) , we need the error δ and the activation h to update the weights in the network . For backpropagation , we could calculate these values very accurately . We use the activations obtained when the input propagates through layers and the errors that back-propagate through the same layers in the reverse order . Since we assume a linear network , equation ( 3 ) is simplified to δl+1 = W Tl+2 δl+2 . Consequently , the learning process in backpropagation ( Fig . 2a ) follows the equations below . hl+1 = Wl+1 hl ( 8 ) δl+2 = W T l+3 δl+3 ( 9 ) ∆Wl+2 = −η δl+2 hTl+1 ( 10 ) However , recent studies suggest that approximate errors could also be used for weight updates . For instance , Feedback Alignment uses approximate errors δ̃ that propagate through random fixed weights . The learning process is described in Fig . 2b and is expressed by hl+1 = Wl+1 hl ( 11 ) δ̃l+2 = B T l+3 δ̃l+3 6= δl+2 ( 12 ) ∆Wl+2 = −η δ̃l+2 hTl+1 6= −η δl+2 hTl+1 ( 13 ) where δ̃L = δL since the error at the output layer due to the loss function is the same for BP and FA . This approximate feedback path successfully trains a neural network by aligning the forward weights W with the random fixed weightsB .

Summary: Methods like backpropagation require the updates to be propagated along the same pathways and weights as the forward computation step (i.e. the "weight transport problem"). To solve this problem, the authors propose activation sharing, which uses random fixed weights both to forwardpropagate activations and to backpropagate error, without bidirectional connections. It is argued that this is more biologically plausible, and can be used to train deep networks (which previous methods could not do).