Towards Biologically Plausible Convolutional Networks

1 Introduction . Convolutional networks are a cornerstone of modern deep learning : they ’ re widely used in the visual domain [ 1 , 2 , 3 ] , speech recognition [ 4 ] , text classification [ 5 ] , and time series classification [ 6 ] . They have also played an important role in enhancing our understanding of the visual stream [ 7 ] . Indeed , simple and complex cells in the visual cortex [ 8 ] inspired convolutional and pooling layers in deep networks [ 9 ] ( with simple cells implemented with convolution and complex ones with pooling ) . Moreover , the representations found in convolutional networks are similar to those in the visual stream [ 10 , 11 , 12 , 13 ] ( see [ 7 ] for an in-depth review ) . Despite the success of convolutional networks at reproducing activity in the visual system , as a model of the visual system they are somewhat problematic . That ’ s because convolutional networks share weights , something biological networks , for which weight updates must be local , can ’ t do [ 14 ] . Locally connected networks avoid this problem by using the same receptive fields as convolutional networks ( thus locally connected ) , but without weight sharing [ 15 ] . However , they pay a price for biological plausibility : locally connected networks are known to perform worse than their 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . convolutional counterparts on hard image classification tasks [ 15 , 16 ] . There is , therefore , a need for a mechanism to bridge the gap between biologically plausible locally connected networks and implausible convolutional ones . Here , we consider two such mechanisms . One is to use extensive data augmentation ( primarily image translations ) ; the other is to introduce an auxiliary objective that allows some form of weight sharing , which is implemented by lateral connections ; we call this approach dynamic weight sharing . The first approach , data augmentation , is simple , but we show that it suffers from two problems : it requires far more training data than is normally used , and even then it fails to close the performance gap between convolutional and locally connected networks . The second approach , dynamic weight sharing , implements a sleep-like phase in which neural dynamics facilitate weight sharing . This is done through lateral connections in each layer , which allows subgroups of neurons to share their activity . Through this lateral connectivity , each subgroup can first equalize its weights via antiHebbian learning , and then generate an input pattern for the next layer that helps it to do the same thing . Dynamic weight sharing doesn ’ t achieve perfectly convolutional connectivity , because in each channel only subgroups of neurons share weights . However , it implements a similar inductive bias , and , as we show in experiments , it performs almost as well as convolutional networks , and also achieves better fit to the ventral stream data , as measured by the Brain-Score [ 12 , 17 ] . Our study suggests that convolutional networks may be biologically plausible , as they can be approximated in realistic networks simply by adding lateral connectivity and Hebbian learning . As convolutional networks and locally connected networks with dynamic weight sharing have similar performance , convolutional networks remain a good “ model organism ” for neuroscience . This is important , as they consume much less memory than locally connected networks , and run much faster . 2 Related work . Studying systems neuroscience through the lens of deep learning is an active area of research , especially when it comes to the visual system [ 18 ] . As mentioned above , convolutional networks in particular have been extensively studied as a model of the visual stream ( and also inspired by it ) [ 7 ] , and also as mentioned above , because they require weight sharing they lack biological plausibility . They have also been widely used to evaluate the performance of different biologically plausible learning rules [ 15 , 19 , 20 , 21 , 22 , 23 , 24 , 25 ] . Several studies have tried to relax weight sharing in convolutions by introducing locally connected networks [ 15 , 16 , 26 ] ( [ 26 ] also shows that local connectivity itself can be learned from a fully connected network with proper weight regularization ) . Locally connected networks perform as well as convolutional ones in shallow architectures [ 15 , 26 ] . However , they perform worse for large networks and hard tasks , unless they ’ re initialized from an already well-performing convolutional solution [ 16 ] or have some degree of weight sharing [ 27 ] . In this study , we seek biologically plausible regularizations of locally connected network to improve their performance . Convolutional networks are not the only deep learning architecture for vision : visual transformers ( e.g. , [ 28 , 29 , 30 , 31 ] ) , and more recently , the transformer-like architectures without self-attention [ 32 , 33 , 34 ] , have shown competitive results . However , they still need weight sharing : at each block the input image is reshaped into patches , and then the same weight is used for all patches . Our Hebbian-based approach to weight sharing fits this computation as well ( see Appendix A.4 ) . 3 Regularization in locally connected networks . 3.1 Convolutional versus locally connected networks . Convolutional networks are implemented by letting the weights depend on the difference in indices . Consider , for simplicity , one dimensional convolutions and a linear network . Letting the input and output of a one layer in a network be xj and zi , respectively , the activity in a convolutional network is zi = NX j=1 wi jxj , ( 1 ) where N is the number of neurons ; for definiteness , we ’ ll assume N is the same in in each layer ( right panel in Fig . 1 ) . Although the index j ranges over all N neurons , many , if not most , of the weights are zero : wi j is nonzero only when |i j| k/2 < N for kernel size k. For networks that aren ’ t convolutional , the weight matrix wi j is replaced by wij , zi = NX j=1 wijxj . ( 2 ) Again , the index j ranges over all N neurons . If all the weights are nonzero , the network is fully connected ( left panel in Fig . 1 ) . But , as in convolutional networks , we can restrict the connectivity range by letting wij be nonzero only when |i j| k/2 < N , resulting in a locally connected , but non-convolutional , network ( center panel in Fig . 1 ) . 3.2 Developing convolutional weights : data augmentation versus dynamic weight sharing . Here we explore the question : is it possible for a locally connected network to develop approximately convolutional weights ? That is , after training , is it possible to have wij ⇡ wi j ? There is one straightforward way to do this : augment the data to provide multiple translations of the same image , so that each neuron within a channel learns to react similarly ( Fig . 2A ) . A potential problem is that a large number of translations will be needed . This makes training costly ( see Section 5 ) , and is unlikely to be consistent with animal learning , as animals see only a handful of translations of any one image . A less obvious solution is to modify the network so that during learning the weights become approximately convolutional . As we show in the next section , this can be done by adding lateral connections , and introducing a sleep phase during training ( Fig . 2B ) . This solution doesn ’ t need more data , but it does need an additional training step . 4 A Hebbian solution to dynamic weight sharing . If we were to train a locally connected network without any weight sharing or data augmentation , the weights of different neurons would diverge ( region marked “ training ” in Fig . 3A ) . Our strategy to make them convolutional is to introduce an occasional sleep phase , during which the weights relax to their mean over output neurons ( region marked “ sleep ” in Fig . 3A ) . This will compensate weight divergence during learning by convergence during the sleep phase . If the latter is sufficiently strong , the weights will remain approximately convolutional . To implement this , we introduce lateral connectivity , chosen to equalize activity in both the input and output layer . That ’ s shown in Fig . 3B , where every third neuron in both the input ( x ) and output ( z ) layers are connected . Once the connected neurons in the input layer have equal activity , all output neurons receive identical input . Since the lateral output connections also equalize activity , all connection that correspond to a translation by three neurons see exactly the same pre and postsynaptic activity . A naive Hebbian learning rule ( with weight decay ) would , therefore , make the network convolutional . However , we have to take care that the initial weighs are not over-written during Hebbian learning . We now describe how that is done . To ease notation , we ’ ll let wi be a vector containing the incoming weights to neuron i : ( wi ) j ⌘ wij . Moreover , we ’ ll let j run from 1 to k , independent of i . With this convention , the response of neuron i , zi , to a k-dimensional input , x , is given by zi = w > i x = kX j=1 wijxj . ( 3 ) Assume that every neuron sees the same x , and consider the following update rule for the weights , wi / 0 @ zi 1 N NX j=1 zj 1 Ax wi winiti , ( 4 ) where winiti are the weights at the beginning of the sleep phase ( not the overall training ) . This Hebbian update effectively implements SGD over the sum of ( zi zj ) 2 , plus a regularizer ( the second term ) to keep the weights near winiti . If we present the network with M different input vectors , xm , and denote the covariance matrix C ⌘ 1M P m xmx > m , then , asymptotically , the weight dynamics in Eq . ( 4 ) converges to ( see Appendix A ) w⇤i = ( C + I ) 1 0 @ C 1 N NX j=1 winitj + w init i 1 A ( 5 ) where I is the identity matrix . As long as C is full rank and is small , we arrive at shared weights : w⇤i ⇡ 1N PN i=1 w init i . It might seem advantageous to set = 0 , as non-zero only biases the equilibrium value of the weight . However , non-zero ensures that for noisy input , xi = x + ⇠i ( such that the inputs to different neurons are the same only on average , which is much more realistic ) , the weights still converge ( at least approximately ) to the mean of the initial weights ( see Appendix A ) . In practice , the dynamics in Eq . ( 4 ) converges quickly . We illustrate it in Fig . 4A by plotting log SNRw over time , where SNRw , the signal to noise ratio of the weights , is defined as SNRw = 1 k2 X j 1 N P i ( wi ) j 2 1 N P i ( wi ) j 1N P i0 ( wi0 ) j 2 . ( 6 ) For all kernel sizes ( we used 2d inputs , meaning k2 inputs per neuron ) , the weights converge to a nearly convolutional solution within a few hundred iterations ( note the logarithmic scale of the y axis in Fig . 4A ) . See Appendix A for simulation details . Thus , to run our experiments with deep networks in a realistic time frame , we perform weight sharing instantly ( i.e. , directly setting them to the mean value ) during the sleep phase .

Summary: This paper designed a convolutional neural network that relaxed the weight-sharing constraint by adding local lateral connections to implement dynamic weight sharing as a "sleep phase". The idea was inspired by the biological neural network, where a strict weight-sharing mechanism does not exist. They further used a Hebbian learning rule for weight update during the sleep phase. Their results suggest such a design may have comparable performance as the traditional CNN on image recognition tasks, and it is more computationally efficient than using data augmentation (e.g., translation) to force weight-sharing.