Towards Biologically Plausible Convolutional Networks

1 Introduction . Convolutional networks are a cornerstone of modern deep learning : they ’ re widely used in the visual domain [ 1 , 2 , 3 ] , speech recognition [ 4 ] , text classification [ 5 ] , and time series classification [ 6 ] . They have also played an important role in enhancing our understanding of the visual stream [ 7 ] . Indeed , simple and complex cells in the visual cortex [ 8 ] inspired convolutional and pooling layers in deep networks [ 9 ] ( with simple cells implemented with convolution and complex ones with pooling ) . Moreover , the representations found in convolutional networks are similar to those in the visual stream [ 10 , 11 , 12 , 13 ] ( see [ 7 ] for an in-depth review ) . Despite the success of convolutional networks at reproducing activity in the visual system , as a model of the visual system they are somewhat problematic . That ’ s because convolutional networks share weights , something biological networks , for which weight updates must be local , can ’ t do [ 14 ] . Locally connected networks avoid this problem by using the same receptive fields as convolutional networks ( thus locally connected ) , but without weight sharing [ 15 ] . However , they pay a price for biological plausibility : locally connected networks are known to perform worse than their 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . convolutional counterparts on hard image classification tasks [ 15 , 16 ] . There is , therefore , a need for a mechanism to bridge the gap between biologically plausible locally connected networks and implausible convolutional ones . Here , we consider two such mechanisms . One is to use extensive data augmentation ( primarily image translations ) ; the other is to introduce an auxiliary objective that allows some form of weight sharing , which is implemented by lateral connections ; we call this approach dynamic weight sharing . The first approach , data augmentation , is simple , but we show that it suffers from two problems : it requires far more training data than is normally used , and even then it fails to close the performance gap between convolutional and locally connected networks . The second approach , dynamic weight sharing , implements a sleep-like phase in which neural dynamics facilitate weight sharing . This is done through lateral connections in each layer , which allows subgroups of neurons to share their activity . Through this lateral connectivity , each subgroup can first equalize its weights via antiHebbian learning , and then generate an input pattern for the next layer that helps it to do the same thing . Dynamic weight sharing doesn ’ t achieve perfectly convolutional connectivity , because in each channel only subgroups of neurons share weights . However , it implements a similar inductive bias , and , as we show in experiments , it performs almost as well as convolutional networks , and also achieves better fit to the ventral stream data , as measured by the Brain-Score [ 12 , 17 ] . Our study suggests that convolutional networks may be biologically plausible , as they can be approximated in realistic networks simply by adding lateral connectivity and Hebbian learning . As convolutional networks and locally connected networks with dynamic weight sharing have similar performance , convolutional networks remain a good “ model organism ” for neuroscience . This is important , as they consume much less memory than locally connected networks , and run much faster . 2 Related work . Studying systems neuroscience through the lens of deep learning is an active area of research , especially when it comes to the visual system [ 18 ] . As mentioned above , convolutional networks in particular have been extensively studied as a model of the visual stream ( and also inspired by it ) [ 7 ] , and also as mentioned above , because they require weight sharing they lack biological plausibility . They have also been widely used to evaluate the performance of different biologically plausible learning rules [ 15 , 19 , 20 , 21 , 22 , 23 , 24 , 25 ] . Several studies have tried to relax weight sharing in convolutions by introducing locally connected networks [ 15 , 16 , 26 ] ( [ 26 ] also shows that local connectivity itself can be learned from a fully connected network with proper weight regularization ) . Locally connected networks perform as well as convolutional ones in shallow architectures [ 15 , 26 ] . However , they perform worse for large networks and hard tasks , unless they ’ re initialized from an already well-performing convolutional solution [ 16 ] or have some degree of weight sharing [ 27 ] . In this study , we seek biologically plausible regularizations of locally connected network to improve their performance . Convolutional networks are not the only deep learning architecture for vision : visual transformers ( e.g. , [ 28 , 29 , 30 , 31 ] ) , and more recently , the transformer-like architectures without self-attention [ 32 , 33 , 34 ] , have shown competitive results . However , they still need weight sharing : at each block the input image is reshaped into patches , and then the same weight is used for all patches . Our Hebbian-based approach to weight sharing fits this computation as well ( see Appendix A.4 ) . 3 Regularization in locally connected networks . 3.1 Convolutional versus locally connected networks . Convolutional networks are implemented by letting the weights depend on the difference in indices . Consider , for simplicity , one dimensional convolutions and a linear network . Letting the input and output of a one layer in a network be xj and zi , respectively , the activity in a convolutional network is zi = NX j=1 wi jxj , ( 1 ) where N is the number of neurons ; for definiteness , we ’ ll assume N is the same in in each layer ( right panel in Fig . 1 ) . Although the index j ranges over all N neurons , many , if not most , of the weights are zero : wi j is nonzero only when |i j| k/2 < N for kernel size k. For networks that aren ’ t convolutional , the weight matrix wi j is replaced by wij , zi = NX j=1 wijxj . ( 2 ) Again , the index j ranges over all N neurons . If all the weights are nonzero , the network is fully connected ( left panel in Fig . 1 ) . But , as in convolutional networks , we can restrict the connectivity range by letting wij be nonzero only when |i j| k/2 < N , resulting in a locally connected , but non-convolutional , network ( center panel in Fig . 1 ) . 3.2 Developing convolutional weights : data augmentation versus dynamic weight sharing . Here we explore the question : is it possible for a locally connected network to develop approximately convolutional weights ? That is , after training , is it possible to have wij ⇡ wi j ? There is one straightforward way to do this : augment the data to provide multiple translations of the same image , so that each neuron within a channel learns to react similarly ( Fig . 2A ) . A potential problem is that a large number of translations will be needed . This makes training costly ( see Section 5 ) , and is unlikely to be consistent with animal learning , as animals see only a handful of translations of any one image . A less obvious solution is to modify the network so that during learning the weights become approximately convolutional . As we show in the next section , this can be done by adding lateral connections , and introducing a sleep phase during training ( Fig . 2B ) . This solution doesn ’ t need more data , but it does need an additional training step . 4 A Hebbian solution to dynamic weight sharing . If we were to train a locally connected network without any weight sharing or data augmentation , the weights of different neurons would diverge ( region marked “ training ” in Fig . 3A ) . Our strategy to make them convolutional is to introduce an occasional sleep phase , during which the weights relax to their mean over output neurons ( region marked “ sleep ” in Fig . 3A ) . This will compensate weight divergence during learning by convergence during the sleep phase . If the latter is sufficiently strong , the weights will remain approximately convolutional . To implement this , we introduce lateral connectivity , chosen to equalize activity in both the input and output layer . That ’ s shown in Fig . 3B , where every third neuron in both the input ( x ) and output ( z ) layers are connected . Once the connected neurons in the input layer have equal activity , all output neurons receive identical input . Since the lateral output connections also equalize activity , all connection that correspond to a translation by three neurons see exactly the same pre and postsynaptic activity . A naive Hebbian learning rule ( with weight decay ) would , therefore , make the network convolutional . However , we have to take care that the initial weighs are not over-written during Hebbian learning . We now describe how that is done . To ease notation , we ’ ll let wi be a vector containing the incoming weights to neuron i : ( wi ) j ⌘ wij . Moreover , we ’ ll let j run from 1 to k , independent of i . With this convention , the response of neuron i , zi , to a k-dimensional input , x , is given by zi = w > i x = kX j=1 wijxj . ( 3 ) Assume that every neuron sees the same x , and consider the following update rule for the weights , wi / 0 @ zi 1 N NX j=1 zj 1 Ax wi winiti , ( 4 ) where winiti are the weights at the beginning of the sleep phase ( not the overall training ) . This Hebbian update effectively implements SGD over the sum of ( zi zj ) 2 , plus a regularizer ( the second term ) to keep the weights near winiti . If we present the network with M different input vectors , xm , and denote the covariance matrix C ⌘ 1M P m xmx > m , then , asymptotically , the weight dynamics in Eq . ( 4 ) converges to ( see Appendix A ) w⇤i = ( C + I ) 1 0 @ C 1 N NX j=1 winitj + w init i 1 A ( 5 ) where I is the identity matrix . As long as C is full rank and is small , we arrive at shared weights : w⇤i ⇡ 1N PN i=1 w init i . It might seem advantageous to set = 0 , as non-zero only biases the equilibrium value of the weight . However , non-zero ensures that for noisy input , xi = x + ⇠i ( such that the inputs to different neurons are the same only on average , which is much more realistic ) , the weights still converge ( at least approximately ) to the mean of the initial weights ( see Appendix A ) . In practice , the dynamics in Eq . ( 4 ) converges quickly . We illustrate it in Fig . 4A by plotting log SNRw over time , where SNRw , the signal to noise ratio of the weights , is defined as SNRw = 1 k2 X j 1 N P i ( wi ) j 2 1 N P i ( wi ) j 1N P i0 ( wi0 ) j 2 . ( 6 ) For all kernel sizes ( we used 2d inputs , meaning k2 inputs per neuron ) , the weights converge to a nearly convolutional solution within a few hundred iterations ( note the logarithmic scale of the y axis in Fig . 4A ) . See Appendix A for simulation details . Thus , to run our experiments with deep networks in a realistic time frame , we perform weight sharing instantly ( i.e. , directly setting them to the mean value ) during the sleep phase .

Summary: The main point of this paper is that we can rest easy about the biological plausibility of convolutional networks, because a simple hebbian mechanism - along with a “sleep phase” - provides  a method for making initially random weights match up, i.e., makes a locally connected network convolutional. The hebbian learning mechanism moves weights at every kth interval to the same value, while keeping them from moving very far from their initial value.   Through experiments on several small datasets, the authors show that this mechanism works better than simply using lots of translations of the image to make the weights similar. In addition, the performance of the hebbian-regularized model is not far from a standard convnet, and on Tiny ImageNet, it actually performs a bit better than a standard convnet.   On ImageNet, the mechanism results in performance similar to a regular convolutional network.  I have read the authors' response, and I am going to stick with my rating. If the paper does not get into NeurIPS, I encourage the authors to continue their work and submit it somewhere else, e.g., WACV or Vision Research.