Predify: Augmenting deep neural networks with brain-inspired predictive coding dynamics

1 Introduction . Deep convolutional neural networks ( DCNNs ) , initially inspired by the primate visual cortex architecture , have taken big strides in solving computer vision tasks in the last decade . State-of-the-art networks can learn to classify images with high accuracy from huge labeled datasets [ 1–6 ] . This rapid progress and the resulting interest in these techniques have also highlighted their various shortcomings . Most widely studied is the sensitivity of neural networks , not only to perturbations specifically designed to fool them ( so-called “ adversarial examples ” ) but also to regular noises typically observed in natural scenes [ 7–9 ] . These shortcomings indicate that there is still room for improvement in current techniques . One possible way to improve the robustness of artificial neural networks could be to take further inspiration from the brain . In particular , one major aspect of the cerebral cortex that is missing from standard feedforward DCNNs is the presence of feedback connections . Recent studies have stressed the importance of feedback connections in the brain [ 10 , 11 ] , and have shown how artificial neural ∗Equal Contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . networks can take advantage of such feedback for various tasks such as object recognition with occlusion [ 12 ] , or panoptic segmentation [ 13 ] . Feedback connections convey contextual information about the state of the higher layers down to the lower layers of the hierarchy ; in this way , they can constrain lower layers to represent inputs in meaningful ways . In theory , this could make neural representations more robust to image degradation [ 14 ] . Merely including feedback in the pattern of connections , however , may not always be sufficient ; rather , it should be combined with proper mechanistic principles . To that end , we explore the potential of recurrent dynamics for augmenting deep neural networks with brain-inspired predictive coding ( supported by ample neuroscience evidence [ 15–19 ] ) . We build large-scale hierarchical networks with both feedforward and feedback connections that can be trained using error backpropagation . Several prior studies have explored this interesting avenue of research [ 20–23 ] , but with important differences with our approach ( see Section 3 ) . We demonstrate that our proposed method adds desirable properties to feedforward DCNNs , especially when viewed from the perspective of robustness . Our contributions can be summarized as follows : • We propose a novel strategy for effectively incorporating recurrent feedback connections based on the neuroscientific principle of predictive coding . • We implement this strategy in two pre-trained feedforward architectures with unsupervised training of the feedback weights , and show that this improves their robustness against different types of natural and adversarial noise . • We suggest and verify that an emergent property of the network is to iteratively shift noisy representations towards the corresponding clean representations—a form of “ projection towards the learned manifold ” as implemented in certain adversarial defense methods . • To facilitate research aimed at using such neuroscientific principles in machine learning , we provide a Python package called Predify that can easily implement the proposed predictive coding dynamics in any convolutional neural network with a few lines of code . 2 Our Approach . 2.1 The proposed predictive coding dynamics . Predictive coding , as introduced by [ 24 ] , is a neurocomputational theory positing that the brain maintains an internal model of the world , which it uses to actively predict the observed stimulus . Within a hierarchical architecture , each higher layer attempts to predict the activity of the layer immediately below , and the errors made in this prediction are then utilized to correct the higher-layer activity . To establish our notation , let us consider a hierarchical feedforward network equipped with generative feedback connections , as represented in Figure 1 . The network contains N encoding layers en ( n ∈ N ) and N corresponding decoding layers dn−1 . The feedforward weights connecting layer n− 1 to layer n are denoted by W fn−1 , n , and the feedback weights from layer n+ 1 to n by W bn+1 , n . For a given input image , we first initiate the activations of all encoding layers with a feedforward pass . Then , over successive recurrent iterations ( referred to as timesteps t ) , both the decoding and encoding layer representations are updated using the following equations ( also refer to Pseudocode 1 ) : dn ( t ) =W b n+1 , nen+1 ( t ) ( 1 ) en ( t+ 1 ) = βnW f n−1 , nen−1 ( t+ 1 ) + λndn ( t ) + ( 1− βn − λn ) en ( t ) − αn∇ n−1 ( t ) , ( 2 ) where βn , λn ( 0 ≤ βn + λn ≤ 1 ) , and αn act as layer-dependent balancing coefficients for the feedforward , feedback , and error-correction terms , respectively . n−1 ( t ) denotes the reconstruction error at layer n − 1 and is defined as the mean squared error ( MSE ) between the representation en−1 ( t ) and the predicted reconstruction dn−1 ( t ) at that particular timestep . Layer e0 is defined as the input image and remains constant over timesteps . All the weights W fn−1 , n and W b n+1 , n are fixed during these iterations . Each of the four terms in Equation 2 contributes different signals , reflected by different arrow colors in Figure 1 : ( i ) the feedforward term ( green arrows ; controlled by parameter β ) provides information about the ( constant ) input and changing representations in the lower layers , ( ii ) the feedback correction term ( red arrows ; parameter λ ) , as proposed in [ 24 , 25 ] , guides activations towards their representations from the higher levels , thereby reducing the reconstruction errors over time , ( iii ) the memory term ( blue arrows ) acts as a time constant to retain the current representation over successive timesteps , and ( iv ) the feedforward error correction term ( black arrows ; controlled by parameter α ) corrects representations in each layer such that their generative feedback can better match the preceding layer . For this error correction term , we directly use the error gradient ∇ n−1 = [ ∂ n−1∂e0n , ... , ∂ n−1 ∂ekn ] to take full advantage of modern machine learning capabilities ( where k is the number of elements in en ) . While the direct computation of this error gradient is biologically implausible , it has been noted before that it is mathematically equivalent to propagating error residuals Pseudocode 1 Predictive Coding Iterations 1 : Input image : e0 2 : for n = 1 toN do 3 : en ← Conv ( en−1 ) 4 : dn−1 ← deConv ( en ) 5 : n−1 ← ||dn−1 − en−1||22 6 : end for 7 : for t = 1 to T do 8 : for n = 1 toN do 9 : ff ← βn · Conv ( en−1 ) 10 : fb← 0 11 : if n < N then 12 : fb← λn · dn 13 : end if 14 : en ← ff + fb+ ( 1− βn − λn ) · en − αn · ∇ n−1 15 : dn−1 ← deConv ( en ) 16 : n−1 ← ||dn−1 − en−1||22 17 : end for 18 : end for up through the ( transposed ) feedback connection weights ( W b ) T , as often done in other predictive coding implementations [ 22 , 24 ] . Together , the feedforward and feedback error correction terms fulfill the objective of predictive coding as laid out by Rao and Ballard [ 24 ] . We discuss the similarities and differences between our equations and those proposed in the original Rao and Ballard implementation in the Appendix A.6 . While it is certainly possible to train such an architecture in an end-to-end fashion , by combining a classification objective for the feedforward weights W f with an unsupervised predictive coding objective ( see Section 2.2 ) for the feedback weights W b , we believe that the benefits of our proposed scheme are best demonstrated by focusing on the added value of the feedback pathway onto a pre-existing state-of-the-art feedforward network . Consequently , we implement the proposed strategy with two existing feedforward DCNN architectures as backbones : VGG16 and EfficientNetB0 , both trained on ImageNet . We show that predictive coding confers higher robustness to these networks . 2.2 Model architectures and training . We select VGG16 and EfficientNetB0 , two different pre-trained feedforward networks on ImageNet , and augment them with the proposed predictive coding dynamics . The resulting models are called PVGG16 and PEfficientNetB0 , respectively . The networks ’ “ bodies ” ( without the classification head ) are split into a cascade of N sub-modules , where each plays the role of an en in equation ( 2 ) . We then add deconvolutions as feedback layers dn−1 connecting each en to en−1 , with kernel sizes accounting for the increased receptive fields of the neurons in en or upsampling layers to match the size of the predictions and their targets ( see Appendix A.2 ) . We then train the parameters of the feedback deconvolution layers with an unsupervised reconstruction objective ( with all feedforward parameters frozen ) . We minimize the reconstruction errors just after the first forward pass , and after a single deconvolution step ( i.e . no error correction or predictive coding recurrent dynamics are involved at this stage ) : L = N−1∑ n=0 ‖ en − dn ‖22 , ( 3 ) where en is the output of the nth encoder after the first forward pass and dn is the estimated reconstruction of en via feedback/deconvolution ( from en+1 ) . For both the networks , after training the feedback deconvolution layers , we freeze all of the weights , and set the values of hyperparameters to βn = 0.8 , λn = 0.1 , and αn = 0.01 for all the encoders/decoders in Equation ( 2 ) . We also explore various strategies for further tuning hyperparameters to improve the results ( see Appendix A.7 for the chosen hyperparameter values ) . 2.3 Predify To facilitate and automate the process of adding the proposed predictive coding dynamics to existing deep neural networks , we have developed an open-source Python package called Predify . The package is developed based on PyTorch [ 26 ] and provides a flexible object oriented framework to convert any PyTorch-compatible network into a predictive network . While an advanced user may find it easy to integrate Predify in their project manually , a simple text-based user interface ( in TOML2 format ) is also provided to automate the steps . For the sake of improved performance and flexibility , Predify generates the code of the predictive network rather than the Python object . Given the original network and a configuration file ( e.g . 'config.toml ' ) that indicates the intended source and target layers for the predictive feedback , three lines of code are enough to construct the corresponding predictive network : from predify import predify net = # load PyTorch network predify ( net , './ config.toml ' ) # config file indicates the layers that # will act as outputs of encoders . 2https : //toml.io/en/ The Appendix A.1 provides further details on the package , along with a sample config file and certain default behaviours . Predify is an ongoing project available on GitHub3 under GNU General Public License v3.0 . Scripts for creating PVGG16 and PEfficientNetB0 from their feedforward instances and reproducing the results presented in this paper , as well as the pre-trained weights are also available on another GitHub repository4 .

Summary: The authors draw inspiration from neuroscience and propose augmenting feedforward neural networks with recurrent dynamics to improve their adversarial and corruption robustness. They provide a PyTorch package that can augment any convolutional neural network with recurrence. Their empirical tests show that recurrence mostly helps with robustness.