Predify: Augmenting deep neural networks with brain-inspired predictive coding dynamics

1 Introduction . Deep convolutional neural networks ( DCNNs ) , initially inspired by the primate visual cortex architecture , have taken big strides in solving computer vision tasks in the last decade . State-of-the-art networks can learn to classify images with high accuracy from huge labeled datasets [ 1–6 ] . This rapid progress and the resulting interest in these techniques have also highlighted their various shortcomings . Most widely studied is the sensitivity of neural networks , not only to perturbations specifically designed to fool them ( so-called “ adversarial examples ” ) but also to regular noises typically observed in natural scenes [ 7–9 ] . These shortcomings indicate that there is still room for improvement in current techniques . One possible way to improve the robustness of artificial neural networks could be to take further inspiration from the brain . In particular , one major aspect of the cerebral cortex that is missing from standard feedforward DCNNs is the presence of feedback connections . Recent studies have stressed the importance of feedback connections in the brain [ 10 , 11 ] , and have shown how artificial neural ∗Equal Contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . networks can take advantage of such feedback for various tasks such as object recognition with occlusion [ 12 ] , or panoptic segmentation [ 13 ] . Feedback connections convey contextual information about the state of the higher layers down to the lower layers of the hierarchy ; in this way , they can constrain lower layers to represent inputs in meaningful ways . In theory , this could make neural representations more robust to image degradation [ 14 ] . Merely including feedback in the pattern of connections , however , may not always be sufficient ; rather , it should be combined with proper mechanistic principles . To that end , we explore the potential of recurrent dynamics for augmenting deep neural networks with brain-inspired predictive coding ( supported by ample neuroscience evidence [ 15–19 ] ) . We build large-scale hierarchical networks with both feedforward and feedback connections that can be trained using error backpropagation . Several prior studies have explored this interesting avenue of research [ 20–23 ] , but with important differences with our approach ( see Section 3 ) . We demonstrate that our proposed method adds desirable properties to feedforward DCNNs , especially when viewed from the perspective of robustness . Our contributions can be summarized as follows : • We propose a novel strategy for effectively incorporating recurrent feedback connections based on the neuroscientific principle of predictive coding . • We implement this strategy in two pre-trained feedforward architectures with unsupervised training of the feedback weights , and show that this improves their robustness against different types of natural and adversarial noise . • We suggest and verify that an emergent property of the network is to iteratively shift noisy representations towards the corresponding clean representations—a form of “ projection towards the learned manifold ” as implemented in certain adversarial defense methods . • To facilitate research aimed at using such neuroscientific principles in machine learning , we provide a Python package called Predify that can easily implement the proposed predictive coding dynamics in any convolutional neural network with a few lines of code . 2 Our Approach . 2.1 The proposed predictive coding dynamics . Predictive coding , as introduced by [ 24 ] , is a neurocomputational theory positing that the brain maintains an internal model of the world , which it uses to actively predict the observed stimulus . Within a hierarchical architecture , each higher layer attempts to predict the activity of the layer immediately below , and the errors made in this prediction are then utilized to correct the higher-layer activity . To establish our notation , let us consider a hierarchical feedforward network equipped with generative feedback connections , as represented in Figure 1 . The network contains N encoding layers en ( n ∈ N ) and N corresponding decoding layers dn−1 . The feedforward weights connecting layer n− 1 to layer n are denoted by W fn−1 , n , and the feedback weights from layer n+ 1 to n by W bn+1 , n . For a given input image , we first initiate the activations of all encoding layers with a feedforward pass . Then , over successive recurrent iterations ( referred to as timesteps t ) , both the decoding and encoding layer representations are updated using the following equations ( also refer to Pseudocode 1 ) : dn ( t ) =W b n+1 , nen+1 ( t ) ( 1 ) en ( t+ 1 ) = βnW f n−1 , nen−1 ( t+ 1 ) + λndn ( t ) + ( 1− βn − λn ) en ( t ) − αn∇ n−1 ( t ) , ( 2 ) where βn , λn ( 0 ≤ βn + λn ≤ 1 ) , and αn act as layer-dependent balancing coefficients for the feedforward , feedback , and error-correction terms , respectively . n−1 ( t ) denotes the reconstruction error at layer n − 1 and is defined as the mean squared error ( MSE ) between the representation en−1 ( t ) and the predicted reconstruction dn−1 ( t ) at that particular timestep . Layer e0 is defined as the input image and remains constant over timesteps . All the weights W fn−1 , n and W b n+1 , n are fixed during these iterations . Each of the four terms in Equation 2 contributes different signals , reflected by different arrow colors in Figure 1 : ( i ) the feedforward term ( green arrows ; controlled by parameter β ) provides information about the ( constant ) input and changing representations in the lower layers , ( ii ) the feedback correction term ( red arrows ; parameter λ ) , as proposed in [ 24 , 25 ] , guides activations towards their representations from the higher levels , thereby reducing the reconstruction errors over time , ( iii ) the memory term ( blue arrows ) acts as a time constant to retain the current representation over successive timesteps , and ( iv ) the feedforward error correction term ( black arrows ; controlled by parameter α ) corrects representations in each layer such that their generative feedback can better match the preceding layer . For this error correction term , we directly use the error gradient ∇ n−1 = [ ∂ n−1∂e0n , ... , ∂ n−1 ∂ekn ] to take full advantage of modern machine learning capabilities ( where k is the number of elements in en ) . While the direct computation of this error gradient is biologically implausible , it has been noted before that it is mathematically equivalent to propagating error residuals Pseudocode 1 Predictive Coding Iterations 1 : Input image : e0 2 : for n = 1 toN do 3 : en ← Conv ( en−1 ) 4 : dn−1 ← deConv ( en ) 5 : n−1 ← ||dn−1 − en−1||22 6 : end for 7 : for t = 1 to T do 8 : for n = 1 toN do 9 : ff ← βn · Conv ( en−1 ) 10 : fb← 0 11 : if n < N then 12 : fb← λn · dn 13 : end if 14 : en ← ff + fb+ ( 1− βn − λn ) · en − αn · ∇ n−1 15 : dn−1 ← deConv ( en ) 16 : n−1 ← ||dn−1 − en−1||22 17 : end for 18 : end for up through the ( transposed ) feedback connection weights ( W b ) T , as often done in other predictive coding implementations [ 22 , 24 ] . Together , the feedforward and feedback error correction terms fulfill the objective of predictive coding as laid out by Rao and Ballard [ 24 ] . We discuss the similarities and differences between our equations and those proposed in the original Rao and Ballard implementation in the Appendix A.6 . While it is certainly possible to train such an architecture in an end-to-end fashion , by combining a classification objective for the feedforward weights W f with an unsupervised predictive coding objective ( see Section 2.2 ) for the feedback weights W b , we believe that the benefits of our proposed scheme are best demonstrated by focusing on the added value of the feedback pathway onto a pre-existing state-of-the-art feedforward network . Consequently , we implement the proposed strategy with two existing feedforward DCNN architectures as backbones : VGG16 and EfficientNetB0 , both trained on ImageNet . We show that predictive coding confers higher robustness to these networks . 2.2 Model architectures and training . We select VGG16 and EfficientNetB0 , two different pre-trained feedforward networks on ImageNet , and augment them with the proposed predictive coding dynamics . The resulting models are called PVGG16 and PEfficientNetB0 , respectively . The networks ’ “ bodies ” ( without the classification head ) are split into a cascade of N sub-modules , where each plays the role of an en in equation ( 2 ) . We then add deconvolutions as feedback layers dn−1 connecting each en to en−1 , with kernel sizes accounting for the increased receptive fields of the neurons in en or upsampling layers to match the size of the predictions and their targets ( see Appendix A.2 ) . We then train the parameters of the feedback deconvolution layers with an unsupervised reconstruction objective ( with all feedforward parameters frozen ) . We minimize the reconstruction errors just after the first forward pass , and after a single deconvolution step ( i.e . no error correction or predictive coding recurrent dynamics are involved at this stage ) : L = N−1∑ n=0 ‖ en − dn ‖22 , ( 3 ) where en is the output of the nth encoder after the first forward pass and dn is the estimated reconstruction of en via feedback/deconvolution ( from en+1 ) . For both the networks , after training the feedback deconvolution layers , we freeze all of the weights , and set the values of hyperparameters to βn = 0.8 , λn = 0.1 , and αn = 0.01 for all the encoders/decoders in Equation ( 2 ) . We also explore various strategies for further tuning hyperparameters to improve the results ( see Appendix A.7 for the chosen hyperparameter values ) . 2.3 Predify To facilitate and automate the process of adding the proposed predictive coding dynamics to existing deep neural networks , we have developed an open-source Python package called Predify . The package is developed based on PyTorch [ 26 ] and provides a flexible object oriented framework to convert any PyTorch-compatible network into a predictive network . While an advanced user may find it easy to integrate Predify in their project manually , a simple text-based user interface ( in TOML2 format ) is also provided to automate the steps . For the sake of improved performance and flexibility , Predify generates the code of the predictive network rather than the Python object . Given the original network and a configuration file ( e.g . 'config.toml ' ) that indicates the intended source and target layers for the predictive feedback , three lines of code are enough to construct the corresponding predictive network : from predify import predify net = # load PyTorch network predify ( net , './ config.toml ' ) # config file indicates the layers that # will act as outputs of encoders . 2https : //toml.io/en/ The Appendix A.1 provides further details on the package , along with a sample config file and certain default behaviours . Predify is an ongoing project available on GitHub3 under GNU General Public License v3.0 . Scripts for creating PVGG16 and PEfficientNetB0 from their feedforward instances and reproducing the results presented in this paper , as well as the pre-trained weights are also available on another GitHub repository4 .

Summary: The authors proposed a new algorithm to implement a popular neuroscience theory, i.e. predictive coding, in deep neural networks (DNN). The feedforward connection weights are pretrained for object recognition and the feedback connection weights are trained for image reconstruction, and then predictive coding introduces recurrent computations with the trained DNNs. The authors use two image recognition DNNs (VGG16 and EfficientNetB0) in experiments. They found that the recurrent computation brings improvement in object recognition and image/feature reconstruction over time given noisy inputs. This property of predictive coding improves DNN robustness to noisy examples and adversarial attacks. The authors also provide a python package to allow community to explore the predictive coding algorithm.