A Biologically Inspired Visual Working Memory for Deep Networks

1 INTRODUCTION . Much of the current effort and literature in deep learning focuses on performance from a statistical pattern recognition perspective . In contrast , we go back to a biological motivation and look to build a model that includes aspects of the human visual system . The eminent computational neuroscientist David Marr posited that vision is composed of stages which lead from a two dimensional input to a three dimensional contextual model with an established notion of object ( Marr , 1982 ) . This higher order model is built up in the visual working memory as a visual sketchpad which integrates notions of pattern and texture with a notion of pose ( Baddeley and Hitch , 1974 ) . Visual attention models often draw inspiration from some of these concepts and perform well at various tasks ( Ablavatski et al. , 2017 ; Ba et al. , 2014 ; Gregor et al. , 2015 ; Jaderberg et al. , 2015 ; Mnih et al. , 2014 ; Sønderby et al. , 2015 ) . Inspired by vision in nature , visual attention corresponds to adaptive filtering of the model input , typically , through the use of a glimpsing mechanism which allows the model to select a portion of the image to be processed at each step . Broadly speaking , visual attention models exist at the crux of two key challenges . The first is to separate notions of pose and object from visual features . The second is to effectively model long range dependencies over a sequence of observations . Various models have been proposed and studied which hope to enable deep networks to construct a notion of pose . For example , transformational attention models learn an implicit representation of object pose by applying a series of transforms to an image ( Jaderberg et al. , 2015 ; Ablavatski et al. , 2017 ) . Other models such as Transformational Autoencoders and Capsule Networks harness an explicit understanding of positional relationships between objects ( Hinton et al. , 2011 ; Sabour et al. , 2017 ) . Short term memories have previously been studied as a way of improving the ability of Recurrent Neural Networks ( RNNs ) to learn long range dependencies . The ubiquitous Long Short-Term Memory ( LSTM ) network is perhaps the most commonly used example of such a model ( Hochreiter and Schmidhuber , 1997 ) . More recently , the fast weights model , proposed by Ba et al . ( 2016 ) provides a way of imbuing recurrent networks with an ability to attend to the recent past . From these approaches , it is evident that memory is a central requirement for any method which attempts to augment deep networks with the ability to attend to visual scenes . The core concept which underpins memory in neuroscience is synaptic plasticity , the notion that synaptic efficacy , the strength of a connection , changes as a result of experience ( Purves et al. , 1997 ) . These changes occur at multiple time scales and , consequently , much of high level cognition can be explained in terms of the interplay between immediate , short and long term memories . An example of this can be found in vision , where each movement of our eyes requires an immediate contextual awareness and triggers a short term change . We then aggregate these changes to make meaningful observations over a long series of glimpses . Fast weights ( Ba et al. , 2016 ) draw inspiration from the Hebbian theory of learning ( Hebb , 1949 ) which gives a framework for how this plasticity may occur . Furthermore , differentiable plasticity ( Miconi et al. , 2018 ) combines neural network weights with weights updated by a Hebbian rule to demonstrate that backpropagation can be used to learn a substrate over which the plastic network acts as a content-addressable memory . In this paper , we propose augmenting transformational attention models with a visual working memory in order to move towards two key goals . Firstly , we wish to understand if visual attention and working memory provide more than just increased efficiency and enable functions that can not otherwise be achieved . Secondly , we wish to understand and seek answers to some of the challenges faced when attempting to model such psychophysical concepts in deep networks . We demonstrate classification performance on MNIST ( LeCun , 1998 ) and CIFAR-10 ( Krizhevsky and Hinton , 2009 ) that is competitive with the state of the art and vastly superior to previous models of attention , demonstrating the value of a working memory . We then demonstrate that it is possible to learn this memory representation in an unsupervised manner by painting images , similar to the Deep Recurrent Attentive Writer ( DRAW ) network ( Gregor et al. , 2015 ) . Using this representation , we demonstrate competitive classification performance on MNIST with self supervised features . Furthermore , we demonstrate that the model can learn a disentangled space over the images in CelebA ( Liu et al. , 2015 ) , shedding light on some of the higher order functions that are enabled by visual attention . Finally , we show that the model can perform multiple tasks in parallel and how a visual sketchpad can be used to produce interpretable classifiers . 2 RELATED WORK AND THE PSYCHOLOGY OF VISUAL MEMORY . In this section we discuss related work on visual attention and relevant concepts from psychology and neuroscience which have motivated our approach . We will use the terms motivation and inspiration interchangeably throughout the paper to capture the notion that our decisions have been influenced by atypical factors . This is necessary as there are several facets to our approach which may seem nonsensical outside of a biological context . Attention in deep models can be broadly split into two types : hard attention and soft attention . In hard attention a non-differentiable step is performed to extract the desired pixels from the image to be used in later operations . Conversely , in soft attention , differentiable interpolation is used . The training mechanism differs between the two approaches . Early hard attention models such as the Recurrent Attention Model ( RAM ) and the Deep Recurrent Attention Model ( DRAM ) used the REINFORCE algorithm to learn an attention policy over non-differentiable glimpses ( Mnih et al. , 2014 ; Ba et al. , 2014 ) . More recent architectures such as Spatial Transformer Networks ( STNs ) ( Jaderberg et al. , 2015 ) , Recurrent STNs ( Sønderby et al. , 2015 ) and the Enriched DRAM ( EDRAM ) ( Ablavatski et al. , 2017 ) use soft attention and are trained end to end with backpropagation . The DRAM model and its derivatives use a two layer LSTM to learn the attention policy . The first layer is intended to aggregate information from the glimpses and the second layer to observe this information in the context of the whole image and decide where to look next . The attention models described thus far predominantly focus on single and multi-object classification on the MNIST and Street View House Numbers ( Goodfellow et al. , 2013 ) datasets respectively . Conversely , the DRAW network from Gregor et al . ( 2015 ) is a fully differentiable spatial attention model that can make a sequence of updates to a canvas in order to draw the input image . Here , the canvas acts as a type of working memory which is constrained to also be the output of the network . A common theme to these models is the explicit separation of pose and feature information . The Two Streams Hypothesis , suggests that in human vision there exists a ‘ what ’ pathway and a ‘ where ’ pathway ( Goodale and Milner , 1992 ) . The ‘ what ’ pathway is concerned with recognition and classification tasks and the ‘ where ’ pathway is concerned with spatial attention and awareness . In the DRAM model , a multiplicative interaction between the pose and feature information , first proposed in Larochelle and Hinton ( 2010 ) , is used to emulate this . The capacity for humans to understand pose has been studied extensively . Specifically , there is the notion that we are endowed with two methods of spatial understanding . The first is the ability to infer pose from visual cues and the second is the knowledge we have of our own pose ( Gibson , 1950 ) . Memory is generally split into three temporal categories , immediate memory , short term ( working ) memory and long term memory ( Purves et al. , 1997 ) . Weights learned through error correction procedures in a neural network are analogous to a long term memory in that they change gradually over the course of the models existence . We can go further to suggest that the activation captured in the hidden state of a recurrent network corresponds to an immediate memory . There is , however , a missing component in current deep architectures , the working memory , which we study here . We will later draw inspiration from the Baddeley model of working memory ( Baddeley and Hitch , 1974 ) which includes a visual sketchpad that allows individuals to momentarily create and revisit a mental image that is pertinent to the task at hand . This sketchpad is the integration of ‘ what ’ and ‘ where ’ information that is inferred from context and/or egomotion . 3 AN ASSOCIATIVE VISUAL WORKING MEMORY . Here we will outline our approach to augmenting models of visual attention with a working memory in order to seek a deeper understanding of the value of visual attention . In Section 3.1 we will detail the ‘ plastic ’ Hebb-Rosenblatt network and update policy that comprise our visual memory . Then in Section 3.2 we will describe the STAWM model of visual attention which is used to build up the memory representation for each image . The key advantage of this approach that will form the basis of our experimentation lies in the ability to project multiple query vectors through the memory and obtain different , goal-oriented , latent representations for one set of weights . In Section 3.3 we will discuss the different model ‘ heads ’ which use these spaces to perform different tasks . 3.1 HEBB-ROSENBLATT REDUX . We will now draw inspiration from various models of plasticity to derive a differentiable learning rule that can be used to iteratively update the weights of a memory space , during the forward pass of a deep network . Early neural network models used learning rules derived from the Hebbian notion of synaptic plasticity ( Hebb , 1949 ; Block , 1962 ; Block et al. , 1962 ; Rosenblatt , 1962 ) . The phrase ‘ neurons that fire together wire together ’ captures this well . If two neurons activate for the same input , we increase the synaptic weight between them ; otherwise we decrease . In a spiking neural network this activation is binary , on or off . However , we can obtain a continuous activation by integrating over the spike train in a given window ∆t . Although the information in biological networks is typically seen as being encoded by the timing of activations and not their number , this spike train integral can approximate it to a reasonable degree ( Dayan and Abbott , 2001 ) . Using this , we can create a novel short-term memory unit that can be integrated with other networks that are trained with backpropagation . Consider the two layer network shown in Figure 1 with an input layer LI and output layer LII , each of size M with weights W ∈ RM×M . For a signal vector e ∈ RM , propagating through LI , LII will activate according to some activation function φ of the projection We . The increase in the synaptic weight will be proportional to the product of the activations in the input and output neurons . We can model this with the outer product ( denoted ⊗ ) to obtain the increment expression W ( t+ 1 ) = W ( t ) + η [ e⊗ φ ( W ( t ) e ) ] − δ [ W ( t ) ] , ( 1 ) where at each step , we reduce the weights matrix by some decay rate δ and apply the increment with some learning rate η . This rule allows for the memory to make associative observations over the sequence of states such that salient information will become increasingly prevalent in the latent space . However , if we initialize the weights to zero , the increment term will always be zero and nothing can be learned . We could perhaps consider a different initialisation such as a Gaussian but this would be analogous to implanting a false memory , which may be undesirable . Instead , a solution is offered in the early multi layer perceptron models of Rosenblatt ( 1962 ) . Here , neurons in the first layer transmit a fixed amount , θ , of their input to a second layer counterpart as well as the weighted signal , as seen in Figure 1 . Combining both terms we obtain the final , biologically inspired learning rule that we will herein refer to as the Hebb-Rosenblatt rule , W ( t+ 1 ) = W ( t ) + η [ e⊗ φ ( W ( t ) e + θe ) ] − δ [ W ( t ) ] . ( 2 ) Note that if we remove the projection term ( We ) and set θ = 0 and φ ( x ) = x we obtain the term used in the fast weights model of Ba et al . ( 2016 ) which also shares some similarities with the rules considered in Miconi et al . ( 2018 ) . Analytically , this learning rule can be seen to develop high values for features which are consistently active and low values elsewhere . In this way the memory can make observations about the associations between salient features from a sequence of images .

Summary: this paper introduces a new  network architecture inspired by visual attentive working memory. The network consists of a recurrent components that generates glimpses of features from a CNN applied to the input (inspired by the DRAW network), and a working memory component that iterative stores memories using Hebbian mechanisms. The authors apply this network to classification tasks (MNIST) as well as using it as a generative model.