Convolutional Bipartite Attractor Networks

1 INTRODUCTION . Under ordinary conditions , human visual perception is quick and accurate . Studying circumstances that give rise to slow or inaccurate perception can help reveal the underlying mechanisms of visual information processing . Recent investigations of occluded ( Tang et al. , 2018 ) and empirically challenging ( Kar et al. , 2019 ) scenes have led to the conclusion that recurrent brain circuits can play a critical role in object recognition . Further , recurrence can improve the classification performance of deep nets ( Tang et al. , 2018 ; Nayebi et al. , 2018 ) , specifically for the same images with which humans and animals have the most difficulty ( Kar et al. , 2019 ) . Recurrent dynamics allow the brain to perform pattern completion , constructing a coherent neural state from noisy , incomplete , and intrinsically ambiguous evidence . This interpretive process is well matched to attractor networks ( ANs ) ( Hopfield , 1982 ; 1984 ; Krotov and Hopfield , 2016 ; Zemel and Mozer , 2001 ) , a class of dynamical neural networks that converge to fixed-point attractor states ( Figure 1a ) . Given evidence in the form of a static input , an AN settles to an asymptotic state—an interpretation or completion—that is as consistent as possible with the evidence and with implicit knowledge embodied in the network connectivity . We show examples from our model in Figure 1b . ANs have played a pivotal role in characterizing computation in the brain ( Amit , 1992 ; McClelland and Rumelhart , 1981 ) , not only perception ( e.g. , Sterzer and Kleinschmidt , 2007 ) , but also language ( Stowe et al. , 2018 ) and awareness ( Mozer , 2009 ) . We revisit attractor nets in light of modern deep learning methods and propose a convolutional bipartite architecture for pattern completion tasks with a novel training loss , activation function , and connectivity constraints . 2 BACKGROUND AND RELATED RESEARCH . Although ANs have been mostly neglected in the recent literature , attractor-like dynamics can be seen in many models . For example , clustering and denoising autoencoders are used to clean up internal states and improve the robustness of deep models ( Liao et al. , 2016 ; Tang et al. , 2018 ; Lamb et al. , 2019 ) . In a range of image-processing domains , e.g. , denoising , inpainting , and super-resolution , performance gains are realized by constructing deeper and deeper architectures ( e.g. , Lai et al. , 2018 ) . State-of-the-art results are often obtained using deep recursive architectures that replicate layers and weights ( Kim et al. , 2016 ; Tai et al. , 2017 ) , effectively implementing an unfolded-in-time recurrent net . This approach is sensible because image processing tasks are fundamentally constraint satisfaction problems : the value of any pixel depends on the values of its neighborhood , and iterative processing is required to converge on mutually consistent activation patterns . Because ANs are specifically designed to address constraint-satisfaction problems , our goal is to re-examine them from a modern deep-learning perspective . Interest in ANs seems to be narrow for two reasons . First , in both early ( Hopfield , 1982 ; 1984 ) and recent ( Li et al. , 2015 ; Wu et al. , 2018a ; b ; Chaudhuri and Fiete , 2017 ) work , ANs are characterized as content-addressable memories : activation vectors are stored and can later be retrieved with only partial information . However , memory retrieval does not well characterize the model ’ s capabilities : like its probabilistic sibling the Boltzmann machine ( Hinton , 2007 ; Welling et al. , 2005 ) , the AN is a general computational architecture for supervised and unsupervised learning . Second , ANs have been limited by training procedures . In Hopfield ’ s work , ANs are trained with a simple procedure—an outer product ( Hebbian ) rule—which can not accommodate hidden units and the representational capacity they provide . Recent explorations have considered stronger training procedures ( e.g. , Wu et al. , 2018b ; Liao et al. , 2018 ) ; however , as for all recurrent nets , training is complicated by the issue of vanishing/exploding gradients . To facilitate training and increase the computational power of ANs , we propose a set of extensions to the architecture and training procedures . ANs are related to several popular architectures . Autoencoding models such as the VAE ( Kingma and Welling , 2013 ) and denoising autoencoders ( Vincent et al. , 2008 ) can be viewed as approximating one step of attractor dynamics , directing the input toward the training data manifold ( Alain et al. , 2012 ) . These models can be applied recursively , though convergence is not guaranteed , nor is improvement in output quality over iterations . Flow-based generative models ( FBGMs ) ( e.g. , Dinh et al. , 2016 ) are invertible density-estimation models that can map between observations and latent states . Whereas FBGMs require invertibility of mappings , ANs require only a weaker constraint that weights in one direction are the transpose of the weights in the other direction . As currently formulated , energy-based models ( EBMs ) are feedforward density-estimation models that learn a mapping from input data to energies and are trained to assign low energy values to the data manifold ( LeCun et al. , 2006 ; Han et al. , 2018 ; Xie et al. , 2016 ; Du and Mordatch , 2019 ) . Whereas AN dynamics are determined by an implicit energy function , the EBM dynamics are driven by optimizing or sampling from an explicit energy function . In the AN , lowering the energy for some states raises it for others , whereas the explicit EBM energy function requires well-chosen negative samples to ensure it discriminates likely from unlikely states . Although the EBM and FBGM seem well suited for synthesis and generation tasks due to their probabilistic underpinnings , we show that ANs can also be used for generation ( maximum-likelihood completion ) tasks . 3 CONVOLUTIONAL BIPARTITE ATTRACTOR NETS ( CBANS ) . Various types of recurrent nets have been shown to converge to activation fixed points , including fully interconnected networks of asynchronous binary units ( Hopfield , 1982 ) and networks of continuousvalued units operating in continuous time ( Hopfield , 1984 ) . Most relevant to modern deep learning , Koiran ( 1994 ) identified convergence conditions for synchronous update of continuous-valued units in discrete time : given a network with state x , parallel updates of the full state with the standard activation rule , x← f ( xW + b ) , ( 1 ) will asymptote at either a fixed point or a limit cycle of 2 . Sufficient conditions for this result are : initial x ∈ [ −1 , +1 ] n , W = W T , wii ≥ 0 , and f ( . ) piecewise continuous and strictly increasing with limη→±∞ f ( η ) = ±1 . The proof is cast in terms of an energy function , E ( x ) = −1 2 xWxT − xbT + ∑ i ∫ xi 0 f−1 ( ξ ) dξ . ( 2 ) With f ≡ tanh , we have the barrier function : ρ ( xi ) ≡ ∫ xi 0 f−1 ( ξ ) dξ = ( 1 + xi ) ln ( 1 + xi ) + ( 1− xi ) ln ( 1− xi ) ( 3 ) To ensure a fixed point ( no limit cycle > 1 ) , asynchronous updates are sufficient because the solution of ∂E/∂xi = 0 is the standard update for unit i ( Equation 1 ) . Because the energy function additively factorizes for units that have no direct connections , parallel updates of these units still ensure non-increasing energy , and hence attainment of a fixed point . We adopt the bipartite architecture of a stacked restricted Boltzmann machine ( Hinton and Salakhutdinov , 2006 ) , with bidirectional symmetric connections between adjacent layers of units and no connectivity within a layer ( Figure 1c ) . We distinguish between visible layers , which contain inputs and/or outputs of the net , and hidden layers . The bipartite architecture allows for units within a layer to be updated in parallel while guaranteeing strictly non-increasing energy and attainment of a local energy minimum . We thus perform layerwise updating of units , defining one iteration as a sweep from one end of the architecture to the other and back . The 8-step update sequence for the architecture in Figure 1c is shown above the network . 3.1 CONVOLUTIONAL WEIGHT CONSTRAINTS . Weight constraints required for convergence can be achieved within a convolutional architecture as well ( Figure 1d ) . In a feedforward convolutional architecture , the connectivity from layer l to l + 1 is represented by weights W l = { wlqrab } , where q and r are channel indices in the destination ( l + 1 ) and source ( l ) layers , respectively , and a and b specify the relative coordinate within the kernel , such that the weight wlqrab modulates the input to the unit in layer l + 1 , channel q , absolute position ( α , β ) —denoted xl+1qαβ—from the unit x l r , α+a , β+b . If W l+1 = { wl+1qrab } denotes the reverse weights to channel q in layer l from channel r in layer l + 1 , symmetry requires that wlq , r , a , b =w l+1 r , q , −a , −b . ( 4 ) This follows from the fact that the weights are translation invariant : the reverse mapping from xl+1q , α , β to x l r , α+a , β+b has the same weight as from x l+1 q , α−a , β−b to x l r , α , β , embodied in Equation 4 . Implementation of the weight constraint is simple : W l is unconstrained , andW l+1 is obtained by transposing the first two tensor dimensions of W l and flipping the indices of the last two . The convolutional bipartite architecture has energy function : E ( x ) = − L−1∑ l=1 ∑ q xl+1q • ( W lq ∗ xl ) + L∑ l=1 ∑ q , α , β ρ ( xlqαβ ) − blq xlqαβ ( 5 ) where xl is the activation in layer l , bl are the channel biases , and ρ ( . ) is the barrier function ( Equation 3 ) , ‘ ∗ ’ is the convolution operator , and ‘ • ’ is the element-wise sum of the Hadamard product of tensors . The factor of 12 ordinarily found in energy functions is not present in the first term because , in contrast to Equation 2 , each second-order term in x appears only once . For a similar formulation in stacked restricted Boltzmann machines , see Lee et al . ( 2009 ) . 3.2 LOSS FUNCTIONS . Evidence provided to our Convolutional Bipartite Attractor Net ( hereafter , CBAN ) consists of activation constraints on a subset of the visible units . The CBAN is trained to fill-in or complete the activation pattern over the visible state . The manner in which evidence constrains activations depends on the nature of the evidence . In a scenario where all features are present but potentially noisy , one should treat them as soft constraints that can be overridden by the model ; in a scenario where the evidence features are reliable but other features are entirely missing , one should treat the evidence as hard constraints . We have focused on this latter scenario in our simulations , although we discuss the use of soft constraints in Appendix A . For a hard constraint , we clamp the visible units to the value of the evidence , meaning that activation is set to the observed value and not allowed to change . Energy is minimized conditioned on the clamped values . One extension to clamping is to replicate all visible units and designate one set as input , clamped to the evidence , and one set as output , which serves as the network read out . We considered using the evidence to initialize the visible state , but initialization is inadequate to anchor the visible state and it wanders . We also considered using the evidence as a fixed bias on the input to the visible state , but redundancy of the bias and top-down signals from the hidden layer can prevent the CBAN from achieving the desired activations . An obvious loss function is squared error , LSE = ∑ i ||vi − yi||2 , where i is an index over visible units , v is the visible state , and y is the target visible state . However , this loss misses out on a key source of error . The clamped units have zero error under this loss . Consequently , we replace vi with ṽi , the value that unit i would take were it unclamped , i.e. , free to take on a value consistent with the hidden units driving it : LSE = ∑ i ||ṽi − yi||2 . An alternative loss , related to the contrastive loss of the Boltzmann machine ( see Appendix B ) , explicitly aims to ensure that the energy of the current state is higher than that of the target state . With x = ( y , h ) being the complete state with all visible units clamped at their target values and the hidden units in some configuration h , and x̃ = ( ṽ , h ) being the complete state with the visible units unclamped , one can define the loss L∆E = E ( x ) − E ( x̃ ) = ∑ i f −1 ( ṽi ) ( ṽi − yi ) + ρ ( yi ) − ρ ( ṽi ) . We apply this loss by allowing the net to iterate for some number of steps given a partially clamped input , yielding a hidden state that is a plausible candidate to generate the target visible state . Note that ρ ( yi ) is constant and although it does not factor into the gradient computation , it helps interpret L∆E : when L∆E = 0 , ṽ = y . This loss is curious in that it is a function not just of the visible state , but , through the term f−1 ( ṽi ) , it directly depends on the hidden state in the adjacent layer and the weights between these layers . A variant on L∆E is based on the observation that the goal of training is only to make the two energies equal , suggesting a soft hinge loss : L∆E+ = log ( 1 + exp ( E ( x ) − E ( x̃ ) ) ) . Both energy-based losses have an interpretation under the Boltzmann distribution : L∆E is related to the conditional likelihood ratio of the clamped to unclamped visible state , and L∆E+ is related to the conditional probability of the clamped versus unclamped visible state : L∆E = − log p ( y|h ) p ( ṽ|h ) and L∆E+ = − log p ( y|h ) p ( ṽ|h ) +p ( y|h ) .

Summary: This paper presents an attractor network (AN) approach for pattern interpretation and completion. The authors propose a convolutional bipartite architecture consisting of visible (input and output) and hidden layers with weight constraints and squared and energy-based losses. To prevent vanishing/exploding gradients, temporal-difference method and leaky sigmoid activation function are exploited. Training is done by stochastic gradient descent. In experimental validation, the proposed model is able to reconstruct missing pixels in the images for bar task and supervised MNIST. And, in OMNIGLOT and CIFAR-10 experiments, the proposed approach outperforms its variants, and in super-resolution results, it outperforms the baselines.