Convolutional Bipartite Attractor Networks
Authors: Michael L. Iuzzolino
Yoram Singer
Michael C. Mozer
Venue:  (2020)
Keywords: attractor network
recurrent network
energy function
convolutional network
image completion
super-resolution

Abstract: In human perception and cognition, a fundamental operation that brains perform is interpretation: constructing coherent neural states from noisy, incomplete, and intrinsically ambiguous evidence. The problem of interpretation is well matched to an early and often overlooked architecture, the attractor network---a recurrent neural net that performs constraint satisfaction, imputation of missing features, and clean up of noisy data via energy minimization dynamics. We revisit attractor nets in light of modern deep learning methods and propose a convolutional bipartite architecture with a novel training loss, activation function, and connectivity constraints. We tackle larger problems than have been previously explored with attractor nets and demonstrate their potential for image completion and super-resolution. We argue that this architecture is better motivated than ever-deeper feedforward models and is a viable alternative to more costly sampling-based generative methods on a range of supervised and unsupervised tasks.

Under review as a conference paper at ICLR 2020
CONVOLUTIONAL BIPARTITE ATTRACTOR NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
human perception and cognition, a fundamental operation that brains perform is interpretation: constructing coherent neural states from noisy, incomplete, and intrinsically ambiguous evidence. The problem of interpretation is well matched to early and often overlooked architecture, the attractor network- recurrent neural net that performs constraint satisfaction, imputation of missing features, and clean up of noisy data via energy minimization dynamics. We revisit attractor in light of modern deep learning methods and propose convolutional bipartite architecture with novel training loss, activation function, and connectivity constraints. We tackle larger problems than have been previously explored with attractor nets and demonstrate their potential for image completion and super-resolution. We argue that this architecture is better motivated than ever-deeper feedforward models and is viable alternative to more costly sampling-basee generative methods on range of supervised and unsupervised tasks.
INTRODUCTION
Under ordinary conditions, human visual perception is quick and accurate. Studying circumstances that give rise to slow inaccurate perception help reveal the underlying mechanisms of visual information processing. Recent investigations of occluded (Tang et al., 2018) and empirically challenging (Kar et al., 2019) scenes have led the conclusion that recurrent brain circuits can play a critical role in object recognition. Further, recurrence can improve the classification performance of deep nets (Tang et al., 2018; Nayebi et al., 2018), 8), specifically for the same images with which humans and animals have the most difficulty (Kar et al., 2019).
Recurrent dynamics allow the brain to perform pattern completion, constructing a coherent neural state from noisy, incomplete, and intrinsically ambiguous evidence. This interpretive process is well matched attractor networks (ANs) (Hopfield, 1982; 1984; Krotov and Hopfield, 2016; Zemel and Mozer, 2001), class of dynamical neural networks that converge to fixed-point attractor states (Figure la). Given evidence in the form static input, an AN settles to an asymptotic state-an interpretation completion-thar is consistent possible with the evidence and with implicit knowledge embodied in the network connectivity. We show examples from our model in Figure 1b.
ANs have played a pivotal role in characterizing computation in the brain (Amit, 1992; McClelland and Rumelhart, 1981), not only perception (c.g., Sterzer and Kleinschmidt, 2007). but also language (Stowe al., 2018) 8) and awareness (Mozer, 2009). We revisit attractor nets in light of modern deep learning methods and propose a convolutional bipartite architecture for pattern completion tasks with novel training loss, activation function, and connectivity constraints.
update order
W
(a)
(d)
W
layer avg. pool
hidden
layer
layer
Figure 1: (a) Hypothetical activation flow dynamics of an attractor net 2D state space; the contours depict an energy landscape. (b) top-to-bottom: original image, completion, and evidence. (c) Bipartite architecture with layer update order. (d) Convolutional architecture with average pooling.

---

problems, our goal is to re-examine them from Under review as a conference paper at ICLR 2020
2 BACKGROUND AND RELATED RESEARCH
Although ANs have been mostly neglected in the recent literature, attractor-like dynamics can be seen in many models. For example, clustering and denoising autoencoders are used to clean up internal states and improve the robustness of deep models (Liao et al., 2016; Tang et al., 2018; Lamb et al., 2019). In range of pnageeprocessing domains, e.g., denoising, inpainting, and super-resolution. performance gains are realized by constructing deeper and deeper architectures (c.g., Lai et al., 2018). Sfate-of-the-ar results are often obtained using deep recursive architectures that replicate layers and weights (Kim et al., 2016; Tai et al., 2017), effectively implementing an unfolded-ing time recurrent net. This approach is sensible because image processing tasks are fundamentally constraint satisfaction problems: the value of any pixel depends on the values of its neighborhood, and iterative processing is required to converge on mutually consistent activation patterns. Because ANs are specifically designed to address a modern deep-learning perspective.
Interest in ANs seems to be narrow for two reasons. First, in both early (Hopfield, 1982: 1984) and recent (Li et al., 2015; Wu et al., 201 8a;b; Chaudhuri and Fiete, 2017) work, ANs are characterized as eadresbla memories: activation vectors are stored and can later be retrieved with only partial information. However, memory retrieval does not well characterize the model's capabilities: like its probabilistic sibling the Boltzmann machine (Hinton, 2007; Welling al., 2005), the AN is general computational architecture for supervised and unsupervised learning, Second, ANs have been limited by training procedures. In Hopfield's work, ANs are trained with a simple procedure an outer product (Hebbian)
cannot accommodate hidden units and the representational capacity they provide. Recent explorations have considered stronger training procedures (e.g.. Wu et al., 201 8b; Liao et al., 2018); however, for all recurrent nets, training is complicated by the issue of aanishing/explodigg gradients. To facilitate training and increase the computational power of ANs, we propose set of extensions to the architecture and training procedures.
ANs are related to several popular architectures. Autoencoding models such as the VAE (Kingma and Welling, 2013) and denoising autoencoders (Vincent et al., 2008) can be viewed as approximating one step of attractor dynamics, directing the input toward the training data manifold (Alain et al., 2012). These models can be applied recursively, though convergence is not guaranteed, nor is improvement in output quality over iterations. Flow-based generative models (FBGMs) (e.g., Dinh et al., 2016) are invertible density-estimation models that can map between observations and latent states. Whereas FBGMs require invertibility of mappings, ANs require only a weaker constraint that weights in one direction are the transpose of the weights in the other direction.
As currently formulated, energy-based models (EBMs) are feedforward eessity-nstimation models that learn a mapping from input data to energies and are trained to assign low energy values to the data manifold (LeCun et al., 2006; Han et al., 2018; Xie et al., 2016; Du and Mordatch, 2019). Whereas AN dynamics are determined by an implicit energy function, the EBM dynamics are driven by optimizing or sampling from an explicit energy function. In the AN, lowering the energy for some states raises it for others, whereas the explicit EBM energy function requires well-chosen negative samples to ensure it discriminates likely from unlikely states. Although the EBM and FBGM seem well suited for synthesis and generation tasks due to their probabilistic underpinnings, we show that ANs can also be used for generation (maximum- wawtmuu--lkkllbaae likelihood completion) tasks.
3 CONVOLUTIONAL BIPARTITE ATTRACTOR NETS (CBANS)
Various types of recurrent nets have been shown to converge to activation fixed points, including fully interconnected networks of asynchronous binary units (Hopfield, 1982) and networks of continuous- valued units operating in continuous time (Hopfield, 1984). Most relevant to modern deep learning, Koiran (1994) identified convergence conditions for synchronous update of continuous-valuee units in discrete time: given a network with state x, parallel updates of the full state with the standard activation rule.
x + (( + b))
(1)
will asymptote at either a fixed point or a limit cycle of 2. Sufficient conditions for this result are: initial x E [11,+ W W wa 2 0, and f-) piecewise continuous and strictly increasing
2

---

Under review as a conference paper at ICLR 2020
with lim, + f(n) =1. The proof is cast in terms of energy function,
1 xW xb + Σ / f (S)
E(x)
(2)
With = lanh, we have the barrier function:
p(x.)
(E) = (1 1x):)[[[l 1 + 24) + (1- (n)1-x))n(11--- (n(1
(3)
To ensure fixed point (no limit cycle 1), asynchronous updates are sufficient because the solution of 0F/0r; = 0 is the standard update for unit (Fquation 1). Because the energy function additively factorizes for units that have no direct connections, parallel updates of these units still ensure non-increasing energy, and hence attainment of a fixed point.
We adopt the bipartite architecture stacked restricted Boltzmann machine (Hinton and Salakhut- dinov, 2006), with bidirectional symmetric connections between adjacent layers of units and no connectivity within a layer (Figure Ic). We distinguish between visible layers, which contain inputs and/or outputs of the net, and hidden layers. The bipartite architecture allows for units within layer to be updated in parallel while guaranteeing strictly non-increasing energy and attainment of local energy minimum. We thus perform layerwise updating of units, defining iteration sweep from one end of the architecture to the other and back. The 8-step update sequence for the architecture Figure lc is shown above the network.
3.1 CONVOLUTIONAL WEIGHT CONSTRAINTS
Weight constraints required for convergence be achieved within convolutional architecture as well (Figure Id). In a feedforward convolutional architecture, the connectivity from layer to + is represented by weights W ( juth where q and r are channel indices in the destination (1 + 1) and source (l) layers, respectively, and a and b specify the relative coordinate within the kemnl,, such that the weight w grab modulates the input to the unit in layer + 1, channel q, absolute position
(ar, denoted 1+1 from the unit of "ra+a,840"* If W (T denotes the reverse weights to channel in layer / from channel r in layer 1, symmetry requires that
w/ q,r.a,b [4 n..-a.-b
This follows from the fact that the weights are translation invariant: the reverse mapping from [+1 "q.0.8 to 8740 has the same weight as from x w/+1 20.00..9-b to " embodied in Equation 4. Implementation of the weight constraint is simple: W is unconstrained, and W is obtained by transposing the first two tensor dimensions of W and flipping the indices of the last two. The convolutional bipartite architecture has energy function:
E(x) = Σ
Σ
(Wi) xx)
Σ
Σ
p(a b/ 2
(5)
1=1
where æt is the activation in layer I, b' are the channel biases, and p(.) is the barrier function (Equation 3), 'x' is the convolution operator, and 'e' is the element-wise sum of the Hadamard product of tensors. The factor of 1 ordinarily found energy functions not present in the first term because, in contrast to Equation 2, each second-orde term in appears only once. For similar formulation stacked restricted Boltzmann machines, see Lee al. (2009).
3.2 LOss FUNCTIONS
Evidence provided to our Convolutional Bipartite Attractor Net (hereafter. CBAN) consists of activation constraints on subset of the visible units. The CBAN is trained to fill-in or complete the activation pattern over the visible state. The manner in which evidence constrains activations depends on the nature of the evidence. In scenario where all features are present but potentially noisy, one should treat them as soft constraints that can overridden by the model; in a scenario where the evidence features are reliable but other features are entirely missing, should treat the evidence as hard constraints.
We have focused on this latter scenario in simulations, although discuss the of soft constraints in Appendix A. For a hard constraint, we clamp the visible units to the value of the evidence, meaning that activation is set to the observed value and not allowed to change. Energy is
3

---

Under review as a conference paper at ICLR 2020
minimized conditioned on the clamped values. One extension clamping is to replicate all visible units and designate one set as input, clamped to the evidence, and one set as output, which serves as the network read out. We considered using the evidence to initialize the visible state, but initialization is inadequate to anchor the visible state and it wanders. We also considered using the evidence as a fixed bias on the input to the visible state, but redundancy of the bias and top-down signals from the hidden layer can prevent the CBAN from achieving the desired activations.
An obvious loss function is squared error, LSE = - |e; - yi where i is an index over visible units, v is the visible state, and y is the target visible state. However, this loss misses out on a key source of error. The clamped units have zero error under this loss. Consequently, we replace v with öy the value that unit i would take were it unclamped, i.c., free to take on value consistent with the hidden units driving it:
LSE = E, l|ē; 1.l..
An alternative loss, related to the contrastive loss of the Boltzmann machine (see Appendix B), explicitly aims to ensure that the energy of the current state is higher than that of the target state. With æ = (y, h) being the complete state with all visible units clamped at their target values and the hidden units in some configuration h, and ã (ë, being the complete state with the visible units unclamped, one can define the loss
CAE = E(z) E( E, (z,) - y4) + p(u)) p((;)-
We apply this loss by allowing the net to iterate for some number of steps given partially clamped input, yielding hidden state that is a plausible candidate generate the target visible state. Note that p(ys) is constant and although it does not factor into the gradient computation, it helps interpret CAE: g: when CAE 0, y. This loss is curious in that it is function not just of the visible state, but, through the term f- (ū:), it directly depends on the hidden state in the adjacent layer and the weights between these layers. variant on LAE is based on the observation that the goal of training is only to make the two energies equal, suggesting a soft hinge loss:
Las+ log (1 + exp( (ax) E())))
Both energy-based losses have an interpretation under the Boltzmann distribution: LAF is related to the conditional likelihood ratio of the clamped to unclamped visible state, and LaR+ is related to the conditional probability of the clamped versus unclamped visible state:
p(aik) LAF = - log p(0(k)
and LAFt = - llog p(y
3.3 PREVENTING VANISHING/EXPLODDII GRADIENTS
Although gradient descent is a more powerful method to train the CBAN than Hopfield's Hebb rule or the Boltzmann machine's contrastive loss, vanishing and exploding gradients are a concern as with any recurrent net (Hochreiter et al., 2001), particularly in the CBAN which may take 50 steps to fully relax. We address the gradient issue in two ways: through intermediate training signals and through soft sigmoid activation function.
The aim of the CBAN is to produce stable interpretation asymptotically. The appropriate way to achieve this is to apply the loss once activation converges. However, the loss can be applied prior to convergence well, essentially training the net to achieve convergence quickly as possible, while also introducing loss gradients deep inside the unrolled net. Assume a slability criterion 0 that determines the iteration 1x at which the net has effectively converged:
t min, mmax, (xr(t) - ., - 1)] < <
Training can be logically separated into pre- and post-convergence phases, which we will refer to as transient and stationary. In the stationary phase. the Almeida/Pineda algorithm (Pineda, 1987; Almeida, 1987) leverages the fact that activation is constant over iterations, permitting a computationally efficient gradient calculation with low memory requirements. In the transient phase, the loss can be injected each step, which is exactly the -efforal-differencee method TD(1) (Sutton, 1988). Casting training as temporal-din -iffereeee learning, one might consider other values of A in TD(A); for example, TD(0) trains the model to predict the visible state at the next time step, encouraging the model to reach the target state quickly as feasible while not penalizing it for being unable to get to the target immediately.
Any of the losses, LSE, LAE, and LAEt, can be applied with a weighted mixture of training in the stationary and transient phases. Although we do not report systematic experiments in this article, we consistently find that transient training with À is efficient and effective weighted mixtures
4

---

Under review as a conference paper at ICLR 2020
including stationary- caaponaarryypph--oooe phass--ollyyy training, and that À outperforms any À < likely due to moving targets during training. We thus conduct all simulations with transient-phass training and = 1.
We propose second method of avoiding vanishing gradients specifically due to sigmoidal activation functions: a leaky sigmoid, analogous to a leaky ReLU, which allows gradients to propagate through the net more freely. The leaky sigmoid has activation and barrier functions
a(2 + 1
-1
2 2<-1
2
+ (1
a) + 2z)]
if x < -1
f(z)
=1 p(x)
if - 1 1 x 111.
a(2 - +1+11 2>1
x +11 -2)1-)x)) 2x)]
if x >
Parameter a specifies the slope of the piecewise linear function outside the |a| interval. As + 0, loss gradients become flat and the CBAN fails to train well. As a - 1, activation magnitudes can blow up and the CBAN fails to reach fixed point. In Appendix C, we show that convergence to a fixed point guaranteed when | < where [[][2,o max; ||w, practice, we have found that restricting W is unnecessary and = 0.2 works well.
SIMULATIONS
We report on a series of simulation studies of increasing complexity. First, we explore fully connected bipartite attractor net (FBAN) on bar imputation task and then supervised MNIST image completion and classification. Second, we apply CBAN to unsupervised image completion tasks on Omniglot and CIFAR- 10 and compare CBAN to CBAN-variants and denoising- VAEs. Lastly, we revision CBAN for the task of super-resolution and report promising results against competing models, such as DRCN and LapSRN. Details of architectures, parameters, and training are in Appendix D. All models trained via SGD (back propagation through time).
4.1 BAR TASK
We studied simple inference task on partial images that have exactly one correct interpretation. Images are 5 x 5 binary pixel arrays consisting of two horizontal bars or two vertical bars. Twenty distinct images exist, shown in the top row of Figure 2. A subset of pixels is provided as evidence; examples are shown in the bottom row of Figure 2. The task is fill in the masked pixels. Evidence generated such that only one consistent completion exists. In some cases, bar must be inferred without any white pixels as evidence (e.g., second column from the right). In other cases, the local evidence is consistent with both vertical and horizontal bars (e.g., first column from left).
An FBAN with one layer of 50 hidden units sufficient for the task. Evidence generated randomly each trial. Evaluating 10k random states, the model is 99.995% correct. The middle in Figure shows the FBAN response after one iteration. The net comes close to performing the task in single shot, but after second iteration of clean up and the asymptotic state is shown in the top row.
Figure shows some visible-hidden weights learned by the FBAN. Each 5 x 5 array depicts weights to/from one hidden unit. Weight sign and magnitude indicated by coloring and area of the square, respectively. Units appear to select one row and one column, either with the same or opposite
completion
iteration 1
evidence
Figure 2: Bar task: Input consists of 5 5 pixel arrays with the target being either two rows two columns pixels present.
Figure 3: Bar task: Weights between visible and first hidden layers
5

---

Under review as a conference paper at ICLR 2020
target
0
2
2
3
3
4
4
5
5
6
6
7
8
8
9
+
completion
2
2
3
3
4
4
5
3
6
6
7
7
8
8
9
evidence
2
5
8
4
2
dream
0
/
2
3
3
4
4
5
5
6
6
7
7
2
8
9
9
Figure 4: MNIST completions. Row I: target test examples, with class label coded in the bottom row.
Row 2: completions produced by the FBAN. Row 3: Evidence with masked regions (including class
labels) in red. Row 4: the top-down 'dream' state produced by the hidden representation.
polarity. Same-polaries weights within a row or column induce coherence among pixels. Opposite- polarity weights between a row and column allow the pixel at the intersection to activate either the row/column depending on the sign of the unit's activation.
4.2 SUPERVISED MNIST
We trained FBAN with two hidden layers on a supervised version of MNIST in which the visible state consists of 28 x 28 array for an MNIST digit and an additional vector to code the class label. For the sake of graphical convenience, we allocate 28 units to the label, using the first 20 by redundantly coding the class label in pairs of units, and ignoring the final 8 umits. Our architecture had 812 inputs, 200 units in the first hidden layer, and 50 units in the second. During training, all bits of the label were masked as well as one-third of image pixels. The image was masked with thresholded Perlin coherent noise (Perlin, 1985), which produces missing patches that are far more difficult to fill in than the isolated pixels produced by Bernoulli masking.
Figure shows evidence provided to FBAN for 20 random test set items the third row. The red masks indicate unobserved pixels; the other pixels are clamped in the visible state. The unobserved pixels include those representing the class label, coded in the bottom row of the pixel array. The top row of the Figure shows the target visible representation, with class labels indicated by the isolated white pixels. Even though the training loss treats all pixels as equivalent, the FBAN does learn to classify unlabeled images. On the test set, the model achieves classification accuracy of 87.5% on Perlin-masked test images and 89.9% on noise-free test images. Note that the 20 pixels indicating class membership are no different than any other missing pixels in the input. The model learns to classify by virtue of the systematic relationship between images and labels. We can train the model with fully observed images and fully unobserved labels, and performance like that of any fully-connected MNIST classifier, achieving an accuracy of 98.5%.
The FBAN does an excellent job of filling in missing features in Figure and further examples in Appendix F. The FBAN's interpretations of the input seem to be respectable in comparison to other recent recurrent associative memory models (Figures Sa,b). We mean no disrespect of other research efforts-which have very different foci than ours-but merely wish to indicate we are obtaining state-of-the-att results for associative memory models. Figure Se shows some weights between visible and hidden units. Note that the weights link image pixels with multiple digit labels. These weights stand apart from the usual hidden representations found in feedforward classification networks.
4.3 UNSUPERVISED OMNIGLOT
We trained a CBAN with the Omniglot data set (Lake al., 2015) which consists of instances of 1623 characters from 50 different alphabets. The CBAN has one visible layer containing the character image, 28 28 x 1, and three successive hidden layers with dimensions and 7 x 7 x 256, all with average pooling between the layers and filters of size 3 x 3. Other network parameters and training details are presented in Appendix D. To vary the masking, we used random square patches of diameter 3-6, which remove average 30% of the white pixels in the image.
x 28 x 128, 14 x 14 256
We compared our CBAN to variants with critical properties removed: one without weight symmetry (CBAN-asym) and one in which the TD(1) training procedure is substituted for standard squared loss at the final step (CBAN-noTD)) We also compare to a convolutional denoising VAE (CD- VAE), which takes the masked image input and outputs the completion. The CBAN with symmetric

---

Under review as a conference paper at ICLR 2020
Target
0
V
E
3
4
5
200
ARL
CBAN
0
2
V
E
n
5
CBAN-asym
0
77
6
A
t
d
6
5
8.5000
CBAN-noTD
2
6
A
1
t
F<
5
CD-VAE
t
3
Chil CBi (MA
Figure 5: Omniglot image completion comparison examples (left) and quantitative results (right). The top two rows of the examples show the target image and the evidence provided to the model (with missing pixels depicted in red), respectively. The subsequent rows show the image completions produced by CBAN, CBAN-asym, CBAN-noTD, and the denoising VAE. The quantitative measures evaluate each model on PSNR and SSIM metries; black lines indicate + standard error of the mean.
weights reaches a fixed point, whereas CBAN-asym appears to attain limit cycles of 2-10 iterations. Qualitatively, CBAN produces the best image reconstructions (Figure 5; additional completions in Appendix F). CBAN-asym and CBAN-noTD tend to hallucinate additional strokes; and CBAN-noTD and CD-VAE produce less crisp edges. Quantitatively. we assess models with two measures of reconstruction quality, peak signal-to-noie ratio (PSNR) and structural similarity (SSIM, Wang al., 2004); larger is better on each measure. CBAN strictly superior to the alternatives on both measures (Figure 5, right panel). CBAN completions are not merely memorized instances; the CBAN has learned structural regularities of the images, allowing to in big gaps in images that-with the missing pixels-are are typically uninterpretable by both classifiers and humans.
4.4 UNSUPERVISED CIFAR-10
We trained CBAN with one visible and three hidden layers on CIFAR-10 images. The visible layer is the size of the input image, 32 x 32 x 3. The successive hidden layers had dimensions 32 x 32 x 40, 16 x 16 120, and 8 x 440, all with filters of size x 3 and average pooling between the hidden layers. Further details of architecture and training can be found in Appendix D. Figure shows qualitative and quantitative comparisons of alternative models. Here, CBAN-asym performs about the same CBAN. However, CBAN-asym typically attains bi-phasic limit cycles, and CBAN-asym sometimes produces splotchy artifacts in background regions (e.g., third image from left). CBAN-noTD and the CD-VAE clearly inferior to CBAN. Additional CBAN image completions can be found in Appendix F.
4.5
SUIPER-ESSOLUTION
Deep learning models have proliferated in many domains of image processing, perhaps none more than image super-resolution, which is concerned with recovering a high-resolution image from
30
CBA
pm
5.077
0.753
Figure 6: CIFAR-10 image completion comparison examples (left) and quantitative results (right). Layout identical to that of Figure 5.
7

---

Under review as a conference paper at ICLR 2020
Sct5
Sct14
BSDIOD
Urban 100
Algorithm
PSNR SSIM
PSNR SSIM
PSNR SSIM
PSNR SSIM
Bicubic (baseline)
32.21 0.9921
29.21 /0911
28.67 0.8
25.63 0.827
DRCN (Kim et al., 2016)
37.63/0.955
32.94 /0,913
31.85 / 0.894
30.76 / 0,913
LapSRN (Lai et al., 2018)
37.52 0,959
33.08 0.913
31.80 / 0.895
30.41 0.910
CBAN (ours)
34.18/ / 0.947
30.79 / 0.953
30.12 0.872
27.49 / 0.915
Table I:
Quantitative benchmark presenting average PSNR/SSIMs for scale factor x on four test sets. Red indicates superior performance of CBAN. CBAN consistently outperforms baseline Bicubic.
Ground
Bicubic
Ground
Scubii
crubiic
Bcubic
Figure 7: Examples of super-resolution, with the columns in given image group comparing high- resolution ground truth, CBAN, and bicubic interpolation (baseline method).
low-resolution image. Many specialized architectures have been developed, and although common test data sets exist, comparisons are not as simple as one would hope due subtle differences in methodology. (For example, even the baseline method, bicubic interpolation, yields different results depending on the implementation.) We set out to explore the feasibility of using CBANs for super-resoluee resputresolution. Our architecture processes 40 x 40 color image patches, and the visible state included both the low- and high-resolution images, with the low-resolution version clamped and the high-resolution version read out from the net. Details can be found in Appendix D.
Table presents two measures of performance, SSIM and PSNR, for the CBAN and various published alternatives. CBAN beats the baseline, bicubic interpolation, both measures, and performs well on SSIM against some leading contenders (even beating LapSRN and DRCN on Sei14 and Urban but poorly PSNR. It is common for PSNR and SSIM to be in opposition: SSIM rewards crisp edges, PSNR rewards averaging toward the mean. The border sharpening and contrast enhancement that produce good perceptual quality and high SSIM score (see Figure 7) are due to the fact that CBAN comes to an interpretation of the images: it imposes edges and textures in order to make the features mutually consistent. We believe that CBAN warrants further investigation for super- resolution; regardless of whether it becomes the winner in this competitive field, one can argue that it performing different type computation than feedforward models like LapSRN and DRCN.
DISCUSSION
This article revisits attractor nets, which are traditionally fully interconnected RNNs trained with recurrent back propagation Liao et al. (2018) the Hebbian rule (Hopfield, 1984). Our key contribu- tion is to endow nets with a combination of properties which allows them to tackle difficult image completion problems; these properties include: convolutional weight constraints, novel loss functions, and methods for preventing vanishing/exploding gradients. In comparison to recent published results on image completion with attractor networks, our CBAN produces far more impressive results (see Appendix, Figure 8, for contrast). The computational cost and challenge of training CBANs is no greater than those of training deep feedforward nets. CBANs seem to produce crisp images, on par with those produced by generative (e.g., energy- and flow-based) models. CBANs have potential to be applied in many contexts involving data interpretation, with the virtue that the computational resources they bring to bear task is dynamic and dependent on the difficulty of interpreting given input. Although this article has focused on convolutional networks that have attractor dynamics between levels of representation, fundamentally feedforward with attractor dynamics within a level. Our current research explores this variant of the CBAN biologically plausible account of intralaminar lateral inhibition.
have recently recognized the value of architectures that are
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
G. Alain, Y. Bengio, and S. Rifai. Regularized auto-encoders estimate local statistics. CoRR, abs/1211.42444 2012.
L. B. Almeida. A learning rule for asynchronous perceptrons with feedback in a combinatorial environment. In JEEE First International Conference Neural Networks, pages 608-18. IEEE Press, San Diego, CA, 1987.
D. J. Amit. Modeling brain function: The world of attractor neural networks. Cambridge University Press, Cambridge, England, 1992.
M. Bevilacqua, A. Roumy, C. Guillemot, and M. L. Alberi-Morel, Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In British Machine Vision Conference. BMVA press, 2012.
R. Chaudhuri and Fiete. Associative content-addressabll networks with exponentially many robust stable states. arXiv preprint arXiv: N-bio.NC, 2017.
L. Dinh, J. Soik-Dicks, and S. Bengio. Density estimation using Real NVP. arXiv preprini arXiv:1605.08803 cs.LG, 2016.
Y. Du and I. Mordatch. Implicit generation and generalization in energy based models. arXiv preprint arXiv: 1903.08899 cs...G, 2019.
T. Han, E. Nijkamp. X. Fang, M. Hill, S.-C. Zhu, and Y. Nian Wu. Divergence triangle for joint training of generator model, energy-based model, and inference model. arXiv e-prints, art. 12Xi:1812.110907 Dec 2018.
G. E. Hinton. Boltzmann machine. Scholarpedia, 2(5):1668, 2007. doi: 10.42499scholamedia..too 1668. revision #91076.
G. E. Hinton and R. R. Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 313(5786):504-507 2006.
S. Hochreiter, Y. Bengio, and P. Frasconi. Gradient flow in recurrent nets: The difficulty of learning long-term dependencies. In Kolen and S. Kremer, editors, Field Guide to Dynamical Recurrent Networks, IEEE Press, 2001.
J. Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, 79(8):2554-25 2558, 1982. doi: 10. 10.1073/pnas.77 779 8.2554.
J. J. Hopfield. Neurons with graded response have collective computational properties like those of two-state neurons. Proceedings of the National Academy of Sciences, 81(10))00883309 1984.
J.B. Huang, A. Singh, and N. Ahuja. Single image super-resolution from transformed self-exemplars. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5197-5206, 2015.
D. Im, S. Ahn, R. Memisevic, and Y. Bengio. Denoising criterion for variational auto-encoding framework. In Proceedings of the Thirty-Firs/ AAAI Conference on Artificial Intelligence, AAAI' pages 2059-2065. AAAI Press, 2017. URL http: /dl. acm. org/ cwt.ttii.tiioncoo cfm?id= 3298483 3298537.
K. Kar, J. Kubilius, K. Schmidt, E. B. Issa, and J. DiCarlo. Evidence that recurrent circuits critical to the ventral stream's 's execution of core object recognition behavior. Nature neuroscience, 2019 Apr 29 2019. ISSN 1546- 1726. doi: 00.1038/441933-019000 nature.. com/ attp//aatiicceeeoeee / /41///4119331 01015593---00000 0011-0992255,
URL https: /www.
J. Kim, J. Kwon Lee, and K. Mu Lee. Deeply-recursive convolutional network for image super- resolution. In Computer Vision and Pattern Recognition, pages 1637-1645, 06 2016. doi: 10.1 10.1109/CVPR.2016.1195

---

Under review as a conference paper at ICLR 2020
D. P. Kingma and M. Welling. Auto-encoding variational Bayes. arXiv preprint arXiv:1312.6114, 2013.
P. Koiran. Dynamics of discrete time, continuous state Hopfield networks. Neural Computation, 6(3): 459 468, 1994. doi: 10.1162/neco.1994
URL httph: htt.:ttp..//.....r///........
1994. 6. 3. 459,
D. Krotov and J. J. Hopfield. Dense associative memory for pattern recognition. In Advances in Neural Information Processing Systems, pages 1172-1180, 2016.
W.-S. Lai, J.-B. Huang, N. Ahuja, and M.-H. Yang. Fast and accurate image super-resolution with deep Laplacian pyramid networks. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2018.
B. M. Lake, R. Salakhutdinov, and J. B. Tenenbaum. Human-Level Concept Learning through Probabilistic Program Induction. Science, 350(6266 2015. ISSN 0036-8075. doi: 10.1 I Satosciience.aa335000
A. Lamb, J. Binas, A. Goyal. S. Subramanian, Mitliagkas, D. Kazakov, Y. Bengio, and M. C. Mozer. State-reification networks: Improving generalization by modeling the distribution of hidden representations.
K. Chaudhuri and R. Salakhutdinov. editors, Proceedings of the 36th International Conference on Machine Learning, volume 96 of Proceedings of Machine Learning Research, Long Beach, CA, 2019.
Y. LeCun, S. Chopra, R. Hadsell, M. Ranzato, and F.J. Huang. A tutorial on energy-based -basd learning. In G. Bakir, T. Hofman, B. Schölkopf, A. Smola, and B. Taskar, editors, Predicting Structured Data. MIT Press, Boston, MA, 2006.
H. Lee, R. Grosse, R. Ranganath, and A. Y. Ng. Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In Proceedings of the 26th International Conference Machine Learning, 2009.
G. Li, K. Ramanathan, N. Ning, L. Shi, and C. Wen. Memory dynamics in attractor networks. Computational Intelligence and Neuroscience, 2015.
R. Liao, A. Schwing, R. 7emel, and R. Urtasun. Learning deep parsimonious representations. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29, pages 5076-5084. Curran Associates, Inc., 2016.
R. Liao, Y. Xiong, E. Fetaya, I.. Zhang, K. Yoon, X. Pitkow, R. Urtasun, and R. Zemel. Reviving and improving recurrent back pook-proppggttin. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 3082-3091, Stockholmsmässann Stockholm Sweden, 10-15 Jul 2018. PMLR. html.
www.pppwessssseeee htt.:/eedddn..........eesseeeeee URL http:// http:://proceedingg mll
D. Martin, C. Fowlkes, D. Tal, J. Malik, et al. database human segmented natural images and its application evaluating segmentation algorithms and measuring ecological statisties. In Proceedings of the International Conference on Computer Vision (ICCV), 2001.
J. L. McClelland and D. E. Rumelhart. An interactive activation model of context effects in letter perception: I. an account of basic findings. Psychological Review, 88(5):375 1981.
M. C. Mozer. Attractor networks. P. Wilken, A. Cleeremans, and Bayne, editors, Oxford Companion Consciousness,, pages 86-89. Oxford University Press, Oxford, UK, 2009.
A. Nayebi, D. Bear, J. Kubilius, K. Kar, S. Ganguli, D. Sussillo, J. DiCarlo, and D. L. Yamins. Task- driven convolutional recurrent models of the visual system. In Advances in Neural Information Processing Systems, pages 5290-5301, 2018.
K. Perlin. An image synthesizer. In Proceedings of the 12th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH '85, pages 287-296, New York, NY, USA, 1985. ACM. ISBN 0-89791-166-0. doi: 10.1145/35334.335527 URL http:// doi.. acm. org/10. 110. 11 325247.
10

---

Under review as a conference paper at ICLR 2020
F. J. Pineda. Generalization of pack-propagation to recurrent neural networks. Letters, 59:2229 2232, 1987.
Physical Review
R. Salakhutdinov and G. Hinton. Deep boltzmann machines, In Artificial intelligence and statistics, pages 448-455, 2009.
Sterzer and A. Kleinschmidt. neural basis for inference in perceptual ambiguity, Proceedings of the National Academy of Sciences, 104(1):323-32) -2222,222,,,0 2007. ISSN 0027-8424. doi: 10. 10.1073/pnas. 0609006104. URL https: /www. pnas. org/ content /tteeenn/111///222
L. A. Stowe, E. Kaan, L. Sabourin, and R. C. Taylor. The sentence wrap-up dogma. Cognition, 176: 232-247, 2018.
R. S. Sutton. Learning predict by the methods of temporal differences. Machine Learning, 3:9 1988. doi: 10. 10.1007/FF00115009 URL ht https:://// 1ttp://00000011100000000/0001110
Y. Tai, J. Yang, and X. Liu. Image super-resolution via deep recursive residual network. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2790-2798, 2017.
H. Tang, M. Schrimpf, W. Lotter, C. Moerman, A. Paredes, O. Caro, W. Hardesty, D. Cox, and G. Kreiman. Recurrent computations for visual pattern completion. Proceedings of the National Academy of Sciences, 15(35):8835-58)) 8840, 2018.
P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th International Conference on Machine Learning, ICML '08, pages 1096-1 103, New York, NY, USA, 2008. ACM. ISBN 978-1-60558- 205-4. doi: 1390294.
URL http: /dot:://oi... doi acm. org/lo.g./ooooooo... 1141111451110 1tt///...14/11301111
Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):000661,, April 2004. doi: 10. 101110911.220038119828
M. Welling, M. Rosen-zvi, and G. E. Hinton. Exponential family harmoniums with application to information retrieval. In L. K. Saul, Y. Weiss, and L. Bottou, edi- tors, Advances in Neural Information Processing Systems 17, pages 1481-1488. MIT Press, 2005. URL http::////ppp nips oc httpp//c.c.ppppppp//eeeeeaoaaaaa harmoniuma wwwtwwithhhae htt.://iww......a.pppoaaaoooooa.
Y. Wu, G. Wayne, A. Graves, and T. Lillicrap. The Kanerva machine: generative distributed memory. In International Conference Learning Representations, 2018a. URL https:// openreview belerrvvewnnet/Eeeee net/forummt id=s1H1A- 2A2.
Y. Wu, G. Wayne, K. Gregor, and T. Lillicrap. Learning attractor dynamics for generative memory. In S. Bengio. H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 9379-9388, Curran Associates, Inc., 2018b. URL http: /ttt::://paaeeeee /wtpppaappssssaaapeeeeeeaaaesees nips cc/ wwtp/pape///eee / 8149-144999 weweeeennnnnntrranooee for -ee-se---oo--eeeeeeeeeeeeeeeeeee
J. Xie, Y. Lu, S.-C. Zhu, and Y. N. Wu. A theory of generative ConvNet. In Proceedings of the 33rd International Conference on International Conference on Machine Learning Volume 48, ICML'16. 16, pages 2635-2644. 2. JMLR.org, 2016. URL http://tt//// /dl. acm, wrggcitation. a momiidd=3453900 3045668.
Yang, Wright, T. S. Huang, and Y. Ma. Image super-resolution via sparse representation. JEEE transactions on image processing, 19(1 (9(1)):881-28733 2010.
R. S. Zemel and M. C. Mozer. Localist attractor networks. Neural Computation, 1365):1055-1064, 2001. doi: 10. 111620899760011111144449
R. Zeyde, M. Elad, and M. Protter. On single image scale-up using spanse-ropresenttaions.. In Proceedings of the International conference on curves and surfaces, pages 7-17730. Springer, 2010.
11

---

Under review as a conference paper at ICLR 2020
APPENDIX
A USING EVIDENCE
The CBAN is probed with observation-a constraint the activation of subset of visible units, For any visible unit, we must specify how an observation is used to constrain activation. The possibilities include:
The unit is clamped, meaning that the unit activation is set to the observed value and is not allowed to change. Convergence is still guaranteed, and the energy is minimized conditional the clamped value. However, clamping unit has the disadvantage that any error signal back propagated to the unit will be lost (because changing the unit's imput does not change its output).
The unit is initialized to the observed value, instead of O. This scheme has the disadvantage that activation dynamics can cause the network to wander away from the observed state. This problem occurs in practice and the consequences are so severe it is not viable approach.
In principle, we might try an activation rule which sets the visible unit's activation to be a convex combination of the observed value and the value that would be obtained via activation dynamics: a x observed + (1 - a) X fnet input). With a = 1 this is simply the clamping scheme; with a 0 and appropriate start state, this is just the initialization scheme.
The unit has external bias proportional to the observation. In this scenario, the net input to a visible unit is:
(6) where Ej x observation. The initial activation can be either or the observation. One concern with this scheme is that the ideal input unit will depend whether not the unit has this additional bias. For this reason the magnitude of the bias should probably be small. However, in order to have an impact, the bias must be larger.
24 - f(ow! + b, + cs),
We might replicate all visible units and designate one set for input (clamped) and one set for output (unclamped). The input is clamped to the observation (which may be zero). The output is allowed to settle. The hidden layer(s) would synchronize the inputs and outputs, but it could handle noisy inputs, which isn't 't possible with clamping. Essentially, the input would serve bias, but on the hidden units, not the inputs directly.
In practice, we have found that external biases work but are not as effective as clamping. Partial clamping with 0 a < has partial effectiveness relative clamping. And initialization is not effective; the state wanders from the initialized values. However, the replicated-visible scheme seems very promising and should be explored further.
B Loss FUNCTIONS
The training procedure for a Boltzmann machine aims maximize the likelihood of the training data, which consist of set of observations over the visible units. The complete states in a Boltzmann machine occur with probabilities specified by
p(x) x e E(()/TT
where T is computational temperature and the likelihood of a visible state is obtained by marginal- izing over the hidden states. Raising the likelihood of a visible state is achieved by lowering its energy.
The Boltzmann machine learning algorithm has a contrastive loss: it tries to maximize the energy of states with the visible units clamped to training observations and minimize the energy of states with the visible units unclamped and free to take on whatever values they want. This contrastive loss is an example of an energy-based loss, which expresses the training objective in terms of the network energies.
In our model, we will define energy-based loss via matched pairs of states: is state with the visible units clamped to observed values, and ğ is a state which the visible units are unclamped, i... they are free take values consistent with the hidden units driving them. Although & could be
12

---

Under review as a conference paper at ICLR 2020
any unclamped state, it will be most useful for training if it is related to (i.e., it is good point of contrast). To achieve this relationship, we propose to compute (æt, x) pairs by:
1. Clamp some portion of the visible units with a training example.
2. Run the net some iteration, at which point the full hidden state h. (The point of this step is to identify a hidden state that is a plausible candidate to generate the target visible state.)
3. Set a to be the complete state in which the hidden component of the state is h and the visible component the target visible state.
4. Set Ø to be the complete state in which the hidden component of the state is h and the visible component is the fully unclamped activation pattern that would be obtained by propagating activities from the hidden units to the (unclamped) visible units.
Note that the contrastive pair at this iteration, (æ;, w:), are states close the activation trajectory that the network is following. We might train the net only after it has reached convergence, but we ve found that defining the loss for every iteration up until convergence improves training performance.
B. LOSS 1: THE DIFFERENCE OF ENERGIES
LAF = E(x) -(())
+ Σ C/
- -2 T -bat + Σ ;
x
2 F Wxx - bx +
f (E)
Σ (wiw + b,)(0: v;)) + 1"
1/) --)(E)
(
00
(6:) 0;) + p(v;)) p(()))
1 (1 - 5) In(1 s) with p(s) = (1 + 8) In(1 + +s +
2
This reduction depends on æ and t sharing the same hidden state, a bipartite architecture in which visible and hidden interconnected, all visible-to-visibe connections zero, tanh activation function, f, for all units, and symmetric weights.
The goal of this loss is move á toward Once they become identical, adjusting their relative energies is not beneficial. Thus, we propose using a hinge version of the loss:
LAF = max(0, E(x) E(Ø))
B.2 LOss 2: THE CONDITIONAL PROBABILITY OF CORRECT RESPONSE
This loss aims to maximize the log probability of the clamped state conditional on the choice between unclamped and clamped states, Framed loss, have negative log likelihood:
LAE+ In P(x V Z)
p(æ))
In
p(क) + p(x)
[(xx) E(E)
In exp
T
The last step attained using the Boltzmann distribution (Equation 7).
13

---

Under review as a conference paper at ICLR 2020
C PROOF OF CONVERGENCE OF CBAN WITH LEAKY SIGMOID ACTIVATION FUNCTION
f(o) 2(0-)+))) +1
x f(W +6) xE
= dl x -1) +1
t uppeed elomast wine
L (8uN88 + +( +1
Ca(211)
---1
s d ( +
f()==
-/
we want the faot te cm to be at most leme
ER
< (2--))+1
where
oL* -1
fosin
l1bbla $
1b1=1
d (1+m)) (omm)+11 1m
d.r 6 (i-dr) m dr 21 =7 d
1 1g1 $14
whene mao
Ny, sv
may, Nag
Ther for 1x lsm to bold we need:
Ia fact thar is degrcz of freedom heee so we can simpl use cearel as the ony prom in the andlyois loy as we olso ie pa storize accortigyy
1l ( 1+m
E/ (uT +671 $ itm
if [z|e| we done
if 200l Canolgourly 201)
In conobsiin given courel the veglon of convergoure must énclude the by popeeahe
(1)
(2)
-1 1- i
6
rc
-- --
if the bander set to be snally then the obove vegian convegouge (3) is guarantedd
D NETWORK ARCHITECTURES AND HYPERPARAMETERS
D. BAR TASK
Our architecture was fully connected bipartite attractor net (FBAN) with one visible layer and two hidden layers having 48 and 24 channels. We trained using CAE+ with the transient TD(1) procedure, defining network stability as the condition in which all changes in unit activation on successive iterations are less than 0.01 for given input, tanh activation functions, batches of 20 examples (the complete data set), with masks randomly generated on each epoch subject to the constraint that only one completion is consistent with the evidence. Weights between layers 1 and + and the biases in layer are initialized from a mean-zero Gaussian with standard deviation 0.1 n2 + 2 1 + 1) 1 where n, is the number of units in layer I. Optimization is via stochastic gradient descent with an initial learning rate of 0.01, dropped to .001; the gradients given layer of weights are L renormalized be 1.0 for batch of examples, which refer SGD--12.
14

---

Under review as a conference paper at ICLR 2020
D.2 MNIST
Our architecture was a fully connected bipartite attractor net (FBAN) with one visible layer to one hidden layer with 200 units second hidden layer with 50. We trained using Car+ with the transient TD(1) procedure, defining network stability as the condition in which all changes in unit activation on successive iterations are less than 0.01 for given input, tanh activation functions. batches of 250 examples. Masks are generated randomly for each example on each epoch. The masks were produced by generating Perlin noise, frequency 7, thresholded such that one third of the pixels were obscured. Weights between layers I and 1 + 1 and the biases in layer are initialized from mean-zero Gaussian with standard deviation 0.1(3 n4 + 2 + 1)) where ny is the number of units in layer I. Optimization is via stochastic gradient descent with learning rate 0.01; the gradients in given layer of weights are a renormalized to be 1.0 for a batch of examples, which refer to as SGI)-Linf. Target activations scaled to lie in [-0.999, 10.999,0.999),
(a)
(b) R
Iterations
Figure 8: (a) Example of recurrent net clean-up dynamics from Liao et al. (2018). Left column is noisy input, right column is cleaned representation. (b) Example of associative memory model of Wu et al. (2))8a). Column is target, column 2 is input, and remaining columns are retrieval iterations. (c) Some weights between visible and first hidden layer in FBAN trained on MNIST with labels.
D.3 OMNIGLOT
The network architecture consists of four layers: one visible layer and three hidden layers. The visible layer dimensions match the input image dimensions: (28, 28, 1). The channel dimensions of the three hidden layers increase by 128, 256, and 512, respectively. We used filter sizes of 3 x 3 between all layers. Beyond the first hidden layer, we introduce half-padded convolution going from layer to layer 1 + 1, and a half-padded convolution followed by a 2 nearest- neighhor interpolation going from layer I + 1 to layer Consequently, the spatial dimensions of the hidden states, from lowest to highest, are (28,28), (14,14) and (7,7). A trainable bias applied per-channel to each layer. All biases are initialized to O, whereas kernel weights are Gaussian initialized with standard deviation of 0.01. The CBAN used tanh activation functions and Lsg with TD(1) transient training, as described in the main text.
2 average pooling operation followed by
We trained our model on 15,424 images from the Omniglot dataset (test set: 3856). The images are noised by caline-generation of squares that mask 20-40% of the white pixels in the image. We optimized our mean-squared error objective using Adam. The learning rate is initially set to 0.0005 and then decreased manually by a factor of 10 every 20 epochs after training epoch 100. For each batch, the network runs until the state stabilizes, where the condition for stabilization is specified as the maximum absolute difference of the full network states between stabilization steps i and i+1 being less than 0.01. The maximum number of stabilization steps was set to 100; the average stabilization iteration per batch over the course of training was 50 stabilization steps.
Masks were formed by selecting patches of diameter 3-6 uniformly, in random, possibly overlapping locations, stopping when at least 25% of the white pixels have been masked.
Our DC-VAE is based on code from http::/gghbbb..oom//kaaa//oooood cnn.ipynb and followed the method of Im et al. (2017).
D.4 CIFAR-10
The network architecture consists of four layers: one visible layer and three hidden layers. The visible layer dimensions match the input image dimensions: (32, 32, 3). The channel dimensions of the three hidden layers increase by 40, 120, and 440, respectively. We used filter sizes of 3 x between all layers. Beyond the first hidden layer, introduce 2 2 average pooling operation followed by
15

---

Under review as a conference paper at ICLR 2020
half-padded convolution going from layer to layer 1 + 1, and half- convolution followed by a 2 x 2 nearest- neiihtt interpolation going from layer 1 + 1 to layer 1. Consequently, the spatial dimensions of the hidden states, from lowest to highest, are (32,32), (16.16) and (8,8). A trainable bias is applied per-channel to each layer. All biases are initialized to 0, whereas kernel weights are Gaussian initialized with standard deviation of 0.0001. The CBAN used tanh activation functions and LSE with TD(1) transient training, as described in the main text.
We trained our model 50,000 images from the CIFARIO dataset (test set 10,000). The images are noised by online-generation of Perlin noise that masks 40% of the image. We optimized our mean- qquanes error objective using Adam. The learning rate is initially set to 0.0005 and then decreased manually by a factor of 10 every 20 epochs beyond training epoch 150. For each batch, the network runs until the state stabilizes, where the condition for stabilization is specified as the maximum absolute difference of the full network states between stabilization steps t and + 1 being less than 0.01. The maximum number of stabilization steps was set to 100; the average stabilization iteration per batch over the course of training was 50 stabilization steps.
D.5 SUPER-RESOLUTION
The network architecture consists of four layers: one visible layer and three hidden layers. The visible layer spatial dimensions match the input patch dimensions, but consists of 6 channels: (40, 40, 6). The low-resolution evidence patch clamped to the bottom channels of the visible state; the top channels of the visible state serve as the unclamped output against which the high-resolution target patch is compared and loss is computed as a mean-squared error. The channel dimensions of the three hidden layers are 300, 300, and 300. We used filter sizes of 5 x 5 between all layers. All convolutions are half-padded and no average pooling operations are introduced in the SR network scheme. Consequently, the spatial dimensions of the hidden states remain constant and match the input patches of (40, 40). A trainable bias is applied per-channel to each layer. All biases are initialized O, whereas kernel weights are Gaussian initialized with a standard deviation of 0.001.
We trained our model on 91 images from the T91 dataset (Yang et al., 2010) scaled at x2. We optimized our mean-squared error objective using Adam. The learning rate is initially set to 0.00005 and then decreased by a factor of 10 every 10 epochs. The stability conditions described in the CBAN models for CIFAR- 10 and Omniglot are repeated for the SR task, except the stability threshold was set to 0.1 half way through training. We evaluated on four test datasets at x scaling: Ser5 (Bevilacqua al., 2012), Set14 (Zeyde al., 2010), DSBI00 (Martin et al., 2001), and 1rhan100 (Huang et al., 2015).
E ADDITIONAL EXPERIMENTS
E CONVERGENCE THRESHOLD
The convergence threshold operates within the forward iterations of network training and determines the point at which the network's state has satisfactorily converged. Smaller thresholds will result in longer training time, whereas large thresholds will yield faster training times. However, the higher thresholds may negatively impact network performance. We examined the effects of convergence thresholds on network performance by training two identical CBAN models on the Omniglot dataset; & D.3 for network architecture and training details. The only difference between the models is the convergence threshold: one model uses convergence threshold of 0.01, whereas the other uses 1.0.
Our experimental results are shown in Table 2. We find that the higher convergence threshold provides both significant improvement training efficiency, with an increased performance on PSNR, SSIM, and LI reconstruction error metrics. The average number of forward iterations drops from 45 when changing from convergence threshold of 0.01 to 1.0.
E.2 LAYER UPDATING SCHEMES
Salakhutdinov and Hinton (2009) apply odd-even rules for layer updating in their work on Restricted Boltzmann Machines (RBMs). We empirically evaluate the odd-even update rule and compare against simple up-down update rule. The up-down update rule sequentially updates the layers,
16

---

Under review as a conference paper at ICLR 2020
Convergence Threshold
PSNR
SSIM
L1
Number Forward Iterations Training (Mean)
Number Forward Itcrations Training (Median)
0.01
24.1136
0.9275
11.7243
45.00
44.63
1.0
24.4494
0.9440
10.3813
3.01
3.00
Table 2: Quantitative comparison of variation over convergence thresholds. Higher values for PSNR and SSIM are better, whereas lower values for LI are better. Bold font indicates superior performance.
starting from the bottom and moving to the top, and then back down. We explored the two alternative update rules by training two CBAN models on the Omniglot dataset; one with the up-down update rule and the other with the odd-even update rule. See $ D.3 for detailed network architecture and training details. Although the odd-even rule is suggested to be a more coooogialll-plausiii our results show the up-down rule achieving superior performance, measured by PSNR, SSIM, and LI reconstruction loss (see Table 3). In addition, qualitative comparison on image completion demonstrates that the odd-even rule is less efficient when compared to the up-down rule, and more frequently fails to fully complete the task (see Figure 9).
mechanism,
Update Rule
PSNR
SSIM
L1
Odd-Even
21.0335
0.9075
14.3044
Up-Down
22.3537
0.9259
12.0323
Table 3: Quantitative comparison of odd-even versus up-down update rules. Higher values for PSNR and SSIM are better, whereas lower values for LI are better. Bold font indicates superior performance.
AR
target
2
E
53
L
odd-even completion
L
453
L
up-down completion
.. 2
4
3
evidence
Figure 9: Update method comparisons on Omniglot. The rows each array show the target image, the odd-even update method completion (reconstruction), the up-down update method completion (reconstruction) and the evidence provided the CBAN with missing pixels depicted red.
17

---

Under review as a conference paper at ICLR 2020
F
ADDITIONAL RESULTS
1
2
2
3
3
4
4
5
5
6
6
7
7
8
8
9
7
O
1
2.
3
3
4
4
5
5
6
6
7
7
8
8
9
7
2
7
5
5
8
7
4
O
I
1
2
7
3
4
4
S
5
6
6
7
7
2
8
9
9
O
(
2
2
3
4
4
5
5
6
6
7
7
8
8
8
9
2
3
3
4
4
5
5
6
6
7
7
8
8
9
1
7
7
22
0
O
O
1
1
3
2
3
3
4
4
5
5
6
6
7
7
8
2
7
9
O
2
2
3
3
4
4
5
5
6
6
7
7
8
8
9 7
9 9
t
2
2
3
>
4
4
5
5
6
6
7
7
7
8
3
3
73
25 4
0
7
RG
02
25
9
/
4
2
3
3
3
4
4
5
5
6
6
7
7
2
8
4
9
Figure 10: Additional examples of noise completion from supervised MNIST
18

---

Under review as a conference paper at ICLR 2020
target
completion
evidence
dream
target
31
P
3
1
21
37
completion
L
0J
3
3
74
3
10
5
evidence
P
B
24
3)
dream
target
completion
evidence
dream
F
V
2
7
7
3
m
b
8
target
S
1
M
2
7
7
3
n
6
5
completion
evidence
7
7
3
T
J
7
7
3
n
b
5
dream
Figure 11: Additional examples of noise completion from color CIFAR-10 images and the letter-like
Omniglot symbols. The rows of each array show the target image, the completion (reconstruction) produced by CBAN, the evidence provided to CBAN with missing pixels depicted in red, and the CBAN "dream" state encoded in the latent representation.
19


Reviews:
[{"review_id": "rJxJ23IAtS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper studies and evaluates variations on attractor networks (which have recurrence and converge to a fixed point) trained end-to-end by SGD on image completion or super-resolution tasks. Several loss functions are discussed and compared.\n\nThe only quantitative results against published state-of-the-art results (on  the super-resolution task) show that the proposed approach wins on some measures and tasks and loses on others (in fact it wins 25% of the time, within a pool of 4 methods...). While the authors claim superior performance against other recent attractor networks, no quantitative comparison was found in the paper. There were quantitative comparisons against the \"convolutional denoising VAE\" but that model was not previously proposed by other authors to be competitive nor was it defined properly (and it is not clear from its name and the description whether this is really a VAE,  since VAEs are generative models and are not trained to denoise). Overall, I am concerned that the experimental  part of  this paper does not warrant the conclusions of superiority. That being said, I like the approach and I believe that these issues can be fixed but I'd like to see those revisions before accepting the paper.\n\nThe experiments on toy problems like the Bar Task are not necessary. Better use the place for real comparisons against published state-of-the-art benchmark baselines. Otherwise (like with your own implementation of the CD-VAE) it is plausible that the comparisons will be biased (one usually works harder on our own method than on improving someone else's architecture for our task). \n\nSimilarly, the experiments on MNIST are not very informative and you might want to push them to appendices. Also, a more interesting comparison would have been with the convolutional version, where convnets do a LOT better than 1.5% error.\n\nThe comparison against Liao et al is not fair because they don't use convolutions, which we know can make a huge difference.\n\nIn addition to the experimental results  issues, there are several places in the paper which are unclear or possibly wrong. See minor points below.\n\nOne theoretical issue which bothers me is with the \"loss\" L_{\\Delta E}. There is no guarantee that it is lower-bounded, i.e. it can diverge which would not be appropriate for sure.\n\nThe training procedure which ends up being used is TD(1) but that procedure (in the context of the setup of the paper) has not been explained at all. It is really necessary to clarify that.\n\nIt should be clarified that the parameters are tuned by gradient descent on the loss (using backprop through time), or not?\n\nMinor points\n\nThe last paragraph of section 2 seems strange to me. Many ANs are also EBMs so the critique of EBMs (as  if this did not apply to ANs) probably needs to be removed or reconsidered. Also the last sentence of the paragraph is clearly wrong,  since EBMs can easily handle inputs (by opposition to the outputs which are to be predicted or constructed), e.g. see conditional Boltzmann machines or the ANs in the style of Equilibrium Propagation - Scellier et al 2017 (which by the way should probably be mentioned in the review, along with follow-up work).\n\nThe 'bipartite' structure is not clear. What are the two \"parts\" when you have more than 2 layers? Clarify.\nNote that the proposed sweep is inefficient compared to the one proposed for Deep Boltzmann Machines (Salakhutdinov et al 2009) which alternates between odd and even layers. The latter is also more biologically plausible than a feedforward-feedback sweep (consider that the differences in update times of different layers are very different depending on their depth, with your proposed scheme).\n\nDefine CBAN.\n\nPlease clarify how many iterations are needed for convergence in the various experiments. My own experience with ANs suggests that convergence can be quite slow, which is problematic both in terms of computational cost (compared to standard feedforward methods with reasonable depth) and in terms of memory. Hence the sentence in the conclusion stating that the computational cost of CBANs is no greater than DNNs is not clearly true until you demonstrate those aspects quantitatively.\n\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HyeggHcsFB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper argues for the use of attractive networks (AN) for the tasks that involve learning from noisy data. Attractor networks are recurrent in nature and use energy minimization dynamics. As motivation, the authors point to studies that give evidence for the usefulness of recurrence for visual tasks. The experiments presented show that the proposed model produces better quality images than a VAE based baseline.\n\nThe main contribution of the paper appears to the scaling of the AN based architectures to large-capacity models. The authors describe several components that are aimed at improving the gradient flow information and training stability. Firstly, a procedure to constraint the convolutional filters as required for ANs is described which allows CNNs based architectures to be used. An energy-based loss function is also described.\n\nTo tackle the vanishing/exploding gradients during training the authors propose to use leaky sigmoid activation. This is applied at the pre and post-convergence stage. How critical is the leaky sigmoid? Ablation studies for the activation choice and the loss choice need to be analyzed in more detail. \n\nThe authors should make clear the novel contributions, it can be a bit hard to extract this from the descriptions of these components. The point that the proposed model achieves state of the art results for associative memory models needs to elaborate in the text. CD-VAE seems like a weak baseline for the denoising task presented. \n\nIt will be useful to get more insights into the learning dynamics of CBAN and the ablated versions apart from the denoising performance. \n\nMinor:\n\nCBAN is not defined in the text. \n\nIt might be easier to follow the main body by providing all the details of a couple of tasks/datasets in the main body and moving other tasks such as super-resolution to the appendix.\n\nUpdate after rebuttal: I agree with the other reviewers that the paper is not ready for publication. My score would be between 4 and 5 after the rebuttal. \n\nThe various techniques described to make the architecture work is useful.  But the experimental validation is still an issue as pointed by Reviewer 1 and 2. I do not quite agree with the authors that CD-VAE is a strong baseline.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJxQVHhGiB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an attractor network (AN) approach for pattern interpretation and completion. The authors propose a convolutional bipartite architecture consisting of visible (input and output) and hidden layers with weight constraints and squared and energy-based losses. To prevent vanishing/exploding gradients, temporal-difference method and leaky sigmoid activation function are exploited. Training is done by stochastic gradient descent. In experimental validation, the proposed model is able to reconstruct missing pixels in the images for bar task and supervised MNIST. And, in OMNIGLOT and CIFAR-10 experiments, the proposed approach outperforms its variants, and in super-resolution results, it outperforms the baselines. \n\nThe approach looks interesting but the motivation of using AN might not be well justified in the current presentation. Interpretation was emphasized in the paper, but it was not clear to me what induce interpretability in the model and how to interpret experimental results. Also, even though the proposed approach shows promising results in experiments, there were no baseline or only its variants as the baseline, except super-resolution task. \n\n\nDetailed comments:\nDo you have any explanation why \\lambda =1 in TD(\\lambda) works best among others?\nIt was not clear to me how the proposed approach uses recurrent networks. \nIn supervised MNIST experiments, it was claimed that it achieved the state-of-the-art results. But figure 8 might not be enough to justify the claim and could you provide more evidence? \n\n\nI am not quite familiar with this area and my understanding might be limited. But, to the best of my judgement, this paper has an interesting idea but is not yet ready to be published in ICLR. \n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to reintroduce bipartite attractor networks and update them using ideas from modern deep net architectures. 

After some discussions, all three reviewers felt that the paper did not meet the ICLR bar, in part because of an insufficiency of quantitative results, and in part because the extension was considered pretty straightforward and the results unsurprising, and hence it did not meet the novelty bar. I therefore recommend rejection. 
Decision: Reject