Meta-Learning Deep Energy-Based Memory Models
Authors: Sergey Bartunov
Jack Rae
Simon Osindero
Timothy Lillicrap
Venue:  (2020)
Keywords: associative memory
energy-based memory
meta-learning
compressive memory

Abstract: We study the problem of learning an associative memory model -- a system which is able to retrieve a remembered pattern based on its distorted or incomplete version.
Attractor networks provide a sound model of associative memory: patterns are stored as attractors of the network dynamics and associative retrieval is performed by running the dynamics starting from a query pattern until it converges to an attractor. 
In such models the dynamics are often implemented as an optimization procedure that minimizes an energy function, such as in the classical Hopfield network. 
In general it is difficult to derive a writing rule for a given dynamics and energy that is both compressive and fast.
Thus, most research in energy-based memory has been limited either to tractable energy models not expressive enough to handle complex high-dimensional objects such as natural images, or to models that do not offer fast writing.
We present a novel meta-learning approach to energy-based memory models (EBMM) that allows one to use an arbitrary neural architecture as an energy model and quickly store patterns in its weights. 
We demonstrate experimentally that our EBMM approach can build compressed memories for synthetic and natural data, and is capable of associative retrieval that outperforms existing memory systems in terms of the reconstruction error and compression rate.

Published conference paper at ICLR 2020
DEEP ENERGY-BASED MEMORY MODELS
Sergey Bartunov
Jack W Rae DeepMind
Simon Osindero
DeepMind
DeepMind London, United Kingdom os osindero@google com
London, United Kingdom bartunov@google... com
London, United Kingdom jwr swraeggoogle. com
Timothy P Lillicrap
DeepMind
London, United Kingdom countzero@poogle.com
ABSTRACT
We study the problem of learning associative memory model system which is able to retrieve remembered pattern based on its distorted or incomplete version. Attractor networks provide sound model of associative memory: patterns are stored attractors of the network dynamics and associative retrieval is performed by running the dynamics starting from query pattern until converges to an attractor. In such models the dynamics are often implemented as an optimization procedure that minimizes network. In general it is difficult to derive a writing rule for given dynamics and energy that is both compressive and fast. Thus, most research in energy- based memory has been limited either tractable energy models not expressive enough to handle complex high-dimensional objects such as natural images, or to models that do not offer fast writing. We present a novel meta-lcarning approach energy- based memory models (EBMM) that allows to neural architecture as an energy model and quickly store patterns in weights We demonstrate experimentally that our EBMM approach can build compressed memories for synthetic and natural data, and is capable of associative retrieval that outperforms existing memory systems in terms of the reconstruction error and compression rate.
energy function, such as in the classical Hopfield
arbitrary
INTRODUCTION
Associative memory has long been of interest to neuroscience and machine learning communities (Willshaw al.] 1969; Hopfield 1982 Kanerval 1988). This interest has generated many proposals for associative memory models, both biological and synthetic. These models address the problem of storing a set of patterns in such a way that a stored pattern can be retrieved based on a partially known distorted version. This kind of retrieval from memory is known as auto-association.
Due the generality of associative retrieval, successful implementations of associative memory models have the potential to impact many applications. Attractor networks provide one well- grounded foundation for associative memory models (Amit & Amit, 1992). Patterns are stored in such way that they become attractors of the update dynamics defined by the network. Then, if query pattern that preserves sufficient information for association lies the basin of attraction for the original stored pattern, a trajectory initialized by the query will converge to the stored pattern.
A variety of implementations of the general attractor principle have been proposed. The classical Hopfield network (Hopfield) 1982), for example, defines simple quadratic energy function whose parameters serve as memory. The update dynamics in Hopfield networks iteratively minimize the energy by changing elements of the pattern until it converges to a minimum, typically corresponding one of the stored patterns. The goal of the writing process is find parameter values such that

---

Published conference paper at ICLR 2020
Ehergy E
Alx;
Energy E
E(x; 0)
Quary
2end 03
x
irputs
Input
Figure schematic illustration EBMM. The energy function modelled by a neural network. The writing rule is then implemented as weight update, producing parameters θ from the initializa- tion 0, such that the stored patterns X1, X2, X3 become local minima of the energy (see Section 3 Local minima are attractors for gradient descent which implements associative retrieval starting from a query &, in this case distorted version of X3.
the stored patterns become attractors for the optimization process and such that, ideally, spurious attractors are created.
Many different learning rules have been proposed for Hopfield energy models, and the simplicity of the model affords compelling closed-form analysis (Storkey & Valabregue| 1999). At the same time, Hopfield memory models have fundamental limitations: (I) is not possible add capacity for more stored patterns by increasing the number of parameters since the number of parameters in a Hopfield network is quadratic in the dimensionality of the patterns. (2) The model lacks means of modelling the higher-order dependencies that exist in real-world data.
domains such as natural images, the potentially large dimensionality of an input makes both ineffective and often unnecessary to model global dependencies among raw input measurements. In fact, many auto-correlations that exist in weal-world perceptual data can be efficiently compressed without significant sacrifice fidelity using either algorithmic (Wallace) 1992, Candes Taoo 2004) or machine learning tools (Gregor et al. 20166] Tooerici et al. (017)). The success of existing deep learning techniques suggests more efficient recipe for processing high -ineeniinnsi inputs by modelling hierarchy of signals with restricted or local dependencies (LeCun et al.) 1995). In this paper we use a similar idea for building an associative memory: use a deep network's ' weights to store and retrieve data.
Fast writing rules A variety of energy-based memory models have been proposed since the original Hopfield network mitigate its limitations (Hinton al. 2006b Du & Mordatch 2019). Restricted Boltzmann Machines (RBMs) (Hinton) (2012) add capacity to the model by introducing latent variables, and deep variants of RBMs (Hinton et al. 2006b; Salakhutdinov & Larochelle 2010) afford more expressive energy functions. Unfortunately. training Boltzmann machines remains challegging, and while recent probabilistic models such as variational auto-encoders (Kingma & Welling 2013 Rezende et al. 2014) are easier to train, they nevertheless pay the price for expressivity the form of slow writing. While Hopfield networks memorize patterss quickly using a simple Hebbian rule, deep probabilistic models are slow in that they rely on gradient training that requires many updates (typically thousands or more) to settle new inputs into the weights of a network. Hence, writing memories via parametric gradient based optimization is straightforwardly applicable memory problems where fast adaptation a crucial requirement. In contrast, and by explicit design, our proposed method enjoys fast writing, requiring few parameter updates (we employ just 5 steps) write new inputs into the weights of the net once meta- meta-learning is complete. It also enjoys fast reading, requiring few gradient descent steps (again just in our experiments) to retrieve a pattern. Furthermore, our writing rules are also fast in the sense that they use O(N) operations to store N patterns in the memory this scaling is the best one can hope for without additional assumptions.
We propose a novel approach that leverages meta-learning enable fast storage of patterns into the weights of arbitrarily structured neural networks, as well as fast associative retrieval. Our networks output a single scalar value which we treat as an energy function whose parameters implement a distributed storage scheme. We use gradient-based reading dynamics and meta-leam writing rule the form of truncated gradient descent over the parameters defining the energy function. We show that the proposed approach enables compression via efficient utilization of network weights, as well fast-converging attractor dynamics.

---

Published conference paper at ICLR 2020
2 RETRIEVAL ENERGY-BASED MODELS
We focus on attractor networks as a basis for associative memory. Autractor networks define update dynamics for iterative evolution of the imput pattern: x(++1) fx)(1))
For simplicity, we will assume that this process discrete in time and deterministic, however there are examples of both continuous- time (Yoon et al. 2013) and stochastic dynamics (Aarts & Korst 1988). A fixed-point attractor of deterministic dynamies can be defined
point for which it converges, i.c. x f(x). Learning the associative memory in the attractor network is then equivalent to learning the dynamics/ f such that its fixed-poiit attractors are the stored patterns and the corresponding basins of attraction are sufficiently wide for retrieval.
An energy-based attractor network is defined by the energy function E(x) mapping an input object x E X' to a real scalar value. A particular model may then impose additional requirements on the energy function. For example if the model has probabilistic interpretation, the energy function usually negative unnormalized logarithm of the object probability log p(x) -E(x) + const, implying that the energy has to be well-behaved for the normalizing constant to exist. In our case no such constraints are put on the energy.
The attractor dynamics in energy based models is often implemented either by iterative energy optimization [Hopfield] 1982) or sampling (Aarts Korst 1988). In the optimization case considered further in the paper, attractors are conveniently defined as local minimizers of the energy function.
While particular energy function may suggest a number of different optimization schemes for local minimum of an arbitrary function is NP-hard. Thus, we consider a class of energy functions that are differentiable on C R", bounded from below and define the update dynamics over k = 1...., K steps via gradient descent:
retrieval, convergence
read ; 0) x(K),
x (A+1)) x1))
V Ex
x X
(I)
With appropriately set step sizes (-()) 1= this procedure asymptotically converges to local minimum of energy E(x) [Nsterov] 2013). Since asymptotic convergence may not enough for truncate the optimization procedure (T) at K steps and treat x(K) result of the retrieval. While vanilla gradient descent (I) is sufficient to implement retrieval, in our experiments we employ a number of extensions, such as the use of Nesterov momentum and projected gradients, which are thoroughly described in Appendix B
practical applications,
Relying on the generic optimization procedure allows us translate the problem designing update dynamics with desirable properties to constructing an appropriate energy function, which in general is equally difficult. In the next section we discuss how to tackle this difficulty.
3 META-LEARNING GRADIENT-BASED WRITING RULES
As discussed in previous sections, our ambition is to able to use any scalar-output neural network as an energy function for associate retrieval. We assume parametric model Exx; 0) differentiable in both and 0 and bounded from below a function of x. These mild assumptions that are often met in the existing neural architectures with an appropriate choice activation functions, e.g. tanh.
The writing rule then compresses input patterns X {x1. X2......... X. into parameters @ such that each of the stored patterns becomes a local minimum of Exx; 0) or. equivalently, creates basin of attraction for gradient descent the pattern space.
This property can be practically quantified by the reconstruction error, e.g. mean squared error, between the stored pattern x and the pattern read 0) retrieved from a distorted version of x:
N Σ Ex [[x; read (1;
C(X, 0) =
(2)
Here we assume a known, potentially stochastic distortion model p(x[x) ()) such as randomly crasing
certain number of dimensions, or salt-and- -adt-and-pepper pepper noise. While one can consider loss (2) as function of network parameters 0 and call minimization of this loss with conventional optimization method

---

Published a conference paper at ICLR 2020
Query
Iteration
Iteration 2 Iteration 3 Iteration
Iteration 5 Original
N
N
N
N
N
2
W09
20I1
2001
2011
2071
1
1
#
3
Query
Iteration
Iteration
Iteration
Iteration 4
Iteration
Original
Figure 2: Visualization of gradient descent iterations during retrieval of Omniglot characters (largest model). Four random images shown from the batch of 64.
a writing rule - it will require many optimization steps obtain satisfactory solution and thus does not fall into our definition of fast writing rules (Santoro al. 2016).
Hence, we explore a different approach to designing a fast writing rule inspired by recently proposed gradient- based meta-learning techniques (Finn et al. (2017) which we call meta-learning energy- based memory models (EBMM). Namely we perform many write and read optimization procedures with a small number of iterations for several sets of write and read observations, and backpropagate into the initial parameters of 0 to learn good starting location for fast optimization. As usual, we assume that have access the underlying data distribution pa( X) over data batches of interest X from which we can sample sufficiently many training datasets, even if the actual batch our memory model will be used to store (at test time) is not available at the training time (Santoro et al. 2016).
The straightforward application of gradient-based meta-leaming to the loss because we generally cannot evaluate or differentiate through the expectation over stochasticity the distortion model in a way that is reliable enough for adaptation, because as the dimensionality of the pattern space grows the number of possible (and representative) distortions grows exponentially.
problematic,
Instead, we define a different writing loss W. minimizing which serves proxy for ensuring that input patterns are local minima for the energy E(x; 0), but does not require costly retrieval exponential number of distorted queries.
W(x, 0) = E(x; 0) + aaloaV/V. E(;; 0) + 3110 011
(3)
As see, the writing loss 43 consists of three terms. The first term is simply the energy value which we would like to small for stored patterns relative to non-stored patterns. The condition for x to be local minimum of (x: 0) is two-fold: first, the gradient at x is zero, which is captured by the second term of the writing loss, and, second, the hessian is positive-definite The latter condition is difficult to express in a form that admits efficient optimization and we found that meta-lcarning using just first two terms in the writing loss is sufficient. Finally, the third term limits deviation from initial prior parameters 0 which we found helpful from optimization perspective (see Appendix D for more details).
We use truncated gradient descent on the writing loss 3 to implement the writing rule:
N
w = (T) g(++1)
g(t) r7
VoW(x, 100), g
(4)
To ensure that gradient updates 4 are useful for minimization of the reconstruction error 2 we train the combination of retrieval and writing rules end-to-end, meta-learning initial parameters 0,

---

Published conference paper at ICLR 2020
30
Hamming error
20
Memory network
MANN
25
Square error
15
EB MM FC
conv
20
10
2K
4K
6K
8K
10K
12K
14K
16K
10K
12K
14K
16K
Memory size
Memory size
(a) Omniglot.
(b) CIFAR.
Figure 3: Distortion (reconstruction error) vs rate (memory size) analysis on batches of 64 images.
learning rate schedules r -= random sets of patterns from the batch distribution Pa(XX):
JK-i {n)(") ((>(0) and mneaa-paramteer == (a, () to perform well
minimize Ex-p=(X) [c(X, write(XX)) for f,
(5)
our experiments, p(X) is simply distribution over batches of certain size N sampled uniformly from the training (or testing during evaluation) set. Parameters = write(XX) are produced by storing X in the memory using the writing procedure + Once stored, distorted versions of X can be retrieved and can evaluate and optimize the reconstruction error
Crucially, the proposed EBMM implements both read 0) and write (X)) operations via truncated gradient descent which can be itself differentiated through in order to set up a tractable meta-learning problem. While truncated gradient descent is guaranteed
converge. reading and writing rules are trained jointly to minimize the reconstruction error (2) and thus ensure that they converge sufficiently fast. This property turns this potential drawback of the method to its advantage over provably convergent, but slow models. It also relaxes the necessity of stored patterns create too well-behave basins attraction because if, example, stored pattern creates nuisance attractor in the dangerous proximity of the main one, the gradient descent ① might successfully pass it with appropriately learned step sizes y.
EXPERIMENTS
In this section we experimentally evaluate EBMM on number real-world image datasets. The performance of FBMM is compared to a set of relevant baselines: Long-Short Term Mem- ory (LSTM) (Hochreiter & Schmidhuber] (197), the classical Hopfield network (Hopfield) 1982).
Memory-Augmentt Neural Networks (MANN) (Santoro et al. (016)) (which are a variant of the Differentiable Neural Computer (Graves et al. 2016)), Memory Networks (Weston et al.) 2014). Differentiable Plasticity model of Miconi al. (2018) generalization of the Fast-weights RNN (Ba et al. 2016)) and Dynamic Kanerva Machine (Wu et al. 2018). Some of these baselines failed to learn all for real-world images. In the Appendix A.2 we provide additional experiments with random binary strings with larger of representative models.
The experimental procedure is the following: we write a fixed-sized batch of images into memory model, then corrupt a random block of the written image to form query and let the model retrieve the originally stored image. By varying the memory size and repeating this procedure, distortion/rate analysis, iee we measure how well memory model can retrieve remembered pattern for given memory size. Meta-learning have been performed on the canonical train splits of each dataset and testing the test splits. Batches were sampled uniformly. see Appendix A.7 for the performance study on correlated batches.
perform
We define memory size as a number float 32 numbers used to represent a modifiable part the model. In the case of EBMM it is the subset of all network weights that are modified by the gradient descent (4), for other models is size of the state, e.g. the number of slots the slot size for a Memory Network. To ensure fair comparison, all models use the same encoder (and decoder, when applicable) networks, which architectures are described in Appendix C In all experiments EBMM used K read iterations and T' 5 write iterations.

---

Published conference paper at ICLR 2020
Query
Iteration
Ineration
Itaratioo 3 Iteiration
Itaration
Original
MarNett
-1 10
180
Ehargy
-2
-200
20
Query
Iteration
Iteratio 2
Iteration 3
Iteration
Iteraion
Original
Figure Visualization gradient descent iterations during retrieval of CIFAR images. The
column contains reconstructions from Memory networks (both models use 10k memory).
4.1 OMNIGLOT CHARACTERS
We begin with experiments the Omniglot dataset (Lake al. 2015) which is now standard evaluation of fast adaptation models. For simplicity of comparison with other models, downscaled the images to 32 x 32 size and binarized them using a 0.5 threshold. We use Hamming distance as the evaluation metric. For training and evaluation we apply 16 x 16 randomly positioned binary distortions (see Figure for example).
We explored two versions of EBMM for this experiment that use parts of fully- (FC. see Appendix C.2) and convolutional (conv, Appendix C.3) layers in a 3-block ResNet (He et al. 2016) writable memory.
Figure conttaiiss the distortion-rate analysis of different models which in this case the Hamming distance as a function memory size. We can see that there are two modes in the model behaviour. For small memory sizes, learning lossless storage becomes a hard problem and all models have be
find efficient compression strategy, where most of the difference between models observed. However, after a certain critical memory size it becomes possible to rely just on the autoencoding which in the case of relatively simple dataset such Omniglot can be efficiently tackled by the ResNet architecture any compression mechanisms beyond using distributed representations can retrieve original images almost perfectly. In this experiment MANN has been able to learn the most efficient compression strategy, but could not make of larger memory. EBMM performed well both in the high and low compression regimes with convolutional memory being more efficient over the fully-connected memory. Further, in CIFAR and ImageNet experiments we only use the convolutional version of EBMM.
using. Hence, even Memory Networks that do not employ
We visualize the process of associative retrieval in Figure 2 The model successfully detected distorted parts of images and clearly managed to retrieve the original pixel intensities. We also show energy levels of the distorted query image, the recalled images through 5 read iterations, and the original image. In most cases found the energy the retrieved images matching energy of the originals, however, an error would occur when they sometimes do not match (see the green example).
REAL IMAGES FROM CIFAR-10
We conducted similiar study on the CIFAR dataset. Here we used the same network architecture as in the Omniglot experiment. The only difference in the experimental setup is that we used squared evaluation metric since the data is continuous RGB images.

---

Published conference paper at ICLR 2020
Query
Original
EBMM
MemNet
DNC
NANN
Square error
Nemory networt
70
Energy memory (coes)
60
50
40
30
4K
7K
9K
12K
14K
16K
18K
Memory size
(a) Distortion-rate analysis on ImageNet.
(b) Retrieval of 64x64 ImageNet images (all models have RJ 18K memory).
Figure 5: ImageNet results.
Figure 3b contains the corresponding distortion-rate analysis. EBMM clearly dominates in the comparison. One important reason for that is the ability of the model to detect the distorted part of the image so can avoid paying the reconstruction loss for the rest of the image. Moreover, unlike Omniglot where images can be almost perfectly reconstructed by an autoencoder with a large enough code, CIFAR images have much more variety and larger channel depth. This makes efficient joint storage of a batch as important as an ability to provide good decoding of the stored original.
Gradient descent iterations shown in Figure demonstrate the successful application to natural images. Due to the higher complexity of the dataset, the reconstructions are imperfect, however the original patterns are clearly recognizable. Interestingly, the learned optimization schedule starts with one big gradient step providing a coarse guess that then gradually refined.
model
4.3 IMAGENET 64x64
We further investigate the ability of EBMM handle complex visual datasets by applying the model to 64 x 64 ImageNet. Similarly to the CIFAR experiment, we construct queries by corrupting a quarter of the image with 32 x 32 random masks. The model is based on a 4-block version of the CIFAR network. While the network itself is rather modest compared to existing ImageNet classifiers, the sequential training regime resembling large-state recurrent networks prevents us from using anything significantly bigger than a CIFAR model. Due to prohibitively expensive computations required by experimenting this scale, also had to decrease the batch size 32.
The distortion- rate analysis (Figure5a) 5a) shows the behaviour similar the CIFAR experiment. EBMM pays less reconstruction error than other models and MANN demonstrates better performance than Memory Networks for smaller memory sizes; however, the asymptotic behaviour of these two models will likely match.
qualitative results are shown Figure 5b] Despite the arguably more difficult images, EBMM able to capture shape and color information, although not in high detail. We believe this could likely be mitigated by using larger models. Additionally, using techniques such perceptual losses (Johnson) et al. 2016) instead of naive pixel-wise reconstruction errors can improve visual quality with the existing architectures, but we leave these ideas for future work.
ANALYSIS OF ENERGY LEVELS
We were also interested in whether energy values provided by EBMM are interpretable and whether they can be used for associative retrieval. We took an Omniglot model and inspected energy levels of different types of patterns. It appears that, despite cases could discriminate between in in-memory and oft-or-memory patterns, see Figure 6 Morcover, distorted patterns had even higher energy than simply unknown patterns. -iit----iittibttion patterns, here modelled binarized CIFAR images, be as clear outliers.
being explicitly trained to, EBMM in many

---

Published conference paper at ICLR 2020
250
Memories
200
Non-memaries
Distorted memories
150
FAR Images
100
50
-15
-100 -50
50
Figure 6: Energyy distributions of different classes of patterns under Omniglot model. Memories are the patterns written into memory, non-memories are other randomly sampled images and distorted memories are the written patterns distorted as during the retrieval. CIFAR images were produced by binarizing the original RGB images and serve as out-of-distribution samples.
5 RELATED WORK
Deep neural networks capable of both compression (Pakhi et al. 2013 Kraska al., 2018). and memorizing training patterns (Zhang et al. 2016). Taken together. these properties make deep networks an attractive candidate for memory models, with both exact recall and compressive capabilities. However, there exists natural trade-off between the speed of writing and the realizable capacity of model (Ba et al. 2016). Approaches similar to ours their use of gradient descent dynamics, but lacking fast writing, have been proposed by Hinton et al. (2006a) and recently revisited by Xie et al.] (2016); Nijkamp et al. (2019): Du & Mordatch| (2019). Krotov & Hopfield] (2016) also extended the classical Hopfield network more capacity. In general it is difficult to derive a writing rule for a given dynamics equation or an energy model which attempt to address in this work.
larger family of non-quadratic energy functions with
The idea of meta-learning (Thrun & Prau 2012 Hochreiter applications few-shot supervised (Santoro et al.) 2016)] Vinyals et al.] (016) and unsupervised learning (Bartunov & Vetroy (016 Reed et al. (2017). Our model is particularly influenced by works of Andrychoowic al. (2016) and Finn et al. (2017). which experiment with meta-learning efficient optimization schedules and, perhaps, can be seen as an ultimate instance of this principle since we implement both learning and inference procedures
2001) has found many successful
optimization. Perhaps the most prominent existing application of meea-leerning for associative retrieval is found in the Kanerva Machine (Wu) ct al. 2018), which combines a variational auto-encoder with a latent linear model to serve as an addressable memory. The Kanerva machine benefits from high-level representation extracted by the auto-encoder. However, its linear madel only represent convex combinations of memory slots and thus less expressive than distributed storage realizable weights of deep network.
We described literature on associative and mnergy-based memory in Section I but other types of memory should be mentioned in connection with our work. Many recurrent architectures aim maintaining efficient compressive memory (Graves al. 2016 |ac et al.) (0188). Models developed by Ba et al. (2016) and Miconi et al. (2018) enable associative recall by combining standard RNNs with structures similar to Hopfield network. And, recently Munkhdalai et al. (2019) explored the idea of using arbitrary feed-forward networks key-value storage.
Finally, the idea of learning a surrogate model to define gradient field useful for a problem interest has a number of incarnations. Putzky & Welling (207)) jointly Iearn an energy model and an optimizer to perform denoising impainting of images. Marino et al. (2018) use gradient descent on an energy defined by variational lower bound for improving variational approximations. And, [Belanger et al.] (2017) formulate a generic framework for energy-based prediction driven by gradient descent dynamics. A detailed explanation of the learning through optimization with applications in control can found in (Amos 2019).
Modern deep learning made a departure from the earlier works on energy- energy-based based models such as Boltzmann machines and approach image manipulation tasks using techniques such as aforementioned variational autoencoders generative adversarial networks (GANs) (Goodfellow al. 2014). While

---

Published conference paper at ICLR 2020
these models indeed constitute state of the art for learning powerful prior models of data that can perform some kind of associative retrieval, they naturally lack fast memory capabiliitis. In contrast, the approach proposed in this work addresses the problem of jointly learning strong prior model and an efficient memory which can be used combination with these techniques. For example, one can replace the plain reconstruction error with a perceptual loss incurred by a GAN discriminator or use EBMM to store representations extracted by VAE.
While both VAEs and GANs can be also be equipped with memory (as done e.g. by Wu et al. (2018)) energy-based formulation allows us to employ arbitrary neural network parameters as an associative storage and make use of generality of gradient-base
meta-learning As we show in the additional experiments binary strings in Appendix AA.2 EBMM is applicable not only high- ligeninnnniional natural data, but also to uniformly generated binary strings where no prior can useful. At the same time, evaluation of non-memory baselines in Appendix demooossssssssss that the combination of a good prior and memory in EBMM achieves significantly better performance than just prior, even suited with a much larger network.
CONCLUSION AND FUTURE WORK
We introduced novel learning method for deep associative memory systems. Our method benefits from the recent progress in deep learning so that we can use very large class of neural networks both for learning representations and for storing patterns in network weights. At the same time, we are not bound by slow gradient learning thanks to meta-learning of fast writing rules. We showed that method is applicable a variety of domains from non-oompeesiibee (binary strings; see Appendix) to highly compressible (natural images) and that the resulting memory system uses available capacity efficiently. We believe that more elaborate architecture search could lead to stronger results par with state-of the--rtertt art generative models.
The existing limitation of EBMM is the batch writing assumption, which is in principle possible to relax. This would enable embedding of the model in reinforeement learning agents or into other tasks requiring online-updating memory. Employing more significantly more optimization steps does not seem to be necessary at the moment, however, scaling up to larger hatches or sequences of patterns will face the bottleneck of recurrent training. Implicit differentiation techniques (Liao et al. 2018). reversible learning (Maclaurin et al. (2015) or synthetic gradients (Jaderberg et al.) 2017) may be promising directions towards overcoming this limitation. It would be also interesting to explore a stochastic variant of EBMM that could return different associations in the presence of uncertainty caused by compression. Finally, many general principles of learning attractor models with desired properties are yet to discovered and we believe that our results provide a good motivation for this line of research.
REFERENCES
Emile Aarts and Jan Korst. Simulated annealing and boltzmann machines. 1988.
Daniel J Amit and Daniel Amit. Modeling brain function: The world of attractor neural networks. Cambridge university press, 1992.
Brandon Amos. Differentiable optimization based modeling for machine learning. PhD thesis, PhD thesis. Camegie Mellon University. 2019.
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. In Advances in Neural Information Processing Systems, pp. 3981 1-9989, 2016.
Antreas Antoniou, Harrison Edwards, and Amos Storkey. How to train your maml. arXiv preprint a1X0v.1810.09502. 2018
Jimmy Ba. Geoffrey E Hinton, Volodymyr Mnih Joel Z Leibo, and Catalin lonescu. Using fast weights to attend to the recent past. In Advances in Neural Information Processing Systems, pp. 4331-4334 2016.

---

Published conference paper at ICLR 2020
Sergey Bartunov and Dmitry P Vetrov. Fast adaptation in generative models with generative matching networks. arXiv preprint urXiv:1612.0211 2016.
David Belanger, Bishan Yang, and Andrew McCallum. End-to-end learning for structured prediction energy networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 429-439. JMLR. org, 2017.
Emmanuel Candes and Terence Tao. Near optimal signal recovery from random projections: Universal encoding strategies? arXiv preprint math/0415522, 2004.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation gated recurrent neural networks on sequence modeling. arXiv preprint arXiv: 2014.
Yilun Du and Igor Mordatch. Implicit generation and generalization in energy-based models. arXiv preprint arXiv: 1xXXvv1903..08 1903,0868, 2019.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Madel-agnostic meta-learning for fast adaptation of deep networks. arXiv preprint 1 030000 2017.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- tion processing systems, pp. 2672-2680 2014.
Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Danihelka, Agnieszka Grabska- Barwiñssa, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. Hybrid computing using neural network with dynamic external memory. Nature, 538(762):471, 2016.
Karol Gregor, Frederic Besse, Danilo Jimenez Rezende, Ivo Danihelka, and Daan Wierstra. Towards conceptual compression. Advances In Neural Information Processing Systems, pp. 3549-3557, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015. 2015.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Geoffrey Hinton, Simon Osindero, Max Welling, and Yee-Whye Teh. Unsupervised discovery of nonlinear structure using contrastive backpropagation. Cognitive science, 30(4):725 7311 2006a.
Geoffrey E Hinton. A practical guide to training restricted boltzmann machines. In Newral networks: Tricks of the trade, pp. 599-619. Springer, 2012.
Geoffrey E Hinton, Simon Osindero, and Yee Shyy Teh. A fast learning algorithm for deep belief nets. Neural computation, 18(7):1527-155, 2006b.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8); 1780, 1997.
Sepp Hochreiter, Steven Younger, and Peter R Conwell. Learning to learn using gradient descent. In International Conference on Artificial Neural Networks, pp. 87-94. Springer, 2001.
John Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(8):5544-2558 1982.
Max Jaderberg, Wojciech Marian Czamneck, Simon Osindero, Oriol Vinyals, Alex Graves, David Silver, and Koray Kavukcuoglu. Decoupled neural interfaces using synthetic gradients. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1627-1635.
JMLR. org, 2017.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision, pp- 694-711. Springer, 2016.
10

---

Published conference paper at ICLR 2020
Pentti Kanerva. Sparse distributed memory. MIT press, 1988.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114 2013.
arXiv preprint
Tim Kraska, Alex Beutel, Ed H Chi, Jeffrey Dean, and Neoklis Polyzotis. The case for learned index structures. In Proceedings of the 2018 International Conference on Management of Data, pp. 489-504. ACM, 2018.
Dmitry Krotov and John Hopfield. Dense associative memory for pattern recognition. In Advances in Neural Information Processing Systems, pp. 1172-1180, 2016.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenhaum. Human-level concept learning through probabilistic program induction. Science, (5006266)):13322222 332-1338, 2015.
Yann LeCun, Yoshua Bengio, al. Convolutional networks images, speech, and time series. handbook of brain theory and neural networks, 3361((0):1995, 1995.
Renjie Liao, Yuwen Xiong, Ethan Fetaya, Lisa Zhang, KiJung Yoon, Xaq Pitkow, Raquel Urta- sun, and Richard Zemel. Reviving and improving recurrent back-propagattin. arXiv preprint urXiv:18033 06396, 2018.
Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. arXiv preprint 1rXiv:1711.01001, 2017.
Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient baadient-based hyperparameter optimization through reversible learning. In International Conference on Machine Learning, pp. 2113-2122, 2015.
Joseph Marino, Yisong Yue, and Stephan Mandt. Iterative amortized inference. arXiv preprint arXiv:/807.000 09356, 2018.
Thomas Miconi, Jeff Clune, and Kenneth O Stanley, Differentiable plasticity: training plastic neural networks with backpropagation. arXiv preprint
2018.
Tsendsuren Munkhdalai, Alessandro Sordoni, Tong Wang, and Adam Trischler. Metalcarned neural memory. ArXiv, abss/190..000 hass/190700920,0 2019.
Yurii Nesterov. Introductory lectures on convex optimization: A basic course, volume 87. Springer Science & Business Media, 2013.
Yurii E Nesterov. method for solving the convex programming problem with convergence rate o (I/K) 2). In Dokl. akad. nauk Sssr, volume 269, pp. 543-547, 1983.
Erik Nijkamp, Bong-Chun Zhu, and Ying Nian Wu. learning non-convergent short-run meme toward -basey- model. arXiv preprint 194iv:1904.09770, 2019.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, al. Deep face recognition. BMVC, volume I, pp. 6, 2015.
Patrick Putzky and Max Welling. Recurrent inference machines for solving inverse problems, arXiv preprint arXiv: 1ee::70066 7706044088 2017.
Jack W Rae, Sergey Bartunov, and Timothy P Lillicrap. Meta-learning neural bloom filters. 2018.
Scott Reed, Yutian Chen, Thomas Paine, Aäron den Oord, SM Eslami, Danilo Rezende, Oriol Vinyals, and Nando Freitas. Few-shot autoregressive density estimation: Towards learning learn distributions. arXiv preprint arXiv:1710. 10304, 2017.
Danilo Jimenez. Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv: 2014.
Ruslan Salakhutdinov and Hugo Larochelle. Efficient learning of deep boltzmann machines. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 693-700, 2010.

---

Published conference paper at ICLR 2020
Adam Santoro. Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with memory-augmentte neural networks. In International conference on machine learning, pp. 1842-1850, 2016.
Amos J Storkey and Romain Valabregue. The basins of attraction of new hopfield learning rule. Neural Networks, 12(6):89--77, 1999.
Sebastian Thrun and Lorien Pratt. Learning to learn. Springer Science & Business Media, 2012.
George Toderici, Damien Vincent, Nick Johnston, Sung Jin Hwang, David Minnen, Joel Shor, and Michele Covell. Full resolution image compression with recurrent neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5306-5314, 2017.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.
2018.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, al. Matching networks for shot learning. In Advances neural information processing systems, pp. 3630-3638, 2016.
Gregory K Wallace. The jpeg still picture compression standard. IEEE transactions consumer electronics, 38(1 (8(1)xxvii-xxxxv 1992.
Jason Weston, Sumit Chopra, and Antoine Bordes. arXiv:1410.3916. 2014.
Memory networks.
arXiv preprint
David Willshaw, O Peter Buneman, and Hugh Christopher Longuet-Higgins. Non-holographic associative memory. Nature, 222(5197);960, 1969.
Yan Wu, Gregory Wayne, Karol Gregor, and Timothy Lillicrap. Learning attractor dynamics for generative memory. In Advances in Neural Information Processing Systems, pp. 9401-9410, 2018.
Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingnian Wu. theory of generative convnet. International Conference on Machine Learning, pp. 2635 2644. PMLR, 2016.
KiJung Yoon, Michael Buice, Caswell Barry, Robin Hayman, Neil Burgess, and lla R Ficte. Specific evidence of low-dimensional continuous attractor dynamics in grid cells. Nature neuroscience, 16 (8):1077, 2013.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint 1...5300 2016.

---

Published conference paper at ICLR 2020
Table 1: Number of error bits in retrieved binary patterns.
# PATTERNS
16
32
48
64
96
METHOD
HOPFIELD NETWORK, HEBB RULE
0.4
5.0
9.8
13.0
16.5
HOPFIELD NETWORK, STORKEY RULE HOPFIELD NETWORK, PSECDO-INVEEN
0.0
0.9
6.3
11.3
17.1
0.0
0.0
0.3
4.3
22.5
DIVERENTIABLE PLASTICITY (MICONI ET AL.
2018
3.0
13.2
26.3
34.9
MANN (SANTORO ET AL. 2016)
0.1
0.2
1.8
4.25
9.6
LSTM [HCCRRETTRR SCHMIDHUBER 1997
30
58
63
64
64
MEMORY NETWORKS (WESTON ET AL. 2014
0.0
0.0
10.5
EBMM RNN
0.0
0.0
0.1
0.5
4.2
A ADDITIONAL EXPERIMENTAL DETAILS
We train all models using AdamW optimizer (Loshchilov & Hutter 2017) with learning rate x 10-3 and weight decay 10 all other parameters set to Adam detaults. We also apply gradient clipping by global norm at 0.03 All models were allowed to train for x 10 gradient updates or week whichever ended first. All baseline models always made more updates than EBMM.
One instance of each model been trained. Error showed on ligures correspond to 95-percentiles computed on 1000 of random hatches.
In all experiments we used initialization scheme proposed by He et al. 2015
A.1 FAILURE MODES OF BASELINE MODELS
Image retrieval appeared to be difficult for a number of baselines.
LSTM failed to train due to quadratic growth of the hidden-to-hidee weight matrix with increase of the hidden state size. Even moderately large hidden states were prohibitive for training modern GPU.
Differential plasticity additionally struggled to train when using a deep representation instead the raw image data. We hypothesize that it was challenging for the encoder-decoder pair to train simultaneously with the recurrent memory, because in the binary experiment, while not performing the best, the model managed to learn a memorization strategy.
Finally, the Kanerva machine could not handle the relatively strong noise we used in this task. By design, Kanerva machine is agnostic to the noise model and is trained simply maximize the data likelihood, without meta- leaanne a particular do-noising scheme. In the presence of the strong noise it failed to train on sequences longer than 4 images.
A.2 EXPERIMENTS WITH RANDOM BINARY PATTERNS
Besides highly-structures patterns such as Omniglot ImageNet images we also conducted experi- ments on random binary patterns- the classical setting in which associative memory models have been evaluated. While such random patterns internal structure, by this experiment we examine the efficiency a learned coding scheme, i.e. how well can each of the models store binary information in the floating point format.
compressible in expectation due lack of any
We generate random 128-dimensional patterns, each dimension of which takes values of -1 or +1 with equal probability, corrupt half of the bits and this query for associative retrieval. We compare EBMM employing a simple fully recurrent network (an RNN using the same input at each iteration, see Appendix C.I as energy model, against classical Hopfield network (Hopfield. 1982) using different writing rules (Storkey & Valabregue 1999) and recently proposed differential plasticity model (Miconi et al. 20.8). It is worth noting the differentiable plasticity model a generalized variant of Fast Weights (Ba et al. 2016), where the plasticity of each activation is modulated separately. We also consider I.S (Hochreiter & Schmidhuber 1997). Memory

---

Published conference paper at ICLR 2020
network (Weston al. 2014) and iet al. (2166) which is variant of the DNC (Graves et al. 2016).
Neural Network (MANN) used by Saantor
Since the Hopfield network has limited capacity that is strongly tied to input dimensionality and that cannot be increased without adding more inputs, constrain all other baseline models to use the same amount of memory. For this task it equals to 128 * (128- 1)/2 + 128 to parametrize a symmetric matrix and frequency vector. We measure Hamming distance between the original and the retrieved pattern for each system, varying the number of stored patterns. We found it difficult to train the recurrent baselines on this task, so we let all models clamp non non-distorted bits to their true values at retrieval which significantly stabilized training.
its memory size reference and
As we can see from the results shown in Table I1 EBMM learned highly efficient associative memory. Only the EBMM and the memory network could achieve near-zero error when storing 64 vectors and even though EBMM could not handle 96 vectors with this number of parameters, was the most accurate memory model.
A.3 EVALUATION OF NON- MEMORY BASELINES
For a more complete experimental study, we additionally evaluate a number of baselines which have no capacity to adapt to patterns that otherwise would be written into the memory but still learn prior over a domain and hence can serve form of associative memory.
One should note, however, that even though such models can, in principle, achieve relatively good performance by pretraining on large datasets and adopting well- architectures, they ultimately fail in situations where strong memory is more important a good prior. One very distinctive example of such setting is the binary string experiment presented in the previous section, where no prior can be useful at all. EBMM, in contrast, naturally learn both the prior in the form of shared features and memory in the form of writable weights.
Our first non- non-memory baseline is Variational Auto-encoder (VAE) model with 256 latent Gaussian variables and Bernouli likelihood. VAE defines an energy equal to the negative joint log-likelihood:
{x, z) = log Log.N(zz) (2((0, log p(x|z).
We consider attractor dynamics similar the one used by Wu et al. (0118) We start from a
configuration x()) 8, (0)) = μ(x), where μ(x) is the output of Gaussian encoder (z)xx)
(x2)x((x), (x))). Then alternate between updating these two parts of the configuration follows:
(1+1) (1) 2z
x (1+1) arg max E(x, E(x, ((+2)).
Since we use simple factorized Bemoulli likelihood model, the exact minimization with respect x can be performed analytically.
One can hope that under certain circumstances a well-trained VAE would assign lower energy levels less distorted versions of & and image We used 50, 100 and 200 iterations for Omniglot, CIFAR and ImageNet experiments respectively, in each case also performing grid search for the learning rate 7.
Another baseline is a Denoising Auto-encoder (DAF) which is trained to reconstruct the original pattern from its distorted version x using bottleneck of 256 hidden units. This model trained exactly as our model using (2) as an objective and hence, in contrast to VAE, can adapt to a particular noise pattern, which makes it a stronger baseline.
Finally, we consider the Deep Image Prior (Ulyanov al. 2018)) method that can be seen very special case of the energy-based model which also iteratively adapts parameters of convolutional network to generate the most plausible reconstruction. We used the network provided by the authors and performed 2000 Adam updates for each of the images. Similarly to how it was implemented in the paper. network was instructed about location of the occluded block which arguably a strong advantage over all other models.
The quantitative results of the comparison are provided in the Table Clearly, the most efficient of the baselines is DAE, largely because it explicitly trained against a known distortion model. In

---

Published conference paper at ICLR 2020
BASELINE
OMNIGLOT
| CIFAR
IMAGENET
25% BLOCK NOISE
VARIATIONAL. AUTO-ENCODER
109.08
544.72 24.66
1889.79
DENOISING AUTO-ENCODER 36.08
141.07
DEEP IMAGE PRIOR
170.36
15% SALT AND PEPPER NOISE
VARIATIONAL. AUTO-ENCODER 84.08 9.57
430.33 8.82
1385.29
DENOISING AUTO-ENCODER DEEP IMAGE PRIOR
69.20
467.92
Table 2: Reconstruction emrors (Hamming for Omniglot, squared for CIFAR and ImageNet) for non-memory baselines. *Location of the distorted block was used.
Hamming error
EBMM canv
DKM
30
20
10
2K
4K
6K
8K
10K
12K
14K
16K
Memory size
Figure 7: Reconstruction error on Omniglot. Dynamic Kanerva Machine compared EBMM with convolutional memory. 15% salt and pepper noise is used.
contrast, VAE failed to recover from noise producing a significantly shifted observation distribution. Deep Image Prior performed significantly better, however, since it only adapts to single distorted image instead of a whole distribution, it could not over-perform DAE with much simpler architecture. As expected, none of the non non-menen memory baselines performed even comparably models with memory we consider in the main experiments section.
A.4 COMPARISON WITH DYNAMIC KANERVA MACHINE
As we reported earlier, Dynamic Kanerva Machine failed to perform better than a random guess under the strong block noise we used in the main paper. In this appendix we evaluate DKM in simpler conditions where can be reasonably compared to EBMM. Thus, we trained DKM batches of 16 images and used a relatively simple 15% salt and pepper noise.
Figure 7 contains the reconstruction error in the same format as Figure 3a One can see that DKM generally performs poorly and does not show consistent improvement with increasing memory size. As can seen on Figure DKM able retrieve patterns that are visually similar to the originally stored ones, but also introduces a fair amount of undesirable variability and occasionally converges to a spurious pattern. The errors increase with stronger block noise and larger batch sizes.
A.5 GENERALIZATION TO DIFFERENT DISTORTION MODELS
In this experiment we assess how FBMM trained with one of block noise of one size performs with different levels of noise. Figure contains the generalization analysis with different kinds of noise Omniglot.
One can see that regardless of noise type, EBMM successfully generalizes to weaker noise and generalization to slightly stronger noise is also observed. One should note though that 20 x 20 block noise already covers almost 40% of the image and hence may completely occlude certain characters.

---

Published conference paper at ICLR 2020
F m 33 w
P
2
M
M
C
w
W1
VCC
w
25
C
21
P
F a La
C
1w)
D in
Figure 8: An example of a retrieval single batch by Dynamic Kanerva Machine. A model with 12K memory used. Top: salt and pepper noise, bottom: 16 16 block noise. First line: original patterns, middle line: distorted queries, bottom line: retrieved patterns.
0.4
10c16
Hamming enoo
0.2
Hamining errar
30
20
10
1x4
16 16
30
10%
20%
40%
50%
60%
Black
Noise rate
Block noise. Batches of 64 images.
(b) Salt and pepper noise. Batches of 16 images.
Figure Generalization different noise intensity on Omniglot. Each line represents model trained with certain noise level. All models use 16K memory.
Generally found salt and pepper noise much simpler which originally motivated us focus block occlusions. However, as we indicate on the figure, the model used experiment with kind of noise was only trained on batches of 16 images and hence the comparison of absolute values between Figures 9 and 9b is possible. The relative performance degradation with increasing noise rate is still representative though.
We did not observe generalization from type of noise to another. Perhaps, the very different kinds of distortions were not compatible with learned basins of attraction. In general, this is not very surprising as, arguably, no model can be expected adapt to all possible distortion models without a relevant supervision of some sort.
For this experiment we did not anyhow fine-tune model parameters, including the meta- parameters of the reading gradient descent. Hence, it is possible that some amount of fine-tuning and, perhaps, custom optimization strategy for modelled energy may lead to an improvement.
A.6 GENERALIZATION TO MNIST
natural question is whether model trained one task dataset strictly overfits its features whether it can generalize to similar, but previously unseen tasks, One of the standard experiments to test this ability is transfer from Omniglot to MNIST, since both datasets consist of handwritten characters, which, however, differ enough to present distributional shift.

---

Published conference paper at ICLR 2020
Iceration
Iteration
Iteration
Iteration
Iteration
Original
!
9
9
4
9
}
7
7
7
3
3
3
3
3
3
Energy
-20
Query
Iteration
Iteration
Iteration 3
Iteration
Iteration
Original
Figure 10: Iterative reading MNIST. The model learned Omniglot.
BATCH SAMPLING
4K MEMORY
5K MEMORY
UNIFORM (UNCORRELATED)
7.18
5.37
2 CLASSES (CORRELATED)
6.22
2.57
Table 3: Hamming error of models trained on differently constructed Omniglot batches of 32 images.
Developing such transfer capabilities is out of scope for this paper, but simple check confirmed that EBMM can successfully retrieve upscaled (to match the dimensionality) and binarized MNIST characters as one can see on Figure T0 One can see that although some amount artifacts introduced the retrieved images are clearly recognizable and the energy levels are as adequate as the Omniglot experiment, As in the previous experiment, we did not fine-tune the model.
A.7 EXPERIMENTS WITH CORRELATED BATCHES
One of the desirable properties of memory model would an efficient consolidation of similar patterns, corresponding to the same class of images, in the interest of better compression.
We do not observe this property in the models trained on uniformly sampled, uncorrelated batches, perhaps because the model was experiment where we trained EBMM on the Omniglot images, making sure that each batch 32 images has characters of exactly two classes. For this purpose we employed a modified convolutional model (see Appendix C.5) in which the memory weights are located in the second residual layer instead of the third as the main architecture. As we found experimentally. this way the model could compress more visual correlations in the batch.
incentivized do However, performed simple
The results be found in Table is evident that training correlated batches enabled better compression and EBMM is able learn an efficient consolidation strategy the meta-learning set up appropriately.
B READING IN EBMM
PROJECTED GRADIENT DESCENT
We described the basic reading procedure in section 2 however, there is a number of extensions we found useful practice.

---

Published conference paper at ICLR 2020
Since in all experiments we work with data constrained the [0, interval, one has to ensure that the read data also satisfies this constraint. One strätegy that is often used in the literature is to model the output as argument to sigmoid function (logits). This may not work well for values close to the interval boundaries due vanishing gradient, so instead we adopted projected gradient descent, i...
x ( proj Vx xxx(x))))
where the proj function clips data to [0, 1] interval.
Quite interestingly, this formulation allows more flexible behavior of the energy function. stored pattern x has one of the dimensions exactly the feasible interval boundary, e.g. ry = 0, then V., [x) does necessarily have to be zero, since z, will not be able to go beyond zero. We provide more information on the properties of storied patterns in further appendices.
B.2 NESTEROV MOMENTUM
Another extension we found useful to employ Nesterov momentum (Neseervv optimization scheme and we use it in all our experiments.
1983 into the
x project(x (R-11 + (1-11)) v(0) = = -1 (E(((()
(k) project(v(k))
B.3
STEP SIZES
To encourage learning converging attractor dynamics constrained step sizes "y be non- increasing sequence:
(K) -0)+)))))
Then the actual parameters to meta-learn is the initial step size (0)) and the logits n. We apply similar parametrization the momentum learning rates vp.
B.4 STEP-WIE WISE RECONSTRUCTION LOSS
As has often been found helpful in the literature (Belanger al. 2017 Antomiou al. 2018 we apply the reconstruction loss 2 not just to the final iterate of the gradient descent, but to all iterates simultaneously:
K
Cx(X,0)
E
x
ARCHITECTURE DETAILS
Below we provide pseudocode for computational graphs of models used in the experiments. All modules containing memory parameters specifically named memory.
C.I
GATED RNN
We used fairly standard recurrent architecture only equipped with an update gate as in (Chung al. 2014). We unroll the RNN for steps and compute the energy value from the last hidden state.
hidden_size = 1024
input_sis size 128
128 * (128 - 1) / 128 parameters in total
dynamic_skee size = (input_st) _sze 1) // 2
state repeat_batce (zercs (hidden_siiee
memory = Linear (input_size dynamic_size)

---

Published conference paper at ICLR 2020
gate Sequential
Linear (input_size hidden_size, [idden_size) sigmoid
}
static - Linear (input_size hidden_size, hidden_size dynamic_size)
for hop in xrange (5) concat (x, state)
dynamic_pan part = memory (x)
static_part = static
tanh (concat (dynamic_ static_part))
u gate (2)
state u
(1 u) state
energy = Linear (1) (state)
C.2 RESNET, FULLY-CONNECTED MEMORY
channels = 32
hidden_size 512
representation 512
static_size = zepresentation_ake _ize dynamic_siz size
state repeat_batcl (zeros (hidden_size))
encoder = Sequential ([
ResBlock (channels
1, kernel=13, 3), stride=2, downscale=false)
ResBlock (channels 2, kernel=(3, [3, 3), stride=2, downscale=false)
ResBlock (channels flatten,
kernel-[3, [3, 3], stride-2, cownscale-False)
Linear (256),
LayerNorm ()
1)
gate Sequential ([
Linear (hidden_size), sigmoid
}}
hidden Sequential Linear (hidcen_size),
tanh
})
x encoder (x)
memory = Linear fimpuu__iiee, dynamic_size)
dynamic_part memory (x)
static_part = Linear (static_siee (x)
tanh (concat (dynamic (dynamic_parts part, static_part))
for hop in xrange (3):
concat (x, state)
hidden (2)
LayerNorm () (c)
(z)

---

Published conference paper at ICLR 2020
state u (1 u)
h = tanh (Linear (1024) ( (state)) energy Linear (1) (h)
The encoder module is also shared with all baseline models together with its transposed version a decoder.
C.3 RESNET, CONVOLUTIONAL MEMORY
channels 32
x = ResBlock (charnels * 1, kernel=(3, 3], stride=2, dcwnscale=True) (x)
ResBlock (channels 2, kernel=(3, 3], stride=2, domnscale=True) (x)
def busblock_bottlenee (x, channels, bottleneck_ srr_leneckkchaneels, downscale=False) static_size channels dynamic_size
2 x
X Conv2D (battDbbbbleeenccchaaanaa channels, [1, 1]) (x)
X LayerNorm () (x)
tanh (x)
if downscale:
memory_pars Conv2D (dynamio_size kernel-[3, 31, stride-2, downscale-True) (x)
static_part Conv2D (static_size, kernel-(3, 3], stride-2, downscale-True) (x)
else:
memory_part. Conv2D (dynamic_size kernel=13, 3], stride=1, downscale=Falsse
static part Conv2D (static_siae kernel=(3, 3], stride=1, downscale=False) (x)
x concat Leclatic_pprr____ memory pamooryparrt,, -1)
LayerNorm) (x)
X tanh (x)
Conv2D (channels, s, kernel=[1, 1]) (2)
if downscale:
= avg_ (z, [3, 3] + (1), stride=2) +=
return
resblock_batttenne (x, channels 4, channels 2, False)
X resblock_bottleneet (x, channels 4, channels 2, True)
recurrent Sequential
Conv2D (hi dden_size kernel=[3, 3], stride=1), LayerNorm(), tanh })
update_gate gate = Sequential
Conv2D (hi dden_size, kernel=(1, 1], stride=1), LayerNorm(). signoid })
hidden_size = 128 hidden_state repeat_batoh (zeros (4, 4, hidden_size)
for hop in xrange (3)
concat (xx, #idden_stater -1)
20

---

Published conference paper at ICLR 2020
candidate recurrent (z)
u = update_gates (z)
hidden_state u candidate (1. u) hidden_state
x Linear (1024) (x)
tanh (x)
energy Linear (1)
C.4 RESNET, IMAGENET
This network is effectively slightly larger version of the ResNet with convolutional memory described above.
channels 64
dynamic_size
ResBlock (charnels 1, kernel=[3, 31, stride=2, downscale=Trus (x)
x
ResBlock (channels 2, kernel-[3, 3], stride-2, downscale-True) (x)
= resblock_bettleneet (x, channels 4, channels 2, True)
x resblock_botteeneed (x, channels 4, channels 2, True)
recurrent Sequential
Conv2D (hi siee kernel-(3, 3], stride-1), LayerNorm(),
tanh })
update Brratt_gaae_aaee gate = Sequential
Conv2D (hi size,, kernel-[1, 1], stride-1), LayerNorm(),
signoid
})
hidden_size = 256
hidden_state repeat_ (zeros (4, 4, sidden_size)
for hop in xrange (3)
z concat ([x, hidden_statel, -1)
candidate = zecurrent (z)
(z) hidden_state
candidate (1. u) hiidde_ssttee
x Linear (1024) (x)
tanh (x)
energy Linear (1)
C.5 RESNET, CONVOLUTIONAL LOWER-LEVER MEMORY
This architecture similar other convolutional architectures with only difference is that dynamic weights are in the second residual block.
channels 32 dynamic_size size =
def resblock (x, channels) static_size channels dynamic_size

---

Published conference paper at ICLR 2020
memory_part Conv2D (dynamic_saae size, kernel=(3, 3), stride=2, downscale=True) (x)
s static_part = Conv2D (static_size, kernel= 3], stride=2, downscale=True) (x)
X concat (staaiic_part, memory_part), -1)
X LayerNorm) (x)
tanh (x)
memory_part = Conv2D (dynamic_size, kernel=(3, 3), stride=2, downscale=True) (x)
atatic_part = Conv2D (static_size) kernel=13, 3], stride=2, downscale=Trun (x)
y concat ([static_part, memory_part]. -1)
y += x
y nonlinear (LayerNorm) (y))
x y
2 Conv2D (channels, [1, 1)) (z)
avq_pool (z, [3, 3], 2)
x 2
return
X ResBlock (channels * 1, kernel-(3, 3], stride-2, downscale-True) (x)
resblock (x, channels 2)
ResBlock (channels 4, kernel=[3, 3], stride=2, downaca l==True) (x)
recurrent Sequential
Conv2D (hi dnidddinsiee kernel=(3, 3], stride=1), LayerNorm(), tanh })
update_gate = Sequential Conv2D (hi dden_size, kernel=[1, 1], stride=1), LayerNorm(),
sigmoid
}}
hidden_size = 256
hidden_state repeat_bate (zeros (4, 4, hidden_size) siz)))
for hop in xrange (3)
concat (xx, sidden_statel] -1)
candidate = zecurrent (2)
u update_gate hidden_state state
candidate (1. u) hidden state
x Linear (1024) (x)
tanh
energy Linear (1)
C.6 THE ROLE OF SKP-CONNECTIOON IN ENERGY MODELS
Gradient-base meta-lcarning and EBMM in particular rely on the expressiveness not just the forward pass of network. but also the backward pass that is used to compute gradient. This may require special considerations about the network architecture.

---

Published conference paper at ICLR 2020
With grad loss
0.030
Without grad loss
0.025
0.020
0.010
0.005
0
Iterations (10*)
Figure II: Effect of including the [xx]]] term in the writing loss
Omniglot.
One may notice all energy models considered above have an element recurrency of some sort. While the recurrency itself is not crucial for good performance, skip-connections, of which recurrency is special case, are.
We can illustrate this by considering energy function of the following form:
E(x) = o(h(x)), h(z) = f(x) + s(f())).
Here think of representation from which the energy is computed. We allow the representation be computed as (() and then refined by adding g(
During retrieval, we use gradient of the energy with respect to x which can be computed as
do dh do
df dg df df de
E(x)
dx
dh dr dh
dx
One can see, that with skip-connection the model is able to refine the gradient together with the energy value.
simple way of incorporating such cxip-connestions via recurrent computation. We allow the model to use gating mechanism that can modulate the refinement and prevent from unneces- sary updates. We found that usually small number of recurrent steps (3-5) is enough for good performance.
D EXPLANATIONS ON THE WRITING LOSS
Our setting deviates from the standard gradient- based meta-learning described (Finn al. 2017). In particular, we are not using the same loss function (naturally defined by the energy function) adaptation and inference phases. As explain in section B writing loss (3) besides just the energy term also contains the gradient term and the prior term.
Even though we found it sufficient use just the energy value as writing loss, perhaps not surprisingly, minimizing the gradient norm appeared to help optimization especially in the early training (see Figure [1] and lead to better final results.
We use an individual learning rate per each writable layer and each of the three loss terms, initialized at 10 and learned together with other parameters. We used softplus function to ensure that all learning rates remain non-negative.
23


Reviews:
[{"review_id": "HyekxA0EcS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Thanks for the extensive answers. I updated my rating based on the provided clarification and extra experiments.\n\n=====================\nMy understanding of eq 1-5 is that the algorithm finds an energy landscape (by modifying \\theta) for each dataset (task) such that in this landscape, the inputs from the distribution are reachable by truncated gradient-descent initiating at a query (distorted input with respect to some distortion model).\n\n1- The connection to meta-learning is unclear in your experiments. Can you elaborate on that? \n\n2- The expectation in eq 5 is over different input patterns, which I assume that a set of input patterns belong to a task. What is that you write in memory?  For each experiment, what are the different input patterns (tasks) that you have written in the memory?\n\n3- What is the \\theta that is feed to the read function at the test time? \n\n4- Are you testing on the tasks that you already trained on?\n\n5- How this approach generalizes to unseen (or relatively close) task? \n\n6- Can it recover any query that is not constructed with respect to the distortion model that is trained on? or what happens if the distorted image at test times comes from a different distortion model? (image blocking, for example)\n\n7- How many distorted samples are used for training?\n\n\n8- For the chosen tasks, I am curious to see the experimental comparison to deep image prior (Ulyanov, 2018). Deep image prior would be very similar to the read operator (although the gradient descent is over the parameter of model) without having write operations when you define the energy as MSE.\n\n\n\nTypos and writing style:\n-- The expectation in eq 2 should independently show what the expectation is taken with respect to. \n-- input patters -> input patterns\n-- figure 3 -> Figure 3\n-- section 1 -> Section 1\n-- 4 random images -> four random images \n-- the Figure 5b -> Figure 5b\n-- Models such as (Ba et al., 2016; Miconi et al., 2018) enable ->  Models such as Ba et al. (2016) and Miconi et al. (2018) enable", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Hye1YmQTqr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposes a new type of energy-based models, a class of non-normalized generative models that relies on an energy function to retrieve patterns that correspond to its minima. The goal that is tackled by the authors is to implement an associative memory system, i.e. a mechanism that is able to retrieve any one of a set of patterns, given a distorted copy of these patterns. This task is traditionally carried out using attractor neural networks like the Hopfield model, a recurrent neural network model endowed with a learning rule that allows it to quickly embed a given set of patterns in its weight matrix such that the patterns become stable fix points of its dynamics. As the authors point out though, models like the Hopfield model are limited in their capacity to assimilate attractor patterns and in term of their expressiveness. On the other hand, more complex models based on deep architectures trained with gradient descent are slow at updating their weights to create new attractors.\nThe authors propose a new method to make up for the weaknesses of these two approaches. Their method is based on meta-learning, and in short consists in meta-training an energy function parametrized as a neural network such that  executing a write dynamics on the weights results in a model whose read dynamics (a gradient descent on the energy function) is able to denoise distorted inputs and retrieve the original ones. In practice, the write dynamics is obtained as a gradient descent procedure on a writing loss (which is itself dependent on the energy function) as a function of the weights. The meta-learning procedure minimizes the discrepancy between the original patterns and the retrieved ones by optimizing end-to-end the learning schedule parameters and initial conditions of the weights, analogously to gradient-based meta-learning methods like MAML.\nThe authors then carry out a series of experiments to check that their model is competitive with Memory-Augmented Neural Network (MANN) and Memory Networks (MemNets) in retrieving samples from Omniglot, CIFAR and ImageNet, in terms of retrieving abilities for a given memory size. In the supplementary material section they in addition compare their model's performance against the Hopfield model and recurrent networks on the classical toy task of retrieving random binary patterns, also with good results for the new model.\n\nDecision:\nThis paper is very clearly and compactly written. The idea of training an energy-based model through gradient-based meta-learning seems novel and innovative. \nOne thing that the the paper is arguably missing, is a convincing motivation section for focusing on energy-based models. The panorama of generative models has radically changed since attractor neural networks and energy-based models were first introduced. At the time powerful methods like variational autoencoders, normalizing flows and GANs didn't exist. But nowadays, one could arguably expect that energy-based models should be contextualized and motivated in the perspective of comparing them with these new breeds of deep generative models. I am absolutely not suggesting that the authors should providing experimental comparisons between their models and GAN or VAE, but simply that they compare them to their style of generative modeling in terms of advantages, disadvantages, use cases, and potential for applications.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BkeZXMcccB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Summary:*\n\nThe authors propose to tackle the associative memory problem by recasting read/write operations to read/write by optimizing the parameters/input of an energy based model. Writing is reformulated as training a parametric energy model (EBMM) to have local minima of energy w.r.t. the parameters at memorized data points. Reading is performed by performing (projected) gradient descent on the corrupted/incomplete input to minimize the energy. To ensure the operations are fast (read and write with minimal gradient steps), the authors propose to take inspiration from modern meta-learning literature and learn initialization parameters of the energy model (and other hyperparameters for GD during read/write) from which writing is fast while ensuring reading is also fast, since the models are trained to maximize read/write performance within a constrained number of gradient steps.  Experimentally, the authors show that EBMM reading performs similar to baseline methods (but better across many memory sizes) on the standard Omniglot task. On CIFAR-10 and downsized ImageNet, they show much better L2 reconstruction error of corrupted images. They also show that the learnt energy \n\n*Recommendation:*\n\nI believe this is a very neat idea, and utilizes large parametric models for \"smart\" overfitting and compression of data for the associative memory task. The proposed meta-learning approach to training the model seems to perform well across multiple simple and challenging datasets, and therefore I would recommend accept. My current recommendation is very borderline (weak accept) because of a lack of some experimental rigour (which I would love clarifications on), and missing related work, which I mention below.\n\n*Discussion Points and Concerns from the Reviewer:*\n\n- Dataset / batching details \nPlease mention how the datasets were split for training and testing the models. How much training data is utilized to meta-learn the EBMM initialization? How is batching performed? I believe these details are very important to mention in the paper for reproducibility of results. \n\nAre there any correlations in the batch selection? Can you evaluate how good the associative memory performs across different correlation levels in the batch (A well learnt algorithm should demonstrate better reconstruction at lower memory levels for correlated batches). \n\n- Experiments across multiple SNR and generalization on noise patterns\nThe authors mention at the beginning of Section 4 that a random block is corrupted, but in the end the experiments are done on a constant corruption size on the CIFAR and ImageNet images. How do the models perform across different signal-to-noise ratios? Similarly, the model is trained on simple noise patterns \n\n- Missing related work\nThere is related work [1] in learning in Hopfield Networks using the implicit function theorem and finding stationary points of the dynamics. This work is not mentioned in the paper, and is a valid baseline for this paper as well.\n\n- Mentioning Appendix D in the main paper\nAppendix D is not mentioned in the main paper and has a short discussion on the mismatch between the reading process and the writing loss during meta-training. It also mentions additional tricks required for the training, and I believe it should be mentioned in the main paper like other sections are appropriately referenced. \n\n- Large batch sizes for ImageNet\nWork from [2] can be utilized to backpropagate through very long optimization sequences and therefore can be utilized to train with larger batch sizes in the ImageNet example. It is important to see how the small model utilized for ImageNet works to compress higher batch sizes, as that is one of the major practical issues with the algorithm.\n\n- Related paper at NeurIPS this year \n[3] is a related paper from Neurips this year, which the authors could consider adding as contemporary work\n\n- Comments on scalability\nThe associative memory papers have often been criticized for lack of scalability, and I think the authors make progress towards making this better with the use of unconstrained energy models in the learning process. It would be nice to have a discussion of the scalability from the authors, highlighting issues in the current model and future directions\n\nReferences:\n[1] Reviving and Improving Recurrent Back-Propagation, ICML '18\n[2] Gradient-based Hyperparameter Optimization through Reversible Learning, Maclaurin et al. ICML '15\n[3] Metalearned Neural Memory, Munkhdalai et al. NeurIPS '19", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1e_L5zpKH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n======================================== Update after revisions ============================================\n\nI appreciate the effort the authors have put into the revision and the rebuttal. I'm happy to increase my score and recommend acceptance based on the revised paper. \n\nHowever, I have to say that some of my worries still linger. With respect to non-memory baselines, the authors have responded that the memory based models will outperform non-memory models in cases where prior structure is less important than memory and provided a demonstration of this with an extreme example, i.e. a case with no structure (random binary strings example). I understand the point being made here, but this is a rather pedantic and uninteresting example. The authors have provided another (more interesting) example in Appendix A3 and shown that the memory-based model outperforms some simpler baselines such as the DAE even in this case. But no explanation is given for this result. Why is the memory based model outperforming the DAE in this case, given that this is an example where prior *is* very important? I'm a bit worried that the DAE results may perhaps be due to a non-optimized architecture or training setup (and what exactly is the architecture used for the DAE here)? I would appreciate it if the authors could clarify these issues in the final version.\n\nI have also spotted several typos. For the final version, please make sure to go through the paper thoroughly at least once and fix all the typos.\n\n========================================================================================================\n\nThis paper proposes a meta-learning approach to learning fast read and write mechanisms in an energy-based model so that a given set of images can be quickly inducted into memory and retrieved from memory with noisy queries. The paper is well-written and the proposed approach seems interesting and novel enough. However, I have some concerns about the paper that need to be addressed. Here are the main issues for me:\n\n1) I am in general not really convinced about the supposed advantages of these attractor memory models (this paper and the earlier Kanerva machine) over more standard and much simpler approaches. For example, for the problem of retrieval from noisy queries, a more standard approach would be a simple autoencoder. Note that in an autoencoder, reading (inference) is already fast. The authors might point out that writing (training) will not be fast, which is correct. However, the meta-learning phase proposed in this paper will also not be fast and perhaps the fair comparison should be between the meta-learning phase of this paper and the standard training phase of an autoencoder. Note that the autoencoder will have additional benefits. For example, with the autoencoder, one is not constrained by memory storage requirements and can make use of a much larger set of images to train the model. This allows the model to learn a richer structure in images. Moreover, with a large enough feedforward net, one can approximate arbitrarily complex dependencies in images. However, in attractor memory models, on the other hand, one necessarily restricts oneself to a particular model class that can be expressed as gradient descent dynamics in an energy landscape both during reading and writing. This seems overly restrictive to me. So, perhaps, the authors can clarify the supposed advantages of these attractor memory models a bit better. For example, I would be interested in seeing some comparative results with, say, a denoising autoencoder model.\n\n2) Currently, the paper only uses a specific type of “block-noise” corruption. One thing that would be nice to see is some results with other noise models. I think this is important to demonstrate that the approach is general enough to handle different kinds of noise. Also, a salt-and-pepper noise will allow the authors to compare their results with the dynamic Kanerva machine (the authors note that the DKM failed to train successfully for the block-noise used here). \n\n3) It would be good to say something about the meta-learned parameters, theta_bar, r, tau. Is there any meaningful structure in these parameters that distinguishes them from their random initial values? Is one of these parameters more important than the others? For example, what happens if you just use generic step size decay rules for gamma and eta (or perhaps no decay at all)?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Four knowledgable reviewers recommend accept. Good job!
Decision: Accept (Poster)