Multigrid Neural Memory
Authors: Tri Huynh
Michael Maire
Matthew R. Walter
Venue:  (2020)
Keywords: multigrid architecture
memory network
convolutional neural network

Abstract: We introduce a novel architecture that integrates a large addressable memory space into the core functionality of a deep neural network.  Our design distributes both memory addressing operations and storage capacity over many network layers.  Distinct from strategies that connect neural networks to external memory banks, our approach co-locates memory with computation throughout the network structure.  Mirroring recent architectural innovations in convolutional networks, we organize memory into a multiresolution hierarchy, whose internal connectivity enables learning of dynamic information routing strategies and data-dependent read/write operations.  This multigrid spatial layout permits parameter-efficient scaling of memory size, allowing us to experiment with memories substantially larger than those in prior work.  We demonstrate this capability on synthetic exploration and mapping tasks, where the network is able to self-organize and retain long-term memory for trajectories of thousands of time steps.  On tasks decoupled from any notion of spatial geometry, such as sorting or associative recall, our design functions as a truly generic memory and yields results competitive with those of the recently proposed Differentiable Neural Computer.

Under review as a conference paper at ICLR 2020
MULTIGRID NEURAL MEMORY
Anonymous authors Paper under double-blind review
ABSTRACT
We introduce novel architecture that integrates large addressable memory space into the core functionality of a deep neural network. Our design distributes both memory addressing operations and storage capacity over many network layers. Distinct from strategies that connect neural networks to external memory banks, our approach co-locates memory with computation throughout the network structure. Mirroring recent architectural innovations in convolutional networks, we organize memory into multiresolution hierarchy, whose internal connectivity enables learning of dynamic information routing strategies and data-dependent read/write operations. This multigrid spatial layout permits parameter-efficien scaling of memory size, allowing us to experiment with memories substantially larger than those in prior work. We demonstrate this capability on synthetic exploration and mapping tasks, where the network is able to self-organizz and retain long-term memory for trajectories of thousands of time steps. On tasks decoupled from any notion of spatial geometry, such sorting associative recall, our design functions as a truly generic memory and yields results competitive with those the recently proposed Differentiable Neural Computer (Graves et al., 2016).
INTRODUCTION
Memory, in the form of generic, high-capacity, long-term storage, is likely to play a critical role in expanding the application domain of neural networks. A trainable neural memory subsystem with such properties could be a transformative technology paapologgyppsiiigggggoge neural networks within grasp of tasks traditionally associated with general intelligence and an extended sequence of reasoning steps. Development of architectures for integrating memory units with neural networks spans good portion of the history of neural networks themselves (e.g. from LSTMs (Hochreiter & Schmidhuber, 1997) to the recent Neural Turing Machines (NTMs) (Graves et al., 2014)). Yet, while useful, none has elevated neural networks be capable of learning from and processing data on size and time scales commensurate with traditional computing systems. Recent successes of deep neural networks, though dramatic, are focused on tasks, such as visual perception or natural language translation, with relatively short latency- e.g. hundreds of steps, which is often also the depth of the network itself.
We present a design for neural memory subsystems that differs dramatically from prior architectures in the aspect of read/write interface, and, as consequence, facilitates parameter-efficcent scaling of memory capacity. Our design draws upon both established ideas, as well a key recent advance: multigrid convolutional networks implicitly capable of learning dynamic routing mechanisms and attentional behavior (Ke et al., 2017). LSTMs serve as a fine-grained component, but are wrapped within a larger multigrid connection topology, from which novel capabilities emerge.
In contrast to NTMs and the subsequent Differentiable Neural Computer (DNC) (Graves et al., 2016), we intertwine memory units throughout the interior of a deep network. Memory is first-class citizen, rather than a separate data store accessed via a special controller. We introduce a new kind of network layer-a multigrid memory layer-and use networks. Contrasting with simpler LSTMs, our memory is truly deep; accessing an arbitrary memory location requires passing through several layers. Figure 1 provides visualization of our approach; we defer the full details to Section 3. There are major benefits to this design strategy, in particular:
stackable building block to create deep memory
Memory scalability. Distributing storage over multigrid hierarchy allows to instantiate large amounts while remaining parameter-efficient. Read and write operations are similarly distributed.
The low-level mechanism underlying these operations is essentially convolution, and we inherit the parameter-sharing efficiencies of convolutional neural networks (CNNs). Our parameterized

---

Under review as a conference paper at ICLR 2020
Conn-LLTM
comV- LLTM
(c.A)
conV -.5TM
(c, A1
Mulligrrd Convolutional Layer y- (10.31 32]
Multigrid Memory Layer
MG cort LS TM
50.1
MO Cone IESTT
MO coney LITT
MG -aIIN LSTTM
MG -00---------- LLSM
de (input)
Multigrid Memory Network
Multigrid memory architecture. Top Left: A multigrid convolutional layer (Ke et al., 2017) transforms input pyramid X, containing activation tensors (In, ,,, into output pyramid y via learned filter sets that act across the concatenated representations of neighboring spatial scales. Top Right: We design an analogous variant of the convolutional LSTM (Xingjian et al., 2015), in which X and y are indexed by time and encapsulate LSTM internals, e.g. memory cells (c), hidden states (h), and outputs (o). Bottom: Connecting many such layers, both in sequence and across time, yields multigrid mesh capable of routing input a, into much larger memory space, updating distributed memory representation, and providing multiple read-out pathways (e.g. 20,t or 23,1).
Figure I:
filters simply act across a spatially organized collection of memory cells, rather than the spatial extent of an image. Increasing feature channels stored in each spatial cell costs parameters, but adding more cells incurs no such cost, decoupling memory size from the number of parameters. The multigrid layout of our memory network provides information routing mechanism that is efficient with respect to overall network depth. We can grow the spatial extent of memory exponentially with depth, while guaranteeing there a connection pathway between the network input and every memory unit. This allows experimentation with substantially larger memories.
Unification of compute and storage. Our memory layers incorporate convolutional and LSTM components. Stacking such layers, we create not only a memory network, but also a generalization of both CNN and LSTM. Our memory networks are standard networks with additional capabilities. Though not our primary experimental focus, Section 4 shows they can learn tasks which require performing classification alongside storage and recall.
A further advantage to this unification is that it opens a wide design space for connecting our memory networks to each other, as well as standard neural network architectures. For example, within a larger system, we could easily plug the internal state of our memory into a standard CNN granting that CNN read-only memory access. Sections 3 and 4 develop and experimentally validate two such memory interface approaches.
diverse array of synthetic tasks serves as our experimental testbed. Mapping and localization, an inherently spatial task with relevance to robotics, is one focus. However, we want to avoid only experimenting with tasks naturally fit to the architecturally-iddeeee Therefore, we also train them to perform the same kind of algorithmic tasks used in analyzing the capabilities of NTMs and DNCs. Throughout all experimental settings, DNC accuracy serves as a comparison baseline. We observe significant advantages for multigrid memory, including:
biases of our memory networks.
Long-term large-capacity retention. On spatial mapping tasks, our architecture retains large, long-term memory. It correctly remembers observations an external environment collected over paths consisting of thousands of time steps. Visualizing internal memory unit activations actually reveals the representation and algorithmic strategy our network learns order to solve the problem. The DNC, in contrast, fails to master this category of task.
2

---

Under review as a conference paper at ICLR 2020
Generality. On tasks decoupled from any notion of spatial geometry, such associative recall or sorting, our memory networks prove equally as capable as DNCs.
Section 4 further elaborates experimental results, while Section 5 summarizes implications. Before turning technical details (Section 3), we review the history of coupling neural networks to memory.
2 RELATED WORK
There is an extensive history of work that seeks to grant neural networks the ability to read to and write from memory (Das et al., 1992; 1993; Mozer & Das, 1993; Zeng al., 1994; Hölldobler et al., 1997). Das et al. (1992) propose neural network pushdown automaton, which performs differential push and pop operations on external memory. Schmidhuber (1992) uses two feedforward networks: one that produces context-dependent weights for the second network, whose weights may change quickly and can be used as a form of memory. Schmidhuber (1993) proposes memory addressing in the form of "self-referential" recurrent neural network that is able to modify its own weights.
Recurrent neural networks with Long Short-Term Memory (LSTMs) (Hochreiter Schmidhuber, 1997) have enabled significant progress translation (Sutskever et al., 2014), speech recognition (Graves et al., 2013), and image caption- ing (Donahue et al., 2017). LSTMs are Turing- corpleec (Siegelmann & Sontag, 1995) and are, in principle, capable of context-dependent storage and retrieval over long time periods (Hermans & Schrauwen, 2013). However, network's is capacity for long-term read-write is sensitive to the training procedure (Collins et al., 2017) and is limited in practice.
variety of sequential prediction tasks, including machine
Grid LSTMs (Kalchbrenner et al., 2015) arrange LSTM cells in a 2D 3D grid, placing recurrent memory links along all axes of the grid. This sense of grid differs from our usage of multigrid, as the latter refers links across multiscale spatial layout. In Kalchbrenner et al. (2015)'s terminology. our multigrid memory networks are not Grid LSTMs, but are variant of Stacked LSTMs (Graves et al., 2013).
In an effort to improve the long-term read-write abilities of recurrent neural networks, several modi- fications have been recently proposed. These include differentiable attention mechanisms (Graves, 2013; Bahdanau et al., 2014; Mnih et al., 2014; Xu et al., 2015) that provide a form of content-based memory addressing, pointer networks (Vinyals et al., 2015) that "point to" rather than blend inputs, and architectures that enforce independence among the neurons within each layer (Li et al., 201 20.
number of methods augment the short- and long-term memory internal to recurrent networks with external "working" memory, in order to realize differentiable programming architectures that can learn to model and execute various programs (Graves et al., 2014; 2016; Weston et al., 2015; Sukhbaatar al., 2015; Joulin & Mikolov, 2015; Reed & de Freitas, 2015; Grefenstette al., 2015; Kurach et al., 2015). Unlike our approach, these methods explicitly decouple memory from computation, mimicking standard computer architecture, A neural controller (analogous to a CPU) interfaces with specialized external memory (e.g., random-access memory or tapes).
The Neural Turing Machine (NTM) augments neural networks with hand-designed attention-based mechanism to read from and write to external memory in a differentiable fashion. This enables the NTM learn to perform various algorithmic tasks, including copying, sorting, and associative recall. The Differential Neural Computer (Graves et al., 2016) improves upon the NTM with support for dynamic memory allocation and additional memory addressing modes. Without a sparsifying approximation, DNC runtime grows quadratically with memory due to the need to maintain the temporal link matrix. Our architecture has no such overhead or approximations, nor does require maintaining an auxiliary state.
Other methods enhance recurrent layers with differentiable forms of a restricted class of memory structures, including stacks, queues, and dequeues (Grefenstette al., 2015; Joulin & Mikolov, 2015). Gemici et al. (2017) augment structured dynamic models for temporal processes with various external memory architectures (Graves et al., 2014; 2016; Santoro et al., 2016).
Similar expory-e architectures have been proposed for deep reinforcement learning (RL) tasks. While deep RL has been applied successfully to several challenging domains (Mnih et al., 2015; Hausknecht & Stone, 2015; Levine et al., 2016), most approaches reason over short-term representations of the state, which limits their ability to deal with partial observability inherent in
3

---

Under review as a conference paper at ICLR 2020
many RL tasks. Several methods augment deep RL architectures with external memory to facilitate long-term reasoning. Oh et al. (2016) maintain a fixed number of recent states in memory and then read from the memory using soft attention- dieention-based read operation. Parisotto & Salakhutdinov (2018)
propose a specialized write operator, together with a hand-designed 2D memory structure, both specifically designed for navigation in maze-like environments.
Rather than learn when to write to memory (e.g., as done by NTM and DNC), Pritzel et al. (2017) continuously write the experience of an RL agent to a dictionary-like memory module queried in a key-based fashion (permitting large memories). Building on this framework, Fraccaro et al. (2018) augment a generative temporal model with specialized form of spatial memory that exploits privileged information, including an explicit representation of the agent's 's position in the environment.
Though we experiment with RL, our memory implementation contrasts with this past work. Our multigrid memory architecture jointly couples computation with memory read and write operations, and learns how to use a generic memory structure rather than one specialized particular task.
MULTIGRID MEMORY ARCHITECTURES
To endow neural networks with long-term memory, we craft an architecture that generalizes modern convolutional and recurrent designs, embedding memory cells within the feed-forward computational flow of a deep network. Convolutional neural networks and LSTMs (specifically, the convolutional LSTM variety (Xingjian et al., 2015)) exist as strict subsets of the full connection set comprising our multigrid memory network. We even encapsulate modern residual networks (He et al., 2016). Though omitted from diagrams for the sake of clarity, in all experiments we utilize residual connections linking the inputs of subsequent layers across the depth (not time) dimension of our memory networks.
Implementing memory addressing behavior is the primary challenge when adopting design philosophy. If the network structure is uniform, how will it be capable of selectively reading from and writing to only sparse, input-dependent subset of memory locations?
common approach is to build an explicit attention mechanism into the network design. Such attention mechanisms, independent of memory, have been hugely influential in natural language processing (Vaswani et al., 2017). NTMs (Graves et al., 2014) and DNCs (Graves et al., 2016) construct memory addressing mechanism by explicitly computing soft attention mask over memory locations. This naturally leads to a design reliant on an external memory controller, which produces and then applies that mask when reading from writing to a separate memory bank.
Ke et al. (2017) recently proposed a multigrid variant of both standard CNNs and residual networks (ResNets). While their primary experiments concern image classification, they also present striking result on a synthetic image to image transformation task: multigrid CNNs (and multigrid ResNets) are capable of learning to emulate attentional behavior. Their analysis reveals that the network's multigrid connection structure is both essential to and sufficient for enabling this phenomenon.
The underlying cause is that bi-directional connections across scale-space hierarchy (Figure 1, left) create exponentially shorter signalling pathways between units at different locations on the spatial grid. Information can be efficiently routed from one spatial location to any other location by traversing only few network layers, flowing up the scale-space hierarchy, and then back down again.
We convert the inherent attentional capacity of multigrid CNNs into an inherent capacity for distributed memory addressing by replacing convolutional subcomponents with convolutional LSTMs (Xingjian et al., 2015). Grid "levels" no longer correspond to operations on a multires- olution image representation, but instead correspond to accessing smaller larger storage banks within a distributed memory hierarchy. Dynamic routing across scale space (in the multigrid CNN) now corresponds to dynamic routing into different regions of memory, according a learned strategy. Appendix A provides an analysis of this dynamic routing capability. This routing mechanism is the foundation that enables the addressing capability in multigrid memory architectures.
3.1 MULTIGRID MEMORY LAYER
Figure diagrams both the multigrid convolutional layer of Ke et al. (2017) and our corresponding multigrid memory, or MG-Cony-LSTM, layer. Activations particular depth in network consist of pyramid X; {oo he,,, (,,)), where j indexes the pyramid level and t indexes time. o", h², denote the output, hidden state, and memory cell contents of convolutional LSTM (Xingjian et al.,
4

---

Under review as a conference paper at ICLR 2020
BA"
Reader
Encoder
TEA
C4
Writer
Decoder
Reader
Figure 2: Memory interfaces. Left: Multiple readers (red, orange) and a single writer (blue) simultancously manipulate a multigrid memory. Readers are multigrid CNNs; each convolutional layer views the hidden state of the corresponding grid in memory by concatenating it as additional input. Right: Distinct encoder (green) and decoder (purple) networks, each structured as deep multigrid memory mesh, cooperate perform a sequence-to- -o-suuence task. We initialize the memory pyramid (LSTM internals) of each decoder layer by copying it from the corresponding encoder layer.
2015), respectively. Following the construction of Ke et al. (2017), outputs o² neighboring scales are resized and concatenated, with the resulting tensors fed as inputs to the corresponding scale- specific convolutional LSTM units in the next multigrid layer. The state associated with conv-LSTM unit at a particular layer and level, say (o), (0,sh ha.h o (,,)). is computed from memory: h²,-- and - and input tensor: fo8-1, @ of, @ Jof where f. 4. and @ denote upsampling, downsampling, and concatenation, respectively. Specifically, multigrid memory layer (Figure 1, top right) operates as:
Of,- (10f-1,)) @ (of,,) @ 11t)+
y 8,2 = (W," -f,, + Wh * h² + W o 5,1-1 + b³)
(2)
f. == o (W Of,,- wha h w cy ,-,1- + b')
(3)
C0, = J o ch --1 + i tan * 0f, + Who * h + b9)
(4)
of o (W O2,- o h h + Wj c/, + +b)
(5)
1H, = o", o tanb(c))))
(6)
Superscripts denote variable roles (e.g. layer x or y, and/or a particular parameter subtype for weights or biases). Subscripts index pyramid level j and time t. * denotes convolution, Hadamard product. Computation resembles Xingjian et al. (2015), with additional input tensor assembly, and repetition over output pyramid levels j. If a particular input pyramid level is not present in the architecture, it is dropped from the concatenation in the first step. Like Ke et al. (2017), downsampling (J) includes max-pon max-poolingg We utilize wooddimensional memory geometry, and change resolution by factor of 2 in each spatial dimension when moving up or down a pyramid level.
Connecting many such memory layers yields memory network or distributed memory mesh, as shown in the bottom diagram Figure 1. Note that a single time increment (from of running an entire forward pass of the network, propagating the input signal o, to the deepest layer z. Though not drawn here, we also incorporate batch normalization layers and residual connections along grids of corresponding resolution (i.e., from of,, to o,,,). These details mirror Ke et al. (2017). The convolutional nature of the multigrid memory architecture, together with its routing capability (detailed in Appendix A) provides parameter-efficient addressing of scalable memory space.
to t) consists
3.2
MEMORY INTERFACES
As our multigrid memory networks are multigrid CNNs plus internal memory units, we are able to connect them to other neural network modules as freely and flexibly as one could do with CNNs. Figure diagrams couple possibilities, which we experimentally explore in Section 4.
On the left, multiple "threads", two readers and one writer, simultaneously access a shared multigrid memory. The memory itself is located within the writer network (blue), which is structured as deep multigrid convolutional LSTM. The reader networks (red and orange), are merely multigrid CNNs, containing internal storage, but observing the hidden state of the multigrid memory network.
5

---

Under review as a conference paper at ICLR 2020
Pollcy Network (Multigni CNN)
Actin
Multigrid Memory Network
Osservation
Obs.: 3
Query: 3
Pyreed
deepeet lppe shown
SUMEROS DAOGOO
Query Natwork (Multigrid CNN)
Obs.: 3 3
Quary: 9
World Mep
Figure 3: Mapping, localization, and exploration. An agent is comprised of a deep multigrid memory, and two deep multigrid CNNs (query and policy subnetworks), which have memory read access. Navigating a maze, the agent makes a local observation at each time step, and chooses next action, receiving reward for exploring unseen areas. The query subnet, given a local patch, must report all previously observed maze locations matching that patch. Subnet colors reflect Figure 2. Left: We view the mean activation, across channels, for LSTM memory cells {cg.e) in the network's deepest layer. Right: Training only memory and query subnets, using fixed action sequence (spiral motion), clearly shows the network learns an interpretable strategy for writing and reading memory. Memory contents (hidden states {hr} } on deepest, righest-resolutionn grid) mirror the explored map.
The right side of Figure 2 diagrams a deep multigrid analogue a standard paired recurrent encoder and decoder. This design substantially expands the amount of addressable memory that can be manipulated when learning common
tasks.
EXPERIMENTS
We first consider a RL-based navigation problem, whereby an agent is tasked with exploring a priori unknown environments with access to only observations of its immediate surroundings. Learning an effective policy requires maintaining consistent representation of that environment (i.e., a map). Using memory form of map, an agent must learn where and when to perform write and read operations as it moves, while retaining the map over long time periods. This task mimics related partially observable spatial navigation scenarios considered by memory-based deep RL frameworks.
Problem Setup: The agent navigates an n X 2D maze with access to only observations of the m m grid (m & n) centered the agent's 's position. It has no knowledge of its absolute position. Actions consist of one-step motion in each of the four cardinal directions. While navigating, we query the network with randomly chosen, previously seen, &o x patch (m S & n) and ask to identify every location matching that patch in the explored map. See Figure 3 (left).
Multigrid Architecture: We use deep multigrid network with multigrid memory and multigrid CNN subcomponents, linked together as outlined in Figure 8. Here, writer consists of 7 MG-conv- LSTM layers, with maximum pyramid spatial scale progressively increasing from 3x3 to 48 48. The reader, structured similarly, has an output attached to its deepest 48 48 grid, and is tasked with answering localization queries. Figure provides an illustration, while Appendix B. provides additional details. Section 4.2 experiments with additional reader network that predicts actions that drive the agent to explore the maze.
4.1 MAPPING & LOCALIZATION
In order to understand the network's ability to maintain a "map" of the environment in memory, we first consider a setting in which the agent executes pre-defined navigation policy and evaluate its localization performance. We consider different policies (spiraling outwards or a random walk), patch sizes for observation and localization (3x3 3 or 9x9), as well different trajectory (path) lengths. We compare against the following baselines:
Differentiable Neural Computer (DNC) (Graves et al., 2016). See Appendix C for details.
Ablated MG: multigrid architecture variant including only the finest pyramid scale each layer.

---

Under review as a conference paper at ICLR 2020
Table 1:
Mapping and localization performance. Our multigrid network for mapping and local- ization, shown in Figure 3. outperforms the DNC and other basclines by significant margin. Large memory capacity, enabled by multigrid connectivity, is essential; the DNC even fails to master smaller 15x 15 mazes. Our system retains memory over thousands of time-steps. Our localization subnet, trained on random motion, generalizes to answering queries for policy-driven agent (bottom row).
Params
Memory
World
Task Definition
Path
Localization Accuracy
Architecture
(x109)
(x10²)
Map
FoV
Motion Query
Length
Prec. (5)
Recall (%)
F
MG Mcm+CNN
0.65
76.97
99.99
99.97
99.98
Ablated MG
1.40
265.54
77.57
51.27
61.73
ConyLSTM-deco
0.38
626.69
25x25
529
43,42
3.52
6,51
ConYLSTM-thick
1.40
626.69
3x3
Spiral
3x3
47,68
1.11
2.16
DNC
0.68
8.00
77.63
14.50
24.44
DNC
0.75
8.00
91.09
87.67
89.35
15x15
169
MG Mem+CNN
0.12
7.99
99.79
99.88
99.83
0.79
76.97
3x3
Spiral 9x9
529
97.34
99.50
98.41
500
96.83
95.59
96.20
MG Mem+CNN
0.65
76.97
x5
3x3
Random 3x3
1500
96.13
91.08
93.54
0.66
78.12
9x9
Random 9x9
500
92.82
87.60
90.14
0.65
76.97
3x3
Policy
3x3
1000
95.65
90.22
92.86
Generalization of localiza- tion. Fixing parameters after training the query subnet on random motion (left), its localization loss remains low while training the exploration policy (middle). whose reward improves (right).
Figure 4:
500
ConvLSTM-deep: deep 23-layer architecture, in which each layer is a convolutional LSTM on a 48 grid. This contains the same total number of grids as our 7-layer multigrid network.
ConvLSTM -thckk: layers of convolutional LSTMs acting on 48 48 grids. We set channel counts to the sum of channels distributed across the corresponding layer of our multigrid pyramid.
We train each architecture using RMSProp. We search over learning rates in log scale from 10-2 2 to 10 and use 10 for multigrid and ConvI and 10-4 for DNC. Randomly generated maps are used for training and testing. Training runs for 8 x 10 steps with batch size 32. Test set size is 5000 maps. We used pixel-wise cross- cross-entropy loss over predicted and true locations (see Appendix B.2).
Table I reports performance in terms of localization accuracy on the test set. For the simplest setting in which the agent moves in spiral (i.e.. predictable) motion and the observation and query are 3x3, our multigrid architecture achieves nearly perfect precision (99.99%), recall (99.97%), and F-score (99.98 (99.8%). 98%). while all baselines struggle. DNC performs similarly to Ablated MG in terms of precision (< 77.6%), at the expense a significant loss in recall (14.50%). If we instead task the DNC with the simpler task of localization in a 15x 15 grid, we that the performance improves, yet the rates still around 10% lower than our architecture on the more challenging x5x environment. Efficiently addressing large memories is required here, and the DNC temporal memory linkage either incurs quadratic cost for the link matrix, placing a limited cap on the addressable memory (8k), or requires sparsifying approximation. Our architecture has no such overhead or approximations, nor does require maintaining an auxiliary state. To further evaluate the effect of the reduced memory size, and help putting Multigrid Memory and DNC on an even playground for comparison, we experimented with reduced version of Multigrid Memory with only ~ 8k memory units, and again found that Multigrid Memory has no issue mastering this task, with near perfect precision (99.79%), recall (99.88%), and F-score (99.83%)
Figure (right) visualizes the contents of the deepest and high-resolution LSTM block within the multigrid memory network of agent moving in spiral pattern. The memory clearly mirrors the contents of the true map, demonstrating that the network has learned a correct, and incidentally, an interpretable, procedure for addressing and writing memory. This further solidifies the addressing capability of the proposed multigrid memory architectures. See Figure 10 for a visualization of the DNC memory, which is not interpretable due its defined addressing mechanism.
7

---

Under review as a conference paper at ICLR 2020
Figure 5: MINST recall & classification. Given a random sequence of images followed by repeat (green), output the class of the next image (red).
5 3 0 6
2
Spe Meer
Nuartyy Saaal Clacabacees
Aocert dise Terel: Clenk
Learning curves. Multigrid memory architectures learn significantly faster than oth- Left: Maze localization task. Middle: Joint priority sort and classification. Right: Joint asso- ciative recall and classification.
Figure 6:
30
Table 2: Algorithmic tasks. Multigrid memory architectures achieve far lower error the classification variants of priority sort and associative recall, while performing comparably to the DNC on the simpler versions of these tasks. Multigrid memory architectures also remain effective when dealing with long sequences.
Params Memory (x10%)
Item
List Length
Architecture
(x10²)
Size
Data
Task
Error Rate + a
MG Enc+Dee
0.89
76,97
Random Patch
Priority Sort 0.004710.0015
3x3
20
DNC
0.76
8.00
0.003910.0016
MG Enc+Dec
1.16
441.94
Priority Sort 0.0604=0.0081
28x28
20
MNIST
DNC
1.05
8.00
+ Classify
0.1659#000188
Standard Sequence
MG Mem+CNNN
0.65
76.97
Random
Assoc. Recall 0.0046=#000011
3x3
10
DNC
0.76
8.00
Patch
0.00441100001
MG Mem+CCNN
0.84
220.97
Assoc. Recall 0,0312:10.005:
28
10
MNIST
DNC
0.90
8.00
+ Classify 0.2016::0.016
MG Enc+Dec
0.89
76.97
50
Random Patch
Priority Sort 0.0067+0.0001
Extended Sequence
3x3
MG Mem+CNN
0.65
76.97
Assoc. Recall 0.0056.10.0003
In more complex settings for motion type and query size (Table 1, bottom) our multigrid network remains accurate. It even generalizes to motions different from those on which it trained, including motion dictated by the learned policy that we describe shortly. Notably, even with the very long trajectory of 1,500 time steps, our proposed architecture has no issue retaining a large map memory.
4.2 JOINT EXPLORATION, MAPPING, AND LOCALIZATION
We next consider a setting in which the agent learns exploration policy via reinforcement, on top of fixed mapping and localization network that has been pre- pre-trained with random walk motion. We implement the policy network another multigrid reader, and intend to leverage the pre-trained mapping and localization capabilities learn more effective policy.
We formulate exploration reinforcement learning problem: the agent receives a reward of 1 when visiting a new space cell, -1 it hits a wall, and 0 otherwise. We use discount factor 7 = 0.99, and the A3C (Mnih et al., 2016) algorithm to train the policy subnet within our multigrid architecture.
Figure (left) depicts the localization loss while pre-training the mapping and localization subnets. Freezing these subnets, but continuing to monitor localization loss, see that localization remains reliable while a rewarding policy is Iearned (Figure 4, right). The results demonstrate that the learned multigrid memory and query subnets generalize trajectories that differ from those in their training dataset, as also conveyed in Table (last row). Meanwhile, the multigrid policy network is able to utilize memory from the mapping subnet in order to learn effective exploration policy. See the video demo linked in Appendix E for a visualization of learned exploratory behavior.
4.3 ALGORITHMICC TASKS
We test the task agnostti nature of our multigrid memory architecture by evaluating on a series of algorithmic tasks, closely inspired by those appearing in the original NTM work (Graves et al., 2014). For each of the following tasks, we consider two variants, increasing in level of difficulty. See Appendix B.3 and B.4 for complete details.
8

---

Under review as a conference paper at ICLR 2020
Priority Sort. In the first variant, the network receives sequence of twenty 3x3 patches, along with their priority. The task is to output the sequence of patches in the order of priority. Training and testing use randomly generated data. Training takes 2 x 10t steps. batch size 32, and testing 5000 sequences. We tune hyper-pan parameters as done for the mapping task. We structure our model as encoder-decoder architecture (Figure 2, right). As revealed in Table 2, our network performs comparably with DNC, with both architectures achieving near- cear-perfect performance.
The second variant extends the priority sort require recognition capability. The input is sequence of twenty 28 MNIST images (Lecun et al., 1998). The goal is to output the class of the input images in increasing order. Table 2 reveals that our architecture achieves much lower error rate compared to DNC on this task (priority sort + classificationn, while also learning faster (Figure 6).
Associative Recall. In the first task formulation, the network receives sequence of ten 3x3 random patches, followed by a second instance of one of the first nine patches. The task is to output the patch that immediately followed the query in the input sequence. We demonstrate this capability using the multigrid reader/writer architecture (Figure 2, left). Training details are similar to the sorting task. Table 2 shows that both DNC and our architecture achieve near-zero error rate.
In the second variant, the input is a sequence of ten 28 x 28 randomly chosen MNIST images (Lecun et al., 1998), where the network needs to output the class of the image immediately following the query (Figure 5). As shown in Table 2 and Figure 6, our multigrid memory network performs this task with significantly greater accuracy than the DNC, and also learns in fewer training steps.
To further test the ability of multigrid memory architectures in dealing with longer sequences, we experimented the sorting and associative recall tasks with sequence length of 50 and 20, respectively. As can be seen in Table 2, multigrid memory architectures remain effective with near-zero error rates.
The harder variants of both priority sort and associative recall require a combination of memory and pattern recognition capability. The success of multigrid memory networks (and notable poor performance of DNCs), demonstrates that they are a unique architectural innovation. They are capable of learning to simultaneously perform representational transformations and utilize a large distributed memory store. Furthermore, as Figure 6 shows, across all difficult tasks, including mapping and localization, multigrid memory networks train substantially faster and achieve substantially lower loss than all competing methods.
5 CONCLUSION
Our multigrid memory architecture represents a new paradigm in the history of designs linking deep neural networks long-term storage. We gain dramatic flexibility and new capabilities by co-locating memory with computation throughout the network structure. The technical insight driving this design an identification of attentional capacity with addressing mechanisms. and the recognition that both can be supported by endowing the network with the right structural connectivity and components: multigrid links across spatial hierarchy. Memory management thereby implicit, being distributed across the network and learned in an end-to-end manner. Multigrid architectures efficiently address large amounts of storage, and are more accurate than competing approaches across a diverse range of memory-intensive tasks.
REFERENCES
Dzmitry Bahdanau, Kyunghyun Cho. and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv: 2014.
Jasmine Collins, Jascha Sohl-Dickstein, and David Sussillo. Capacity and trainability in recurrent neural networks. ICLR, 2017.
Sreerupa Das, C. Lee Giles, and Guo- Zheng Sun. Learning context-free grammars: Capabilities and limitations of recurrent neural network with external stack memory. In CogSei, 1992.
Sreerupa Das, C. Lee Giles, and Guo-Zheng Sun. Using prior knowledge an NNPDA to learn context- languages, In NIPS, 1993.
Jeff Donahue, Lisa Anne Hendricks, Mareus Rohrbach, Subhashini Venugopalan, Sergio Guadarrama, Kate Saenko, and Trevor Darrell. Long-term recurrent convolutional networks for visual recognition and description. IEEE Transactions Pattern Analysis and Machine Intelligence, 2017.

---

Under review as a conference paper at ICLR 2020
Marco Fraccaro, Danilo Jimenez Rezende, Yori Zwols, Alexander Pritzel, S. M. Ali Eslami, and Fabio Viola. Generative temporal models with spatial memory for partially observed environments, ICML, 2018.
Mevlana Gemici, Chia-Chun Hung, Adam Santoro, Greg Wayne, Shakir Mohamed, Danilo Rezende, David Amos, and Timothy Lillicrap. Generative temporal models with memory. arXiv: 2017.
Alex Graves, Generating sequences with recurrent neural networks.
2013.
Alex Graves, Abdel rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recurrent neural networks. In IEEE International Conference Acoustics, Speech and Signal Processing, 2013.
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv:1410.5401, 2014.
Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grahska- Barwiñska, Sergio Gómez Colmenarejo. Edward Grefenstette, Tiago Ramalho, John Agapiou, Adrià Puigdomenech Badia, Karl Moritz Hermana, Yori Zwols, Georg Ostrovski, Adam Cain, Helen King. Christopher Summerfield, Phil Blunsom, Koray Kavukcuoglu, and Demis Hassabis. Hybrid computing using neural network with dynamic external memory. Nature, 2016.
Edward Grefenstette, Karl Moritz Hermann, Mustafa Suleyman, and Phil Blunsom. Learning to transduce with unbounded memory. In NIPS, 2015.
Matthew Hausknecht and Peter Stone. Deep recurrent Q-leaming for partially observable MDPs. In AAAI Fall Symposium, 2015.
Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
Michiel Hermans and Benjamin Schrauwen. Training and analysing deep recurrent neural networks. NIP,, 2013.
Sepp Hochreiter and Jurgen Schmidhuber. Long short-term memory. Neural Computation, 1997.
Steffen Hölldobler, Yvonne Kalinke, and Helko Lehmann. Designing counter: and activation landscapes in recurrent networks. Annual Conference on Artificial Intelligence, 1997.
Another case study of dynamics
Armand Joulin and Tomas Mikolov. Inferring algorithmic patterns with augmented recurrent nets. In NIPS, 2015.
Kalchbrenner, Ivo Danihelka, and Alex Graves. Grid long short-term memory. arXiv:1507.01526. 2015.
Tsung-Way Wei Ke, Michael Maire, and Stella X. Yu. Multigrid neural architectures. CVPR, 2017.
Karol Kurach, Marcin Andrychowicz, and Ilya Sutskever. Neural random-access machines. arXiv:1511.0639 2015.
Yann Lecun, Lcon Bottou, Yoshuo Bengio, and Patrick Haffner. Gradient- based learning applied to document recognition. Proceedings of the IEFE, 1998.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 2016.
Shuai Li, Wanqing Li, Chris Cook, Ce Zhu, and Yanbo Gao. Independently recurrent neural network (indmn); Building a longer and deeper mn. CVPR, 2018.
Volodymyr Mnih, Nicolas Heess, Alex Graves, et al. Recurrent models of visual attention. NIPS, 2014.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Marc Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza. Alex Graves. Timothy P. Lillicrap. Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. ICMI., 2016.
Michael C. Mozer and Sreerupa Das. A connectionist symbol manipulator that discovers the structure of context-free languages. In NIPS, 1993.
Junhyuk Oh, Valliappa Chockalingam, Satinder Singh, and Honglak Lee. Control of memory, active perception, and action Minecraft. arXiv: 1605.09/28 2016.
10

---

Under review as a conference paper at ICLR 2020
Emilio Parisotto and Ruslan Salakhutdinov. Neural map: Structured memory for deep reinforcement learning. ICLR, 2018.
Alexander Pritzel, Benigno Uria, Sriram Srinivasan, Adria Puigdomenech, Oriol Vinyals, Demis Hassabis, Daan Wierstra, and Charles Blundell. Neural episodic control. JCML, 2017.
Scott Reed and Nando de Freitas. Neural programmeer-inneeppileesss arXiv:151 2015.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. One-shot learning with memory-augmentte neural networks. crXiv:1605.06066 2016.
Jürgen Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation, 1992.
Jürgen Schmidhuber. A 'self-referentil weight matrix. In ICANN, 1993.
Hava T. Siegelmann and Eduardo D. Sontag. On the computational power of neural nets. Journal of computer and system sciences, 1995.
Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. End-to-en memory networks. In NIPS, 2015.
Ilya Sutskever, Oriol Vinyals, and Qune L.c. Sequence to sequence learning with neural networks. NIPS, 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NewrIPS, 2017.
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. In NIPS, 2015.
Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. ICLR, 2015.
SHI Xingjian, Zhourong Chen, Hao Wang. Dit- Yan Yeung, Wai-Kin Wong. and Wang-chun Woo. Convolutional LSTM network: machine learning approach for precipitation nowcasting. NIPS, 2015.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. Show, attend and Neural image caption generation with visual atten- tion. M.XX::15022.00444 2015.
Zheng Zeng, Rodney Goodman, Padhraic Smyth. Discrete recurrent neural networks for grammatical inference. IEEE Transactions on Neural Nenworks, 1994.

---

Under review as a conference paper at ICLR 2020
Convolution (3x3 fliter)
Layer
Layer 2
Layer
Loval
Operators
Bujduess-uncg
Convolution
Level 2
Up-sempling
Level 3
Figure Information routing. Shown are example paths of information flow in multigrid architecture. Progressing from one layer to the next, information flows between grids at the same level (via convolution, green), as well as to adjacent grids at higher resolution (via up-sampling and convolution, red) lower resolution (via down-sampling and convolution, not shown). Information from location (1,1) (blue) of the source grid at [layer 1, level I] can be propagated to all the locations marked as blue in subsequent layers and levels, following the indicated paths. With two more layers (not shown), it could reach any location in any level. Similarly rapid routing occurs along down-sampling links (in combination with up-sampling): information fine-scale levels can quickly flow to coarser levels and then to any location. a real multigrid network, the portion of this routing capacity actually used is determined by the learned network parameters (convolutional filters).
A INFORMATION ROUTING
Proposition 1: For the setup in Figure 7, suppose that the convolutional kernel size is 3 x 3. and up-sampling 2x nearest-neighbor sampling. Consider location (1, 1) of the source grid at [layer 1, level 1]. For target grid at [layer m, level nt,, where source location can be routed to any locations (i, where 1 < i, S (m n + 2) 2n- 1.
n, the information from the
Proof: Induction proof level n.
For level 1: Each convolution of size 3 can direct information from location (i,j) layer & to any of its immediate neighbors (i', where i y-1 S < 11j, - j < layer + 1. Therefore, convolutional operations direct information from location (1, in layer 1 to any locations (i',j') ) in layer k = m where < i', .' S m (m 1 2) 2 - 2-
in
Assume the proposition is true for level n (Vm 2 n),, we show that it is true for level n Consider any layer + 1 in level + 1, where m + n 1:
We have, m + 2 n + + m 2 n. Therefore, we have that at [layer m, level n]. the information from the source location can be routed to any locations (i,j) where 1 < i, < (m - 71 2) 2n-1 1. Now consider the path from [layer m, level n) to [layer m + 1, level n + 1]. This path involves the upsampling followed by convolution operator, as illustrated in Figure
Nearest-neighbor up-sampling directly transfers information from index i to 2 and 2 i 1, and j to 2 and 2-j - by definition. For simplicity, first consider index separately. By transferring to 2 information from location 1 < (m n + 2) 2n-1
in level will be 1) -2] at level n + 1. By transferring to 2. information from location 1 < (m 2) - in level will be transferred
transferred to all even indices in [2, ((m - n n 2) 2
12

---

Under review as a conference paper at ICLR 2020
Groundtruth Locations
Groundtruth Locations (9x9 Query)
Map Re-centered
(333 Query)
agent's
Explored Map
location
S Query
Query
12
Reader
Iosa
Iown
Figure 8: Multigrid memory writer-reader(s) architecture for spatial navigation. At each time step, the agent moves a new location and observes the surrounding 3 3 patch. The writer receives this 3 x 3 observation along with the agent's relative location (with respect to the starting point), updating the memory with this information. Two readers receive randomly chosen queries, view the current map memory built by the writer, and infer the possible locations of those queries.
and 9 x
to all odd indices in [1, ((m - ni + 2) 2000)22-1111 1) -1 at level n + Together, with 2 and 2 i transferring, the nearest- neighbo up-sampling transfers information from location
1 1 i S (m n + 2) 2- in level n to all indices [1, ((m - n +)) 2n - 1)- 2] at level n + Furthermore, the following convolution operator with 3 x kernel size can continue to transfer information from ((m
2) 2n 2n-1-
2+1 (m + - 1011--n111 11)+))2)-22 1)) 1)
[1, ((m n 2) 2)- 10--22m11111
1] level Taking together
n 11 We have ((m- - n + 2)- 22))2n11-111 -)
indices and j. information from location (i.j) where 1 < i,j (m (m-n+2)) -2n in level
n can be transferred to (i',j") in level n + 1, where 1 < f' -
+ 2) 2" 1.
B EXPERIMENT DETAILS
B.1 SPATIAL NAVIGATION: ARCHITACTURES
All experiments related to spatial navigation tasks use multigrid writer-reader(s) architectures. Fig- ure 8 visualizes this architecture and problem setup. At each time step during training, the agent takes a one-step action (e.g., along a spiral trajectory) and observes its observation, together with its location relative to the starting point, are fed into the writer, which must learn to update its memory. The agent has no knowledge of its absolute location in the world map. Two random 3 3 and 9 9 patches within the explored map presented to the agent to queries (some experiments use only x 3 queries). These queries feed into two readers, each viewing the same memory built by the writer; they must infer which previously seen locations match the query.
surroundings. This
13

---

Under review as a conference paper at ICLR 2020
8
Resice
1311
Encoder
115011
188
x2
x2
x1
target
E171
E/21
E/11
197
Decoder
1000
10000
109091
9000
Figure Multigrid memory encoder-decoder architecture for MNIST sorting. After processing the input sequence. the encoder (top) transfers memory into the decoder, which predicts the sequence of classes of the input digits sorted order.
Since the agent has no knowledge of its absolute location in the world map, the agent builds map relative to its initial position (map re-centered Figure 8) as it navigates.
During training, the writer learns organize and update memory from localization losses simultane- ously backpropagated from the two readers. During inference, only the writer updates the memory at each time step, and the readers simply view (i.e., without modification) the memory to infer the query locations. It also worth noting that only x 3 patches are fed into the writer at each time step; the agent never observes 9 patch. However, the agent successfully integrates information from the 3 x patches into a coherent map memory, in order to correctly answer queries much larger than its observations. Figure (right) shows that this learned memory strikingly resembles the actual world map.
B.2 SPATIAL NAVIGATION: LOSSES
Given predicted probabilities and the ground-truth location mask (Figure 8), we employ pixel-wise cross entropy loss as localization loss. Specifically, letting be the set of pixels, Pt the predicted probability pixel i, and ys be the binary ground-truth pixel i, the pixel-wise cross entropy loss is computed as follows:
Σ ye log(p:) + (1 y:) log(1 - p.)
YES
B.3 ALGORITHMIC TASKS: ARCHITECTURES
We employ writer-reader architectures for the associative recall tasks. The architectures are similar to those of the spatial navigation tasks depicted Figure 8, with some modifications appropriate the tasks:
Associative Recall Tasks
Standard variant. In the standard version of the task, we use the same writer architecture shown in Figure 8. For the reader, after progressing to the finest resolution (48 x 48) corresponding to the memory in the writer, the second half of MG-conv layers progressively scale down the output to x 3 to match the expected output size (instead of 48 48 as in the spatial navigation tasks).
14

---

Under review as a conference paper at ICLR 2020
5
10
15
20
25
30
35
40
Figure 10: Visualization of DNC mem- ory in mapping task. Due to its defined addressing mechanism, the DNC always al- locates new continuous memory slot at each ttme-step. does not appear to main- tain interpretable structure of the map.
MNIST recall + classification. For the MNIST variant, since the input is 28 28 images, we resize imput images to 5 scales from x to 48 x 48 and maintain the same S-scale structure for layers of the writer. The writer architecture is the same the encoder architecture in MNIST priority sort task, as depicted in Figure 9. The reader for the MNIST variant is similar to the reader in the standard variant, with the final layer followed fully connected layer to produce 10-way prediction vector over MNIST classes.
Priority Sort Tasks
Our priority experiments utilize encoder-decoder architectures.
Standard variant. The encoder has the same architecture as the writer used in the associative recall and spatial navigation tasks (Figure 8). For the decoder, the first half of the layers (MG-conv-LSTM) resemble the encoder, while the second half employ MG-conv layers to progressively scale down the output to 3 x
MNIST sort + classification. Figure depicts the encoder-decoder architecture for the MNIST variant.
B.4
ALGORRITMICC TASKS: LOSSES
Standard variants
We use pixel-wise cross-entropy loss for the standard variants, described in
Section B.2.
MNIST variants For MNIST variants, cross-entropy loss softmax prediction of the classes. Specifically, letting be the set of available classes, Pe the softmax output for class c, and one-hot vector of the ground-truth label, compute the loss as:
Σ CEC
log(pe)
(8)
C DNC DETAILS
We use the official DNC implementation (https: http::///gttht /hw.ttb..h.gggggggggggggg cwm/ deepmi nd/dnc), with controller heads (4 read and write), a memory vector length of 16 elements, and 500 memory slots (8K total), which is the largest memory size permitted by GPU resource limitations. Controllers are LSTMs, with hidden state sizes chosen to make total parameters comparable to other models in Table and Table 2. DNC imposes relatively small cap on the addressable memory due to the quadratic cost of the temporal linkage matrix blob/ htt///aater//eeee http//dn//////deeeeeeeeeeeeeeaae addres s ing..... py #L1 63).
com/ deepmineennmm nd/ dnc/
A visualization of DNC memory in the spatial mapping task (15 x 15 map) is provided in Figure 10.
D RUNTIME
On spatial mapping (with 15 15 world map), the runtimes for one-step inference with the Multigrid Memory architecture (0.12 parameters and 8 K memory) and DNC (0.75 M parameters and 8 K memory) are (mean + std): 0.018 + 0,003 sec and 0.017 + 0.001 see, respectively. These statistics computed over 10 runs NVIDIA Geforce GTX Titan X.
15

---

Under review as a conference paper at ICLR 2020
E DEMOS
Instructions for interpreting the video demos:
attps://drive.r http:://ddiiee...l.....oo
Mapping & localization in spiral trajectory, with 3 x queries:
https://drive.. wtt.::/.ww....oot.//////////taat
Mapping & localization in spiral trajectory, with 3 and x 9 queries:
https://driae http:///............o.oooa/aaaaa
Mapping & localization in wwnnomtttaeetttttooo
https://dria http://.www.aaeeeoooooocccosctpa
Joint exploration, mapping & localization:
16


Reviews:
[{"review_id": "rJlrkMmpYH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a multigrid memory architecture by introducing multigrid CNN [1] into convolutional LSTM network [2]. The method extends the convolutional LSTM with bigger memory capacity in forms of multigrid CNNs. Some specific designs such as multiple threads and encoder-decoder are also proposed. The model is validated with synthetic tasks such as spatial mapping, associative recall and priority sort. \n\nPros: \n* The method is well-motivated. Utilizing multigrid hierarchy enables convolutional LSTM to operate across scale space and thus may achieve richer representation for the hidden state memory.\n* The experiments are well designed (especially RL tasks), demonstrating the advantage of the proposed model.\n\nCons:\n* The proposed model is not presented clearly. The paper does not show details on how [1] and [2] are integrated, which requires the reader to refer back to the old works and make inference on the integration. Besides graphic illustration, the authors should include a brief review on [1, 2] and introduce some basic formulas describing the combination between the two.\n* Section 3.2 is supposed to contain the most important design considerations, but details seem missing. For example, what does the Writer do to the memory?\n* The contribution is rather incremental. Without detailed description, it seems that the proposed model is a straightforward replacement of the vanilla CNN with another CNN (multigrid CNN) in the convolutional LSTM architecture.\n* The experiments are not very satisfactory for the “generality” claim that the paper makes. \n\nQuestions and concerns:\n* Could you explain the term “addressable memory space”? It seems that your network’s memory comes from the internal states of LSTM. How is the memory addressable? \n* The analysis on information routing seems interesting. However, how does it relate to the memorization capacity? Is there any guarantee that the information from source grid is preserved in higher levels/layers? How does it differ from using vanilla multiple-layer neural networks?\n* As DNC is not originally designed for image inputs, how did you feed the images to DNC? Did you tune DNC carefully by adjusting the number of elements per memory slot? Also, DNC seems not a really strong baseline. Other solutions to increase memory capacity of MANNs exist [3, 4]\n* For algorithmic tasks, why don’t you include ConvLSTM as a baseline? Also, NTM maybe a better baseline than DNC for these tasks. \n* What is the model size and computational complexity compared to other MANNs?\n* The model is naturally fit for visual inputs. Is there any advantage when applying it to other sequential data (NLP, time-series)? \n\nReference\n[1] Ke, Tsung-Wei, Michael Maire, and Stella X. Yu. \"Multigrid neural architectures.\" In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6665-6673. 2017.\n[2] Xingjian, S. H. I., Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. \"Convolutional LSTM network: A machine learning approach for precipitation nowcasting.\" In Advances in neural information processing systems, pp. 802-810. 2015.\n[3] Rae, Jack, Jonathan J. Hunt, Ivo Danihelka, Timothy Harley, Andrew W. Senior, Gregory Wayne, Alex Graves, and Timothy Lillicrap. \"Scaling memory-augmented neural networks with sparse reads and writes.\" In Advances in Neural Information Processing Systems, pp. 3621-3629. 2016.\n[4] Hung Le, Truyen Tran, and Svetha Venkatesh. Learning to remember more with less memorization. In International Conference on Learning Representations, 2019. URL\nhttps://openreview.net/forum?id=r1xlvi0qYm.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJgP4SU0YB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Recurrent neural networks that can grow their memory capacity independent of the number of training parameters are an interesting topic. DNC, memory networks and NTM (all cited in this work) are some examples.\n\nThis work proposes an architecture inspired by an approach used in the computer vision literature, a multi-scale CNN. However, each cell of the CNN here is a convolutional LSTM.\n\nThis approach allows the memory capacity of the architecture to be increased (by increasing the number of cells) while maintaining a fixed number of parameters. The multi-scale nature of it allows memory operations across multiple scales the 2D grid in an efficient manner.\n\nThey test this architecture on a mapping and localization task (a natural fit for the multi-scale architecture) and find it outperforms other architectures including a single scale version of the same architecture.\n\nThey also compare against the DNC on tasks similar to that used in the original paper (priority sort) and associative recall and again find it learns in fewer iterations and achieves good performance.\n\nOverall, this architecture, while not groundbreaking, is novel in this context and the results show empirical gains. The paper is fairly well written.\n\nThis work could be improved by providing more detail (e.g. in the appendix) on the losses and approach used in the navigation task (the only explicit discussion of the loss used in the navigation task is in figure 3). It would also be helpful to provide more detail on the other tasks in the appendix.\n\nFinally, there is little analysis (either theoretical or empirical) on the runtime and memory requirements of this model. For example, figure 6 would seem to imply this model is running slower than the DNC (already quite a slow model) since it has completed less iterations? At a minimum, some empirical numbers of run time speed and memory usage compared with the DNC would be helpful.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1lPj6V3tr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes the multigrid memory networks which combine multigrid convolutional layers with LSTMs and evaluates its performance on a reinforcement learning-based navigation task and two algorithmic tasks of priority sorting and associative recall. \n\nThe authors claim that by integrating LSTM within the layers of the network, it affords larger memory size while remaining parameter efficient compared to other memory-augmented networks like the DNC, which abstract its memory module as a separate unit from its computational units. \n\nThe authors show with their experiments that multigrid memory networks outperform DNC and other models that lack its multigrid inference property.\n\n\nWhile the experiments show the proposed networks’ superior performance over baselines, my main concern about this paper is the lack of comparison with more recent memory-augmented models [1,2]. \n\nMoreover, in all the experiments, the DNC baseline has memory sizes that are much smaller than the multigrid memory networks. To my understanding, the memory module of DNC can be scaled up without increasing the number of computational parameters. Why is the DNC’s memory not scaled to match that of the multigrid memory networks? It would seem unfair to compare against a baseline with much smaller memory in memory-intensive tasks.\n\n\nOther comments:\n\n\n1) How are the input tensors upsampled in the multigrid memory layers?\n\n2) How is the visualization on the right of Figure 3 generated? It\nwould be more convincing to compare it with that of DNC.\n\n3) Clarity of some parts, especially Section 3.1 & 3.2, could be\nimproved with more formal mathematical statements about the multigrid\nmemory networks.\n\n4) How would multigrid memory networks generalize to other tasks that\ndo not involve input images?\n\n\n\n[1] Emilio Parisotto and Ruslan Salakhutdinov. Neural map: Structured\nmemory for deep reinforcement learning. ICLR, 2018.\n\n\n[2] Arbaaz Khan, Clark Zhang, Nikolay Atanasov, Konstantinos Karydis,\nVijay Kumar, Daniel D. Lee. Memory augmented control networks. ICLR,\n2018.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates convolutional LSTMs with a multi-grid structure. This idea in itself has very little innovation and the experimental results are not entirely convincing.
Decision: Reject