Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Recurrent neural networks that can grow their memory capacity independent of the number of training parameters are an interesting topic. DNC, memory networks and NTM (all cited in this work) are some examples.

This work proposes an architecture inspired by an approach used in the computer vision literature, a multi-scale CNN. However, each cell of the CNN here is a convolutional LSTM.

This approach allows the memory capacity of the architecture to be increased (by increasing the number of cells) while maintaining a fixed number of parameters. The multi-scale nature of it allows memory operations across multiple scales the 2D grid in an efficient manner.

They test this architecture on a mapping and localization task (a natural fit for the multi-scale architecture) and find it outperforms other architectures including a single scale version of the same architecture.

They also compare against the DNC on tasks similar to that used in the original paper (priority sort) and associative recall and again find it learns in fewer iterations and achieves good performance.

Overall, this architecture, while not groundbreaking, is novel in this context and the results show empirical gains. The paper is fairly well written.

This work could be improved by providing more detail (e.g. in the appendix) on the losses and approach used in the navigation task (the only explicit discussion of the loss used in the navigation task is in figure 3). It would also be helpful to provide more detail on the other tasks in the appendix.

Finally, there is little analysis (either theoretical or empirical) on the runtime and memory requirements of this model. For example, figure 6 would seem to imply this model is running slower than the DNC (already quite a slow model) since it has completed less iterations? At a minimum, some empirical numbers of run time speed and memory usage compared with the DNC would be helpful.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The paper proposes a multigrid memory architecture by introducing multigrid CNN [1] into convolutional LSTM network [2]. The method extends the convolutional LSTM with bigger memory capacity in forms of multigrid CNNs. Some specific designs such as multiple threads and encoder-decoder are also proposed. The model is validated with synthetic tasks such as spatial mapping, associative recall and priority sort. 

Pros: 
* The method is well-motivated. Utilizing multigrid hierarchy enables convolutional LSTM to operate across scale space and thus may achieve richer representation for the hidden state memory.
* The experiments are well designed (especially RL tasks), demonstrating the advantage of the proposed model.

Cons:
* The proposed model is not presented clearly. The paper does not show details on how [1] and [2] are integrated, which requires the reader to refer back to the old works and make inference on the integration. Besides graphic illustration, the authors should include a brief review on [1, 2] and introduce some basic formulas describing the combination between the two.
* Section 3.2 is supposed to contain the most important design considerations, but details seem missing. For example, what does the Writer do to the memory?
* The contribution is rather incremental. Without detailed description, it seems that the proposed model is a straightforward replacement of the vanilla CNN with another CNN (multigrid CNN) in the convolutional LSTM architecture.
* The experiments are not very satisfactory for the “generality” claim that the paper makes. 

Questions and concerns:
* Could you explain the term “addressable memory space”? It seems that your network’s memory comes from the internal states of LSTM. How is the memory addressable? 
* The analysis on information routing seems interesting. However, how does it relate to the memorization capacity? Is there any guarantee that the information from source grid is preserved in higher levels/layers? How does it differ from using vanilla multiple-layer neural networks?
* As DNC is not originally designed for image inputs, how did you feed the images to DNC? Did you tune DNC carefully by adjusting the number of elements per memory slot? Also, DNC seems not a really strong baseline. Other solutions to increase memory capacity of MANNs exist [3, 4]
* For algorithmic tasks, why don’t you include ConvLSTM as a baseline? Also, NTM maybe a better baseline than DNC for these tasks. 
* What is the model size and computational complexity compared to other MANNs?
* The model is naturally fit for visual inputs. Is there any advantage when applying it to other sequential data (NLP, time-series)? 

Reference
[1] Ke, Tsung-Wei, Michael Maire, and Stella X. Yu. "Multigrid neural architectures." In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6665-6673. 2017.
[2] Xingjian, S. H. I., Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. "Convolutional LSTM network: A machine learning approach for precipitation nowcasting." In Advances in neural information processing systems, pp. 802-810. 2015.
[3] Rae, Jack, Jonathan J. Hunt, Ivo Danihelka, Timothy Harley, Andrew W. Senior, Gregory Wayne, Alex Graves, and Timothy Lillicrap. "Scaling memory-augmented neural networks with sparse reads and writes." In Advances in Neural Information Processing Systems, pp. 3621-3629. 2016.
[4] Hung Le, Truyen Tran, and Svetha Venkatesh. Learning to remember more with less memorization. In International Conference on Learning Representations, 2019. URL
https://openreview.net/forum?id=r1xlvi0qYm.

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper proposes the multigrid memory networks which combine multigrid convolutional layers with LSTMs and evaluates its performance on a reinforcement learning-based navigation task and two algorithmic tasks of priority sorting and associative recall. 

The authors claim that by integrating LSTM within the layers of the network, it affords larger memory size while remaining parameter efficient compared to other memory-augmented networks like the DNC, which abstract its memory module as a separate unit from its computational units. 

The authors show with their experiments that multigrid memory networks outperform DNC and other models that lack its multigrid inference property.


While the experiments show the proposed networks’ superior performance over baselines, my main concern about this paper is the lack of comparison with more recent memory-augmented models [1,2]. 

Moreover, in all the experiments, the DNC baseline has memory sizes that are much smaller than the multigrid memory networks. To my understanding, the memory module of DNC can be scaled up without increasing the number of computational parameters. Why is the DNC’s memory not scaled to match that of the multigrid memory networks? It would seem unfair to compare against a baseline with much smaller memory in memory-intensive tasks.


Other comments:


1) How are the input tensors upsampled in the multigrid memory layers?

2) How is the visualization on the right of Figure 3 generated? It
would be more convincing to compare it with that of DNC.

3) Clarity of some parts, especially Section 3.1 & 3.2, could be
improved with more formal mathematical statements about the multigrid
memory networks.

4) How would multigrid memory networks generalize to other tasks that
do not involve input images?



[1] Emilio Parisotto and Ruslan Salakhutdinov. Neural map: Structured
memory for deep reinforcement learning. ICLR, 2018.


[2] Arbaaz Khan, Clark Zhang, Nikolay Atanasov, Konstantinos Karydis,
Vijay Kumar, Daniel D. Lee. Memory augmented control networks. ICLR,
2018.
review_title: Official Blind Review #3

Meta-review: This paper investigates convolutional LSTMs with a multi-grid structure. This idea in itself has very little innovation and the experimental results are not entirely convincing.
Decision: Reject