Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: --------------------------------------------------------------------------------------------------------------------------------
Summary:

In this paper, the authors propose the Continual Memory (CM) targeted towards a reasoning scenario called “reasoning after memorization”. The main goal of CM is to enable long-term memorization as opposed to memory networks that suffer from gradual forgetting. They evaluate their model both on synthetic data as well as a few downstream benchmarks. 

--------------------------------------------------------------------------------------------------------------------------------
Overall assessment:

I really struggled with this one and I think there are some interesting ideas in there. However, it was very hard for me to understand the main motivation and story behind the proposed model and its design choices. Moreover, the task itself is not clearly defined until the experiments section making it really hard to understand the claims and motivations of the work. I will provide detailed feedback below. 

--------------------------------------------------------------------------------------------------------------------------------
Feedback:

(1) One thing that can improve the paper substantially is re-structuring the introduction to clearly state the motivation, studied task, proposed solution and the main contributions of the work. 

(1-1) For example, the authors briefly mention QA/VQA/Recommendation in the beginning of the introduction and then do not formally present/discuss their studied task is in the introduction. The QA/VQA/Recommendation are large research areas with many different benchmarks and approaches. Some references to reasoning has also been mentioned in the introduction, but what area in reasoning is this paper specifically studying? It would be very helpful for the reader to understand early on what the target of the paper is.

(1-2) Some concepts used in the introduction are not well defined. For example, the authors refer multiple times to “Reasoning while experiencing” and “reasoning after memorizing” without formally defining them. I was not familiar with these notions and wasn’t able to find any pointers through online search. However, if these are known concepts in a sub-area, it would be very helpful if the authors can add a citation to where they were originally defined. If not, it would be helpful to formally define them. Another vague concept is "raw content". It is not clear what it is referring to. Is it the input? perhaps some source of knowledge? If the task is defined the authors can use examples to make these concepts more clear.

(1-3) The introduction makes some connections to human cognition all throughout that read a bit subjective and are stated without any citations. For example paragraph 2 in the introduction. 

It is really hard for me to understand the main contributions of the paper and to make a fair assessment until the paper text has been revised. If the authors are willing to submit a modified version during the author response period, I will re-read and re-evaluate my score. 
review_title: Review of "Continual Memory: Can We Reason After Long-Term Memorization?"

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes the continual memory machine, which learns to compress an input stream with a continually-learned memory module for reasoning after (long-term) memorization. Specifically, they first model an input stream as a sequence of fixed-length segments (of individual items).  

Then, they use the Transformer architecture with multi-head attention to learn slot-to-item attention scores for updating the memory (consisting of K memory slots) with input features. The memory updates are controlled by a GRU network. For improving memorization over long-term memories, they also propose self-supervised training inspired by masked-language-modeling (of BERT).  Their target scenarios are long-term (text) question answering and lifelong sequence recommendation. The paper also conducts experiments on its syntectic dataset.  I believe a key contribution of this paper is that they connect super-supervised learning and memory-augmented neural networks. However, I am not exactly convinced that this helps the memorization of long-term memorization and "reasoning after memorizing." The current experiments cannot justify this as well. 

The major weakness of this paper is the experiment design. The main experiments are done on the synthetic dataset, but the construction of the synthetic dataset is not clear to me. For example, how do you create a series of the logic chain here? And can you connect your synthetic data with a real application by giving some illustrative examples? Can you at least show some examples of your synthetic dataset? In this current presentation, I cannot justify the use of synthetic data is reasonable or not for evaluating the methods.  For the realistic datasets, can you also show some case studies, such as the learned reasoning chains, and how the memory updates during memorization? 

I don't see a clear benefit of the so-called "reasoning after memorizing" versus "reasoning while experiencing." The authors aslo didn't use any clear mathematical formulation to distinguish between these two settings. The mentioned related works can also save their memories after training and then use them for reasoning and inference only, what are the main disadvantage of doing that? Can you show the differences in math and in experiments?


Some minor points:
- Is R_f the number of all facts or the number of all fact types? Or do you use fact and fact type interchangeably here?
- Why are there R_q * R_a different chains for each query-answer pair? I thought R_q is the number of all queries, no? 
- Why do you claim using segments can lead to bi-directional context (Sec 3.2). 
- Please use "their" instead of his or her for making ICLR a more inclusive community. 

 




review_title: Need more illustrative examples for the motivation; Experiments are less convincing.

final_score: rating: 6: Marginally above acceptance threshold
review_content: To approach "reasoning after memorization", the paper presents a Continual Memory (CM) framework using a memory-augmented neural network (MANN) and self-supervised training. In particular,  the CM  compresses the input sequence into a matrix memory using self-attention mechanisms and gated recurrent update (GRU). Then together with a Transformer decoder, the memory is used for downstream reasoning tasks without the need of referring to the original input sequence.  Moreover, the framework is simultaneously trained with auxiliary losses to enforce memorization capability. A variety of experiments demonstrate promising results, in which the CM outperforms two MANN baselines and shows competitive performance against state-of-the-art methods. 

Pros:
- "Reasoning after memorization" is an interesting problem and the proposed solution generally makes sense under this setting
- The proposed solution combines several techniques, some of which seem novel and useful
- The experiments are diverse with good results and SOTA for recommendation task

Cons:
- The writing sometimes is misleading and vague
- There is no major novel contribution
- The synthetic task is poorly described 
- The experiments lack details of baselines and hyper-parameters

Detailed comments and questions:

- In the introduction, "Castatrophic forgetting" [1] is about continual learning over multiple tasks and thus, different from the problem the paper is addressing. Please explain the relation here. 
- In Sec. 3.2, the memory writing looks overcomplicated. Any explanation for the choice of designing it this way? 
- As in Eq. 3, the memory slots seem to be updated independently. That is, there is no memory-memory interaction in determining the content of the memory slot for the next time-step. Classical MANNs allow reading from memory during encoding and thus enable using other memory slots to write to a memory slot. The CM seems not to have this property, which may be a disadvantage. Please elaborate more on this point or correct me if I misunderstand.
- In Sec. 3.3, to construct negative fragments, 50% of unmasked items were replaced with what? 
- Self-supervised training to improve memorization in MANN is not new. Please review other related works [2,3].
- There is little description of how the decoder uses the memory for inference. Please consider using explicit equations to describe clearly the process.
- There is no concrete example of the data used in the synthetic task. The task seems to be more like a memorization benchmark rather than "reasoning after memorization". The authors should consider known synthetic tasks that test both memorization and reasoning such as N-farthest [4], Relational Associative Recall [5] or bAbI [6]. Using known tasks makes comparison with other approaches easier. 
- The memory-based baselines are inadequate. Please consider stronger baselines that can reason and remember [4,5]. Also, the related work is incomprehensive without these methods. 
- Did the authors tune critical hyper-parameters of DNC (e.g., number of heads) or NUTM (e.g., number of cores)? Also, the authors should consider a comparison between baselines' number of hyper-parameters
- The authors claim that segment-level memorization is better.  How to choose a good segment size N? If possible, please conduct an ablation study to verify the performance with different N. 
- The name of the baseline "Two-Stream" is misleading. It is unclear how the baseline works. 
- Some writing format problems: add space after CM in baseline name (e.g., page 6: "CM(Only Reason)" --> "CM (Only Reason)"); please use \citet when appropriate (e.g., page 2:  "(Le et al., 2019a) proposes" --> "Le et al. (2019a) proposes" );  

I may raise the score if the authors improve the writing clarity and add more content (baselines, synthetic tasks, hyper-parameters) to the experiments.


[1] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521–3526, 2017

[2]  Munkhdalai, Tsendsuren, Alessandro Sordoni, Tong Wang, and Adam Trischler. "Metalearned neural memory." In Advances in Neural Information Processing Systems, pp. 13331-13342. 2019.

[3] Park, Taewon, Inchul Choi, and Minho Lee. "Distributed Memory based Self-Supervised Differentiable Neural Computer." arXiv preprint arXiv:2007.10637 (2020).

[4] Santoro, Adam, Ryan Faulkner, David Raposo, Jack Rae, Mike Chrzanowski, Theophane Weber, Daan Wierstra, Oriol Vinyals, Razvan Pascanu, and Timothy Lillicrap. "Relational recurrent neural networks." In Advances in neural information processing systems, pp. 7299-7310. 2018.

[5] Hung Le, Truyen Tran, and Svetha Venkatesh. Self-attentive associative memory. In Proceedings of Machine Learning and Systems 2020, pages 2363–2372. 2020.

[6] Weston, J., Bordes, A., Chopra, S., Rush, A. M., van Merrienboer, B., Joulin, A., and Mikolov, T. Towards ¨ ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698, 2015.
review_title: Improve memorization and reasoning in MANN using a bag of techniques, yet the method lacks details and the baselines are inadequate.

Meta-review: This paper proposes an approach to allow a neural network to memorize and reason over a long time horizon. Experiments on synthetic datasets, question answering, and sequence recommendation are presented to evaluate the proposed method. 

The paper addresses an important problem of processing long sequences. However,  all reviewers agree that the writing of the paper can be improved (i.e., motivation, details of experiment design/setup, and others below). Importantly, I think the authors need to elaborate the differences of continual memory with existing episodic memory methods. The authors added a paragraph about continual learning during the rebuttal period, and mentioned that their continual memory focuses on remembering infinite information stream without forgetting. Episodic memory models can be applied/adapted for this purpose, so the authors should at least compare with one of them (ideally more).
Decision: Reject