Continual Memory: Can We Reason After Long-Term Memorization?

Abstract: Existing reasoning tasks often follow the setting of ‘’end-to-end reasoning'', which has an important assumption that the input contents can be always accessed while reasoning. However, human beings frequently adopt another reasoning setting in daily life, referred to ‘’reasoning after memorizing''. Concretely, human beings have the ability to unconsciously memorize their experiences within limited memory capacity, from which they can recall and respond to subsequent tasks. In this setting, the input contents are no longer available during reasoning, thus we need to compress and memorize the input stream in one pass, trying to answer general queries that are unseen before. Memory augmented neural networks introduce a write-read memory to perform such human-like memorization and reasoning, but they continually update the memory from current information and inevitably forget the early contents, failing to answer the queries relevant to early information.  In this paper, we propose the Continual Memory (CM) to explore this ability of reasoning after long-term memorization.  To alleviate the gradual forgetting of early information, we develop self-supervised memorization training with item-level and sequence-level objectives. We demonstrate several interesting characteristics of our continual memory via synthetic data, and evaluate its performance by several downstream tasks, including long-term text QA, long-term video QA and recommendation with long sequences.

Reviews and author responses:

Thanks for your comments. (1) Actually, our synthetic task requires the long-term memorization to support elaborate recall for any given query. This task may depend on the pattern matching between the compressed memory and arbitrary query. (2) To further validate the "reasoning after memorization" ability of our method, we conduct abundant experiments on long-term text QA, long-term video QA and recommendation with long sequences. These tasks contain more complex reasoning logic and can support our claim. (3) Different from existing synthetic tasks (e.g. copy, associative recall), our synthetic task, whose question-answer-evidence triples are drawn from synthetic patterns, has a consistent form with the subsequent QA and recommendation tasks.

I appreciate the authors' responses to my concerns. Adding more baselines makes the experiment much stronger now.  Hence, I have raised my score to 6. 
Reasons for not giving a clear accept: 
- The idea is not novel enough
- I am not fully convinced that the synthetic task is a good testbed for "reasoning after memorization".  Given the current description, it looks like an associative recall task wherein memorization is the main element. 

To approach "reasoning after memorization", the paper presents a Continual Memory (CM) framework using a memory-augmented neural network (MANN) and self-supervised training. In particular,  the CM  compresses the input sequence into a matrix memory using self-attention mechanisms and gated recurrent update (GRU). Then together with a Transformer decoder, the memory is used for downstream reasoning tasks without the need of referring to the original input sequence.  Moreover, the framework is simultaneously trained with auxiliary losses to enforce memorization capability. A variety of experiments demonstrate promising results, in which the CM outperforms two MANN baselines and shows competitive performance against state-of-the-art methods. 

Pros:
- "Reasoning after memorization" is an interesting problem and the proposed solution generally makes sense under this setting
- The proposed solution combines several techniques, some of which seem novel and useful
- The experiments are diverse with good results and SOTA for recommendation task

Cons:
- The writing sometimes is misleading and vague
- There is no major novel contribution
- The synthetic task is poorly described 
- The experiments lack details of baselines and hyper-parameters

Detailed comments and questions:

- In the introduction, "Castatrophic forgetting" [1] is about continual learning over multiple tasks and thus, different from the problem the paper is addressing. Please explain the relation here. 
- In Sec. 3.2, the memory writing looks overcomplicated. Any explanation for the choice of designing it this way? 
- As in Eq. 3, the memory slots seem to be updated independently. That is, there is no memory-memory interaction in determining the content of the memory slot for the next time-step. Classical MANNs allow reading from memory during encoding and thus enable using other memory slots to write to a memory slot. The CM seems not to have this property, which may be a disadvantage. Please elaborate more on this point or correct me if I misunderstand.
- In Sec. 3.3, to construct negative fragments, 50% of unmasked items were replaced with what? 
- Self-supervised training to improve memorization in MANN is not new. Please review other related works [2,3].
- There is little description of how the decoder uses the memory for inference. Please consider using explicit equations to describe clearly the process.
- There is no concrete example of the data used in the synthetic task. The task seems to be more like a memorization benchmark rather than "reasoning after memorization". The authors should consider known synthetic tasks that test both memorization and reasoning such as N-farthest [4], Relational Associative Recall [5] or bAbI [6]. Using known tasks makes comparison with other approaches easier. 
- The memory-based baselines are inadequate. Please consider stronger baselines that can reason and remember [4,5]. Also, the related work is incomprehensive without these methods. 
- Did the authors tune critical hyper-parameters of DNC (e.g., number of heads) or NUTM (e.g., number of cores)? Also, the authors should consider a comparison between baselines' number of hyper-parameters
- The authors claim that segment-level memorization is better.  How to choose a good segment size N? If possible, please conduct an ablation study to verify the performance with different N. 
- The name of the baseline "Two-Stream" is misleading. It is unclear how the baseline works. 
- Some writing format problems: add space after CM in baseline name (e.g., page 6: "CM(Only Reason)" --> "CM (Only Reason)"); please use \citet when appropriate (e.g., page 2:  "(Le et al., 2019a) proposes" --> "Le et al. (2019a) proposes" );  

I may raise the score if the authors improve the writing clarity and add more content (baselines, synthetic tasks, hyper-parameters) to the experiments.


[1] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521–3526, 2017

[2]  Munkhdalai, Tsendsuren, Alessandro Sordoni, Tong Wang, and Adam Trischler. "Metalearned neural memory." In Advances in Neural Information Processing Systems, pp. 13331-13342. 2019.

[3] Park, Taewon, Inchul Choi, and Minho Lee. "Distributed Memory based Self-Supervised Differentiable Neural Computer." arXiv preprint arXiv:2007.10637 (2020).

[4] Santoro, Adam, Ryan Faulkner, David Raposo, Jack Rae, Mike Chrzanowski, Theophane Weber, Daan Wierstra, Oriol Vinyals, Razvan Pascanu, and Timothy Lillicrap. "Relational recurrent neural networks." In Advances in neural information processing systems, pp. 7299-7310. 2018.

[5] Hung Le, Truyen Tran, and Svetha Venkatesh. Self-attentive associative memory. In Proceedings of Machine Learning and Systems 2020, pages 2363–2372. 2020.

[6] Weston, J., Bordes, A., Chopra, S., Rush, A. M., van Merrienboer, B., Joulin, A., and Mikolov, T. Towards ¨ ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698, 2015.

**Q7.** There is no concrete example of the data used in the synthetic task. The task seems to be more like a memorization benchmark rather than "reasoning after memorization". The authors should consider known synthetic tasks that test both memorization and reasoning such as N-farthest [4], Relational Associative Recall [5] or bAbI [6]. Using known tasks makes comparison with other approaches easier.

**Response:** We rewrite the details of dataset construction in Section 4.2 to make it easy to understand and display the concrete example of the data sample in Figure 3. Since our work focuses primarily on the reasoning after **long-term memorization** rather than on any specific task of reasoning, we need to validate our method and baselines on the long-term reasoning data. So known synthetic tasks such as N-farthest [4], Relational Associative Recall [5] or bAbI [6] are not appropriate for our work.


**Q8.** The memory-based baselines are inadequate. Please consider stronger baselines that can reason and remember [4,5]. Also, the related work is incomprehensive without these methods.

**Response:** Thanks for your suggestion and we have added the stronger baseline [3,5] in our synthetic experiments and long-term text question answering experiments. We can find our CM still achieves the best performance under the setting of "reasoning after memorization". We also added these related works [2,3,4,5] in Section 2. We will conduct more experiments on these stronger baselines later.


**Q9.** Did the authors tune critical hyper-parameters of DNC (e.g., number of heads) or NUTM (e.g., number of cores)? Also, the authors should consider a comparison between baselines' number of hyper-parameters

**Response:** We have tuned critical hyper-parameters of these baselines and displayed the hyper-parameter setting in Section 4.1.


**Q10.** The authors claim that segment-level memorization is better. How to choose a good segment size N? If possible, please conduct an ablation study to verify the performance with different N.

**Response:** Thanks for your suggestion and we have conducted an ablation study to verify the performance with different N in Appendix 2. 


**Q11.** The name of the baseline "Two-Stream" is misleading. It is unclear how the baseline works. Some writing format problems: add space after CM in baseline name (e.g., page 6: "CM(Only Reason)" --> "CM (Only Reason)"); please use \citet when appropriate (e.g., page 2: "(Le et al., 2019a) proposes" --> "Le et al. (2019a) proposes" );

**Response:** Thanks for your suggestion. We have renamed this method to "Directly Reason" and added more details of it in Section 4.2. And we have modified these writing format problems carefully.

We’d like to thank the reviewers for valuable comments. We have modified our paper according to the comments and marked the modified content in blue font.

In summary, we improve the writing clarity in Section 1 to make the motivation and contribution of this paper clearer. We review more related works in Section 2 according to the comments. And we rewrite the details of dataset construction in Section 4.2 to make it easy to understand. Finally, we add stronger memory-based baselines in the experiments. Note that we rename "reasoning while experiencing" to “end-to-end reasoning”in the new submission.


**Q1.** In the introduction, "Castatrophic forgetting" [1] is about continual learning over multiple tasks and thus, different from the problem the paper is addressing. Please explain the relation here.

**Response:** Thanks for your comment. We incorrectly cited the paper [1] related to continual learning and have modified it. Moreover, we clarify the difference between continual learning and our task in Section 2. Continual learning aims to learn from an infinite stream of data and gradually extend acquired knowledge without catastrophic forgetting of early knowledge. But our task focuses on remembering the infinite information stream and try to overcome gradually forgetting of long-term information.
 

**Q2.** In Sec. 3.2, the memory writing looks overcomplicated. Any explanation for the choice of designing it this way?

**Response:** In the model design, we mainly refer to existing excellent components in Transformer. The Multi-Head setting aims to improve the memorization ability and we verify its effectiveness by ablation study in Section 4.2. And the GRU-based slot update is to make the memory able to memorize the long-term information by gate mechanism.


**Q3.** As in Eq. 3, the memory slots seem to be updated independently. That is, there is no memory-memory interaction in determining the content of the memory slot for the next time-step. Classical MANNs allow reading from memory during encoding and thus enable using other memory slots to write to a memory slot. The CM seems not to have this property, which may be a disadvantage. Please elaborate more on this point or correct me if I misunderstand.

**Response:** As you think, there is no memory-memory interaction in our continual memory and each memory slot is updated independently. Because we hope each memory slot can capture different information. When different queries are given, we expect to activate the query-relevant slot and infer the answer. That is,  memory-memory interaction is not necessary in our design. And the experiments validate the our continual memory with independent slots can achieve excellent performance.


**Q4.** In Sec. 3.3, to construct negative fragments, 50% of unmasked items were replaced with what?

**Response:** Thanks for your comment and We have added the detail in Section 3.3. The replacement items are sampled from other input stream to make the negative fragments distinguishable. 


**Q5.** Self-supervised training to improve memorization in MANN is not new. Please review other related works [2,3].

**Response:** Thanks for your suggestion and we have added the related works [2,3] in Section 2. The method [2] designs the meta-learned neural memory instead of the conventional array-structured memory. To write the current values into the neural memory, they develop a meta loss to pull close between the prediction values of the memory function and the target values. That is, their mete loss is indispensable for neural memory writing and is not for long-term memorization training. But our self-supervised loss is designed for long-term memorization. So the meta loss is different from our self-supervised loss in both design philosophy and methods. And the method [3] introduces a self-supervised memory loss to ensure how well the current input is written to the memory, but it only focuses on remembering the current information and ignoring the long-term information forgetting, which is our main focus. We also compare [3] in our experiments, showing that our method is more powerful with long sequence tasks.


**Q6.** There is little description of how the decoder uses the memory for inference. Please consider using explicit equations to describe clearly the process.

**Response:** Thanks for your comment and we have added more description of the decoder in Section 3.3. Concretely, the decoder is a standard Transformer decoder without the future masking. The memory M is input to the “encoder-decoder multi-head attention sub-layer” in each decoder layer, where the queries come from the previous decoder layer and the memory M are regarded as the keys and values. This allows each item in the decoder to attend over all slot features in the memory. 


We’d like to thank the reviewers for valuable comments. We have modified our paper according to the comments and marked the modified content in blue font.


**Q1.** the construction of the synthetic dataset is not clear.

**Response:** Thanks for your comment and we have rewritten the details of dataset construction in Section 4.2 to make it easy to understand. We describe the general concepts of synthetic dataset, connect the synthetic data with real reasoning tasks and describe the process of sample synthesis. Moreover, we give an example to illustrate the synthetic sample.


**Q2.** I don't see a clear benefit of the so-called "reasoning after memorizing" versus "reasoning while experiencing." The authors also didn't use any clear mathematical formulation to distinguish between these two settings. 

**Response:** We have rewritten the introduction to clarify the differences two settings and illustrate the benefit of "reasoning after memorizing" in Section 1. Moreover, we modify the problem formulation in Section 3.1 to distinguish two settings. Note that we rename "reasoning while experiencing" to “end-to-end reasoning” to make it clearer. We briefly make a summary here:
* Differences: In the setting of "end-to-end reasoning", the original input X can be always accessed when answering query Q, so that complex interaction between X and Q can be designed to extract query-relevant information, even when X is extremely long. But under the setting of “reasoning after memorizing”, it has the restrictions that the raw input X is not available at the time of answering Q, requires the model to first digest X in a streaming manner, i.e., incrementally compress the current subsequence of X into a memory M with very limited capacity. In the inference phase, the model can only capture query-relevant clues from the limited states M to infer the answer when the query is known. 
* Benefit: The methods in the setting of "end-to-end reasoning" require unlimited storage resources to store the original input and have to consume much time to infer the answer from scratch when the query is known. But in some practical applications, e.g., long-term sequence recommendation, we only have limited storage resources and require a short reasoning delay. And under the setting of “reasoning after memorizing”, we only need to store the compressed states M of input contents and can answer any possible query based on M with a short reasoning response time. 




**Q3.** The mentioned related works can also save their memories after training and then use them for reasoning and inference only, what are the main disadvantage of doing that? Can you show the differences in math and in experiments?

**Response:** We have pointed out the main disadvantage of MANNs in Section 1 and verify the disadvantage in experiments. Concretely, existing MANNs learn how to maintain the memory only by back-propagated losses to the final answer and do not design specific training target for long-term memorization, which inevitably leads to the gradual forgetting of early contents. That is,  the memory may only focus on short-term contents and naturally neglect long-term clues. Thus, existing MANNs fail to answer the query relevant to early information. And we alleviate the gradual forgetting by extra self-supervised tasks, which recall the recorded history contents from continual memory. In experiments, we can find the MANNs (DNC, NUTM, STM and DMSDNC) achieve the terrible early performance due to the gradual forgetting. But by the self-supervised memorization training, our CM (Full) significantly improves the early accuracy.


**Q4.** Some minor points:

**Q4.1.** Is R_f the number of all facts or the number of all fact types? Or do you use fact and fact type interchangeably here. Why are there R_q * R_a different chains for each query-answer pair? I thought R_q is the number of all queries, no?

**Response:** We have rewritten the details of dataset construction in Section 4.2 to make it clearer. R_f is the number of all fact types. we denote a fact by a R_f-d one-hot vector and can obtain the fact feature by a trainable embedding layer. R_q is the number of all queries. And for each query, we set the number of answer types to R_a, so we have R_q * R_a different query-answer pairs. For each query-answer pair, we synthesize the fixed evidence to generate a logic chain, that is, we regard the evidence-query-answer triple as the logic chain. The more details can be found in Section 4.2.

**Q4.2.** Why do you claim using segments can lead to bi-directional context (Sec 3.2).

**Response:** In segment-level modeling, each item in the segment can interact with all other items, i.e., can develop the relation with the future items. So we call it bi-directional context modeling.

**Q4.3.** Please use "their" instead of his or her for making ICLR a more inclusive community.

**Response:** Thanks for your suggestion and we have revised the contents.




We’d like to thank the reviewers for valuable comments. We have modified our paper according to the comments and marked the modified content in blue font. 

In summary, we completely re-structure the Introduction to make the studied task, motivation and contribution of this paper clearer. We review more related works in Section 2. And we rewrite the details of dataset construction in Section 4.2 to make it easy to understand. Finally, we add stronger memory-based baselines in the experiments. Note that we rename "reasoning while experiencing" to “end-to-end reasoning” in the new submission.


**Q1.** For example, the authors briefly mention QA/VQA/Recommendation in the beginning of the introduction and then do not formally present/discuss their studied task is in the introduction. The QA/VQA/Recommendation are large research areas with many different benchmarks and approaches. Some references to reasoning has also been mentioned in the introduction, but what area in reasoning is this paper specifically studying? It would be very helpful for the reader to understand early on what the target of the paper is.

**Response:** Thanks for your suggestion and we have rewritten the Introduction to formally present our studied tasks at the beginning. Concretely, we aims to tackle the standard reasoning tasks (including QA/VQA/Recommendation) under the unconventional setting of "reasoning after memorization". That is, our work can be applied to any reasoning task under the setting of "reasoning after memorization" rather than limited to one specific task. 


**Q2.**  Some concepts used in the introduction are not well defined. For example, the authors refer multiple times to “Reasoning while experiencing” and “reasoning after memorizing” without formally defining them. I was not familiar with these notions and wasn’t able to find any pointers through online search. However, if these are known concepts in a sub-area, it would be very helpful if the authors can add a citation to where they were originally defined. If not, it would be helpful to formally define them. Another vague concept is "raw content". It is not clear what it is referring to. Is it the input? perhaps some source of knowledge? If the task is defined the authors can use examples to make these concepts more clear.

**Response:** We have rewritten the Introduction and give clearer definitions of “reasoning after memorizing”and “reasoning while experiencing” (renamed to "end-to-end reasoning").  And we carefully compare two reasoning settings and clarify the motivation of our work. Moreover, we give the more concrete description in Section 3.1 about two settings. Briefly, the reasoning tasks aim to infer the answer for any query Q according to the input contents X. Under the setting of “end-to-end reasoning”, the raw input contents X is available at the time of answering Q. In this setting, complex interaction between X and Q can be designed to extract query-relevant information from X with little loss. But this setting requires unlimited storage resources to hold the original input X. And it also needs to encode the whole input and develop the elaborate interaction from scratch, which are time consuming. Another setting of “reasoning after memorization”, which has the restrictions that the raw input X is not available at the time of answering Q, requires the model to first digest X in a streaming manner, i.e., incrementally compress the current subsequence of X into a memory M with very limited capacity (size much smaller than |X|). Under such constraints, in the inference phase, we can only capture query-relevant clues from the limited states M (rather than X) to infer the answer to Q, where the information compression procedure in M is totally not aware of Q, posing great challenges of what to remember in M.
As for the "raw content", it refers to the unprocessed input sequence, e.g., the article in text question answering and user behavior sequence in recommender systems. To avoid misunderstanding, we rename it to "raw input".


**Q3.** The introduction makes some connections to human cognition all throughout that read a bit subjective and are stated without any citations. For example paragraph 2 in the introduction.

**Response:** Thanks for your suggestion and we have added the citations while illustrating the connections between our work and human cognition.


--------------------------------------------------------------------------------------------------------------------------------
Summary:

In this paper, the authors propose the Continual Memory (CM) targeted towards a reasoning scenario called “reasoning after memorization”. The main goal of CM is to enable long-term memorization as opposed to memory networks that suffer from gradual forgetting. They evaluate their model both on synthetic data as well as a few downstream benchmarks. 

--------------------------------------------------------------------------------------------------------------------------------
Overall assessment:

I really struggled with this one and I think there are some interesting ideas in there. However, it was very hard for me to understand the main motivation and story behind the proposed model and its design choices. Moreover, the task itself is not clearly defined until the experiments section making it really hard to understand the claims and motivations of the work. I will provide detailed feedback below. 

--------------------------------------------------------------------------------------------------------------------------------
Feedback:

(1) One thing that can improve the paper substantially is re-structuring the introduction to clearly state the motivation, studied task, proposed solution and the main contributions of the work. 

(1-1) For example, the authors briefly mention QA/VQA/Recommendation in the beginning of the introduction and then do not formally present/discuss their studied task is in the introduction. The QA/VQA/Recommendation are large research areas with many different benchmarks and approaches. Some references to reasoning has also been mentioned in the introduction, but what area in reasoning is this paper specifically studying? It would be very helpful for the reader to understand early on what the target of the paper is.

(1-2) Some concepts used in the introduction are not well defined. For example, the authors refer multiple times to “Reasoning while experiencing” and “reasoning after memorizing” without formally defining them. I was not familiar with these notions and wasn’t able to find any pointers through online search. However, if these are known concepts in a sub-area, it would be very helpful if the authors can add a citation to where they were originally defined. If not, it would be helpful to formally define them. Another vague concept is "raw content". It is not clear what it is referring to. Is it the input? perhaps some source of knowledge? If the task is defined the authors can use examples to make these concepts more clear.

(1-3) The introduction makes some connections to human cognition all throughout that read a bit subjective and are stated without any citations. For example paragraph 2 in the introduction. 

It is really hard for me to understand the main contributions of the paper and to make a fair assessment until the paper text has been revised. If the authors are willing to submit a modified version during the author response period, I will re-read and re-evaluate my score. 

This paper proposes the continual memory machine, which learns to compress an input stream with a continually-learned memory module for reasoning after (long-term) memorization. Specifically, they first model an input stream as a sequence of fixed-length segments (of individual items).  

Then, they use the Transformer architecture with multi-head attention to learn slot-to-item attention scores for updating the memory (consisting of K memory slots) with input features. The memory updates are controlled by a GRU network. For improving memorization over long-term memories, they also propose self-supervised training inspired by masked-language-modeling (of BERT).  Their target scenarios are long-term (text) question answering and lifelong sequence recommendation. The paper also conducts experiments on its syntectic dataset.  I believe a key contribution of this paper is that they connect super-supervised learning and memory-augmented neural networks. However, I am not exactly convinced that this helps the memorization of long-term memorization and "reasoning after memorizing." The current experiments cannot justify this as well. 

The major weakness of this paper is the experiment design. The main experiments are done on the synthetic dataset, but the construction of the synthetic dataset is not clear to me. For example, how do you create a series of the logic chain here? And can you connect your synthetic data with a real application by giving some illustrative examples? Can you at least show some examples of your synthetic dataset? In this current presentation, I cannot justify the use of synthetic data is reasonable or not for evaluating the methods.  For the realistic datasets, can you also show some case studies, such as the learned reasoning chains, and how the memory updates during memorization? 

I don't see a clear benefit of the so-called "reasoning after memorizing" versus "reasoning while experiencing." The authors aslo didn't use any clear mathematical formulation to distinguish between these two settings. The mentioned related works can also save their memories after training and then use them for reasoning and inference only, what are the main disadvantage of doing that? Can you show the differences in math and in experiments?


Some minor points:
- Is R_f the number of all facts or the number of all fact types? Or do you use fact and fact type interchangeably here?
- Why are there R_q * R_a different chains for each query-answer pair? I thought R_q is the number of all queries, no? 
- Why do you claim using segments can lead to bi-directional context (Sec 3.2). 
- Please use "their" instead of his or her for making ICLR a more inclusive community. 

 





Meta-review: This paper proposes an approach to allow a neural network to memorize and reason over a long time horizon. Experiments on synthetic datasets, question answering, and sequence recommendation are presented to evaluate the proposed method. 

The paper addresses an important problem of processing long sequences. However,  all reviewers agree that the writing of the paper can be improved (i.e., motivation, details of experiment design/setup, and others below). Importantly, I think the authors need to elaborate the differences of continual memory with existing episodic memory methods. The authors added a paragraph about continual learning during the rebuttal period, and mentioned that their continual memory focuses on remembering infinite information stream without forgetting. Episodic memory models can be applied/adapted for this purpose, so the authors should at least compare with one of them (ideally more).

Decision: withdrawn-rejected-submissions