Continual Memory: Can We Reason After Long-Term Memorization?
Authors: Zhu Zhang
Chang Zhou
Zhou Zhao
Zhijie Lin
Jingren Zhou
Hongxia Yang
Venue:  (2021)
Keywords: Memory Augmented Neural Networks
Continual Memory
Reasoning After Long-Term Memorization

Abstract: Existing reasoning tasks often follow the setting of ‘’end-to-end reasoning'', which has an important assumption that the input contents can be always accessed while reasoning. However, human beings frequently adopt another reasoning setting in daily life, referred to ‘’reasoning after memorizing''. Concretely, human beings have the ability to unconsciously memorize their experiences within limited memory capacity, from which they can recall and respond to subsequent tasks. In this setting, the input contents are no longer available during reasoning, thus we need to compress and memorize the input stream in one pass, trying to answer general queries that are unseen before. Memory augmented neural networks introduce a write-read memory to perform such human-like memorization and reasoning, but they continually update the memory from current information and inevitably forget the early contents, failing to answer the queries relevant to early information.  In this paper, we propose the Continual Memory (CM) to explore this ability of reasoning after long-term memorization.  To alleviate the gradual forgetting of early information, we develop self-supervised memorization training with item-level and sequence-level objectives. We demonstrate several interesting characteristics of our continual memory via synthetic data, and evaluate its performance by several downstream tasks, including long-term text QA, long-term video QA and recommendation with long sequences.

Under review as a conference paper at ICLR 2021
CONTINUAL MEMORY: CAN WE REASON AFTER LONG- MEMORIZATION?
Anonymous authors Paper under double-blind review
ABSTRACT
Existing reasoning tasks often follow the setting of "end-to-end reasoning", which has an important assumption that the input contents can be always accessed while reasoning. However, human beings frequently adopt another reasoning setting in daily life, referred to "reasoning after memorizing". Concretely, human be- ings have the ability to unconsciously memorize their experiences within limited memory capacity, from which they can recall and respond to subsequent tasks, this setting, the input contents are no longer available during reasoning, thus we need to compress and memorize the input stream in pass, trying swer general queries that are unseen before. Memory augmented neural networks introduce write-read memory to perform such human-like memorization and reasoning, but they continually update the memory from current information and inevitably forget the early contents, failing to answer the queries relevant early information. In this paper, we propose the Continual Memory (CM) to explore this ability of reasoning after long-term memorization. To alleviate the gradual forgetting of early information, we develop self-supervised memorization training with item-levee and sequence-level objectives. We demonstrate several interesting characteristics of our continual memory via synthetic data, and evaluate its per- formance by several downstream tasks, including long-term text QA, long-term video QA and recommendation with long sequences.
INTRODUCTION
In recent years, the tremendous progress of neural networks has enabled machines to perform rea- soning given a query Q and the input contents X, e.g., infer the answer of given questions from the text/video stream in text/video question answering (Seo all [016 [e et al.] 2020b), or pre- dict whether user will click the given item based on the user behavior sequence in recommender systems (Ren et al. 2019 Pi al.) 2019). Studies that achieve top performances at such reason- ing tasks usually follow the setting of "end-to-en reasoning", where the input contents X is available at the time of answering Q. In this setting, complex interaction between X and Q can be designed to extract query-relevant information from X with little loss, such co-attention interac- tion (Xiong et al. 2016). Though these methods (Seo et al.) 2016 Le et al.) 2020h) can effectively handle these reasoning tasks, they require unlimited storage resources to hold the original input X. Further, they have to encode the whole input and develop the elaborate interaction from scratch, which are time consuming. This is not acceptable for online services that require instant response such as recommender systems, as the input sequence becomes extremely long (Ren et al. 2019).
Another setting of "reasoning after memorization", which has the restrictions that the raw input X is not available at the time of answering Q, requires the model to first digest X in a streaming manner, i.e., incrementally compress the current subsequence of X into memory M with very limited capacity (size much smaller than [X]). Under such constraints, in the inference phase, we can only capture query-relevant clues from the limited states M (rather than X) to infer the answer to Q, where the information compression procedure in M is totally not aware of Q, posing great challenges of what to remember in M. This setting is very similar to the daily situation of our human beings, i.e., we may not even know the tasks that we will answer in the future when we experiencing current events, and we also cannot go back to replay when we are solving problems at hand. However, it's our instincts, which continually process information during our entire life with

---

Under review as a conference paper at ICLR 2021
limited and compressed memory storages, that allow us to recall and draw upon past events to frame our behaviors given the present situations (Moscovitch et al. 2016 Baddeley 1992).
Compared to "end-to-end reasoning", "reasoning after memorization" though may not achieve bet- ter precisions at regular tasks with short sequences according to literatures (Park et al. 2020). is naturally better choice for applications like long-sequence recommendation (Ren et al., 2019) and long-text understanding (Ding et al.) 2020). Maintaining M can be incremental with only a small part of inputs at each timestep while inference over M and Q is also tractable for online service. Memory augmented neural networks (MANNs) (Graves et al., 2014; 2016) introduce a write-read memory that already follows the setting of "reasoning after memorization". which compress the input contents into fixed-size memory and only read relevant information from the memory dur- ing reasoning. However, existing works do not emphasize on using MANNs to perform long-term memory-based reasoning. They learn how to maintain the memory only by back pack-propagated losses to the final answer and do not design specific training target for long-term memorization, which in- evitably lead to the gradual forgetting of early contents (Le et al., 2019a). That is, when dealing with the long-term input stream, the memory may only focus on current contents and naturally neglect long-term clues. Thus, existing MANNS fail to answer the query relevant to early information due to the lack of long-teri term memorization training.
In this paper, we propose the Continual Memory (CM) to further explore this ability of reasoning af- ter long-term memorization. Specifically, we compress the long-term input stream into the continual memory with fixed-size capacity and infer subsequent queries based the memory. To overcome gradual forgetting of early information and increase the generalization ability of the memorization technique, we develop the extra self-supervised task recall the recorded history contents from the memory. This is inspired by the fact that human beings can recall details nearby some specific events and distinguish whether a series of events happened in the history, which respectively correspond to two different memory process revealed in cognitive, neuropsychological. and neuroimaging studies, ie., recollection and familiarity (Yonelinas) 2002 Moscovitch et al., 2016). Concretely, we de- sign the self-superviser memorization training with item-level and sequence-level objectives. The item-level objective aims to predict the masked items in history fragments, which are sampled from the original input stream and parts of items are masked as the prediction target. This task tries to endow the recollection ability that enables one to relive past episodes. And the sequence-level objective tries to distinguish whether a historical fragment ever appears in the input stream, where we directly sample positive fragments from the early input stream and replace parts of the items in positive ones as negative fragments. This task enables the familiarity process that can recognize experienced events or stimulus as familiar. We also give implementations on segment-level mainte- nance of memory to better capture context clues and improve the modeling efficiency. We illustrate the long-term memorization ability of our continual memory via a synthetic task, and evaluate its performance at solving real-world downstream tasks, including long-term text QA, long-term video QA and recommendation with long sequences, showing that it achieves significant advantages over existing MANNs in the "reasoning after memorizing" setting.
RELATED WORKS
Memory Augmented Neural Networks (MANNs) introduce external memory store and access the past information by differentiable write-read operators. Neural Turing Machine (NTM) (Graves) al. 2014) and Differentiable Neural Computer (DNC) (Graves et al. 2016) are the typical MANNS for human- like reasoning under the setting of "reasoning after memonzing", whose inference relies only on the memory with limited capacity rather than starting from the original input data. In this line of research, |Re et al. (2016) adopt the sparse memory accessing to reduce computational cost. [sordás & Schmidhuber 2019) introduce the key/value separation problem of content-based addressing and adopt a mask for memory operations a solution. Le et al. (2019)) manipulate both data and programs stored in memory to perform universal computations. And Santoro et al. (2018); Le et al. 2020a) consider the complex relational reasoning with the information they remember.
However, these works exploit MANNs mainly to help capture long-range dependencies in deal- ing with input sequences, but not paying efforts in dealing with the gradual forgetting issue in MANNs (Le et al., 2019a). They share the same training objective as those methods developed for the setting of "end-to-end reasoning", inevitably incurring gradual forgetting of early contents (Le
2

---

Under review as a conference paper at ICLR 2021
Self-Supervised Memorization Training
m
X
Continual Memory Machine
Task: Specifii Reasoning Training
mg
Information Stream X
Continual Memory M
Sequence Loss
Item Loss
f
m
m
[ck]
Transformer Encoder
Memory Update Module
BI-Directional
Memory
Transformer Decoder
X
m2
the t-th segment x
continual memory 3t at 1-th
history fragment
Continual Memory Machine
Self-Supervised Memorization Training
Figure 1:
The Framework of Continual Memory and Self-Supervised Memorization Training.
et al. 2019a). Recently, there are a few works trying to alleviate this problem in training MANNs. Le al." 2019a) propose measure "remember" ability by the final gradient the early input, and adopt a uniform writing operation on the memory. Rae et al. (2019) minimize the difference between successive memory states along with the reasoning objective, they assume the steadily changed memory states will benefit remembering older information. And Munkhdalai et al. (2019) design the meta-learned neural memory instead of the conventional urray-structured memory and memo- rize the current and past information by reconstructing the written values via the memory function. Note that our approach different and parallel to these techniques, since we give no assumptions on what behavior will remember the most. Instead, we optimize towards this goal directly by designing auxiliary tasks in a self-supervised manner. A recent work (Park et al.] (0200 also introduces a self- supervised memory loss to ensure how well the current input is written the memory, but it only focuses on remembering the current information and ignoring the long-term information forgetting.
Continual learning (Kirkpatrick et al. 2017 Lopez-Par Paz & Ranzato 2017 Chaudhry et al. 2018
de Masson d' l'Auttme et al. 2019) is another field about the forgetting problem of neural networks, which aims to learn from infinite stream of data and gradually extend acquired knowledge without catastrophic forgetting of early knowledge. But our continual memory focuses on remembering the infinite information stream and try overcome gradually forgetting of long-term information.
3 CONTINUAL MEMORY
3.1 PROBLEM FORMULATION
(x1, ".. and query Q, the methods under the setting of "end- to-end reasoning" directly learn the reason model T(X, Q) to predict the answer A. These is an important assumption that the input stream X can be always accessed while reasoning. And complex interaction between and Q can designed to extract query-relevant information in T(X, Q)- Obviously, these methods have to store the original input and infer the answer from scratch when the query known. But under the setting of "reasoning after memorizing". we compress the input stream X into fixed-size memory M {m with K memory slots and then infer the answer for any relevant query Q by A = RM, Q). Here we only need to store the compressed memory M. which can be updated in real-time and reused for series of queries. Since the slot number K in the memory is irrelevant to the input length |X), this setting only requires ((1) storage space rather than O(XX] in the setting of "end-to-end reasoning".
Given the input stream X

---

Under review as a conference paper at ICLR 2021
3.2 CONTINUAL MEMORY MACHINE
As shown in Figure given the input stream X = [x1, X2, "x,".... we apply a continual memory machine Ge(+) to compress them into continual memory M [mx) ) with K memory slots. By self-supervised memorization training, we try to overcome the gradual forgetting of early in- formation and make it possible to capture the clues any time in the stream. Concretely, based on continual memory, we develop the history recall model H-(-) to reconstruct the masked history fragments and distinguish positive history fragments from negative ones. Simultaneously. we train the task- ses--specffic reason model (2(()) based on continual memory. Under the setting of "reasoning after memorizing", we can develop the continual memory M Ge(X) and then infer the answer for any relevant query Q by A (M,M, Q).
We deal with the input stream X from the segment level rather than item level, i.c., cut the input sequence into fixed-length seg- ments and memorize them into the memory slots segment-by-segmenn ing MANNs (Graves al. 2014 2016). which store the input stream item-by-itent orderly with RNN-based controller, our segment- level memorization can further capture the bi- directional context of each item and improve the modeling efficiency. We denote the 1-th seg-
M
f2
Stat-to-ltem Attention
Mult--hee Gate Sate-Sased Update
Compared to exist-
Zhead
Memory Update Module
the 1-th segment features p!
mS]
cD3
continual memory M at the t-th step
Figure 2: The Process of Memory Updating.
ment as X (x4, )x] and the current memory as M = (m where we have recorded f-1 segments in M The x,, and m have the dimension d, and dm respectively.
We first model the t-th segment by a Transformer encoder (Vaswani et al.) 2017) and obtain the sequence features F = [f with dimension dz. After it, we apply a memory update module to write F' into M As shown in Figure 2 we apply slot-to-item attention to align the sequence features to slot features in the current memory M and then develop multi-head gate-based up- date. Concretely, we first calculate the slot-to-item attention matrix where each element means the relevance of slot-item pair, and then learn aligned features L'
(1
for each slot, given by
exp(ain) Σx exp(a
am W., (a.thhnnWim + W bb), af
1
=
Ce Ft
where WT E Rddmodt Xdm. W2 Rdmoda and ba E Rdmmoder are the projection matrices and bias. w is the row vector. After it, we project slot features and aligned features into Z subspaces, which is similar to the Muadi-Head setting in Transformer (Vaswani et al. 2017). given by
(2) where W. E 1ddmote/22xd and W! E / are the projection matrices. The m and Le are the slot and aligned sub-features in the z-th subspace. Next, the k-th slot sub-feature m is updated with the corresponding sub-feature 1/_ based on the z-th GRU unit with dmodet -d hidden states, given by
W. mi: li w
2
1+1 = GRU 1( m k=
where 1/_ is the current input of the z-th GRU unit. Next, the new slot feature m (+1 is aggregated from Z subspaces by mf+1 = W.Concat(m c+1 .... ml where W E Idd Xdmotet is the aggregation matrix. After the memorization of T segments, we can obtain continual memory M and we denote it by M for convenience. Note that, the inference stage, we can develop and update the continual memory in real time by Go(+), thus we do not need the input contents during subsequent reasoning and have the ability to reason after long-term memorization.
3.3 SELF-SUPERVISED MEMORIZATION TRAINING
After memorizing the input stream, conduct self-supervised memorization training alleviate the gradual forgetting of early information by the history recall model Hg(+) with item-level and sequence-level objectives, where the item-level objective aims reconstruct the masked positive
4

---

Under review as a conference paper at ICLR 2021
history fragments and the sequence-level objective tries distinguish positive history fragments from negative ones.
Concretely, sample the preceding segment from the imput stream the positive history frag- ment {ht. .... , with N items, where each item h corresponds to a feature x, in the
stream. We then mask 50% of items in the fragment and add an especial item [cls] the be- ginning to obtain the masked positive history fragment H =
(css) hi [mask]]"" , In order to guarantee that the model Hg(-) reconstructs the masked fragment by utilizing continual memory rather than only relying on fragment context, we set the mask ratio to 50% instead of 15% in BERT (Devlin al. 2019). Moreover,
construct the masked negative history fragment bN) by replacing 50% of unmasked items the positive frag- ments, where the replacement items are sampled from other input stream to make the negative frag- ments distinguishable. Here we construct masked positive fragments with corresponding nega- tive ones. Next, we adopt bidirectional Transformer decoder (Vaswani al., 2017) without the fu- ture masking to model each history fragment H /H" In the decoder, each history item can interact with all other items. The continual memory M is input the "encoder-decoder multi-head attention sub-layer" in each decoder layer, where the queries come from the previous decoder layer and the memory M are regarded the keys and values. This allows each item in the decoder to attend over all slot features the memory. Finally, we obtain the features {ridb] ri +/- +/- [[mask]]"" .NN /
H- = (Miclo) "4 I: [maakk]]
where each r +/- has the dimension dz.
Item-Level Reconstruction. We first predict the masked items of positive history fragments to build the item-level loss. Considering there are too many item types, we apply the contrastive training He et al. (2020); Chen et al. (2020) based on the ground truth and other sampled items. For the N/2 masked items, we compute the item-level loss by
N/2
exp(o(r) (Tmakk,) y:))
Litem
Σ log
(4)
exp(o(rfmass) [mask,] Y:)) + Σj= (xp(o(Tmaskki) Vi))
where y, E dd is the feature of ground truth of the i-th masked item, y, E Rd is the feature of sampled items and o(-) is the inner product.
Sequence-Leve Prediction. Next, we predict whether the masked history fragment ever appears in the current input stream, i.e. distinguish positive history segment from negative ones. Concretely, we project the feature r +/- into confident score + and develop the sequence-level loss by [cts]
Laeg
log(s
log(s))
Σ log(1 $7),
(5)
where B is the number of positive and negative history fragments.
3.4 TASK-SPECIFIC REASONING TRAINING
Besides self-supervised memorization training, we simultaneously develop task-specific reasoning training. For several downstream tasks, propose different task-specific reason model Ra (M, Q) for any query Q based on continual memory M. Here we adopt the simple and mature components in the reason model for fair comparison. The details are introduced in Appendix [A. Briefly, first learn the query representation by a task-specific encoder and then perform the multi-hop attention-based reasoning. Finally, we obtain the reason loss L, from (M, Q).
Eventually, we combine the memorization and reason losses to train our model, given by
Lum X₁ Litem + XgLsseq Ashy,
where \1. X2 and Ag are applied to adjust the balance of three losses.
EXPERIMENTS
We validate continual memory synthetic data and several downstream tasks, including long- term text question answering, long-term video question answering and recommendation with long sequences.
5

---

Under review as a conference paper at ICLR 2021
4.1 EXPERIMENT SETTING
Model Setting, We first introduce the common model settings for all downstream tasks. We set the layer number of the Transformer encoder and bi-directional Transformer decoder to 3. The head number in Multi-Head Attention is set to 4. And the subspace number Z in the memory update module is also set to 4. We set Å₁ 22 and A to 1.0, 0.5 and 1.0, respectively. The number of apply an Adam optimizer (Duchi al. 2011) to
history fragments is set to 5. During training, minimize the multi-task loss Lem, where the initial learning rate is set to 0.001.
Baseline. We compare our continual memory with the end-to-end reasoning methods and the memory-based reasoning approaches under the "reasoning after memorizing" setting. The end- to-end baselines are different in downstream tasks and the memory based baselines mainly DNC (Graves et al., 2016), NUTM (Le et al.) 2019b), STM (Le et al.) (020a) and DMSDNC (Park] al. 2020). For fair comparison, we modify the reason module of memory- baseed baselines to be consistent with our continual memory, i.e. we conduct multi-hop attention- based reasoning based on the built memory. And the number of memory slots in these baselines is also set to K. Besides, we set the core number of NUTM to 4, the query number of STM to 8 and the memory block number of DMSDNC to 2.
4.2 SYNTIETICC EXPERIMENT
Synthetic Dataset, We first introduce the setting of the synthetic task. Here we abstract the general concepts of reasoning tasks (AAVOA//coomeenaation)) to construct the synthetic task. We define the input sequence as a Stream and each item in the sequence as a Fact, where the stream and fact can correspond the text sequence and word token in text QA. We set the number of fact types to Ry. that is, each fact can be denoted by a R Ry-d one-hot vector and obtain the fact feature by a trainable embedding layer. Different facts can correspond to different words in text QA. Considering reasoning tasks often need to retrieve vital clues related to the query from the given input and then infer the answer, we define the query-relevant facts in the stream as the Evidence and regard the Evidence- Query-Answe
triple as the Logic Chain. As shown in Figure 3 given a stream and a query, we need to infer the answer if the stream contains the evidence. Specifically, we set the number of query types to R. and each query can be denoted by Ry one-hot vector. For each query, we set the number of answer types to Rg. That is, there are totally Rg * R. quere- pairs and we need to synthesize Rg * Ra corresponding evidences of each pair. Concretely, each evidence is denoted by a sequence of facts [fact].... fact J. which orderly appear in the input stream. And Rs is the length of the evidence. During the evidence synthesis, we first define 20 different group and uniformly split these facts and queries to 20 groups Next, if a query belongs to group k, we randomly sample R. facts from the group as the evidence, and then assign the evidence to query-answer pair to generate a fixed logic chain.
evidence
+
infer
stream
query
answer
Figure 3: A Data Sample with Consecutive Evidence.
Eventually, we synthetic 400 data samples for each logic chain to train the models. Each sample contains the imput stream with Ri items, a query and an answer. Concretely, we first sample R; facts as sequence and then place the evidence in the sequence, where we guarantee each stream-query pair corresponds to unique answer. Moreover, we synthesize two datasets with the consecutive and discrete evidence, respectively. The facts in the consecutive evidence continuously appear in the stream, but facts in the discrete evidence are distributed different parts of the stream, where we still make these facts exist in certain interval that does not exceed 20% length of the input stream.
Baselines and Model Details. The Directly Reason method first models the input stream by RNN to obtain the stream feature, then concatenates the stream feature with the query feature and predicts the answer by a linear layer. The Multi- Reason method further applies multiple attention layers after ENN-based stream modeling capture the query-relevant clues. In the main experiment,

---

Under review as a conference paper at ICLR 2021
Table 1: Performance Comparisons on Synthetic Data. R0=400, R,=200, R,,40, Ra=30 R5=5.
Discrete Evidence
Consecutive Evidence
Method
Setting
Early
Later
All
Early
Later
All
Directly Reason Multi-Hop Reason
End-to-End End-to-End
9.45
9.32
9.39
13.57
13.41
13.49
32.16
33.42
32.29
34.38
34.50
34.44
DNC
Memory-Based
13.37
22.54
17.95
20.56
26.59
23.58
NUTM
Memory-Based
17.84
23.59
20.72
24.31
29.71
27.01
STM DMSDNC
Memory-Basee -ased
17.90
23.47
20.68
23.55
29.64
26.59
Memory- Memory-Based
18.17
24.21
21.19
24.92
30.74
27.83
CM (Only Reason)
Memory-Based Self-Sup. Memory
Memory- Based
18.71
25.13
21.92
25.79
31.38
28.63
CM (Full)
22.14
24.98
23.56
28.42
31.71
30.07
CM (Only Litem)
Self-Sup. Memory
21.79
24.46
23.13
27.88
31.12
29.50
CM (Only Lseg)
Self-Sup. Memory
19.75
23,81
21.78
26.63
30.25
28.44
CM (Single Head)
Self-Sup. Memory
21.24
24.12
22.68
27.79
31.14
29.47
10
Multi-Hop Reason
26
DNC
CM (Only Reason)
CM (Full)
24
Early Later
Acc(%)
Acc (%)
22
2
20
5
10
15
20 25
0-255% 25-50% 50-75% 75-100% Evidence Location in the Stream
Slot Number K
Figure 4: Effect of the Memory Slot Number. Figure 5:
Performance on Synthetic Data with Longer Stream and Evidence.
set the dataset The facts of the evidence may appear in different stages of the input stream. Early means the facts appear in the preceding 50% of the stream and Later means the facts appear in the subsequent 50%. For our continual memory, we set the d, dm and d'nodel 128. The number K of memory slots and length of segments are set to 20 and 10, respectively. And we sample all other facts as negative items in Litem-
Ry, R₁, Rg. R. and Rg to 400, 200, 40, 30, and 5, respectively.
Evaluation Results. Table reports the performance comparison between our method and base- lines, where CM (Full) the full model with the memorization and reasoning training and CM (Only Reason) only employ the task-specific reasoning training. Overall, end-to-end methods have the close early and later performance, but memory-based approaches DNC, NUTM, STM, DMS- DNC and CM (Only Reason) achieve the terrible early performance due to the gradual forgetting. By the sulf-supervi memorization training, our CM (Full) significantly improves the early accu- racy and achieves the best memory-based reasoning performance. This fact suggests our proposed memorization training can alleviate the gradual forgetting of early information. Besides, CM (Only Reason) outperforms other memory-based methods, which indicates continual memory machine can better memorize the long-term information even without the memorization training. Moreover, find the Directly Reason approach achieves the worst performance but the Multi-Hop Rea- son method has high accuracy, which demonstrates the performance of end-to-end methods mainly depends the complicated interaction between the input contents and queries.
Ablation Study. We next perform ablation studies on the memorization loss and multi-heate update strategy. Concretely, first remove the sequence-level item-level loss to produce ablation

---

Under review as a conference paper at ICLR 2021
models CM (Only Litem) and CM (Only Lreq). As shown in Table I CM (Full) outperforms two ablation models on all metrics, which illustrates two supe--uue losses are both helpful for long- term memorization. Moreover, CM (Only Litem) achieves better accuracy than CM (Only Lseg), demonstrating the importance of the item-level loss. Next, we discard the multi-head setting from the memory update module to generate the ablation model CM(Single Head). From the results, can find the CM(Single Head) has the performance degradation, indicating the multi-head update can improve the memorization ability of continual memory.
Prraee-arraeters Analysis. We then explore the effect of the slot number K. We set K to [5, 10, 15, 20, 25] and display the results in Figure 4 We note that the model performance gradually improves with the increase of slot number and it reaches the bottleneck when the slot number is set to 25. Besides, the early performance has more significant improvement than the later performance.
Longer Stream and Evidence. To further validate the characteristics of our continual memory, we synthesize a more complicated dataset where Ry, Ri, Rg. R, and Re are set to 2000, 1000, 40, 30, and 10, respectively. That is, the dataset contains the longer input stream and more complex evidence. We set the number K memory slots and length N of segments to 20. As shown in Figure we display the reasoning performance of each model when the facts in the evidence appear in different locations of the imput stream. For example, 25~550% means the facts appear between 25% and 50% of the stream. We can observe that CM (Full) has an obvious performance improvement compared to memory-based methods in the 0~25% and 25~500% stage, but achieves slight improvement in the 75~100% stage. This verifies our self-supervised training is beneficial for long-term memorization.
Table Performance Comparisons Long-Term Text
Table 3:
Performance Comparisons for Long-Term Text Question Answering on TVQA.
Question Answering on NarrativeQA.
Method
Setting
Val
Test
Method
Setting
Acc.
AS Reader E2E-MN
End-to-Ean End
26.9
25.9
Murei-Stream
End-to-Eat
63.14 14
End-to-End
29.1
28.6
MSAN
End-to-End
65.77
DNC
Memory-Baat Based
25.8
25.2
DNC NUTM
Memory-Basee Based
55.92
NUTM
Memory-Basce
27.7
27.2
Memory-Bace -aaer
58.41
STM DMSDNC
Memory-Base -asse
27.2
26.7
CM (Only Reason)
Memory-Ban -ased
59.87
Memory-Baan -aase
28.1
27.5
CM (Full)
Self-Sup. Memory
61.12
CM (Only Reason) CM (Full)
Memory-Baseee
28.0
27.6
Self-Sup. Memory
28.8
28.1
4.3 LONG-TERM TEXT QUESTION ANSWERING
Dataset, Baseline and Model Details. We apply two multi-choice datasets Narrative (Kočiský et al. 2018) and TVQA (Lei et al., 2018) for long-term text question answering. The two datasets provide the answer candidates thus they are suitable for the setting "reasoning after memorizing". Note that the TVQA dataset also provides the video contents input but we only use the subtitles in the videos. For NarrativeQA, the AS Reader (Kadlec et al. 2016) applies a pointer network to generate the answer and E2E-MN (Sukhbaatar et al.J 2015) employs the end-to-end memory network to conduct multi-hop reasoning. For TVQA, the Multi-Stream (Lei et al. 2018) method develops the query-subtitle interaction for reasoning. And MSAN (Kim et al. 2020) first localizes the clues relevant to the question and then predicts the answer. For our continual memory, we set the dz, dm and dmmodei to 256. The number K of memory slots and length N of segments are both set to 20. And we sample all other words as negative items in Litem-
Evaluation Results. As shown in Table 2 and Table 3 the results are similar to synthetic ex- periments, i.e. our continual memory obtains the best performance among the memory-based approaches. And CM (Full) achieves the further improvement compared to CM (Only Reason), demonstrating the self-supervised memorization training can boost the reasoning ability of contin- ual memory. Moreover, our proposed method achieves the accuracy close to the excellent end-to-end method E2E-MN in the NarrativeQA dataset, but still has large performance gap with MSAN in the TVQA dataset. This is mainly because the MSAN method applies the two-stage reasoning with elaborate interactions between texts and queries.
8

---

Under review as a conference paper at ICLR 2021
4.4 LONG-TERM VIDEO QUESTION ANSWERING
The Activity QA dataset YYu al. 2019) contains 5,800 videos from the ActivityNeet (Caba Heilbron et al. 2015). The average video duration of this dataset about 180s and is the longest in VQA datasets. We compare our method with three basic end- to-end baselines E-VQA E-MN, E-SA from (Yu et al.) 2019) and the SOTA end-to-end model HCRN (Le al. 2020 For our continual memory, we set the d,. dm and dmmodet to 256. The number K of memory slots and length N of segments are both set to 20, And in Litem, we select 30 other frame features from the video as the sampled items.
Dataset, Baseline and Model Details:
Evaluation Results. As shown in Table 4 the CM (Only Reason) obtains the better performance than other memory-based models DNC and NUTM, and CM (Full) further achieves the 1.1% abso- lute improvement, showing the effectiveness of our model designs and self-supervised memoriza- tion training. Moreover, our method outperforms the basic end-to-end baselines and slightly worse than the SOTA method HCRN. This suggests continual memory can reduce the gap between memory-based and end-to-end reasoning paradigms.
Table 4: Performance Comparisons for Long- Term Video Question Answering on ActivityNet-QA QA.
Table Performance Comparisons for Lifelong Se- quence Recommendation on XLong,
Method
Setting
Acc.
Method
Setting
AUC
E-VQA
End-to-End
25.2
GRU4REC
End-to-End
0.8702
E-MN
End-to-End
27.9
Caser
End-to-End
0.8390
E-SA
End-to-End
31.8
RUM
End-to-Ean
0. 0.8649
HCRN
End-to-End
37.6
DIEN
End-to-Enn Lnn
0.8793
DNC
Memory- -ased
30.3
HPMN
Memory-Base Based
0.8645
NUTM
Memory- Besee
33.1
MIMN
Memory-Bane Based
0.8731
CM (Only Reason)
Memory- Based
34.6
CM (Only Reason)
Memory-Basee
0.8756
CM (Full)
Self-Sup. Memory
35.7
CM (Full)
Self-Sup. Memory
0.8824
4.5
LIFELONG SEQUENCE RECOMMENDATION
Dataset, Baseline and Model Details. The lifelong sequence recommendation aims to predict whether the user will click a given item based on long sequences, thus it can be regarded as a long- term reasoning task. The XLong dataset (Ren al. 2019) is sampled from the click logs on Alibaba. The length of historical behavior sequences in this dataset 1000. We compare our method with four end-to-end methods GRU4REC (Hidasi al. 20155). Caser (Tang & Wangl 2018), DIEN (Zhou
et al. 2019) RUM (Chen et al. 2018) and two memory-bases methods HPMN (Ren et al. 2019) and MIMN (Pi et al. (2019), where the HPMN method builds the memory by hierarchical RNNs and
the MIMN method introduces write-read memory as in (Graves et al. 2014). For our continual memory, we set the dæs dm and dmodee to 64. The number K of memory slots and length of segments both set to 20. And in Litem. we select 200 items from the large item set as the sampled items.
Evaluation Results. As shown in Table 5 our CM (Full) method not only outperforms other memory-based approaches, but also achieves better performance than state-of-the-ar end-to-end baselines. This is because continual memory can aggregate and organize the long- term interests from user behavior sequences and these interests can be activated during next-item prediction. But the end-to-end approaches may fail to learn such informative interest representations.
5 CONCLUSIONS
In this paper, we propose the continual memory to explore the ability of reasoning after long-term memorization. We compress the input stream into continual memory with the self-supervised mem- orization training and task-specific reasoning training. Based on continual memory, we can capture the clues for the subsequent queries and give the correct responses. Extensive experiments on series of downstream tasks verify the effectiveness of the continual memory. For future work, we will further explore the property of continual memory.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Alan Baddeley. Working memory. Science, 255(3044):556-559 1992.
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 961-970, 2015.
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny, Efficient lifelong learning with a-gem. arXiv preprint /8/200/0/ 2018.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint (xxi::0022.05099 2020.
Xu Chen, Hongteng Xu, Yongfeng Zhang, Jiaxi Tang, Yixin Cao, Zheng Qin, and Hongyuan Zha. Sequential recommendation with user memory networks. In Proceedings of the eleventh ACM international conference on web search and data mining, pp. 108-116, 2018.
Róbert Csordás and Juergen Schmidhuber. Improving differentiable neural computers through memory masking. de-allocation, and link distribution sharpness control. arXiv preprint arXiv:1904. 10278, 2019.
Cyprien Masson Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. Episodic memory in lifelong language learning. In Advances in Neural Information Processing Systems, pp. 13143-13152, 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference on The North American Chapter of the Association for Computational Linguisties, 2019.
Ming Ding, Chang Zhou, Hongxia Yang, and Jie Tang. Cogltx: Applying bert to long texts. Ad- vances Neural Information Processing Systems, 33, 2020.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 120ul):211122159, 2011.
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. 1rXiv:1410.540) 2014.
arXiv preprint
Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska- Barwiñska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. Hybrid computing using neural network with dynamic external memory. Nature, 538 (7626):471-44) 2016.
Kaiming He, Haoqi Fan, Yuxin Wa, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. Session-based rec- ommendations with recurrent neural networks. arXiv preprint arXiv:1511.06993 2015.
Rudolf Kadlec, Martin Schmid, Ondrej Bajgar, and Jan Kleindienst. Text understanding with the attention sum reader network. arXiv preprint 1603v:1603.011 01 547, 2016.
Junyeong Kim, Minuk Ma, Trung Pham, Kyungsu Kim, and Chang D Yoo. Modality shifting atten- tion network for multi- modal video question answering. Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pp. 10106-10115, 2020.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan. John Quan, Tiago Ramalho, Agnieszka Grabska Barwinska, al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):35113322 2017.
10

---

Under review as a conference paper at ICLR 2021
Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317-328, 2018.
Hung Le, Truyen Tran, and Svetha Venkatesh. Learning to remember more with less memorization.
arXiv preprini arXiv: 1901i 01111990111777 2019a.
Hung Le, Truyen Tran, and Svetha Venkatesh. Neural stored-program memory. arXiv preprint arXiv: 1eXiv:/906.00 08862, 2019b.
Hung Le, Truyen Tran, and Svetha Venkatesh. Self-attentive associative memory. arXiv preprint arXiv:2002.035/9 2020a.
Thao Minh Le, Vuong Le, Svetha Venkatesh, and Truyen Tran. Hierarchical conditional relation networks for video question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9972-9981, 2020b.
Jie Lei, Licheng Yu, Mohit Bansal, and Tamara Berg. Tvqa: Localized, compositional video question answering, arXiv preprint arXiv: 180900169, 2018.
David Lopez-Paz. and Marc Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, pp. 6467-6476, 2017.
Morris Moscovitch, Roberto Cabeza, Gordon Winoeur, and Lynn Nadel. Episodic memory and beyond: the hippocampus and neocortex in transformation. Annual review of psychology, 67: 105-134, 2016.
Tsendsuren Munkhdalai, Alessandro Sordoni, Tong Wang, and Adam Trischler. Metalearned neural memory. Advances in Neural Information Processing Systems, pp. 13331-13342, 2019.
Taewon Park. Inchul Choi, and Minho Lee. Distributed memory based self-supervised differentiable neural computer. arXiv preprint arXiv:2007.10637 2020.
Qi Pi, Weijie Bian, Guorui Zhou, Xiaoqiang Zhu, and Kun Gai. Practice long sequential user behavior modeling for click-through rate prediction. In Proceedings of the 25th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp. 2671-2679, 2019.
Jack Rae, Jonathan Hunt, Ivo Danihelka, Timothy Harley, Andrew W Senior, Gregory Wayne, Alex Graves, and Timothy Lillicrap. Scaling memmey-augmented neural networks with sparse reads and writes. In Advances in Neural Information Processing Systems, pp. 3621-3629, 2016.
Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling. arXiv preprint arXiv:1911.05500 2019.
Kan Ren, Jiarui Qin, Yuchen Fang, Weinan Zhang, Lei Zheng, Weijie Bian, Guorui Zhou, Jian Xu, Yong Yu, Xiaoqiang Zhu, et al. Lifelong sequential modeling with personalized memorization for user response prediction. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 565-574, 2019.
Adam Santoro, Ryan Faulkner, David Raposo, Jack Rae, Mike Chrzanowski, Theophane Weber, Daan Wierstra, Oriol Vinyals, Razvan Pascanu, and Timothy Lillicrap. Relational recurrent neural networks. In Advances neural information processing systems, pp. 7299-7310, 2018.
Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. Bidirectional attention flow for machine comprehension. arXiv preprint 1xX1:1611.0060) 2016.
Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. End-to-end memory networks. In Advances in Neural Information Processing Systems, pp. 2440-2448, 2015.
Jiaxi Tang and Ke Wang. Personalized top-n sequential recommendation via convolutional sequence embedding. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, pp. 565-573, 2018.

---

Under review as a conference paper at ICLR 2021
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances Neural Infor- mation Processing Systems, pp. 5998-6008, 2017.
Caiming Xiong, Victor Zhong, and Richard Socher. Dynamic coattention networks for question answering. arXiv preprint 1rXiv:161..0666 2016.
Andrew P Yonelinas. The nature of recollection and familiarity: A review of 30 years of research.
Journal of memory and language, 46(3):441 2002.
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet- qa: A dataset for understanding complex weh videos via question answering. In Proceedings of the American Association for Artificial Intelligence, volume 33, pp. 9127-9134, 2019.
Guorui Zhou, Na Mou, Ying Fan, Qi Pi, Weijie Bian, Chang Zhou, Xiaoqiang Zhu, and Kun Gai. Deep interest evolution network for click-through rate prediction. In Proceedings of the American Association for Artificial Intelligence, volume 33, pp. 5941-5948, 2019.
A APPENDIX
A.1 TASK- SPECIFCC REASON MODELS
In this section, we introduce the task-specific reason model Ra(M, Q). where M is the built mem- ory and Q is the given query. Specifically, we first model the query feature q E Rddmadel by a task-specific encoder. For synthetic experiments, the given query Q one-hot vector and we di- rectly obtain q by an embedding layer. For long-term text and video QA tasks, the query Q is sentence and we apply a bi-directional GRU to learn the sentence feature q. As for the recommen- dation task with long sequences, the given query is target item with the unique id and we likewise learn an embedding layer to obtain the feature q.
Next, we develop the multi-hop attention-based reasoning on continual memory M. Concretely, at each step e, we capture the importance memory feature e E Rdm from M based on the current query q- using an attention method, given by
exp(11)
The = w tanh(nnWWe (Wiq + WSm + b*), 18
e"
m
ΣØ= exp(15)
where Wi E (gdmodetxdmodd WS E RRdmodaiddm and be E Rdmadel are the projection matrices and bias. And w. the row vector. We then produce the next query qa Rdmmodes where W E [Rdmodee x(mm +mmodte)) the projection matrix and qd is the original q. After C steps, we obtain the reason feature q.. The hyper hyper-parameter C' is set to 2, 2, 2 and for synthetic experiments, text QA, video QA and sequence recommendation, respectively.
q" E
After it, design the final reasoning layer for different tasks. For synthetic experiments and long- term video QA with fixed answer sets, we directly apply a classification layer to select the answer and develop the cross-entropy loss L,- But text QA datasets NarrativeQA and TVQA provide different candidate answers for each query, we first model each candidate feature a; by another bi-directional GRU and then concatenate a; with q to predict the conference score for each candidate. Finally, we also learn the cross-entropy loss L, based on answer probabilities. As for the sequence recom- mendation task, can directly compute confidence score based on q c by linear layer and build the binary loss function L,.
A.2 PHEER-PRRMETER ANALYSIS OF SEGMENT LENGTH
We explore the effect of the segment length N in the main experiment of the synthesis task. We set N to [5, 10, 15, 20] and display the results in Figure 6 We can find that when the segment length is set to 5, the model achieves a terrible result and the performance is relatively stable when the length changes between 10 and 20. This is because when the segment is too short, important evidence may be scattered in different segments, and the model cannot effectively capture the evidence and infer the answer.
12

---

Under review as a conference paper at ICLR 2021
Early Later
Acc( (%)
22
20
Segment Length N
Figure 6: Effect of the Segment Length.
13


Reviews:
[{"review_id": "jrFqyQajzF", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "To approach \"reasoning after memorization\", the paper presents a Continual Memory (CM) framework using a memory-augmented neural network (MANN) and self-supervised training. In particular,  the CM  compresses the input sequence into a matrix memory using self-attention mechanisms and gated recurrent update (GRU). Then together with a Transformer decoder, the memory is used for downstream reasoning tasks without the need of referring to the original input sequence.  Moreover, the framework is simultaneously trained with auxiliary losses to enforce memorization capability. A variety of experiments demonstrate promising results, in which the CM outperforms two MANN baselines and shows competitive performance against state-of-the-art methods. \n\nPros:\n- \"Reasoning after memorization\" is an interesting problem and the proposed solution generally makes sense under this setting\n- The proposed solution combines several techniques, some of which seem novel and useful\n- The experiments are diverse with good results and SOTA for recommendation task\n\nCons:\n- The writing sometimes is misleading and vague\n- There is no major novel contribution\n- The synthetic task is poorly described \n- The experiments lack details of baselines and hyper-parameters\n\nDetailed comments and questions:\n\n- In the introduction, \"Castatrophic forgetting\" [1] is about continual learning over multiple tasks and thus, different from the problem the paper is addressing. Please explain the relation here. \n- In Sec. 3.2, the memory writing looks overcomplicated. Any explanation for the choice of designing it this way? \n- As in Eq. 3, the memory slots seem to be updated independently. That is, there is no memory-memory interaction in determining the content of the memory slot for the next time-step. Classical MANNs allow reading from memory during encoding and thus enable using other memory slots to write to a memory slot. The CM seems not to have this property, which may be a disadvantage. Please elaborate more on this point or correct me if I misunderstand.\n- In Sec. 3.3, to construct negative fragments, 50% of unmasked items were replaced with what? \n- Self-supervised training to improve memorization in MANN is not new. Please review other related works [2,3].\n- There is little description of how the decoder uses the memory for inference. Please consider using explicit equations to describe clearly the process.\n- There is no concrete example of the data used in the synthetic task. The task seems to be more like a memorization benchmark rather than \"reasoning after memorization\". The authors should consider known synthetic tasks that test both memorization and reasoning such as N-farthest [4], Relational Associative Recall [5] or bAbI [6]. Using known tasks makes comparison with other approaches easier. \n- The memory-based baselines are inadequate. Please consider stronger baselines that can reason and remember [4,5]. Also, the related work is incomprehensive without these methods. \n- Did the authors tune critical hyper-parameters of DNC (e.g., number of heads) or NUTM (e.g., number of cores)? Also, the authors should consider a comparison between baselines' number of hyper-parameters\n- The authors claim that segment-level memorization is better.  How to choose a good segment size N? If possible, please conduct an ablation study to verify the performance with different N. \n- The name of the baseline \"Two-Stream\" is misleading. It is unclear how the baseline works. \n- Some writing format problems: add space after CM in baseline name (e.g., page 6: \"CM(Only Reason)\" --> \"CM (Only Reason)\"); please use \\citet when appropriate (e.g., page 2:  \"(Le et al., 2019a) proposes\" --> \"Le et al. (2019a) proposes\" );  \n\nI may raise the score if the authors improve the writing clarity and add more content (baselines, synthetic tasks, hyper-parameters) to the experiments.\n\n\n[1] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521–3526, 2017\n\n[2]  Munkhdalai, Tsendsuren, Alessandro Sordoni, Tong Wang, and Adam Trischler. \"Metalearned neural memory.\" In Advances in Neural Information Processing Systems, pp. 13331-13342. 2019.\n\n[3] Park, Taewon, Inchul Choi, and Minho Lee. \"Distributed Memory based Self-Supervised Differentiable Neural Computer.\" arXiv preprint arXiv:2007.10637 (2020).\n\n[4] Santoro, Adam, Ryan Faulkner, David Raposo, Jack Rae, Mike Chrzanowski, Theophane Weber, Daan Wierstra, Oriol Vinyals, Razvan Pascanu, and Timothy Lillicrap. \"Relational recurrent neural networks.\" In Advances in neural information processing systems, pp. 7299-7310. 2018.\n\n[5] Hung Le, Truyen Tran, and Svetha Venkatesh. Self-attentive associative memory. In Proceedings of Machine Learning and Systems 2020, pages 2363–2372. 2020.\n\n[6] Weston, J., Bordes, A., Chopra, S., Rush, A. M., van Merrienboer, B., Joulin, A., and Mikolov, T. Towards ¨ ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698, 2015.", "extra_scores": {}, "extra_text": {}}, {"review_id": "d42ZV-Kck_", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes the continual memory machine, which learns to compress an input stream with a continually-learned memory module for reasoning after (long-term) memorization. Specifically, they first model an input stream as a sequence of fixed-length segments (of individual items).  \n\nThen, they use the Transformer architecture with multi-head attention to learn slot-to-item attention scores for updating the memory (consisting of K memory slots) with input features. The memory updates are controlled by a GRU network. For improving memorization over long-term memories, they also propose self-supervised training inspired by masked-language-modeling (of BERT).  Their target scenarios are long-term (text) question answering and lifelong sequence recommendation. The paper also conducts experiments on its syntectic dataset.  I believe a key contribution of this paper is that they connect super-supervised learning and memory-augmented neural networks. However, I am not exactly convinced that this helps the memorization of long-term memorization and \"reasoning after memorizing.\" The current experiments cannot justify this as well. \n\nThe major weakness of this paper is the experiment design. The main experiments are done on the synthetic dataset, but the construction of the synthetic dataset is not clear to me. For example, how do you create a series of the logic chain here? And can you connect your synthetic data with a real application by giving some illustrative examples? Can you at least show some examples of your synthetic dataset? In this current presentation, I cannot justify the use of synthetic data is reasonable or not for evaluating the methods.  For the realistic datasets, can you also show some case studies, such as the learned reasoning chains, and how the memory updates during memorization? \n\nI don't see a clear benefit of the so-called \"reasoning after memorizing\" versus \"reasoning while experiencing.\" The authors aslo didn't use any clear mathematical formulation to distinguish between these two settings. The mentioned related works can also save their memories after training and then use them for reasoning and inference only, what are the main disadvantage of doing that? Can you show the differences in math and in experiments?\n\n\nSome minor points:\n- Is R_f the number of all facts or the number of all fact types? Or do you use fact and fact type interchangeably here?\n- Why are there R_q * R_a different chains for each query-answer pair? I thought R_q is the number of all queries, no? \n- Why do you claim using segments can lead to bi-directional context (Sec 3.2). \n- Please use \"their\" instead of his or her for making ICLR a more inclusive community. \n\n \n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ILghL-IzS3y", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "--------------------------------------------------------------------------------------------------------------------------------\nSummary:\n\nIn this paper, the authors propose the Continual Memory (CM) targeted towards a reasoning scenario called “reasoning after memorization”. The main goal of CM is to enable long-term memorization as opposed to memory networks that suffer from gradual forgetting. They evaluate their model both on synthetic data as well as a few downstream benchmarks. \n\n--------------------------------------------------------------------------------------------------------------------------------\nOverall assessment:\n\nI really struggled with this one and I think there are some interesting ideas in there. However, it was very hard for me to understand the main motivation and story behind the proposed model and its design choices. Moreover, the task itself is not clearly defined until the experiments section making it really hard to understand the claims and motivations of the work. I will provide detailed feedback below. \n\n--------------------------------------------------------------------------------------------------------------------------------\nFeedback:\n\n(1) One thing that can improve the paper substantially is re-structuring the introduction to clearly state the motivation, studied task, proposed solution and the main contributions of the work. \n\n(1-1) For example, the authors briefly mention QA/VQA/Recommendation in the beginning of the introduction and then do not formally present/discuss their studied task is in the introduction. The QA/VQA/Recommendation are large research areas with many different benchmarks and approaches. Some references to reasoning has also been mentioned in the introduction, but what area in reasoning is this paper specifically studying? It would be very helpful for the reader to understand early on what the target of the paper is.\n\n(1-2) Some concepts used in the introduction are not well defined. For example, the authors refer multiple times to “Reasoning while experiencing” and “reasoning after memorizing” without formally defining them. I was not familiar with these notions and wasn’t able to find any pointers through online search. However, if these are known concepts in a sub-area, it would be very helpful if the authors can add a citation to where they were originally defined. If not, it would be helpful to formally define them. Another vague concept is \"raw content\". It is not clear what it is referring to. Is it the input? perhaps some source of knowledge? If the task is defined the authors can use examples to make these concepts more clear.\n\n(1-3) The introduction makes some connections to human cognition all throughout that read a bit subjective and are stated without any citations. For example paragraph 2 in the introduction. \n\nIt is really hard for me to understand the main contributions of the paper and to make a fair assessment until the paper text has been revised. If the authors are willing to submit a modified version during the author response period, I will re-read and re-evaluate my score. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an approach to allow a neural network to memorize and reason over a long time horizon. Experiments on synthetic datasets, question answering, and sequence recommendation are presented to evaluate the proposed method. 

The paper addresses an important problem of processing long sequences. However,  all reviewers agree that the writing of the paper can be improved (i.e., motivation, details of experiment design/setup, and others below). Importantly, I think the authors need to elaborate the differences of continual memory with existing episodic memory methods. The authors added a paragraph about continual learning during the rebuttal period, and mentioned that their continual memory focuses on remembering infinite information stream without forgetting. Episodic memory models can be applied/adapted for this purpose, so the authors should at least compare with one of them (ideally more).
Decision: Reject