Continual Memory: Can We Reason After Long-Term Memorization?

1 INTRODUCTION . In recent years , the tremendous progress of neural networks has enabled machines to perform reasoning given a query Q and the input contents X , e.g. , infer the answer of given questions from the text/video stream in text/video question answering ( Seo et al. , 2016 ; Le et al. , 2020b ) , or predict whether a user will click the given item based on the user behavior sequence in recommender systems ( Ren et al. , 2019 ; Pi et al. , 2019 ) . Studies that achieve top performances at such reasoning tasks usually follow the setting of “ end-to-end reasoning ” , where the raw input contents X is available at the time of answering Q . In this setting , complex interaction between X and Q can be designed to extract query-relevant information from X with little loss , such as co-attention interaction ( Xiong et al. , 2016 ) . Though these methods ( Seo et al. , 2016 ; Le et al. , 2020b ) can effectively handle these reasoning tasks , they require unlimited storage resources to hold the original input X . Further , they have to encode the whole input and develop the elaborate interaction from scratch , which are time consuming . This is not acceptable for online services that require instant response such as recommender systems , as the input sequence becomes extremely long ( Ren et al. , 2019 ) . Another setting of “ reasoning after memorization ” , which has the restrictions that the raw input X is not available at the time of answering Q , requires the model to first digest X in a streaming manner , i.e. , incrementally compress the current subsequence of X into a memory M with very limited capacity ( size much smaller than |X| ) . Under such constraints , in the inference phase , we can only capture query-relevant clues from the limited states M ( rather than X ) to infer the answer to Q , where the information compression procedure in M is totally not aware of Q , posing great challenges of what to remember in M . This setting is very similar to the daily situation of our human beings , i.e. , we may not even know the tasksQ that we will answer in the future when we are experiencing current events , and we also can not go back to replay when we are solving problems at hand . However , it ’ s our instincts , which continually process information during our entire life with limited and compressed memory storages , that allow us to recall and draw upon past events to frame our behaviors given the present situations ( Moscovitch et al. , 2016 ; Baddeley , 1992 ) . Compared to “ end-to-end reasoning ” , “ reasoning after memorization ” though may not achieve better precisions at regular tasks with short sequences according to literatures ( Park et al. , 2020 ) , is naturally a better choice for applications like long-sequence recommendation ( Ren et al. , 2019 ) and long-text understanding ( Ding et al. , 2020 ) . Maintaining M can be incremental with only a small part of inputs at each timestep while inference over M and Q is also tractable for online service . Memory augmented neural networks ( MANNs ) ( Graves et al. , 2014 ; 2016 ) introduce a write-read memory that already follows the setting of “ reasoning after memorization ” , which compress the input contents into a fixed-size memory and only read relevant information from the memory during reasoning . However , existing works do not emphasize on using MANNs to perform long-term memory-based reasoning . They learn how to maintain the memory only by back-propagated losses to the final answer and do not design specific training target for long-term memorization , which inevitably lead to the gradual forgetting of early contents ( Le et al. , 2019a ) . That is , when dealing with the long-term input stream , the memory may only focus on current contents and naturally neglect long-term clues . Thus , existing MANNs fail to answer the query relevant to early information due to the lack of long-term memorization training . In this paper , we propose the Continual Memory ( CM ) to further explore this ability of reasoning after long-term memorization . Specifically , we compress the long-term input stream into the continual memory with fixed-size capacity and infer subsequent queries based on the memory . To overcome gradual forgetting of early information and increase the generalization ability of the memorization technique , we develop the extra self-supervised task to recall the recorded history contents from the memory . This is inspired by the fact that human beings can recall details nearby some specific events and distinguish whether a series of events happened in the history , which respectively correspond to two different memory process revealed in cognitive , neuropsychological , and neuroimaging studies , i.e. , recollection and familiarity ( Yonelinas , 2002 ; Moscovitch et al. , 2016 ) . Concretely , we design the self-supervised memorization training with item-level and sequence-level objectives . The item-level objective aims to predict the masked items in history fragments , which are sampled from the original input stream and parts of items are masked as the prediction target . This task tries to endow the recollection ability that enables one to relive past episodes . And the sequence-level objective tries to distinguish whether a historical fragment ever appears in the input stream , where we directly sample positive fragments from the early input stream and replace parts of the items in positive ones as negative fragments . This task enables the familiarity process that can recognize experienced events or stimulus as familiar . We also give implementations on segment-level maintenance of memory to better capture context clues and improve the modeling efficiency . We illustrate the long-term memorization ability of our continual memory via a synthetic task , and evaluate its performance at solving real-world downstream tasks , including long-term text QA , long-term video QA and recommendation with long sequences , showing that it achieves significant advantages over existing MANNs in the “ reasoning after memorizing ” setting . 2 RELATED WORKS . Memory Augmented Neural Networks ( MANNs ) introduce external memory to store and access the past information by differentiable write-read operators . Neural Turing Machine ( NTM ) ( Graves et al. , 2014 ) and Differentiable Neural Computer ( DNC ) ( Graves et al. , 2016 ) are the typical MANNs for human-like reasoning under the setting of “ reasoning after memorizing ” , whose inference relies only on the memory with limited capacity rather than starting from the original input data . In this line of research , Rae et al . ( 2016 ) adopt the sparse memory accessing to reduce computational cost . Csordás & Schmidhuber ( 2019 ) introduce the key/value separation problem of content-based addressing and adopt a mask for memory operations as a solution . Le et al . ( 2019b ) manipulate both data and programs stored in memory to perform universal computations . And Santoro et al . ( 2018 ) ; Le et al . ( 2020a ) consider the complex relational reasoning with the information they remember . However , these works exploit MANNs mainly to help capture long-range dependencies in dealing with input sequences , but not paying efforts in dealing with the gradual forgetting issue in MANNs ( Le et al. , 2019a ) . They share the same training objective as those methods developed for the setting of “ end-to-end reasoning ” , inevitably incurring gradual forgetting of early contents ( Le et al. , 2019a ) . Recently , there are a few works trying to alleviate this problem in training MANNs . Le et al . ( 2019a ) propose to measure “ remember ” ability by the final gradient on the early input , and adopt a uniform writing operation on the memory . Rae et al . ( 2019 ) minimize the difference between successive memory states along with the reasoning objective , as they assume the steadily changed memory states will benefit remembering older information . And Munkhdalai et al . ( 2019 ) design the meta-learned neural memory instead of the conventional array-structured memory and memorize the current and past information by reconstructing the written values via the memory function . Note that our approach is different and parallel to these techniques , since we give no assumptions on what behavior will remember the most . Instead , we optimize towards this goal directly by designing auxiliary tasks in a self-supervised manner . A recent work ( Park et al. , 2020 ) also introduces a selfsupervised memory loss to ensure how well the current input is written to the memory , but it only focuses on remembering the current information and ignoring the long-term information forgetting . Continual learning ( Kirkpatrick et al. , 2017 ; Lopez-Paz & Ranzato , 2017 ; Chaudhry et al. , 2018 ; de Masson d ’ Autume et al. , 2019 ) is another field about the forgetting problem of neural networks , which aims to learn from an infinite stream of data and gradually extend acquired knowledge without catastrophic forgetting of early knowledge . But our continual memory focuses on remembering the infinite information stream and try to overcome gradually forgetting of long-term information . 3 CONTINUAL MEMORY . 3.1 PROBLEM FORMULATION . Given the input stream X = { x1 , x2 , · · · } and a query Q , the methods under the setting of “ endto-end reasoning ” directly learn the reason model T ( X , Q ) to predict the answer A . These is an important assumption that the input stream X can be always accessed while reasoning . And complex interaction between X and Q can be designed to extract query-relevant information in T ( X , Q ) . Obviously , these methods have to store the original input and infer the answer from scratch when the query is known . But under the setting of “ reasoning after memorizing ” , we compress the input stream X into a fixed-size memory M = { mk } Kk=1 with K memory slots and then infer the answer for any relevant query Q by A = R ( M , Q ) . Here we only need to store the compressed memory M , which can be updated in real-time and reused for a series of queries . Since the slot number K in the memory is irrelevant to the input length |X| , this setting only requires O ( 1 ) storage space rather than O ( |X| ) in the setting of “ end-to-end reasoning ” . 3.2 CONTINUAL MEMORY MACHINE . As shown in Figure 1 , given the input stream X = { x1 , x2 , · · · } , we apply a continual memory machine GΘ ( · ) to compress them into continual memory M = { mk } Kk=1 with K memory slots . By self-supervised memorization training , we try to overcome the gradual forgetting of early information and make it possible to capture the clues at any time in the stream . Concretely , based on continual memory , we develop the history recall model Hξ ( · ) to reconstruct the masked history fragments and distinguish positive history fragments from negative ones . Simultaneously , we train the task-specific reason model RΩ ( · ) based on continual memory . Under the setting of “ reasoning after memorizing ” , we can develop the continual memory M = GΘ ( X ) and then infer the answer for any relevant query Q by A = RΩ ( M , Q ) . We deal with the input stream X from the segment level rather than item level , i.e. , we cut the input sequence into fixed-length segments and memorize them into the memory slots segment-by-segment . Compared to existing MANNs ( Graves et al. , 2014 ; 2016 ) , which store the input stream item-by-item orderly with a RNN-based controller , our segmentlevel memorization can further capture the bidirectional context of each item and improve the modeling efficiency . We denote the t-th seg- ment as Xt = { xtn } Nn=1 and the current memory as Mt = { mtk } Kk=1 , where we have recorded t-1 segments in Mt . The xtn and m t n have the dimension dx and dm , respectively . We first model the t-th segment by a Transformer encoder ( Vaswani et al. , 2017 ) and obtain the sequence features Ft = { f tn } Nn=1 with dimension dx . After it , we apply a memory update module to write Ft into Mt . As shown in Figure 2 , we apply a slot-to-item attention to align the sequence features to slot features in the current memory Mt , and then develop multi-head gate-based update . Concretely , we first calculate the slot-to-item attention matrix where each element means the relevance of a slot-item pair , and then learn aligned features Lt = { ltk } Kk=1 for each slot , given by αtkn = w > a tanh ( W a 1m t k +W a 2f t n + b a ) , α̂tkn = exp ( αtkn ) ∑K j=1 exp ( α t jn ) , ltk = N∑ n=1 α̂tknf t n , ( 1 ) where Wa1 ∈ Rdmodel×dm , Wa2 ∈ Rdmodel×dx and ba ∈ Rdmodel are the projection matrices and bias . w > a is the row vector . After it , we project slot features and aligned features into Z subspaces , which is similar to the Multi-Head setting in Transformer ( Vaswani et al. , 2017 ) , given by mtkz = W M z m t k , l t kz = W L z l t k , ( 2 ) where WMz ∈ Rdmodel/Z×dm and WLz ∈ Rdmodel/Z×dx are the projection matrices . The mtkz and ltkz are the slot and aligned sub-features in the z-th subspace . Next , the k-th slot sub-feature m t kz is updated with the corresponding sub-feature ltkz based on the z-th GRU unit with dmodel Z -d hidden states , given by mt+1kz = GRUz ( m t kz , l t kz ) , ( 3 ) where ltkz is the current input of the z-th GRU unit . Next , the new slot feature m t+1 k is aggregated from Z subspaces by mt+1k = W oConcat ( mt+1k1 , · · · , m t+1 kZ ) , where W o ∈ Rdm×dmodel is the aggregation matrix . After the memorization of T segments , we can obtain continual memory MT and we denote it by M for convenience . Note that , at the inference stage , we can develop and update the continual memory in real time by GΘ ( · ) , thus we do not need the input contents during subsequent reasoning and have the ability to reason after long-term memorization .

Summary: To approach "reasoning after memorization", the paper presents a Continual Memory (CM) framework using a memory-augmented neural network (MANN) and self-supervised training. In particular,  the CM  compresses the input sequence into a matrix memory using self-attention mechanisms and gated recurrent update (GRU). Then together with a Transformer decoder, the memory is used for downstream reasoning tasks without the need of referring to the original input sequence.  Moreover, the framework is simultaneously trained with auxiliary losses to enforce memorization capability. A variety of experiments demonstrate promising results, in which the CM outperforms two MANN baselines and shows competitive performance against state-of-the-art methods.