Continual Memory: Can We Reason After Long-Term Memorization?

1 INTRODUCTION . In recent years , the tremendous progress of neural networks has enabled machines to perform reasoning given a query Q and the input contents X , e.g. , infer the answer of given questions from the text/video stream in text/video question answering ( Seo et al. , 2016 ; Le et al. , 2020b ) , or predict whether a user will click the given item based on the user behavior sequence in recommender systems ( Ren et al. , 2019 ; Pi et al. , 2019 ) . Studies that achieve top performances at such reasoning tasks usually follow the setting of “ end-to-end reasoning ” , where the raw input contents X is available at the time of answering Q . In this setting , complex interaction between X and Q can be designed to extract query-relevant information from X with little loss , such as co-attention interaction ( Xiong et al. , 2016 ) . Though these methods ( Seo et al. , 2016 ; Le et al. , 2020b ) can effectively handle these reasoning tasks , they require unlimited storage resources to hold the original input X . Further , they have to encode the whole input and develop the elaborate interaction from scratch , which are time consuming . This is not acceptable for online services that require instant response such as recommender systems , as the input sequence becomes extremely long ( Ren et al. , 2019 ) . Another setting of “ reasoning after memorization ” , which has the restrictions that the raw input X is not available at the time of answering Q , requires the model to first digest X in a streaming manner , i.e. , incrementally compress the current subsequence of X into a memory M with very limited capacity ( size much smaller than |X| ) . Under such constraints , in the inference phase , we can only capture query-relevant clues from the limited states M ( rather than X ) to infer the answer to Q , where the information compression procedure in M is totally not aware of Q , posing great challenges of what to remember in M . This setting is very similar to the daily situation of our human beings , i.e. , we may not even know the tasksQ that we will answer in the future when we are experiencing current events , and we also can not go back to replay when we are solving problems at hand . However , it ’ s our instincts , which continually process information during our entire life with limited and compressed memory storages , that allow us to recall and draw upon past events to frame our behaviors given the present situations ( Moscovitch et al. , 2016 ; Baddeley , 1992 ) . Compared to “ end-to-end reasoning ” , “ reasoning after memorization ” though may not achieve better precisions at regular tasks with short sequences according to literatures ( Park et al. , 2020 ) , is naturally a better choice for applications like long-sequence recommendation ( Ren et al. , 2019 ) and long-text understanding ( Ding et al. , 2020 ) . Maintaining M can be incremental with only a small part of inputs at each timestep while inference over M and Q is also tractable for online service . Memory augmented neural networks ( MANNs ) ( Graves et al. , 2014 ; 2016 ) introduce a write-read memory that already follows the setting of “ reasoning after memorization ” , which compress the input contents into a fixed-size memory and only read relevant information from the memory during reasoning . However , existing works do not emphasize on using MANNs to perform long-term memory-based reasoning . They learn how to maintain the memory only by back-propagated losses to the final answer and do not design specific training target for long-term memorization , which inevitably lead to the gradual forgetting of early contents ( Le et al. , 2019a ) . That is , when dealing with the long-term input stream , the memory may only focus on current contents and naturally neglect long-term clues . Thus , existing MANNs fail to answer the query relevant to early information due to the lack of long-term memorization training . In this paper , we propose the Continual Memory ( CM ) to further explore this ability of reasoning after long-term memorization . Specifically , we compress the long-term input stream into the continual memory with fixed-size capacity and infer subsequent queries based on the memory . To overcome gradual forgetting of early information and increase the generalization ability of the memorization technique , we develop the extra self-supervised task to recall the recorded history contents from the memory . This is inspired by the fact that human beings can recall details nearby some specific events and distinguish whether a series of events happened in the history , which respectively correspond to two different memory process revealed in cognitive , neuropsychological , and neuroimaging studies , i.e. , recollection and familiarity ( Yonelinas , 2002 ; Moscovitch et al. , 2016 ) . Concretely , we design the self-supervised memorization training with item-level and sequence-level objectives . The item-level objective aims to predict the masked items in history fragments , which are sampled from the original input stream and parts of items are masked as the prediction target . This task tries to endow the recollection ability that enables one to relive past episodes . And the sequence-level objective tries to distinguish whether a historical fragment ever appears in the input stream , where we directly sample positive fragments from the early input stream and replace parts of the items in positive ones as negative fragments . This task enables the familiarity process that can recognize experienced events or stimulus as familiar . We also give implementations on segment-level maintenance of memory to better capture context clues and improve the modeling efficiency . We illustrate the long-term memorization ability of our continual memory via a synthetic task , and evaluate its performance at solving real-world downstream tasks , including long-term text QA , long-term video QA and recommendation with long sequences , showing that it achieves significant advantages over existing MANNs in the “ reasoning after memorizing ” setting . 2 RELATED WORKS . Memory Augmented Neural Networks ( MANNs ) introduce external memory to store and access the past information by differentiable write-read operators . Neural Turing Machine ( NTM ) ( Graves et al. , 2014 ) and Differentiable Neural Computer ( DNC ) ( Graves et al. , 2016 ) are the typical MANNs for human-like reasoning under the setting of “ reasoning after memorizing ” , whose inference relies only on the memory with limited capacity rather than starting from the original input data . In this line of research , Rae et al . ( 2016 ) adopt the sparse memory accessing to reduce computational cost . Csordás & Schmidhuber ( 2019 ) introduce the key/value separation problem of content-based addressing and adopt a mask for memory operations as a solution . Le et al . ( 2019b ) manipulate both data and programs stored in memory to perform universal computations . And Santoro et al . ( 2018 ) ; Le et al . ( 2020a ) consider the complex relational reasoning with the information they remember . However , these works exploit MANNs mainly to help capture long-range dependencies in dealing with input sequences , but not paying efforts in dealing with the gradual forgetting issue in MANNs ( Le et al. , 2019a ) . They share the same training objective as those methods developed for the setting of “ end-to-end reasoning ” , inevitably incurring gradual forgetting of early contents ( Le et al. , 2019a ) . Recently , there are a few works trying to alleviate this problem in training MANNs . Le et al . ( 2019a ) propose to measure “ remember ” ability by the final gradient on the early input , and adopt a uniform writing operation on the memory . Rae et al . ( 2019 ) minimize the difference between successive memory states along with the reasoning objective , as they assume the steadily changed memory states will benefit remembering older information . And Munkhdalai et al . ( 2019 ) design the meta-learned neural memory instead of the conventional array-structured memory and memorize the current and past information by reconstructing the written values via the memory function . Note that our approach is different and parallel to these techniques , since we give no assumptions on what behavior will remember the most . Instead , we optimize towards this goal directly by designing auxiliary tasks in a self-supervised manner . A recent work ( Park et al. , 2020 ) also introduces a selfsupervised memory loss to ensure how well the current input is written to the memory , but it only focuses on remembering the current information and ignoring the long-term information forgetting . Continual learning ( Kirkpatrick et al. , 2017 ; Lopez-Paz & Ranzato , 2017 ; Chaudhry et al. , 2018 ; de Masson d ’ Autume et al. , 2019 ) is another field about the forgetting problem of neural networks , which aims to learn from an infinite stream of data and gradually extend acquired knowledge without catastrophic forgetting of early knowledge . But our continual memory focuses on remembering the infinite information stream and try to overcome gradually forgetting of long-term information . 3 CONTINUAL MEMORY . 3.1 PROBLEM FORMULATION . Given the input stream X = { x1 , x2 , · · · } and a query Q , the methods under the setting of “ endto-end reasoning ” directly learn the reason model T ( X , Q ) to predict the answer A . These is an important assumption that the input stream X can be always accessed while reasoning . And complex interaction between X and Q can be designed to extract query-relevant information in T ( X , Q ) . Obviously , these methods have to store the original input and infer the answer from scratch when the query is known . But under the setting of “ reasoning after memorizing ” , we compress the input stream X into a fixed-size memory M = { mk } Kk=1 with K memory slots and then infer the answer for any relevant query Q by A = R ( M , Q ) . Here we only need to store the compressed memory M , which can be updated in real-time and reused for a series of queries . Since the slot number K in the memory is irrelevant to the input length |X| , this setting only requires O ( 1 ) storage space rather than O ( |X| ) in the setting of “ end-to-end reasoning ” . 3.2 CONTINUAL MEMORY MACHINE . As shown in Figure 1 , given the input stream X = { x1 , x2 , · · · } , we apply a continual memory machine GΘ ( · ) to compress them into continual memory M = { mk } Kk=1 with K memory slots . By self-supervised memorization training , we try to overcome the gradual forgetting of early information and make it possible to capture the clues at any time in the stream . Concretely , based on continual memory , we develop the history recall model Hξ ( · ) to reconstruct the masked history fragments and distinguish positive history fragments from negative ones . Simultaneously , we train the task-specific reason model RΩ ( · ) based on continual memory . Under the setting of “ reasoning after memorizing ” , we can develop the continual memory M = GΘ ( X ) and then infer the answer for any relevant query Q by A = RΩ ( M , Q ) . We deal with the input stream X from the segment level rather than item level , i.e. , we cut the input sequence into fixed-length segments and memorize them into the memory slots segment-by-segment . Compared to existing MANNs ( Graves et al. , 2014 ; 2016 ) , which store the input stream item-by-item orderly with a RNN-based controller , our segmentlevel memorization can further capture the bidirectional context of each item and improve the modeling efficiency . We denote the t-th seg- ment as Xt = { xtn } Nn=1 and the current memory as Mt = { mtk } Kk=1 , where we have recorded t-1 segments in Mt . The xtn and m t n have the dimension dx and dm , respectively . We first model the t-th segment by a Transformer encoder ( Vaswani et al. , 2017 ) and obtain the sequence features Ft = { f tn } Nn=1 with dimension dx . After it , we apply a memory update module to write Ft into Mt . As shown in Figure 2 , we apply a slot-to-item attention to align the sequence features to slot features in the current memory Mt , and then develop multi-head gate-based update . Concretely , we first calculate the slot-to-item attention matrix where each element means the relevance of a slot-item pair , and then learn aligned features Lt = { ltk } Kk=1 for each slot , given by αtkn = w > a tanh ( W a 1m t k +W a 2f t n + b a ) , α̂tkn = exp ( αtkn ) ∑K j=1 exp ( α t jn ) , ltk = N∑ n=1 α̂tknf t n , ( 1 ) where Wa1 ∈ Rdmodel×dm , Wa2 ∈ Rdmodel×dx and ba ∈ Rdmodel are the projection matrices and bias . w > a is the row vector . After it , we project slot features and aligned features into Z subspaces , which is similar to the Multi-Head setting in Transformer ( Vaswani et al. , 2017 ) , given by mtkz = W M z m t k , l t kz = W L z l t k , ( 2 ) where WMz ∈ Rdmodel/Z×dm and WLz ∈ Rdmodel/Z×dx are the projection matrices . The mtkz and ltkz are the slot and aligned sub-features in the z-th subspace . Next , the k-th slot sub-feature m t kz is updated with the corresponding sub-feature ltkz based on the z-th GRU unit with dmodel Z -d hidden states , given by mt+1kz = GRUz ( m t kz , l t kz ) , ( 3 ) where ltkz is the current input of the z-th GRU unit . Next , the new slot feature m t+1 k is aggregated from Z subspaces by mt+1k = W oConcat ( mt+1k1 , · · · , m t+1 kZ ) , where W o ∈ Rdm×dmodel is the aggregation matrix . After the memorization of T segments , we can obtain continual memory MT and we denote it by M for convenience . Note that , at the inference stage , we can develop and update the continual memory in real time by GΘ ( · ) , thus we do not need the input contents during subsequent reasoning and have the ability to reason after long-term memorization .

Summary: In this paper, the authors propose the Continual Memory (CM) targeted towards a reasoning scenario called “reasoning after memorization”. The main goal of CM is to enable long-term memorization as opposed to memory networks that suffer from gradual forgetting. They evaluate their model both on synthetic data as well as a few downstream benchmarks.