A Memory Model for Question Answering from Streaming Data Supported by Rehearsal and Anticipation of Coreference Information
Year: 2023

Abstract:
Existing question answering methods often assume that the input content (e.g., documents or videos) is always accessible to solve the task. Alternatively, memory networks were introduced to mimic the human process of incremental comprehension and compression of the information in a fixed-capacity memory. However, these models only learn how to maintain memory by backpropagating errors in the answers through the entire network. Instead, it has been suggested that humans have effective mechanisms to boost their memorization capacities, such as rehearsal and anticipation. Drawing inspiration from these, we propose a memory model that performs rehearsal and anticipation while processing inputs to memorize important information for solving question answering tasks from streaming data. The proposed mechanisms are applied self-supervised during training through masked modeling tasks focused on coreference information. We validate our model on a short-sequence (bAbI) dataset as well as large-sequence textual (NarrativeQA) and video (ActivityNet-QA) question answering datasets, where it achieves substantial improvements over previous memory network approaches. Furthermore, our ablation study confirms the proposed mechanisms’ importance for memory models.

heading: 1 Introduction
text: The question answering (QA) task is one of the most important and challenging tasks in natural language processing (NLP). A significant advance has been seen in this subject thanks to models based on deep learning (Hermann et al., 2015; Seo et al., 2017; Chen et al., 2017; Devlin et al., 2019). However, these models assume that the whole input (e.g., sentences, paragraphs, etc.) can always be accessed while answering the question. This is a reasonable and practical approach if the input sequence is short but becomes less effective and efficient as the input length grows. These models
are also cognitively implausible, lacking incremental human language processing (Tanenhaus et al., 1995; Keller, 2010) (or visual (Goldstone, 1998)), which could enable the models for online tasks where data input is a stream, such as discourse, dialog or video processing.
Memory-augmented neural networks (MANNs) (Graves et al., 2014) offer a solution to this problem as they introduce mechanisms to compress and remember the input contents. MANNs have shown effectiveness in various tasks (Graves et al., 2016; Liu et al., 2019; Le et al., 2020), including question answering (Miller et al., 2016; Xiong et al., 2016; Han et al., 2019; Zhang et al., 2021). In the latter, given a data sequence, the models process the input incrementally, capturing relevant information in the memory to answer a given query. This is akin to the daily behavior of human beings. For instance, our mental state is updated when we encounter a new text segment while reading (Traxler et al., 1997). Later, we can use the memorized information to solve a specific task (Moscovitch et al., 2016).
Despite the success of MANNs and their similarity to human memory processes, they still have one
13124
significant limitation. Most of these models rely on learning a single task to maintain their memory; for example, a MANN for QA must learn what to store in memory while learning to answer questions. Instead, some studies have suggested that human beings are endowed with specific mechanisms for memorization. On the one hand, rehearsal (Waugh and Norman, 1965; Craik and Watkins, 1973) is a mechanism humans use to memorize information more effectively by repeating information over and over to be remembered. On the other hand, anticipation (Hawkins and Blakeslee, 2004; Wittmann et al., 2007; Cole et al., 2015) suggests that our memory could lead to predictions of upcoming material. Furthermore, both processes are potentially guided by coreference information (Jaffe et al., 2020), an integral part of discourse comprehension.
In this work, we propose a Rehearsal and Anticipatory Memory model (RAM) consisting of a memory module that uses a fusion module to integrate incoming information from a data stream to solve the QA task. This model is supported by two pretext tasks respectively inspired by human rehearsal and anticipation processes to enhance memory representation. These tasks are based on masked modeling (Devlin et al., 2019) of coreference information, allowing the model to anticipate a coreferent and link it to the past through memory and rehearsal. We validate our proposed model with datasets of short synthetic text sequences, long realistic text sequences, and video question answering. Results show that our model significantly outperforms previous well-known memory approaches due to the enhanced memory representation achieved by using the pretext tasks.

heading: 2 Proposed Method
text: In this section, we introduce our model, which processes inputs and builds memory representations incrementally from a data stream, used later to provide an answer to a question (Figure 1). RAM leverages attention and gating mechanisms along with masked modeling-based self-supervision to create a simple but effective method to improve memory representation and memorization. We first describe the problem formulation of QA with MANNs. We then present our basic memory architecture for encoding input and decoding an answer. Finally, we introduce our novel self-supervised mechanisms inspired by rehearsal and anticipation guided by coreference information.

heading: 2.1 Problem Formulation
text: Let X be an input stream and Q a question related to the content of X . Standard QA approaches consist of a model G(X,Q) that is trained to predict an answer A. In an incremental processing setup (Han et al., 2019), the model does not have access to the whole input, so a fixed-size memory M is used to compress the input stream X one step at a time. Then a model G(M,Q) learns to infer the answer A for any relevant Q.

heading: 2.2 Memory Model Overview
text: Here we present our basic model that incrementally processes the inputs and integrates the relevant information into memory to then answer a question.
Input Encoding: The input to our model is a sequence of segments (sequence of sentences or images in our experiments). A segment Xt = {xtn}Nn=1 at step t with N tokens is encoded by projecting each token into the latent space by using a linear function F . To inject information about the position of the tokens in the segment, we add learnable position embeddings PE. In this way, we obtain the final token embeddings of the segment Et = {etn}Nn=1 with dimension dmodel. We use this procedure to encode the question as well.
Et = F (Xt) + PE (1)
Note that this form of representation allows the model to granularly capture and memorize the relevant elements of each segment, which differs from previous approaches that rely on a sequence of sumpooled representations (Limbacher and Legenstein, 2020; Le et al., 2020).
Memory Module: Our model is augmented with a parametric memory M = {mk}Kk=1 with K slots
intended to compress the information from the input stream effectively. For this we implement a module that borrows the information fusion idea from (Dou et al., 2022) and extends it to integrate information into the memory incrementally.
As shown in Figure 2, at step t, our module receives two inputs: a segment representation Et and a memory M t to be updated. First, a selfattention layer (SA) is used over the memory to allow the slots to interact with each other. Then, a cross-attention (CA) operation is performed to merge the tokens’ information into memory (Eq. 2). Note that the query matrix QM (with dimension dk) is computed from the memory and the key KE and value VE matrices from the segment. We employ a residual connection around the two layers.
As a result, we obtain an intermedial memory state M̂ t (Eq. 3) with K slots, which has been updated with information from step t. Intuitively each slot can, for example, describe an object or an entity in the input or composition of them.
CA(M,E) = softmax
( QMK
⊤ M√
dk
) VE (2)
M̂ t = CA(SA(M t), Et) (3)
To allow the model to “forget,” an essential ability for memory networks, we employ a gating mechanism proposed by Hutchins et al. (2022). First, the memory candidates are computed (Eq. 4). Unlike the standard LSTM gating, the candidate only is computed using the intermediate memory state M̂ t as it indirectly depends on M t. Later, the input (Eq. 5) and forget gates (Eq. 6) which are used to obtain the new memory state (Eq. 7) are calculated. Note that the matrices W and bias vectors b are trainable.
zkt = tanh(Wzm̂ k t + bz) (4)
ikt = σ(Wim̂ k t + bi − 1) (5)
fkt = σ(Wfm̂ k t + bf + 1) (6)
mkt+1 = m k t ⊙ fkt + zkt ⊙ ikt (7)
Finally, the new memory state M t+1 is projected with a feedforward layer (FFN). We replace the standard residual connection with the gating mechanism explained above. Note that the parameters are not tied, so this gating mechanism is not attached to the previous one.
Output Decoding: After the model has processed the entire input stream and obtained our memory fed MT , we use a decoder to obtain the answer (Figure 3). We use a standard transformer decoder (Vaswani et al., 2017), which allows contextualizing the question token embeddings and aggregating question-relevant clues from memory (Eq. 8). Then, we perform a global average pooling (GAP) of the resulting representations HQ and use a classifier Wclf to predict the answer (Eq. 9) by optimizing a loss Lqa. To allow HQ to capture enough information to answer the question, we perform multi-hop reasoning by iteratively updating HQ states with the same decoder. This is equivalent to having a multilayer decoder with tied weights.
HQ = Decoderqa(EQ,M T ) (8)
A = Wclf (HQ) (9)

heading: 2.3 Rehearsal and Anticipation Mechanisms
text: Rehearsal and anticipation are important processes that occur in our brain to refresh memory and prevent forgetting. On the one hand, rehearsal consists of continually bringing to our mind information already experienced to strengthen those memories (Waugh and Norman, 1965; Craik and Watkins, 1973). On the other hand, anticipation is like imagining the future. It has been found that anticipation can fire memory-forming regions of the brain — even before an event has occurred (Mackiewicz et al., 2006; Wittmann et al., 2007). Furthermore, rehearsal and anticipation were found to be related, might be using the same machinery (Cole et al., 2015), and guided by coreference information (Jaffe et al., 2020).
Motivated by these findings, we propose implementing rehearsal as the reconstruction of the past
(Zhang et al., 2021) and anticipation as the prediction of the future (Oord et al., 2018; Araujo et al., 2021). To use the same machinery, we pose these processes as masked modeling tasks that predict past and future coreference-related tokens.
Rehearsal: This process is performed at step t by randomly selecting a previous (< t) segment to mask some of its tokens. Using a standard transformer decoder (Figure 4), we compute an input segment representation Er to obtain a contextual representation Hr. This process aggregates relevant clues through cross-attention from the updated memory M t+1 to the representations (Eq. 10).
Anticipation: This process is performed similarly to rehearsal (Eq. 10), but the next (t + 1) segment embedding Ea is used instead. We use the same decoder to compute the representation Ha.
Hr/a = Decoderr/a(Er/a,M t) (10)
where r and a denote rehearsal and anticipation, respectively. The final hidden vectors (Hr or Ha) corresponding to masked tokens are fed into a classifier to predict the actual token.
What to rehearse and anticipate: Our model has to store relevant information to predict future elements and, at the same time, avoid forgetting crucial information to predict the elements of previous segments. For this to happen, it is important to mask significant tokens. Unlike the standard masked modeling, we do not mask tokens randomly but instead focus on masking coreference-related tokens (for instance, person, object, or event).
The idea stems from the suggestion that coreference information may be helpful in retrieving probable antecedents from memory and in inform-
ing expectations about future words in language comprehension (Jaffe et al., 2020). Intuitively, by focusing on coreference relationships, our model can anticipate a coreferent and link it to the past through memory and rehearsal, also connecting the related pieces of discourse information.
Separating the tasks: Since the task is the same for both mechanisms, the model would learn to predict the mask regardless of whether it is past or future information. Therefore, we include an additional signal to teach the model to distinguish where the segment comes from. We use the [CLS] token to create a binarized prediction task to classify whether the segment belongs to the past or the future. As a result, we obtain three loss functions that we additively combine to obtain our final selfsupervised modeling loss Lssm (Eq. 11). Our loss function is used in conjunction with the QA loss Lqa to optimize the whole model (Eq. 12).
Lssm = Lanticipation+Lrehearsal+Lbinary (11)
L = Lqa + Lssm (12)
Novelty of our proposed method: Although our model builds on the RM setup and model (Zhang et al., 2021) incorporating anticipation along with rehearsal, we introduce important novelties.
RM trains an additional QA model for each downstream task (history sampler), using the complete input text to determine the important items to answer a question and then use them to rehearse. A key novelty of our method is that the memory uses coreference-related tokens that can be obtained by general-purpose part-of-speech (POS) taggers, making it task agnostic as we do not need to train additional models. By rehearsing and anticipating this information, we induce the model to softly learn coreference relationships, which is important for discourse comprehension.
In addition, RM updates its memory using singlehead cross-attention and a disaggregated GRU network. Instead, our model implements self-attention, cross-attention, and gating in the same memory module, providing better integration of the incoming information with the one in the memory. Finally, our model is light and efficient due to the above features. Our memory module is parallelizable as it does not use a sequential GRU model and does not need to train additional QA models to decide what to rehearse or anticipate.

heading: 3 Experiments

heading: 3.1 Experimental Settings
text: Implementation Details: We use PyTorch to implement our model. We set the head number in all attention layers to 8, our decoders have 3 weightsharing layers and K=20 memory slots. We tie the input embedding and output embedding as it has been demonstrated that it helps to improve language models (Press and Wolf, 2017) and memory networks (Liu et al., 2017). Also, this avoids having multiple output embeddings for the masked and QA modeling, reducing the model size.
We initialize the weights and bias of the gating mechanism using a truncated normal distribution with a mean of 0 and a standard deviation of 0.1, and add a constant of -1 and +1 to the input (Eq. 5) and forget gates (Eq. 6). We use Adam optimizer with a learning rate of 4e-4 to train our model for 300 epochs with a batch size of 128. RAM for short sequence QA has dmodel = 128, and for long sequence QA has dmodel = 256, resulting in 1.2M and 3M parameters, respectively.
For the anticipation and rehearsal masked modeling, we use a POS tagging model to compute coreference-related tokens (nouns, pronouns, and verbs) for all the datasets. We rely on the FLAIR library (Akbik et al., 2019), which provides fast and accurate POS tagging models. We mask coreference-related tokens up to a maximum of 40% of segment tokens, which have been shown to be beneficial (Wettig et al., 2023).
Baselines: We compare our model with very well-known and recent memory networks that incrementally process the input data: DNC (Graves et al., 2016), NUTM (Le et al., 2019), H-Mem (Limbacher and Legenstein, 2020), DAM (Park et al., 2021), STM (Le et al., 2020), CT (Rae et al., 2020), and RM (Zhang et al., 2021). We borrowed the setup of Zhang et al. (2021), in which baselines and memory size were adapted for a fair comparison with our model. Regarding the specific hyperparameters, the NUTM core number was set to 4, the STM query number was set to 8, and the DAM memory block number was set to 2. The CT model uses a 3-layer transformer and a compression ratio of 5. Besides, all the models were set to K=20.
We also include direct methods that solve the task by accessing the entire input. We specify them in the section of each task explored.

heading: 3.2 Short Sequence QA
text: For short sequence QA, we use the bAbI dataset (Weston et al., 2015), a synthetic benchmark widely used to evaluate memory networks. This dataset consists of 20 short-sequence reasoning tasks (less than 100 words) that have to be solved with a common architecture. We use the percent error rate as the metric, which would be the complement of accuracy. Table 1 shows the results. Our model achieves the best result compared to all baselines over 10 runs. Also, RAM has a low variance, being comparable with the most competitive models.

heading: 3.3 Long Sequence QA
text: For long sequence QA, we use NarrativeQA (Kočiský et al., 2018), a dataset with long input contents. It contains about 1,5000 stories and corresponding summaries (more than 600 words). In addition, it includes around 47,000 questions. We adopt the multiple-choice form proposed by the authors. Given a question associated with a summary, the idea is to retrieve the correct answer from a pool of answer candidates drawn from the associated questions. We use mean reciprocal rank (MRR) (Voorhees and Tice, 2000) as the metric to measure how far down the ranking the first relevant answer is. We include two direct models: AS Reader (Kadlec et al., 2016) and E2E-MN (Sukhbaatar et al., 2015) as additional baselines.
Table 2 shows the results for the validation and test sets. Our model achieves 8.46% and 7.38% percentage differences of improvement with respect to the memory baseline RM for validation and test set, respectively. Note that our model also outperforms direct QA methods, being 9.49% and 7.73%, the percentage difference for validation and test set.
These results demonstrate that incremental memory processing constitutes an effective and efficient approach over direct methods when well-designed.

heading: 3.4 Video QA
text: For video QA, we use ActivityNet-QA (Yu et al., 2019a), which consists of 58,000 QA pairs about 5,800 complex web videos, derived from the popular ActivityNet dataset. We used the same procedure as for the bAbI task to predict the answer. As evaluation metric for the test set, we use accuracy.
As the input stream is video, we adapt our network to make it work with the visual modality. Following the original setup of ActivityNet-QA (Yu et al., 2019a), we use fixed sampling to obtain 20 frames as the sequence that our model will process incrementally. To encode the images and decode the masked tokens, we closely follow the framework proposed by Xie et al. (2022) for masked
image modeling. The images of the stream are linearly projected to obtain a sequence of patch embeddings with a 32 × 32 pixel resolution, to which position embeddings are added.
For the rehearsal and anticipation tasks, we use a learnable mask token vector to replace each masked patch. To decode the masked patches, we use a linear layer to yield patch logits to compute the L1 loss considering only the masked patches. The inputs are images, so we no longer have content words to mask. As an alternative, we propose using the objects in the scene as tokens to mask. Therefore, we use the object detector YOLO (Jocher et al., 2022) to compute the patches of the objects in a frame. We use all the object categories but constrain the masking up to 40% of tokens.
Table 3 shows the test set results. Our model achieves a percentage difference improvement of 2.98% with respect to the model RM and 5.49% with respect to the model CT. Note that our model performs almost comparable to the best direct baseline, HCRN, having only a 0.5% percentage difference of improvement.

heading: 4 Further Experimentation

heading: 4.1 Ablation Study
text: The ablation study allows us to understand which components or configurations impact our model’s performance most. We are interested in analyzing the impact of the proposed pretext tasks and masking strategy.
Table 4 shows the four ablation results we performed compared with the default model. First, we found that randomly masking tokens in the masked modeling task slightly affects the results for both bAbI and NarrativeQA. However, for ActivityNetQA, the impact is negligible, suggesting that random masking is good enough to capture relevant visual information, as shown in (Xie et al., 2022).
Also, we found that when we remove the pretext tasks entirely (w/o Lssm), the performance drops dramatically, resulting in being competitive with the no-self-supervised memory STM. By removing only the anticipation task (w/o Lanticipation) or only the rehearsal task (w/o Lrehearsal), the performance approaches the RM model. This means that the backbone of our model is more robust, possibly due to the gating mechanism or the fusion module. Note that removing the anticipation task impacts performance more than removing the rehearsal task, but both mechanisms help to boost
performance. This supports that rehearsal is a crucial mechanism for memory networks (Park et al., 2021; Zhang et al., 2021) and confirms that anticipation is a helpful signal to improve memorization.

heading: 4.2 Effect of Memory Size
text: Figure 5 shows the results of our model and some of the more robust memory-based baselines for NarrativeQA and ActivityNet-QA. We evaluate all the models using 10 to 25 memory slots. We do not include the results of the bAbI task because we found that the performance does not vary substantially when changing the memory size of the models.
We found that the performance for all models gradually increases with growing memory size. As shown, the CT model is the most affected when the size is reduced, followed by the STM model. The behavior of STM, RM, and RAM is slightly similar but with a difference in performance. We found that our model with low memory is almost equivalent to the performance of RM, but as memory capacity increases, the performance gap also increases. We noticed that memory slots higher than 25 do not show substantial improvements, suggesting that 20-25 slots are adequate for these tasks.

heading: 4.3 What is the Attention Capturing?
text: This section aims to reveal what kind of interactions are taking place between the inputs and the
memory slots by visualizing the attention weights. Our approach uses cross-attention to both integrate information in the memory and also to retrieve query-relevant information from the memory to obtain the answer. Therefore, we analyze the attention of those layers obtained from random examples.
Figure 6 shows an example from bAbI task 5. In this example, the objective is reasoning over argument relations. We found that 20 slots of memory capture several repeated patterns, so we analyze some of them. We notice that slot 8 is specialized to capture relationships between entities (e.g., fred, jeff), as more attention is seen on entity names and action verbs across all the steps. Besides, the attention from the query to slot 8 was related to the entities and the activity they carried out.
Regarding slot 18, we notice that this slot is specialized to capture the relation between entities and objects. We found that some slots became more
specialized to capture specific items, such as this one for the "football" object. We can see that all entities have high attention and the word "football", which suggests that this slot is tracking the state of the football through the steps. Interestingly, we found that the question attention over the slots is very high for the word "what", showing that slot 8 contains information about the football.
Figure 7 shows an example from NarrativeQA. In this case, the example consists of several sentences, so we only include some relevant ones to solve the question. The example shows an outcome question inquiring about the specific result of a particular event. We notice that slot 2 mainly tracks people of the story, for instance, "Ryan", "Jeff" and "Anne". It is also possible to see that attention is paid to the pronoun tokens.
Regarding slot 6, we find that the attention is a bit more distributed across the tokens. However,
more attention is paid to verbs (for example, "try" and "help"), which suggests that this slot specializes in events and participants. Note that attention to the question also presents the same behavior.
Figure 8 shows an example from ActivityNetQA. This example presents a counting task across the steps. In this case, the input stream is images. Our visualizations show bright regions when attention is high. We find a behavior similar to textbased models since some slots specialize in entities or the interaction between them or with objects.
We see that slot 7 is paying attention to the objects and the people around, suggesting that this slot captures the interaction between entities and objects across the steps. Note that slot 7 got the most attention for the words "how many", which supports the hypothesis that this slot tracks entity information.
As for slot 12, we find that it is specialized to capture information about the objects (cups, in this case). The focus is mainly on the cups; however, sometimes, in other places in the image. This suggests that attention tracks the interaction between objects in the scene.
These results support our initial hypothesis that slots could capture information about entities, objects, and relations between them. Also, it is possible to see that our model can track the state of those entities or objects across the steps, suggesting it can represent sequentially coreference-related information when updating the memory.

heading: 5 Related Work

heading: 5.1 Question Answering
text: In recent years there has been a significant advance in QA. This is thanks to the new datasets and novel deep learning architectures proposed to solve them. One of the most important datasets is SQuAD (Rajpurkar et al., 2016), which proposes span prediction as the task for QA about a given paragraph. SQuAD has been one of the reasons for progress in the field (Liu et al., 2021), as several attention models have been proposed for this task. For example, BiDAF (Seo et al., 2017) proposes a specialized bi-directional attention mechanism.
Recently, more complex QA tasks were proposed, such as long-range QA (Kočiský et al., 2018; Pang et al., 2022) and commonsense QA (Talmor et al., 2019). Also, other modalities were explored, using images (Johnson et al., 2017; Goyal et al., 2017) or videos (Lei et al., 2018; Yu et al., 2019b)
as input. For these, large pre-trained models (Devlin et al., 2019) have achieved outstanding results.
The models mentioned above always have access to the whole input. However, this is ineffective when the input is a data stream or a long sequence. For this reason, memory networks have been proposed to perform incremental processing to accumulate knowledge into memory and afterward perform the QA task (Miller et al., 2016; Xiong et al., 2016; Han et al., 2019; Zhang et al., 2021).

heading: 5.2 Memory-augmented Neural Networks
text: MANNs introduce models with external memory to store and access the past contents by differentiable operators. These models attempt to mimic human memory processes, bringing them closer to biological plausibility. NTM (Graves et al., 2014) and DNC (Graves et al., 2016) are well-known memorization and reasoning models which use memory instead of original input to generate inferences.
In this line of research, several papers have proposed new approaches for different NLP tasks, such as language modeling (Santoro et al., 2018), dialogue (Zhang et al., 2019), machine translation (Kaiser et al., 2017), coreference resolution (Liu et al., 2019), question answering (Henaff et al., 2017), and visual question answering (Xiong et al., 2016). These approaches propose new ways of storing and retrieving information (Henaff et al., 2017; Banino et al., 2020), relational or associative mechanisms (Santoro et al., 2018; Limbacher and Legenstein, 2020; Le et al., 2020), and methods to improve memory representation and prevent forgetting (Park et al., 2021; Zhang et al., 2021).
In recent years, some works have been proposed with the latter objective. DAM (Park et al., 2021) introduced a rehearsal task to ensure a current input is well stored by reproducing it using the memory. RM (Zhang et al., 2021) was proposed to tackle the long-term memorization problem, as this model can rehearse a current input and previous ones employing a recollection and familiarity task. Our work extends these approaches and builds on recent findings from neurocognitive science that posit that our memory is guided by rehearsal and anticipation of coreference information, and they might use the same machinery to improve human memorization (Cole et al., 2015). Unlike DAM and RM, our model proposes to perform not only rehearsal but also anticipation using masked modeling and a task to detect whether the information belongs to the future or the past. These mechanisms aim to improve memorization and prevent forgetting.

heading: 6 Conclusion
text: This article presents RAM, a QA memory model supported by two self-supervised pretext tasks to improve memory presentation. It uses masked modeling to mimic the human processes of rehearsal and anticipation of coreference information. Our experiments on short sequence QA, long sequence QA, and video QA show that our model substantially outperforms previous memory models. Our further experimentation demonstrates that the rehearsal and anticipation tasks help to improve memorization and that the model leverages coreference information to support the processes.
Limitations
This work has some limitations regarding the architecture and the data used: Our model assumes that masked modeling could be used as rehearsal and anticipation tasks; however, other approaches could also be effective. We use transformer layers, so our model scalability is tied to the scalability of the transformer model. Also, our approach relies on obtaining additional annotation from pre-trained models for the masking process, so we are limited to the misprediction of those models. We only tested our model with the English language; further exploration of other languages would be valuable to validate the language-independent functionality of the model.
Ethics Statement
Our article presents a novel approach and has not been published in whole or in part elsewhere. The data used to train the model does not imply any violation of privacy. The potential negative social impacts from this work are similar to any other NLP models. Question answering models could potentially be used to create malicious chatbots. This work does not include experimentation with humans or animals.

heading: Acknowledgements
text: This work was partially funded by European Research Council Advanced Grant 788506, FONDECYT grant 1221425, and National Center for Artificial Intelligence CENIA FB210017, Basal ANID.

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Left blank.

heading: 3 A2. Did you discuss any potential risks of your work?
text: Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Left blank.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Left blank.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Left blank.

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Left blank.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.
C 3 Did you run computational experiments? Left blank.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Left blank.
7 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Not reported for all the experiments.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: Unlike DAM and RM, our model proposes to perform not only rehearsal but also anticipation using masked modeling and a task to detect whether the information belongs to the future or the past. These mechanisms aim to improve memorization and prevent forgetting