GrowOVER: How Can LLMs Adapt to Growing Real-World Knowledge? (2024)

Abstract: In the real world, knowledge is constantly evolving, which can render existing knowledge-based datasets outdated. This unreliability highlights the critical need for continuous updates to ensure both accuracy and relevance in knowledgeintensive tasks. To address this, we propose GrowOVER-QA and GrowOVERDialogue, dynamic open-domain QA and dialogue benchmarks that undergo a continuous cycle of updates, keeping pace with the rapid evolution of knowledge. Our research indicates that retrieval-augmented language models (RaLMs) struggle with knowledge that has not been trained on or recently updated. Consequently, we introduce a novel retrieval-interactive language model framework, where the language model evaluates and reflects on its answers for further re-retrieval. Our exhaustive experiments demonstrate that our training-free framework significantly improves upon existing methods, performing comparably to or even surpassing continuously trained language models.

heading: 1 Introduction
text: In natural language research, many knowledgeintensive tasks have been actively studied, including open-domain QA (Kwiatkowski et al., 2019; Joshi et al., 2017; Yang et al., 2018), factchecking (Thorne et al., 2018), entity linking (Hoffart et al., 2011), and open-domain dialogue (Dinan et al., 2018), to name a few (Petroni et al., 2021; Levy et al., 2017). Such a knowledge-intensive task is mostly to utilize world knowledge to generate a proper answer for a given query (Lewis et al., 2020). However, the amount of real-world knowledge is often too enormous for models to fully store them in the parameters. Thus, in most scenarios, a retriever is employed to seek relevant para-
graphs or documents in a knowledge bank, such as a vector database, and a generator provides answers based on the retrieved passages or documents (Lewis et al., 2020; Guu et al., 2020). Previous benchmarks (Kwiatkowski et al., 2019; Dinan et al., 2018; Petroni et al., 2021) annotate gold answers and the evidence text needed to predict them, and evaluate the retriever using the evidence text and the generator using the gold answers.
In the real world, new knowledge is constantly being created, and existing knowledge is changing over time, causing annotated benchmarks to become quickly outdated. To handle this issue, Kim et al. (2023) and Margatina et al. (2023) respectively suggest dynamic QA and cloze query benchmarks, which are automatically generated by comparing two Wikipedia (or Wikidata) snapshots at different times. However, they provide no annotated evidence text for the gold answers, which may make it difficult to evaluate retrievers in opendomain knowledge-intensive tasks. In the retrieval-augmented generation (RAG) framework, it is crucial to accurately measure the performance of each component as well as end-to-end performance. This enables precise identification of error sources and inaccuracies,
3282
allowing for less frequent updates to Large Language Models (LLMs). Also, the tasks of these benchmarks require the model to provide only direct answers, which mainly consist of entities. In contrast, real-world knowledge can’t be structured as simple question-andanswer pairs. Instead, it’s more accurately represented as a vast, interconnected knowledge graph. Therefore, there are limitations in evaluating the generator’s ability to provide contextually appropriate and informative answers that incorporate relevant background knowledge.
To overcome such limitations, we propose novel open-domain dynamic benchmarks, GrowOVER-QA and GrowOVER-Dialogue (Growing Open-domain knowledge benchmarks for retrieVal-augmented genERation). As illustrated in Figure 1, GrowOVER provides the evidence text along with the gold answers, which can be used to evaluate the retriever. In addition, we utilize the evidence text to verify the validity of previously generated datasets and maintain valid ones in the succeeding time steps. Consequently, our benchmarks continue to grow from their initial creation as new Wikipedia snapshots continue to come in. Furthermore, it introduces a dialogue task to better evaluate the generator. The opendomain dialogue task challenges models to adapt to the user’s responses and potentially shift topics while still responding accurately. This demands a more sophisticated understanding and application of world knowledge, allowing for a more extensive evaluation. Table 1 presents the comparison of GrowOVER with other benchmarks.
To enable intermittently updated LLMs to cope with the rapidly evolving world, recent research has explored two approaches: retrieval (Kasai et al., 2022; Ram et al., 2023) and
continual pretraining (Jang et al., 2022; Kim et al., 2023). The retrieval approach employs a retriever to supply LLMs with new information from an updated database, leveraging their in-context learning capabilities. The continual pertaining approach updates outdated knowledge within the LLMs’ parameters, thereby preventing hallucinations. However, constantly updating LLMs can be costly and prone to performance degradation, while relying solely on the retriever can be vulnerable. Therefore, we propose a retrieval-interactive LLM (RiLM). In RiLM, the LLM evaluates its own answers and, if found unreliable, provides feedback to the retriever to locate more relevant documents. The LLM then uses the feedback to generate improved answers.
Finally, our contributions are as follows. 1. We introduce GrowOVER, a set of dy-
namic QA and dialogue benchmarks that evaluate both retrievers and generators by annotating the evidence text and introducing a challenging dialogue task.
2. We propose RiLM, a framework where the LLM evaluates its own answers and provides feedback to the retriever to correct retrievals, thereby regenerating better answers.
3. We empirically demonstrate the effectiveness of our method without requiring additional pre-training of the LLM.

heading: 2 Related Work
text: Temporal sensitivity. Temporal misalignment occurs when training and test datasets originate from different time periods. Past studies (Lazaridou et al., 2021; Luu et al., 2021) report poor performance in downstream tasks when making predictions beyond the training period. Thus, recent research (Dhingra et al.,
2022; Liska et al., 2022; Saxena et al., 2021; Jang et al., 2022; Kim et al., 2023) efforts to evaluate how LLMs handle time-sensitive information. In particular, there have been approaches to utilize a retriever for time-sensitive knowledge. Zhang and Choi (2021) and Longpre et al. (2021) report that, even with an updated evidence corpus, language models trained on previous data struggle to respond to questions in the present. However, Kasai et al. (2022) show that LLMs can adjust their generated responses to recently retrieved documents provided by prompting. Still, when failing to retrieve appropriate documents, LLMs may produce outdated answers. To address this issue, we propose a retrieval-interactive LLM framework that allows the LLM to provide feedback to the retriever to fetch more relevant documents when the answer is less reliable.
Retrieval augmented LLM. In the initial stages, language models had limited capacity to store a vast amount of factual details. Hence, prior studies (Lewis et al., 2020; Guu et al., 2020) introduce RAG, where the generator responds based on the passages provided by the retriever. As LLMs grow larger and are pre-trained on huge text corpora, Ram et al. (2023) propose, instead of training LLMs, to combine retrieved content with a query into a prompt for LLMs to generate an answer. Additionally, Shi et al. (2023) use an ensemble scheme that provides multiple documents to the LLM, which determines the next token by summing the probability of the next token for each document. Recently, studies have focused on when or what to retrieve. For instance, Asai et al. (2023) uses special tokens to decide when to retrieve, and then generates and reflects on the passages and generated answers. Similarly, Jiang et al. (2023) generates a sentence, and if the generated tokens have low probabilities, it retrieves passages using the generated sentence for a long-form generation task.
Continual Knowledge Learning. Continual Learning (CL) focuses on training a model on multiple sequential tasks, while retaining knowledge from previously learned tasks and adapting to new ones (Chen and Liu, 2018; He et al., 2021; Chen et al., 2020; Xu et al., 2023; Hu et al., 2021; Wang et al., 2020). In the realm of knowledge-intensive tasks, there is an additional imperative for knowledge revision.
Addressing this, Jang et al. (2021) introduce the concept of continual knowledge learning (CKL) to manage the dynamic nature of world knowledge. It involves not just retaining previous knowledge but also embracing new information and adapting to updates. These objectives align with the goal of our benchmarks.
3 The GrowOVER Dataset
GrowOVER comprises two distinct datasets: QA and Dialogue. GrowOVER-QA is designed to evaluate the ability to recall entities, while GrowOVER-Dialogue features userexpert interactions over 3-4 turns to highlight generation capabilities. Each instance from both datasets is annotated with the evidence text and the type: Unchanged, Changed, or New. Our goal is to evaluate the retention of unchanged knowledge, the updating of changed knowledge, and the acquisition of new knowledge, aligning with the objectives of CKL.
Article selection. GrowOVER is based on Wikipedia snapshots1, which contain a vast amount of world knowledge. Although there is no limitation to applying our algorithm to generate QA and conversation for any articles, we select the articles linked to Portal:Current Events Wikipedia article from January 2023 to December 2023 (About 12K articles).
Overall process. We create initial QA and dialogue instances using the 2023-08-20 Wikipedia snapshot using GPT-4. For each subsequent snapshot, we label each sentence in articles as unchanged, changed, or new by comparing it to the previous month’s snapshot. Then, we retain QA and dialogue instances when the evidence text is labeled as unchanged, and create new instances from new or changed sentences. This process is repeated for each month’s new snapshot from September through December. Algorithms and prompt templates for sentence labeling and data generation are detailed in Appendix C. Also, the statistics of datasets are provided in Appendix E.

heading: 3.1 Initial Generation
text: GrowOVER-QA. Each article in snapshots in the initial month is split into paragraphs. Then,
1We download Wikipedia data dumps from https:// dumps.wikimedia.org/enwiki/ and use monthly snapshots from 2023-08-20 to 2023-12-20.
we select up to four paragraphs, for each of which we prompt GPT-4 to generate QA. The sentences used for generating QA become the evidence text. To ensure dataset quality, we experimentally set the criteria for paragraph selection. First, we select paragraphs with less than five sentences and between 300 and 600 characters. If there are more than four satisfying paragraphs, we use K-Means clustering to group them into four clusters and randomly select one paragraph from each cluster. This creates semantically diverse QAs while avoiding too similar QAs. Afterward, we guide GPT-4 to satisfy the following: i) the question should be directly answered without the context (e.g. no "According to the context), ii) the answers must be short and be entities, and iii) return a bounding box indicating the sentence(s) that include the answer as the evidence text.
GrowOVER-Dialogue. The paragraph selection process is the same as QA. For each paragraph, we ask GPT-4 to create a dialogue involving user-expert interactions spanning 3- 4 turns. We guide it to return the sentence used for generating each turn and then annotate it as the evidence text for each turn.

heading: 3.2 Sentence Labeling
text: Unchanged. Each time a new snapshot of Wikipedia becomes available, we first identify unchanged sentences for each pair of old and new articles. We compute sentence similarity using SimCSE (Berant et al., 2013) to localize semantically identical sentence pairs, (sold, snew). If the similarity score exceeds a threshold of 0.99, we label (sold, snew) as unchanged.
Additionally, we group sentences and com-
pare them to find all unchanged sentences. Wikipedia articles generally maintain the order of sentences even after editing. Hence, we group old and new sentences into Sold and Snew, respectively, that lie between pairs of previously identified unchanged sentences. Then, we compute the similarity between all subsets of Sold and Snew. If any subset of Snew matches with a subset of Sold with a similarity score above the threshold, those sentences are labeled as unchanged. For instance, if we have matched (sold1 , snew1) and (sold4 , snew5) as unchanged, we then compare similarity between all subsets of Sold = {sold2 , sold3} and Snew = {snew2 , snew3 , snew4}. If the similarity score between {sold2} and {snew2 , snew3} exceeds the threshold, we label (sold2 , {snew2 , snew3}) as unchanged as well. After identifying unchanged sentences in the groups, we update the groups enclosed by the new pairs of unchanged sentences. In the previous example, Sold becomes {sold3} and Snew becomes {snew4}.
NLI. Next, we classify unlabeled sentences in Sold and Snew using a natural language inference (NLI) task with RoBERTa (Liu et al., 2019) fine-tuned on the MultiNLI dataset (Williams et al., 2018). The NLI task, given a premise and a hypothesis, classifies the hypothesis as entailment, neutral, or contradiction. In this step, we provide the model with each sentence pair (sold, snew) where sold ∈ Sold and snew ∈ Snew treating sold as the premise and snew the hypothesis. If snew is classified as entailed by any sold, we label that snew as unchanged.
Changed. Else if snew is classified as contradiction with any sold, we label the (sold, snew) as changed. Besides, we check whether their simi-
larity is higher than τ2 set to 0.6 since changed sentences share some content, not entirely new. After that, we double-check whether sold and snew are contradictory with GPT-4 to ensure that (sold, snew) is changed.
New. Otherwise, if snew is classified as neutral, we further check their similarity scores. If the similarity with all sold ∈ Sold is lower than τ1 set to 0.7, we classify that snew as new since new sentence should not have similar counterparts in the old document.

heading: 3.3 Temporal Updates
text: If an article is newly added in the new snapshot, we perform the initial generation as done in 3.1. Otherwise, based on the results of sentence labeling, we update GrowOVER in two ways: i) maintain or exclude existing QA and dialogue turn instances, and ii) generate new instances.
Maintenance. Each QA and turn instance is annotated with evidence sentences and their indices within the article. If all evidence sentences sold are labeled as unchanged, we keep the corresponding QA or turn as Unchanged and update the evidence sentences and their indices as of the new snapshot. Otherwise, we exclude the instance since it is not guaranteed as Unchanged. For QA, we simply delete the instance from our dataset. But, for dialogue, we only exclude the turn when evaluation instead of the whole dialogue.
Generation. We generate new QA and dialogues with new and changed sentences. For QA, we find consecutive new sentences and split them into multiple groups if more than six sentences. For each group, we prompt GPT-4 to generate a New QA instance and annotate the evidence text as done before. For changed sentences, we provide GPT-4 with both the original and the revised sentences and prompt it to generate a Changed question with contradictory answers based on each sentence and annotate the revised sentences as evidence text. For dialogue generation, similarly to the initial generation process, we select informative paragraphs with changed or new sentences. We then prompt GPT-4 to generate dialogues and annotate the used sentences as the evidence text for each turn. If the evidence text is changed or new, we label the generated turn as Changed and New, respectively.

heading: 4 Approach
text: If the LLM has been trained on outdated data or has never been trained on new data, it may not be able to answer the questions on new information correctly. However, updating the parameters of such models should be conducted with the greatest caution to avoid potential side effects, such as catastrophic forgetting. Therefore, our approach ensures that neither the LLM nor the retriever is continuously trained with new data.
To adapt LLMs to rapidly evolving world knowledge, we propose RiLM, as shown in Figure 3. While freezing both the LLM and the retriever, we introduce the Decision Gate, which decides whether to accept the LLM’s answer based on its certainty score (section 4.1). If the answer is not sufficiently confident, the retrieval-generation process is performed again, termed Adaptive Re-Retrieval (section 4.2). In this process, the LLM’s previous output is fed back to the retriever to fetch documents again, enabling the LLM to generate better answers.
Similar to our approach, Asai et al. (2023) propose using reflection tokens to confirm output relevance, support, or completeness. However, their method requires training LLMs to predict reflection tokens. In contrast, RiLM only involves training the certainty classifier. Additionally, for long-form text generation, Jiang et al. (2023) generate the next sentence and then concatenate it with the question to form a retrieval query. They exclude tokens generated with low probability to avoid interrupting retrieval. Our approach also uses generated answers for re-retrieval but considers all generated tokens only to the extent that the LLM is certain about the answer.

heading: 4.1 The Decision Gate
text: Previous work (Ram et al., 2023) has often concatenated top-k retrieved documents {D1 · · ·Dk} with a query Q into a single prompt to the LLM. However, some irrelevant content may degrade performance. Thus, following Shi et al. (2023), we concatenate each document Di with the query Q into a prompt to LLM in parallel. To select the best answer from these prompts, we add a certainty classifier to the last multi-head attention layer of the LLM.
More specifically, we pass the query and each document through the LLM to obtain the last hidden state vector hLLM(Q,Di). For each hidden state, the certainty classifier predicts either of the following three labels: i) reliable: the LLM confidently knows the answer, ii) misleading: the LLM knows the answer but could be wrong, or iii) uncertain: the LLM does not know the answer exactly.
To train the classifier, we assume that the LLM knows the answer if the data it has been trained on remains unchanged and the retrieval succeeds. Conversely, the LLM might incorrectly know the answer if the data it has been trained on has been updated and the retrieval fails. Lastly, the LLM does not know the answer if it has never been trained on the data and the retrieval fails. Based on this assumption, we train the classifier to predict: i) reliable: given Unchanged QA/turn with correct retrieval, ii) misleading: given Changed QA/turn with wrong retrieval, iii) uncertain: given New QA/turn with wrong retrieval. We train the classifier separately for QA and dialogue tasks. We use 512, 245, and 512 data points of Unchanged, Changed, and New GrowOVER-QA, respectively, and 512, 133, and 512 Unchanged, Changed, and New turns of GrowOVER-Dialogue from September.
We choose the hidden state hLLM(Q,D∗) with which the certainty classifier outputs the highest reliable probability:
D∗ = argmax D ∈ {Di} pCLF (reliable | hLLM(Q,D)). (1)
Afterward, LLM generates the answer yLLM based on the hLLM(Q,D∗). If the certainty classifier predicts the label as reliable given hLLM(Q,D
∗), the Decision Gate adopts the answer; otherwise, we return to the retrieval step.

heading: 4.2 Adaptive Re-Retrieval
text: If the classifier does not predict reliable, we reretrieve documents since all top-k documents are unlikely to be helpful in generating correct answers. Instead of simply retrieving the next set of top-k documents, we propose an Adaptive Re-Retrieval (ARR) method. This method feeds the LLM’s answer and certainty value back to the retriever to improve relevance.
In ARR, the retriever relies on the LLM’s answer to the extent that it is reliable. The reliable probability is computed by
ω = λ pCLF (reliable | hLLM(Q,D∗)). (2)
The hyperparameter λ is set to optimize reretrieval relevance with the training dataset for the certainty classifier. Therefore, the relevance score in the ARR is calculated as a weighted sum of two components: cosine similarity between the query and the document, and the similarity between the document and a concatenation of the query with the generated answer yLLM. This is formulated as:
score = (1− ω) sim(E(Q),E(D)) + ω sim(E([Q : yLLM]),E(D)).
(3)
The retriever re-retrieves top-k documents based on Eq.(3). We adjust the reflection of the generated answer based on the LLM’s predicted reliable probability: the less reliable an answer is, the less likely it is to be used. After the re-retrieval, we again choose the last hidden state vector from the document with the highest reliable probability among the reretrieved documents. In the final step, the Decision Gate compares the reliable probabilities of the initial and newly generated answers, selecting the one with the higher probability.

heading: 5 Experiments

heading: 5.1 Experimental Setup
text: Baselines. We use five types of baselines: i) Vanilla: LLM without retrievals, ii) SelfRAG: an adaptive RAG baseline (Asai et al., 2023), RaLM: vanilla LLM with concatenated retrievals (Ram et al., 2023), iii) RaLM-CP: continuously pre-trained LLM (Jang et al., 2022) with concatenated retrievals, iv) RaLM-D∗: LLM generates an answer with the classifier’s selected document, and v) RiLM. We use top-k (k=3) documents for retrievals.
Database. Since it requires much computation to use the entire snapshot (6M articles) for the database, we randomly select 100K articles in addition to the 12K articles selected for GrowOVER generation. We first split these articles using the LangChain document loader2, which semantically segments given documents. Then, we index documents using FAISS (Johnson et al., 2019) following Shi et al. (2023).
Retriever. Even though all retrievers can be plugged into our framework, we use SentBERT (Reimers and Gurevych, 2019) since it reduces time cost using Siamese encoders with strong performance. It uses cosine similarity to calculate sentence similarity. We also test Contriever (Izacard et al., 2021) and present the results in Appendix B.1. We use questions as retrieval queries for the QA task, and user queries combined with the chat history for the dialogue response task.
LLM. We use Llama2-7B (Touvron et al., 2023). We initially pre-train the model with the selected articles in the 2023-08-20 snapshot. For RaLM-CP, we continuously pre-train the model on each new snapshot available every month, using only the selected articles from these snapshots for dataset generation. For the other baselines, we freeze the initially pretrained model. In the QA task, the prompt consists of the retrieved documents and questions, while in the dialogue task, the prompt concatenates the chat history, retrieved documents, and the user query.
Metric. We use the F1 score to evaluate QA, following Petroni et al. (2021) and use the BLEU (Papineni et al., 2002) score for the dialogue task, follwing (Chan et al., 2021).
2https://python.langchain.com/docs/modules/ data_connection/document_loaders/.

heading: 5.2 Experimental Results
text: We report not only the end-to-end performance of the QA and dialogue tasks, but also the performance of the classifier and ARR to demonstrate the effectiveness of each component. For all experiments, we exclude the training dataset of the classifier.

heading: 5.2.1 Results of the Classifier
text: Experimental results of the classifier are presented in Table 2. We evaluate the accuracy of all data points from all months. The accuracy is approximately 75 for GrowOVER-QA and 58 for GrowOVER-Dialogue. Since chat history is included in the prompts, it is harder for the classifier to gauge certainty based on documents and queries, so performance tends to suffer slightly. Also, to show the actual effectiveness of the classifier, we separately calculate the average F1 / BLEU scores for adopted answers and not-adopted answers in the Decision Gate. The gap between the average F1 / BLEU score of adopted and notadopted answers is approximately 25.0 for the QA task and 2.7 for the dialogue task. This significant gap indicates the classifier can predict the certainty and reliability of LLM.

heading: 5.2.2 Results of Adaptive Re-Retrieval
text: We compare the accuracy of ARR and two baselines: i) search with only query (Q) and ii) always append yLLM to Q (Q:yLLM), each of which are using sim(E(Q),E(D)) and sim(E([Q:yLLM]),E(D)), where ω = 0 and

heading: New
text: ω = 1 in equation (3), respectively. The gap between ARR and the former implies the degree to which reliable yLLM improves retrieval relevance, while the gap between ARR and the latter represents the degree to which ARR ignores potentially incorrect yLLM. We exclude the first retrieved documents from re-retrieval to avoid using duplicate retrievals. The results are shown in table 3. In the QA task, ARR improves Q and Q:yLLM by approximately 1.2 and 0.3, respectively, while in the dialogue task, the performance enhances by 1.0 and 0.2, respectively. This result indicates that the output of LLM with reliable probability can aid in re-retrieval. Moreover, the effect of retrieval relevance on the final answer is demonstrated in the subsequent end-to-end results.
5.2.3 Results of GrowOVER-QA Table 4 shows the performance on GrowOVERQA from September to December. It displays the F1 score for New, Changed, and Unchanged QAs, as well as All types. When comparing Vanilla to other baselines, it’s clear that the retrieval significantly enhances performance, highlighting its crucial role in opendomain tasks. Self-RAG shows modest improvement over Vanilla but underperforms
compared to other retrieval-augmented models. Also, in general, RiLM demonstrates outstanding performance over other baselines. For New, RiLM improves RaLM from 1.2 in September to 2.7 in November. It shows a higher F1 score over all months than RaLM-CP, even though RiLM is not continuously trained. For Changed, our method outperforms other baselines, except for October, when RaLMCP exceeds our method by 1.3. Nevertheless, our RiLM shows much more robust performance across the other months, with improvements ranging from 0.8 to 4.7. Moreover, for Unchanged and All, our method surpasses the performance of other baselines. However, over several months, all baselines show performance degradation over months, indicating the need for the model update in the future.
5.2.4 Results of GrowOVER-Dialogue Table 5 displays the consequences on GrowOVER-Dialogue from September to December. Similar to QA task, RiLM generally exhibits superior performance over other baselines. For instance, RiLM improves upon RaLM-CP by an average of 0.3 for New and 0.4 for Changed. Also, RiLM enhances performance on both the Unchanged and All
datasets, indicating its overall effectiveness. However, for Changed in November, not only RiLM but also RaLM-D∗ underperforms compared to RaLM. This performance drop only occurs on Changed, which may indicate the limitation in predicting misleading cases. From the consistent performance improvements on New and Unchanged, we can infer that LLM can identify what it knows and what it doesn’t know. On the other hand, it occasionally has difficulty assuming that its answer might be wrong and verifying it.

heading: 5.2.5 Label-Based Analysis
text: In addition to the monthly results, we also report the average experimental results for each label across the months. Table 6 shows ARR accuracy for New, Changed, and Unchanged labels. For the Changed label in the QA task, Q:yLLM performs 1.6 points lower than Q, indicating that relying entirely on misleading yLLM can significantly degrade performance. Conversely, in the dialogue task, misleading yLLM for Changed improves performance by 1.2 points. This suggests that the detailed sentences generated in the dialogue task benefit from common knowledge between the old and new data, aiding re-retrieval. For New, the LLM struggles to generate detailed answers, resulting in a small gain of 0.1 points. For Unchanged in both tasks, yLLM improves performance by 1.4 and 0.8 points, respectively, though still lower than ARR. Overall, ARR demonstrates robust and improved performance in all scenarios.
To analyze errors corrected by Decision Gate and ARR, we report the performance at each stage of the RiLM pipeline, averaged by label type, as shown in Table 7. This includes results for answers adopted by the decision gate, not adopted answers before and after ARR and after the final selection, and the final aggregated result. The classifier performs well across all types, as evidenced by the gap between adopted and not-adopted answers before ARR. The efficiency of ARR and the decision gate is highlighted by the performance improvements from before ARR to after the final selection. Although performance decreases after ARR due to the exclusion of top-k documents from re-retrieval, the final selection results for not-adopted answers improve, indicating
the classifier’s effectiveness in selecting more reliable answers. In the QA task, F1 scores for not-adopted answers improve by over 2.0 points across all types. In the dialogue task, BLEU scores significantly improve for New and Changed by 1.0 and 0.6 points, respectively, with little improvement in Unchanged.

heading: 6 Conclusion
text: To evaluate whether LLMs can adapt to the fast-evolving world knowledge, we propose GrowOVER-QA and GrowOVER-Dialogue. Our benchmarks annotate the evidence text and introduce dialogue tasks to evaluate retriever-augmented RaLM comprehensively. Furthermore, we suggest RiLM, an interactive retriever-generator framework by simply training a classifier for LLM to predict reliability itself. Through rigorous experiments, we show that our method can be on par with or surpass continuously pre-trained LLMs even without pre-training. However, even with retrieval, we observe that the performance degrades over time. Thus, we hope our benchmarks can be valuable resources to detect when to update the retriever or LLM in future work. Additionally, we anticipate further research into optimizing the use of retrievers to reduce the frequency of model updates.

heading: Limitations
text: We highlight a few considerations for readers regarding potential limitations. We rely on various models to label each sentence in Wikipedia snapshots. Although we thoroughly designed the sentence labeling process to label sentences accurately, it can occasionally be faulty. To address this limitation, we append the MTurk study in the Appendix, showing the results are within acceptable bounds. Also, our dataset is primarily built from knowledge based on single articles, which may restrict its effectiveness for tasks that necessitate combining information from multiple sources. Further research would be beneficial to generate benchmarks that enable the evaluation of frameworks handling complex reasoning tasks with a timesensitive nature. Moreover, the information extracted from Wikipedia may not promptly reflect real-world knowledge updates. This means that knowledge modifications in the real world might not be immediately mirrored in the dataset, resulting in some degree of outdated or inaccurate information. Finally, the characteristics of our dataset may be influenced by the features of GPT-4 that we utilized during its generation, as well as the prompts we used. To enhance the effectiveness and mitigate bias in datasets, further research should focus on generating datasets for a variety of using Large Language Models employed across diverse fields, aligned with a wide range of prompts.

heading: Ethics Statement
text: We have manually reevaluated the dataset we created to ensure it is free of any potential for discrimination, human rights violations, bias, exploitation, and any other ethical concerns.

heading: Acknowledgments
text: We sincerely thank Jaewoo Ahn, Soochan Lee, Yeda Song, Heeseung Yun, Junik Bae, and other anonymous reviewers for their valuable comments. This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2022-II220156, Fundamental research on continual meta-learning for quality en-
hancement of casual videos and their 3D metaverse transformation), the SNU-Global Excellence Research Center establishment project, the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. 2023R1A2C2005573), Basic Science Research Program through the National Research Foundation of Korea(NRF) funded by the Ministry of Education(RS-2023-00274280), and Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2021-II211343, Artificial Intelligence Graduate School Program (Seoul National University)).

heading: A Experimental Details
text: Dataset. For dataset generation, we use OpenAI gpt-4-1106-preview model and set the temperature as 0 and max_new_token as 256. Also, we use Huggingface princeton-nlp/sup-simcseroberta-large for NLI task in the sentence labeling process. Additionally, when checking contradiction with GPT-4, we use gpt-41106-preview and set the temperature as 0 and max_tokens as 256.
Database. We use the Langchain document loader (RecursiveCharacterTextSplitter) to split each article in the Wikipedia snapshot into several passages. We set chunk_size as 1500 and chunk_overlap as 10 characters.
Continual Pretraining of LLM. For initial training on August, we train Llama2 for four epochs with a learning rate of 1e-06, a learning rate decay of 0.8, a cosine learning rate scheduler, an AdamW optimizer, and a batch size of 64, using FSDP (fully sharded data parallel). After that, for the c-RaLM baseline, we continuously pre-train the model with 12K articles on each month. We set all hyperparameters the same with the initial training except the epoch. To prevent catastrophic forgetting, we trained the model for only one epoch, following Jang et al. (2022).
Classifier Training. For the classifier, we trained a linear layer with dimensions (4096, 3). In both QA and dialogue response generation tasks, we set the learning rate as 0.0001, weight decay as 1e-07, and 10 epochs. We use a cosine learning rate scheduler. Also, we train the model using cross-entropy loss.
Adaptive Re-Retrieval. In both QA and dialogue tasks with SentBERT, we set the hyperparameter λ as 2.0. In the other cases, we set the λ as 1.0.
Answer Generation. We set the max_new_token of Llama2 as 10 for QA and 50 for dialogue, and load the model using bfloat16.
Self-RAG. We used short-form generation version with always_retrieve mode, since all data points of GrowOver contain evidence texts. We used same retrievals as RiLM.

heading: B Additional Experimental Results

heading: B.1 Contriever
text: We additionally perform experiments with Contriever (Izacard et al., 2021), following Shi et al. (2023) and Izacard et al. (2022). We report classifier performance, ARR results, as well as end-to-end QA results.

heading: B.1.1 Classifier Performance
text: Table 8 shows the experimental results of the certainty classifier. Despite using the same certainty classifier as SentBERT, the accuracy of all cases is above 65.0, demonstrating its effectiveness. Also, across all months, F1 / BLEU scores for adopted and not-adopted answers show a significant gap.

heading: B.1.2 Adaptive Re-Retrieval results
text: As shown in Table 9, ARR significantly improves retrieval relevance. In the QA task, performance increased by more than 3 times, and in the dialogue task, it increased by more than 2 times.

heading: B.1.3 QA Results
text: The results of GrowOVER-QA are presented in Table 10. It demonstrates the robustness of our method. For New, Unchanged, and all types of QA, our RiLM shows the highest score
across all months. Also, for Changed, RiLM surpasses the other baselines except in September. In September, the performance degrades after re-retrieval, which indicates the DG possibly struggles with selecting correct answers between two different answers. Nonetheless, across all the other months, RiLM improves RaLM-D∗ about by 3.0. Also, it significantly outperforms other baselines.

heading: New

heading: C Algorithms
text: Algorithm 1 Initial Generation for GrowOVER-QA Require: WPcur := Wikipedia snapshots of the initial month P := An empty array to store valid paragraphs S := An empty array to store selected paragraphs Q := An empty array to store generated QA pairs ∗ article in WP has attributes id, title and text
for all article ar ∈ WPinit do for paragraph pr ∈ ar.text do
if pr is of adequate length then P .append(pr)
end if end for S ← ClusterParagraphs(P ) Q← Q + GenerateQA(S, New)
end for
function ClusterParagraphs(P ) Extract features and obtain embeddings of P using SimCSE Compute cluster assignment using KMeans algorithm S ← Randomly selected paragraphs from each cluster return S end function
function GenerateQA(S, type) // initialize QA as an empty array for selected text p ∈ S do
qa← generate QA pairs with p qa.type← type QA.append(qa)
end for return QA
end function
Algorithm 2 Initial Generation for GrowOVER-Dialogue Require: WPinit := Wikipedia snapshots at initial month P := An empty array to store splitted paragraphs D := An empty array to store generated Dialogue ∗ article in WP has attributes id, title and text
for article ar ∈ WPinit do P ← SplitArticleIntoParagraph(ar) for paragraph p ∈ P do
if p is informative paragraph then D.append(GenerateDialogue(p, ar .title))
end if end for
end for
function SplitArticleIntoParagraph(a) for paragraph p ∈ a.text do
P .append(split p into sentence) end for return P
end function
function GenerateDialogue(p, a.title) d← generate dialogue with p and a.title for turn t ∈ d do
t.type← NEW end for return d
end function
Algorithm 3 Sentence Labeling Require: sentences_old:= sentences in the old article, sentences_new:= sentences in the new article
▷ Identify unchanged sentence pairs for sold in sentences_old do
for snew in sentences_new do if sim(sold, snew) > thrs(=0.99) then
(sold, snew)← unchanged end if
end for end for ▷ Identify unchanged sentence groups for each group of sentences (Sold, Snew) enclosed by unchanged pairs do
for subold in P(Sold) do for subnew in P(Sold) do
if sim (concat(subold), concat(subnew)) > thrs(=0.99) then (subold, subnew)← unchanged
end if end for
end for end for ▷ NLI for changed & new for each group of sentences (Sold, Snew) enclosed by unchanged pairs do
for snew in Snew do preds← an empty list for sold in Sold do preds.append(NLI.classify(sold, snew))
end for if “entailment” in preds then
sold ← the entailed old sentence (sold, snew)← unchanged
else if “contradiction” in preds then sold ← the contradicted old sentence if sim (sold, snew) > τ1 (=0.6) then
if GPT-4.contradict(sold, snew) then (sold, snew)← changed
end if end if
else ▷ all elements in preds are “neutral” sim_res← [sim(sold, snew) for sold in Sold)] max_similarity← max(sim_res) if max_similarity < τ2 (=0.7) then
snew ← new end if
end if end for
end for
Algorithm 4 Temporal Updates for GrowOVER-QA Require: Wt := Wikipedia snapshots of month t QAt-1 := QAs of month t-1 QAt := An empty set for QAs of month t
for article at ∈ Wt do for at’s QA qa ∈ QAt-1 do
qa← UpdateQA(qa) if qa.type is Unchanged then
QAt.append(qa) end if
end for Gnew ← groups of new sentences in at for group g ∈ Gnew do
QAt.append(GenerateQA(g, New)) end for for changed sentence st ∈ at do
st-1 ← contradictory sentence in at-1 QAt.append(GenerateQA(st-1, st, Changed))
end for end for
function UpdateQA(qa) if all sentences ∈ qa.evid_text are unchanged then
qa.type← Unchanged qa.index← indices of qa.evid_text in Wt
else qa.type← Deleted end if return qa
end function
Algorithm 5 Temporal Updates for GrowOVERDialogue Require: Wt := Wikipedia snapshots on month t Dt-1 := Dialogues of month t-1 Dt := An empty set for dialogues of month t
for article at ∈ Wt do for at’s dialogue d ∈ Dt-1 do
d← UpdateDialog(d, Lt-1) Dt.append(d)
end for P ← SplitArticleIntoParagraph(at) for paragraph p ∈ P do
if p is not informative paragraph then continue else if changed or new sentence in p then Dt.append(GenerateDial(p, at.title))
end if end for
end for
function UpdateDialog(d) for turn t ∈ d do
if s is unchanged sentence then t.type← Unchanged t.index← the index of t.evid_text in WPcur else t.type← Deleted
end if end for return d end function

heading: D Mturk Study
text: We thoroughly designed the sentence labeling process to accurately label sentences (e.g., setting a high threshold for selecting semantically identical sentences). To further validate using sentence similarity scores and natural language inference, we employed Amazon Mechanical Turk (AMT) workers to assess the sentence labels during the rebuttal process.
We randomly sampled 30 new and 30 changed sentences and asked AMT workers to classify whether each sentence in the new article was supported, not supported, or uncertain given the old article. For new sentences, the labeling is incorrect when classified as “supported,” while for changed sentences, the labeling is correct when classified as “not supported.” Each sentence was evaluated by three workers, and the majority vote was used. The human quality check results are shown in the table 12.
Although new sentences have lower accuracy since they are not verified with GPT-4, the results are still within acceptable bounds. Evaluating the correct label by humans requires a review of the entire article, which is extremely time-consuming. This becomes critical, especially for our dataset, which requires regular updates to reflect the ever-changing knowledge of the real world. Therefore, as evidenced by the human quality check results, our fully automated sentence labeling alone can efficiently provide reasonably accurate labels with no human effort at all.

heading: E Data Analysis

heading: F Dataset Examples
text: Table 15 shows the examples of QA. Table 16 and 17 show the examples of Dialogue.

heading: G Case Study
text: Table 18 and Table 19 present a case study from GrowOVER-QA. Predictions denote the answers generated by each retriever-generator framework. Table 20 presents a case study from GrowOVER-Dialogue.

heading: H Prompt examples
text: Table 21 shows the prompt used for the initial generation of the QA pair. Table 22 and 23 show the prompts used to generate New QA pairs, without and with Source Content respectively. Table 24 shows the prompt used to generate Changed QA pairs. Table 25 shows the prompt used for Dialogue generation.
Article : Politics of Cambodia Type : Changed Question: Who is the current prime minister of Cambodia from the Cambodian People’s Party (CPP)? Answer: Hun Manet Previous Answer: Hun Sen Evidence Text: The current prime minister is Cambodian People’s Party (CPP) member Hun Manet. Evidence Index: 80 Article : The Eras Tour Type : New Question: Who is directing the concert film Taylor Swift: The Eras Tour? Answer: Sam Wrench Evidence Text: On August 31, 2023, Swift announced the concert film Taylor Swift: The Eras Tour, directed by Sam Wrench. Recorded at SoFi Stadium in Los Angeles, the film is scheduled for release to theaters in North America on October 13. Evidence Index: 263 Article : Airline Type : Unchanged Question: Which airport would most Manhattan travelers prefer for its proximity? Answer: LaGuardia Airport Evidence Text: For example, LaGuardia Airport is the preferred airport for most of Manhattan due to its proximity, while long-distance routes must use John F. Kennedy International Airport’s longer runways. Evidence Index: 297
Table 15: An Example of QA
Article : 2030 Winter Olympics

heading: Paragraph :
text: 20A decision on the 2030 Winter Olympics host city has been delayed until 23 July 2024, to allow the IOC more time to carefully plan the future of the Winter Olympics. 21Due to impacts from climate change, the IOC is considering a number of changes to future games, such as rotating host cities, limiting the games to existing or temporary venues, and establishing minimum average temperature requirements. 22It also considered awarding the 2030 and 2034 winter games simultaneously at the next IOC session, but IOC President Thomas Bach has ruled out that possibility because "it would not be the right thing to do". 23However, at the 141st IOC Session in Mumbai, where these games were originally set to be awarded, the IOC approved to award both the 2030 and 2034 Winter Olympics at the 142nd IOC Session in Paris on the eve of the 2024 Summer Olympics.

heading: Dialogue :
text: Type: Unchanged User: I was looking forward to finding out which city will host the 2030 Winter Olympics, but I heard there’s been a delay. What’s going on with that? Expert: Yes, that’s correct. The decision has been postponed until 23 July 2024. The International Olympic Committee (IOC) wants to take more time to plan the future of the Winter Games with careful consideration. Evidence Index: 20 Type: Unchanged User: I guess the delay must be frustrating for the candidate cities. I’ve heard that the changing climate is causing some issues? Expert: Indeed, the changing climate is a significant concern. The IOC is exploring various adjustments for future games, including the possibility of having rotating host cities and using existing or temporary venues to mitigate the impact. Evidence Index: 21 Type: Unchanged User: I also caught a rumor that they might announce the hosts for both 2030 and 2034 at the same time. Is that true? Expert: That was a consideration at one point. However, IOC President Thomas Bach has stated that awarding both the 2030 and 2034 Winter Olympics simultaneously wouldn’t be appropriate. Evidence Index: 22 Type: New User: So, when can we expect the announcement for the 2030 host city? Expert: The announcement is set to be made at the 142nd IOC Session in Paris, which will happen right before the 2024 Summer Olympics. They will also award the 2034 Winter Olympics at that session. Evidence Index: 23
Table 16: An Example of Dialogue
Article : Tornadoes of 2023

heading: Paragraph :
text: 357More rounds of severe storms occurred on June 17, including a powerful MCS that pushed through Northern and Central Oklahoma. 358Both the Oklahoma City and Tulsa metropolitan areas had widespread wind damage with the latter metropolitan area seeing wind gusts of up to and multiple weak tornadoes. 359On June 18, more tornadoes were confirmed in Florida and Mississippi, including an EF3 tornado which caused significant damage in Louin, Mississippi, where at least one fatality and 20 injuries occurred. 360Scattered tornadoes occurred across the Southeast on June 19 as well, including an EF2 tornado that injured six people in Moss Point, Mississippi. 361Overall, 87 tornadoes were confirmed during this outbreak sequence.

heading: Dialogue :
text: Type: Unchanged User: I heard there were some intense storms in Oklahoma last June. What happened exactly? Expert: Indeed, on June 17, Oklahoma experienced severe storms, including a particularly strong Mesoscale Convective System that affected the northern and central parts of the state. Evidence Index: 357 Type: Unchanged User: That sounds serious. Did the storms cause a lot of damage in the area? Expert: Yes, the storms led to extensive wind damage, especially in the Oklahoma City and Tulsa areas. Tulsa even experienced wind gusts of up to 80 mph and several weak tornadoes. Evidence Index: 358 Type: Unchanged User: I also caught wind of some tornado activity in other states around that time. Expert: That’s correct. The following day, on June 18, Florida and Mississippi reported tornadoes. Mississippi was hit by an EF3 tornado in Louin, which resulted in one death and 20 injuries. Evidence Index: 359 Type: Unchanged User: Was that the end of it, or did the tornadoes continue? Expert: The tornadoes continued. On June 19, the Southeast saw more, including an EF2 tornado in Moss Point, Mississippi, which injured six people. Evidence Index: 360 Type: Changed User: It must have been a large outbreak to affect so many areas. Expert: It was indeed a significant outbreak. In total, there were 87 confirmed tornadoes during that sequence of storms. Evidence Index: 361
Table 17: Another Example of Dialogue
Article : Football player Type : Unchanged Question: What was the average annual salary for goalkeepers in Major League Soccer during the 2013 season? Answer: $85,296 Evidence Text: For example, the average annual salary for footballers in Major League Soccer for the 2013 season was $148,693, with significant variations depending on the player position (goalkeepers for example earned $85,296, whereas forwards earned $251,805.

heading: Predictions :
text: Vanilla: $110,000 RaLM: $148,693 RaLM-CP: $148,693 RaLM-D∗ (Ours): $85,296 RiLM (Ours): $85,296 Retrieved Documents : [Top-3] (1) Hit: However, only a fraction of men’s professional football players is paid at this level. ... For example, the average annual salary for footballers in Major League Soccer for the 2013 season was $148,693, with significant variations depending on the player position (goalkeepers for example earned $85,296, whereas forwards earned $251,805. Popularity and average salaries in women’s leagues are far lower. For example, players in ... (2) Miss: MLS has a set of pool goalkeepers who are signed to a contract with the league and are loaned to teams during emergencies in which they are missing a goalkeeper due to injuries or suspensions. ... These initiatives have brought about an increase in on-field competition. (3) Miss: According to "France Football", Messi was the world’s highest-paid footballer for five years out of six between 2009 and 2014; ... In 2020, Messi became the second footballer, as well as the second athlete in a team sport, after Cristiano Ronaldo, to surpass $1 billion in earnings during their careers.
[RaLM-D∗] Hit: However, only a fraction of men’s professional football players is paid at this level. ... For example, the average annual salary for footballers in Major League Soccer for the 2013 season was $148,693, with significant variations depending on the player position (goalkeepers for example earned $85,296, whereas forwards earned $251,805. Popularity and average salaries in women’s leagues are far lower. For example, players in ...
[RiLM] Hit: However, only a fraction of men’s professional football players is paid at this level. ... For example, the average annual salary for footballers in Major League Soccer for the 2013 season was $148,693, with significant variations depending on the player position (goalkeepers for example earned $85,296, whereas forwards earned $251,805. Popularity and average salaries in women’s leagues are far lower. For example, players in ... Article : Benjamin Netanyahu Type : New Question: What city was Benjamin Netanyahu born in? Answer: Tel Aviv Evidence Text: Netanyahu was born in Tel Aviv, to Benzion Netanyahu (original name Mileikowsky) and Tzila (Cela;

heading: Predictions :
text: Vanilla: Jerusalem, Israel RaLM: Tel Aviv RaLM-CP: Tel Aviv RaLM-D∗ (Ours): Tel Aviv RiLM (Ours): Tel Aviv Retrieved Documents : [Top-3] (1) Miss: Netanyahu was the second of three children. He was initially raised and educated in Jerusalem, where he attended ... the liberal sensibilities of the Reform synagogue, Temple Judea of Philadelphia, that the family attended. (2) Hit: Netanyahu was born in Tel Aviv, to Benzion Netanyahu (original name Mileikowsky) and Tzila (Cela; née Segal). His mother was born in 1912 in Petah Tikva, then in Ottoman Palestine, now Israel. Though all his grandparents were born in ... (3) Miss: Netanyahu made his closeness to Donald Trump, a personal friend since the 1980s, central to his political appeal in Israel from 2016. ... He claims descent from the Vilna Gaon.
[RaLM-D∗] Hit: Netanyahu was born in Tel Aviv, to Benzion Netanyahu (original name Mileikowsky) and Tzila (Cela; née Segal). His mother was born in 1912 in Petah Tikva, then in Ottoman Palestine, now Israel. Though all his grandparents were born in ...
[RiLM] Hit: Netanyahu was born in Tel Aviv, to Benzion Netanyahu (original name Mileikowsky) and Tzila (Cela; née Segal). His mother was born in 1912 in Petah Tikva, then in Ottoman Palestine, now Israel. Though all his grandparents were born in ...
Table 18: Case Study for QA
Article : Kyrylo Budanov Type : Changed Question: What is Kyrylo Budanov’s military rank? Answer: Lieutenant general Evidence Text: He holds the rank of lieutenant general.

heading: Predictions :
text: Vanilla: Kyrylo Budanov is a Major General in the Ukrainian Armed Forces. RaLM: Lieutenant General RaLM-CP: Lieutenant General RaLM-D∗ (Ours): Lieutenant General RiLM (Ours): Lieutenant General Retrieved Documents : [Top-3] (1) Hit: Kyrylo Oleksiyovych Budanov (; born 4 January 1986) is a Ukrainian military leader who is the chief of the ... Budanov previously served as the Deputy Director of one of the Departments of the Foreign Intelligence Service of Ukraine. He holds the rank of lieutenant general. ... as head of the Main Intelligence Directorate of the Ministry of Defense. (2) Miss: The Austrian military has a wide variety of equipment. Recently, Austria has spent considerable amounts of money modernizing its military arsenal. ... (3) Miss: Soon after the start of the German invasion of the Soviet Union, he was soon re-drafted into the Red Army on 4 July 1941 and initially deployed to the front as part of the 50th Cavalry Regiment. ...
[RaLM-D∗] Hit: Kyrylo Oleksiyovych Budanov (; born 4 January 1986) is a Ukrainian military leader who is the chief of the ... Budanov previously served as the Deputy Director of one of the Departments of the Foreign Intelligence Service of Ukraine. He holds the rank of lieutenant general. ... as head of the Main Intelligence Directorate of the Ministry of Defense.
[RiLM] Hit: Kyrylo Oleksiyovych Budanov (; born 4 January 1986) is a Ukrainian military leader who is the chief of the ... Budanov previously served as the Deputy Director of one of the Departments of the Foreign Intelligence Service of Ukraine. He holds the rank of lieutenant general. ... as head of the Main Intelligence Directorate of the Ministry of Defense. Article : Darwin, Northern Territory Type : Unchanged Question: What is the name of the passenger train service that connects Darwin to Adelaide? Answer: The Ghan Evidence Text: The first service ran in 2004. "The Ghan" passenger train service from Adelaide via Alice Springs and Katherine runs once per week in each direction, with some exceptions.

heading: Predictions :
text: Vanilla: The Ghan RaLM: The Ghan RaLM-CP: The Ghan RaLM-D∗ (Ours): The Ghan RiLM (Ours): The Ghan Retrieved Documents : [Top-3] (1) Hit: The Territory’s public transport services are managed by the Department of Lands and Planning, ... The first service ran in 2004. "The Ghan" passenger train service from Adelaide via Alice Springs and Katherine runs once per week in each direction, with some exceptions. Historically, ... (2) Miss: Darwin and its suburbs, Palmerston and Katherine, are powered by the Channel Island Power Station, the Northern Territory’s largest power plant, and the ... (3) Miss: The platform layout at Flinders Street is unusual among Australian terminal stations for being almost entirely composed of through tracksa product ...
[RaLM-D∗] Hit: The Territory’s public transport services are managed by the Department of Lands and Planning, ... The first service ran in 2004. "The Ghan" passenger train service from Adelaide via Alice Springs and Katherine runs once per week in each direction, with some exceptions. Historically, ...
[RiLM] Hit: The Territory’s public transport services are managed by the Department of Lands and Planning, ... The first service ran in 2004. "The Ghan" passenger train service from Adelaide via Alice Springs and Katherine runs once per week in each direction, with some exceptions. Historically, ...
Table 19: Another Case Study for QA
Article : Marshall Islands Turn 1 Type : Changed User: I’ve heard that the Marshall Islands were inhabited for a very long time. What kind of influence did the early settlers have on these islands? Expert: Indeed, the islands have been inhabited since the 2nd millennium BC. The early Austronesian settlers brought with them Southeast Asian crops like coconuts and breadfruit, as well as domesticated chickens, which were crucial for making the islands habitable. Evidence Text: Austronesian settlers reached the Marshall Islands as early as the 2nd millennium BC and introduced Southeast Asian crops, including coconuts, giant swamp taro, and breadfruit, as well as domesticated chickens which made the islands permanently habitable.

heading: Predictions :
text: Vanilla: The Marshall Islands were inhabited for a very long time. The early settlers had a big influence on the islands. They built houses, farmed, and fished. They also made tools and weapons. The early settlers also had a big influence on the culture of the Marshall Islands. They taught the people how to make clothes, how to cook food, and how to make music. The early settlers also taught the people how to make weapons and how to fight. The early settlers RaLM: The early settlers of the Marshall Islands introduced Southeast Asian crops, including coconuts, giant swamp taro, and breadfruit, as well as domesticated chickens which made the islands permanently habitable. They developed a complex technique for interisland piloting by reading disruptions in ocean swells caused by low-lying coral atolls. RaLM-CP: The early settlers of the Marshall Islands introduced Southeast Asian crops, including coconuts, giant swamp taro, and breadfruit, as well as domesticated chickens which made the islands permanently habitable. They developed a complex technique for interisland piloting by reading disruptions in ocean swells caused by low-lying coral atolls. RaLM-D∗ (Ours): The early settlers introduced Southeast Asian crops, including coconuts, giant swamp taro, and breadfruit, as well as domesticated chickens which made the islands permanently habitable. RiLM (Ours): The early settlers introduced Southeast Asian crops, including coconuts, giant swamp taro, and breadfruit, as well as domesticated chickens which made the islands permanently habitable. Retrieved Documents : [Top-3] (1) Miss: The majority of the citizens of the Republic of Marshall Islands are of Marshallese descent, ... around the 1st century CE at the village of Laura on Majuro and on Kwajalein Atoll. (2) Hit: Austronesian settlers reached the Marshall Islands as early as the 2nd millennium BC and introduced Southeast Asian crops, including coconuts, giant swamp taro, and breadfruit, as well as domesticated chickens which made the islands permanently habitable. They developed a complex technique for ... (3) Miss: The British sea captains John Marshall and Thomas Gilbert visited the islands in 1788. Their vessels had been part of the First Fleet taking convicts from England to Botany Bay ...
[RaLM-D∗] Hit: Austronesian settlers reached the Marshall Islands as early as the 2nd millennium BC and introduced Southeast Asian crops, including coconuts, giant swamp taro, and breadfruit, as well as domesticated chickens which made the islands permanently habitable. They developed a complex technique for ...
[RiLM] Hit: Austronesian settlers reached the Marshall Islands as early as the 2nd millennium BC and introduced Southeast Asian crops, including coconuts, giant swamp taro, and breadfruit, as well as domesticated chickens which made the islands permanently habitable. They developed a complex technique for ... Turn 2 Type : New User: That’s fascinating. I wonder how they navigated between the islands back then. Expert: They actually developed a sophisticated method for navigating the ocean. They could read the disruptions in ocean swells, which were caused by the coral atolls, to pilot between the islands. Evidence Text: They developed a complex technique for interisland piloting by reading disruptions in ocean swells caused by low-lying coral atolls.

heading: Predictions :
text: Vanilla: The early settlers developed a complex technique for interisland piloting by reading disruptions in ocean swells caused by low-lying coral atolls. RaLM: The early settlers developed a complex technique for interisland piloting by reading disruptions in ocean swells caused by low-lying coral atolls. RaLM-CP: The early settlers developed a complex technique for interisland piloting by reading disruptions in ocean swells caused by low-lying coral atolls. RaLM-D∗ (Ours): The early settlers developed a complex technique for interisland piloting by reading disruptions in ocean swells caused by low-lying coral atolls. RiLM (Ours): The early settlers developed a complex technique for interisland piloting by reading disruptions in ocean swells caused by low-lying coral atolls.
Retrieved Documents : [Top-3] (1) Hit: ... which made the islands permanently habitable. They developed a complex technique for interisland piloting by reading disruptions in ocean swells caused by low-lying coral atolls. In 1526, Spanish explorer Alonso de Salazar ... (2) Miss: The majority of the citizens of the Republic of Marshall Islands are of Marshallese descent, ... around the 1st century CE at the village of Laura on Majuro and on Kwajalein Atoll. (3) Miss: The Austronesian settlers introduced Southeast Asian crops, including coconuts, giant swamp taro, and breadfruit, as well as domesticated chickens throughout the Marshall Islands. ...
[RaLM-D∗] Hit: ... which made the islands permanently habitable. They developed a complex technique for interisland piloting by reading disruptions in ocean swells caused by low-lying coral atolls. In 1526, Spanish explorer Alonso de Salazar ...
[RiLM] Hit: ... which made the islands permanently habitable. They developed a complex technique for interisland piloting by reading disruptions in ocean swells caused by low-lying coral atolls. In 1526, Spanish explorer Alonso de Salazar ... Turn 3 Type : Unchanged User: I guess that means they were pretty isolated until Europeans arrived. Expert: That’s correct. The first European who might have sighted the islands was Spanish explorer Alonso de Salazar in 1526, and the first confirmed European landing was by Álvaro de Saavedra Cerón two years later on an uninhabited island. Evidence Text: In 1526, Spanish explorer Alonso de Salazar may have been the first European to sight the islands, and Álvaro de Saavedra Cerón made landfall on an uninhabited island two years later.

heading: Predictions :
text: Vanilla: That’s correct. The Marshall Islands were not discovered by Europeans until 1529, when Spanish explorer Alvaro de Mendana de Neira arrived. He named the islands after his patron, the Spanish admiral Álvaro de Mendaña de Neyra. RaLM: That’s right. The islands were largely untouched by European explorers until the late 16th century. RaLM-CP: That’s right. The islands were largely untouched by European explorers until the late 16th century. RaLM-D∗ (Ours): That’s correct. The islands were largely untouched by European explorers until the 18th century. RiLM (Ours): That’s correct. The islands were largely untouched by European explorers until the 18th century. Retrieved Documents : [Top-3] (1) Hit: ... by low-lying coral atolls. In 1526, Spanish explorer Alonso de Salazar may have been the first European to sight the islands, and Álvaro de Saavedra Cerón made landfall on an uninhabited island two years later. Several other Spanish ... (2) Miss: The majority of the citizens of the Republic of Marshall Islands are of Marshallese descent, ... around the 1st century CE at the village of Laura on Majuro and on Kwajalein Atoll. (3) Miss: The British sea captains John Marshall and Thomas Gilbert visited the islands in 1788. Their vessels had been part of the First Fleet taking convicts from England to Botany Bay in New South Wales, ...
[RaLM-D∗] Hit: ... by low-lying coral atolls. In 1526, Spanish explorer Alonso de Salazar may have been the first European to sight the islands, and Álvaro de Saavedra Cerón made landfall on an uninhabited island two years later. Several other Spanish ...
[RiLM] Hit: ... by low-lying coral atolls. In 1526, Spanish explorer Alonso de Salazar may have been the first European to sight the islands, and Álvaro de Saavedra Cerón made landfall on an uninhabited island two years later. Several other Spanish ...
Turn 4 Type : Unchanged User: And how did the Marshall Islands get their name? Expert: The name comes from the British captains John Marshall and Thomas Gilbert who landed on Mili Atoll in 1788. It was the first recorded European presence in over 200 years, and subsequent maps began to refer to the islands by Captain Marshall’s name. Evidence Text: Later maps and charts named the islands after Captain Marshall.

heading: Predictions :
text: Vanilla: The islands were named after British explorer John Marshall, who was the first European to map them in 1788. RaLM: The islands were named after Captain John Marshall, who was the first European to sight them. RaLM-CP: The islands were named after Captain John Marshall, who was the first European to sight them. RaLM-D∗ (Ours): The islands were named after Captain John Marshall, who was the first European to map the islands in 1788. RiLM (Ours): The islands were named after Captain John Marshall, who was the first European to map the islands in 1788. Retrieved Documents : [Top-3] (1) Hit: ... in the archipelago in over 200 years. Later maps and charts named the islands after Captain Marshall. ... (2) Miss: The majority of the citizens of the Republic of Marshall Islands are of Marshallese descent, ... around the 1st century CE at the village of Laura on Majuro and on Kwajalein Atoll. (3) Miss: The British sea captains John Marshall and Thomas Gilbert visited the islands in 1788. Their vessels had been part of the First Fleet taking convicts from England to Botany Bay in New South Wales, ...
[RaLM-D∗] Hit: ... in the archipelago in over 200 years. Later maps and charts named the islands after Captain Marshall. ...
[RiLM] Hit: ... in the archipelago in over 200 years. Later maps and charts named the islands after Captain Marshall. ...
Table 20: Case Study for Dialogue
Generate a Q&A pair based on a given context, where the context is understood but NOT DIRECTLY VISIBLE to the person answering the question. Assume the person answering the question has common sense and is aware of the details and key points in the paragraph, but the paragraph itself is not quoted or referenced directly.
Paragraph (a list of sentences): {paragraph}
Use the following instructions for generating a Q&A pair: 1) Provide a question, an answer, and a bounding box. 2) DON’T use phrases such as ‘according to the paragraph’ in your question. 3) An answer should be an entity or entities. Provide a SHORT ANSWER. 4) The bounding box for a paragraph is defined as (starting sentence index, ending sentence index): the bounding box should be sufficiently large to encompass all the information necessary for a reader to FULLY infer the answer to the question. 5) The sentence index starts from 0. 6) Generate a SINGLE Q&A pair.
Be sure to follow the following format and write your answer within curly brackets. The format is as follows: {Question}{Answer}{starting sentence index}{ending sentence index}
Table 21: Sample prompt for initial generation of a QA pair

heading: Future Work
text: The International Olympic Committee (IOC) wants to take more time to plan the future of the Winter Games with careful consideration. Evidence Index: 20 Type: Unchanged User: I guess the delay must be frustrating for the candidate cities. I’ve heard that the changing climate is causing some issues? Expert: Indeed, the changing climate is a significant concern. The IOC is exploring various adjustments for future games, including the possibility of having rotating host cities and using existing or temporary venues to mitigate the impact. Evidence Index: 21 Type: Unchanged User: I also caught a rumor that they might announce the hosts for both 2030 and 2034 at the same time. Is that true? Expert: That was a consideration at one point. However, IOC President Thomas Bach has stated that awarding both the 2030 and 2034 Winter Olympics simultaneously wouldn’t be appropriate. Evidence Index: 22 Type: New User: So, when can we expect the announcement for the 2030 host city? Expert: The announcement is set to be made at the 142nd IOC Session in Paris, which will happen right before the 2024 Summer Olympics. They will also award the 2034 Winter Olympics at that session. Evidence Index: 23
Table 16: An Example of Dialogue
Article : Tornadoes of 2023