ChroKnowledge: Unveiling Chronological Knowledge of Language Models in Multiple Domains

## Abstract

Abstract Large language models (LLMs) have brought significant changes to many aspects of our lives.
However, assessing and ensuring their chronological knowledge remains challenging.
Existing approaches fall short in addressing the accumulative nature of knowledge, often relying on a single time stamp.
To overcome this, we introduce ChroKnowBench , a benchmark dataset designed to evaluate chronologically accumulated knowledge across three key aspects: multiple domains, time dependency, temporal state.
Our benchmark distinguishes between knowledge that evolves (e.g., scientific discoveries, amended laws) and knowledge that remain constant (e.g., mathematical truths, commonsense facts).
Building on this benchmark, we present ChroKnowledge (Chronological Categorization of Knowledge), a novel sampling-based framework for evaluating and updating LLMs’ non-parametric chronological knowledge.
Our evaluation led to the following observations:
(1) The ability of eliciting temporal knowledge varies depending on the data format that model was trained on.
(2) LLMs partially recall knowledge or show a cut-off at temporal boundaries rather than recalling all aspects of knowledge correctly.
Thus, we apply our ChroKnowPrompt , an in-depth prompting to elicit chronological knowledge by traversing step-by-step through the surrounding time spans.
We observe that our framework successfully updates the overall knowledge across the entire timeline in both the biomedical domain (+11.9%) and the general domain (+2.8%), highlighting its positive effect in refining temporal knowledge.
This non-parametric approach also enables knowledge updates not only in open-source models but also in proprietary LLMs, ensuring comprehensive applicability across model types.
We perform a comprehensive analysis based on temporal characteristics of ChroKnowPrompt and validate the potential of various models to elicit intrinsic temporal knowledge through our method. 1 1 1 Our datasets and code are publicly available at https://github.com/dmis-lab/ChroKnowledge

## 1 Introduction

Do large language models (LLMs) possess the ability to understand and track the history of knowledge as time progresses?
In other words, can these models, which represent the cutting edge of modern artificial intelligence, reason appropriately about questions that involve evolving facts?
Although some details remain controversial, knowledge—like science—is built upon accumulation (Zeigler, ; Picho et al., ).
From raw data to information and to knowledge, every bit is cumulative which contributes to progress across all domains.
This accumulation forms the foundation for higher-level reasoning, which is akin to wisdom in navigating the complexities of our world (Rowley, ).
Given that LLMs are trained on vast and diverse corpora and are now integral to numerous applications in our daily lives, they must remain accurate and up-to-date to ensure reliability.
Early versions of ChatGPT (OpenAI, ), for instance, sometimes produced inaccurate or absurd responses like the infamous example of “The happening of King Sejong (1397-1450) throwing MacBook (2016-)”(^2^22[https://english.hani.co.kr/arti/english$_$edition/e$_$international/1095956](https://english.hani.co.kr/arti/english_edition/e_international/1095956)).
These errors still give us a lesson that we need more precise model recalling knowledge correctly.

Figure: Figure 1: The overview of time variant dataset generation in ChroKnowBench. We accumulate knowledge in three key aspects: (1) multiple domains: general, biomedical, legal, commonsense, and mathematics; (2) time dependency: as time goes by, changeable knowledge; (3) temporal state: dynamic (has evolved over period) and static (no change occurred during period).
Refer to caption: x1.png

When we examine the issue closely, it’s not just a matter of hallucination but also about whether the alignment of knowledge, particularly regarding dates, is accurate.
Ensuring that LLMs maintain current and contextually relevant knowledge over time is crucial and researchers have explored various ways to investigate and verify the knowledge within these models (Zhang et al., ).
Pioneering works investigate whether language model has an ability of knowledge base or not in diverse domains (Petroni et al., ; Sung et al., ).
Many subsequent studies analyze how LLMs define knowledge (Dai et al., ; Mishra et al., ); exploit how LLMs represent their knowledge (Geva et al., ; Zheng et al., ) with temporal context (Kasai et al., ; Fatemi et al., );
edit the misleading aspects of knowledge (Manakul et al., ; Wang et al., ).

Here, we raise a question: “Do these methods sufficiently address the accumulative nature of knowledge?”.
Current temporal-related approaches for evaluating and updating LLMs often focus on single time stamps, struggling to address the accumulative characteristics (Jang et al., ; Ge et al., ), which are especially important in specialized domains such as scientific discoveries and amended laws.
This limitation can lead to outdated or incomplete information, undermining the models’ effectiveness and safety.
Addressing these challenges requires a comprehensive approach is needed—one that accounts for both the temporal evolution and the accumulative nature of knowledge.

Thus, we introduce ChroKnowBench, a benchmark dataset designed to evaluate chronologically accumulated knowledge across three dimensions: time dependency (time variant and time invariant), multiple domains (general, biomedical etc.), and temporal state (dynamic and static).
ChroKnowBench differentiates between knowledge that is subject to evolution (e.g., transfer situation of a soccer player, scientific discoveries, and amended legal regulations) and knowledge that remains invariant (e.g., mathematical truths and commonsense facts).
We then classify domains based on whether they are influenced by the flow of time, considering the domain specificity.
Finally, we set the time frame to categorize the knowledge as either changeable or steady (Section 3).

Building on this benchmark, we also present ChroKnowledge (Chronological Categorization of Knowledge), a novel framework for assessing and enhancing the non-parametric chronological knowledge of LLMs.
So, we start from analyzing how current open-source and proprietary LLMs work.
As we expected, the time invariant knowledge shows steady in all time frame.
However, the domain-specific characteristics significantly influence the representation of temporal knowledge from LLMs.
More stable domains exhibit consistent performance, while more variable domains show more fluctuations.
These observations highlight that we need a comprehensive approach to enhance representing temporal knowledge from LLMs (Section 5).

To this end, our ChroKnowPrompt approach utilizes an in-depth chronological prompting strategy that traverses and updates knowledge across adjacent time spans, effectively addressing issues of partial recall and temporal boundaries (Section 6).
In knowledge recall, our evaluation reveals significant improvements in the biomedical domain (+11.9%) and modest gains in the general domain (+2.8%).
Our non-parametric approach allows for direct updates to both proprietary and open-source LLMs without extensive retraining, highlighting its practicality and broad applicability (Section 7).
Our work emphasizes the importance of temporal context in eliciting LLMs knowledge, introducing a novel framework for updating their chronological knowledge.
We perform a comprehensive analysis based on temporal characteristics of ChroKnowPrompt and validate the potential of various models to elicit intrinsic temporal knowledge through our method.

**Table 1: Knowledge categorization with a temporal component. We classify responses into Correct, Partial Correct, and Incorrect to specify eliciting predictions in diverse way by comparing them with the answer set $A$. We use a temperature set $\mathcal{T}\in{0,0.7}$ to capture variations in prediction, where $\mathcal{T}$ includes both greedy decoding and temperature sampling. We set $n$ as 5, meaning that we evaluate using five distinct combinations of few-shot exemplars to ensure the robust assessment.**
| Category | Definition | Description |
| --- | --- | --- |
| Correct | $\{\hat{o_{i}}~{}|~{}M(D_{i},s,r,t)=\hat{o_{i}};M,\tau=0\}_{i=1}^{n}\subseteq A$ | All objects generated with greedy decoding are entirely<br>included within the answer set. |
| Partial Correct | $\bigcup\limits_{\tau\in\mathcal{T}}\left\{\hat{o_{i}}\mid M(D_{i},s,r,t)=\hat{<br>o_{i}};M,\tau\right\}_{i=1}^{n}\cap A\neq\emptyset$ | At least one generated object from greedy decoding or<br>temperature sampling is in the answer set. |
| Incorrect | $\bigcup\limits_{\tau\in\mathcal{T}}\left\{\hat{o_{i}}\mid M(D_{i},s,r,t)=\hat{<br>o_{i}};M,\tau\right\}_{i=1}^{n}\cap A=\emptyset$ | None of the generated objects, either from greedy decoding<br>or temperature sampling, are included in the answer set. |

## 2 Preliminaries

### 2.1 Knowledge Categorization with a Temporal Component

To distinguish and evaluate the knowledge levels of language models, we utilize the Sampling-based Knowledge Categorization (SliCK) framework (Gekhman et al., ).
This approach starts by sampling the model $M$’s answers to questions using various few-shot exemplar sets $D$.
The sampling is conducted under two temperature conditions: $\tau=0$ and $\tau>0$.
Then, it categorizes the degree to which the model knows each piece of knowledge into four levels: HighlyKnown, MaybeKnown, WeaklyKnown, and Unknown.

Based on Gekhman et al. (), we make the following modifications as follows:
(1) We append a temporal component $t$ to the conventional $\{subject~{}(s),relation~{}(r),object~{}(o)\}$ triplet structure, allowing us to evaluate the model’s knowledge across different time stamps;
(2) We merge the two categories (MaybeKnown and WeaklyKnown) that represent recallable knowledge states(^3^33We refer a recallable knowledge as the presence of at least one answer in the answer set, generated using either the greedy decoding or the temperature sampling method.) into a single category (Partial Correct);
(3) By using time attribute, we also renamed the HighlyKnown and Unknown to the Correct and Incorrect, respectively.
Our detailed definitions and descriptions are provided in Table 1.
Although this setting allows us to categorize the model’s sampled responses more precisely regarding time attribute, it only captures the model’s knowledge at specific time points $t$, limiting our ability to observe changes over time.
We address this limitation in Section 6.

### 2.2 Eliciting Knowledge using diverse Templates

Since models prefer different formats when eliciting their knowledge, it is important to use varied approaches to accurately assess their understanding ().
While we initially evaluate the model’s knowledge using a standard triplet format, relying on a single template may not sufficiently capture the full extent of the model’s knowledge.
Thus, following Hendrycks et al. (), we also employ a well-known format, multiple-choice question answering (MCQA), to elicit the model’s knowledge more effectively.
As a result, we propose two templates for measuring how much knowledge the model holds: triplets (hereafter referred to as Generation) and MCQA.
Each template is designed with appropriate few-shot exemplars and corresponding matching rules.
For example, in Generation, due to the complexity of evaluating responses, we apply fuzzy matching techniques to compare the generated responses against predefined labels.
See Appendix A.1 for further details of few-shot exemplars, fuzzy matching rules, and examples of two templates.

## 3 ChroKnowBench: Constructing a Benchmark Dataset

In this section, we enumerate the details of constructing ChroKnowBench, a chronologically accumulated knowledge benchmark dataset.
The ChroKnowBench dataset encompasses three key aspects: time dependency (time variant and invariant), multiple domains (general, biomedical, legal, commonsense, and math), and temporal state (dynamic and static).
We first categorize knowledge into two groups: knowledge that remains unchanged over time (time invariant) and knowledge that changes over time (time variant).
Additionally, we classify domains based on whether they are influenced by the flow of time, considering the specificity of each domain.
Finally, we categorize knowledge as either changeable (dynamic) or steady (static) within the time frame we have set.

### 3.1 Task definition

Our primary focus is a time variant knowledge across three domains (general, biomedical, and legal) with comparisons to time invariant knowledge across two domains (commonsense and mathematics).
What knowledge would be the difference between time variant and invariant?
The time variant knowledge has a specific object changing across a stream of time.
For example, “Cristiano Ronaldo (s) was a member of sports team of (r) Manchester United F.C. ($o_{1}$) in 2009 ($t_{1}$) and Real Madrid CF ($o_{k}$) in 2018 ($t_{k}$)”.
Likewise, we identify subject and object alias for each relation, then gather yearly changed objects.
After accumulating object lists {$o_{1},o_{2}$, …, $o_{m}$}, we de-duplicate and fill out the missing data in specific years based on available data; objects between Manchester United F.C. ($o_{1}$) in 2009 ($t_{1}$) and Real Madrid CF ($o_{k}$) in 2018 ($t_{k}$), missing objects between 2010 ($t_{2}$) to 2017 ($t_{k-1}$) filled with existing object of 2009 ($t_{1}$).
The statistic of ChroKnowBench dataset is in Table 2.

### 3.2 Dataset Generation

To construct dataset, we select annual knowledge sources for each domain, possible to be aligned with each elements even though the corpus does not specifically mention about time stamp.
For sources with structured triplets, we identify temporal affect-able relations that typically change over time, such as “position held”.
As time variant knowledge refers to the knowledge that has the potential to change over time, we divide it into two temporal states for more fine grained results:
(1) dynamic, where the knowledge has evolved over the accumulated period.
(2) static, where no change occurred during the accumulated period though it has potential to be changed.
Following the methodology outlined in Section 3.1, we track changes in objects to build the dynamic dataset, employing normalization and de-duplication for verification.
Each object is checked with strict exact string match, then add into objects pool.
Simultaneously, we identified unchanged objects over the same time frame to construct the static dataset.
At the end, all data elements consist with an associated object pool ${\{o_{1},o_{2},\ldots,o_{m}\}}$ over time frames ${\{t_{1},t_{2},\ldots,t_{m}\}}$.

**Table 2: Statistics of our benchmark dataset. We categorize whether knowledge changes over time (Time Variant) or remains constant (Time Invariant). We provide five domains to measure the knowledge from LMs. We set the temporal state with dynamic (knowledge that changes within the time frame we have set) and static (knowledge that do not change within the time frame we have set).**
| Time | Domain | # of | Structured | Format | Temporal | # of | Source |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Dependency | (Time Frame) | Relations | State | Examples |  |  |  |
| Time Variant | general | 8 | ✓ | (s, r, o, t) | dynamic | 8,330 | Wikidata |
| (2010 - 2023) | static | 8,302 |  |  |  |  |  |
| biomedical | 14 | ✓ | (s, r, o, t) | dynamic | 7,345 | UMLS |  |
| (2020 - 2024) | static | 7,345 |  |  |  |  |  |
| legal | 6^* | ✗ | QA | dynamic | 3,142 | CFR |  |
| (2010 - 2023) | static | 3,142 |  |  |  |  |  |
| Time Invariant | commonsense | 8 | ✓ | (s, r, o) | invariant | 24,788 | CSKG |
| math | 12 | ✓ | (s, r, o) | invariant | 2,585 | Math-KG |  |

### 3.3 Time Variant & Invariant Knowledge

We sourced time variant knowledge from the general, biomedical, and legal domains.
In general domain, we utilize Wikidata (Vrandečić & Krötzsch, ) dump to track object changes among the time frame using suggested time quantifiers.
Collecting similar amounts of dynamic and static instances across eight relations, the result is formatted as $\{{s,r,o,t\}}$ quadruplet for each object and accompanied time stamp.
For biomedical domain, we parse Unified Medical Language System (UMLS) (Bodenreider, ) metathesaurus data, where suggest yearly updated research in the domain, following previous work of BIOLAMA (Sung et al., ).
Due to the slow pace of change in biomedical research, object pools in this domain shows slight expansions or contractions over time frame.
In the legal domain, we employ the Code of Federal Regulations (CFR) () to track regulatory changes, as they suggest collection and accumulation of change in regulations at the end of year.
Starting from pre-processing unstructured xml data, we adopt a QA-like format with placeholder for object, tracked among time frame which ends to dynamic or static whether it change or not.
Time invariant knowledge, which remains constant regardless of temporal context, is drawn from common-sense and mathematical domains.
We process the CSKG (Ilievski et al., ) dataset of commonsense knowledge, and Math-KG (Wang, ) for covering areas like data structures and pure mathematics.
Further details of our source data are provided in Appendix A.2.

## 4 Experimental Setup

We conduct the experiments on a variety of proprietary and open-source LLMs.
Each model utilizes either an instruction-tuned or chat version to enhance instruction following during sampling.
We enumerate the eight open-source and one proprietary LLMs: Llama-3.1-8B-Instruct (Meta, ), Llama-3-8B-Instruct (Dubey et al., ), Llama-2-7b-chat-hf (Touvron et al., ), Mistral-7B-Instruct-v0.3 (Jiang et al., ), Phi-3.5-mini-instruct (Abdin et al., ), SOLAR-10.7B-Instruct-v1.0 (Kim et al., ), gemma-2-9b-it (Team et al., ), gemma-7b-it (Team et al., ), and GPT-4o mini (OpenAI, ).
We focus on analyzing trends in the chronological knowledge captured by suggested models from various companies.
These models differ in corpus coverage reflecting their respective cut-off dates.
Details of our inference setups are in Appendix A.3.

## 5 ChroKnowledge : Chronological Categorization of Knowledge

Figure: Figure 2: Performance analysis of general domain. The left two figures show the tendency of dynamic knowledge in temporal results, with more fluctuations in recent knowledge compared to static knowledge, which is more stable but still shows slight variation across the plots. The template-wise results reveal a trend of minimal reliance on internal representation (top figures). Meanwhile, the MCQA templates are influenced by the model’s specialized capabilities, even managing to overcome the training cutoff in recent years (bottom figures).
Refer to caption: x2.png

In this section, we introduce ChroKnowledge (Chronological Categorization of Knowledge), a sampling-based framework designed to evaluate and update LLMs’ non-parametric chronological knowledge.
Our methodology assesses the temporal capabilities of LLMs using two distinct templates, detailed in Section 2.2, and explores how current LLMs encapsulate temporal information.

### 5.1 Results of Representing Knowledge from Large Language Models

For testing model’s knowledge within the categorization, we sample five times for each knowledge to elicit it as possible in dynamic and static dataset.
We present our findings across three different aspects: temporal-wise, template-wise, and domain-wise results.
In Figure 2 and 3, we depict the results for all domains and especially address general and biomedical domains in main section.
Legal and invariant datasets are presented in Appendix (Figure 6 and 7).

##### Temporal-wise Results.

By comparing the left and right side would provide the tendency of temporal-wise results in Figure 2 and 3.
A common trend across models is a decline in recent knowledge, aligning with pretraining corpus cutoff dates.
In dynamic datasets, models show reduced knowledge in multiple-choice question answering (MCQA) settings, highlighting the impact of formatting and prompting on temporal abilities.
Static datasets exhibit less fluctuation, though the result of general domain suggests domain-specific characteristics that each relations are more influenced via temporal sensitivity.

##### Template-wise Results.

By comparing the top and bottom side would provide the tendency of template-wise results in Figure 2 and 3.
As we utilized two templates: generation and MCQA, the result is also checked in template specificity.
Generation templates reveal a greater decline in recent knowledge, as models rely on internal representations without predefined answers.
In contrast, MCQA templates help models select correct answers from options, mitigating some gaps in recent knowledge.
Notably, GPT-4o Mini performs best in generation tasks, while gemma-2-9b-it excels in MCQA.
In addition, the result of Phi-3.5-mini-instruct indicates specialized abilities can offset smaller model sizes, as it is more than twice small billion size compared to other models.
This highlights the importance of template design in eliciting temporal knowledge, with generation benefiting from larger models and MCQA leveraging structured formats to enhance knowledge recall.

Figure: Figure 3: Performance analysis of biomedical domain. Across all the cases, there is little variability, highlighting a domain-specific feature. Both the left and right figures show similar aspects, though the failure to track updates between 2022 and 2023 is more clearly shown in the dynamic dataset. When comparing the top two generation figures with the MCQA results at the bottom, the generation templates show a narrower gap between models, while the gap in MCQA is more distinct. However, in terms of temporal states, the generation settings reveal clearer distinctions between dynamic and static knowledge, whereas in MCQA, the task-specific abilities may compensate for the differences.
Refer to caption: x3.png

##### Domain-wise Results.

Comparing the Figure 2 and 3 provide the tendency of domain-wise results.
In general domain, the models show a decline in recent knowledge, especially with generation templates.
It comes from domain’s nature, such as changes in relation ‘position held’ or ‘member of sports team’ which has sensitivity to temporal cues, leading to higher variability.
However, in biomedical domain, models fail to track recent updates between 2022 and 2023, particularly in generation tasks.
Static datasets remain consistent, reflecting the stable nature of scientific knowledge.
Here, MCQA setting offers some resilience though still limited to specific model’s improvement.
We provide the overall trends of legal domain and time-invariant knowledge in Appendix A.4.

Overall, domain-specific characteristics significantly influence LLMs’ temporal knowledge representation.
More stable domains like biomedical and legal exhibit consistent performance, while more variable domains like general show more fluctuations.
These insights underscore the need for tailored strategies to enhance LLMs’ temporal knowledge capabilities.

## 6 ChroKnowPrompt: Chronological Knowledge Prompting

### 6.1 Chronological Categorization

We represent ChroKnowPrompt (Chronological Knowledge Prompting), an in-depth prompting of non-parametric method to elicit chronological knowledge which has gap between each time stamp, by traversing step-by-step through the surrounding time spans.

Figure: Figure 4: Chronological categorization based on each answer with its time stamp. If the model answer correctly for all, it is re-categorized as Known. The target of ChroKnowPrompt is Partial Known, which confuses its knowledge among the whole time stamps.
Refer to caption: x4.png

In previous section, we demonstrate whether open-source and proprietary models possess specific knowledge at various time stamps.
However, this does not sufficiently assess the models’ understanding of knowledge within a chronological progression.
As Zhao et al. () suggest, knowledge influenced heavily by temporal factors such as general domain can still vary in more stable situation like static dataset.
To address this issue, we first reclassify the models’ responses using a refined categorization scheme, allowing for a more comprehensive evaluation of temporal knowledge across all years.

Figure 4 illustrates how it works:
(1) We refer it as Known for the precise temporal alignment if the model correctly identifies all relevant objects for a given knowledge category at each specific year;
(2) If model fails to match the correct objects for every year, we refer it as Unknown for indicating incomplete or misaligned temporal knowledge;
(3) The model accurately responds either just before or after a specific year but fails for others, signifying outdated information or forgotten legacy knowledge due to continuous updates (Cut-off);
(4) The model correctly identifies some objects for a given year but not others, reflecting an incomplete understanding of the temporal knowledge (Partial Known).

Our main focus is on the Partial Known category, where models demonstrate substantial temporal knowledge but fail to answer correctly for all years, often showing confusion between correct answers.
For example, Nana Akwasi Asare ($s$) was a member of sports team of ($r$) FC Utrecht ($o_{n}$) in 2011 ($t_{n}$), but the model incorrectly identifies the team as FC Groningen, despite answering correctly with FC Utrecht for 2010 ($t_{n-1}$) and in 2012 ($t_{n+1}$).
At this point, we hypothesize that when the model gets one time stamp wrong, a more explicit focus on the temporal aspects surrounding that time span could help it generate more accurate answers. This is the core idea behind ChroKnowPrompt, which we will discuss details in the following section.

Figure: Figure 5: Overview of ChroKnowPrompt. The algorithm systematically traverses step by step, appending each span’s result as few shot for each steps. The range of each previous and next span is predefined, with the order of nearest time stamp from target $T_{n}$. The model suggests last candidate answer $C_{n}$, verified and refined through several steps, which ends to be checked with the object $o_{n}$ in benchmark.
Refer to caption: x5.png

### 6.2 Method

We introduce a chronological prompting technique for non-parametric knowledge editing, aimed at bridging knowledge gaps by utilizing multiple temporal snapshots.
This method enhances the model’s reasoning by systematically integrating knowledge from different time stamps, enabling for in-depth eliciting.
Figure 5 illustrates an example of the chronological prompting process.

Our process begins with a target year $t_{n}$.
The algorithm systematically traverses the preceding years ($t_{n-1},t_{n-2},\ldots$) in the ‘Previous Span’ and the subsequent years ($t_{n+1},t_{n+2},\ldots$) in the ‘Next Span’.
For each traversed year, the most representative object $\hat{o}_{k}$ is selected from the Correct (represented by circle) and Partial Correct (represented by triangle) categories in Table 1 using majority voting.

Starting with the initial prompt containing the target time $t_{n}$, subject $s_{n}$, and relation $r_{n}$, the nearest year in the previous span is appended above the initial prompt with the selected object, forming the first step.
The model then generates a candidate answer $C_{1}$ for $t_{n}$ using this augmented prompt.
Next, our ChroKnowPrompt iteratively adds prompts from progressively earlier years, refining the candidate answer by verifying its consistency across contexts (from $C_{1}$ to $C_{2}$).
This backward traversal continues until a predefined span range is reached.
Once the previous span is completed, our algorithm performs forward traversal by appending objects from subsequent years below the target year, further generating and verifying candidate answers.
If there are no previous or next years available, the process proceeds on only one side.

Upon completing all traversals, the final candidate answer $C_{n}$ is compared against the benchmark object for $t_{n}$.
If the candidate answer aligns correctly with the object for $t_{n}$, appropriately reflecting the temporal contexts, the knowledge categorization for $t_{n}$ is updated to Chrono-Correct, which is equivalent to Correct for chronological assessments.
In Appendix A.5, we provide the detailed steps.

## 7 Experimental Results & Analysis

##### Task Configurations.

We apply our method to both Incorrect and Partial Correct categories, as the latter may still lack definitive answers.
The test set consists of 10% of the total dataset from each domain.
Evaluation employs fuzzy matching with a temperature of 0 for strict assessment, classifying an answer as Chrono-correct only if the last candidate answer matches the object.

### 7.1 Results of ChroKnowPrompt

Table 3 presents the effect of chronological prompting on knowledge elicitation across different models.
Results are shown as the percentage of Known category, with parenthetical values indicating the increase by Chrono-correct.
Significant improvements are observed in the biomedical domain (average increase of 11.9%), while general and legal domains show smaller gains (2.8% and 2.7%, respectively).
This indicates that chronological prompting effectively enhances knowledge recall without requiring external retrieval systems.
In the legal domain, models struggle to update knowledge due to the complexity of unstructured, context-rich data.
Among the models, proprietary GPT4o-mini perform best overall, while open-source models like Llama-2-7b-chat-hf also show notable improvements despite outdated.
These findings suggest that chronological prompting can enhance both current and legacy models by leveraging temporal context.

**Table 3: Result of ChroKnowPrompt. The order of open-sources LLM is sorted by release date, starting from the latest model to the most outdated model. The numeric score is the level of Known in chronological categorization and the increase in parentheses is from the ratio of Chrono-correct which was confusing Partial correct before. Each result presents both in total span and previous span.**
| Models | general | biomedical | Model Increase |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| total span | previous span | total span | previous span | total | previous |  |  |  |  |  |
| dynamic | static | dynamic | static | dynamic | static | dynamic | static | span | span |  |
| Proprietary Large Language Models |  |  |  |  |  |  |  |  |  |  |
| GPT4o-mini | 28.7 (+7.7) | 33.2 (+4.7) | 26.6 (+5.7) | 31.7 (+3.3) | 51.9 (+23.0) | 51.6 (+27.8) | 41.8 (+12.8) | 36.7 (+13.0) | 15.8 | 8.7 |
| Open-Source Large Language Models |  |  |  |  |  |  |  |  |  |  |
| Phi3.5 Mini | 17.3 (+2.1) | 25.5 (+2.5) | 16.5 (+1.2) | 24.1 (+1.1) | 45.4 (+18.7) | 41.3 (+20.3) | 36.6 (+10.0) | 31.5 (+10.5) | 10.9 | 5.7 |
| LLaMA3.1 | 20.6 (+3.1) | 27.1 (+1.7) | 19.4 (+1.9) | 26.4 (+1.0) | 36.9 (+9.2) | 33.6 (+7.9) | 32.0 (+4.2) | 29.1 (+3.4) | 5.5 | 2.6 |
| Gemma2 | 19.6 (+4.0) | 26.7 (+2.3) | 17.8 (+2.2) | 24.7 (+0.4) | 32.5 (+6.2) | 31.7 (+9.0) | 27.9 (+1.5) | 26.7 (+4.1) | 5.4 | 2.1 |
| Mistral v0.3 | 18.6 (+1.8) | 26.9 (+1.6) | 18.3 (+1.6) | 26.8 (+1.5) | 26.6 (+4.2) | 24.3 (+5.6) | 24.6 (+2.2) | 21.3 (+2.6) | 3.3 | 2.0 |
| LLaMA3 | 20.9 (+2.7) | 28.0 (+1.7) | 20.8 (+2.5) | 27.2 (+0.9) | 31.4 (+5.7) | 25.7 (+3.8) | 28.7 (+3.0) | 24.2 (+2.3) | 3.5 | 2.2 |
| Gemma | 18.9 (+1.0) | 25.9 (+1.5) | 18.8 (+0.8) | 25.3 (+0.8) | 18.3 (+6.0) | 12.6 (+5.3) | 16.0 (+3.7) | 9.60 (+2.3) | 3.5 | 1.9 |
| SOLAR | 16.5 (+0.8) | 24.9 (+0.9) | 16.7 (+1.1) | 25.1 (+1.1) | 26.5 (+4.1) | 20.3 (+4.5) | 27.7 (+5.3) | 19.7 (+3.8) | 2.6 | 2.8 |
| LLaMA2 | 18.1 (+5.2) | 26.6 (+5.0) | 15.9 (+3.0) | 23.1 (+1.5) | 44.3 (+25.2) | 37.2 (+26.3) | 32.5 (+13.4) | 23.3 (+12.4) | 15.4 | 7.6 |
| Temporal Increase | 3.2 | 2.4 | 2.2 | 1.3 | 11.4 | 12.3 | 6.2 | 6.0 |  |  |
| Domain Increase | 2.8 | 1.8 | 11.9 | 6.1 |  |  |  |  |  |  |

### 7.2 Effects of chronological span

To elucidate the mechanisms of chronological prompting, we conduct a detailed analysis across different settings, focusing on the impact of incorporating the next span in chronological contexts.
Table 3 compares performance when using only the previous span versus the total span (both previous and next).
Overall, the total span approach yield higher scores compared to using only the previous span.
However, the degree of improvement varies by domain.
In the biomedical domain, the total span results in nearly double the score increase compared to the previous span alone (11.9 vs. 6.1).
Conversely, general domain shows a modest increase (2.8 vs. 1.8).

Model-specific temporal sensitivity also varies.
For instance, Llama-2-7b-chat-hf demonstrates significant score increases by effectively utilizing temporal cues, particularly from next spans.
On the other hand, SOLAR-10.7B-Instruct-v1.0 shows improvements when only the previous span was used, suggesting differing responses to temporal information across models.
These findings indicate that models differ in their responsiveness to past and future temporal contexts.
Another possible explanation is the variation in the temporal coverage of each model’s pretraining corpus, which may influence their temporal sensitivity and affect knowledge recall across different time frames.

## 8 Related work

### 8.1 Deriving and Exploring Knowledge from Language Models

Since the emergence of LMs, deriving knowledge from language model is extensively studied, such as probing tasks (Hewitt & Manning, ), LAMA (Petroni et al., ) and BioLAMA (Sung et al., ).
Then, many subsequent studies follows to exploit, (1) how LLMs define knowledge (; Zhang et al., ; Gottesman & Geva, ), (2) how these models represent it (Chen et al., ; ; Wang et al., ), and (3) how manipulate misleading part (Wang et al., ; Gutiérrez et al., ; Wu et al., ).
Based on them, recent researches on investigating knowledge highlight the dynamic nature of evolving facts and suggests that contradictions within the training data may lead to knowledge conflicts (Marjanović et al., ; Chang et al., ; Wang et al., ; Xu et al., ; Jin et al., ).
And knowledge overshadowing (Zhang et al., ) reveals phenomena where certain conditions overshadow other facts, leading to misleading information (i.e., hallucinations).

In other view point, exploring temporal knowledge starts from using Wikidata, a static format of knowledge in triplet: subject, relation, and object, originated from extracting literature-based knowledge (Hahn-Powell et al., ).
Following pioneers like TimeQA (Chen et al., ) and TemporalWiki (Jang et al., ), many works dealing with temporal and continuous knowledge flow (Zhang & Choi, ; Dhingra et al., ; Jang et al., ; Liska et al., ; Nylund et al., ; Zhu et al., ; Khodja et al., ; Zhang et al., ) consist in line of it.
Building upon their achievement, CarpeDiem (Kim et al., ) emerges to simply identify whether knowledge is outdated or not, and DyKnow (Mousavi et al., ) maps various models’ knowledge distribution.
Also, (Zhao et al., ) makes dramatic work: align model into one fixed age.
Though those impressive works, we try to broad the coverage of temporal knowledge.
We utilize various templates to elicit the knowledge of LLMs, broaden the coverage of time stamps, and differentiate domains that should change based on a temporal perspective with those that should remain constant.

### 8.2 Parametric & Non-parametric Knowledge Update

Considering update of LLMs are in two types, parametric and non-parametric (Wang et al., ), a classical way of parametric update is using fine-tuning (Ghosal et al., ; Mecklenburg et al., ; Ge et al., ).
While the method extended to LoRA (), QLoRA (), Melo (Yu et al., ) to continual learning (GRACE (Hartvigsen et al., ) and WISE (Wang et al., )), by using the parameter accessibility of white-box LLMs like Llama series (Touvron et al., ), MEND (), ROME (Meng et al., ), MEMIT () emerges.
Those local editable methods are effective, and still try to improve specificity and generalizability.

In contrast, for black-box LLMs, updates rely on non-parametric knowledge methods (Onoe et al., ), such as SERAC (Mitchell et al., ), MeLLo (Zhong et al., ), and IKE (Zheng et al., ).
They align with two key trends:
(1) Mitigating catastrophic forgetting, where the model loses previous knowledge, by not directly updating parameters.
(2) Exploiting abilities of prominent black-box LLMs like GPT-o1 (OpenAI, ) and Gemini (Team et al., ), as we cannot access to parameters.
Another concern in knowledge update is they often focus only structured format, pointed out that current methods struggle to update unstrctured data effectively (Wu et al., ).
In this paper, we focus on non-parametric knowledge updates accomodating a broad range of input formats (structured and unstructured) to represent knowledge across diverse domains, depending on the use of various white-box and black-box LLMs.

## 9 Conclusion, Limitation, and Future Work

Overall, our work highlights the critical role of temporal context in knowledge evaluation and introduces a robust framework for improving the temporal capabilities of future language models.
We present ChroKnowBench, a comprehensive benchmark designed to assess the temporal knowledge of large language models (LLMs) across diverse domains, time dependencies, and temporal states.
Through our novel ChroKnowledge framework, we systematically evaluate and enhance the chronological knowledge of various LLMs, revealing that the effectivenesss of knowledge elicitation depends on the specific training data formats of each model.
Our findings indicate that while models often associate facts with specific time stamps, they frequently exhibit partial recall or struggle to fully capture temporal boundaries.
By implementing an in-depth chronological prompting, ChroKnowPrompt, we achieve significant improvements in knowledge recall, with the most notable increase seen in the biomedical domain.
This non-parametric approach proves effective for both proprietary and open-source models, demonstrating its potential as a practical method for updating and refining the temporal accuracy of LLMs without relying on external retrieval systems.

While our approach demonstrates significant improvements in certain domains, it shows limited gains in general domain.
This could be a limitation of our non-parametric approach to effectively update knowledge within complex and context-rich fields.
Specifically, domains like general and legal require more definitive updates that extend beyond the capabilities of prompt-based techniques alone.
This underscores the need for practical knowledge updating and editing methods that are intrinsically aligned with temporal dynamics in line of parametric approach.

To address this limitation, our future work will focus on developing methods for updating the parametric knowledge of open-source LLMs.
Current knowledge update techniques do not adequately handle temporal aspects, which are essential for maintaining the accuracy and relevance of information over time.
We aim to create comprehensive updates synchronized with various time stamps and explore how different temporal attributes affect both structured and unstructured data.
By refining the temporal alignment of knowledge updates, we hope to enhance the LLMs’ ability to recall and integrate information accurately across different time frames, thereby overcoming the current limitations observed in domains that require a deeper understanding of temporal complexity.

## Appendix A Appendix

### A.1 Details of Eliciting Knowledge: Few-shot Exemplars, Fuzzy Matching Rules, and Examples of Two Templates

#### A.1.1 Few-shot Exemplars

To obtain the few-shot exemplar pool $D$, we leverage additional data collected using the same process as in ChroKnowBench. Specifically, for each individual relation type, We gather four exemplars for the general domain, and eight for the biomedical and legal domains. We then generate the few-shot exemplar set $D_{i}$ by sampling four exemplars from $D$, which serves as actual demonstrations within a prompt. This process is repeated for every timestamp to ensure comprehensive temporal coverage.

#### A.1.2 Fuzzy Matching

We utilize the rapidfuzz library to compare the model’s responses with the predefined labels.
As the model’s answer may a little bit different with complicated objects in specialized domains, such as the difference in order of words or upper and lower cases, using fuzzy match enables more rapid but still reliable quality without facilitating external NLI mechanisms.

Specifically, we employ a token_set_ratio metric with a threshold value set to 70 to determine a match.
token_set_ratio is a metric used for comparing the similarity of two strings in a flexible manner, extending the functionality of the token_sort_ratio.
In the preprocessing stage, the strings undergo tokenization, removal of punctuation, and conversion to lowercase.
The tokens are then sorted in alphanumeric order before the similarity ratio is computed.
This makes it useful for comparing strings where the word order may differ but the content is similar.

The key distinction of token_set_ratio lies in its incorporation of set operations, where duplicate words are removed.
After eliminating repeated tokens, the same preprocessing steps as in token_sort_ratio are applied.
When performing the comparison, the method checks if all tokens from the shorter string are contained within the longer string, making the approach particularly suited for cases where one string is a subset of the other.
This flexible matching often results in higher accuracy for comparing strings with similar content but different structures, as illustrated by the example where a score of 100 is achieved when all tokens from the second string are present in the first.

#### A.1.3 Examples of Two Templates

We provide two templates of Generation and MCQA in the end of the Appendix for the better readability.
For example, in Table 5, our target year is 2020 (t) to generate answer candidate of position held (r) by Donald Tusk (s).

#### A.1.4 Iterative Distractor Generation

For the Commonsense dataset, the objects corresponding to a given subject and relation are often ambiguous.
When constructing compelling distractors, there is a higher likelihood (about 20%) of creating options that are actually correct answers rather than intended incorrect ones, compared to other datasets.
Therefore, we include an additional verification process after generating the distractors, as outlined in Algorithm 1.
Specifically, we formulate multiple-choice questions using the problem and the generated distractors, then use GPT-4o to select all correct answers.
If it identifies more than one correct answer, we refine the distractors based on a prompt to recreate incorrect options.

Figure: Algorithm 1 Iterative Distractor Generation Algorithm

### A.2 Details of our source of benchmark dataset

#### A.2.1 Biomedical Domain

In the biomedical domain, we follow previous work of BIOLAMA (Sung et al., ) framework to parse Unified Medical Language System (UMLS) yearly metathesaurus data.
In the range of 2020 to 2024, we gather instances in 14 relations, resulting 7k for each dynamic and static dataset.
Here, by considering domain specificity that the slow pace of change typical of long-term research, the object pool is slightly expanded or narrowed in that period;
Autonomic nerve structure, with the relation has indirect procedure site, has a slightly broader scope in 2024, including additional objects like Neurolytic autonomic nerve block alongside previous objects such as Intravenous regional autonomic block.
The format is same with general domain, $\{s,r,o,t\}$ quadruplet.

#### A.2.2 Legal Domain

In the legal domain, we create a benchmark dataset based on the Code of Federal Regulations (CFR) from 2010 to 2023. We first extract paragraph-level data from regulatory documents for each year and employ Python’s difflib library to detect changes between paragraphs across adjacent years (e.g., 2011 to 2012). Careful filtering is applied to ensure that only paragraphs with minor modifications (e.g., single-word updates or subtle phrasing changes) are retained.

To further analyze the dataset, we utilize the spaCy en_core_web_lg model to detect named entities in the paragraphs and assess whether these changes involve modifications to the detected entities. Despite noise introduced by the NER model, we initially identify around 56K changes for near-year comparisons. These changes are grouped into sequences of years to track alterations over time, while filtering out paragraphs that are introduced or removed in intermediate years. Ultimately, we focus on paragraphs present in all years between 2010 and 2023, resulting in 8,793 paragraphs.

We then apply GPT-4o-mini to assess whether the detected changes are semantically meaningful, excluding minor corrections like typographical fixes or abbreviations. This results in a refined set of 4,362 meaningful updates. Additionally, we select 4,746 unchanged paragraphs containing entities detected by the NER model. For each paragraph, we format the changes as fill-in-the-blank tasks, where the modified part is replaced with a blank, providing a rich resource for studying legal text evolution over time.

#### A.2.3 Commonsense and mathematics

In the commonsense domain, we utilized the CSKG dataset presented in the CSKG paper. Unlike the BIO dataset, the object lists for each triplet in this dataset consist of synonymous terms, allowing multiple triplets to share the same subject and relation. In such cases, the objects appearing in each triplet carry distinct meanings. Out of the 6 million triplets, we merged the objects of triplets that have the same subject and relation into a single set, and then sampled x number of triplets from this collection.

In the mathematics and data structure/algorithm domain, we utilized the Math-KG dataset introduced in the Math-KG paper. This dataset, originally in Chinese, stores multiple objects with the same subject and relation across different triplets. Each object was translated into English using GPT-4, after which the objects from triplets sharing the same subject and relation were merged to construct a final dataset consisting of 22k triplets.

For the CommonSense dataset, the answers (objects) corresponding to a given subject and relation are often ambiguous. Consequently, when constructing compelling distractors, there is a higher likelihood (about 20%) of creating options that are actually correct answers rather than intended incorrect ones, compared to other datasets. Therefore, we include an additional verification process after generating the distractors, as outlined in Algorithm 2. Specifically, we formulate multiple-choice questions using the problem and the generated distractors, then ask GPT-4o to select all correct answers. If it identifies more than one correct answer, we refine the distractors based on a prompt to recreate incorrect options.

### A.3 Inference Setting

We evaluates all models using vLLM (Kwon et al., ) system, supporting features of efficient KV cache memory management that dramatically decrease inference time.
All white box LM inference is conducted by vLLM with hyper-parameter: BFloat16, fixed seed, two kinds of temperature based on each sampling setting(greedy decoding with 0.0, and high temperature with 0.7).
The precision is done with eight NVIDIA A100 GPUs(80GB).

### A.4 Tendency of Legal Domain and Time-Invariant Knowledge

Figure: Figure 6: Performance analysis of legal domain. Among time variant domains, legal domain shows the most stable results of static, besides, there still exists fluctuation and steep cutoff in dynamic dataset. When it comes to each templates, generation shows a less variation among models, while MCQA setting shows cut-off in many models between 2015 and 2016. It is also observed that in this domain, model’s capability of task is still influential such as the result of gemma-2-9b-it or SOLAR-10.7B-instruct-v1.0, but the parametric knowledge seems more important as small model like Phi-3.5-mini-instruct cannot overcome the gap.
Refer to caption: x6.png

Figure: Figure 7: Performance analysis of common-sense and mathematics domains. All model shows clearly the domain specific characteristics, which is invariant knowledge even it comes with temporal attributes.
Refer to caption: x7.png

Figure 6 shows the result of legal domain.
Among time variant domains, legal domain shows the most stable results of static, also minimal decline in dynamic dataset, especially generation setting.
This indicates the domain specificity, which has less frequent yearly changes and change continues across many time stamps.
In the template wise viewpoint, however, MCQA templates exhibit cutoffs between 2015 and 2016, suggesting selection tasks are more sensitive in this domain’s temporal boundaries.
Also, model’s capability for task is still influential in legal domain like the result of gemma-2-9b-it and SOLAR-10.7B-instruct-v1.0 shows, but the parametric knowledge seems more important here as small model cannot overcome the gap by technical skills, like Phi-3.5-mini-instruct.

For commonsense and mathematics in Figure 7, arbitrary years based on the biomedical domain were used, from 2020 to 2024.
The left side of result shows the tendency of generation templates, and the right side is the tendency of MCQA templates, in both domains.
Results show minimal variation, aligning with the stable nature of these knowledge types.
This consistency confirms that time-invariant knowledge is well-preserved across models.
For template wise comparison, generation cases show a way little gap between models, while MCQA tasks show the different between models, which is aligned with the findings from other time variant domains: the ability of each model’s specialized task affects its knowledge recall ability.

About the overall performance quality, the result of time invariant shows lower performance as the model show generate appropriate object in each knowledge, even though the time invariant knowledge’s coverage is wider than other domains.
This tendency is alleviated by using MCQA templates, which ends of the rationale for using multiple templates.

### A.5 Algorithm of ChroKnowPrompt

The overall scheme of ChroKnowPrompt is down below.
As described in Section 6.2, the algorithm starts from making initial prompt with target time $t_{n}$, subject $s_{n}$ and relation $r_{n}$ from target triplet.
As the initialized candidate answer is None that model cannot properly answer for that target year, the algorithm also starts with make a empty list of candidate answer list $A$. and accumulated prompt $\mathcal{P}$.
Then, the algorithm checks the correct object within each time span $P$ and $N$.
If one of those span has no correct object, the algorithm passes that side of traversal.
It the preparation is all done, the first step in previous span (if no previous span exists, the nearest next span) begins with selecting object $\hat{o}$ by majority voting.
Appending prompts in each step, the model is asked to generate or verify and refine the answer $C$ of each step, like in Figure 5.
After all step is done, the last candidate answer, which is the most refined result, is being checked with the original target object $o_{n}$ coming from target triplet.
If it is matched (we also used fuzzy match in here), the category of Incorrect is updated to Chrono-correct.

Figure: Algorithm 2 Chronological Prompting Algorithm

### A.6 Evaluation of ChroKnowPrompt in Legal domain

Table 4 shows the evaluation of ChroKnowPrompt in legal domain.
While our approach demonstrates significant improvements in certain domains, it shows limited or negligible gains in the legal domain.
Overall score in dynamic dataset is mimic, as the highest gain is 1.9 increasing.
However, the increase in static dataset is still impressive as the highest increase is comparable level as much as biomedical domain’s results (more than 10 in the proprietary model).
Another finding is that although the increase in Table 3’s result in general domain is not higher than the static figures in the legal domain, the variation in figures between models is significantly larger in the legal domain.
As the format of legal dataset is the unstructured format with long context, this would be one factor of low edit quality.

**Table 4: Result of ChroKnowPrompt in Legal domain. The order of open-source LLMs follows the same sequence as in Table 3, starting with the latest model and progressing to the most outdated one. The numeric score represents the level of Known in chronological categorization, and the increase in parentheses reflects the ratio of Chrono-correct answers, considering total span in the left side and previous span in the right side.**
| Models | legal | Model Increase |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| total span | previous span | total span | previous span |  |  |  |
| dynamic | static | dynamic | static |  |  |  |
| Proprietary Large Language Models |  |  |  |  |  |  |
| GPT4o-mini | 3.2 (+1.9) | 51.9 (+14.1) | 2.6 (+1.3) | 48.4 (+10.6) | 8.0 | 6.0 |
| Open-Source Large Language Models |  |  |  |  |  |  |
| Phi3.5 Mini | 0.6 (+0.3) | 14.2 (+4.5) | 0.6 (+0.3) | 11.9 (+2.3) | 2.4 | 1.3 |
| LLaMA3.1 | 0.3 (+0.0) | 13.8 (+1.3) | 0.3 (+0.0) | 12.5 (+0.0) | 0.7 | 0.0 |
| Gemma2 | 2.9 (+0.6) | 44.6 (+2.6) | 2.6 (+0.3) | 43.9 (+1.9) | 1.6 | 1.1 |
| Mistral v0.3 | 1.3 (+0.6) | 21.1 (+7.0) | 1.0 (+0.3) | 19.2 (+5.1) | 3.8 | 2.7 |
| LLaMA3 | 1.0 (+0.3) | 18.9 (+0.6) | 1.3 (+0.6) | 18.9 (+0.6) | 0.5 | 0.6 |
| Gemma | 0.3 (+0.0) | 8.70 (+0.0) | 0.3 (+0.0) | 8.70 (+0.0) | 0.0 | 0.0 |
| SOLAR | 0.6 (+0.0) | 26.8 (+1.3) | 0.6 (+0.0) | 28.4 (+2.9) | 0.7 | 1.5 |
| LLaMA2 | 0.3 (+0.0) | 21.8 (+12.8) | 0.3 (+0.0) | 17.3 (+8.3) | 6.4 | 4.2 |
| Temporal Increase | 0.4 | 4.9 | 0.3 | 3.5 |  |  |
| Domain Increase | 2.7 | 1.9 |  |  |  |  |

**Table 5: Example of two templates: Generation and MCQA. Domain: general, Subject: Donald Tusk, Relation: position held (P39), year to target: 2020.**
| Template: Generation |
| --- |
| Q. In 2020, Pedro Braillard Poccard, position held, [Object] |
| A. member of the Argentine Chamber of Senators |
| Q. In 2020, Ana Santos Aramburo, position held, [Object] |
| A. Spain National Library general manager |
| Q. In 2020, James E. McPherson, position held, [Object] |
| A. United States Secretary of the Navy |
| Q. In 2020, Jesús Ávila de Grado, position held, [Object] |
| A. chief scientific officer |
| Q. In 2020, Donald Tusk, position held, [Object] |
| : (generate from here …) |
| Template: MCQA |
| In 2020, what office does Pedro Braillard Poccard hold? |
| (a) member of the Argentine Chamber of Senators, (b) Minister of Foreign Affairs, (c) Governor of Corrientes Province, (d) Mayor of Buenos Aires |
| (a) member of the Argentine Chamber of Senators |
| In 2020, what office does Ana Santos Aramburo hold? |
| (a) Minister of Culture and Sports of Spain, (b) Director of the Prado Museum, (c) Spain National Library general manager, (d) President of the Spanish Royal Academy |
| (c) Spain National Library general manager |
| In 2020, what office does James E. McPherson hold? |
| (a) United States Secretary of Homeland Security, (b) United States Attorney General, (c) United States Secretary of the Navy, (d) United States Secretary of Defense |
| (c) United States Secretary of the Navy |
| In 2020, what office does Jesús Ávila de Grado hold? |
| (a) President of the National Research Council, (b) Minister of Health, (c) Director of the World Health Organization, (d) chief scientific officer |
| (d) chief scientific officer |
| In 2020, what office does Donald Tusk hold? |
| (a) President of the European Commission, (b) President of Poland, (c) Chancellor of Germany, (d) chairperson |
| : (generate from here …) |

**Table 6: 2-shot prompt for generating three distractors in general domain.**
| Domain: General |
| --- |
| [System] |
| You are an expert in natural language processing and logic puzzles, skilled at generating plausible yet misleading distractor options that challenge users to distinguish between correct and incorrect answers. Pay special attention to questions that involve negative phrasing, such as those containing “not” or “which does not,” to ensure that the distractors do not confuse users into overlooking the negative aspect of the question. |
| [User] |
| The question “What office does ‘Mitt Romney’ hold?” can be answered with ‘United States senator’. Create three plausible incorrect distractors for this question. |
| [Assistant] |
| 1. Governor of Massachusetts |
| 2. Secretary of State |
| 3. Speaker of the House |
| [User] |
| The question “Which sports team is ‘Yann MVila’ a member of?” can be answered with ‘Rubin Kazan’, ‘France national association football team’, ‘Sunderland A.F.C.’, ‘France national under-21 association football team’, ‘Inter Milan’, ‘Stade Rennais F.C.’. Create three plausible incorrect distractors for this question. |
| [Assistant] |
| 1. Paris Saint-Germain F.C. |
| 2. Olympique Lyonnais |
| 3. AS Monaco FC |
| [User] |
| The question “[Q]” can be answered with ‘[O1]’, ‘[O2]’, … , ‘[On]’. Create three plausible incorrect distractors for this question. |
| [Assistant] |
| : (generate from here …) |

**Table 7: 4-shot prompt for generating three distractors in biomedical domain.**
| Domain: Biomedical |
| --- |
| [System] |
| You are an expert in natural language processing and logic puzzles, skilled at generating plausible yet misleading distractor options that challenge users to distinguish between correct and incorrect answers. Pay special attention to questions that involve negative phrasing, such as those containing “not” or “which does not,” to ensure that the distractors do not confuse users into overlooking the negative aspect of the question. |
| [User] |
| The question “What is not the primary anatomic site of ‘Rhabdomyosarcoma of the orbit’?” can be answered with ‘Bones set’, ‘structure bone’, ‘Bone structure’, ‘os’, ‘Skeletal bone’, ‘Bone structure (body structure)’, ‘bones structure’, ‘bone structures’, ‘Ossa’, ‘Set of bone organs’, ‘Bone organ’, ‘Skeleton system’, ‘Osseous’, ‘Bone’, ‘NOS’, ‘Bones’, ‘bones structures’, ‘Bone (organ)’, ‘Bone’, ‘skeletal’, ‘Set of bones’, ‘Bones’, ‘NOS’. Create three plausible incorrect distractors for this question. |
| [Assistant] |
| 1. Orbital connective tissue |
| 2. Extrinsic ocular muscles |
| 3. Orbital soft tissue |
| (two more shots) |
| [User] |
| The question “What negatively regulates ‘Vitellogenesis’?” can be answered with ‘downregulation of yolk production’, ‘downregulation of vitellogenesis’, ‘down regulation of vitellogenesis’, ‘inhibition of yolk production’, ‘negative regulation of vitellogenesis’, ‘down regulation of yolk production’, ‘down-regulation of yolk production’, ‘inhibition of vitellogenesis’, ‘down-regulation of vitellogenesis’, ‘negative regulation of yolk production’. Create three plausible incorrect distractors for this question. |
| [Assistant] |
| 1. Partial left salpingectomy |
| 2. Unilateral oophorectomy |
| 3. Hysterectomy |
| [User] |
| The question “[Q]” can be answered with ‘[O1]’, ‘[O2]’, … , ‘[On]’. Create three plausible incorrect distractors for this question. |
| [Assistant] |
| : (generate from here …) |

**Table 8: 2-shot prompt for generating three distractors in legal domain.**
| Domain: Legal |
| --- |
| [System] |
| You are a legal expert skilled in crafting challenging fill-in-the-blank questions and generating plausible yet misleading distractor options. You will receive a question and answer where part of a legal text has been blanked out. For the provided question, create three plausible incorrect distractors that challenge users to distinguish between correct and incorrect answers. |
| [User] |
| The question |
| """ |
| … (beginning of text omitted) |
| Subpart B—Certification of Substantially Equivalent Agencies |
| Substantial equivalency certification is granted if the ____ determines that a state or local agency enforces a law that is substantially equivalent to the Fair Housing Act with regard to substantive rights, procedures, remedies, and the availability of judicial review. The Department has developed a two-phase process of substantial equivalency certification. |
| """ |
| can be answered with ‘Department’. Create three plausible incorrect distractors for this question. |
| [Assistant] |
| 1. Secretary |
| 2. Commission |
| 3. Board |
| [User] |
| The question |
| """ |
| … (beginning of text omitted) |
| Subpart B—Minimum Standards for Substantial Compliance by States |
| Within the period defined in § 383.73( ___ : |
| """ |
| can be answered with ‘f) of this title, the State shall’, ‘h) of this subchapter, the State must’. Create three plausible incorrect distractors for this question. |
| [Assistant] |
| 1. g) of this chapter, the State will |
| 2. e) of this section, the State is required to |
| 3. d) of this part, the State should |
| [User] |
| The question |
| """ |
| [Q] |
| """ |
| can be answered with ‘[O1]’, ‘[O2]’, … , ‘[On]’. Create three plausible incorrect distractors for this question. |
| [Assistant] |
| : (generate from here …) |

**Table 9: 4-shot prompt for generating three distractors in commonsense domain.**
| Domain: CommonSense |
| --- |
| [System] |
| You are an expert in natural language processing and logic puzzles, skilled at generating plausible yet misleading distractor options that challenge users to distinguish between correct and incorrect answers. Pay special attention to questions that involve negative phrasing, such as those containing “not” or “which does not,” to ensure that the distractors do not confuse users into overlooking the negative aspect of the question. |
| [User] |
| The question “What properties does ‘human’ not have?” can be answered with ‘strong enough to lift horse’, ‘strong enough to lift truck’, ‘strong enough to lift car’, ‘strong enough to lift cow’, ‘strong enough to lift building’. Create three plausible incorrect distractors for this question. |
| [Assistant] |
| 1. strong enough to lift a motorcycle |
| 2. strong enough to lift a large suitcase |
| 3. strong enough to lift a heavy stone |
| [User] |
| The question “What is synonym for ‘salix tristis’?” can be answered with ‘dwarf gray willow’. Create three plausible incorrect distractors for this question. |
| [Assistant] |
| 1. green weeping willow |
| 2. dwarf green willow |
| 3. gray weeping willow |
| [User] |
| The question “What does ‘kids’ desire?” can be answered with ‘play games’, ‘play’, ‘find crabs in tidal pools’, ‘read games magazines’, ‘play with’, ‘play with toys’, ‘learn things’. Create three plausible incorrect distractors for this question. |
| [Assistant] |
| 1. eat vegetables |
| 2. do homework |
| 3. fold laundry |
| [User] |
| The question “What is ‘oil and water’ not capable of?” can be answered with ‘mix’. Create three plausible incorrect distractors for this question. |
| [Assistant] |
| 1. boil |
| 2. freeze |
| 3. emulsify |
| [User] |
| The question “[Q]” can be answered with ‘[O1]’, ‘[O2]’, … , ‘[On]’. Create three plausible incorrect distractors for this question. |
| [Assistant] |
| : (generate from here …) |

**Table 10: 2-shot prompt for generating three distractors in mathematics, data structure, and algorithm domain.**
| Prompt: Mathematics / Data Structure & Algorithm |
| --- |
| [System] |
| You are an expert in mathematics and computer science, skilled at generating plausible yet misleading distractor options that challenge users to distinguish between correct and incorrect answers. Pay special attention to questions that involve negative phrasing, such as those containing “not” or “which does not,” to ensure that the distractors do not confuse users into overlooking the negative aspect of the question. |
| [User] |
| The question “Which of the following is unrelated to ‘Output’?” can be answered with ‘Left child’, ‘Set’, ‘Depth’, ‘Modify’, ‘Post-order traversal’, ‘Dictionary’, ‘Predecessor’, ‘Array’, ‘Delete’, ‘Sparse matrix’, ‘Infix expression’, ‘Leaf node’, ‘Shortest path’, ‘Right subtree’, ‘Node’, ‘In-order traversal’, ‘Hashing’, ‘Head node’, ‘Pointer’, ‘Level-order traversal’, ‘Keyword’, ‘Pattern string’, ‘Record’, ‘Determinism’, ‘Linked list’, ‘Critical Path’, ‘Function name’, ‘Connected component’, ‘Loop statement’, ‘Robustness’, ‘Inverted index’, ‘Preorder traversal’, ‘Memory’, ‘Dequeue’, ‘Singly Linked List’, ‘Image’. Create three plausible incorrect distractors for this question. |
| [Assistant] |
| 1. Print statement |
| 2. Display buffer |
| 3. Output stream |
| [User] |
| The question “What is synonym for ‘Ancestor’?” can be answered with ‘Parent’. Create three plausible incorrect distractors for this question. |
| [Assistant] |
| 1. Descendant |
| 2. Sibling |
| 3. Offspring |
| [User] |
| The question “[Q]” can be answered with ‘[O1]’, ‘[O2]’, … , ‘[On]’. Create three plausible incorrect distractors for this question. |
| [Assistant] |
| : (generate from here …) |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Q1: What is the evaluation score for the "Generation" experiments in Figures 2 and 3? How do you extract the answer from the model output? Is it an exact match?

Q2: You mention that you use the original MMLU prompting strategy (Hendrycks 2021). Is this with the 5-shot setting or with the zero-shot and Chain-of-Thought generation?

Q3: Are the results for the "Time Invariant" data being reported?
soundness: 3
strengths: - The paper is very well written, and most of the claims are well-motivated and justified. 
- The benchmark covers various domains and temporal knowledge dimensions. 
- The authors transform the problem into both open generation and MCQ tasks.
summary: This paper introduces CHROKNOWBENCH, a benchmark used to evaluate chronological knowledge in LLMs by distinguishing between evolving and static information. The authors also introduce a sampling-based prompting technique to improve LLMs’ recall of time-dependent knowledge across multiple domains. Their approach enables effective temporal non-parametric knowledge updates, even in proprietary models, showing some improvement in biomedical and general knowledge accuracy.
weaknesses: W1: The overall results appear incomplete across the dataset's various dimensions. A table summarizing average results for each benchmark dimension—Domain, Format, Temporal State, and Time Dependency—would help clarify performance. Currently, it’s unclear whether the performance limitations stem from temporal aspects or the models' domain-specific capabilities.

W2: The proposed prompting method does not clearly demonstrate its effectiveness. Section 5.1 shows stable performance in the biomedical domain, while Section 7.1 suggests the prompting method mainly improves results for biomedical questions, which were not influenced by dynamic changes. How do the authors justify these outcomes as evidence of the prompting method’s efficiency?

W3: In Line 313, the authors state that "GPT-4 performs best in generation tasks." Do they provide any rationale for this observation?

W4: To strengthen the analysis, the paper would benefit from more detailed error analysis and examples. In which cases does the model fail to generate correct answers, and are there discernible patterns in these errors?

---

confidence: 3
contribution: 3
initial_rating: 3
presentation: 3
questions: * The ChroKnowBench considers only changes on `object` while keeping `subject` and `relation` unchanged. How does it capture the change of `relation` between the `subject` and `object`? For example, `Zidane` was a `player` at `Real Madrid` in 2001, then became `coach` in 2010.
* Figures 2 and 3: It is hard to interpret these results; suggest using the same y-axis range
* Section 5.1: `In dynamic datasets, models show reduced knowledge in multiple-choice question answering (MCQA) settings, highlighting the impact of formatting and prompting on temporal abilities.` What does this mean?
* Section 5.1: `models fail to track recent updates between 2022 and 2023`; `Static datasets remain consistent, reflecting the stable nature of scientific knowledge` suggest clarifying what these statements mean
soundness: 3
strengths: * The studied problem is important and interesting.
* The authors build a benchmark for evaluating LLM and propose a method for recalling/editing knowledge?
summary: This paper investigates how well large language models recall chronological knowledge. The paper makes several contributions. First, the authors describe a new benchmark dataset covering different domains (e.g., biomedical, legal) and different types of time dependency and temporal state. Then, the authors benchmark several LLMs (both open-sourced and closed-sourced) and show that LLMs have varied abilities to elicit temporal knowledge. Finally, the authors present a prompting method to elicit chronological knowledge.
weaknesses: * The authors evaluate whether LLM can track `object changes`. This problem formulation may not capture the `accumulative nature of knowledge` as discussed in the paper. For example, one drug may have multiple side effects, each of which was identified at different times. I am unsure how the proposed benchmark and evaluation strategy addresses this accumulative nature. For example, a model that always generates the old knowledge will be considered `correct` based on the definition in Table 1.
* The result sections are hard to follow: it is very difficult to understand how their results support these statements in Section 5 (Figures 2, 3).
* The proposed prompting strategy seems to be disconnected from the benchmark sections. For example, the authors show that the proposed prompting strategy improves in the biomedical domain, whereas results in Section 5 and Figure 3 show little variability across times. Any intuitive explanations of why?

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1.	Why would you include Legal data? The rest of the data are of the same type in format and structure, but only legal is unstructured and of a different type.
2.	For your results in section 4, is it possible that the quality of your collected data in different time points is a more influential factor? 
3.	Suggestions for the weaknesses mentioned above: 
a.	Do more fine-grained classification of the “static” and “dynamic” data.
b.	Update Figure 2 and 3 for more clarity.
c.	Move some information from the appendix to section 7, including the extra descriptions for ChronoPrompt and some extra summaries for the algorithm.
soundness: 3
strengths: The paper targets a novel field, which studies how language models learn and adapt to knowledge in different time spans. Previously, not many researchers formalized and paid attention to the question. The authors did well in defining the key aspects of testing and evaluating chronological knowledge in large language models, including putting forward a comprehensive dataset and benchmark that covers a variety of time ranges and domains, as well as developing novel prompting techniques. They also conducted fairly comprehensive experiments to demonstrate their claims.
summary: This paper proposes a new aspect of evaluating Large Language Models, which focuses on chronologically accumulated knowledge. The authors set up a new benchmark, ChroKnowBench, for chronological knowledge evaluation across three dimensions: time dependency, domain difference, and temporal state. They also put forward a novel evaluation framework called ChroKnowledge, and a novel prompting technique called ChroKnowPrompt, which aims to improve the model’s performance using temporal neighboring data.
weaknesses: 1.	Results in section 7 does not necessarily support the soundness and validity of the ChroKnowPrompt framework, but instead hints at the deficiency in the ChroKnowledge dataset. There is another possible explanation to the large amount of performance increase in the biomedical section of the dataset: domain-specific data are more static / less dynamic than general data, and your method on distinguishing dynamic and static knowledge needs to be updated to a more fine-grained extent. For example, over the span of 10 years, some data might changed 5 to 6 times, but there could be other data that only changed once. I currently don’t see that taken into account. The improvement from the “dynamic” portion of your results could be over-exaggerated.
2.	The figures need more work. For figure 2 and 3, I understand that the authors are trying to demonstrate the relationship between time and model performance, but the graph needs some extra work on clarity to deliver the message. Figure 4 and 5 in section 6 also needs more work. The most useful explanations for Figure 4 are in the appendix, making the figure a little confusing. Figure 5 needs more clear textual hints on the overall framework of ChroKnowPrompt. The figure itself is not illustrative enough for the entire framework.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 4
questions: See weakness.
soundness: 3
strengths: This work addresses the sensitivity of LLM to time dependencies and constructs dynamic and static datasets for evaluation.
This paper introduces an iterative approach to integrate knowledge across different temporal spans, enabling dynamic knowledge updates.
summary: This paper introduces CHROKNOWBENCH, a benchmark dataset designed to evaluate chronologically accumulated knowledge across three key aspects: multiple domains, time dependency, and temporal state. Besides, the authors develop CHROKNOWPROMPT, an in-depth prompting to elicit chronological knowledge by traversing step-by-step through the surrounding time spans. The motivation behind this work is clear, however, the paper majorly lacks in the quality of the experiments and its setup.
weaknesses: The paper mentions evaluating knowledge across multiple domains; however, it only introduces knowledge from the medical and legal domains, where changes of knowledge along the timeline are relatively subtle.
This paper aims to explore the ability of LLMs to capture temporal dependencies within knowledge. However, the constructed benchmark dataset and evaluation methodology do not effectively demonstrate this capability of the models in an intuitive manner.
Although the paper divides knowledge into dynamic and static datasets, the knowledge within these datasets may have already been learned by LLMs. As a result, the benchmark evaluation results are likely to primarily reflect the memory capacity of the LLM regarding this knowledge, rather than its ability to handle time dependencies.
For the dynamic dataset, this paper essentially transforms knowledge into a temporal knowledge graph (TKG). However, there are established benchmarks in TKGs. The authors should consider providing comparisons with these benchmarks and clarifying the specific improvements made.

Meta-review: This paper introduces a benchmark dataset ChroKnowBench to evaluate chronologically accumulated knowledge of LLMs, and presents a prompting method to elicit chronological knowledge. The problem studies in this paper is important and interesting. The benchmark constructed in this paper may be a valuable resource for future research. Various LLMs have been evaluated on the benchmark. The paper is generally easy to follow, but there are some non-trivial writing issues in the paper that can be addressed by incorporating the content in the authors' responses into the next version.  All reviewers finally hold a positive opinion of the paper, although no one strongly supports it.

Decision: Accept (Poster)