ChroKnowledge: Unveiling Chronological Knowledge of Language Models in Multiple Domains
Authors: Yein Park
Chanwoong Yoon
Jungwoo Park
Donghyeon Lee
Minbyul Jeong
Jaewoo Kang
Venue: ICLR 2025 Poster (2025)
Keywords: Temporal knowledge
Knowledge update
Primary area: datasets and benchmarks

Abstract: Large language models (LLMs) have brought significant changes to many aspects of our lives.
However, assessing and ensuring their chronological knowledge remains challenging.
Existing approaches fall short in addressing the temporal adaptability of knowledge, often relying on a fixed time-point view. 
To overcome this, we introduce ChroKnowBench, a benchmark dataset designed to evaluate chronologically accumulated knowledge across three key aspects: multiple domains, time dependency, temporal state.
Our benchmark distinguishes between knowledge that evolves (e.g., personal history, scientific discoveries, amended laws) and knowledge that remain constant (e.g., mathematical truths, commonsense facts). 
Building on this benchmark, we present ChroKnowledge (Chronological Categorization of Knowledge), a novel sampling-based framework for evaluating LLMs' non-parametric chronological knowledge.
Our evaluation led to the following observations: 
(1) The ability of eliciting temporal knowledge varies depending on the data format that model was trained on.
(2) LLMs partially recall knowledge or show a cut-off at temporal boundaries rather than recalling all aspects of knowledge correctly.
Thus, we apply our ChroKnowPrompt, an in-depth prompting to elicit chronological knowledge by traversing step-by-step through the surrounding time spans.
We observe that it successfully recalls objects across both open-source and proprietary LLMs, demonstrating versatility, though it faces challenges with dynamic datasets and unstructured formats.

Published as a conference paper at ICLR 2025
CHROKNOWLEDGE: UNVEILING CHRONOLOGICAL KNOWLEDGE OF LANGUAGE MODELS IN MULTIPLE DOMAINS
Yein Park', Chanwoong Yoon', Jungwoo Park 1.3 Donghyeon Lee Miibbyl Jeong" Jaewoo Kang 1,3+ Korea University! Upstage A1 AIGEN Sciences
(522yein, jungwoo-rkkk dong9733, kangj) @korea.ac.kr
ABSTRACT
Large language models (LLMs) have brought significant changes to many aspects of our lives. However, assessing and ensuring their chronological knowledge remains challenging. Existing approaches fall short in addressing the temporal adaptability of knowledge, often relying on fixed time-point view. To overcome this, we introduce CHROKNOW BENCH, a benchmark dataset designed to eval- uate chronologically accumulated knowledge across three key aspects: multiple domains, time dependency, temporal state. Our benchmark distinguishes between knowledge that evolves (e.g., personal history, scientific discoveries, amended laws) and knowledge that remain constant (e.g., mathematical truths, commonsense facts). Building on this benchmark, we present CHROKNOWLEDGA (Chrono- logical Categorization of Knowledge), novel sampling baased framework for evaluating L.LMs' non-parametric chronological knowledge. Our evaluation led to the following observations: (1) The ability of eliciting temporal knowledge varies depending on the data format that model was trained on. (2) LLMs partially recall knowledge or show a cut-off temporal boundaries rather than recalling all aspects of knowledge correctly. Thus, we apply our CHROKNOW BrRiOKNOOPROMIS PROMPT, an in de-thhh prompting to elicit chronological knowledge by traversing step-by-step through the surrounding time spans. We observe that it successfully recalls objects across both open-souree source and proprietary LLMs, demonstrating versatility, though faces challenges with dynamic datasets and unstructured formats
BENNEC,,
INTRODUCTION
Do large language models (LLMs) possess the ability to understand and track the history of knowledge as time progresses? In other words, can these models, which represent the cutting edge of modern artificial intelligence, reason appropriately about questions that involve evolving facts? Although some details remain controversial, knowledge- like science- is built upon accumulation (Zeigler) 2012, Picho et al., 2016). From raw data to information and to knowledge, every bit is cumulative which contributes to progress across all domains. This accumulation forms the foundation for higher- level reasoning. which is akin to wisdom in navigating the complexities of our world (Rowley (007). Given that LLMs are trained on vast and diverse corpora and are now integral to numerous applications in our daily lives, they must remain accurate and up-to-date ensure reliability. Early versions of ChatGPT (OpenAl 2022). ) for instance, sometimes produced inaccurate or absurd responses like the infamous example of "The happening of King Sejong (1397-1450) throwing MacBook (2016-) These errors still give us a lesson that we need more precise model recalling knowledge correctly.
When we examine the issue closely, it's not just matter of hallucination but also about whether the alignment of knowledge, particularly regarding dates, is accurate. Ensuring that LLMs maintain current and contextually relevant knowledge over time is crucial and researchers have explored
* *Corresponding authors
"Our datasets and code are publicly available at https//iital..oommmdaiillhccoooo
hhtps://english.aani http:/sss.........aaaaaiiioetaae /r/att///i// edition/e Imternationaa/109555566

---

Published as a conference paper at ICLR 2025
Time-variant
Yemporalstate
Dynamic
Doeal Teok partic
of Frland
preskdrut
Chelppersoa
S,R, 0,, F.
Generation Quaaiooiin)
8, 9
2033
1814
2013
Q,0
Muiti-choice QA Truee/ / Ralse
Tempooral affoci-able
Static
Associate Jurtioe
Associate Juetica
Asendtate Juatica of...
of...
of..
Q.
2014
General
Biomedical
Legal
Dynamic
Static
Dynamic
Static
Dynamic
Static
Known
Known
Xnoww
Khown
Anoway
2010
2023
2010
2020
2024
2000
2034
2010
2023
2010
2005
Time-Invariant
Math
5, R.
Invariant
Root Mode,..
Multt-chaice Qf Trae/ / Palse
Ganeration Qactrpont
The overview of ChroKnowBBnnhh We gather knowledge with time stamps and separate them in three key aspects: (1) multiple domains: general, biomedical, legal, commonsense, and mathematics; (2) time dependency: as time goes by, changeable knowledge or not; (3) temporal state: dynamic (has evolved over period) and static (no change occurred during period). Here, trends of Correct (62.)) for each years represented by line plots show difference among domains and temporal states. And each highlighted portions are chronologically Known following $ various ways to investigate and verify the knowledge within these models (Zhang et al. 2024b). Pioneering works investigate whether language model has ability of knowledge base not in diverse domains (Petroni et al., 2019 (Snn et al. (021). Many subsequent studies analyze how
Figure I:
LLMs define knowledge (Dai al. 2022 Mishra et al. 2024); exploit how LLMs represent their
knowledge (Geva et al., 2023 Zheng et al. 2024) with temporal context (asai et al. 2023 Fatemi
al. 2024] ; edit the misleading aspects of knowledge (Manakul et al., 2023: Wang et al. 2024c).
Here, we raise a question: "Do these methods sufficiently address the temporal adaptability of knowledge?". Current temporal-related approaches for evaluating and updating LLMs often focus on single time stamps, struggling to address the adaptable characteristics of knowledge over time (Jang al. 2022a| Ge et al. 2024). which are especially important in specialized domains such as scientific discoveries and amended laws. This limitation can lead to outdated or incomplete information, under- mining the models' effectiveness and safety. Addressing these challenges requires comprehensive approach-tempont evolution, a core component of the knowledge accumulation.
Thus, we introduce CHROKNOW BENCH, a benchmark dataset designed to evaluate chronolog- ically accumulated knowledge across three dimensions: time dependency (time variant and time invariant), multiple domains (general, biomedical etc.), and temporal state (dynamic and static). CHROKNOW BENCH differentiates between knowledge that is subject to evolution (e.g., transfer situation of a soccer player, scientific discoveries, and amended legal regulations) transformations in object-specific attributes such as roles affiliations while keeping the subject and relation fixed-and and knowledge that remains invariant (e.g., mathematical truths and commonsense facts). This object-level focus allows for precise and interpretable assessments of temporal knowledge dynamics by isolating changes in object-specific attributes. We then classify domains based on whether they are influenced by the flow of time, considering the domain specificity. Finally, we set the time frame to categorize the knowledge as either changeable or steady (Section
on
Building on this benchmark, we also present CHROKNOWLEDGE (Chronological Categorization of Knowledge), novel framework for assessing and enhancing the non- non- parametric chronological knowledge of I.LMM. So, we start from analyzing how current open-source and proprietary L.LMs work. As we expected, the time invariant knowledge shows steady in all time frame. However, for time variant dataset, the domain-specific characteristics significantly influence the representation of temporal knowledge from LLMs. More stable domains exhibit consistent performance, while more variable domains show more fluctuations. These observations highlight that we need a comprehensive approach to enhance representing temporal knowledge from LLMs (Section (5).
To this end, our CHROK PROMPT approach utilizes an in-depth chronological prompting strategy that traverses knowledge across adjacent time spans, effectively addressing issues of partial recall and
2

---

Published as a conference paper at ICLR 2025
Table 1:
Knowledge categorization with a temporal component. We classify responses into Correct, Partial Correct, and Incorrect to specify eliciting predictions in diverse way by comparing them with the answer set A. We use a temperature set T E O, 7 to capture variations in prediction, where T includes both greedy decoding and temperature sampling. We set n as 5, meaning that we evaluate using five distinct combinations of few-shot exemplars to ensure the robust assessment.
Category
Definition
Description
All objects generated with greedy decoding are entirely included within the answer set.
Correct
(4 IMM( d, 0];_, CA
At least one generated object from greedy decoding or temperature sampling is the answer set.
Partial Correct
U. |4| IIMMM,,,,, M, n A
None generated objects, either from greedy decoding temperature sampling, are included in the answer set.
Incorrect
U. {4 ()
AA= 13
temporal boundaries (Section 6 In knowledge recall, our evaluation reveals improvements in the biomedical domain (1 (1.55) and the general domain (2.8%), shifting knowledge category from Partial Known to Known for unchanged objects. Our non-parametric approach allows for direct updates to both proprietary and open-source LLMSs without extensive retraining, highlighting practicality and broad applicability (Section 7 Our work emphasizes the importance of temporal context in eliciting LLMs knowledge while identifying challenges with dynamic datasets and unstructured, context-rich formats like in the legal domain. A comprehensive analysis advocates for integrating parametric techniques to complement prompting and achieve more robust temporal knowledge handling.
2
PRELIMINARIES
2.1 KNOWLEDGE CATEGORIZATION WITH A TEMPORAL COMPONENT
To distinguish and evaluate the knowledge levels of language models, we utilize the Sampling- based Knowledge Categorization (SliCK) framework (Gekhman et al. 2024). This approach starts by sampling the model M's answers to questions using various few-shot exemplar sets D. The sampling is conducted under two temperature conditions: т = 0 and 0. Then, it categorizes the degree to which the model knows each piece of knowledge into four levels: HighlyKnown, MaybeKnown, WeaklyKnown, and Unknown.
Based on Gekhman et al. (2024), we make the following modifications as follows: (1) We append a temporal component f the conventional {subject (s), relation (r), object {o)} triplet structure, allowing us to evaluate the model's 's knowledge across different time stamps; (2) We merge the two categories (MaybeKnown and WeaklyKnown) that represent recallable knowledge state into a single category (Partial Correct); (3) By using time attribute, we also renamed the HighlyKnown and Unknown to the Correct and Incorrect, respectively. Our detailed definitions and descriptions are provided in Table Although this setting allows to categorize the model's sampled responses more precisely regarding time attribute, it only captures the model's ' knowledge at specific time points t, limiting our ability observe changes over time. We address this limitation in Section 6
2.2 ELICITING KNOWLEDGE USING DIVERSE TEMPLATES
Since models prefer different formats when eliciting their knowledge, it is important to use varied approaches to accurately assess their understanding (Zhou et al. 2024). While we initially evaluate the model's 's knowledge using standard triplet format, relying on single template may not sufficiently capture the full extent of the model's 's knowledge. Thus, following Hendrycks et al. (2021): Huang al. (2024), we also employ well-known format, multiple-choice question answering (MCQA) with 4 options, and True/False to elicit the model's 's knowledge more effectively. As a result, we propose three templates for measuring how much knowledge the model holds: triplets (hereafter referred to as Generation), MCQA, and TF. Each template is designed with appropriate few-shot exemplars and corresponding matching rules. For example, in Generation, due to the complexity of evaluating responses, we apply fuzzy matching techniques to compare the generated responses against predefined labels. See Appendix A.2 for further details of few-shot exemplars, fuzzy matching rules, and examples of three templates.
We refer a recallahle knowledge the presence of at least one answer in the answer set, generated using either the greedy decoding or the temperature sampling method.
3

---

Published as a conference paper at ICLR 2025
Table 2: Statistics of our benchmark dataset. We categorize whether knowledge changes over time (Time Variant) or remains constant (Time Invariant). Among five domains, we set the temporal state with dynamic (knowledge that changes within the time frame we have set) and static (knowledge that do not change within the time frame we have set). The number parentheses represents the average change in objects per element within dynamic dataset. See details in Appendix A.3.1
Time Dependency
Domain
# of Relations
Temporal State
#
Structured
Format
Source
(Time Frame)
Examples
general
dynamic (2.6)
8,330
8
(s. r,
Wikidata
(2010- 2023)
static
8,302
biomedical
dynamic (2.3)
7.345
Time Variant
(2020 2024)
14
(s. o,
static
7,345
UMLS
legal
dynamic (1.1)
3,142
6'
X
QA
CFR
(210 2010-2023)
static
3,142
Time Invariant
commonsense
(s, r, o)
invariant
24,788
CSKG
math
12
(5, r. 0)
invariant
2,585
Math-KG
3 CHROKNOWBENCE CONSTRUCTING A BENCHMARK DATASET
In this section, we enumerate the details of constructing CHROKNOW accumulated knowledge benchmark dataset. The CIROKNOW dataset encompasses three key aspects: time dependency (time variant and invariant), multiple domains (general, biomedical, legal, commonsense, and math), and temporal state (dynamic and static). We first categorize knowledge into two groups: knowledge that remains unchanged over time (time invariant) and knowledge that changes over time (time variant). Additionally, we classify domains based on whether they are influenced by the flow of time, considering the specificity of each domain. Finally, we categorize knowledge either changeable (dynamic) steady (static) within the time frame we have set.
a chronologically
3.1 TASK DEFINITION
Our primary focus is time variant knowledge across three domains (general, biomedical, and legal) with comparisons to time invariant knowledge across two domains (commonsense and mathematics). What knowledge would be the difference between time variant and invariant? The time variant knowledge has a specific object changing across a stream of time. For example, "Cristiano Ronaldo (s) was a member of sports team of (r) Manchester United F.C. (or) in 2009 (t,) and Real Madrid CF (ox) ) 2018 (t,))., Particularly, we adopts an object-level focus, emphasizing changes in object attributes such as roles or affiliations. This approach ensures scalability by simplifying temporal dynamics, precision by capturing nuanced updates, and flexibility by supporting fine-grained real- world transformations without rigid relational constraints. Likewise, we identify subject and object alias for each relation, then gather yearly changed objects. After accumulating object lists for 02, ). we de-duplicate and fill out the missing data in specific years based on available data; objects between Manchester United F.C. (0)) ) in 2009 (f,) and Real Madrid CF (oR) in 2018 (tz), missing objects between 2010 (12) to 2017 (lx-1 1) filled with existing object of 2009 (41). The statistic of
NOWBOCNWWENCH dataset is in Table 2 and detail of object-level focus is in Appendix A.3.2
3.2 DATASET GENERATION
To construct dataset, select annual knowledge sources for each domain, possible to be aligned with each elements even though the corpus does not specifically mention about time stamp. For sources with structured triplets, we identify temporal affect-able relations that typically change over time, such as "position held". As time variant knowledge refers to the knowledge that has the potential to change over time, we divide it into two temporal states for more fine grained results: (1) dynamic, where the knowledge has evolved over the period. (2) static, where no change occurred during the period, though has potential be changed. Following the methodology outlined in Section 3.1 we track changes in objects to build the dynamic dataset, employing normalization and de- de-duplication for verification. Each object is checked with strict exact string match. then add into objects pool. Simultaneously, we identified unchanged objects over the same time frame to construct the static dataset. At the end, all data elements consist with an associated object pool {or, O2...., 2m) over time frames { t2, ...,...,. 12, }.
4

---

Published as a conference paper at ICLR 2025
(A) Model-wise
Dynamic
(B) Template-wise
Dynamic Generation
Dynamic NCQA
2010
f1.1
09.1
41.1
47.7
15.6
2016
Gemter
30.3
15.0
35.3
29.4
50.1
23,6
2523
30.3
50.2
55.2
53.1
50.1
41.0
Meee instraa
40.0
37.5
93.1
35.2
375
31.7
thi Semminattaac
31.5
31.4
36.1
30.0
25.2
21.0
Static
Dynamic
35.6
37.5
25.1
40.6
31.3
37
33.1
3033
2013
2036
2002
2822
2023
Static
47540 min
35.5
38.5
20.5
26.2
23.9
31.4
(C) Object Change
31.2
24.7
26.1
MCQA
Generation
38.9
43.3
64.9
46.9
39
40.0
Uamad 240-rattuc
12.8
34.4
34.
28.8
28.5
26.1
27.2%
Makes 10.3
33.2
34.6
29.8
29.0
25.3
29.2
28.9
23.7
18.0
15.1
13.3%
28.5
12.6
31.9
28.9
28.5
25.7
2013
2013
2006
2033
1822
2003
Percontage
Change
Figure 2: Performance analysis of general domain. (A) Heatmap in Generation template. For both dynamic and static datasets, a common trend across models is that performance stronger in the intermediate years but decline recent years, reflecting the data-cutoff point. Dynamic knowledge shows more variation compared to static. Full results of total time frame Figure T0 (B) Template- wise performance for selected years. As time goes by, performance in generation goes low, the other hand, MCQA and TF appeal to be rising. (C) Distribution object changes in dynamic dataset.
3.3 TIME VARIANT & INVARIANT KNOWLEDGE
We sourced time variant knowledge from the general, biomedical, and legal domains. In general domain, we utilize Wikidata (Vrandečić & Krötzsch 2014) dump to track object changes among the time frame using suggested time quantifiers. Collecting similar amounts of dynamic and static instances across eight relations, the result is formatted as (8, .,()) quadruplet for each object and accompanied time stamp. For biomedical domain, we parse Unified Medical Language System (UMLS) (Bodenreider) 2004) metathesaurus data, where suggest yearly updated research in the domain, following previous work of BIOLAMA (Sung et al. 2..I). Due to the slow pace of change in biomedical research, object pools in this domain shows slight expansions contractions over time frame. In the legal domain, we employ the Code of Federal Regulations (CFR) (U.S. Government) to track regulatory changes, they suggest collection and accumulation of change in regulations at the end of year. Starting from pre-processing unstructured xml data, we adopt QA-like format with placeholder for object, tracked among time frame which ends dynamic or static whether change or not. Time invariant knowledge, which remains constant regardless of temporal context, drawn from common-sense and mathematical domains. We process the CSKG (llievski al.) 2021) dataset of commonsense knowledge, and Math-KG (Wang 2022) for covering areas like data structures and pure mathematics. Further details are provided Appendix A.33 especially the sources and the mechanism of object-level focus. Especially, compared to TKGs (Zhang et al.) 2024a), which focus on temporal snapshots, our CHROO BONNNOWIBBIIIII emphasizes the detailed temporal evolution of individual knowledge elements, offering better adaptability for gradual changes; see Appendix A.3.3
EXPERIMENTAL SETUP
We enumerate the nine open-source and two proprietary LLMs: Llama-3. 1-70B-Insttrcc and Llama-
11-8B-Instrue Meta 2024). L-ama-3-8B-- 8B-Inttruct (Dubey et al. 2024). Llama-2 2-rb-chat--hht (Touvron
et al.) 2023b), Mistral Instrucct-OO (Jiang et al. 2023).
(Abdin et al. 2024).. SOLAR -.AAR-00.75 10.7--0BB--- --..B--Instuuceee Instrnnt-ttt.....m (Kim et al. (023), gemma-2-9b-i (Team et al.) (024b). gemma-7h-
it TTeam et al. 2024a) for major open-source models, and GPT-40 mini (OpenAl) 2024a), Gemini- 1.5-flash (Ded 2024) for proprietary models. Each model utilizes either an instruction-tun or chat version to enhance instruction following during sampling. We focus on anlayzing trends in the chronological knowledge captured by those models, differ in corpus coverage. Details of our inference setups are in Appendix A.4
5

---

Published as a conference paper at ICLR 2025
(A) Model-wise
Dynamic
Template-wise
Dynamic Generation
Dynamic NCQA
20.00
48.3
2013
Oeninl Posk
02.3
6.1
63.5
51.3
57.7
3324
41.1
746
18.1
42.
41A
47.2
478
47.0
41.3
41.7
Static
Dynamic
430
65.4
671
16.0
36.1
35.5
2021
1222
Static
40.3
39.7
40.1
31.3
37.0
(C) Object Change
MCDA
Generation
Morw Instvat
53.1
53.9
$3.0
50.5
96.9
Mirro
31.4
304
13.7
30.2
31.6
Melde siinttus
40.7
41.3
39.1
37.2
28.25
41.1
40.8
39.4
36.3
36.2
300
2801
720
200
2001
Irtentage
Nurber Changes
Figure 3: Performance analysis of biomedical domain. The format of figure is same Figure 2 (A) Compared to the general domain, both dynamic and static datasets show lower variability, reflecting a domain--pecific tendency toward consistency in knowledge changes. Both of them shows performance decrease between 2022 and 2023, aligning with the cutoff pattern noted in the general domain. (B) As time goes by, performance in generation declines, but MCQA and TF continue to perform well.
5 CHROKNOWLEDGE: CHRONOLOGICAL CATEGORIZATION OF KNOWLEDGE
In this section, introduce CHROKNOWLEDGE (Chronological Categorization of Knowledge), saapeing-basedd
sampling based framework designed to evaluate LLMs' pon-parmmetric chronological knowledge. Our methodology assesses the temporal capabilities of LLMSs using two distinet templates, detailed in Section 2.2 and explores how current LLMs encapsulate temporal information.
5.1 RESULTS OF REPRESENTING KNOWLEDGE FROM LARGE LANGUAGE MODELS
For testing model's knowledge within the categorization, we sample five times for each knowledge elicit it as possible in dynamic and static dataset. We present our findings across three different aspects: temporal-wise template-wise and domain-wise results. In Figure 2 and 4 we depict the results for all time variant domains; general, biomedical and legal domains in main section. Time invariant datasets are presented in Appendix A.5 Figure
Comparing the upper and lower panels of (A) in Figure and and provides the tendency of temporal-wiss results based on the generation results. common trend across models is decline in performance recent knowledge, reflecting the pretraining corpus cutoff dates. Particularly dynamic datasets, models demonstrate strong performance on earlier knowledge but experience steeper decline in later periods, particularly in both general and biomedical domains. In contrast, static datasets show less fluctuation, with more stable yet weaker performance, highlighting limited temporal sensitivity due to reliance single timestamp. These results emphasize the need for frequent model updates, especially for dynamic knowledge, to ensure temporal relevance.
Temporal-wise Results.
Template- wise Results. (B) of Figure and a provide the average scores of template-wiss results, for more template specificity checking three templates: generation, MCQA, and TF. Generation templates reveal greater decline in recent knowledge, as models rely on internal information without predefined answers. contrast, MCQA and TF templates help models select correct answers from structured options and predefined formats, mitigating some gaps in recent knowledge. This trend more evident in the biomedical domain with MCQA templates and the legal domain with TF, revealing how domain-specifi knowledge is more effectively elicited through specific formats. Also, dynamic dataset the general domain more sensitive to temporal shifts than static, highlighting the importance of task-specific templates in eliciting and improving temporal robustness.

---

Published as a conference paper at ICLR 2025
(A) Model-wise
Dynamic
(B) Template-wise
Dynamic Generation
Dynamic
NCQA
2010
2014
S-ftut
11.3
11.2
12.9
11.6
114
114
2523
Came 31-35 intruct
6.6
82
102
9.0
9.5
9.6
105
95
92
intrvst
5.2
62
69
6.3
63
6.1
Static
Dynamic
152
15.5
15.
14.9
13.1
16.6
19.4
19.4
20.0
20.
303
2016
208
2822
203
Static
CFF-4
40.4
40.0
49.1
41.1
48
42.4
(C) Object Change
MCQA
Generation
34.3%
Uame 2-000 surrutt
58.9
$0.0
59.3
$8.0
58.6
58.7
Notul 18 2
24.2
23.8
23.7
24.7
24.4
23.5
20.9
20.9
20.8
SOLAN insteat o
38.7
35
2
39.2
38.1
38.1
90/6 3016
Vha
1999
3003
Charges
Figure 4: Performance analysis of legal domain. The format of figure same as Figure (A) Among time variant domains. legal domain shows the most stable results of static, while the gap between dynamic and static datasets is the largest among domains. (B) When comes to each template, generation shows the lowest performance, while TF settings perform extraordinarily well in answering correctly. (C) In the legal domain, the distribution shows the lowest number of object changes over time, supporting the conclusion of the stable results in the heatmap.
Domain-wise Results. Comparing the Figure 2 3 and 4 provide the tendency of domain-wise
results, demonstrating distinet domain-specific characteristies of temporal knowledge change. In general domain, the models show a decline in recent knowledge, with a more unstable distribution of scores, which stems from domain's 's nature; changes in relations 'position held' or 'member of sports team' are more sensitive to temporal cues, leading to higher variability. Here, MCQA setting offers some resilience dynamic and static datasets over different years. Every domain shows similar distribution of object changes with benchmark statistics, comparing (C) in each figure with Figure g This indicates that the models perform robustly intended by the benchmark, without bias toward specific low changed objects. We provide more details for legal domain and time-invariant knowledge in Appendix A.5)
mitigates the knowledge decline observed in generation templates across
Overall, domain-specific characteristics significantly influence LLMs' temporal knowledge repre- sentation. More stable domains like biomedical and legal exhibit consistent performance with time invariant knowledge, while general domain shows more inconsistency. These insights underscore the need for tailored strategies to enhance LLMs' temporal knowledge capabilities.
CHROKNOW PROMPT: CHRONOLOGICAL KNOWLEDGE PROMPTING
6.1 CHRONOLOGICAA CATEGORIZATION
In previous section, we demonstrate whether open-source and proprietary models possess specific knowledge at various time stamps. However, this does not sufficiently assess the models' understand- ing of knowledge within chronological progression. As Zhao et al. (2024) suggest, knowledge influenced heavily by temporal factors as general domain can still vary in more stable situation like static dataset. To address this, we first reclassify the models' responses using refined categorization scheme, allowing for a more comprehensive evaluation of temporal knowledge across all years.
@illustrates how it works: (1) Known for the precise temporal alignment if the model correctly identifies all relevant objects for a given knowledge category at each specific year; (2) If model fails to match the correct objects for every year, we refer it as Unknown for indicating incomplete or misaligned temporal knowledge; (3) The model accurately responds either just before or after a specific year but fails for others, signifying outdated information forgotten legacy knowledge due to continuous updates (Cut-off); (4) The model correctly identifies some objects for a given year but not others, reflecting incomplete understanding of the temporal knowledge (Partial Known).
7

---

Published as a conference paper at ICLR 2025
Partial Known Set
Prompt
ChroKnowBench
Timeline
Q. Ta Akwasi
sports team, [Ooject]
Previius Span
Candidate None
S. o T
n2
0n-2
Step
Candidate
In-1, Akwasi Asare, member sports team, [Object]
0111
FC Utrecht
n3
Q. In Tn, Nana Akwasi Asare, member of sports team, [Object]
T,
Target
Candidate
generate C₁
O n
Step 2: Verify Refine Candidate
AP
0x+1
Q. In-z, Nana Akwasi Asare, member sports team, [Object]
FC Utrecht
C
n12
LLM
Q. In 1,-1, Nana Alewasi Asare, member of sports team, [Object]
Span
A. FC Utrecht
mn
In Ta, Nana Akwasi Asare, member of sports team, [Object]
Candidate A. Cj ( (verify & refine C2
Lsing Coneet Areoow for Sew shot
LLM Generates the Outpu
Stop n:
Spans
Cn
Match Answor with Bendhmark
Figure 5: Overview of ChroKnow Prompt. ChropnowPrompt. The algorithm systematically traverses step by step, appending each span's 's correct answer few shot for each steps. The range of each previous and next span is predefined, with the order of nearest time stamp from target Tr. The model suggests last candidate answer C.. verified an refined through several steps, which ends to be checked with the object o, original ChroKnowBeneeh
Our main focus on the Partial Known cat- egory, where models demonstrate substantial temporal knowledge but fail to answer correctly for all years, often showing confusion between correct answers. For example, Nana Akwasi Asare (s) was member of sports team of (r) FC Utrecht (o,) ) in 2011 (t,)) but the model in- correctly identifies the team FC Groningen, despite answering correctly with FC Utrecht for 2010 (fn-1) and in 2012 (fn+1). At this point, we hypothesize that when the model gets one time stamp wrong, more explicit focus on the temporal aspects surrounding that time span could help generate more accurate an- swers. This is the core idea behind CHRO- KNOWPROMPT.
Answwer
Correct A Partial Correct X Incorrect
Cut-OH
Incomplete Temporal Knowledge
A
O
x
X
Partial
Known
O
O
x
A
Known
Targetiof "ChroKnowPrompr"
T
T
T
T
T
Figure 6: Chronological categorization based on each answer with its time stamp. If the model answer rectly for all. it re-categorized Known. The target Khoooootpo Partial Known, which con- fuses its knowledge among the whole time stamps.
6.2 METHOD
We introduce a chronological prompting technique for non- parametraaee method to elicit chronological knowledge, aimed at bridging knowledge gaps by utilizing multiple temporal snapshots. This method enhances the model's 's reasoning by systematically integrating knowledge from different time stamps, enabling in-depth traverse. Our method is inspired by non-pan non-paraeetric editing techniques, such as Zhong et al. (2023) and Zheng et al. 2023), described detail in Appendix A.
Figure illustrates example of the chronological prompting process. From target year In the algorithm systematically traverses the preceding years (nc-1,4-22...3 in the 'Previous Span' and the subsequent years (fn+1, fx+2.... .) in the 'Next Span'. For each traversed year, the most representative object òø is selected from the Correct (represented by circle) and Partial Correct (represented by triangle) categories in Table T Jusing majority voting.
Starting with the initial prompt containing the target time t.,. subject sn, and relation Tn, the nearest year in the previous span is appended above the initial prompt with the selected object, forming the first step. The model then generates a candidate answer C₁ for t, using this augmented prompt. Next, Now PROMPT iteratively adds prompts from progressively earlier years, refining
8

---

Published as a conference paper at ICLR 2025
General Domain
Biomedical Domain
Logal Domain
9pt-4o-mini
4.7
-2.2
+20
114
1+141
gentni-1.--ttah
3165
1354
141
Phi3.5_Mini Mini
-1.8%
20.1
467
120.1
116.7
43
Ua0
4.7
1-12
-1.2
-1.
U
2.8
312.
Gemmaz 98
1.2.
911.
11.7
3-3.8
-0.1
Gemma 78
1244
+-14
-3.1
-15.2
10.71
-44
j-4
-1.1
Uamm2 70
1121
-2.8
26
66
26
25
Dynamic Known Before Dynamic Known Afat
Dynamic Partisl Before
Staric Known Before
Static Partal Before
Dynamic Partial After
Static Known After
Static Paria After
across multiple domains with unchanged objects. For each domain, the left space represents the percentage of Partial Known, and the right represents the percentage of Known. Each model includes results for both dynamic (yellow-blue bar) and static (red-green) datasets, with arrows indicating the actual increase. As shown in plots, the most effective results are observed in the biomedical domain, where the unchangeable characteristic is stronger than the general domain. While the static dataset of the legal domain shows improvement, many models struggle with unstructured format, resulting in the lowest performance among the dynamic dataset.
Figure 7: Results of ChroKnowProme
the candidate answer by verifying its consistency across contexts (from Ci to C2). This backward traversal continues until predefined span range is reached. Once the previous span is completed, our algorithm performs forward traversal by appending objects from subsequent years below the target year, further generating and verifying candidate answers. If there are no previous next years available, the process proceeds on only one side.
Upon completing all traversals, the final candidate answer C, is compared against the benchmark object for fn. If the candidate answer aligns correctly with the object for bn appropriately reflecting the temporal contexts, the knowledge categorization for i, is updated to Chrono-Correct, which is equivalent to Correct for chronological assessments. In Appendix A.7 we provide the detailed steps.
EXPERIMENTAL RESULTS& & ANALYSIS
7.1 RESULTS OF SHIROKNNWPROMPPT
Details of task configuration is in Appendix A.8 preseetss the effect of chronological prompt- ing on unchanged objects across different models. Results show the rise of percentage in Known category with decreasing Partial Known, indicating the increase by Chrono-correct. Significant improvements are observed in the biomedical domain (average increase of 11.5%), while general and legal domains show smaller gains (2.8% and 3.1%, respectively). Both proprietary models (GPT4o- mini, Gemini- 1.5-flash) and the massive open-source model (Llama-3. 1- Instruct perform well, while smaller open-source models like Mlama---B-chaa-2 also show notable improvements despite being outdated. This indicates that chronological prompting effectively enhances knowledge recall without requiring external retrieval systems. But in the legal domain, models struggle to recall knowledge due to the complexity of unstructured, context-rich data, especially for dynamic dataset.
7.2 ANALYSIS
Results in changed objects Taalee5 results for changed and unchanged objects in dynamic datasets across domains. As shown in Section 77.1 ShroKnowPrompp effectively aids models in recalling unchanged objects in both dynamic and static datasets. However, its performance on changed objects in dynamic datasets remains limited, achieving only 10-30% performance of unchanged object cases (an average of 0.4 in general domains). This highlights, despite the helpfulness and applicability of ChroKnowPrompp in addressing the chronological gaps of knowledge, recalling all historical changes of objects using prompting alone remains an exceptionally challenging problem. particularly in complex contexts such as the legal domain. These findings emphasize the need for further research into effective parametric editing techniques to assist temporal knowledge handling.
9

---

Published as a conference paper at ICLR 2025
Effects of chronological span To elucidate the mechanisms of chronological prompting, analyze the impact of incorporating the next span in chronological contexts. As shown in Table 6 the total span (both previous and next) yields higher scores than using only the previous span with the degree of improvement varying by domain. In the biomedical domain, the total span nearly doubles the score of the previous span alone (12.0 6.7), while the general domain shows a modest increase (2.8 vs. 1.8.. Model-specific temporal sensitivity also varies: (lama-2-7b-chat-h effectively utilizes next spans, whereas Gemini-L.5-lash and SOLAR-I0 7B-Instruct-v 1.0 benefit more from previous spans. These suggest that differences in temporal context utilization and the coverage of pretraining corpus may influence models' sensitivity and knowledge recall across time frames.
Effects of chat prompting Additionally, we analyze various chat models (before instruction- tuning), including Llama-2-7b-chat-hh as chronological prompting may enhance both current and legacy models by leveraging temporal context. We evaluate three open-source chat models: mpt-7b- chat (Research] 2023). Pythia-Chat-Base-
(Biderman et al., 2023), and mennotron---b-chat--ee and ChroknowPromm is not partieularly effective for chat models. Only mpt-7b-chat achieves a comparable peformance to Llama-2- 2-bb-chat-hhh (an average increase of 1 11.4), while Satii-Chat-BBae-- shows almost no improvement.
hf (Zhang et |l.]|2223ai 2023a). As shown in Table 6
RELATED WORK
Since the emergence of LMs, deriving knowledge from language model is extensively studied, such as probing tasks (Hewitt& & Manning 2019). LAMA (Petroni et al. 2019) and BioLAMA (Sung et al, 2021 Then, many subsequent studies follows exploit, (1) how LLMs define knowledge Yu et al.
2023;
Zhang et al. [223b] Gottesman & Geva 2024). (2) how these models represent (Chen et al.
2024a bJ Wang et al. [022dd) and (3) how manipulate misleading part (Wang al. 2023 Gutiérrez et al. 2024 Wu et al. (024a). Based on them, recent investigations of knowledge highlight the dynamic nature of evolving facts and suggests that contradictions within the training data may lead to knowledge conflicts (Marjanović et al. 2024; Chang et al., (224] Waag et al. [024aa] Xu et al. 2024
Jin et al., 2024). And knowledge overshadowing (Zhang et al. (024c) reveals phenomena where certain conditions overshadow other facts, leading to misleading information (i.e., hallucinations).
In other view point, exploring temporal knowledge starts from using Wikidata, a static format of knowledge in triplet: subject, relation, and object, originated from extracting literature- based knowledge (Hahn-Powell et al.] (017)). Following pioneers like TimeQA (Chen et al. 202I) and TemporalWite (Jang et al. 2022a), many works dealing with temporal and continuous knowledge
flow (Zhang& & Choi 2021 Dhingra et al., 2022; Jang et al. 2022b; Liska et al. 2022 Nylund et al.)
2023 Zhu et | [023] Khodja et al.| 2024 Zhang et al. 2024d) consist in line of it. Building upon their achievement, CirpeDiem (Kim et al. 2024) emerges to simply identify whether knowledge is outdated or not, and DyKnow (Mousavi et al.) (024) maps various models' knowledge distribution. Also, (Zao et al. (224)) makes dramatic work: align model into one fixed age. Though those impressive works, we try to broad the coverage of temporal knowledge. We utilize various templates to elicit the knowledge of LLMs, broaden the coverage of time stamps, and differentiate domains that should change based on a temporal perspective with those that should remain constant.
CONCLUSION, LIMITATION, AND FUTURE WORK
Overall, our work highlights the critical role of temporal context in knowledge evaluation and introduces a framework for improving the temporal capabilities of future language models. We present STROKNOWBBR
a benchmark for assessing temporal knowledge across diverse domains, and our CHROKNOWLEDGG framework. which evaluates LLMs' chronological knowledge through three types of templates. Our findings indicate that while models often recall facts with time stamps, they struggle with capturing full temporal boundaries, with reliance on rigid formats like MCQA and TF. By using CHROKNOW PROMPT, we improve knowledge recall by reducing ambiguous Partial Known and increasing Known, particularly in biomedical domains with strong performance on unchanged objects in both proprietary and open-source models. However, challenges persist in dynamic datasets and in unstructured, context-rich formats, which amplify the difficulty of capturing temporal evolution solely with prompting. Future work will focus on the need for parametric techniques to complement prompting, enabling better alignment with changing objects and complex temporal dependencies to enhance I.LMs' temporal accuracy across various domains.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported in part by the National Research Foundation of Korea [NRF- 2023R T2230AA2C300176, RS 20-2002-00262002), the Ministry of Health & Welfare, Republic of Korea [HR20C002103] and the ICT Creative Consilience program through the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the MSIT [IITP-2025- 2020-0-01819).
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404. /42 2024.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
Olivier Bodenreider. The unified medical language system (umls); integrating biomedical terminology. Nucleic acids research, 2004.
Hoyeon Chang, Jinho Park, Seonghyeon Ye, Sohce Yang, Youngkyung Seo, Du-Seong Chang, and Minjoon Seo. How do large language models acquire factual knowledge during pretraining? arXiv preprint arXiv:2406. / 1813, 2024.
Jianhao Chen, Haoyuan Ouyang. Junyang Ren, Wentao Ding, Wei Hu, and Yuzhong Qu Timeline- based sentence decomposition with in context learning for temporal fact extraction. In Lun-Wet Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand, August 2024a. Association for Computational Linguisties.
Wenhu Chen, Xinyi Wang, and William Yang Wang. dataset for answering time-sensitive questions. arXiv preprint arXiv: 100.0000000664 /4,44,22406 2021.
Yuheng Chen, Pengfei Cao, Yubo Chen, Yining Wang. Shengping Liu, Kang Liu, and Jun Zhao Cracking factual knowledge: A comprehensive analysis of degenerate knowledge neurons in large language models. arXiv preprint arXiv:2402. 13731, 2024b.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, 2022.
Google DeepMind. Gemini flash: Lightweight models, two variants, both optimized for speed and efficiency. 2024.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finetuning of quantized L.Mss. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL URL htt.hhttts///oeeeeiiiieeee nwt. forum?i S=O
Bhuwan Dhingra, Jeremy Cole, Julian Martin Eisenschlos, Dan Gillick, Jacob Eisenstein, and William Cohen. Time-aware language models as temporal knowledge bases. Transactions of the Association for Computational Linguistics, 10, 2022.
Abhimanyu Dubey, Abhinay Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The laaa 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
Bahare Fatemi, Mehran Kazemi, Anton Tsitsulin, Karishma Malkan, Jinyeong Yim, John Palowitch, Sungyong Sco, Jonathan Halcrow, and Bryan Perozzi. Test of time: A benchmark for evaluating lims temporal reasoning. arXiv preprint Xxxv:2406.001770 2024.

---

Published as a conference paper at ICLR 2025
Xiou Ge, Ali Mousavi, Edouard Grave, Armand Joulin, Kun Qian, Benjamin Han, Mostafa Arefiyan, and Yunyao Li. Time sensitive knowledge editing through efficient finetuning. arXiv preprint arXiv:2406. 200e:2406.44496, 044904449666. 2024.
Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig. Does fine-tuning Ilms on new knowledge encourage hallucinations? arXiv preprint arXiv:2405.05904, 2024.
Mor Geva, Jasmijn Bastings, Karja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. In Proceedings of the 2023 Conference on EMNLP, 2023.
Gaurav Rohit Ghosal, Tatsunori Hashimoto, and Aditi Raghunathan. Understanding finetuning for factual knowledge extraction. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research. PMLR, 21-27 Jul 2024.
Daniela Gottesman and Mor Geva. Estimating knowledge in large language models without generating single token. arXiv preprint arXiv:2406. /2673, 2024.
Bernal Jiménez Gutiérrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. Hipporag: Neurobio- logically inspired long-term memory for large language models. arXiv preprint 14X1::2405.14811, 2024.
Gus Pownll Marco A Ailenzucla-Escárcege and Mihai Surdeanu. Swanson linking revisited: Accelerating literature- based discovery across domains using a conceptual influence graph. In Proceedings of ACL 2017, System Demonstrations, 2017.
Tom Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. Aging with grace: Lifelong model editing with discrete key-value adaptors. Advances in Neural Information Processing Systems, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Confer- ence on Learning Representations, 2021. URL https: http:::tt////ootttet http::::///peneeiee... net fareet/oooumiiommm
d7KB jmI 3GmQ,
John Hewitt and Christopher D Manning. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers),
2019.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https:t//////...ttt /otp:rsi///peeeviee.. net/ forum?
id=nZevKeeFYff9
Xiusheng Huang, Jiaxiang Liu, Yequan Wang, and Kang Liu. Reasons and solutions for the decline in model performance after editing, arXiv preprint arXiv:2410. 233443, 2024.
Filip Ilievski, Pedro Szekely, and Bin Zhang. Cskg: The commonsense knowledge graph. In The Semantic Web: 18th International Conference, ESWC 2021, Virtual Event, June 6-10, 2021, Proceedings 18. Springer, 2021.
Joel Jang, Seonghyeon Ye, Changho Lee, Sohee Yang, Joongbo Shin, Janghoo Han, Gyconghun Kim, and Minjoon Seo. Temporalwiki: A lifelong benchmark for training and evaluating ever- evolving language models. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, 2022a.
Joel Jang, Seonghyeon Ye, Sohee Yang. Joongbo Shin, Janghoon Han, Gyeonghun Kim, Jungkyu Choi, and Minjoon Seo. Towards continual knowledge learning of language models. In 10th International Conference on Learning Representations, ICLR 2022. International Conference on Learning Representations, 2022b.
12

---

Published as a conference paper at ICLR 2025
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
Zhuoran Jin, Pengfei Cao. Yubo Chen, Kang Liu, Xiaojian Jiang, Jiexin Xu, Li Qiuxia, and Jun Zhao. Tug-of-war between knowledge: Exploring and resolving knowledge conflicts in retrieval- augmented language models. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC- 2024), 2024.
Jachun Jung, Jinhong Jung, and U Kang. T-gap: Learning to walk across time for temporal knowledge graph completion. arXiv preprint 20Ki::2012.105955 2020.
Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Velocity Yu, Dragomir Radev, Noah A Smith, Yejin Choi, and Kentaro Inui. Realtime qa: what's the answer right now? In Proceedings of the 37th International Conference on Neural Information Processing Systems, 2023.
Hichem Ammar Khodja, Frédéric Bechet, Quentin Brabant, Alexis Nasr, and Gwénolé Lecorvé. Wikifactdiff: A large, realistic, and temporally adaptable dataset for atomic factual knowledge update in causal language models. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation COLLIG 2024), 2024.
Dahyunn Kim, Chanjun Park, Sanghoon Kim, Wonsung Lee, Wonho Song, Yunsu Kim, Hyeonwoo Kim, Yungi Kim, Hyeonju Lee, Jihoo Kim, et Solar 10.7 b: Scaling large language models with simple yet effective depth up-scaling. arXiv preprint arXiv:23122 2.1.. 15 166, 2023.
Yujin Kim, Jaehong Yoon, Seonghyeon Ye, Sangmin Bae, Namgyu Ho. Sung Ju Hwang, and Se- Young Yun. Carpe diem: On the evaluation of world knowledge in lifelong language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Pupers), 2024.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems
Principles, pp. 61 611-226, 2023.
Zixuan Li. Xiaolong Jin, Wei Li, Saiping Guan, Jiafeng Guo, Huawei Shen, Yuanzhuo Wang, and Xueqi Cheng. Temporal knowledge graph reasoning based on evolutional representation learning. In Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval, pp. 408-417, 2021.
Adam Liska. Tomas Kocisky, Elena Gribovskaya, Tayfun Terzi, Eren Sezener, Devang Agrawal, D' Autume Cyprien De Masson, Tim Scholtes, Manzil Zaheer, Susannah Young, et al. Streamingqa: benchmark for adaptation to new knowledge over time in question answering models. In International Conference on Machine Learning. PMLR, 2022.
Potsawee Manakul, Adian Liusie, and Mark Gales. Selfcheckgpt: Zero-resource black-box hallucina- tion detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 9004-9017, 2023.
Sara Vera Marjanović, Haeun Yu, Pepa Atanasova, Maria Maistro, Christina Lioma, and Isabelle Augenstein. From internal conflict to contextual adaptation of language models, arXiv preprint arXiv:2407.17023, 2024.
Nick Mecklenburg, Yiyou Lin, Xiaoxiao Li, Daniel Holstein, Leonardo Nunes, Sara Malvar, Bruno Silva, Ranveer Chandra, Vijay Aski, Pavan Kumar Reddy Yannam, al. Injecting new knowledge into large language models via supervised fine-tuning, arXiv preprint arXiv:2404.00213, 2024.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35, 2022.
Kevin Meng, Amab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. In The Eleventh International Conference on Learning Representations, 2023.
13

---

Published as a conference paper at ICLR 2025
Meta. Introducing Ilama 3.1: Our most capable models date. 2024.
Abhika Mishra, Akari Asai, Vidhisha Balachandran, Yizhong Wang, Graham Neubig, Yulia Tsvetkov, and Hannaneh Hajishirzi. Fine-grained hallucination detection and editing for language models. arXiv preprint 0tti::2401.068555 2024.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing at scale. In International Conference on Learning Representations, 2022a. URL
https: wwwpreeeeeeeeeeeeeeeeeeeeeeeeeee net / forummiiieooooo..m (rttumidd=DDcca forumiid=0DczzeWEorr
Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and Chelsea Finn. Memory- based model editing at scale. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, Proceedings of Machine Learning Research. PMLR, 17-23 Jul 2022b.
Seyed Mahed Mousavi, Simone Alghisi, and Giuseppe Riccardi. Is your IIm outdated? benchmarking Ilms & aligmment algorithms for time-sensitive knowledge. arXiv preprint arXiv:240..08700, 2024.
Kai Nylund, Suchin Gururangan, and Noah Smith. Time is encoded in the weights of finetuned language models. arXiv preprint 1rXxix23322.34400 2023.
Yasumasa Onoe, Michael Zhang, Shankar Padmanabhan, Greg Durrett, and Eunsol Choi. Can lms learn new entities from descriptions? challenges in propagating injected knowledge. In Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023.
OpenAL. Introducing chatgpt. 2022.
OpenAI. Gpt-4o mini, advancing cost-efficient intelligence. 2024a.
OpenAl. Openai ol system card. 2024b.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)) 2019.
Katherine Picho, Lauren A Maggio, and Anthony R Artino. Science: the slow march of accumulating evidence. Perspectives on medical education, 2016.
Mosaic AI Research. Introducing mpt-7b: A new standard for open-source, commercially usable 1lms. 2023.
Jennifer Rowley. The wisdom hierarchy: representations of the dikw hierarchy. Journal of information science, 2007.
Mujeen Sung, Jinhyuk Lee, S Yi Sean, Minji Jeon, Sungdong Kim, and Jaewoo Kang. Can language models be biomedical knowledge bases? In 2021 Conference on EMNLP 2021. Association for
Computational Linguisties (ACL), 2021.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk. Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based gemini research and technology. arXiv preprint 20X.X:::00000295.. 2024a.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at practical size. arXiv preprint 208.0.2408.00111, 2024b.
14

---

Published as a conference paper at ICLR 2025
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXhv:2302.13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307. 09288, 2023b.
U.S. Government. Electronic code of federal regulations. URL htt.::t::/// www. wwtp:/ww.w.cc.oo////oo/
Denny Vrandečié and Markus Krötzsch, Wikidata: a free collaborative knowledgehase. Communica- fions of the ACM, 2014.
Jianing Wang. Math-kg: Construction and applications of mathematical knowledge graph. arXiv preprint arXiv:2205.03777: 2022.
Mengru Wang, Yunzhi Yao. Ziwen Xu, Shuofei Qiao, Shumin Deng, Peng Wang, Xiang Chen, Jia-Chen Gu, Yong Jiang, Pengjun Xie, et al. Knowledge mechanisms in large language models: A survey and perspective. arXiv preprint arXiv:2407.15017, 2024a.
Peng Wang, Zexi Li, Ningyu Zhang, Ziwen Xu, Yunzhi Yao, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. Wise: Rethinking the knowledge memory for lifelong model editing of large language models. arXiv preprini 2405 14768, 2024b.
Peng Wang, Ningyu Zhang, Bozhong Tian, Zekun Xi, Yunzhi Yao, Ziwen Xu, Mengru Wang. Shengyu Mao, Xiaohan Wang. Siyuan Cheng, Kangwei Liu, Yuansheng Ni, Guozhou Zheng, and Huajun Chen. EasyEdit: An easy-to- use knowledge editing framework for large language models. In Yixin Cao, Yang Feng, and Deyi Xiong (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, August 2024c. Association for Computational Linguistics.
Xiaohan Wang, Shengyu Mao, Ningyu Zhang, Shumin Deng, Yunzhi Yao, Yue Shen, Lei Liang, Jinjie Gu, and Huajun Chen. Editing conceptual knowledge for large language models. arXiv preprint arXiv:2403.0665 06259, 2024d.
Yike Wang, Shangbin Feng, Heng Wang, Weijia Shi, Vidhisha Balachandran, Tianxing He, and Yulia Tsvetkov. Resolving knowledge conflicts in large language models. arXiv preprint 1rX1v:2310.00935. 2023.
Kevin Wu, Eric Wu, and James Zou. How faithful are rag models? quantifying the tug-of-war between rag and Ilms' internal prior. arXiv preprint arXiv:2404. 10198, 2024a.
Xiaobao Wu, Liangming Pan, William Yang Wang, and Anh Tuan Luu. Updating language models with unstructured facts: Towards practical knowledge editing. arXiv preprint arX1v:2402.18909, 2024b.
Rongwu Xu, Zehan Qi, Cunxiang Wang, Hongru Wang, Yue Zhang, and Wei Xu. Knowledge conflicts for IIms: A survey, arXiv preprint arXiv:2403.08319 2024.
Lang Yu, Qin Chen, Jie Zhou, and Liang He. Melo: Enhancing model editing with neuron-indexed dynamic lora. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024.
Qinan Yu, Jack Merullo, and Ellie Pavlick. Characterizing mechanisms for factual recall in language models. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
David Zeigler. Evolution and the cumulative nature of science. Evolution: Education and Outreach, 2012.
Jinchuan Zhang, Bei Hui, Chong Mu, Ming Sun, and Ling Tian. Historically relevant event structuring for temporal knowledge graph reasoning. arXiv preprint arXiv:2405.106211 2024a.
Michael Zhang and Eunsol Choi. Situatedqa: Incorporating extra-linguistic contexts into qa. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7371-7387, 2021.
15

---

Published as a conference paper at ICLR 2025
Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang, Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, et al. A comprehensive study of knowledge editing for large language models. arXiv preprint arXiv:2401.01286 2024b.
Vivienne Zhang, Shashank Verma, Neal Vaidya, Abhishek Sawarkar, and Amanda Saunders. Nvidia ai foundation models: Build custom enterprise chatbots and co-pilots with production-reate Ilms. 2023a.
Yuji Zhang, Sha Li, Jiateng Liu, Pengfei Yu, Yi Fung, Jing Li, Manling Li, and Heng Ji. Knowledge overshadowing causes amalgamated hallucination in large language models. arXiv preprint arXiv:2407.08039, 2024c.
Zhihan Zhang, Yixin Cao, Chenchen Ye, Yunshan Ma, Lizi Liao, and Tat-Seng Chua. Analyzing temporal complex events with large language models? benchmark towards temporal, long context understanding. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume Long Papers), Bangkok, Thailand, August 2024d. Association for Computational Linguistics.
Zihan Zhang, Meng Fang, Ling Chen, Mohammad-Reza Namazi-Rad, and Jun Wang. How do large language models capture the ever-changing world knowledge? a review of recent advances. In Proceedings of the 2023 Conference EMNLP, pp. 8289 8311, 2023b.
Bowen Zhao, Zander Brumbaugh, Yizhong Wang, Hannaneh Hajishirzi, and Noah Smith. Set the clock: Temporal alignment of pretrained language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics ACL 2024, Bangkok, Thailand and virtual meeting, August 2024. Association for Computational Linguistics.
Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. Can we edit factual knowledge by in-context learning? Proceedings of the 2023 Conference on EMNLP, 2023.
Danna Zheng, Mirella Lapata, and Jeff Z Pan. Large language models as reliable knowledge bases? arXiv preprint wxXi:2247.133788 2024.
Zexuan Zhong, Zhengxuan Wu, Christopher Manning, Christopher Potts, and Danqi Chen. MQUAKE: Assessing knowledge editing in language models via multi-hop questions. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, Singapore, December 2023, Association for Computational Linguistics.
Yuxuan Zhou, Xien Liu, Chen Ning, and Ji Wu. Multifaceteval: Multifaceted evaluation to probe Ilms in mastering medical knowledge, In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24. International Joint Conferences on Artificial Intelligence Organization, 2024. doi: 10.249663fjcai URL httpp:://oi..oog///////o 24963/1 ijcai. wca..........31111..
Xinyu Zhu, Cheng Yang, Bei Chen, Siheng Li, Jian-Guang Lou, and Yujiu Yang. Question answering as programming for solving time-sensitive questions. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, 2023.
A APPENDIX
A.1 SUPPLEMENTARY STUDIES IN KNOWLEDGE EDITING
A.1.1 PARAMETRIC KNOWLEDGE UPDATE
Considering update of LLMs are in two types, parametric and non-parametric WWng et al. 2024c),
classical way of parametric update is using fine-tuning (Ghosal et al. 2024 Mecklenburg et al. 2024
Ge et al. 2024). While the method extends to LoRA (Hu al. 2022). QLoRA (Dettmers al. 2023)
and Melo (Yu et al. 2024), as well as continual learning approaches such as GRACE (Hartvigsen
et al. 2024) and WISE (Wang al. 2024b), the parameter accessibility of open-source L. like
16

---

Published as a conference paper at ICLR 2025
the Llama series (oovrro al. 2023a) enables techniques such as MEND (Mitchell al. (022a), ROME (Meng et al. 2022), and MEMIT Meng et al. 2023) to emerge. Those local editable methods effective, and still try improve specificity and generalizability.
A.1.2 NON-PARAMETRIC KNOWLEDGE UPDATE
In contrast, for black-bon box LLMs, updates rely on non- con-paraettric knowledge methods (Onoe et al. 2023), such SERAC (Mitchell al. 2022b), MeLLo (Zhog al., 2023), and IKE (Zheng et al. 2023). They align with two key trends: (1) Mitigating catastrophic forgetting, where the model loses previous knowledge, by not directly updating parameters.
(2) Exploiting abilities of prominent black-box LLMs like GPT-ol (OpenAl 2024b) and Gemini (Team et al.) 2023), as we cannot access to parameters. Another concern in knowledge update they often focus only structured format, pointed out that current methods struggle to update unstrctured data effectively (Wu et al. (024b). In this paper, we focus on non- non-parametric knowledge updates accomodating broad range of input formats (structured and unstructured) to represent knowledge across diverse domains, depending the use of various white-box and black-box LLMs.
A.2 DETAILS ELICITING KNOWLEDGE: FEW-SHOT EXEMPLARS, FUZZY MATCHING RULES, AND EXAMPLES OF THREE TEMPLATES
A.2.1 FEW-SHOT EXEMPLARS
To obtain the few-shot exemplar pool D, we leverage additional data collected using the same process BENKC.BBENC.....
as in CHROKNOW BENCH. Specifically, for each individual relation type, We gather four exemplars for the general domain, and eight for the biomedical and legal domains. We then generate the few-shot exemplar set D, by sampling four exemplars from D, which serves actual demonstrations within prompt. This process is repeated for every timestamp to ensure comprehensive temporal coverage.
A.2.2 FUZZY MATCHING
We utilize the rapi dfuzz library to compare the model's responses with the predefined labels. As the model's answer may little bit different with complicated objects in specialized domains, such as the difference in order of words or upper and lower cases, using fuzzy match enables more rapid but still reliable quality without facilitating external NLI mechanisms.
Specifically, we employ a woken-set.rate io metric with threshold value set to 70 to determine a match. token set. ratio is metric used for comparing the similarity of two strings in flexible manner, extending the functionality of the token. sort carrtraaiioo In the preprocessing stage, the strings undergo tokenization, removal of punctuation, and conversion lowercase. The tokens then sorted in alphanumeric order before the similarity ratio is computed. This makes it useful for comparing strings where the word order may differ but the content is similar.
The key distinction of token_set -raei lies in its incorporation of set operations, where du- plicate words removed. After eliminating repeated tokens, the same preprocessing steps as in boken_sort_rate io are applied. When performing the comparison, the method checks if all tokens from the shorter string are contained within the longer string, making the approach particularly suited for cases where one string is a subset of the other. This flexible matching often results in higher accuracy for comparing strings with similar content but different structures, as illustrated by the example where score of 100 is achieved when all tokens from the second string are present in the first
A.2.3 EXAMPLES OF THREE TEMPLATES
We provide three templates of Generation, MCQA, and TF in the end of the Appendix for the better readability. For example, in Table g and Table 9 our target year is 2020 (r) to generate answer candidate of position held (r) by Donald Tusk (s).
17

---

Published as a conference paper at ICLR 2025
A.2.4 ITERATIVE DISTRACTOR GENERATION
For the Commonsense dataset, the objects corresponding given subject and relation are often ambiguous. When constructing compelling distractors, there is a higher likelihood (about 20%) of creating options that are actually correct answers rather than intended incorrect ones, compared to other datasets. Therefore, we include an additional verification process after generating the distractors, as outlined Algorithm T Specifically, we formulate multiple-choice questions using the problem and the generated distractors, then use GPT-4o to select all correct answers. If it identifies more than one correct answer, we refine the distractors based on a prompt to recreate incorrect options.
Algorithm 1: Iterative Distractor Generation Algorithm
Data: Subject s, Relation r, Set of correct objects Comeet
Result: Refined multiple-choiee question
Initialize conversation history H.
2 Initialize number selected options n 4-
5 while a > 0 do
4
D + L...Respoosc((+ (s, r. H) // Generate three incorrect options
g + ComposeQuestion(s, D) // Compose question using the generated distractors
Append y
7 <- LLMRespouse (q) // LLM generates a response by solving the question
S + LLMResponse(-))
Extract selected options using LLM
n < [S] Number of options selected by LLM
if > 0 then
p CreatePrompt(s,r)
Create another prompt for regenerating distractors Append p to H. Add regeneration prompt to conversation history
12
Dnen I.I.Respooss((,, H) Generate new set distractors
14
D[S] + D [1 rt] Replace selected options
return g
A.3
DETAILS OF BENCHMARK DATASET
A.3.1 STATISTICS OF OBJECT CHANGES IN DYNAMIC DATASET
The statistics of object changes among dynamic dataset in three time variant domains are in Figure g The figure shows how many objects have been changed among the time frame of each elements, and its distribution is proposed with the percentage of that elements across total number of dataset. The average number of object changes is 2.6 and 2.3 for general and biomedical dynamic dataset, while most of the element in legal domain has only one object changes among time frame. The biomedical domain shows the least skewness with a balanced cumulative distribution of changes, unlike the general domain, which is moderately skewed with broader change frequencies. The legal domain is highly skewed, with most changes concentrated in a single occurrence, lacking cumulative progression.
General Domain
Blomedicall Domain (p=2.34 24, sktw=0018))
Legal Domain (u=11.09, d=0.45,
1,52.58,
45%
605
60/4
60%
40%
40%
21:01
45%
1441
14
3.4
2,0
2.5
10
3342
Figure 8: Statistics of object changes among dynamic dataset in three time variant domains.
18

---

Published as a conference paper at ICLR 2025
A.3.2 OBJECT-LEVEL FOCUS IN BENCHMARK GENERATION
Our CIROKNNWBENCI emphasizes object-level changes the core metric for assessing temporal knowledge dynamics. This approach reflects a deliberate balance between scalability, precision, and interpretability, aligning with the methodological goals of the benchmark.
Design and Scope The benchmark evaluates how well models can align and reason about temporal knowledge by focusing on object-level transformations at specific time points. For instance: Query 1: "In 2001, Zidane was player at Real Madrid." " and Query 2: "In 2010, Zidane was coach." " By treating these as distinct evaluations, CIROK BENC roles iifiliations-whie keeping the subject and relation fixed. This design ensures structured and scalable evaluation process that captures significant, interpretable changes in temporal knowledge.
BENCH isolates object-level transitions-
as
Reason of focusing on Object-Level Changes
The object-level focus simplifies the complexity of tracking temporal dynamics in subject-relation-objeee triples. By avoiding the combinatorial challenges associated with relational changes, this approach ensures clear and scalable evaluations.
Scalability:
Precision:
Object-level changes, such as shifts in roles or affiliations, capture more nuanced updates compared relational changes, which often default binary states (e.g., "is player" + "is not a player"). This granularity enhances the depth of the evaluation.
Unlike traditional Temporal Knowledge Graphs, which may impose rigid relational structures, the object-centered approach accommodates fine- fine-grained changes in knowledge. This flexibility enables precise and interpretable assessments, particularly for dynamic, real-world transformations.
Flexibility:
By centering evaluations on object-level changes, CHROK BENCH delivers a robust frame- work for measuring temporal knowledge dynamics, balancing methodological rigor with practical scalability.
A.3.3 COMPARISON WITH TEMPORAL KNOWLEDGE GRAPII
Temporal Knowledge Graphs (TKGs) are one of the well known approach for addressing temporal knowledge (Jung et al. 2020 Li et al. 2021 Zhang al. 2024a Our CHROK BENCH also aim to model time-sensitive Enowledge, but we adopt different approaches to structuring and interpreting temporal information. In this section, we compare these two paradigms across several dimensions, with particular attention to their handling of temporal snapshots, knowledge dynamics, and suitability for domains such as biomedical data.
Temporal Snapshot vs. Knowledge- CCetrrcc Tracking TKGs organize events based temporal snapshots, incorporating multiple events occurring within the same timestamp into a single graph representation. This approach emphasizes the relationships between events at specific time point, making it highly suitable for scenarios where the context of concurrent events is critical (e.g.. (North America, Host visit, Business Africa, 2010), (Barack Obama, Consult, North America, 2010)) (Zhang et al.) (2244)). By connecting these events, TKGs enable reasoning about their inter-dependencies and broader temporal patterns.
In contrast, CHROK BENCH adopts a knowledge-centric perspective, focusing on the temporal evolution of individual knowledge elements (s, r, 0) over time. Instead of aggregating multiple events into a single snapshot, across a timeline like the example in Section 3.1 This approach highlights the evolution of specific knowledge and ensures comprehensive tracking of its temporal progression.
BROOKNOWBENTT tracks changes in object (o) values for each relation (r)
Handling Specific Domains A key limitation of TKGs lies in their reliance on well-defined temporal snapshots. While this approach is effective for aggregating and reasoning about concurrent events, it becomes less suitable for domains like biomedical data, where changes often unfold gradually over time and not tied to distinct temporal snapshots. For instance, the development of a medical treatment over a decade may involve incremental advancements that cannot be neatly encapsulated within discrete, event-based snapshots.
19

---

Published as a conference paper at ICLR 2025
Table 3: Comparison of TKGs and CHROKNOW BENCH based on their temporal modeling approaches. TKGs focus on temporal snapshots aggregating multiple events, while CHROKNOWBENCC empha- sizes tracking the temporal evolution of individual knowledge elements. This table highlights their strengths, weaknesses, and domain applicability.
Aspect
TKGs
BENOKNOWBENCH
Temporal snapshots aggregating
Temporal evolution of individual knowledge elements
Temporal Focus
multiple events
Suitable for well- defined, event-rich domains
Applicable not only temporal, but also gradual or implicit changes (e.g.. biomedical, legal)
Domain Applicability
(e.g,, geopolitical, social networks)
Handling of Gradual Changes
Limited by snapshot granularity
Effective through continuous tracking
May overlook fine- tre-ggaiied
May overlook broader
Limitations
changes in individual knowledge
event interdependencies
CHROKNOWBENCE overcomes this limitation not merely by dividing time into yearly intervals, but by prioritizing the temporal progression of individual knowledge elements. This distinction lies in its focus on tracking and organizing the dynamic and static changes of specific objects over time. While TKGs aggregate multiple concurrent events within single snapshot, CHROKNOW KNowkENNWC constructs yearly object pools that capture the fine- fine-graine grained evolution of a specific knowledge element across its temporal trajectory. These object pools allow CHROK NoTOKNOWBBRRR BENCH explicitly track updates and fill gaps in data, ensuring a cohesive and complete representation of knowledge.
Furthermore, CHOO gorizing knowledge based on its temporal variability. This approach enables detailed modeling of knowledge that evolves gradually while preserving distinctions from knowledge that remains unchanged over time. By avoiding the rigid aggregation of unrelated events and instead focusing on the chronological development of individual elements, CHROKNOW provides more precise framework for fine- treeggraned analysis in those specialized domains.
incorporates the concept of dynamic and static datasets, cate-
Comparative Summary TKGs excel at modeling inter-event relationships within temporal snap- shots, making them effective for domains where concurrent event dependencies are critical, such as geopolitical analysis. However, they struggle with gradual changes and unstructured data, limiting their applicability in multiple domains. In contrast, temporal evolution of individual knowledge units, leveraging object pools to capture gradual changes effectively, particularly in specialized domains like biomedical, and legal regulations.
KNOW>BNCC focuses on the detailed
Table 3 provides a comparative overview of the two paradigms.
A.3.4 SOURCE AND APPROACH OF BIOMEDICAL DOMAIN
In the biomedical domain, follow previous work of BIOLAMA (Sung et al. 2021) framework to parse Unified Medical Language System (UMLS) yearly metathesaurus data. In the range of 2020 to 2024, gather instances in 14 relations, resulting 7k for each dynamic and static dataset. Here, by considering domain specificity that the slow pace of change typical of long-term research, the object pool is slightly expanded or narrowed in that period; Autonomic nerve structure, with the relation has indirect procedure site, has a slightly broader scope in 2024, including additional objects like Neurolytic autonomic nerve block alongside previous objects such as Intravenous regional autonomic block. The format is same with general domain, {s, r, o, f quadruplet.
A.3.5 SOURCE AND APPROACH OF LEGAL DOMAIN
In the legal domain, create benchmark dataset based on the Code of Federal Regulations (CFR) from 2010 to 2023. We first extract paragraph-level data from regulatory documents for each year and employ Python's difflib library to detect changes between paragraphs across adjacent years (e.g., 2011 1 to 2012). Careful filtering is applied to ensure that only paragraphs with minor modifications (e.g., single-wore word updates or subtle phrasing changes) retained.
To further analyze the dataset, we utilize the spaCy en core web Ig model to detect named entities in the paragraphs and assess whether these changes involve modifications to the detected entities.
20

---

Published as a conference paper at ICLR 2025
Despite noise introduced by the NER model, initially identify around 56K changes for near- comparisons. These changes are grouped into sequences of years to track alterations over time, while filtering out paragraphs that are introduced removed in intermediate years. Ultimately, we focus paragraphs present in all years between 2010 and 2023, resulting in 8,793 paragraphs.
We then apply GPT--4o-inni to assess whether the detected changes semantically meaningful, excluding minor corrections like typographical fixes or abbreviations. This results a refined set of 4,362 meaningful updates. Additionally, we select 4,746 unchanged paragraphs containing entities detected by the NER model. For each paragraph, we format the changes as fill-in-the-blant tasks, where the modified part is replaced with a blank. providing a rich resource for studying legal text evolution over time.
A.3.6 SOURCE AND APPROACH OF COMMONSENSE AND MATHEMATICS
In the commonsense domain, we utilized the CSKG dataset presented in the CSKG paper. Unlike the BIO dataset, the object lists for each triplet in this dataset consist of synonymous terms, allowing multiple triplets to share the same subject and relation. In such cases, the objects appearing in each triplet carry distinct meanings. Out of the 6 million triplets, we merged the objects of triplets that have the same subject and relation into a single set, and then sampled x number of triplets from this collection.
In the mathematics and data structure/aloorihmm domain, we utilized the Math-KG dataset introduced in the Math-KG paper. This dataset, originally Chinese, stores multiple objects with the same subject and relation across different triplets. Each object was translated into English using GPT-4, after which the objects from triplets sharing the same subject and relation were merged to construct a final dataset consisting of 22k triplets.
For the CommonSense dataset, the answers (objects) corresponding to given subject and relation are often ambiguous. Consequently, when constructing compelling distractors, there is higher likelihood (about 20%) creating options that are actually correct answers rather than intended incorrect ones, compared to other datasets. Therefore, we include additional verification process after generating the distractors, as outlined in Algorithm 2. Specifically, we formulate multiple-choice questions using the problem and the generated distractors, then ask GPT-44o to select all correct answers. If identifies more than one correct answer, we refine the distractors based on a prompt to recreate incorrect options.
A.4 INFERENCE SETTING
We evaluates all models using vLLM (Kwon et al. (023) system, supporting features of efficient KV cache memory management that dramatically decrease inference time. All white box LM inference conducted by vLLM with hyper-parameter BFloat16 16, fixed seed, two kinds of temperature based on each sampling setting(greed decoding with 0.0, and high temperature with 0.7). The precision is done with eight NVIDIA A100 GPUs(80GB)
A.5 DETAILS OF CHROK IN LEGAL AND TIME-INVARIANT DOMAIN
Figure 4 shows the result of legal domain. Among time variant domains, legal domain shows the most stable results of static, also minimal decline in dynamic dataset. This indicates the domain specificity. which has less frequent yearly changes (almost cases has one change of object in total time frame) and change continues across many time stamps. Also, model's capability for specific task setting is influential in legal domain like the result of MCQA and TF shows, where gap between generation and other templates are many times larger than in other domains.
For commonsense and mathematics in Figure 9, arbitrary years based on the biomedical domain were used, from 2020 to 2024. The left side of result shows the tendency of generation templates, and the middle side is the tendency of MCQA templates, and the last one for TF templates. Each results measure the percentage of Correct answer, represented as line plots. Results show minimal variation, aligning with the stable nature of these knowledge types. This consistency confirms that inveiiinai knowledge is well-preserved across models. For template wise comparison, generation cases show way little gap between models, while MCQA tasks show the different between models,
21

---

Published as a conference paper at ICLR 2025
Common Sense Multi-choice QA
Gemeration
True/False
60
10
60
24
25
Mathematics
2000
2001
2032
2023
2004
2000
2021
2022
2023
2004
2022
2021
2233
2323
2004
Cissuct
Pri-3. S-meriintter
Iaana 1-258
Figure 9: Performance analysis of common-sense and mathematics domains. Three line plots represent each template's results: Generation, MCQA and TE. All model shows clearly the domain specific characteristics, which is invariant knowledge it comes with temporal attributes. Overall results are lower in generation templates, as is challenging for models to correctly recall exactly one object in these domains (e.g., 'subject': 'Parent', 'relation': 'Synonym' has more objects later than Ancestor')
which is aligned with the findings from other time variant domains: the ability of each model's specialized task affects its knowledge recall ability.
About the overall performance quality, the result of time invariant shows lower performance the models generate one object per each knowledge, while the time invariant knowledge's is coverage is wider than other domains. This tendency is alleviated by using MCQA and TF templates, which ends of the rationale for helpfulness of using multiple templates to check knowledge.
A.6 TOTAL TIME FRAME RESULT OF CHROKNOWLEDGE
Figure 12 represent the total results with total time frames general, biomedical and legal domain, including chat template models like mpt-7B, Figure 13 17 represents the total results of template-wise performance in Chroknowledge. Each domain's s result is separated into two temporal state: Dynamic and Static. Every spider plots consist with three template: Generation, Multi-choice QA, and True/False. Each statistics refer the percentage Correct answer, same as Figure]
12
A.7 ALGORITHM OF CHROKNOW PROMPT
The overall scheme of ChroKnowPrompi is down below. As described in Section 6.2 the algorithm starts from making initial prompt with target time ly, subject 8, and relation from target triplet. As the initialized candidate answer is None that model cannot properly answer for that target year, the algorithm also starts with make empty list of candidate answer list A. and accumulated prompt P. Then, the algorithm checks the correct object within each time span P and N. one of those span has no correct object, the algorithm passes that side of traversal. It the preparation is all done, the first step in previous span (if no previous span exists, the nearest next span) begins with selecting object ò by majority voting. Appending prompts in each step, the model is asked to generate or verify and refine the answer C' of each step. like Figure 5 After all step is done, the last candidate answer, which is the most refined result, is being checked with the original target object on coming from target triplet. If is matched (we also used fuzzy match in here), the category of Incorrect updated Chrono-correct.
22

---

Published as a conference paper at ICLR 2025
Algorithm Chronological Prompting Algorithm
Data: Correct set C = {(t,, c: target time f, triplet (s, o), Prev span P. Next span N
Result: List of candidate answers A. Updated Category
Initialize accumulated prompt 7>
:
Initialize candidate answer
8 Initialize candidate answer list A +-
4
Tpeev + time before f in C up to span
// Find correct ohject in previous time
Tnext < time after / in C up to span N
// Find correct object in next time
if T prev 0 then
7
Skip backward traversal and process next years only
8 Thexr @ then
Skip forward traversal and process previous years only
for Ep Tary do
Process previous years first
MajorityVote(C(A, (ot,))(((,)) Get the correct object by majority voting (, P. a, 'previous') Augment prompt by adding above // Generate or verify answer based on system prompt
12
M < PromptAugment(1yg
Onew + LLMResponse
Gext + ExtractAnswer(Anv)
if Gen 1 and der + then
Append a
Update accumulated prompt P with M
for tu Thear do
Process next after previous years á + Majority Vote((((t,)) Get the correct object by majority voting M < PromptAugment(tn s, r. P, P, a, 'next') Augment prompt by adding below // Generate verify answer hased on system prompt
Onew f -L.MRssponse
dest ExtractAnswer((css
If Uen
+ and dez of then
Oext
Append a
Update accumulated prompt P with M
if va; E A. = then
Update knowledge categorization to Chrono-Correct
return A. Updated Category
A.8 TASK CONFIGURATIONS OF CHROKNOWPROMPT
We apply our method to both Incorrect and Partial Correct categories, as the latter may still lack definitive answers. The test set consists of 10% of the total dataset from each domain. Evaluation employs fuzzy matching with a temperature of 0 for strict assessment, classifying an answer as Chrono-correct only if the last candidate answer matches the object. As described in Section|6.2 the system prompt for each case (generation or verification refinement) works as follows Table
Generation Case
[System]
Answer 'Candidate A. [Object]' based on the timestamp. Output only the answer: 'A. [Object]'.
Verification & Refinement Case
[System]
Answer CCandidate A. [Object]' based on the timestamp. it is correct, repeat the same [Object]. If wrong, generate new [Object]. Output only the answer: 'A. [Object]'.
Table System prompts for ChroKnowPrompp.
23

---

Published as a conference paper at ICLR 2025
Table Result of ChroKnowPrompt for both object changed and unchanged cases. The order of open-sources LLM is sorted by release date, starting from the latest model to the most outdated model. The numeric score represents the level of Known increase chronological categorization, and the increase is due to the transition of previously confusing Partial correct responses to Chrono-correct. The parenthesis score the total percentage in dynamic or static dataset, including both changed and unchanged cases. Showing almost 10% to 30% performance of object unchanged cases, the results gives observations that only prompting method has limitations in editing diversely changing temporal knowledge.
Models
general
hiomedical
legal
dynamic
dynamic
dyynmiic
statie unchanged
Object
changed
unchanged
unchanged
changed
unchanged
unchanged
changed
unchanged
Proprictory Large Language Models
Go-immmini
+0.7 (28.7) +7.0 (28.7)
(3..2
+1.1 (519) (49.0)
(51.9
+27.8 (51.)
+0.0 (3.2)
(51.) (3,2
Gemini-
(15.6)
(15.6)
(22.1)
(49.0)
16.0 (48.8)
(1.3)
(1.0
(16.3)
Open-Source Large Language Models
Phi3 5 Mini
+0.3 (17.3)
+1.8
+2.5 (25.5)
+2.1 (45.4)
-16.7 (45.4)
+20.3 (41.3)
+0.0 (0.6)
40.3(00.6)
+4.5 (14,2)
(26.0)
(49.5
[LaMA3.] 8B
+0.2 (20.6)
+2.8(20.6)
(27.1)
+1.1
(36.9)
(33.6)
+0.0 (0.3)
+0.0
+13.8)38)
Gemma2
(19.6)
+3.0()9.))
(2.31)
(32.5
(32.5)
(31.7)
(2.9)
(2.9)
(2.6644.))
Mistral v0.3
+0.4 (18.6)
(18.))
+1.5 (26.9
+0.4
(2.6
+2.6
+0.0
+0.6
(1.0
LLaMA3
(20.9)
(2.77
(18.)
Gemma
+0.2 (18.9)
+0.8 (18.9)
+1.5 (25.9)
+0.3 (18.3)
(18.3
(1.6
+0.0 (0.3)
(0.0
(8.70)
SOLAR
+0.1 (16.5)
+0.7 (16.5)
(24.)
+0.3 (26.5)
(26.5)
(20.)
+0.0 (0.6)
(0.0
(26.)
LLaMA2
+0.3
(18.)
+5.0 (26.6)
(4,.)
-23,2 (143)
(37.)
+0.0 (0.3)
+0.0 (0.3)
(2L8)
Object Inerease
Temporal Inerease
1.7
2.6
6.0
12.3
5.7
Domain Increase
2.0
8.1
A.9 DETAILS IN SPAN-WISE RESULTS OF CHROKNOWPROMPT
6 and present the evaluation of ChroKnow Prompt in span-wise comparisons. While our approach demonstrates significant improvements in certain domains, it shows limited or negligible gains the legal domain. Overall scores in the dynamic dataset remain modest, with the highest impressive results, with the highest
gain being only 1.9. However, the static dataset yields increase exceeding 10% in proprietary models, a level comparable to the biomedical domain's results. Another finding that although the increase in Table o result in general domain is not higher than the static figures in the legal domain, the variation in figures between models significantly larger in the legal domain. As the format of legal dataset is the unstructured format with long context, this would be one factor of low edit quality.
24

---

Published as a conference paper at ICLR 2025
Table Result of ChroKnowPrompt in span-wiie comparison for general and biomedical domain. The order of open-sources LLM is sorted by release date, starting from the latest model to the most outdated model. The numeric score is the level of Known chronological categorization and the increase in parentheses is from the ratio of Chrono-corred which was confusing Partial correct before. Each result presents both in total span and previous span.
general
blomedical
Model Inerease
Models
total span
previous span
total span
previous span
total
previous
dynanaic
static
dynamic
static
dynamic
static
dynamic
static
Propristary Language Models
(+4.))
26.(-5
(+3.3)
(+27.8)
1-5hhshh
(46.5
(44.5)
(+63)
(14.))
(+15.))
48.8 (416.))
(+14.)
(+188)
Open-Sonrce Large Language Modely
Phil Mini
(+2.1)
25.5 (+2.5)
(-1.))
24.1 (+1.1)
45.4 (+18.7)
41.3 (+20.3)
36.5 (+10.)
(+10,))
10.9
5.7
LLaMAS.1 70B
(+1.8)
33,9 (42.1)
26.1 (+19)
33.5 (+1.6)
49.5 (+12.66
46.7 (48.7)
44.9 (+7.9)
41.7 (+3.7)
6.3
3.8
LLaMA3.1
20.6 (+3.1)
27.1 (+1.7)
(-1.9
(+1.0
(69.2)
33.6
(+4.2)
29.1
(+1.0)
(-22
(+1.8
(+1())
(-.36-
26.6(14.2)
LLaMA3
20.9 (+2.7)
(+.07+
(-25)
(+0.)
(1..2
25.7 (+3.8)
28.7 (+3.0)
24.2
Glemma
18.9 (+1.0)
(4..(41..))
(-08)
(40.8)
18.3 (463)
(45.3)
16.0 (45.7)
(.60)))3)
3.5
1.9
SOLAR
16.5 (40.8)
24.9 (40.9)
(s11)) (-..1-
(64.1)
20.3 (14.5)
27.7 (45.3)
(4.7))
2.6
2.8
LLaMA2
(65.2)
(-.00)
(x1.))
37,2
32.5 (+134)
233.(4
Soweee
Mpt
18.3
25.6
(7.0(-3.5)
(42.1)
45.3 (430.3)
(+104)
266((1116)
Pythia
13.8 (+0.0)
20.8 (+0.1)
13.8 (+0.0)
20.7 (+0.0)
13.1 (+0.0)
10.2 (+0.1)
13.1 (+0.0)
102 (+0.1)
0.1
Nemotran3
(412()
18.3 (+1.8)
10.1 (-0.5)
16.7 (+0.1)
22.1 (49.0)
19.4 (48.3)
17.9.-48)
(5.44))4)
5.2
2,5
Temsoral Increase
3.1
2.5
23
1.4
11.6
12.4
6.7
6.6
Domain Increase
2.8
1.8
12,0
6.7
Table 7: Result of ChroknowProopt in span-wise comparison for legal domain. The order of open-source LLMs follows the same sequence as in Table 6 starting with the latest model and progressing to the most outdated one. The numeric score represents the level of Known in chronological categorization, and the increase in parentheses reflects the ratio of Chrono-correct answers, considering total span in the left side and previous span in the right side.
legal
Model Increase
Models
total span
previous span
total span
previous span
dynamic
static
dynamic
static
Proprictary Large Language Models
GPT4o-mini
(+1.9)
(119.1
(+1.3)
(+10.))
8.0
6.0
Gemini-I.5 1.5-flash
1.3 (+1.0)
16.3 (+14.1)
1.6 (+1.3)
18.5 (+16.3)
7.6
8.8
Open-Source Large Language Models
Phi3.5 Mini
0.6 (+0.3)
(+4.5)
(+0.3)
(+2.3)
2.4
1.3
LLaMA3.1 70B
(+1.0)
(+4.5)
(40.3)
53.9 (+2.2)
1.3
1 8B
0.3 (+0.0)
13.8 (+1.3)
0.3.00000
12,5 (+0.0)
0.7
0.0
Gemma2
(+0.6)
44.6
(.0.3)
43.9 (+1.9)
1.6
1.1
Mistral v0.3
1.3 (+0.6)
(47.0)
(+0.3)
19.2 (+5.1)
3.8
2.7
LLaMA3
(+0.3)
(+0.6))))
(40.6)
18.9 (40.6)
0.5
0.6
Gemma
0.3 (+0.0)
8.70 (+0.0)
(+0.0)
8.70 (+0.0)
0.0
0.0
SOLAR
0.6 (+0.0)
(+1.3)
0.6 (40.0)
(42.9)
LLaMA2
0.3 (+0.0)
21.8 (+12.8)
0.3 (+0.0)
17.3 (48.3)
6,4
4.2
Open-Source Chat Models
Mpt
1.0 (+0.6)
8.4 (+5.1)
0.6 (+0.3)
4.5 (+1.3)
2.9
0.8
Pythia
0.3 (+0.0)
3.2 (+0.0)
0.3 (+0.0)
3,2 (+0.0)
0.0
0.0
3
0.3 (+0.0)
5.1 (+1.0)
0.3 (+0.0)
4.8 (+0.6)
0.5
0.3
Temporal Increase
0.5
4,9
0.3
3.7
Domain Increase
2.7
2.0
25

---

Published as a conference paper at ICLR 2025
39
36.8
37.1
35.0
33.7
35.4
35.9
33.9
30.6
29
28.9
31.0
30.1
23.6
30.7
29.7
29.9
29.5
30.4
33.0
34.1
34.4
33.4
30.4
28.6
27.5
24.0
20.4
40.1
37.2
37.5
37.9
38.2
40.5
42.6
42.5
39.7
36.6
35.3
38.3
34.5
32.3
50.9
50.4
50.2
50.2
50.0
51.5
55.2
55.7
53.2
53.1
52.1
52.1
50.1
41.0
40.7
Dynamic
38
35.5
35.9
36.6
37.5
40.5
44.7
44.8
39.6
36.7
35.5
39.6
36.8
30.1
31.5
30.1
30.8
31.4
31.8
33.8
36.1
34.9
32.5
30.0
28.6
28.8
25.2
21.0
30.0
31.5
32.1
28.9
29.4
31.2
32.5
31.3
28.4
25.5
22.8
24.8
22.7
16.0
37.2
35.7
35.7
41.1
39.6
40.2
36.3
31.5
37.1
41.3
32.2
33.6 6
34.7
36.0
38.9
42.9
42.7
36.8
37.5
30.7
3.5
28.5
25.5
30.3
26.4
26.6
28.5
28.6
28.8
31.4
30.1
29.7
28.3
25.1
26.2
21.6
15.44
26
25.9
26.1
25 4
26.3
29.8
31.3
30.0
30.1
28.1
26.3
30.1
25.3
16.4
33.5
35.5
37.0
38.5
38.6
39.0
39.6
38.5
37.2
36.2
33.4
35.9
33.9
31.4
30.5
31.6
33.1
32 /
32.2
31.6
31.2
29.6
27.7
24.7
23.9
25.5
24.7
26.1
25.6
27.2
28.1
28.7
29.8
29.0
25.9
24.5
22.5
22.1
18.9
14.7
33.5
28.7
30.2
38.0
41.0
42.6
43.3
43.7
44.5
44.9
44.1
41.8
40.9
39.5
40.6
39.2
40.0
30.8
32.2
33.3
34.4
34.4
35.8
34.6
33.7
30.6
28.8
28.2
31.1
28.5
26.1
Static
29.4
31.6
32.8
33.2
33.3
34.3
34.6
33
31.0
29.8
28.4
30.2
29.0
23.7
27.6
26.6
27.5
26.4
27.4
25.9
23.7
21.1
18.7
15.6
12.9
9.4
29.9 .88
31.9
33.6
32.9
32.8
34.0
34.2
33.2
30.3
30.2
29.5
31.1
25.6
20.4
29.5
30.7
31.6
32.6
31.6
32.5
31.9
31.3
29.3
28.9
27.3
30.9
28.5
25.7
26.6
28.6
29.7
31.1
30.6
32.0
31.1
28.0
23.5
22.4
18.3
19.6
18.7
13.5
22.7
23.6
24.5
25.4
24.9
26.6
24.1
22.2
22.4
20.3
18.0
20.2
19.2
15.1
2010
2011
2012
2013
2014
2015
2016
2017
2016
2019
2020
2021
2022
2023
10
72 Percentage of Correct
Po min instruc
Liare 3-d5-insrrv instrcct pemman-774
1o0-7b-chu Pyrlil
gmm44 mini
GamW-1355 3.5--a tutrost
gammu
instoot-11
Figure 10: Total result of performance heatmap in general domain for all models
26

---

Published as a conference paper at ICLR 2025
47.4
41.7
62.3
63.1
63.5
57.7
57.7
38.4
39.3
38.9
33.1
32.9
46.8
46.5
46.9
39.6
40.7
57.9
49.6
49.4
42.5
43.4
Dynamic
40.8
41.3
41.1
34.7
35.7
47.2
47.8
47.9
41.3
41.7
33.2
34.5
32.8
28.2
27.7
46.4
47.1
46.8
39.1
39.5
29.5
29.9
28.6
23.2
23.6
39.7
41.1
40.5
34.1
34.3
33.2
33.7
33.9
27.8
27.4
40.5
39.7
40.1
37.3
37.0
63.0
62.5
61.7
59.2
59.9
27.2
27.6
26.0
23.9
24.1
53.1
53.9
53.0
50.5
50.9
44.9
44.8
43.3
41.0
41.7
Static
34.4
35.4
33.7
30.2
31.6
40.7
41.3
39.1
37.0
37.2
33.3
33.3
31.6
29.0
28.9
41.1
40.8
39.4
36.3
36.2
20.1
20.3
18.5
16.8
16.9
34.5
35.6
33.5
31.1
32.4
2020
2021
2022
2023
2024
Percentage of Carrect
Ubno Invorut
mon-7b-cna
GPT-4x
Jamed
corma
Crih Buso-1
Figure 11: Total result of performance heatmap in biomedical domain for all models
27

---

Published as a conference paper at ICLR 2025
11.3
10.5
11.8
12.2
12.3
12.2
12.9
14.2
14.2
13.6
13.0
14.0
13.4
13.4
5.2
5.5
6.4
6.8
7.4
7.7
7.7
8.6
8.5
7.5
7.3
7.3
7.1
7.5
7.6
7.1
9.4
10.0
11.2
12.1
13.5
14.3
14.4
12.8
12.6
13.3
12.7
12.4
19.7
19.6
24.0
23.6
24.5
27.5
26.4
27.7
27.7
26.7
28.0
29.0
28.7
27.7
Dynamic
7.2
8.4
8.3
9.4
10.1
10.5
10.7
10.6
9.5
9.2
9.8
9.2
9.3
5.2
5.2
6.3
6.2
6.9
7.0
6.9
8.0
7.5
6.3
5.9
6.4
6.3
6.1
2.5
2.7
2.4
2.0
2.2
2.1
2.3
2.1
2.3
1.3
1.3
1.4
1.6
1.6
9.2
12.2
12.0
13.6
14.5
15.2
15.7
15.6
15.5
15.4
16.0
15.5
14.9
13.1
16.5
20.3
20.4
4.3
4.3
5.1
4.6
5.0
4.9
5.0
4.9
4.1
4.1
3.8
4.2
3.0
3.1
3.0
3.2
3.3
3.3
3.3
3.5
3.6
3.1
3.0
3.2
3.2
3.7
4.4
4.6
5.2
5.0
6.3
6.0
6.2
6.3
6.5
5.3
5.5
5.8
5.8
5.5
48 4
48.8
48.4
49.0
49.4
48.7
49.1
49.2
48.7
48.3
48.1
48.6
48.1
48.4
40.5
38 4
37.3
37
36.3
36.8
37
36.8
37.2
37.7
37.2
37.3
37.5
37.0
19.7
19.7
19.4
19.3
19.4
19.3
19.3
18.8
19.4
19.8
19.4
29.9
29.0
29.7
29.9
29
29.
29.7
58
58.6
58.6
59.0
59.1
58.8
59.3
58
59.3
59.0
58.3
59.3
58.6
58.7
25.3
25.9
26.0
25.9
25.7
25.8
25.8
26.2
25.7
26.1
25.1
25.5
25.8
26.2
Static
24.2
23.6
23.5
23.8
24.1
24.5
23.7
24.6
24.1
24.7
23.9
24.4
24.4
23.5
20.4
20.9
20.4
9.1
9.6
9.6
10.1
9.1
9.2
9.8
9.5
9.3
9.6
38.7
37.7
37.8
38.4
38.3
37.7
36.2
38.6
37.8
39.2
36.1
38.4
38.1
38.1
48.1 .
48.
48.0
48.7
48.1
48.2
48.4
48.6
48.3
49.1
48.6
48.5
48.7
48.9
18.6
19.0
17.8
18.8
18.8
18.6
18.7
18.7
18.0
18.0
18.6
18.1
18.3
18.5
12.4
12.1
12.5
12.1
12.8
12.3
11.9
12.5
12.3
11.9
12.4
12.2
13.1
12.3
2010
2011
2012
2013
2014
2015
2016
2017
2016
2019
2020
2021
2022
2023
44
59
Percentage of Correct
Ps Ph-33mmn instrct
Liare 3-d5-inatru instrcct pemman-774
mob-7b-ched Pyrtil
gmm4g mini
GamWi3355 tutrost
garmu
instoot112
nemetron 3-5t ww---hh-hh-h-. 4k-1
Figure 12: Total result of performance heatmap in legal domain for all models
28

---

Published as a conference paper at ICLR 2025
Fus7 tbc
P.olebe
2010
2011
2012
Mutt-chess...
TI T.s7aha
2013
2014
2015
General - Dynamic
THRNee
T.o7ibe
T.orob
2016
2017
2018
TueTabe
TueTabi
V.ofebe
2019
2020
2021
T Justhlk
2022
2023
Laam3 58
SOLAR_10.73
got-4o-mini
Mitral78
Gemma2_96
gomini-1 5-flshh
PHi..55,MM
Uema331_708
Figure 13: Total result of template-wise performance in general domain, dynamic dataset
29

---

Published as a conference paper at ICLR 2025
Fus7 abc
F.olebe
2010
2011
2012
Mutt-chess..A
TI T.a7Phe
2013
2014
2015
General - Static
T.o7ibe
T.orbb
2016
2017
2018
Multichosg.cc
TueTabe
TueTabi
V.ofebe
2019
2020
2021
2022
2023
Lamm3 53
SOLA3_10.73
got-4o-mini
Mitral78
Cemma2_96
gomini-1 5-flshh
Phi..55
Uema331_708
Figure 14: Total result of template-wise performance in general domain, static dataset
30

---

Published as a conference paper at ICLR 2025
Nuttccoon
Nuttehars
Biomedical - Dynamic
oration
ad oration
oehoreenn
TranTaha
ThmnTahi
T.eTTbe
2020
2021
2022
MUhi choice
Nuht choice Lt
Peratioo
eraiion
Lama3 L_83
SOLAT_10.75
got-4o-mhi gemini flash
Mitrra78
Gemma22 96
Phi3.5, Mini
Laam33117CC 7C6
Nuttichoice.oo GA
Nutt-cholce.co Cal
Nutichokce.o Oa
Biomedical - Static
resettton
Take
ToweTake
2023
2024
Lama3
SOLA3_10.78
got-4o-min
Misrra78
Gemma2_96
gemini flash
Phi3.5
Uamm3311 708
Figure 15: Total result of template-wise performance in biomedical domain, dynamic and static dataset
31

---

Published as a conference paper at ICLR 2025
FusT abc
F.olebe
2010
2011
2012
Nult-chess..A
T: T.ssPabe
2013
2014
2015
Legal - Dynamic
T.Oroe
2016
2017
2018
TucTabe
TueTabic
V.ofebe
2019
2020
2021
Multi
T: TussPrla
2022
2023
Lamm3 53
SOLA9_10.73
got-4o-mini
Mitral78
Cemma2_96
gomini-1 5-fash
PhiB.55
Uema331_708
Figure 16: Total result of template-wise performance in legal domain, dynamic dataset
32

---

Published as a conference paper at ICLR 2025
FusT abc
F.olebe
2010
2011
2012
Mutt-chess...
2013
2014
2015
Legal - Static
T.oribe
T.orbe
2016
2017
2018
TucTabe
TueTabi
V.ofebe
2019
2020
2021
Multi
T. J.s
2022
2023
Liam33 53
SOLA9_10.73
got-4o-mini
Mitral78
Gemma2_96
gomini-1 5-fashh
PHi..5,
Uema331_708
Figure 17: Total result of template-wise performance in legal domain, static dataset
33

---

Published as a conference paper at ICLR 2025
Template: Generation
Q. In 2020, Pedro Braillard Poccard, position held, [Object]
A. member of the Argentine Chamber of Senators
In 2020, Ana Santos Aramburo, position held, [Ohject] Spain National Library general manager
Q. In 2020, James F. McPherson, position held, [Object] United States Secretary of the Navy
In 2020, Jesús Ávila de Grado, position held, [Object] chief scientific officer
Q. In 2020, Donald Tusk, position held, [Ohject] (generate from here
Template: MCQA
In 2020, what office does Pedro Braillard Poccard hold? (a) member of the Argentine Chamber of Senators, (b) Minister of Foreign Affairs, (c) Governor of Corrientes Province, (d) Mayor of Buenos Aires
(a) member of the Argentine Chamber of Senators
In 2020, what office does Ana Santos Aramburo hold? (a) Minister of Culture and Sports Spain, (b) Dircctor the Prado Museum, (c) Spain National Library general manager, (d) President of the Spanish Royal Academy
(c) Spain National Library general manager
In 2020, what office does James E. McPherson hold?
(a) United States Secretary of Homeland Security, (b) United States Attorney General, (c) United States Secretary the Navy, (d) United States Secretary of Defense
(c) United States Secretary of the Navy
In 2020, what office does Jesús Ávila de Grado hold? (a) President of the National Research Council, (b) Minister of Health, (c) Director of the World Health Organization, (d) chief scientific officer
(d) chief scientific officer
2020, what office does Donald Tusk hold? (a) President of the European Commission, (b) President of Poland, (c) Chancellor of Germany, (d) chairperson
(generate from here ...)
Table 8: Example of two templates: Generation and MCQA. Domain: general, Subject: Donald Tusk, Relation: position held (P39), year to target: 2020.
34

---

Published as a conference paper at ICLR 2025
Template: TF
Q. In 2020, Pedro Braillard Poccard, position held, member of the Argentine Chamber of Senators true
Q. In 2020, Ana Santos Aramburo, position held, Director of the Prado Museum
A. false
Q. In 2020, James E. McPherson, position held, United States Secretary of Defense
A. false
Q. In 2020, Jesús Ávila de Grado, position held, chief scientific officer
A. true
Q. In 2020, Donald Tusk, position held, Prime Minister Poland (generate from here ...)
Table Example of TF templates. Domain: general, Subject: Donald Tusk, Relation: position held (P39), year to target: 2020.
35

---

Published as a conference paper at ICLR 2025
Domain: General
[System]
You expert natural language processing and logic puzzles, skilled generating plausible yet misleading distractor options that challenge users to distinguish between correct and incorrect answers. Pay special attention to questions that involve negative phrasing, such as those containing "not" or "which docs not," to ensure that the distractors do not confuse users into overlooking the negative aspect of the question.
[User]
The question "What office does 'Mitt Romney' hold?" can be answered with "United States senator'. Create three plausible incorrect distractors for this question.
[Assistant]
1. Governor of Massachusetts
Secretary State
3. Speaker of the House
[User]
The question "Which sports team "Yann MVila' member of?" can answered with 'Rubin Kazan', 'France national association foothall team', 'Sunderland A.F.'', 'France national under-21 association foothall team', 'Inter Milan', 'Stade Rennais F.C., Create three plausible incorrect distractors for this question.
[Assistant]
1. Paris Saint-Germain F.C.
2. Olympique Lyonnais
3. AS Monaco FC
[User]
The question " "[Q]" can be answered with '[01]', '[02)',... ",..,'" '[On]', Create three plausible incorrect distractors for this question.
[Assistant]
(generate from here ...)
Table 10: 2-shot prompt for generating three distractors in general domain.
36

---

Published as a conference paper at ICLR 2025
Domain: Biomedical
[System]
You are an expert natural language processing and logic puzzles, skilled generating plausible yet misleading distractor options that challenge users to distinguish between correct and incorrect answers. Pay special attention to questions that involve negative phrasing, such as those containing "not" or "which does not," ensure that the distractors do not confuse users into overlooking the negative aspect of the question.
[User]
The question "What is not the primary anatomic site of 'Rhabdomyosarcomaa of the orbit'?" be answered with Bones set', 'structure bone', 'Bone structure', 'os', 'Skeletal bone', 'Bone structure (body structure)', "bones structure', 'bone structures', 'Ossa', 'Sct of bonc organs', 'Bone organ', 'Skeleton system', 'Osscous', 'Bone', 'NOS', 'Bones', 'bones structures', 'Bone (organ)', 'Bonc', 'skeletal', 'Sct of bones", 'Bones', 'NOS'. Create three plausible incorrect distractors for this question.
[Assistant]
1. Orbital connective tissue Extrinsic ocular muscles
3. Orbital soft tissue
(two more shots)
[User]
The question "What negatively regulates "Vitellogenesis? " can be answered with 'downregulation of yolk production', "downregulation of vitellogenesis', 'down regulation of vitellogenesis', 'inhibition of yolk production' 'negative regulation vitellogenesis, "down regulation of yolk production', 'down-regulation of yolk production' 'inhibition of vitellogenesis', "down-regulation of vitellogenesis' 'negative regulation of yolk production'. Create three plausible incorrect distractors for this question.
[Assistant]
1. Partial left salpingectomy
2. Unilateral oophorectomy
3. Iysterectomyy
[User]
The question " "[Q]" be answered with '[01]', (002)",..,."""" Create three plausible incorrect distractors for this question.
[Assistant]
(generate from here ...)
Table 11: 4-shot prompt for generating three distractors in biomedical domain.
37

---

Published as a conference paper at ICLR 2025
Domain: Legal
[System]
You are a legal expert skilled in crafting challenging fillin-the-blank questions and generating plausible yet misleading distractor options. You will receive question and answer where part of legal text has been blanked For the provided question, create three plausible incorrect distractors that challenge users to distinguish between correct and incorrect answers.
[User]
The question
(beginning of text omilled)
Subpart B Cerrttiffiiation of Substantially Equivalent Agencies Substantial equivalency certification is granted if the determines that a state local agency enforces law that is substantially equivalent to the Fair Housing Act with regard to substantive rights, procedures. remedies, and the availability of judicial review. The Department has developed two-phase process of substantial equivalency certification. m
can be answered with 'Department'. Create three plausible incorrect distractors for this question.
[Assistant]
Secretary
2. Commission
3. Board
[User]
The question
(beginning of text omitted)
Subpart B Minimum Standards for Substantial Compliance by States Within the period defined in 383.73(
can answered with 'f) this title, the State shall', 'h) of this subchapter, State must', Create plausible incorrect distractors for this question.
[Assistant]
1. g) of this chapter, the State will
2. c) of this scction, the State is required to
3. d) of this part, the State should
[User]
The question
[Q]
***
can be answered with [[0]", '[02]', "O.)",..., '[On]', ". Create three plausible incorrect distractors for this question.
[Assistant]
(generate from here ...)
Table 12: 2-shot prompt for generating three distractors legal domain.
38

---

Published as a conference paper at ICLR 2025
Domain: CommonSense
[System]
You are an expert in natural language processing and logic puzzles, skilled at generating plausible yet misleading distractor options that challenge users distinguish between correct and incorrect answers. Pay special attention to questions that involve negative phrasing, such as those containing "not" or "which does not," to ensure that the distractors do not confuse users into overlooking the negative aspect of the question.
[User]
The question "What properties does 'human' not have?" can be answered with "strong enough to lift horse', sstonn enough to lift truck', 'strong enough lift car', 'strong enough to lift cow', 'strong enough to lift building'. Create three plausible incorrect distractors for this question.
[Assistant]
1. strong enough to lift motorcycle
2. strong enough to lift large suitcase
3. strong enough to heavy stone
[User]
The qucstion "What is synonym 'salix tristis"?" can be answered with 'dwarf gray willow', Create three plausible incorrect distractors for this question.
[Assistant]
green weeping willow
2. dwarf green willow
3. gray weeping willow
[User]
The question "What does 'kids' desire?" can be answered with 'play games", 'play', "find crabs in tidal pools', 'read games magazines', 'play with', 'play with toys', 'learn things'. Create three plausible incorrect distractors for this question.
[Assistant]
1. cat vegetables
2. do homework
3. fold laundry
[User]
The question "What 'oil and water' not capable of?" can answered with 'mix'. Create three plausible incorrect distractors for this question.
[Assistant] boil
2. freeze
3. emulsify
[User]
The question " "[Q]" can he answered with '[O1]', "[O2]',-.. '[On]' '. Create three plausible incorrect distractors for this question.
[Assistant]
(generate from here ...)
Table 13: 4-shot prompt for generating three distractors commonsense domain.
39

---

Published as a conference paper at ICLR 2025
Prompt: Mathematics/ Data Structure Algorithm
[System]
You expert in mathematics and computer science, skilled generating plausible yet misleading distractor options that challenge users to distinguish between correct incorrect answers. Pay special attention to questions that involve negative phrasing, such as those containing "not" or "which docs not," ensure that the distractors do not confuse users into overlooking the negative aspect of the question.
[User]
The question "Which of the following is unrolated to "Output"?" can be answered with 'Left child', 'Sct', 'Depth'. 'Modify', 'Post-order traversal', 'Dictionary', 'Predecessor', 'Array', 'Delete', 'Sparse matrix', 'Infix expression', 'Leaf node', 'Shortest path', 'Right subtree', 'Nodc', 'In-order traversal', 'Hashing' 'Head node', 'Pointer', "Level-order traversal', 'Keyword', 'Pattern string', 'Record', 'Determinism', 'Linked list', 'Critical Path', 'Function name", "Connected component", 'Loop statement', 'Robustness', 'Inverted index', 'Prcorder traversal', 'Memory', 'Dequeue', 'Singly Linked List", 'Image'. Create three plausible incorrect distractors for this question.
[Assistant]
1. Print statement
2. Display buffer
3. Output stream
[User]
The question "What is synonym for 'Ancestor'?" can be answered with 'Parent', Create three plausible incorrect distractors for this question.
[Assistant]
Descendant
2. Sibling
3. Offispring
[User]
The question " "[Q]" can be answered with '[01]', '[02)', ([o,)"...."" "a"]",..,''".... '[On] Create three plausible incorrect distractors for this question.
[Assistant]
(generate from here ...)
Table 14: 2-shot prompt for generating three distractors mathematics, data structure, and algorithm domain.
40


Reviews:
[{"review_id": "I0uaK7JS3c", "reviewer": "Reviewer_Yuxp", "summary": "This paper introduces CHROKNOWBENCH, a benchmark used to evaluate chronological knowledge in LLMs by distinguishing between evolving and static information. The authors also introduce a sampling-based prompting technique to improve LLMs’ recall of time-dependent knowledge across multiple domains. Their approach enables effective temporal non-parametric knowledge updates, even in proprietary models, showing some improvement in biomedical and general knowledge accuracy.", "questions": "Q1: What is the evaluation score for the \"Generation\" experiments in Figures 2 and 3? How do you extract the answer from the model output? Is it an exact match?\n\nQ2: You mention that you use the original MMLU prompting strategy (Hendrycks 2021). Is this with the 5-shot setting or with the zero-shot and Chain-of-Thought generation?\n\nQ3: Are the results for the \"Time Invariant\" data being reported?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is very well written, and most of the claims are well-motivated and justified. \n- The benchmark covers various domains and temporal knowledge dimensions. \n- The authors transform the problem into both open generation and MCQ tasks.", "weaknesses": "W1: The overall results appear incomplete across the dataset's various dimensions. A table summarizing average results for each benchmark dimension—Domain, Format, Temporal State, and Time Dependency—would help clarify performance. Currently, it’s unclear whether the performance limitations stem from temporal aspects or the models' domain-specific capabilities.\n\nW2: The proposed prompting method does not clearly demonstrate its effectiveness. Section 5.1 shows stable performance in the biomedical domain, while Section 7.1 suggests the prompting method mainly improves results for biomedical questions, which were not influenced by dynamic changes. How do the authors justify these outcomes as evidence of the prompting method’s efficiency?\n\nW3: In Line 313, the authors state that \"GPT-4 performs best in generation tasks.\" Do they provide any rationale for this observation?\n\nW4: To strengthen the analysis, the paper would benefit from more detailed error analysis and examples. In which cases does the model fail to generate correct answers, and are there discernible patterns in these errors?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mKodZpDn67", "reviewer": "Reviewer_P36m", "summary": "This paper investigates how well large language models recall chronological knowledge. The paper makes several contributions. First, the authors describe a new benchmark dataset covering different domains (e.g., biomedical, legal) and different types of time dependency and temporal state. Then, the authors benchmark several LLMs (both open-sourced and closed-sourced) and show that LLMs have varied abilities to elicit temporal knowledge. Finally, the authors present a prompting method to elicit chronological knowledge.", "questions": "* The ChroKnowBench considers only changes on `object` while keeping `subject` and `relation` unchanged. How does it capture the change of `relation` between the `subject` and `object`? For example, `Zidane` was a `player` at `Real Madrid` in 2001, then became `coach` in 2010.\n* Figures 2 and 3: It is hard to interpret these results; suggest using the same y-axis range\n* Section 5.1: `In dynamic datasets, models show reduced knowledge in multiple-choice question answering (MCQA) settings, highlighting the impact of formatting and prompting on temporal abilities.` What does this mean?\n* Section 5.1: `models fail to track recent updates between 2022 and 2023`; `Static datasets remain consistent, reflecting the stable nature of scientific knowledge` suggest clarifying what these statements mean", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The studied problem is important and interesting.\n* The authors build a benchmark for evaluating LLM and propose a method for recalling/editing knowledge?", "weaknesses": "* The authors evaluate whether LLM can track `object changes`. This problem formulation may not capture the `accumulative nature of knowledge` as discussed in the paper. For example, one drug may have multiple side effects, each of which was identified at different times. I am unsure how the proposed benchmark and evaluation strategy addresses this accumulative nature. For example, a model that always generates the old knowledge will be considered `correct` based on the definition in Table 1.\n* The result sections are hard to follow: it is very difficult to understand how their results support these statements in Section 5 (Figures 2, 3).\n* The proposed prompting strategy seems to be disconnected from the benchmark sections. For example, the authors show that the proposed prompting strategy improves in the biomedical domain, whereas results in Section 5 and Figure 3 show little variability across times. Any intuitive explanations of why?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JPNXWuVSjy", "reviewer": "Reviewer_D8Co", "summary": "This paper proposes a new aspect of evaluating Large Language Models, which focuses on chronologically accumulated knowledge. The authors set up a new benchmark, ChroKnowBench, for chronological knowledge evaluation across three dimensions: time dependency, domain difference, and temporal state. They also put forward a novel evaluation framework called ChroKnowledge, and a novel prompting technique called ChroKnowPrompt, which aims to improve the model’s performance using temporal neighboring data.", "questions": "1.\tWhy would you include Legal data? The rest of the data are of the same type in format and structure, but only legal is unstructured and of a different type.\n2.\tFor your results in section 4, is it possible that the quality of your collected data in different time points is a more influential factor? \n3.\tSuggestions for the weaknesses mentioned above: \na.\tDo more fine-grained classification of the “static” and “dynamic” data.\nb.\tUpdate Figure 2 and 3 for more clarity.\nc.\tMove some information from the appendix to section 7, including the extra descriptions for ChronoPrompt and some extra summaries for the algorithm.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper targets a novel field, which studies how language models learn and adapt to knowledge in different time spans. Previously, not many researchers formalized and paid attention to the question. The authors did well in defining the key aspects of testing and evaluating chronological knowledge in large language models, including putting forward a comprehensive dataset and benchmark that covers a variety of time ranges and domains, as well as developing novel prompting techniques. They also conducted fairly comprehensive experiments to demonstrate their claims.", "weaknesses": "1.\tResults in section 7 does not necessarily support the soundness and validity of the ChroKnowPrompt framework, but instead hints at the deficiency in the ChroKnowledge dataset. There is another possible explanation to the large amount of performance increase in the biomedical section of the dataset: domain-specific data are more static / less dynamic than general data, and your method on distinguishing dynamic and static knowledge needs to be updated to a more fine-grained extent. For example, over the span of 10 years, some data might changed 5 to 6 times, but there could be other data that only changed once. I currently don’t see that taken into account. The improvement from the “dynamic” portion of your results could be over-exaggerated.\n2.\tThe figures need more work. For figure 2 and 3, I understand that the authors are trying to demonstrate the relationship between time and model performance, but the graph needs some extra work on clarity to deliver the message. Figure 4 and 5 in section 6 also needs more work. The most useful explanations for Figure 4 are in the appendix, making the figure a little confusing. Figure 5 needs more clear textual hints on the overall framework of ChroKnowPrompt. The figure itself is not illustrative enough for the entire framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4bZ5pHI4Cn", "reviewer": "Reviewer_1hbk", "summary": "This paper introduces CHROKNOWBENCH, a benchmark dataset designed to evaluate chronologically accumulated knowledge across three key aspects: multiple domains, time dependency, and temporal state. Besides, the authors develop CHROKNOWPROMPT, an in-depth prompting to elicit chronological knowledge by traversing step-by-step through the surrounding time spans. The motivation behind this work is clear, however, the paper majorly lacks in the quality of the experiments and its setup.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This work addresses the sensitivity of LLM to time dependencies and constructs dynamic and static datasets for evaluation.\nThis paper introduces an iterative approach to integrate knowledge across different temporal spans, enabling dynamic knowledge updates.", "weaknesses": "The paper mentions evaluating knowledge across multiple domains; however, it only introduces knowledge from the medical and legal domains, where changes of knowledge along the timeline are relatively subtle.\nThis paper aims to explore the ability of LLMs to capture temporal dependencies within knowledge. However, the constructed benchmark dataset and evaluation methodology do not effectively demonstrate this capability of the models in an intuitive manner.\nAlthough the paper divides knowledge into dynamic and static datasets, the knowledge within these datasets may have already been learned by LLMs. As a result, the benchmark evaluation results are likely to primarily reflect the memory capacity of the LLM regarding this knowledge, rather than its ability to handle time dependencies.\nFor the dynamic dataset, this paper essentially transforms knowledge into a temporal knowledge graph (TKG). However, there are established benchmarks in TKGs. The authors should consider providing comparisons with these benchmarks and clarifying the specific improvements made.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)