KoLA: Carefully Benchmarking World Knowledge of Large Language Models
Authors: Jifan Yu
Xiaozhi Wang
Shangqing Tu
Shulin Cao
Daniel Zhang-Li
Xin Lv
Hao Peng
Zijun Yao
Xiaohan Zhang
Hanming Li
Chunyang Li
Zheyuan Zhang
Yushi Bai
Yantao Liu
Amy Xin
Kaifeng Yun
Linlu GONG
Nianyi Lin
Jianhui Chen
Zhili Wu
Yunjia Qi
Weikai Li
Yong Guan
Kaisheng Zeng
Ji Qi
Hailong Jin
Jinxin Liu
Yu Gu
Yuan Yao
Ning Ding
Lei Hou
Zhiyuan Liu
Xu Bin
Jie Tang
Juanzi Li
Venue: ICLR 2024 poster (2024)
Keywords: Large Language Model
World Knowledge
Evolving Benchmark
Primary area: datasets and benchmarks
TL;DR: A carefully designed evolving benchmark for evaluating LLMs' world knowledge.

Abstract: The unprecedented performance of large language models (LLMs) necessitates improvements in evaluations. Rather than merely exploring the breadth of LLM abilities, we believe meticulous and thoughtful designs are essential to thorough, unbiased, and applicable evaluations. Given the importance of world knowledge to LLMs, we construct a Knowledge-oriented LLM Assessment benchmark (KoLA), in which we carefully design three crucial factors: (1) For ability modeling, we mimic human cognition to form a four-level taxonomy of knowledge-related abilities, covering 19 tasks. (2) For data, to ensure fair comparisons, we use both Wikipedia, a corpus prevalently pre-trained by LLMs, along with continuously collected emerging corpora, aiming to evaluate the capacity to handle unseen data and evolving knowledge. (3) For evaluation criteria, we adopt a contrastive system, including overall standard scores for better numerical comparability across tasks and models, and a unique self-contrast metric for automatically evaluating knowledge-creating ability. We evaluate 21 open-source and commercial LLMs and obtain some intriguing findings. The KoLA dataset will be updated every three months to provide timely references for developing LLMs and knowledge-related systems.

Published as a conference paper at ICLR 2024
KoLA: CAREFULLY BENCHMARKING WORLD KNOWLEDGE OF LARGE LANGUAGE MODELS
Jifan Yu, Xiaozhi Wang. Shangqing Tu, Shulin Cao, Daniel Zhang-Li, Xin Lv, Hao Peng, Zijun Yao, Xiaohan Zhang, Hanming Li, Chunyang Li, Zheyuan Zhang, Yushi Bai, Yantao Liu, Amy Xin, Nianyi Lin, Kaifeng Yun, Linlu Gong, Jianhui Chen, Zhili Wu, Yunjia Qi, Weikai Li, Yong Guan, Kaisheng Zeng, Ji Qi, Hailong Jin, Jinxin Liu, Yu Gu, Yuan Yao, Ning Ding, Lei Hou, Zhiyuan Liu, Bin Xu, Jie Tang, Juanzi Li Tsinghua University, Beijing, China, 100084 kola-bes gell-benchmarrlgoooggggggoooo
ABSTRACT
The unprecedented performance of large language models (LLMs) necessitates improvements in evaluations. Rather than merely exploring the breadth of LLM abilitics, we believe meticulous and thoughtful designs are essential to thorough, unbiased, and applicable evaluations. Given the importance of world knowledge LLMs, we construct a Knowledge-oriented LLM Assessment benchmark (KoLA), in which we carefully design three crucial factors: (1) For ability modeling, we mimic human cognition to form a four-level taxonomy of knowledge-relatee abilities, covering 19 tasks. (2) For data, to ensure fair comparisons, Wikipedia, a corpus prevalently pre-trained by LLMs, along with continuously collected emerging corpora, aiming evaluate the capacity to handle unseen data and evolving knowledge. (3) For evaluation criteria, we adopt contrastive system, including overall standard scores for hetter numerical comparability across tasks and models and a unique self-contrast metric for automatically evaluating knowledge-creating ability. We evaluate 28 open-source and commercial LLMs and obtain some intriguing findings. The KoLA dataset will be updated every three months to provide timely references for developing LLMs and knowledge systems.
use both
INTRODUCTION
Recent remarkable breakthroughs achieved by large language models (LLMs) like GPT-4 (OpenAI, 2023) have elicited widespread astonishment. Considering the extensive and profound natural language understanding and generation abilities exhibited by LLMs (Bubeck et al., 2023), the conventional benchmarks (Wang et al., 2018; 2019) focusing on relatively narrow and superficial abilities are no longer helpful for testing them. It has become necessary to construct better benchmarks for effectively comparing LLMS and providing valuable diagnostic results. To this end, various benchmarks are proposed, focusing extending the evaluation scope to cover broader abilities (Hendrycks et al., 2021; Zhong et al., 2023; Huang et al., 2023) or more challenging tasks (Srivastava et al., 2022; Suzgun al., 2022).
In addition to broadening the evaluation scope to explore the breadth of LLM abilities, we believe meticulous designs are also necessary build evaluations that facilitate in-depth insights, maintain impartiality towards different LLMs, and have high applicability for audiences interested in selecting and enhancing LLMs. Designing benchmark requires careful consideration of three key factors: (1) Ability Modeling. A benchmark should not only define the scope of desired abilities but also model the inherent connections between the evaluated abilities, which allows for diagnostic insights on how to acquire and improve these abilities. (2) Data. Given the extremely broad range of training data for L.LMs, which might include annotated data of certain tasks and often undisclosed, ensuring
"Equal Contribution.
"Corresponding author.

---

Published as a conference paper at ICLR 2024
Date: Known and Evolving
Ability Cognitive Takonomy
Evaluation: Contrastive Metrios
desace KASKING
CARATING
Consicetin both form and Lelaa Arooehddg
ARPLING
SAR EEASTNN
KAA
IPONDREATION
Figure 1: KoLA's careful design on three key factors for LLM evaluation.
that differences in training data do not impact the evaluation fairness is critical and challenging. (3) Evaluation Criteria. For high applicability, evaluation metries should enable audiences to easily understand and gain helpful observations. Moreover, there are many well-known issues (Theis et al., 2016; Sajjadi al., 2018; Ji et al., 2023) for evaluating tasks with large search spaces like the generative tasks. Evaluations for related abilities still heavily rely on human evaluation, which is time-consuming and not easily reproducible (Belz et al., 2022: 2023).
In this paper, we propose a Showledge-orienter LLM Assessment benchmark (KoLA), which aims at carefully benchmarking the world knowledge of LLMs by undertaking meticulous designs considering the aforementioned three factors:
For ability modeling, we evaluate world knowledge of LLMs and design four-level cognitive ability taxonomy. We chose world knowledge as our evaluation scope because: (i) World Knowledge is widely recognized as playing a fundamental role in the impressive performance of LLMs (Hendrycks et al., 2021; Zhong et al., 2023; Huang et al., 2023), and deeper grasp of knowledge enables LLMs to better assist humans; (ii) Recent work has shown that understanding and generating structural world knowledge remain challenging for LLMs. Unlike previous work focusing on expanding the evaluation breadth by covering diver tasks and disciplinary knowledge to test the knowledge boundaries of LLMs (Hendrycks al., 2021; Zhong al., 2023; Huang al., 2023), we focus more the "depth" of evaluation, i.e., modeling the intrinsic connections between knowledge-related abilities and ensuring reliable evaluation results. Inspired by the human cognitive processes in learning theory, such as Bloom's 's taxonomy (Krathwohl, 2002), we organize evaluated abilities into four levels: Knowledge Memorization, Knowledge Understanding, Knowledge Applying, and Knowledge Creating. This taxonomy helps to provide more specific and helpful evaluation results, detailing which aspect of knowledge the evaluated models may be deficient in. also facilitates a preliminary exploration of the similarities and differences between the learning mechanisms of L.LMs and humans. To coordinate with our data design considerations introduced later, we selected 19 tasks, primarily focusing on world knowledge about entities, concepts, and events.
For data, we obtain both known and evolving data sources. Some studies adopt unpublished or machine-unreadable data (Zhong et al., 2023; Huang et al., 2023) to reduce the possibility that the test data has been learned by LLMs. However, considering the intense competition between L.M.s, those data may also be trained by LLMs in the near future'. We believe the ideal approach is to do evaluations on newly emerging data and maintain a continuously evolving benchmark, like the attempts that include time-sensitive evolving data (Kasai et al., 2022; Dhingra et al., 2022). In KoLA, we host a new competition season every three months. For each season, we crawl and annotate 500 recently published articles as the evolving data. The evolving data source allows us (i) evaluate models more fairly, even if some models can rapidly update their knowledge, thereby demonstrating their power, and (ii) better track the model development. Besides evolving data, we also consider the known data of LLMs, which means the data sources that all models have learned. Evaluations on known data enable us (i) fairly compare the learning efficiency of LLMs by comparing the different knowledge they acquire from the same training data and (ii) assess the generalization ability by comparing L.MMs' performance on known data and evolving data. We chose Wikipedia as our known data source due to its common use. Considering the limitations of Wikipedia and our annotation capabilities on the evolving data, we are unable to cover a very wide range of tasks.
Some cases have been noted (https: //c/a/////ev// wew.evvaabnccaar........ com wtt.atttiieieeeearrrrrraaaaaa htmll)
2

---

Published as a conference paper at ICLR 2024
For evaluation criteria, we design a contrastive evaluation system, including an overall standard score system and a self-contrast knowledge creating metric. Conventional benchmarks report absolute metrics for different tasks separately. The incomparability of scores across tasks makes it difficult for audiences to intuitively compare the proficiency levels across different abilities. Additionally, the sensitivity of different metrics varies, which may lead less experienced audiences misinterpret the ability differences represented by numerical differences. In the KoLA main leaderboard, we report standard scores across different tasks, determined by the relative level compared other evaluated LLMs. This makes KoLA applicable to broader range of audiences. Experienced audiences can still refer to absolute metrics if desired. Furthermore, evaluating knowledge creation is particularly challenging as it involves distinguishing the correctly created knowledge and knowledge hallucinations (Ji et al., 2023). We design self-contrast metric for evaluating knowledge hallucination by contrasting freely created completions and ghowledge-groundee completions of an LLM given the same beginnings. This metric eliminates the influence of writing styles and focuses on whether the generated completions are consistent with the actually presented knowledge.
In the first two seasons of KoLA, we evaluate 28 widely-used LLMs, including API-access commercial LLMs, such GPT-4 (OpenAI, 2023) and Cohere-commante and 20 open-source LLMs including GLM-130B (Zeng et al., 2022). LLaMa (Touvron et al., 2023), etc. From the experimental results, obtain some intriguing observations, such as larger base models tend to memorize more knowledge, alignment unleashes the potential of larger models in higher-level abilities but may harm memorization, and open-source models exhibit overall inferiority compared to commercial models.
We welcome the participation of more LLMs in KoLA evaluation and encourage contributions to the new seasons of KoLA. The data, leaderboard, participation information, and supporting tools are publicly available upon acceptance. We hope KoLA can serve as a diagnostic tool to facilitate the development of increasingly knowledgeable LLMs, and also help practitioners select LLMs.
2
KoLA BENCHMARK
2.1 ABILITY MODELING
Within the context of Artificial Intelligence (AI), Knowledge has long been employed to signify information encompassing facts, events, and skills (Feigenbaum, 1977), serving as an indicator for the intelligence level of Al. Hence various mnowledge-intensivg tasks (Petroni al., 2019; 2021) are proposed to examine language models' knowledge-related abilities. Recently, the impressive perfor- mance of L.MMs has encouraged the development of more comprehensive benchmarks (Srivastava et al., 2022: Suzgun et al., 2022) with broad human-subject exams (Hendrycks et al., 2021; Zhong al., 2023; Huang et al., 2023).
Cognitive Ability Taxonomy. Confronted with such vast array of evaluation datasets, we ad- vocate for considering the stratification and connection of abilities, rather than organizing them discretely (Wang et al., 2019; Hendrycks et al., 2021; Srivastava et al., 2022; Suzgun et al., 2022) or straightforwardly based on disciplines (Zhong al., 2023) or difficulties (Huang et al., 2023). Such viewpoints have also been upheld by cognitive scientists for several decades, giving rise series of cognitive learning theories (Lewis Smith, 1993). Considering the ongoing debates surrounding high-order thinking (Miri et al., 2007: Collins, 2014), we simplify and select four widely accepted processes in Bloom's 's taxonomy (Krathwohl, 2002) for organizing the tasks in KoLA benchmark.
1. Knowledge Memorization (KM) aims to gauge the model's ability in faithfully recalling known facts, exemplified by the previous knowledge probing task (Petroni et al., 2019).
2. Knowledge Understanding (KU) focuses on evaluating the model's 's ability in understand- ing the underlying knowledge within texts, instantiated by the conventional information extraction tasks (Yao et al., 2019; Wang et al., 2020; Ding et al., 2021; Peng et al., 2022).
3. Knowledge Applying (KA) reflects the ability of agents in employing knowledge accom- plish reasoning and problem-solving tasks. Consequently, this level is evaluated by various knowledge reasoning tasks (Yang et al., 2018; Trivedi et al., 2022; Cao et al., 2022).
4. Knowledge Creating (KC) denotes the ability to create novel and reasonable knowledge given known facts. This is evaluated by the knowledge coherence and correctness (Chen
3

---

Published as a conference paper at ICLR 2024
et al., 2020; Bang et al., 2023) of contents generated by the model. It is worth noting that the evaluation goes beyond merely assessing the generation quality (fluency, etc.).
2.2 DATA SOURCE AND SELECTED TASKS
Known & Evolving Data: A common concern in evaluating LLMs is the fairness issue brought by variations in training data and the potential test data leakage risk. To minimize these biases, we propose the design of the following distinctive data sources:
(1) Known Data Source. Wikipedia" is an acknowledged high-quality corpus containing over 6.6 million English articles, which has been used in pre-training by numerous pre-trained models since BERT (Devlin et al., 2019; Brown et al., 2020; Shuster et al., 2022) and is widely included in open pre-training corpora (Gao al., 2021). Hence we believe assuming every L.LM has been trained on Wikipedia is reasonable and adopt it as our known data source. Considering that many LLMs state they can only provide answers based on "Content before 2021" 3, we select Wikidata5M (Wang et al., 2021a), high-quality subset of Wikidata, as the basis, which allows linking to the 2019 version of Wikipedia dump, thus enabling the selection or reconstruction of downstream tasks' datasets.
(2) Evolving Data Source. Considering the time required for model training (Zeng al., 2022), it is less unlikely for newly emerged data to be timely trained by LLMs. Therefore, we have devised an evolving evaluation mechanism that continuously retrieves the web content published in around recent 90 days as the data source and constructs new datasets on them. This approach ensures fair assessment of LLMs' performance on unseen content and whether they "secretly" involve knowledge updating modules like the external search. Each update (we call it Season of KoLA) requires crawling a minimum of 500 articles to support building test sets. For the first season reported in this paper, we adopt two kinds of data: factual news and fictional novels? 5, We intend to persist for an additional 4 seasons (approximately year) to promptly integrate the fortheoming top LLMs. We anticipate that the consistently released reports can further support relevant researchers.
Built upon these two data sources, we finally select and construct 19 tasks in KoLA, as shown in Table 1. To ensure both the quality and efficiency of annotations for each season, we randomly select one task at each level to annotate the new evolving evaluation dataset. For the existing datasets, we try to ensure most of the test sets are not public, and this rigorous setting ensures a high level of fairness. The data collection and task construction details are shown in Appendix C. We briefly introduce the tasks of the four levels below. It is noteworthy that, due to the limitations of data distribution and collection processes, the absolute numerical values of the model on Evolving data are not necessarily destined to be lower than those on Known data.
We follow LAMA (Petroni et al., 2019) to evaluate knowledge memorization by probing facts from LLMs but re-construct the datasets on our data sources. Given a triplet in Wikidata5M (Wang et al., 202 we transform it into a sentence with a relation- specific template and let LLMs complete its tail entity. Additionally, we want to explore whether the knowledge memorization of LLMs correlates with training frequency. We sort the entities in Wikidata5M according to their frequency of occurrence in Wikipedia (Jin et al., 2019), resulting in the creation of two test sets: (1-1) HlghhFFreqeeny Knowledge. Randomly selecting 100 entities from the top 2, 000 entities with the highest frequency and construct data with triplets of them; (1-2) Low- now-Frequency Knowledge. Similarly, we randomly select 100 entities from the lowest-frequent entities and construct a more challenging evaluation set; (1-3) Evolving Test of Memorization (ETM). From the articles in evolving data sources, we annotate the knowledge triplets shown in them and only preserve 100 triplets that cannot be inferred from previously available corpora.
Knowledge Memorization Tasks:
Knowledge Understanding Tasks: Knowledge understanding is evaluated by whether LLMs can understand various genres of knowledge from texts, including concepts, entities, entity relations, events, and event relations. (2-1/2-2/2-3) Concept Probing employs the three probing tasks (CSJ, CPJ, CiC) of COPEN (Peng et al., 2022) to evaluate the models' understanding of conceptual knowledge. (2-4) Named Entity Recognition utilizes the FewNERD dataset (Ding et al., 2021), from which we
Phttps: openai. com *An open source news API at Githuh. URL: http:://ggthuu.../rraaaaaaateaie /GNews
'A well-known open license novel creating community. UN..http:://rrchhrrrooooor
4

---

Published as a conference paper at ICLR 2024
Table 1: The tasks in KoLA (Season 1st and 2nd). Metrics in bold are selected for calculating standardized scores. Exclusive task means their test sets are newly developed or sponsored by the original authors and were not publicly disclosed. Test Set and Pool correspond to the testing instances used in each season and the overall available instances.
Level
ID
Dutaset
Metries
Exclusive
Context Type
Test Set
Poul
Source
1-1
High-Freq.
EM, F1 EM,
Triple
100
20.6M
KM
1-2
Low-Freq.
Triple
100
20.6M
Known
1-3
FTM
EM, F1
Triple
100
2.7k
Evolving
2-1
COPEN-CSI
Acc.
Entity, Concept
100
3.9k
2-2
COPEN-CPI
Acc.
Conceps
100
4.7k
2-3
COPEN-CIC
Acc.
Concept
2.3k
Known
KU
2-4
FewNERD
Sentence
188.24
2-5
DocRED
Documeut, Eatity
12k
2-6
MAVEN
Document
100
20.4k
2-7
MAVEN-ERE
Document(s), Event
199
1.3M
2-8
ETU
F1
Document, Eatity
100
1.6k
Evolving
3-1
HorporQA
F1
X
Document(s)
100
7.4k
3.2
2WikiMulé.
F1
Document(s)
100
12.6k
KA
3-3
MuSiQuee
F1
Document(s)
100
2.5k
Known
3-4
KQA Pro
KG
100
1.2k
3.5
KoRC
Documen((s), KG
100
5.2k
3-6
ETA
Document(s), KG
1.6k
Evolving
KC
4-1
Encyclopodic
BLBU, Rouge
Document, Event
4.5k
Known
4-2
ETC
BLBU, Ronge
Document, Event
100
Evolving
randomly select 300 examples in our evaluation. (2-5) Relation Extraction selects the undisclosed test set from the challenging document- level relation extraction dataset, DocRED (Yao al., 2019). (2-6) Event Detection adopts the undisclosed test set of the finely annotated MAVEN (Wang et al., 2020) dataset. (2-7) Event Relation Extraction involves the undisclosed test from MAVEN-ERE (Wang et al., 2022), which consists of 113k examples of coreference, temporal, causal, and subevent relations between events. (2-8) Evolving Test of Understanding (ETU). For the articles in evolving data, conduct the entity recognition and follow the same relation schema of DocRED to annotate brand new test set containing 100 relation instances from 50 articles. is worth noting that apart from the evolving test, the other datasets are all based on Wikipedia texts.
Knowledge applying ability is evaluated by LLMs' multi-hop reasoning capabilities, specifically over world knowledge. This differs from several recent studies (Lu et al., 2023; Mialon al., 2023), which cover more general reasoning, such as mathematical reasoning. Therefore, the following progressive Wikipedia-based datasets are included in KoLA: (3-1) Hot- PoiQA (Yang et al., 2018) a question-answert
Knowledge Applying Tasks:
dataset that involves a substantial number of natural language questions written by native speakers, examining machine's abilities in comparison, multi-hop reasoming, and more. However, a limitation of HotpotQA is that some questions can be answered through shortcuts. To address this, (3-2) 2 2WikiMulrihoPAA (Ho et al., 2020) ensures that questions cannot be solved through shortcuts by haaually-designed templates, but their questions lack naturalness in language. Furthermore, the (3-3) MuSiQue (Trivedi al., 2022) dataset tackles the challenges of shortcuts and naturalness simultaneously. Its questions are composed of simple questions from existing datasets, with up to four-hop complex reasoning. (3-4) KQA Pro (Cao et al., 2022) is large-scale dataset, whose questions are relatively complex, allowing for more fine-grained evaluation of LLMs' multi-hop reasoning with logical operations and modifiers. (3-5) KoRC' (Yao et al., 2023) is dataset that requires joint reasoning between the text and knowledge base. differs from the aforementioned four datasets as it requires implicit rather than explicit reasoning. (3-6) Evolving Test of Applying (ETA) takes the same construction approach KoRC, producing 49 questions upon 350 annotated knowledge triplets and 40 articles in the evolving data.
Knowledge Creating Tasks: As the highest level of Bloom's 's Cognitive Taxonomy (Krathwohl, open and challenging question. The capacity for knowledge creation is evident in open-ended generation tasks. Traditional text generation evaluation metrics (Theis et al., 2016) are based on textual similarities between model model-generated generated content and tumttewritteen references, which do not solely focus knowledge creation ability but
2002), how to evaluate knowledge creation a
5

---

Published as a conference paper at ICLR 2024
cover other skills, such as text style and fluency (Naeem et al., 2020). Ideally, human evaluators should be employed to solely assess whether the content generated by models contains novel and reasonable Lamb et al., 2018). However, manually evaluating diverse open open-domain knowledge is labor-intensive, costly, and lacks scalability. Inspired by the knowledge- segelddggggguudedd text generation tasks (Yu et al., 2022), KoLA proposes feasible automatic evaluation protocol that specifically contrasts the model-generated knowledge with that in human references. First, we limit the generation scope narrative texts such history, news, and fiction. This is because the knowledge creating in generating narrative texts has a clear focus on envisioning plausible subsequent events and articulating them in a reasonable way. As shown in Figure 2, we then conduct human annotation on the reference texts to obtain reference fine-grained event knowledge. The annotated events enable a dedicated self-contrast metric (elaborated below) that emphasizes the quality of event knowledge in generated content. This approach offers an effective assessment of knowledge creation abilities compared to traditional text generation metrics encompassing many other factors (Akter al., 2022). We conduct annotation on both Wikipedia texts and evolving articles, which constructs two evaluation datasets: (4-1) Encyclopedic Knowledge Creation, which is based on the narrative Wikipedia articles selected by MAVEN (Wang et al., 2020) and (4-2) Open Knowledge Creation, which is based on unseen news and novels, serving as the Evolving Test of Creating (ETC).
knowledge (Maher, 2010;
Table I presents the features and statistics of each selected task. Further details regarding annotation processes and task demonstrations are correspondingly presented in Appendix D.
2.3 CONTRASTIVE EVALUATION SYSTEM
Our contrastive evaluation system includes standardized overall scores based on relative model com- parisons and a unique self-contrast metric, which can automatically evaluate knowledge hallucination and enhance generation evaluation.
Standardized Overall Scoring. Since the metrics of different KoLA tasks are incomparable and differently sensitive, less experienced audiences cannot easily compare and interpret results, which is also prevalent in recent LLM benchmarks like Big-Bench- Hard (Suzgun et al., 2022) and MMLU (Hendrycks et al., 2021). Therefore, we propose to introduce standardized scores (Dyck et al., 2005) to enhance the applicability of KoLA results. Specifically, given task set D= = { and the evaluated model set M = | we first select the most representative metric for each task, allowing us to compute the performance score Tij of model m, on task d,. Then the standardized score z can be calculated as:
;=1
Zij 1 (Ia, (ijar)) m.jarr)
(1)
(Ial, -., s((a)
where (-) and a (-) denote the mean and standard deviation. Subsequently, we apply Min-Max scaling (Patro & Sahu, 2015) adjust all the results to the range of [0, 100), further enhancing the correlation and readability of scores across tasks. The final scores are presented as:
min (2)
100
(2)
max
min
where the functions max (z) and min (z) correspond to the maximum and minimum of all zi scores.
Self-contrast Metric. Evaluating knowledge creating is not only about evaluating generation quality (Theis et al., 2016), but more about assessing whether the generated knowledge is faithful and reasonable, i.e., avoiding knowledge hallucination (Ji et al., 2023). We develop a unique self-contrast metric for this, which is defindd by contrasting two completions generated by the same model.
As illustrated in Figure 2, C' denotes the given preceding context, R denotes the writte- succeeding completion, and K refers to the annotated event knowledge in R. Each model is required to generate two completions: (a) Given only context C, generate a version of completion T, which requires the model to freely imagine the possible events and may have knowledge hallucination like the negotiation event in Figure 2: (b) Given both the context C and the foreknowledge K, generate another completion Tk. which only requires the model to reasonably compose the given events. If T and T2 demonstrate a strong resemblance, it implies that the model can create highly reasonable events that are consistent with human provided references and has less knowledge hallucination. The distinct advantage of this self-contrast method is that, since both completions are generated by the same model, factors external knowledge creation, such as writing style, are highly likely to remain consistent,

---

Published as a conference paper at ICLR 2024
Given Contrct
LLM
42
The Sartle of Eveslam nustectre
Knowledgn
Model Comp
Loicect ced qus wooe
Eovead
Elger Aorriiatee (Arg
on betoverss tooes magmmba ou Antinnnoee Juees flom The
lnvened fuentamn, Unternnaecarr
deliccioon
sssape
ora rarinng banor aroor uty groams evett corfict
Vidler
Assolatten
Hentlort
Prince Gavand
Reference Completion
Mert stant aad you contyp sfroull
engage
kiled
Completoon with Forekroowllee Sedgge
casaral asuans the escape from
Cauplt jalsi uoerghh Motrffrtt
Mbeetortt
Vertlort
of rrgported
Puee su facil nure of eisente be whet vineal was leflet
eeg regayroent Pucchre vi recement bai foriter to privee
aading
Figure 2: Illustration of the Knowledge Creating (KC) tasks.
minimizing their influence on the evaluation. Moreover, to cover more comprehensive aspects of knowledge creation abilities and prevent evaluation collapse caused by the model's disregard for the knowledge K in the prompt of the process (b), the overall knowledge creating score is defined the mixture of multiple contrasts:
I avg (3 (T, (Te, R)).
where avg (-) denotes the average. Function (-) is to calculate the similarity of the two texts, which we employ the widely-used Rouge-I. (F1) (Lin, 2004) in this work. The 0 (T, R) is the conventional text generation metric. While it captures broad range of knowledge creation abilities (spanning multiple genres of knowledge beyond events), it also includes undesired factors unrelated knowledge creation, such as writing styles and text fluency. Hence we add 0 TT) and (Ti, It) to emphasize the abilities about creating event-related knowledge, which is important for generating narrative texts. The a (T, Tk) is the newly-pres prrpposed self-contrast metric focusing on whether the generated event knowledge is reasonable. The 0 (Ti, R) takes inspiration from the knowledge- grounded generation tasks (Chen et al., 2020; Ghazvininejad et al., 2018). reflects the ability to create knowledge about the relationships between events, which is required in reasonably composing the given events into story. For instance, the Tę in Figure 2 implies that the death of Simon de Montfort caused the rebels to lose the battle, while this is hallucinated causal relation inconsistent with the narrative in R.
3 EXPERIMENT
Evaluated Models. In the first two seasons of KoLA, we evaluate LLMs of two categories: (1) Open- source Model, including GPT-J (6B) (Wang & Komatsuzaki. 2021), GPT-JT (6B) (Computer. 2022),
GPT-NeoX (20B) (Black et al., 2022), BLOOM (7B) (Scao et al., 2022), TO++ (1 (11B) (Bang et al., 2023), LLaMa (65B) (Touvron et al., 2023), GLM (130B) (Zeng et al., 2022), UL.2 (20B) (Tay et al., 2022), FLAN-T5 (11B) (Chung et al., 2022), FLAN-UL. (20B) (Research, 2022), Alpaca (7B) (Taori et al., 2023), ChatGLM (6B) (Du et al., 2022), Dolly-v2 (12B) (Conover et al., (023)).eePajjam- Instruct (7B) (Computer, 2023), Tulu (7B) (Wang al., 2023), Vicuna (13B) (Zheng al., 2023),
Llama2-chat (7B) (Touvron et al., 2023), CharGLMM-32K (6B) (Zeng et al., 2022), Internlm-chat-88 (7B) (Team, 2023);; (2) API service: GPT-3 curie v1 (6.7B) and davinci v1 (175B) (Brown et al., 2020), InstructGPT curie v1 (6.7B*)6 and davinci v2 (175B*) (Ouyang et al., 2022), ChatGLM (130B) (Zeng et al., 2022), Cohere-command (52.4B)². 12-Jumbo-Instrutt (178B*) (Studio, 2023), GPT3.5-turbo and GPT-4 (OpenAI, 2023). (*) indicates the size has not been confirmed.
Overall Performance. We report the standardized scores of all models in Table 2 and 3, where indicates that the result is unavailable due to the input is longer than the model context length. All results are from Season 2nd (Sept. 2023), and the comparison with the rankings from Season 1st (June 2023, Appendix F) is shown in the "Rank" column. Despite the overall consistency in rankings across different levels, we can still obtain some intriguing findings from the results:
(1) For models without alignment or instruction tuning (c.g.. GPT-J and BLOOM), there is a strong correlation (Spearman's 's coefficient of 0. 0.79) between the ranking of the Knowledge Memory (KM)
/https://plat form. cpt.ra.pppei....mmmmmmm com wttp:i.ccooooovvvooooooo/ overview "https://docs cohere. com /wc.///o////eoooo ttee commannd-mdde
7

---

Published as a conference paper at ICLR 2024
Table 2: Standardized performance of Knowledge Memorization and Understanding level.
Model
Level I: KM
Level 2: KU
I-1
I-2
1-3
Rank
2-1
2-2
2-3
24
2-5
2-6
2-7
2-8
Rank
GPT-
64.3
68.9
41.5
(-)
69.5
48.6
51.4
66.9
100.0
77.2
78.9
81.3
1st
GPT-33-
60.0
18.3
2nd (721
49.4
54.0
51.3
50.2
56.6
25.3
InstucstiPT devinci (1755))
42.4
41.8
62.3
31.3
35.7
30.9
20.5
22.2
25.8
Cobere-commad (52.4)
54.5
33.9
(191
40.0
47.1
40.3
26.6
38.6
20.6
46-9
25.0
PLAN-U1 (20B) 12 Jumbe Instruct (1758*)
42.6
30.7
(L11
47.1
53.0
16.0
25.0
20.5
23.2
25.0
6th
32.6
33.7
19.2
12th
27.3
23.5
31.2
37.1
32.0
22.7
$1.3
25.0
7th
Chanta M (1508)
38.0
56.5
36.1
5th (72)
30.5
47.8
$1.9
16.0
25.0
25.5
30.3
25.0
Kth
FLAN-T3 (IB)
56.
51.5
32.9
4th (L1)
63.2
47.8
49.1 1
18.7
25.0
5th
InstrustiPT curie (6.78*)
28.1
43.8
28.9
10th (13)
29.4
41.2
41.8
22.3
25.4
22.0
25.8
25.0
9th
1.aMM (65R)
24.2
25.6
19.5
15th (1))
22.0
18.4
18.3
55.6
31.4
30.1
25.5
25.0
(11)
CbaiGLM2 32k (6B)
25.3
22.8
20.1
16th
22.0
40.8
20.0
19.0
26.5
20.5
22.7
25.4
17th
Alpaca (78)
17th
26.4
31.4
22.2
25.0
21th
21.6
19.7
17.9
22th
18.4
24.5
37.4
20.5
25.2
25.6
14th
Chentill (6R)
16.0
25.0
32.0
22.8
25.0
Vicuna
17.4
26th
29.8
26.0
35.4
30.5
25.0
22.9
14th
49.6
33.2
29.7
22.2
12th (12)
GPT-3 (6R)
20.8
18.8
18.0
25th (11)
22.0
18.4
18.3
22.2
25.0
31.0
250
25th
TU+- (11B)
41.9
38.4
23.9
9tb (?1)
30.5
29.2
21.8
16.0
25.0
16th
Dally-s2 (128)
21.1
21.0
18.9
200h (72)
22.0
18.4
18.8
28.5
25.0
20.5
27.1
25.0
25th
GPT-JT (SB)
19.8
18.9
19.0
24th (1)
22.0
18.4
18.3
19.2
25.0
36.7
25.0
22th
Internim chat 8k (7B)
25.5
20.4
17.2
19th (71)
22.0
18.4
18.3
19.0
27.2
20.5
26,1
25.0
27th
11E.2 (20R)
26.5
28.1
18.9
15th
22.0
18.4
18.3
18.0
25.0
28th
GPT davines vI (1750)
17.9
16.9
27th
22.0
18.7
18.3
30.2
25.0
29.5
22.3
25.0
19th
GPT-NNoX (2UB)
32.1
25.0
BLLOMM
28.0
10.3
21.7
250
curie (6.70)
16.8
25th
18.4
18.3
21.5
25.0
25.6
23.9
25.0
MedPPjjaa--ussruutt (78)
21.8
21.2
16.4
21th
(-]
22.0
18.4
18.3
29.8
25.0
30.6
25.4
25.0
24th
Table 3: Standardized performance of Knowledge Applying, Creating level and all the 4 levels.
Model
Level 3: KA
Level 4: KC
Overall (1,2,3,4
3-1
3-2
3-3
3-4
3-5
3-6
Rank
4-1
4-2
Rank
Avg
Rank
GPT4
59.8
00.7
76.8
22.8
60.9
58.1
1st (-))
45.6
58.9
2nd (11)
2.33
lst
GPT-3.5-mabh
58.5
42.6
53.9
45.6
30.9
19.6
5th (11)
52.2
46.4
1st (11)
1.54
2nd
InttrusGPPT davinci x2 (1758*)
30.6
39.7
44.2
23.6
50.3
23.4
7th (11)
54.4
36.6
1.12
3nd
(7B)
42.3
54.8
(12)
32.8
42.5
(12
0.04
Cohoro-ommadd (52.4)
44.0
(L1)
17.1
15.1
(511(
0.54
FLAN (20B)
43.5
53.3
(-)
28.3
20th (13)
(11)
12-Jambo-Inti Instract (1788*)
452
31.6
31.6
38.3
28.3
25.5
8th (-)
43.7
53.3
4th (-)
0.47
(11)
CharGLM (1308)
36.4
34.1
28.0
26.4
36.7
21.2
9th
24.4
28.4
10th (12)
0.28
SUL (73)
FI-AN-T5 (1 1B)
45.0
49.0
32.9
51.1
39.7
16.1
6ih (11)
20.2
0.0
28th (16)
0.22
9th (13)
InstrusGPT curie vi (6.70*)
31.6
37.2
24.3
29.1
31.2
27.2
11th (-)
27.7
29.6
12th (15)
0.05
10th
LLaMa (65B)
16.4
35.4
41.7
25.4
21.7
17.6
16th (12)
44.7
37.1
5th (-)
0.011
11th
ChaeGi M2-32k (6R)
357
31.1
24.0
40.1
20.7
17.3
13th (11)
54.5
41.5
8th
-0.09
17th
Alpuca (7B)
15.1
19.3
20.9
18.1
45.5
50.7
35.7
41.0
7th (13)
13th (12)
Llanna-chhat (78)
14.4
4015
51.7
14th
38.1
(14)
14h
ChartiLM (08)
212
23.5
22.2
19.9
19.5
23.5
20th (-)
17.7
30.8
(16)
150h (15)
Vieuna (138)
254
18.1
21.0
16.4
24th (L))
35.0
45.9
6th (11)
.0.26
16th (1))
GLM (130B)
23.5
13.0
18.4
21.7
45.0
35.1
17th (14)
29.3
19.1
19th (13)
-0.33
17th (1L)
GPT-J (68)
38.8
39.4
26.8
49.3
17.5
16.5
10th (11)
30.5
24.0
14th (19)
-1.53
1801 (73)
T0++ (1 IB)
22.4
23.0
23.8
14.4
39.7
16.1
19th (-)
18.1
3.7
27th
-0.44
19th (16)
Daily- (128B)
14.1
20.5
18:4
18.1
26.4
25.2
22th
29.5
31.9
11th (12)
-0.45
20th (11)
GPT-JT (6B)
514
38.2
23.0
32.8
18.5
17.5
15th (-)
21.2
19.5
(15)
-0.54
21th (13)
Insernim dhal Sk (7B)
193
20.8
22.8
14.4
17.]
22.2
23th (11
26.1
27.8
15th
-0.56
22th
U11.2 (20B)
16.1
(-)
26.9
(19)
23th
(1758)
21.5
17.0
25th
3233
22.6
(-)
.0.57
24th
GPT-NeoX (20R)
18.1
22.7
16.7
28th (L1)
322
19.2
(71)
-0.61
25th
BLOOM (7B) GPE-3 curte vi (6.78)
186
22.6
17.1
19.9
27.4
23,3
21th ()
179
21,4
22t6
-0.61
26th
22.6
15.2
20.5
18.1
19.1
16.4
26th (-)
2577
10.1
24th (11)
-0.81
27th
RedPPaama--nnnne (7R)
12.6
10.0
17.1
14.4
26.1
23.2
270h
137
12.3
-0.85
28th
and the model size. This suggests that model size has an obvious positive impact on memorizing seen knowledge, which corroborates some of the viewpoints from previous studies (Liang et al., 2022).
(2) For models after instruction tuning, there is significant increase in the correlations between higher-level abilities and model size (exemplified by KA, whose Spearman's 's coefficient 0.02 to 0.53). This suggests that alignment unleashes the greater potential of LLMs in higher-level capabilities. However, the correlation between size and low-level KM performance exhibits a decline (0.34), potentially demonstrating the widely discussed "alignment tax" (Ouyang et al., 2022).
(3) Compared to the commercial closed-source models like GPT4 and GPT-3.5-turbo, there is still a noticeable gap in the performance of open-source models. Open-source models obtain an average z-score of -0.29, which is below the overall average. Comparing the second-season results with the first season, the rankings of most source models have declined. This suggests that static open-source models struggle to maintain comparable level with potentially continuously updated commercial models in the long run. The open-source community should advocate for stronger collaborations to support larger and up-to-date models that are crucial for future research purposes.
8

---

Published as a conference paper at ICLR 2024
1,00
COBEST
275
1.
21
114
0.75
Lite of And Ft, 3 0.00
93% Carfiderce Inte
15.
Contidence Mitinal
Sed fi,, 0.11
0.50
30
0.25
20
20-
1
0.00
20
30
40
50
50
70
20
50
40
50
50
lord-3::4
Iryd-::
3/-
Contareat
-9.25
40-
Iine of Bed Fi, 37 0.53
455 Contidence Intercs
10-
95% Cantidence itencal
Uine of Ted fit, Al 0.66
35
A
-9.30
30
30
-9.75
25
20
-1.00
20
30
50
60
20
30
50
Figure 3: Left: The spearman correlation coefficient. Each cell represents the correlation of model rankings on two tasks. Right: Scatter plots of rolling task vs. corresponding non-rolling task (e.g.. 3-5 v.s. 3-6). The x-axis and y-axis of each subplot represent the standard scores correspondingly.
Design Analysis. We further discuss several new observations brought by KoLAA design factors.
First, there is a high correlation among tasks within each level, indicating that the abilities of LLM indeed possess some inherent hierarchical structure. The knowledge memorization (KM) level shows notable correlations with other levels, especially with the concept tasks in the understanding level (2-1, 2-2. 2-3), as well as with the reasoning tasks (from 3-1 to 3-5) in the applying level, which indicates that these high-level tasks rely heavily knowledge memory. Moreover, in order to obtain a more dissociated assessment of the L.MMs competence in higher-order cognitive tasks, it is still recommended to design tasks that exhibit substantial disparities from the pre-training corpus to alleviate the potential biases stemming from data.
Second, the results of the models on evolving and non-evolving tasks show obvious linear correlation, indicating the reliability of our construction of evolving datasets. The performance gap between known and evolving data is more prominent for shallower levels (KM, KU), whereas it is less pronounced in higher-level tasks (KA, KC). The convergence of performance between and
evolving settings suggests a potential enhancement in the model's generalization capability and may support the opinion about the model's acquisition of divergent and reasoning abilities that go beyond simple data fitting (Bubeck et al., 2023; Zhong et al., 2023).
Third, we conduct manual annotation (Appendix F.1 for more details about annotation settings and results) on the results in the knowledge creating tasks, where each annotator is required to read the contexts C' and foreknowledge K, and then evaluate the model's outputs T in two aspects: overall quality and faithfulness. Ratings are assigned on scale of 1 (the worst rating) to 5 (the best rating). We calculate Spearman's 's correlation between the mamually annotated results and the metrics introduced in $ 2.3. We find that there is notable correlation (0.61) observed between the self-contrast metric Ø (T,Th) and the faithfulness of created content, while removing the self-contrast metric from the overall metric x in Eq. (3) brings significant 32% decrease in the correlation with the human-judged overall quality. We believe this metric can contribute to future explorations on the assessment of generation abilities (Theis et al., 2016; Pillutla et al., 2021: Fu et al., 2023).
4 CONCLUSION AND FUTURE WORK
This paper presents KoLAA, a carefully designed Knowledge-oriented LLM assessment benchmark. We design cognitive ability taxonomy for more helpful diagnostic results, adopt both known and evolving data sources for better fairness, and employ contrastive metrics for high applicability. In the first season of KoLA, we evaluate 28 open and commercial LLMs and get some intriguing findings. such as larger models tend to memorize more knowledge, and alignment unleashes the potential of higher-level abilities but may harm the low-level knowledge memorization, etc. In the future, we will continually host more seasons of KoLA to facilitate knowledgeable LLMs, help select backbones for developing knowledge-related applications, and track the development of LLMs with evolving evaluations. KoLA will always welcome open participation and contributions.
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This research project is supported by grant from the Institute for Guo Qiang, Tsinghua University 2019GQB0003). Jie Tang is supported by Technology and Innovation Major Project of the Ministry of Science and Technology of China under Grant 2022ZD01 18600, NSFC for Distinguished Young Scholar 61825602 and the New Cornerstone Science Foundation through the XPLORER PRIZE.
This project also appreciate the providing of the test set of MusiQue (Trivedi et al., 2022) by Harsh Trivedi, Stony Brook University and the test set of 2Multiwikihopp (Ho et al., 2020) by Xanh Ho, National Institute of Informatics, Tokyo, Japan.
ETHICS STATEMENT
In this section, discuss the ethical considerations regarding our data construction and leave the broader impact to Appendix A.2. (1) Data Risk Control. Regarding the collected evolving data source, we have filtered out content that is inappropriate for presentation and the relevant details are outlined in Appendix C.I. Seven of the authors manual check all the newly constructed evolving test datasets well random samples of all the previously released datasets included in KoLA. No instances of personally identifiable information. discriminatory content, explicit, violence, or offensive content were found. (2) Annotator Treatment and Consent. We hire crowdsourced annotators in the annotation of evolving test data and the human evaluation for knowledge creating. The details are introduced Appendix C.2. We have signed work contracts with all the annotators and provided compensation according to mutually agreed-upon wage standards and working hours. All employment arrangements are in compliance with local regulations. (3) Copyright. Our known data source is Wikipedia, which is licensed under CC BY-SA 3.0 and allows for free research use. For all the previously released datasets included in KoLA. Our evolving data source contains public news and fictions. The news data is from The Guardian and access it strictly following the terms and conditions 10, The fiction data is from Archive Our Own (AO3)!! fan-fiction archive site. Although AO3 data has been used in some previous works Cao & Daumé (2020); Yoder et al. (2021); Sun et al. (2022); Krishna et al. (2022), there remains some ambiguity regarding its copyright status. We believe our use of AO3 is appropriate because: (i) A03 exhibits an open attitude towards data crawling 12 (ii) We pledge that KoLA will always remain non-commercial and non-proe proffi, and we do not redistribute the crawled data (only samples are provided our platform). According to the description' 13 provided by the Organization for Transformative Works, the operator of AO3, such usage falls under fair use the context of the U.S. copyright law.
general audience,
REPRODUCIBILITY STATEMENT
To promote reproducibility, we provide details about our data collection in Appendix C, all the used task instructions in Appendix D, and experimental details in Appendix E. The evaluation source codes and data samples for all the tasks are submitted as supplementary material. The results of future seasons will be presented the Github and our platform website.
REFERENCES
Mousumi Akter, Naman Bansal, and Shubhra Kanti Karmaker. Revisiting automatic evaluation of extractive summarization task: Can we do better than rouge? In Findings of the Association for Computational Linguistics: ACL 2022, pp. 1547-1560, 2022.
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. multitask. multilingual, multimodal evaluation of
10

---

Published as a conference paper at ICLR 2024
chatgpt reasoning, hallucination, and interactivity. arXiv preprint arxiv:3302.04023, 2023. URL /tap:s:///rxiii .ow.vvorg///eeeee htt///df//// 2302. 1..02..pd.
Anya Belz, Maja Popovic, and Simon Mille. Quantified reproducibility assessment of NLP results.
In Proceedings of ACL, pp. 16-28, 2022. doi: 10. 10.18663//1/22222acl-- URL https:
//ta///aaaaaa //calannthlloyy org/ 2022. acl acl-long.2.
Anya Belz, Craig Thomson, and Ehud Reiter. Missing information, unresponsive authors, experi- mental flaws: The impossibility of assessing the reproducibility of previous human evaluations in NLP. In The Fourth Workshop on Insights from Negative Results in NLP, pp- 1 1-10, 2023. URL https:/ aacaannaoooooyyy org/ 22/202233 insight 1..1.
Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He. Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, Usvsn Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. GPT-NeoX- 20B: An open-source autoregressive language model. In Proceedings of BigScience Episode #5 Workshop on Challenges & Perspectives Creating Large Language Models, pp. 95-136, 1.9. URL https: http:::s///aaaatth.loooooo /aclannnnnoogyyyyy. wwwpohol.o..........o...// orq/2022.
2022. doi: 10.18653/v
bigscience-i
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Mierbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Proceed- ings of NIPS, 2020. URL https: proceedoceeeeeeeooooocooccoeeeec ngs... neurips ips. cc 1457c0d 674 Bbfbaac
20t0//2220/hhh2//// /
ract httm..
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with GPT-4. arXiv preprint arXiv:2303. 12712, 2023. URL https: //axivv..vvvv org/pdf/2303. 12712.12222222 pdf.
Shulin Cao, Jiaxin Shi, Liangming Pan, Lunyiu Nie, Yutong Xiang, Lei Hou, Juanzi Li, Bin He, and Hanwang Zhang. KQA Pro: A Dataset with Explicit Compositional Programs for Complex Question Answering over Knowledge Base. In Proceedings of ACL. pp. 6101-6119, 2022. doi: 10..18663/11/2022.ac1-ooooo 422.
URL htttss:/acclatttatt wwwp://oooo....o22oo/oo/22 act-2o22...clonnoog
Yang Trista Cao and Hal Daumé. Toward inclusiee coreference ACL, 2020. URL https : aalaaaaaanooooooooeeooooooo wwwpyy..rgg//./0 http:/22200.ccl11000000 acl-ma 418 ....f.
oew.reuusssseeeerrrrrrrrrrerrooi resolutto... In Proceedings of
Wenhu Chen, Yu Su, Xifeng Yan, and William Yang Wang. KGPT: wwwwledge-groundee pre-training for data-to-text generation. In Proceedings of EMNLP, pp. 8635 01455,864,,446 2020. doi: 10. 18653/v1v 4020.emnlp-main.69 URL https: http///cccaattooogoo..ooooo org/ 22000.000022 em ma in. 697.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. Scaling instruction-finetuu language models. arXiv preprint arxiv:2210.11416 2022. URL https: //roiii///xvvv.// org atg/pol /pff/ 221 0. 11 14141666 p..
Robyn Collins. Skills for the 21st century: teaching higher-order thinking. Curriculum & Leadership Journal, 12(14), 2014.
Together Computer. Releasing GPT-JT powered by open-source AI. https::/www. 2022.
xyz/blog/ http:///wo....eaain-.--g-------e -t--:-ttt-oooeeee---------eeeeee
Together Computer. Redpajama: An open source recipe to reproduce Ilama training dataset, 2023.
URL https::/p//......... //gl thub. com httpb//..o/mttgeeeercooooouteott /eeeinuuuuttteeeraaeeeeee RedPa MedPajamas jama-
11

---

Published as a conference paper at ICLR 2024
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instruction-tuned Ilm, 2023. URL https: //www wwtp::.aaabbbaiooaa cttp:////cccc//c/o////////////// commercicci.o..o.m yy inn
1mm.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL- pp. 4171- 4186, 2019. doi: 10.18653/v 1011////////11.... 1423. URL https
http:////aaaaatoaaaaaeeaaa /
Bhuwan Dhingra, Jeremy R Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W Cohen. Time-awar aware language models as temporal knowledge bases. Transactions of the Association for Computational Linguistics, 10:257 2022.
Ning Ding, Guangwei Xu, Yulin Chen, Xiaobin Wang, Xu Han, Pengjun Xie, Haitao Zheng, and Zhiyuan Liu. Few-NERD; A few-shot named entity recognition dataset. In Proceedings of ACL, pp. 3198 3213, 2021. doi: 10. 10.18653/112021...2. .all-long.245 www.://..........oo...2.. org/2021. acl long 248.
URL https:://ccaaneeeeett http:///ccaaatoorryyo
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 320-335, 2022.
Peter Dyck, Christopher Boes, Donald Mulder, Clark Millikan, Anthony Windebank, P James B Dyck, and Raul Espinosa. History of standard scoring, notation, and summation of neuromuscular signs. a current survey and recommendation. Journal of the Peripheral Nervous System, 10(2): 173, 2005.
Edward A Feigenbaum. The art of artificial intelligence: Themes and case studies of knowledge engineering. In Proceedings of IJCAI, volume 2. Boston, 1977.
Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate you desire. arXiv http://aaaaa..../////////////// /2302.
preprint arxiv: 166, 2023. URL https: /ttp://aaivvvv//..//aa//e.a/.aaa
0..00.004160 04166. pdf.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arxiv:2101.0000 000027 2021. URL https: / http//axxvvv.rg/////e
0tt/2100..00021 pdf.
Marjan Ghazvininejad, Chris Brockett, Ming-Wac Wei Chang, Bill Dolan, Jianfeng Gao, Wen-tau Yih, and Michel Galley. A groouddddegg neural conversation model. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, In Proceedings of ICLR, 2021. net URL https::t//// /tttps://openrevviiii wttprrvveeeieeeeoounttt... forum? id= jmI 3GmQ.
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of COLING, pp- 580n URL htttts://acaaatttooua
6609-6625, 2020. doi: www111663/1/2220..o. org/2020- www.ccoonnnooo maann.mmaii.55. 580.
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, et al. C-Eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arxiv:2305.08322, 2023. URL https::/aaae wwwooorg//......eeee 22003 08322 .dfff
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1-38, 2023.
Hailong Jin, Chengjiang Li, Jing Zhang, Lei Hou, Juanzi Li, and Peng Zhang. XLORE2: large- cross-lingual knowledge graph construction and application. Data Intelligence, 1(1):77~98, 2019.
12

---

Published as a conference paper at ICLR 2024
Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A Smith, Yejin Choi, and Kentaro Inui. Realtime qu: What's the answer right now? arXiv preprint ar arXiv:2207.13332, 2022. URL https: /atas://arrie /ttp//iivvv./////////////a .ww./vor//eaae has///b///// 2207 13332.
David R Krathwohl. A revision of bloom's taxonomy: An overview. Theory into practice, 41(4): 212-218, 2002.
Kalpesh Krishna, Yapei Chang, John Wieting, and Mohit lyyer. RankGen: Improving text generation with large ranking models. In Proceedings of EMNLP, pp. 199-232, 2022. URL https: //aclanthology org//022
Carolyn Lamb, Daniel G Brown, and Charles LA Clarke. Evaluating computational creativity: An interdisciplinary uutorial. ACM Computing Surveys (CSUR), 51(2):1-34, 2018.
Arthur Lewis and David Smith. Defining higher order thinking. Theory into practice 32(3):131-337, 1993.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arxiv:2211.091101 2022. URL https: /arxiv.vvvvvv//////////////aa wtt/od//g/p//ooooooooooo 09110. pdf.
Chin-Yew Lin. ROUGE: package for automatic evaluation summaries. Text Summarization Branches Out, pp. 74-81, 2004. URL http:: / //acl//ccccc anthology .eayy.rr//aa org/w04-100
Pan Lu, Baolin Peng. Hao Cheng. Michel Galley, Kai- Wei Chang, Ying Nian Wu, Song-Chun Zhu, and lianfeng Gao. Chameleon: Plug-and-play compositional reasoning with large language models. arXiv preprint arXiv:2304.09842, 2023. URL https: / //tpxiaxvvv////////////////aavvl ortoorggg//..... 2230/ 09842. pdf.
petttss://rxxt
Mary Lou Maher. Evaluating creativity in humans, computers, and collectively intelligent systems. In Proceedings the 1st DESIRE Network Conference on Creativity and Innovation Design, pp. 22-28, 2010.
Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi- Yu, Asli Celikyilmaz, et al. Augmented language models: a survey, arXiv preprint arXiv:2302.07842 2023. URL httpt:ht/saa:t::: //axxiv. org/pdf 2202. 07842.
Barak Miri, Ben-Chaim David, and Zoller Uri. Purposely teaching for the promotion of higher-order thinking skills: A case of critical thinking. Research in science education, 37:353-369, 2007.
Muhammad Ferjad Nacem, Scong Joon Oh, Youngjung Uh, Yunjey Choi, and Jacjun Yoo. Reliable fidelity and diversity metrics for generative models. In International Conference on Machine Learning, pp. 7176-7185. PMLR, 2020.
OpenAl. Gpt-4 technical report. arXiv preprint arxiv22033.08774. 2023. URL org/pdf http:/df/33333 08774.0... pdf.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. et al. Training language models to follow instructions with human feedback. In Proceedings of NIPS, pp. 27730- http://pwcceeceeeeerrrrrrreeeeee co wwwpeeuwwwpps.pppppeeeeeeee 27744, 2022. URL h00.:ffiieheeeeeeeeeeeee20200101
SGOPAL Patro and Kishore Kumar Sahu. Normalization: A preprocessing stage. arXiv preprint arXiv: 1503v:1503.06442. 2015. URL httpttps///xxxx..rrg////e//
064 pd..
Hao Peng, Xiaozhi Wang. Shengding Hu, Hailong Jin, Lei Hou, Juanzi Li, Zhiyuan Liu, and Qun Liu. COPEN: Probing conceptual knowledge in pre- me-trrined language models. In Proceedings of EMNLP, pp. 5015-5035, 2022. htRpss: / //al//aaeeeeeee anthology org/202220222 emn 1-mpp-mm-- n. 335.
13

---

Published as a conference paper at ICLR 2024
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language Models as Knowledge Bases? In Proceedings of EMNLP, pp. 2463- 2473, 2019. doi: 10.18653/v1/D19- 1/509-1250. URL https: http:///aaaaatttoaaaaaaoao
Btrrg//D1--a 1250.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. KILT: Benchmark for Knowledge Intensive Language Tasks. doi: 10. 10.18853/vvl/ 1/203/ mnaccl-main.000.
In Proceedings of NAACL-HLI, pp. 2544,,4,, 2021. naac -main. 200.
URL wtt.tttt/////. /tttps://aclase www.hhtss///cccaaaatcc http:////aaaaancloooorrrrro
Krishna Pillutla, Swabha Swayamdipta, Rowan 7ellers, John Thickstun, Sean Welleck, Yejin Choi, and 7aïd Harchaoui. MAUVE: measuring the gap between neural text and human text using divergence frontiers. In Proceedings of NIPS, pp. 4816- 4828, 2021. URL httpa::/proceeddnng: wwwpiiinnnnnn.iupigoo neurips cc/paper//02//haae Abstraact........aa....aaaaac... html.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
Google Research. A new open source flan 20b with u12, 2022. URL httpt:::/// htttttttt/www.tteeee net/ blog/ Bolog/llan-us ul2 20b.
Mehdi S. M. Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. In Proceedings of NIPS, pp- 5234-5243, 2018. URL https::/ -Astsaa centtrcct.ttl...
weweiiiinnnnneuiissseeeeeeeeieee nwwrips pccpppppr/22188888. 0aa.0/h//hhh//// hash/ Abstract
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié. Danicl Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100 2022. URL
ort. http:/ff//22222 2211 051 0510000 .d...f.ffff....
Kurt Shuster, Jing Xu, Mojtaba Komeili, Da Ju, Eric Michael Smith, Stephen Roller, Megan Ung, Moya Chen, Kushal Arora, Joshua Lane, et al. Blenderbot 3: a deployed conversational agent that continually learns to responsibly engage. arXiv preprint 2tti::2008.0318888 2022. URL /arxiv. org otg//df//////oooo.////p//////epr 2208 001888.11...
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arxiv:2206.04615, 2022. URL https: /ttpss////avvvve/eaeeeeeeeeeeeaa httpp//////////// 2206. 04615. 04615.pdf.
A121 Studio. Announcing jurassic-2 and task-specific apis, 2023. URL https::///// /ttp:///www............ commbbog// httto//ciiooottuiinngngogog j2.
Simeng Sun, Katherine Thai, and Mohit lyyer. ChapterBreak: A challenge dataset for long-range language models. In Proceedings of NAACL-HLT, pp. 3704-3714, 2022. doi: 10. 10.18633v1//22... URL https::////// aate:://llllaaa lwt.laaaaanntoogoo owtp:/olloooo.ooo/////aa 2202 maacc-maiin 271.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging BIG-Bench tasks and whether chain-of-thought can solve them. arXiv preprint arxiv:2210.09261, 2022. URL https: //pp:://aaaa http://axxxvv...gd...../ 2210 09261.pdf.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpaca: A strong, replicable instruction-follooiing model. Stanford Center for Research on Foundation Models., 2023. URL htw.hhtp/////tteeeoe stanfordd edu/2023/ 03 13/ alpaca html.
Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. Unifying language learning paradigms. arXiv preprint arXiv:2205.051/ 05131, 2022. URL https: /ttp//avvrrgg//eeeeeee /2205 051 31 .pff.
14

---

Published as a conference paper at ICLR 2024
InterLLM Team. Internlm: A multilingual language model with progressively enhanced capabilities. https: hettps://ithubs github. www.ccommmnniieee Int.rnIneeei//mmmm InternIM, 2023.
Lucas Theis, Aäron van den Oord, and Matthias Bethge. A note on the evaluatiion of generative models. In Proceedings of ICLR, 2016. URL http: /ttttp://axxv...o wttp://gppppe//eee//te 01011.0111000 01844 pdff.
Hugo Touvron, Thibaut Lavril, Gautier Izacard. Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arxiv:2302.33971, 2023. URL https:
// //taxxiv.orgg/ppele///// 2tt.//222..........1
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. MuSiQue: Multihop questions via single-hop question composition. Transactions of the Association for Computational Lingmistics, 10:539-554, 2022. doi: 10. N0.1162/acl_a_00470 URL
org/2022 taactaalllaa 1..1.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE:
A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of EMNLP Workshop BlackboxNLP, pp. 353-355, 2018. URL ptttps://aclanthol.ne
org/ Bxg/W11 /t////184444
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Fe- lix Hill, Omer Levy, and Samuel R. Bowman. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. In Proceedings of NIPS, pp. neurips cc/ pccppaap///01////////epp http//2219/hhhhh// html.
3261-3275, 2019. URL 4496bf24afed (at.AAAAATTT.aa0000066222266-ee-
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A Billion Parameter Autoregressive Language Model. https: /itthb..g com/kingofll htwp://www..eeeeettaassfoooooooe
jax, 2021.
Xiaozhi Wang. Ziqi Wang, Xu Han, Wangyi Jiang, Rong Han, Zhiyuan Liu, Juanzi Li, Peng Li, Yankai Lin, and Jie Zhou MAVEN: A Massive General Domain Event Detection Dataset. In Proceedings of EMNLP, pp. 1652-1671, 2020. doi: 10.1 06633/112020.emm///p http:///2222.....11mm0mmmmm 11..1mmmmi..229 129.
1299. URL
Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. KEPLER: A unified model for knowledge embedding and pre-trained language representation. Transactions of the Association for Computational Linguistics, 9:176-194, 2021a. doi: 10.1162/ tael a 00360. URL. https: / ac anchhllot org 2021 tacl-1.11
Xiaozhi Wang, Yulin Chen, Ning Ding, Hao Peng. Zimu Wang, Yankai Lin, Xu Han, Lei Hou, Juanzi Li, Zhiyuan Liu, Peng Li, and Jie Zhou. MAVEN-ERE: A unified large-scale dataset for event coreference, temporal, causal, and subevent relation extraction. Proceedings of EMNLP, pp- 926-941, 2022. URL https: hta/aaaaannthhoocooooooooo
org 201/22222.o0 emnlp main. 60.
Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far can camels go? exploring the state of instruction tuning on open resources. arXiv preprint arX1v:2306.04751, 2023.
Ziqi Wang, Xiaozhi Wang, Xu Han, Yankai Lin, Lei Hou, Zhiyuan Liu, Peng Li, Juanzi Li, and Jie Zhou. Cleve: Contrastive pre-training for event extraction. In Proceedings of ACL, 2021b. URL
https: / //aaaclaanthhooo org/ /2/g//2222 2021 acl - -ong. 491/
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of EMNLP, pp. 2369-2380, 2018. doi: 10. 10.18653/1/D18-1111 URL
.rg 1259.
Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. DocRED: A large-scale document-level relation extraction dataset. In Proceedings of ACL, pp. 764-777, 2019. doi: 10.18653/v1/P19- 10744 URL https:/ acl anthalogy..org/1199-1
15

---

Published as a conference paper at ICLR 2024
Zijun Yao, Yantao Liu, Xin Lv, Shulin Cao, Jifan Yu, Juanzi Li, and Lei Hou. KoRC: Knowledge Oriented Reading Comprehension Benchmark for Deep Text Understanding. In Findings of ACL, 2023. URL http::ttt////. http::/ttrassriuss github. io/ wwwes/keolet ptf.
Michael Yoder, Sopan Khosla, Qinlan Shen, Aakanksha Naik, Huiming Jin, Hariharan Muralidharan, and Carolyn Rosé. FanfictionNLP: A text processing pipeline for fanfiction. In Proceedings of the Third Workshop on Narrative Understanding, pp. 13-23, 2021. doi: 10.18653/v1/2021 .nuse- 1.2. URL https::/aclanntoooogy.ooooo org org/2021 mausee1....
Jifan Yu, Xiaohan Zhang, Yifan Xu, Xuanyu Lei, Xinyu Guan, Jing Zhang, Lei Hou, Juanzi Li, and
Jie Tang. XDAI: A tuning-free framework for exploiting pre-trained language models knowledge grounded dialogue generation. In Proceedings of KDD, pp. 4422 4432, 2022.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang. Yifan Xu, Wendi Zheng, Xiao Xia, et al. Glm-130b: An open bilingual pre-trained model. arXiv preprint arxiv:2210.02414. 2022. URL https: arxiv. org/ pdf /2ff 02414. pdf.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng. Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint urXiv:2306. 0568855300.0688, 2023.
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu,, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. AGIEval: A arXiv preprint arxiv:2304003644, 2023. URL https: / /awxiv.vvvvvv/////////a////aaa ortoorrg////.. httgg/ppp/334/// 0634440..30444 06364. pdf.
Benchmark for Evaluating Foundation Models.
Wenxuan Zhou, Kevin Huang, Tengyu Ma, and Jing Huang. Document-level relation extraction with adaptive thresholding and localized context pooling. In Proceedings of AAAI, 2021. URL https:/ //ojs. aaai. org/ index. .htp/phhpppee AAAI AAt./atticl//eeee///1111
16

---

Published as a conference paper at ICLR 2024
APPENDICES
A BROADER DISCUSSION
A.1 LIMITATION
The major limitation of KoLA is that our coverage is not as extensive as some other recent works (Hendrycks et al., 2021; Zhong et al., 2023; Huang et al., 2023). KoLA evaluates LLMs' world knowledge about concepts, entities, and events and only covers 19 English datasets now. While there is no doubt that expanding the evaluation "breadth" to test the boundaries of LLM abilities valuable, emphasis lies more the "depth" of evaluation. Due to careful design considerations regarding data sources and our annotation capacity to host a new competition season every 90 days, is not easy to significantly broaden evaluation coverage. The first season results reported in this paper involve 21 LLMs. Although we strive to cover diverse and representative LLMss, it is challenging cover the ever-emerging and evolving LLMs with solely our own efforts. We sincerely welcome community contributions and participations to introduce new tasks or LLMs.
Another limitation pertains to our evaluation of knowledge creating abilities. To avoid human evaluations. as detailed in Section 2.2 and Section 2.3, we design an automatic evaluation method based on the self-contrast metric. Experiments in Section validate the efficacy of our metric. However, our automatic evaluation still relies on contrasting model-generated content with human- provided knowledge. If model produces knowledge that is novel and reasonable but just not aligned with human-provee provided knowledge, its capabilities might be underestimated. We encourage future work to actively explore this significant avenue and endeavor to develop more effective automatic evaluation methods.
A.2 POTENTIAL IMPACT
As benchmark, the intended use of KoLA is not to construct applications or train LLMs, but rather evaluate the foundational abilities about world knowledge of LLMs. Our evaluation tasks do not involve speculating personal sensitive information, making judgments on social issues, or interacting with the real world. Therefore, we believe the likelihood of our benchmark directly leading to negative impacts on safety, security, discrimination, surveillance, deception & harassment, human rights, bias and fairness is very low. However, effective benchmarks will facilitate the development of powerful LLMs, which poses wide and serious risk of misuse. Although beyond the scope of this paper, carnestly call for strengthened cooperation from various sectors society in enhancing the regulation and safety control of LLMs.
Our evaluation needs to do inference with many LLMs on various datasets, which naturally results in carbon emissions and potential environmental issues. The total carbon emissions can be estimated based on the data provided in Appendix E. As our evaluation does not involve the pre-training and fine-tuning of I.LMs, we believe the impact caused is relatively marginal and controllable. In the participation guidelines on the platform, we also state that we discourage improving evaluation scores through repeated submissions training specifically on benchmark-relate data. This ensures the reliability of the evaluation results while minimizing carbon emissions as much as possible.
17

---

Published as a conference paper at ICLR 2024
B AUTHOR CONTRIBUTION
Data Collection. Xiaozhi Wang, Shulin Cao, Xin Lv, Hao Peng, Zijun Yao collected the Known Data Source from the open-source projects, private academic datasets and Wikidata. Jifan Yu, Daniel Zhang-Li, Nianyi Lin, Linlu Gong and Kaifeng Yun collected and pre-processed the first season's Evovling Data Source. Hailong Jin supported the data from Xlore (Jin al., 2019). Yuan Yao provided the test set of DocRED (Yao et al., 2019) and Ning Ding helped the construction of FewNERD (Ding et al., 2021) test set.
Data Annotation. Xiaozhi Wang organized the crowdsourcing annotation of fine-grained event arguments for knowledge creating (KC) tasks. Kaisheng Zeng and Yong Guan assisted in the quality control process. Jifan Yu and Daniel Zhang-Li organized the annotation of knowledge triples from the evolving articles, as well as the human evaluation of creating results.
Task Construction. Xin Lv designed the dataset and instruction of the knowledge memorization (KM) tasks, (1-1) and (1-2). Hao Peng organized the knowledge understanding (KU) tasks and constructed instructions for (2-1/2-2/2-3). Zhili Wu, Yunjia Qi, Jianhui Chen and Weikai Li correspondingly constructed instructions for task (2-4) to (2-7), Shulin Cao and Zijun Yao organized the knowledge applying (KA) tasks and constructed instructions for (3-1), (3-3), (3-4) and (3-5). Yantao Liu and Amy Xin constructed task (3-2) and (3-6). Nianyi Lin and Daniel Zhang-Li organized the knowledge creating (KC) tasks (4-1), (4-2) and most of the evolving tasks. Kaifeng Yun and Linlu Gong constructed instructions for task (1-3) and (2-8).
Model Evaluation. Shangqing Tu organized the whole process model evaluation. Hanming
Li deployed and conduct experiments of GPT-J (6B), GPT-JT (6B), GPT-NeoX (20B), BLOOM (7B), T0++ (11B), LLLMa (65B), GL.M (130B), U1.2 (20B), FLAN-T5 (11B), Alpaca (7B), FLAN- UL2 (20B), ChatGLM (6B). Chunyang Li evaluated GPT-3 curie v1 (6.7B) and davinci v1 (175B), InstructGPT curie v1 (6.7B*) and davinci v2 (175B*), ChatGLM (130B), and GPT3. -tturb. Zheyuan Zhang evaluated Cohere-commant (52.4B) and 12-Jumbo-Instreut Instreut (178B*), while Yushi Bai evaluated GPT-4. Qi, Daniel Zhang-Li Li and Jinxin Liu assisted the data analysis and presentation on the inference results. Yu Gu supported some of the candidate APIs.
Platform Development. Xiaohan Zhang's 's team organized the overall platform development. Jifan Yu and Daniel Zhang-Li respectively contributed to the visualization design and backend development.
Paper Writing. Jifan Yu and Xiaozhi Wang hosted the paper writting. Shangqing Tu, Ji Qi, Daniel Zhang-Li supported the part of experimental analysis. All the authors contributed to their corresponding working details for completing this paper (including Appendix).
Advising. Lei Hou, Zhiyuan Liu, Bin Xu, Jie Tang and Juanzi Li take advisor roles in this project. Juanzi Li is the main advisor, initializing, supporting and organizing this project.
This project also express appreciations to other supporters. Special thanks go to Gang Wang for his outstanding product prototype design. The artistic design, including the beautiful illustrations for the paper, was skillfully provided by Shanshan Wang. The platform development and feature implementation were carried out by Zhenfang Lu, Shuai Xie, Shuaiming Wang, Liangliang Cui and Dingxiao Liu. The coordination of crowdsourcing affairs was effectively managed by Jupeng Zhang and Yue Yang. Finally, the long-term assistance and support from Yini Chen to the AIGC research team are greatly acknowledged.
C DETAILS OF DATA COLLECTION
There are two data sources used in KoLA: Known and Evolving, which are correspondingly from Wikipedia and newly crawled corpus. In this section, erview of the collection and maintenance of them, as well as the annotation process involved, including essential statistical information.
C.I RAW DATA COLLECTION
Known. We collect the corresponding Wikipedia articles for the entities in Wikidata5M (Wang et al., 2021a) from Xlore2 (Jin et al., 2019), a cross-lingual knowledge base Chinese and English, using its 2019 version, and align them accordingly. This process generates a dataset of 5 million
18

---

Published as a conference paper at ICLR 2024
articles. Given that language model training is reliant textual data, depart from the conventional graph-based based methods typically employed in knowledge graphs to calculate entity frequencies. Instead, conduct statistical analysis to determine and rank the occurrence frequencies of these entities and their aliases within the text corpus. Subsequently, we establish two sets of high-frequency and low-frequency entities, each containing 000 entities, to fulfill the requirements of the knowledge memorization task 1-1 and 1-2.
Evolving. The data collection process for the Evolving dataset in the first season of KoLA concludes on April 15, 2023. Therefore, we are collecting data from the preceding 90 days (January 15, 2023, to April 14. 2023). In terms of news data, we are experimenting with multiple open-source news scraping interfaces. Our primary focus is on gathering articles that have rich event elements, such as factual news and entertainment news. We have collected total of 1 000 such articles, aiming to avoid sensitive news categories like politics. As for the novel data, we randomly selected 1, 000 works from a renowned creative platform striving to achieve a balanced representation of various writing types (fan fiction and original works). Afterward, we employ open-source tool to filter out chapters containing explicit, violent, or other inappropriate content, sorting them based on the number of views. Following this filtering process, we ultimately retain 250 articles each from the news and novel categories as candidates (500 in total) for high-quality texts.
C.2 DATA ANNOTATION
During the construction of the KoLA benchmark, we have two key annotation tasks: (1) Fact Triple Annotation and (2) Event Argument Annotation. The annotated triples from (1) will subsequently be utilized for the construction of the evolving test sets for Task 1-3, 2-8, and 3-6. Meanwhile, the annotated event attributes from (2) will be employed for Task 4-1 and 4-2. Throughout the annotation process, we meticulously provided comprehensive instruction documents and implemented reasonable model are-annotations, aiming to direct the annotators' focus to the most essential parts, thereby striving to enhance the quality of the primary annotated data. Subsequent to this, we also conducted multiple rounds of quality checks to ensure the reliability of the final constructed test set.
Annotation Team. We hire a 21-member annotation team (based on market rates) comprising experienced annotators. With their permission, we gather some basic information about the annotation team and present it in Table 4. In general, the annotation team consists mostly of individuals with graduate-level qualifications. The validators are three Ph.D. holders from the KoLA team. With the collabo- ration of this high-qualit, team, strive to ensure the efficiency and quality of data annotation. Before initiat- ing the annotation work, we enter into legally binding contract with the team to protect the rights of the annota- tors. We also develop a dedicated platform specifically for the annotation process, which facilitates efficient review, publishing, and exporting of the annotation results. The platform allows annotators to have flexibility in choosing their working hours, including the ability to save intermediate results, retrieve relevant resources, and log or log out at any time.
Table 4: Statistics of the annotation team for dataset construction in KoLA.
Gender
Rate
Female Male
85.7%
14.3%
Education
Rate
Bachelor
47.6%
Master
52.4%
Annotation Process and Quality Control. For the aforementioned annotation tasks, we carefully employ methods for data pre-processing to facilitate the annotation process and quality control.
For Task (1), the text to be annotated includes only the Evolving data. We take the following steps to obtain the factual knowledge triples and whether they can be infered from the previous corpus.
1. Named Entity Recognition. We first utilize named entity recognition tool to extract entities from the articles. This model exhibits strong performance on four entity types (PER, ORG, LOC, MISC) with a Precision of 90. 90.7%, Recall of 91.9%, and an F1-score of 91.3%.
"hrctps://archhieeooorromm
co/ "herpss:/huugiingaccc.c ds/ wwwp:///ettteeemeeeeeeeee
19

---

Published as a conference paper at ICLR 2024
After removing small percentage of incompletely recognized ones (< 5.3%), these results are used as input for subsequent processes.
2. Relation Extraction. Subsequently, we replicate a renowned document-level relation extraction model, ATLOP (Zhou et al., 2021), using these entities and documents to extract potential relationships and organized them into triple formats. Due to the task's complexity, the model achieves only an F1 score of 63.4%. Therefore, during replication, we reduce its prediction threshold to 0.4 further enhance the model's Recall (to 84.5%), aiming to minimize the omission of triples during the annotation process.
3. Triple A Annotation. After the aforementioned preprocessing, a total of 23.1k candidate triples entered the annotation phase. Each annotator is permitted to use online searches and is instructed to categorize each candidate triple into the following three classes: a) incorrect triples, b) correct triples that can be known prior to the current season, and c) correct triples that can only be known after the current season began. Each triple is guaranteed to be annotated by two annotators. Eventually, the Cohen's Kappa for the correctness of the triple annotations (a vs b+c) is 0.71, and the Cohen's ' Kappa for whether the triple could be discovered before January 15th (b vs is 0.55, indicating fair good agreement such issues. In cases of classification discrepancies, the decision is deferred to the quality check lead.
For Task (2), the text to be annotated includes both Known and Evolving data. As for the Evolving data, we also conduct processing as:
Event Detection. We also employ the named entity recognition results (as in the annotation task (1)) to reproduce the Omni-Event toolkit"" Specifically, we replace the backbone model with CLEVE (Wang et al., 2021b) for event detection, which achieves a Recall of 81.5% on the ACE2005 dataset and 72.6% on MAVEN dataset. The detected events are subsequently used as candidates for event argument annotation.
The Known data portion utilizes articles from the MAVEN dataset (Wang et al., 2020), where the event trigger words and event types have already been annotated, requiring no additional pre-processing. Then, articles with detected events from both known and evolving data are processed into annotations.
2. Event Argument Annotation. Annotators are required to perform the following tasks: a) annotate candidate attributes for each event, and b) correct or delete events if the event trigger words are incorrect. To guarantee the annotation quality, we build an online annotation platform that indexes examples for all 159 event types (with average of 7 argument roles per type). The annotated results subsequently underwent two rounds of quality checks. Annotations deemed unsatisfactory in each round are returned for re-annotation. The first- time pass rate for the first round of quality checks is 67%, and 91 % for the second round. The overall Fleiss' Kappa for the annotation is 0.62. Ultimately, on average, 57.8 events and 235.3 arguments were annotated per article.
C.3 AVAILABILITY
Platform. We develop an online platform to offer range of services to the community, such as competition news updates, visualizations of evaluation results, and convenient access submit new models or modify previous submissions. Due to the dynamic nature of the KoLA evolving data source, new results and rankings will be generated in each season, and a selection of results from previous seasons will also be publicly available.
Participate in KoLA. Researchers can participate in KoLA evaluations in two roles. (1) Competitor: KoLA welcomes open participation for each season by providing the model's APIs parameters, and provides high-quality examples for each task to help participants debug. It worth noting that KoLA does not allow the local evaluation to prevent test set leakage and unfairly overfitting datasets. (2) Contributor: We maintain a special interest group (invitation link shown at the "About" page of the platform) where volunteers who have ideas for result analysis, model refinement, and benchmark improvement can discuss, propose suggestions, and participate in task construction.
Whttps:// //t::////tti thub. www.coom///oooooo THO//THU--T XKG/Omm Event
20

---

Published as a conference paper at ICLR 2024
Figure The distribution of the top-20 annotated triples terms of relation type. Blue bars represent the correct triples that can be known prior to the current season, and red bars represent the correct triples that can only be known after the current season begins.
1,000
70-
50
400
Figure 5: The distribution of the events and text length in the articles from Wikipedia (MAVEN). Blue bars and red bars correspond to the number of events and the tokens in each article.
Supporting Tools. We release a toolkit to support KoLA-related functions Github, including: (1) Easy-to-submit. Competitors can employ this function to independently maintain the in-context prompts for each task while providing a single model API, making the submission and modification convenient. (2) Resuli Reproduction. We provide the code and developed tools that used in our data visualization and standardization, which support result reproduction and other analyses. Acquisition. We also provide data access API, which assists authorized users in getting the evolving data and results of previous seasons.
(3) Data
C.4 DISTRIBUTION OF THE ANNOTATED RESULTS
For Task (I), we annotate all 500 articles and retained 2.7K correct triples, out of which only 459 triples cannot be found in earlier corpora. Figure 4 illustrates the distribution of relations for all correct triplets, where blue bars represent category b) correct but can be known before the certain season; and red bars are the number of category e) correct and cannot be known before the certain season. It can be observed that even the Evolving data, the triplets still exhibit a long-tail distribution. Most of the triplets, such as "country of citizen" and "country", do not effectively convey the main content of the articles. This further reinforces our goal of annotating fine-grained event-level knowledge.
For Task (2), we specifically select 100 articles from the MAVEN and Evolving datasets, ensuring they possess extensive knowledge (i.e., a substantial number of entities and event triggers that are not within the lowest 20% frequency range). After completing the annotation process, we examine the number of valid events (events containing at least one argument) and the article lengths, as illustrated in Figure 5 and Figure 6.
21

---

Published as a conference paper at ICLR 2024
000
Figure 6: The distribution of the events and text length in the articles from Evolving Data.
In general, the distribution of event knowledge between the two datasets is quite similar. The articles in the Evolving dataset are generally longer compared to the Wikipedia articles MAVEN, resulting challenges
in higher number of valid events. Due this difference, models may encounter when performing tasks on the Evolving dataset, but they may also benefit from greater exposure knowledge. Therefore, the varying performance of models on the tasks upon these data sources may be influenced by factors such as the model's parameter size or training adequacy. This phenomenon warrants further analysis and discussion.
C.5 ANNOTATION COST
To sustain the project over the long term, we have kept the budget for each season below USD 2,000. Generally, each season's expenditure comprises two main parts: 1) the cost data annotation, and 2) the expenses for model deployment and API calls.
The data annotation includes labeling knowledge triplets and event arguments. As introduced in Appendix C.2, to reduce the difficulty and cost of annotation, have pre-labeled the collected text automatically, allowing annotators to focus on the core tasks. Overall, each season requires approximately USD 660-700 for triple annotation and USD 400-420 for event annotation.
Due to the scale of the KoLA test sets, the costs for model deployment and API calls have been kept within an acceptable range. The deployment expenses, estimated from GPU usage time, are approximately USD 200-240, while the API incurs an additional cost of USD 150-180.
Overall, for the seasons completed thus far, the costs have exceeded USD 1,600 per season. With the anticipated increase in the number of models in future seasons, we believe that a budget of USD 2,000 per season should be sufficient maintain the project.
22

---

Published as a conference paper at ICLR 2024
D DETAILS OF TASK INSTRUCTION
After completing the data collection, we proceed to construct separate test sets for tasks at different levels, thereby transforming knowledge-related tasks into language tasks driven by instructions, which facilitates the execution by large-scale models. In this section, we first present the design principles for each level of tasks, followed by specific approaches to constructing detailed instructions, and provide corresponding task examples.
D.1 CONVERTING TO TASK FORMAT
There are tasks' test sets that need to be constructed from scratch ((1-1) High-Freq. (1-2) Low- Freq., (4-1) Encyclopedic Knowledge Creating and Evolving Test tasks). These tasks require the reconstruction and quality control of the data based on our annotations. As for the other 12 tasks, only focus designing how they be transformed into sequence tasks that be solved by language models, considering the dataset construction methods provided in the original text as references. Overall, we follow two principles during the process of constructing instructions: a) Simplicity: We aim to describe the task objectives using the least amount of text, thus saving the model's 's in-context length; b) Standardization: We use special markers to identify all structured knowledge, assisting the model in quickly capturing the knowledge objectives.
D.2 KNOWLEDGE MEMORIZATION TASKS
Knowledge Memorization (KM) level primarily assesses the model's ability to retain knowledge triples. However, this format is not inherently suitable for large models. Therefore, transform the triple prediction task into a duestionn-nsweele task that considers 1-to-N relationships. For each type of relationship, we design specific templates to facilitate this transformation.
INSTRUCTION: Please give answers to the following questions about knowledge. Note: If there are more than answer, print them all and separate them with semicolon (:). Please do not give anything other than the answers.
QUESTION: What is the occupation of Wang Guozhen?
ANSWER: poet
Table 5: The instruction and an example of Task 1-1 High-Freq. K.M.
Please give answers to the following questions about knowledge. Note: If there is more than one answer, print them all and separate them with semicolon (:). Please do not give anything other than the answers.
INSTRUCTION:
QUESTION: Which country does White Hall Township belong to?
ANSWER: United States America
Table 6: The instruction and an example of Task 1-2 Low-Freq. KM.
Please give answers to following questions about knowledge. Note: there is more than one answer, print them all and separate them with semicolon (:). Please do not give anything other than the answers.
INSTRUCTION:
QUESTION: Will Messi still serve in Paris Saint-Germain?
ANSWER: No
Table 7: The instruction and an example of Task 1-3 ETM KM.
23

---

Published as a conference paper at ICLR 2024
D.3 KNOWLEDGE UNDERSTANDING TASKS
Knowledge Memorization (KM) level involves various levels of structured information, such as concepts, entities, relationships, and events. It also incorporates multiple documents, even the multi-document level, which can easily overwhelm the model and distract from the main task. Therefore, we have placed strong emphasis on standardizing the input and output of this layer to facilitate the model's comprehension of the task objectives. Additionally, we strive to balance the simplicity of the input while ensuring comprehensive understanding.
Specifically, the instructions for each task are as follows:
INSTRUCTION: Conceptual similarity judgment
Among Tutu Chengeui, The Pierre, Waddesdon Manor, Astro Orbitor, Heian period, 2019 Canadian federal election, Paradiski, Tenughat Dam, Gros Michel banana, Reedy Glacier, Gangotri Glacier, Pinatuho, Interwar period, djon djon. Qiu Shiliang, Caciotta, Firth of Forth, 2011 Rugby World Cup. Cheng Yuanzhen, Pliocene, Sri Maha Bodhi, which one is the most conceptually similar with Botryosphaeria stevensii? Please answer the entity name only.
QUERY:
ANSWER: djon djon
Table 8: The instruction and an example of Task 2-1 COPEN-CSJ, KU.
INSTRUCTION: Conceptual property judgment
QUERY: Is the statement "Specieses have a cellulose wall and other polysaccharides." true or false? Please true or fälse.
ANSWER: False
Table The instruction and an example Task 2-2 COPEN-CPJ, KU.
INSTRUCTION: Conceptualization contexts
Given the context "The next year, he made his stock car racing debut the American Speed Association, where he won pole at Winchester Speedway and had four top-tens.", neglect your knowledge about Winchester Speedway and select the most contextually related concept for from the concept set: Racecourse, Place, ArchitectuallStmacure, Infrastructure, RaceTrack, Venue, Road, SportFacility, RouicOfTransporaaiion, Building. Please answer the concept name only.
QUERY:
ANSWER: Racecourse
Table 10: The instruction and an example of Task 2-3 COPEN-CIC CCC, KU.
24

---

Published as a conference paper at ICLR 2024
INSTRUCTION:
Please recognize entities for the given text and classify them into suitable type. The collection of types is as follows: <set of types>
QUERY: Agrippa succeeded in blocking more manoeuvrable ships of Sextus and, after a long and bloody fight, to defeat his enemy.
A ANSWER: Agrippa: person-politician; Sextus: person-politician;
Table 11: The instruction and an example of Task 2-4 FewNERD, KU.
INSTRUCTION: Please follow the above demonstration, and extract relations from the [Question text]. Note the relation needs to be the predefined set of relations. The output format required to is the same as demonstration, format(<<ntin) ID>, relation, <entity ID>). The predefined set of relations: <set of types>
<entity_0h 0t> Rickon Stark </entity_Os 00> is a fictional character in the <entity_ A Song of Ice and Fire </entity_ series of fantasy novels by <entity_22 American </entity_ author <cntity_3> George R. Martin </entity_3: and its television adaptation <entity_4e _4 Game of Thrones </entity pentity_44____ Introduced in <entity_Ss 5> 1996 </entity_St _5> <entity A Game of Thrones </entity 6>, <entity_ 0> Rickon </entity O> is the youngest child of <entity_7> 7> Eddard Stark </entity_ 7>, the honorable lord of <entity _>> Winterfell <lentity_ 8>, an ancient fortress in the <entity 9> North </entity_ 9> of the fictional kingdom of <entity 10> Westeros </entity_ 10>> He subsequently appeared in centity 3> Martin </entity 3> 's <entity_ _ltitt____>>__ 11> A Clash Kings </entity_ 11> (<entity 12> 1998 </entity_ (22n) The <entity 13> Publishers Weekly </entity 13> review of <entity 6> A Game of Thrones </entity 6> noted, It is fascinating to watch <entity 3> Martin <lentity 3> 's characters mature and grow, particularly <entity 14> Stark </entity 14> 's children, who stand at the center of the book. <entity_ 0> Rickon </entity 0>> is played by <entity_ 15> Art Parkinson </entity_ 15> in the <entity_16> HBO </entity_ 16> television adaptation.
QUERY:
ANSWER: (<entity_ 0>, father, <entity <>>); (<entity 0>,, present work, <entity_ 4>); (<entity _0siiit_________ creator, <entity 3>); (<entity_7>> 7>, child, <entity 0>); (<entity_7s 7>, present in work, <entity_ 6>); (<entity_ present in work, <entity_45 4>); (<entity_6>> publication date, Sentity_5>>) (<entity_6e ___,,_,,,0__0 characters, <entity_ __>;: (<en- tity_6 6_, author, <entity_3>>> icentity_4>, publication date, <entity_55 (<entity 4>, characters, <entity_ _>)); (<entity_ 4>, has part, centity_6>>) (centity_44 4>, author, <entity_3x 3>>) (<entity_45 __, screenwriter, <entity_33 (centity_ _4b original network, contit__6<y) (<entity_ _3,, notable work, <entity_6n (<entity_ _0>, present in work, <entity_ 4>); (centity_ l>, publication date, <entity_129> (centity_ 11yyy11111 11,, series, <entity <_>); (centity_ 11>>, follows, <entity_6>> (<entity_ 11>, series, <entity 4>); (<entity_ 11>,, author, <entity_3>) (centity_ 14>, present work, <entity 4>): (<entity 14>, creator, <entity 3>);
Table 12: The instruction and an example of Task 2-5 DocRED, KU.
Please identify the events in the text and classify them into appropriate categories; The collection of categories is <set of types>
INSTRUCTION:
QUERY: The ruling National Command of the Arab Socialist Ba' ath Party removed from power by union of the party's 's Military Committee and the Regional Command, under the leadership of Salah Jadid.
ANSWER: removed-Removing
Table 13: The instruction and an example of Task 2-6 MAVEN, KU.
25

---

Published as a conference paper at ICLR 2024
Please classify the relation between two Timets/"Time" in given document. There 10 types of relations: ["before", "overlap", "contains", "simultancous", "begins-on", "endsson", "cause", "precondition", "subeven", and "coreference"]. In each document, 2 "vents/"Timex' are marked as "<Event> event name </Event>" or "<Timex> Timex name </Timex>". ". If there is a relation type or multiple relation types, the answer form is "Answer: [relation type 1, relation 2ype ...""
INSTRUCTION:
The Central Park jogger case was criminal case in the United States based on the assault and <Event> rape </Event> of Trisha Mcili, 28-year-old white woman who was jogging the park, and attacks on eight other persons, in areas ranging from the North Woods of Manhattan's 's Central Park to the Reservoir, on the night of April 19, 1989. Three of the victims were black or Latino. Mcili was so injured that she was in a coma for 12 days. "The New York Times" in 1990 described the attack on her as "one of the most widely publicized crimes of the 1980s". Attacks in Central Park that night were allegedly committed by loose group of 30201332 leenagers, and police attempted to apprehend suspects after crimes began to be reported between 9 and 10 p.m. The brutally beaten Meili was not found until 1:30 a.m., after which the police hunt greatly intensified. They took into custody 14 or more other suspects over the next few days, and arrested a total of ten suspects who were ultimately tried for the attacks. Among them were four African American and two Hispanic American teenagers who were indicted on May 10 on charges of assault, robhery, riot, rape, sexual abuse, and attempted murder of Meili and an unrelated man, John Loughlin. The prosecutor planned to try the defendants in two groups, then scheduled the sixth defendant to he tried last. The latter pleaded guilty in January 1991 on lesser charges and received reduced sentence. Prosecution of the five remaining defendants in the rape and assault case was based primarily on confessions which they had made after police interrogations. None had counsel during this questioning. Within weeks, they each withdrew these confessions, pleaded not guilty, and refused plca deals on the rape and assault charges. None of the suspects DNA matched the DNA collected from the crime seene: two semen samples that both belonged to one unidentified man. No substantive physical evidence connected any of the five tecnagers to the rape scene, but each was convicted in 1990 of related assault and other charges. Subsequently known as the Central Park Five, they received stiff sentences ranging from 5 to 15 years. Four of the defendants appealed their convictions, but these were affirmed by appellate courts. The four juvenile defendants served 620137 years each; the 16-year-old, tried and sentenced as an adult, served 13 years in adult prison. The five other defendants, <Event> indicted </Event> for assaults of other victims, pleaded guilty to reduced charges and received less severe sentences. In 2001, Matias Reyes, convicted murderer and serial rapist serving life prison, confessed officials that he had raped the female jogger. His DNA matched that found at the scene, and he provided other confirmatory evidence. He said he committed the rape alone. Reyes could not be prosecuted for raping Mcili, because the statute of limitations had passed. In 2002 Robert Morgenthau, District Attorney for New York County, had his office conduct investigation and recommended to the state court that the convictions of the five men on all charges be vacated. The court vacated their convictions in 2002, and the state withdrew all charges against the men. In 2003, the five men sued the City of New York for malicious prosecution, racial discrimination, and emotional distress. The city refused to settle the suits for a decade, because its lawyers believed that the city could win a court case. After a change in administration, the city settled in 2014 with the five plaintiff's for S41 million. The five men also filed suit against the State of New York for additional damages; this case was settled 2016 for a total $3.9 million. The first Timex"" <Event> rape </Event>, The second event/"Timex" <Event>> indicted </Event>.
QUERY: Document:
A ANSWER: before
Table 14: The instruction and an example of Task 2-7 MAVEN-ERE, KU.
Please follow the above demonstration, extract relations from the [Question text]. Note the relation need to be in the predefined set of relations. The output format required to is the same as the demonstration, format(<<entity ID>, relation, <entity ID>). The predefined set of relations: <set of types>
INSTRUCTION:
Less than four months removed from the entity0 World Cup entityO's bright lights the U.S. men's national soccer team visited an 8,000 seat bayside stadium on tiny entityl Caribbean entityl 1 island Friday to face an opponent with players from regional leagues and some of entity? England entity2's 's lowest divisions entity3. <text continued> Relations in the predefined set of relations in the above text:?
QUERY:
Text:
ANSWER: (cntity4, part of,ynti 13); (entity6,pan part of, ontityt 13); (entity of.entityO); (entity 15,country of citizenship.entity 9): (entity 16,country of citizenship.entit9); (entity 17,cuuntry of detizennhhi.eenityyy (entity 18, country of citizenship.entity)): (entity 19,nouy of citizenship.entig (en- tity20, of citizenshipp.eniityy)
13, part,entity:4; (entity 1 15, p.aatticipant
Table 15: The instruction and an example of Task 2-8 ETU, KU.
26

---

Published as a conference paper at ICLR 2024
D.4 KNOWLEDGE APPLYING TASKS
Knowledge Applying (KA) level naturally involves multi-hop reasoning in the form of question answering, which suitable for large-scale models to perform inference. One major challenge is that the contexts required for these reasoning steps are not consistent. One particular aspect is the KoRC task and ETA task, which assume that the model has access to a corresponding knowledge base. The original dataset for KoRC confirms Wikidata as the knowledge base. However, in the Evolving data, many pieces of knowledge cannot be directly found. Therefore, we construct a virtual knowledge base using the annotated triples and generate questions based on this knowledge base.
Finally, we adopt the following task instructions:
INSTRUCTION: Please answer the following question.
QUERY: Jeremy Theobald and Christopher Nolan share what profession?
ANSWER: Jeremy Thenbald Therefore, they both share the profession of being producer. So the answer is: producer.
and producer. Christopher Nolan director, producer, and screenwriter.
Table 16: The instruction and an example of Task 3-1 HotpotQA, KA.
INSTRUCTION: Please answer the following question.
QUERY:
Which film came out first, Blind Shaft or The Mask Of Fu Manchu?
ANSWER: Blind Shaft is a 2003 film. The Mask Of Fu Manchu is 1932 film. So the answer is: The Mask Of Fu Manchu.
Table 17: The instruction and an example Task 3-2 2WikiMulthoopQA, KA.
INSTRUCTION: Please the following question.
QUERY: When did the first large winter carnival take place the city where CIMI FM licensed broadcast?
ANSWER: CIMI FM is licensed to broadcast Quebec City. The first large winter carnival in Quebec City took place in 1894. So the answer is: 1894.
Table 18: The instruction and example of Task 3-3 MuSiQue, KA.
27

---

Published as a conference paper at ICLR 2024
INSTRUCTION: Please answer the following question.
QUERY: When Neville A. Stanton's employer founded?
ANSWER: The employer of Neville Stanton is University of Southampton. The University of Southampton was founded in 1862. So the answer is: 1862.
Table 19: The instruction and an example of Task 3-4 KQA Pro, KA.
INSTRUCTION: You are given one document and one anonymized real-world entity with one more mentions in the passage. Then we will ask your question about this anonymized entity. The questions cannot be answered solely within the document or the background knowledge. Your task is leverage world knowledge you have like Wikipedia or Wikidata as background knowledge combined with the given document to answer the question related to the anonymized entity. You must output all answers in the end.
Allen is county in the U.S. state of Ohio. As of the 2010 census, the population was 106,331. The county seat is Lima. The county was created in 1820 and organized in 1831. The county is named for Colonel [a human being], who was killed leading his men at the Battle Frenchtown, during the War 1812. has also been claimed the county was named for Revolutionary War soldier Ethan Allen, but the weight of the evidence in favor of [the human being] led the General Assembly to declare in 1976 that the county was named for him. Allen comprises the Lima, OH Metropolitan Statistical Arca, which is also part of the Lima- Van Wert Wapakoneta, OH Combined Statistical Arca.
QUERY:
QUESTION: Which place was this human being born?
ANSWER: Rackbridge County.
Table 20: The instruction and an example of Task 3-5 KoRC, KA.
INSTRUCTION: You are given one document and one anonymized real- world entity with one or more mentions in the passage. Then we will ask your a question about this anonymized entity. The questions cannot be answered solely within the document or the background knowledge. Your task is to leverage world knowledge you have like Wikipedia or Wikidata as background knowledge combined with the given document to answer the question related to the anonymized entity. You must output all answers in the end.
Six months after its New Shepard rocket suffered a failure during flight, Blue Origin said Friday its review of the incident pinpointed problem with its engine nozzle and that it is expecting to return to flight "soon." In September, the rocket lifted off and flew for just over minute before bright flames flashed from the booster and the capsule's 's emergency abort system kicked in, propelling it away from the rocket. The mission carried only science experiments; no one was on board, and no one was injured on the ground. In a statement Friday. Blue Origin, the space venture founded by Amazon executive chairman Jeff Bezos, said that it would refly the mission, again carrying scientific payloads. (Bezos owns [daily newspaper].) A flight with people could come later. The vehicle designed carry as many six people the edge of space and back suborbital tourist trips that allow passengers to experience weightlessness and view the earth from above. In the statement, Blue Origin said its investigation, which was overseen by the Federal Aviation Administration and included members of the National Transportation Safety Board, concluded that the problem was caused by failure of the engine nozzle, which experienced "temperatures that exceeded the expected and analyzed values of the nozzle matcrial." Engineers are "implementing corrective actions, including design changes the combustion chamber and operating parameters," the statement said, "Additional design changes to the nozzle have improved structural performance under thermal and dynamic loads." The FAA said in statement that it is reviewing Blue Origin's 's mishap report but that the investigation remains open. "FAA approval is required to close the investigation and for the New Shepard system return flight." It was unclear how long that could take. While the booster was lost, the capsule and the 36 payloads it was carrying landed safely under parachutes and can fly again, Blue Origin said. The booster, which under normal circumstances falls back to Earth and touches down softly on a landing pad so that it can be reused, was a total loss. The company was able to recover all the debris from the rocket within the designated hazard area, it said. Bezos flew the first flight with people in 2021. It had since flown five other missions with people on board, including one with Star Trek actor William Shatner and television commentator Michael Strahan. It has not flown since the September incident.
QUERY:
QUESTION: What is the home country of [daily newspaper]?
ANSWER: United States
Table 21: The instruction and an example of Task 3-6 ETA, KA.
28

---

Published as a conference paper at ICLR 2024
D.5 KNOWLEDGE CREATING TASKS
The Knowledge Creation (KC) level is particularly unique as each task involves two generation processes. Here, we present the process that considers generating subsequent knowledge, which is the most informative. However, for direct generation, the instruction can be replaced with "Complete the following generate" without specifying the "TRIPLETS" item.
The example instructions of creating tasks are shown below:
INSTRUCTION: Complete the following texts and make sure contain all the events provided.
Title Death Freddie Gray:##h Known Events Event Trigger: charges: Event Type: Judgment communication,#fttit Event Arguments; Agent: Marilyn Mosby; Patient: six police officers; Reason: the medical examiner's 's report ruled Gray's 's death a homicide; Event Trigger: Hifthd Event Type: ; Event Arguments; Speaker: The prosccutors; Message: they had probable cause to file criminal charges against the six police officers; Details: who were believed be...
EVENTS:
### Text To Be Completed: On April 12, 2015, Freddie Carlos Gray, Jr., a 25-year-old black man, was arrested by the Baltimore Police Department for possessing what the police alleged was an illegal knife under Baltimore law. While being transported in police van, Gray fell into a coma and was taken to trauma center Gray died on April 19, 2015: his death ascribed to injuries to his spinal cord On April 21, 2015, pending an investigation of incident, Baltimore police officers were suspended with pay...
GIVEN CONTEXT:
On May 1, 2015, the Baltimore City State's Attomey, Marilyn Mosby, announced her office had filed charges against six police officers after the medical examiner's report ruled Gray's death homicide. The prosecutors stated that they had probable cause to file criminal charges against the six police officers who were believed be involved in his death...
REFERENCE COMPLETION:
Table 22; The instruction and an example of Task 4-1 Encyclopedic, KC.
INSTRUCTION: Complete the following texts and make sure contain all the events provided.
EVENTS: ## <Title Containeds;##H Known #vents:##l# Event Trigger: uncovered, Event Type: Reveal secret;##### Event Arguments: Speaker: Alston & Bird; Message: no facts to show that Soccer knew of the 1992 Incident when it hired Mr. Berhalter; Receiver: Gregg Berhalter, Event Trigger: harm; Event Type: Bodily harm;#### Event Arguments: Agent: Grege Berhalter: Cause: others; Location: United States;
GIVEN CONTEXT: ### Text To Be Completed; Details of sordid rift between two prominent S. soccer familics one that included allegations of domestic abuse against men' s national team coach Gregg Berhalter and parental complaints about Gio Reyna' s playing time at the 2022 World Cup - continued to spill out Monday when the findings of an independent investigation were released...
REFERENCE COMPLETION: Earnie Stewart left the job last month. Anthony Hudson, World Cup assistant, the interim coach. The next coach will begin preparing the U.S. team for the 2026 World Cup. which will take place the United States, Mexico and Canada. Berhalter guided the United States for four years, leading young squad to two regional championships and a place in the World Cup, where it finished second in group play and lost to the Netherlands in the round of 16...
Table 23: The instruction and an example of Task 4-2 ETC, KC
29

---

Published as a conference paper at ICLR 2024
E DETAILS OF RESULT INFERENCE
Given the instructions and test sets for each task, we evalute a total of 21 models in the first season of KoLA. Here, we present some of the deployment environments of the models that participated in our first season, as well as some specific solutions implemented during the evaluations.
E. DEPLOYMENT ENVIRONMENT AND MODEL INFORMATION
The participating models in the evaluation include two types: closed-source models that return answers through API calls, and open-source models that are deployed directly for inference (with a temperature set to 0). Here, we primarily introduce the software and hardware environment used for deploying the models. We utilize the widely-used PyTorch and transformers library to load open- source models. The evaluation experiments are conducted on an Ubuntu 20.04.4 server equipped with 112 Intel Xcon(R) Platinum 8336C CPU cores, and graphic cards that contained 8 NVIDIA A100 SXM S0GB GPUs Besides, The CUDA version is 11.4, the Python version is 3.10.0, the PyTorch version is 2.0.0 and the transformers version is 4.28.1.
Table 24 presents the features of the selected LLMs in the first and second season. For the open-source models, we deploy them using their official versions, with particular emphasis on the HuggingFace versions. As for the closed-source models, we utilize the various model APIs available as of May 15, 2023. We also conduct thorough checks and re-inferencing in case of any network-related errors.
Table 24: Selected LLMs. Instruct and |Contextl correspond to whether the model is with instruction tuning and the input context's 's length limitation. indicates the size has not been officially confirmed.
Model
Size
Type
Iasinaci
ICunted
Welneine Uil
Secuin
Cquon-roure Modele
Oyen
2,048
hangges / /uugs in ingt es ce/itti bankk
Open
2,085
heago / mwww. cuccher aw/artittttt////pt
GPE-NooX
2011
2,085 heaps
hucher wttpe//attttttt///eaaea
hempes/ geod hwggtnggggnnnn
T0
11B
512
LL
658
2,045
2,045
otan /ddddddd//2222//////////
U1.
208
Oyen
512
he p
ing mighaiie wet.ee///pooee /gogle//1222222
FLAN-TS
11B
Open
512
hes /gishub con /gogglg www.szchh//1aa
ULA
2001
Open
w/
2,005
ht
//www. aty net 221lo///aa-11222000
GLM
1308
Open
win
2,055
he pus htttthhobbbboo.ooooo thib con//oortteeee /T///////GG--- 1303
CleeCil
6R
Open
w/
2,045
he mttpps/// //sg////tt http///ttttt chob cone (rt/ 5A
Seomme (Coord-Soury Madris
ChaGGLM
2,045
Publidty Avullsblee, Comming Socn
GPT.3 curle
API
2,045
he www.:aac...ooaaaaa...... owe....enai........mm..........m con/everviev
GPT daxinci
heopas //l waw..oocnaa..co.mm con//vorviee
Issimes[PP
[Ps davinai
1758
API
GPT3.5 S-uubo
API
2.045
fas..opena
GPP-4
API
2,085
foom openal /wweeveeeeee
Cohern-ommened
$2.48
API
//doce. ootereeeeee.... con/ /doos/// htwp///-eoomddddd command-nodde
12 Jumbo Instuct
178B
API
w/
S,292
htipp:/ // /www. www.aa12..... www...con/oo..........oooooooo.a con/ /oan///o//////////oo///o//ooo/// intzoouuiagg 52
Seome 2 New Midells
LLaM22 shat
Open
w/
5,500
hutps: Tougcinggase
calatta- Jaana /Llaaa-2----- 2-77--chat-bdd
Dolls v2
7B
Open
w/
2.045
heapo: /hugeiiee faces col /o/dat dad aet Leko/ dol.y-
Vicuna
138
Open
w/
2,048
ht:po:/ /huggii nig faon col /o/////y/////e hancys/riouaaaa 13--21.3
RealPyjaam--nnmmmc
Open
1.021
he Thugg hngeteeeee........... cn hepet he jaddajjaaa IICITTE 70 Instruct
Open
2.043
hecpe: /ruggiingaace co ianen /tulua
Chaglm2-53k
Open
heape// // //uggg nn faice. co/THOOM//het whw.://hatttnt... 02-32k
Open
hempet/ /hugg ing migfaceeco ct/ tettrnll Internln--hoee
E.2 SOLUTION FOR RUN-TIME EXCEPTIONS
Apart from issues such as user permissions and network environment when invoking the model API, the main challenges we encountered during the model evaluation process were limited input length for some models and output inconsistencies with the required format. Therefore, we have devised the following strategies to handle these exceptional cases during evaluation:
Due to the length limitations of certain models, performing 5-shot zero-shot inference becomes challenging for tasks with lengthy instructions. Therefore, we have devised the following strategies to enable the models to produce desired outputs: a) Reduce the number of examples until the input-output length requirements are met; b) If reducing the number of examples to one still fails to meet the requirements of all cases, skip the non-compliant cases and treat them as 0; c) If a model skips a substantial number of examples (over 90%) on a particular task, consider it a failure on that task and record it as "_"
Over-length Issue:
30

---

Published as a conference paper at ICLR 2024
Disregarding Instruction: Another factor that significantly affects the model's performance is its potential inability to comprehend the task instructions, resulting in failure to produce outputs in the specified format. This poses challenges for evaluating the generation -rrrrttoo--beedd open-ended answers. Therefore, we attempt to extract key information from the answers using techniques such as regular expressions and perform fuzzy matching. Unfortunately, there still many scenarios where certain models fail to generate correct and valid answers. For models that fail to provide reasonable answers on over 90% of the test cases for a particular task, we mark them as N/A.
The models may also trigger or bypass their safety mechanisms in certain tasks, such as refusing to provide answers (due mistakenly perceiving the given information as containing unsafe content) or generating sensitive content. For cases where the model refuses to provide an answer, we conduct manual checks to ensure that the data in the test set does not contain explicit, violent, discriminatory, or other inappropriate content. In the secure test set, for situations where the model refuses to answer exhibits similar behavior, we handle them in the same way as mentioned above. For models that cannot provide reasonable answers over 90% of the test cases for particular task, we mark them as N/A.
Sensitive Result:
In general, for all tasks, if a model's performance is marked as "-" or "N/A" on particular task, we do not include that score in the calculation of the standard score. However, when calculating the overall rankings of the models across different levels, we consider these cases as "missing" and assign them score of O. There is still room for improvement this handling approach. Nonetheless, we firmly believe that the model's ability to handle input length, adhere guidelines, and handle sensitive information is an important foundational skill for dealing with real-world knowledge problems.
Table 25: The Task Adaption Parameters in KoLA (Season 1st). "Max Train" and "Max Eval" corresoond to the maximum number of examples and test cases. "Output" corresponds to the output format. We both include necessary and additional parameters required to fully specify evaluation (i.e. the number of evaluation instances and runs, which influence the statistical validity and reliability of the results), though they are not strictly part of the adaptation process, as HELM (Liang et al., 2022) presents.
Level
ID
Dataset
Temperature
Max Tokens
Stop sequence(s)
Max Train
Max Eval
Output
1-1
High-Freq. Low-Freq.
64
None
$
100
List
KM
1-2
64
None
5
100
1-3
ETM
64
None
5
100
List
2-1
COPEN
128
None
5
100
2-2
COPEN CPJ
128
None
5
100
2-3
COPEN-CIC
128
None
5
100
24
FewNERD
128
None
5
100
String
KU
2-5
DocRED
2-6
MAVEN
None
2-7
MAVEN-ERE
200
None
100
2-8
ETU
256
String
3-1
HotporQA
128
5
100
3.2
2WikiMulti.
128
3-3
MuSiQue
128
In
100
String
KA
3-4
KQA Pro
128
In
100
3-5
KORC
50
<stop>
S
100
3-6
ETA
20
<stop>>
5
100
String
4-1
Eneyclopedic
256
None
100
String
KC
4-2
ETC
None
String
E.3
ANSWER NORMALIZATION AND SCORING IMPLEMENTATION
Besides, for different tasks, we adjust some parameters used the inference process based on the model's overall performance, as detailed in Table 25, Once obtain the model inference results, we select various evaluation metrics and answer normalization methods according to the task's characteristics.
31

---

Published as a conference paper at ICLR 2024
Inference Implementation. Following the approach of HELM (Liang al., 2022), we first designed a unified dataset format template to store instructions, provided examples, test inputsoutputs, and decoding parameters. These parameters, akin to HELM, are adapted based on the specific task, as detailed in Table 25. After model inference, we document the following information: a) the text completed by model inference; b) the duration of the inference; c) the timestamp of the inference; d) each token inferred by the model, and f) the probability of each token after log softmxx, which is used for further analysis and score computation.
Answer Normalization. Since all tasks are set up as open-ended generative question-answering formats, to best reflect the model's true performance, select different evaluation metries tailored to the attributes of various task levels for pooo-sroes the answers. Specifically, for the tasks that utilize as the major evaluation metric, we employ relaxed F1 (token match rate), i.e., after tokenizing the model's predicted results and the reference answers using the GPT2Tokenizer (Radford al., 2019), we compute whether each token in the prediction contained the corresponding position of the gold standard. For the tasks are formed as classification tasks, we employ Accuracy of the final result as the major evaluation metric, while the generative tasks utilize the widely-accepted Rouge (Lin, 2004). We also make this portion of the code publicly available in github repository, for introducing more detailed tricks to lift the reproducibility.
Mechanism for Standardized Scores. To make the model scores more comparable, we calculate the standardized scores by pairing each Evolving Task with its corresponding Known Task within the same level, such 2-5 with 2-8, and 3-5 with 3-6. Notably, the three tasks the first level are computed together. After determining the standardized scores for each level, we then compute the average standardized score for each level. The final overall score is derived from the average of these average standardized scores across levels. In this manner, we aim to mitigate potential biases caused by the varying number of tasks across different levels.
32

---

Published as a conference paper at ICLR 2024
F MORE EVALUATION RESULT (SEASON 1)
Due to the length constraints of the paper, including the manual annotation process the experimental section, series of specific results are not fully presented. In this section, we first introduce the annotation process and results for the knowledge creation level. Then, we provide a detailed list of all absolute performance values for each model in the first season and discuss some notable findings.
F.1 ANNOTATION OF CREATING TASKS
Annotation Team for Evaluation. We recruit members from the annotation team who are willing to participate in result annotation and further confirm their compensation and bonuses to protect their earnings and rights. Currently, 14 annotators participate in the annotation of knowledge creation results. The composition of these individuals closely resembles that the overall annotation team, with all members having a bachelor's degree or higher. The an- notation results undergo quality checks conducted by three Ph.D. students from KoLA teams, ensuring both efficiency and quality in the annotation process. We use Streamlit to build an annotation platform where annotators can si- multaneously view the text to be annotated, the context, and the historical annotations.
Table 26: Statisties of the annotation team for creating evaluation in KoLA.
Gender
Rate
Female
71.4%
Male
28.6%
Education
Rate
Bachelor
57.1%
Master
42.9%
Annotation Process and Result. For each model, we provide the following information: a) preceding context, b) continuation generated by the model, c) event knowledge content of the actual subsequent text, and d) actual subsequent text. the continuation without being informed of the knowledge in the subsequent text. Annotators required to evaluate two aspects: i) the overall quality of the text, which includes assessing the novelty, coherence, and fluency of the model's creative content; and ii) the plausibility of the generated content in terms of knowledge hallucination, ensuring it does not conflict with real-world situations. The latter aspect subtask of the former. The annotation is conducted on 5-point scale, where 1 represents the worst and represents the best. We collect 4.2K scoring results and calculate the average score for each model. Table 27 presents the results and variances of the two ratings for each model. It is based on these scores and the model's rank that we calculate various correlation coefficients.
important to note that in item b), the model generates
Table 27: The human evaluation results of Knowledge Creating (KC).
Model
Overall
Knowledge
Model
Overall
Knowledge
Model
Overall
Knowledge
FLAN-TS
2.25
3.00
LLaMa
2.25
2.75
InstructGPT curie
1.75
2.75
UL2
2,00
2.25
Alpaca
1.25
2.00
InstrucGPP davinci v2
2.50
2.75
FLAN-UL2
3.00
2.00
D.Jumbo-Insttrer
2.50
3.00
GLM
2.75
3.00
GPTJ
2.00
Cobere-command
2.75
3.00
GPT-4
2.75
3.75
GPT-NeoX
3.25
3.25
G-r--.5-turb
2.50
3.00
CharGLM
2.50
2.00
3.25
GPT
1.75
ChaiGLM (130B)
3,00
TO++
2.25
2.75
GPT-3 davinci vl
2.50
2.50
GPT-JT
2.50
2.25
innttps://stt /tttpp:::///rremiiii io
33

---

Published as a conference paper at ICLR 2024
Table 28: Season I's standardized performance of Knowledge Memorization and Understanding level.
Level I: KM
Level 2: KU
Model
2-3
GPT-4
50.1
54.0
53.1
1st
63.3
42.5
45.2
60.7
100.0
70.9
72.7
59.4
1st
GPu-3.5.turo
41.2
47.0
InstrucGGPT davinci (175B*)
31.1
37.0
32.6
8th
27.5
42.1
36.2
35.6
52.9
56.1
34.5
31.0
3nd
Cobere-commannd (52.4B)
FLAN-UL2 (20B)
40.8
32.2
51.6
Sth
52.8
40.9
46.9
9.9
18.4
14.6
16.1
18.4
6th
PLAN-T5
43.4
39.5
48.5
3rd
57.0
41.7
43.0
12.6
5th
Tulu (7B)
31.3
37.2
41.1
6th
15.9
19.3
36.8
29.6
24.8
14.6
16.1
19.0
12th
J2-Jumbo-Insten (178B*)
23.9
24.8
18.9
12th
21.2
17.4
25.1
31.0
26.1
26.5
45.1
21.8
7th
CharGLM (130B)
28.4
43.7
36.0
24.4
41.7
45.7
9.9
18.4
17.2
24.2
18.4
InstractGPT curie vi (6,78*)
20.2
33.2
33.2
9th
23.3
35.0
35.7
16.2
18.9
15.9
19.7
18.4
9th
(65B)
18.1
24.0
LlamaZ-char (7B)
14.8
13.2
13.9
19th
19.1
12.3
18.4
31.2
26.6
14.6
21.1
18.9
14th
Chatglm2.33 (6B)
17.9
T0+- (11B)
31.6
28.7
26.0
10th
24.4
33.1
21.7
9.9
16th
Alpaca
14.7
25.2
GLM (1308)
15.0
17.7
11.3
17th
15.9
12.3
12.2
43.4
27.4
23.6
16.1
27.4
10th
Vicuna (13R)
12.4
12.7
11.3
26th
12.7
17.2
23.7
19.5
29.3
24.3
18.4
15th
UL. (20B)
18.9
20.2
12.7
13th
15.9
12.3
12.2
11.9
28th
Dolly-v2 (12B)
14.4
14.3
12.2
22th
15.9
12.3
12.8
22.4
18.4
14.6
21.0
18.4
23th
CharGLM (6B)
23.3
24.2
20.7
11th
9.9
18.4
15.9
16.7
18.4
13th
GPT-J (6B)
14.1
12.4
10.9
24thh
15.9
12.3
12.2
16.1
18.4
24.9
18.4
25th
(175B)
11.9
Internin-chut-kke
16.3
13.8
11.6
20th
15.9
12.3
12.2
13.0
20.8
14.6
20.0
18.5
27th
GPT-JT
13.3
25th
18.4
30.5
22th
GPT-NeoX (20B)
13.4
14.0
11.0
23th
15.9
12.3
12.2
19.6
18.4
25.9
18.4
21th
BLOOM (7B)
14.3
15.1
13.0
18th
15.9
12.3
12.2
24.0
21.7
23,1
16.1
18.4
18th
GPT-3 curie (6.7B)
11.1
11.6
10.5
28th
15.9
12.3
12.2
15.4
18.4
19.6
17.9
18.4
26th
RedPajama- Instruct (7B)
15.0
14.5
12.3
21th
15.9
12.3
12.2
23.7
18.4
14.6
19.3
18.4
24th
Table 29: Season I's standardized performance of Knowledge Applying, Creating level and all the levels.
Level 3: KA
Level 4: KC
Overall (1,2,3,4)
Model
4.2
Avg
GPT-4
53.6
54.5
70.6
26.7
54.1
53.3
1st
44.8
50.0
3rd
2.32
1st
GPT-3.5-turb
52.3
36.4
47.7
39.5
24.5
24.1
4th
48.7
52.0
2nd
1.51
2nd
InstructGPT davinci v2 (1758*)
24.5
33.6
38.1
17.5
43.6
42.4
6th
51.1
50.7
1st
1.20
3rd
Cohere-command (52.4B)
35.5
39.1
51.6
10.9
33.7
FLAN-UL2 (20B)
43.4
41.4
33.3
45.0
37.0
49.4
2nd
22.9
14.5
17th
0.67
5th
FLAN-T5 (IIR)
42.8
26.8
7th
16.2
6th
Tulu (7B)
36.4
39.4
34.5
48.6
35.8
10.0
5th
27.8
25.4
11th
0.48
7th
12-Jumbo-Instrun (178B*)
25.5
25.5
14.8
40.6
4th
ChatGLM (130B)
30.2
27.9
21.9
30.3
30.2
10.0
10th
18.8
16.9
18th
0.19
9th
InstructGPT curie (6.7B*)
31.0
18.2
25.6
22.4
21.5
LLaMa (65B)
10.4
29.3
35.6
19.3
15.5
1.8.3
14th
40.6
30.7
5th
0.02
11hh
Llama2-chat (7B)
10.8
8.3
43.6
26.6
32.3
6th
Chatglm2.324 (6B)
29.6
25.0
18.0
34.0
14.5
13..2
12th
29.6
25.9
8th
40.20
13th
T0++ (I]B)
16.4
16.9
17.7
8.3
12.0
23.5
Alpaca (7B)
9.0
13.2
14.8
12.0
38.9
25.3
17th
31.0
22.6
10th
-0.29
15th
(1308)
17.4
12.3
38.4
15.0
Vicuna (1.18)
19.3
3.9
18.6
12.0
14.8
11.4
23th
30.1
28.7
7th
-0.36
17th
(20B)
22.6
Dolly-v2 (12R)
8.1
14.4
12.3
12.0
20.1
1.8.2
22th
24.2
30.0
9th
-0.41
19th
ChaiGLM
21.4
18.6
GPT-J (6B)
32.7
33.2
20.7
43.1
11.3
12.7
9th
25.3
1.4
23th
-0.48
21th
GPT:3 davinci (175B)
Internlm chat----ct-kk Sk (7B)
13.2
14.7
16.7
8.3
10.9
14.5
24th
20.7
10.8
20th
-0.67
23th
GPT-JT
32.0
13.0
GPT-NeoX (208)
7.9
7.9
11.0
12.0
16.4
13.0
27th
27.2
3.8
21th
-0.70
25th
BLOOM (7B)
GPT-3 curie v1 (6.78)
16.5
9.1
14.4
12.0
12.9
10.1
26th
18.0
6.2
25th
-0.79
27th
RedPajanaa (7B)
17.5
F.2 STANDARD RESULTS
Table 28 and Table 29 present the standard performance of models the data of Season 1, which the initial data annotated before June, 2023. The tested models exhibit different rank with the ranking result in Season 2, which shows the changing feature of evolving dataset.
34

---

Published as a conference paper at ICLR 2024
F.3 DETAILED RESULTS OF EACH TASK
Knowledge Memorization (KM). Table 30 present the absolute performance on the tasks of knowl- edge memorization. This level employs two main scoring methods, namely Exactly Match (EM) and Token-level F1. A key observation is that due to the limited control over generation by many models, the scores obtained using EM are often lower, resulting in numerous cases where score cannot be assigned. During the analysis of the results, we observe that some models, even without access to external resources, can achieve good performance on evolving task. However, this often requires substantial scale and instruction tuning. This may be attributed to the fact that certain new knowledge can be inferred from existing knowledge, which also relies on the model's memorization.
Table 30: Absolute Performance all metrics on task (1-1), (1-2), and (1-3), KM.
1-1
1-2
1-3
Model
EM
F1
EM
F1
FM
F1
FLAN-TS (IIB)
13.6
20.1
12.5
17.7
21.8
23.2
UL2 (20B)
N/A
5.1
N/A
5.9
N/A
1.4
FLAN-UL2 (20B)
14.0
18.5
9.5
13.2
23.7
25.1
GPT-JT (6B)
N/A
N/A
1.3
N/A
0.4
GPT-J (6B)
N/A
2.3
N/A
1.2
N/A
0.3
GPT-NeoX (20B)
N/A
1.8
N/A
2.2
N/A
0.3
BLOOM (7B)
N/A
2.3
N/A
2.8
N/A
1.5
TO++ (1IB)
12.9
11.1
LLaMa (65B)
0,7
4.0
N/A
4.7
N/A
0.8
Alpaca (7B)
N/A
2.6
1.0
4.5
N/A
1.5
12-Jumbo-Instnn (178B*)
4.6
8.2
5.0
8.8
1.0
5.2
Cohere-command (52.4B)
17.0
21.5
12.5
19.3
24.7
27.3
GPT-3.5-turbo
18.7
18.0
22.1
13.8
18.9
GPT-3 curie v1 (6.7B)
N/A
0.4
N/A
0.7
N/A
N/A
GPT-3 davinci vl (175B)
N/A
0.9
N/A
0.8
N/A
N/A
InstructGPT curie vl (6.7B*)
1.3
5.9
10.5
13.9
7.9
13.9
InstructGPT davinci (175B%)
12.6
13.0
13.5
GLM (130B)
N/A
2.8
N/A
4.4
N/A
0.5
GPT-4
17.1
24.2
20.8
26.5
21.0
26.0
ChatGLM (130B)
7.4
10.9
16.5
20.3
13.8
15.6
35

---

Published as a conference paper at ICLR 2024
Knowledge Understanding (KU). The results in this level unexpected, as significant amount of knowledge understanding relies on longer texts or generating highly structured content. There- fore, in complex tasks such as document-level relation extraction and event relation extraction, the performance of many models is not satisfactory. This aspect deserves further exploration.
Table 31: Absolute Performance of accuracy on COPEN (2-1), (2-2), (2-3), KU.
Model
2-1
2-3
FLAN-TS (1IB)
39.0
75.0
55.0
UL2 (20B)
N/A
N/A
N/A
FLAN-UL2 (20B)
35.0
73.0
62.0
GPT-JT (6B)
N/A
N/A
N/A
GPT-J (6B)
N/A
N/A
N/A
GPT-NeoX (20B)
N/A
N/A
N/A
BLOOM (7B)
N/A
N/A
N/A
TO++ (1IB)
8.0
53.0
17.0
LLaMa (65B)
N/A
N/A
N/A
Alpaca (7B)
N/A
N/A
1.0
12-Jumbo-Instruct (178B*)
13.0
23.0
Cohere-command (52.4B)
17.0
73.0
50.0
GPT-3.5-turbo
21.0
79.0
57.0
GPT-3 curie v1 (6.7B)
N/A
N/A
N/A
GPT-3 davinci (175B)
N/A
N/A
InstructGPT curie vl (6.7B*)
7.0
58.0
42.0
InstructGPT davinci v2 (175B*)
11.0
76.0
43.0
GLM (130B)
N/A
N/A
N/A
GPT-4
45.0
77.0
59.0
ChaIGLM (130B)
75.0
60.0
Table 32: Absolute Performance on RE tasks, i.e., FewNERD (2-4), DocRED (2-5), ETU (2-8), KU.
2-4
2-5
2-8
Model
P
R
F1
P
R
F1
P
R
F1
FLAN-T5 (IIB)
7.8
0.4
0.7
UL2 (20B)
16.7
0.3
0.5
FLAN-UL2 (20B)
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
GPT-JT (6B)
20.0
0.4
0.8
N/A
N/A
N/A
N/A
N/A
N/A
GPT-J (6B)
1.9
1.7
1.6
N/A
N/A
N/A
N/A
N/A
N/A
GPT-NeoX (20B)
2.6
2.5
2.5
N/A
N/A
N/A
N/A
N/A
N/A
BLOOM (7B)
3.3
4.6
3.7
L.
1.8
1.3
N/A
N/A
N/A
T0++ (11B)
N/A
N/A
N/A
LLaMa (65B)
11.1
10.4
2.2
3.8
2.8
N/A
N/A
N/A
Alpaca (7B)
2.1
3.2
2.4
0.6
0.7
0.6
N/A
N/A
N/A
-2-tuuo--nstrrutt (178B*)
5.7
5.5
5.5
3.0
3.0
3.0
1.5
1.2
1.3
Cohere-command (52.4B)
5.2
1.9
2.8
5.2
6.6
5.8
N/A
N/A
N/A
GPT-3.5-turbo
9.5
11.2
GPT-3 curie v1 (6.7B)
1.3
1.7
1.4
N/A
N/A
N/A
N/A
N/A
N/A
GPT-3 davinci vl (175B)
3.6
3.9
3.7
N/A
N/A
N/A
N/A
N/A
N/A
InstructGPT curie v1 (6.7B*)
2.4
1.3
1.6
0.3
0.2
0.2
N/A
N/A
N/A
InstructGPT davinci v2 (175B*)
6.0
7.9
6.8
13.2
13.9
13.6
5,3
4.6
5.0
GLM (130B)
7.9
8.8
2.8
4.6
3.5
2.8
4.6
3.5
GPT44
12.4
14.5
13.4
35.5
29.2
32.0
19.8
13.6
16.1
ChatGLM (130B)
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
36

---

Published as a conference paper at ICLR 2024
Table 33: Absolute Performance of all metrics two sub-tasks of MAVEN (2-6), KU.
2-6
Model
Identification
Classification
P
R
F1
P
R
F1
FLAN-TS (1IB)
UL2 (20B)
FLAN-UL2 (20B)
N/A
N/A
N/A
N/A
N/A
N/A
GPT-JT (6B)
21.8
12.1
15.6
15.1
8.4
10.8
GPT-J (6B)
25.0
8.4
12.5
13.9
4.7
GPT-NeoX (20B)
11.8
12.6
12.2
7.5
7.9
7.7
BLOOM (7B)
30.6
8.8
13.7
12.9
3.7
5.8
T0++ (11B)
LLaMa (65B)
52.9
16.7
25.4
13.2
4.2
6.4
Alpaca (7B)
48.6
7.9
13.6
25.7
4.2
7.2
2-JuubooInstrruc (178B*)
28.0
10.7
15.5
14.6
5.6
8.1
Cohere-command (52.4B)
N/A
N/A
N/A
N/A
N/A
N/A
GPT-3.5-turbo
54.7
34.9
42.6
25.5
16.3
19.9
GPT-3 curie v1 (6.7B)
52.2
5.6
10.1
17.4
1.9
3.4
GPT-3 davinci v1 (175B)
44.2
10.7
17.2
15.4
3.7
6.0
InstructGPT curie v1 (6.7B*)
100.0
0.5
0.9
100.0
0.5
InstructGPT davinci v2 (175B*)
52.4
45.1
48.5
30.3
26.0
28.0
GLM (130B)
52.5
19.5
28.5
11.2
4.2
6.1
GPT-4
66.3
58.6
62.2
40.5
35.8
38.0
ChatGLM (130B)
50.0
2.3
4.4
20.0
0.9
1.8
Table 34: Absolute Performance of all metrics four sub-tasks of MAVEN-ERE ERE (2-7), KU.
2-7
Model
Temporal
Causal
Subevent
Coreference
R
P
FLAN-TS (11B)
UL2 (20B)
FLAN-UL2 (20B)
NA
NA
N/A
N/
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
GPT-JT (6B)
GPT-J (6B)
GPT-NeoX (20B)
BLOOM (7B)
NA
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
T0++ (IIB)
LLaMa (65B)
N/A
N/A
N/A
33.3
5.0
8.7
N/A
N/A
N/A
N/A
N/A
N/A
Alpaca (7B)
NA
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
12-Jumbo-Instrue (178B*)
10.7
3.4
5.2
25.0
30.0
27.3
22.2
20.0
21.1
25.0
20.0
22.2
Cohere-command (52.4B)
7.5
7.2
15.0
15.0
15.0
N/A
N/A
44.4
40.0
GPT-3.5-turo
18.6
14,9
15.7
24.4
60.0
GPT-3 curie v1 (6.7B)
4.7
4.6
4.7
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
GPT-3 davinci vl (175B)
0.3
1.1
0.5
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
InstructGPT curie vl (6.78*)
6.9
14,9
9.4
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
InstructGPT davinci v2 (175B(*)
22.0
14,9
17.8
11.8
30.0
16.9
20.0
10.0
13.3
N/A
N/A
(130B)
N/A
N/A
N/A
N/A
N/A
N/A
GPT-4
26.4
22.1
23,8
25.0
24.4
21.4
30.0
25.0
80.0
76.2
MhatGLM (130B)
18.7
3.4
5.8
N/A
N/A
N/A
N/A
N/A
N/A
33.3
10.0
15.4
37

---

Published as a conference paper at ICLR 2024
Knowledge Applying (KA). In the evaluation of the KA level, notable phenomenon is that knowledge graph (KG)- based reasoning question answering tasks are almost impossible to complete without the use of KG. This phenomenon is evident in the three tasks (3-4)-(3-6). Furthermore, due to the clear quality progression exhibited by multiple tasks in this layer, the performance of models generally follows decreasing trend.
Table 35; Absolute Performance of Fl --core on task (3-1), (3-2) and (3-3), K.A.
Model
3-1
3-2
3-3
FLAN-T5 (11B)
23.7
35.3
7.5
UL2 (20B)
9.2
16.3
4.2
FLAN-UL2 (20B)
27.1
34.0
10.6
GPT-JT (6B)
13.8
25.5
2.8
GPT-J (6B)
19.2
26.6
4.6
GPT-NeoX (20B)
1.0
3.6
N/A
BLOOM (7B)
4.4
11.4
N/A
T0++ (1IB)
7.2
11.8
3.2
LLaMa (65B)
2.8
23.0
11.7
Alpaca (7B)
1.8
8,4
1.8
Jumbo-Instrucc (178B*)
23.9
19.6
6.9
Cohere-command (52.4B)
17.3
28.7
13.4
GPT-3.5-turbo
33.6
29.5
GPT-3 curie vl (6.7B)
7.3
4.7
1.6
GPT-3 davinci v1 (175B)
4.2
3.2
2.7
InstructGPT curie v1 (6.7B*)
13.9
24.6
3.4
InstructGPT davinci v2 (175B*)
13.2
26.9
12.9
GLM (130B)
8.0
2.7
0.6
GPT-4
34.6
45.9
28.4
ChatGLM (130B)
17.4
21.8
5.2
Table 36: Absolute Performance of accuracy of different types of questions KQA Pro (3-4), KA.
3-4
Model
All
Multi,
Quali.
Comp.
Logi.
Count.
Veri.
FLAN-T5 (11B)
20.0
21.3
19.4
29.4
15.4
N/A
64.3
UL2 (20B)
13.0
13.3
22.6
17.6
15.4
N/A
14.3
FLAN-UL2 (20B)
20.0
21.3
19.4
41.2
19.2
N/A
42.9
GPT-JT (6B)
10.0
12.0
19.4
N/A
15.4
N/A
28.6
GPT-J (6B)
19.0
17.3
22.6
35.3
19.2
N/A
42.9
GPT-NeoX (20B)
2.0
N/A
N/A
N/A
14.3
BLOOM (7B)
3.0
4.0
6.5
N/A
3.8
N/A
7.1
T0+ (11B)
N/A
N/A
N/A
N/A
N/A
N/A
N/A
LLaMa (65B)
6.0
8.0
12.9
S.9
11.5
N/A
7.1
Alpaca (7B)
2.0
N/A
5,9
N/A
N/A
N/A
12-Jumbo-Instracc (178B*)
13.0
12.0
12.9
17.6
11.1
28.6
Cohere-command (52.4B)
19.0
21.3
22.6
23.5
23.1
11.1
50.0
GPT-3.5-turbo
17.0
16.0
19.4
41.2
19.2
N/A
21.4
GPT-3 curie v1 (6.7B)
2.0
1.3
3.2
5.9
3.8
N/A
N/A
GPT-3 davinci (175B)
3.0
3.2
N/A
N/A
14.3
InstructGPT curie vl (6.7B*)
8.0
8.0
3.2
29.4
7.7
N/A
14.3
InstructGPT davinci v2 (175B")
5.0
4.0
6.5
11.8
7.7
N/A
N/A
GLM (130B)
4.0
5.3
6.5
N/A
7.7
N/A
7.1
GPT-4
10.0
12.0
5,9
N/A
42,9
ChatGLM (6B)
3.0
6.5
5.9
N/A
N/A
ChatGLM (130B)
12.0
10.7
9.7
35.3
7.7
N/A
21.4
38

---

Published as a conference paper at ICLR 2024
Table 37: Absolute Performance of all metrics KoRC (3-5) and ETA (3-6), KA.
3-5
3-6
Model
EM
EM
FLAN-TS (IIB)
20.0
23.3
UL2 (20B)
N/A
N/A
FLAN-UL2 (20B)
21.0
27.1
32.6
39.6
GPT-JT (6B)
N/A
2.3
N/A
3.0
GPT-J (6B)
N/A
1.4
N/A
2.7
GPT-NeoX (20B)
3.1
BLOOM (7B)
7.0
11.2
N/A
7.3
T0++ (11B)
18.0
23.4
LLaMa (65B)
2.0
5.6
N/A
8.4
Alpaca (7B)
23.0
15.4
12-Jumbo-Instruct (178B*)
12.1
2.0
4.9
Cohere-command (52.4B)
28.0
38.1
36.7
41.8
GPT-3.5-turbo
10.0
14.6
10.2
14.2
GPT-3 curie v1 (6.7B)
2.0
3.0
N/A
0.2
GPT-3 davinci (175B)
InstrucIGPT curie v1 (6.7B*)
9.0
14.9
8.2
15.7
InstructGPT davinci v2 (175B*)
25.0
33.8
22.4
32.6
GLM (130B)
22.0
28.5
22.0
28.5
GPT-4
33.0
ChatGLM (6B)
3.3
8.7
ChatGLM (130B)
17.0
20.3
N/A
N/A
39

---

Published as a conference paper at ICLR 2024
Knowledge Creating (KC). Here, present the scores of three sub-criteria used to calculate the overall score for each model. If only these scores are considered, it is observed that some well-regarded models such as GPT4 and GPT-3.5-turbo do not necessarily demonstrate superiority.
Table 38: Absolute Performance of the key metrics Encyclopedia Creating task (4-1), KC.
Model
0 (Tr, R)
a (T, R))
a (T, Tc)
FLAN-T5 (IIB)
15.6
9.7
22.5
UL2 (20B)
23.3
18.3
31.6
FLAN-UL2 (20B)
17.3
8.7
14.5
GPT-JT (6B)
17.8
36.0
GPT-J (6B)
27.5
19.4
32.9
GPT-NeoX (20B)
28.2
19.2
31.6
BLOOM (7B)
25,4
19.5
44.6
T0++ (11B)
16.3
10.0
25.7
LLaMa (65B)
30.9
24.0
25.5
Alpaca (7B)
26.5
20.0
26.9
J2- Jumbo-lIntruun (178B*)
27.5
17.8
17.0
Cohere-command (52.4B)
27.6
19.3
47.5
GPT-3.5-turbo
45.6
21.3
29.2
GPT-3 curic vl (6.7B)
26.5
22.1
42.0
GPT-3 davinci v1 (175B)
28.0
21.7
33.7
InstructGPT curic vl (6.7B*)
20.2
18.8
28.0
InstructGPT davinci v2 (175B*)
43.7
22.4
26.0
GLM (130B)
20.3
37.7
GPT-4
37.9
14.8
19.0
CharGLM (6B)
20.0
15.4
35.3
ChatGLM (130B)
17.5
16.0
26.1
Table 39: Absolute Performance of the key metrics on ETC (4-2), KC
Model
Ô (Tr, R)
a (T, R)
a (T, Tc)
FLAN-TS (1IB)
1.0
7.3
3.6
UL2 (20B)
1.2
15.6
5.6
FLAN-UL2 (20B)
9.1
6.1
12.1
GPT-JT (6B)
17.4
14.9
43.9
GPT-J (6B)
17.7
16.0
44.1
GPT-NeoXX (20B)
18.0
16.5
42.3
BLOOM (7B)
19.5
16.4
44.3
T0++ (I1B)
0.9
15.8
4.5
LLaMa (65B)
26.0
20.0
26.6
Alpaca (7B)
14.9
20.7
24.5
J2- Juumo--Instruee (178B*)
27.0
17.8
15.4
Cohere-command (52.4B)
26.1
15.0
18.6
GPT-3.5-turbo
45.7
19.1
23,9
GPT-3 curie vl (6.7B)
18.9
17.4
41.7
GPT-3 davinci v1 (175B)
24.0
20.3
39.0
InstruciGPT curie v1 (6.7B*)
20.8
21.1
31.8
InstructGPT davinci v2 (175B*)
43.1
21.3
24.7
GLM (130B)
25.8
17.1
39.4
GPT44
43.8
17.5
22.3
CharGLM (6B)
17.7
14.9
31.1
ChatGLM (130B)
10.0
8.5
13.0
40

---

Published as a conference paper at ICLR 2024
G MORE EVALUATION RESULT (SEASON 2)
Due to the length constraints of the paper, the results of season 2 are not fully presented. In this section, provide detailed list of all absolute performance values for each model in the second season and discuss some notable findings.
G.1 1 DETAILED RESULTS OF ROLLING TASKS FOR ALL MODELS
Table 40: Absolute performance of all metries on rolling tasks: ETM (2-8), ETU (2-8), ETA (3-6)
and ETC (4-2), Season 2.
1-3
2-8
3-6
4-2
Model
EM
P
EM
F1
(T4, R)
(T, R)
a ( T
FLAN-TS (11B)
6.9
N/A
N/A
N/A
N/A
N/A
UL2 (20B)
NIA
N/A
NA
N/A
N/A
0.0
13.7
23.2
FLAN-UL2 (20B)
5.7
7.2
N/A
N/A
N/A
35,0
36.8
1.6
4.7
4.8
GPT-JT (6B)
NIA
1.3
N/A
N/A
N/A
NIA
1.4
9.4
15.6
23.0
GPT-J (6B)
N/A
0.8
NIA
NA
N/A
N/A
0.4
10.6
17.3
20.9
GPT-NeuX (20B)
N/A
0.9
N/A
N/A
N/A
N/A
0.6
17.1
17.8
33.3
BLOOM (7B)
N/A
N/A
24.6
26.3
T0++ (11B)
2.3
N/A
NA
N/A
N/A
10.6
26.4
LLaMa (65B)
NIA
1.6
N/A
N/A
N/A
N/A
1.5
27.2
30.0
36.0
Alpaca (7B)
NIA
0.9
N/A
N/A
N/A
30.0
34.2
3.4
26,9
4.9
12-Jumbo-Instruct (178B*)
N/A
1.4
N/A
N/A
N/A
S.0
9.3
30.8
29.0
21.0
Cohere-commande (52.4B)
8.0
N/A
N/A
N/A
11.7
27.6
17.7
GPr--,5-turbo
11.1
35,9
25.4
GPT-3 curie v1 (6.7B)
N/A
N/A
NA
N/A
NIA
19.8
19.4
47.3
GPT-3 davinci vl (175B)
N/A
0.2
N/A
N/A
N/A
N/A
0.9
22.7
22.0
39.4
InstructGPT curie vl (6.7B*)
4,6
6.3
N/A
N/
N/A
3,3
10.9
18.5
21.4
26.9
InstructGPT davinci v2 (175B*)
8.6
10.0
2.4
3.1
2.7
N/A
7.2
45.9
31.3
34,6
GLM (130B)
21.1
16.6
GPT-4
12.7
47.7
23.1
ChatGLM (130B)
8.0
10.0
N/A
N/A
N/A
NIA
5.0
28.3
15.3
31.9
ChatGLM (6B)
5.7
7.1
NIA
NAA
N/A
S.0
12.2
21.3
16.9
23.8
Dolly-v2 (12B)
N/A
1.3
N/A
N/A
N/A
N/A
9.0
21.1
20.7
26.4
RedPajama-Insttuut (7B)
N/A
N/A
N/A
N/A
N/A
N/A
7.0
15.7
15.3
36.7
Tulu (7B)
3.4
38.3
2.6
28.2
3.7
Vicuna (13B)
N/A
N/A
N/A
31.2
Llama2-chat (7B)
N/A
0.8
0.3
0.3
0.3
28.3
35.2
32.9
26.5
37.1
Chatglm2-32k (6B)
1.1
1.9
0.1
0.3
0,2
N/A
1.1
32.8
30.3
37.2
Internim-chat-8k (7B)
N/A
0.4
N/A
N/A
N/A
N/A
6.1
24.0
21.3
34.3
Table 40 presents the absolute performance for all models on the rolling tasks with season 2's data. We find that most models still fail to get right results 2-8 as the task require the model's ability understanding the long context and complex structures.
G.2 DETAILED RESULTS OF ALL TASKS FOR NEW MODELS
Knowledge Memorization (KM). Table 41 presents the absolute performance on the tasks of knowledge memorization with data from season to test the 7 new models in season 2.
Knowledge Understanding (KU), The results this level are similar with results in season 1, where most new models in season 2 are not able to get right results due to the long context and highly structured content.
Knowledge Applying (KA). Season 2's new models don't perform well on KQA Pro(3-4) tasks as shown in Table 47.
41

---

Published as a conference paper at ICLR 2024
Table 41: Absolute Performance of all metrics for new models of Season 2 on task (1-1), (1-2), and (1-3), KM.
1-1
1-2
1-3
Model
EM
F1
EM
F1
EM
F1
Dolly-v2 (12B)
0.5
2,4
N/A
2.4
N/A
1.1
RedPaaama--nstruue Inntiiama---ntuuttttcco (7B)
N/A
2.8
N/A
2.4
N/A
1.1
Tulu (7B)
9.3
12.7
9.5
16.3
17.9
18.7
Vicuna (13B)
1.2
N/A
1.4
N/A
0.5
Llama2-chat (7B)
1.0
2.7
N/A
1.7
N/A
2.1
Chatglm2-32K (6B)
2.1
4.5
1.0
3.2
N/A
0.1
Internlm-chat-8. (7B)
0.1
3.6
N/A
2.0
N/A
0.7
Table 42: Absolute Performance of accuracy for models of Scason 2 on COPEN (2-1), (2-2), (2-3), KU.
Model
2-1
2-2
2-3
Dolly-v2 (12B)
N/A
N/A
1.0
RedPajama-Instruel (7B)
Tulu (7B)
N/A
18.0
44.0
Vicuna (13B)
N/A
1.0
9.0
Llama2-chat (7B)
3.0
N/A
11.0
Chatglm2- 32k (6B)
N/A
3.0
Intemlm-chat-8k (7B)
Table 43: Absolute Performance for new models of Season on RE tasks, i.c., FewNERD (2-4),
DocRED (2-5), ETU (2-8), KU.
2-4
2-5
2-8
Model
R
R
Dolly-v2 (12B)
3.6
N/A
N/A
N/A
N/A
N/A
RedPajama-Instren (7B)
3.3
4.0
3.6
N/A
N/A
N/A
N/A
N/A
N/A
Tulu (7B)
7.4
4.0
5.2
2.2
2.9
2.5
3.3
0.1
0.2
Vicuna (13B)
3.2
4.3
3.6
0.5
0.4
0.4
N/A
N/A
N/A
Llama2-chat (7B)
4.4
7.8
5.6
3.5
3.2
0.2
0.2
0.2
Chatglm2-32k (6B)
0.7
0.9
0.3
0.2
Internlm-chat-8k (7B)
0.6
1.2
0.8
1.5
0.7
0.9
N/A
0.1
N/A
Table 44: Absolute Performance of all metrics for new models of Season on two sub-tasks of MAVEN (2-6), KU.
2-6
Model
Identification
Classification
R
F1
P
F1
Dolly-v2 (12B)
N/A
N/A
N/A
N/A
N/A
N/A
RedPajama-Instrenn (7B)
N/A
N/A
N/A
N/A
N/A
N/A
Tulu (7B)
N/A
N/A
N/A
N/A
N/A
N/A
Vieuna (13B)
40.2
20.0
26.7
15.0
7.4
9.9
Llama2-chat (7B)
N/A
N/A
N/A
N/A
N/A
N/A
Chatglm2-32k (6B)
N/A
N/A
N/A
N/A
N/A
N/A
Internlm-chat-80 (7B)
N/A
N/A
N/A
N/A
N/A
N/A
42

---

Published as a conference paper at ICLR 2024
Table 43: Absolute Performance of all metrics for new models Season 2 on four sub-tasks of MAVEN-ERE (2-7), KU.
2-7
Model
Temporal
Causal
Subevent
Coreference
Fl
Fl
F1
P
F1
Dolly-v2 (12B)
5.4
14.9
8.0
1.3
5.0
2.0
1.7
10.0
2.9
N/A
N/A
N/A
RodPajama-Instruet (7B)
3.2
6.9
4.4
N/A
N/A
N/A
L.1
10.0
1.9
1.2
10.0
2.2
Tulu (7B)
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
Vicuma (13B)
21.7
5.7
9.1
16.7
10.0
12.5
N/A
N/A
N/A
N/A
N/A
N/A
Llama2-chat (7B)
10.7
17.2
13.2
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
Chatglm2-32k (6B)
2.4
1.1
1.6
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
Intemlm-chat-8k (7B)
3.7
4.6
4.1
3.4
25.0
6.1
N/A
N/A
N/A
N/A
N/A
N/A
Table 46: Absolute Performance of Fl-score for models of Season 2 on task (3-1), (3-2) and (3-3), KA.
Model
3-1
3-2
3-3
Dolly-v2 (12B)
1.1
9.5
0.6
RedPajama-Insiin Instructt (7B)
N/A
N/A
N/A
Tulu (7B)
21.9
32.2
11.2
Vicuna (13B)
9.4
N/A
3.6
Llama2-chat (7B)
3,6
6.2
3.0
Chatglm2-32K (6B)
16.9
19.1
3.3
Internlm-chat-e (7B)
4.9
9.8
2.7
Table 47: Absolute Performance of accuracy of different types of questions for new models of Season 2 on KQA Pro (3-4), KA.
3-4
Model
All
Multi.
Quali.
Comp.
Legi.
Count.
Veri.
Dolly-v2 (12B)
2.0
2.7
3.2
N/A
N/A
N/A
7.1
RedPajamm--nstruet (7B)
N/A
N/A
N/A
N/A
N/A
N/A
N/A
Tulu (7B)
22.0
22.7
25.8
52.9
11.5
11.1
35.7
Vicuna (13B)
2.0
2.7
N/A
5.9
N/A
N/A
N/A
Llama2-chat (7B)
N/A
N/A
N/A
N/A
N/A
N/A
N/A
Chatglm2-324 32k (6B)
14.0
14.7
12.9
35.3
7.7
N/A
28.6
Intemlm-chat--8 (7B)
N/A
N/A
N/A
N/A
N/A
N/A
N/A
Table 48: Absolute Performance of all metrics for new models of Season 2 on KoRC (3-5) and ETA (3-6), KA.
3-5
3-6
Model
EM
F1
EM
F1
Dolly-v2 (12B)
5.0
10.2
N/A
8.2
RedPajama-Instruu (7B)
7.0
9.9
N/A
7.6
Tulu (7B)
19.0
25.9
N/A
N/A
Vicuna (13B)
2.0
4,9
1.4
Llama2-chan (7B)
18.0
24.1
26.5
33.8
Chatglm2-32k (6B)
3.0
4,6
N/A
3.3
Intemlm-chat-8k (7B)
N/A
0.9
N/A
4.6
43

---

Published as a conference paper at ICLR 2024
Knowledge Creating (KC). Here, present the scores of three sub-criteria used to caleulate the overall score for each new model in season
Table 49: Absolute Performance of the key metrics for new models of Season 2 on Encyclopedia Creating task (4-1), KC
Model
d (Tk,
(T, R)
0 (T, Tk)
Dolly-v2 (12B)
26.8
19.8
33.8
RedPajama-Instnn (7B)
18.7
Tulu (7B)
3.7
4.1
Vicuna (13B)
2.9
18.7
2.8
Llama2-chat (7B)
20.2
21.4
26.3
Chatglm2-32K (6B)
26.2
18.7
26.6
Internlmchhat-8 (7B)
11.4
15.4
17.6
Table 50: Absolute Performance of the key metries on ETC (4-2), KC.
Model
a (Tk, R)
0 (T, R)
a (T, Th)
Dolly-v2 (12B)
24.9
21.7
28.0
Red|Pajama-Instruct (7B)
16.2
17.3
45.0
Tulu (7B)
0.1
14.1
0.1
Vicuna (13B)
24.9
20.5
28.0
Llama2- -hat (7B)
0.7
21.4
1.0
Chatglm2-32k (6B)
23.6
17.8
26.9
Intemlm-chat-88 (7B)
6.3
7.8
14.9
44


Reviews:
[{"review_id": "MLJop0Utwh", "reviewer": "Reviewer_1m8m", "summary": "The paper presents KoLA, an interesting Knowledge-oriented LLM Assessment benchmark. It assesses LLMs on four-level knowledge-related abilities: knowledge memorization, knowledge understanding, knowledge applying and knowledge creating, with known and evolving data sources.\n\nThe authors plan to make available their data, leaderboard, participation information, and supporting tools upon acceptance. They plan to host  a new competition season every three months, updating their evolving data sources, inviting participations from both open and commercial LLMs. The paper reports analysis of two searsons run comparing 28 open-source and commercial LLMs. \n\nI found the framework to be very interesting and insightful. The community will benefit from such a large scale analysis over knowledge-related abilities with known and evolving sources.  The paper is very well written.", "questions": "I would have liked to see the human evaluation results in the main part of the paper. It would strengthen the analysis.\n\nDo authors discuss the costs involved with these studies for both seasons?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The tools and data from the paper will be released upon acceptance. \n\nThe community will benefit from such a large scale analysis over knowledge-related abilities with known and evolving sources. The presented analysis is already very insightful.\n\nThe breakdown of the task using knowledge-related abilities with known and evolving sources is compelling to assess LLMs evolving capabilities.", "weaknesses": "I don’t see any major weaknesses in the paper. \n\nOne could argue that it is just an analysis paper, some of the insights that were drawn here might not be novel. But I feel that the presented framework will be valuable to the community. The authors have done a very good job explaining the framework in detail.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2mTTJZa647", "reviewer": "Reviewer_VBLJ", "summary": "The authors proposes a world knowledge assessment benchmark KoLA that consists of factual knowledge from Wikidata to evaluate the world knowledge of large language models (LLMs). KoLA consists of a four-level taxonomy: knowledge memorization/knowledge understanding/knowledge applying/knowledge creating. The first two tasks focus on directly extracting/generating the information associated with the corresponding world knowledge, and the last two focus on the application of knowledge in reasoning and text generation. \n\nThe data comes from both the data already exposed to LLMs (known data) and the data appeared afterwards (evolving data). Results show that pre-alignment/instruction-tuning models has higher correlation between model size and knowledge memory performance. However, instruction-tuning empowers the models with high-level abilities (e.g. knowledge applying tasks) and commercial models still have leading performance in general.", "questions": "Comments\n- The caption of the figures should include necessary details to understand the them if the space allows (e.g. Figure 4).", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The knowledge benchmark fills the blank of thoroughly evaluating world knowledge of current large language models. The taxonomy is carefully designed and rich experiments on model choices are conducted.\n- The ever evolving setup has long-term benefit in considering the generalization problem in knowledge-intensive tasks.\n- The self-contrast metric has a good motivation in balancing the hallucination in knowledge-based generation.\n- The annotation team is of strong educational background.", "weaknesses": "- The knowledge-wise strength of Rouge-L used in Eq. 3 doesn't look to be strong enough in capturing the knowledge association especially when T is a free-generation result, maybe replacing the measurement with another model (e.g. a entailment model) would be better? (w/ additional computational cost)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UgYxmbT5fP", "reviewer": "Reviewer_ryRT", "summary": "The paper presents a world knowledge benchmark for LLM, focusing on three aspects: (1) ability modeling; (2) evolving benchmark built upon emerging corpora and (3) evaluation criteria. The author also presented metrics of SOTA LLM's performance on the benchmark and provided insights observed from the evaluation results.", "questions": "Q1. The paper identified four knowledge capabilities. It is clear on knowledge memorization and knowledge creation. However, it is vague to distinguish knowledge understanding and knowledge application. Take knowledge understanding as an example, would reading comprehension a task of knowledge understanding or knowledge attention? Why knowledge understanding only has extraction tasks?\nQ2. Benchmark on emerging corpora is a great idea and it is quite encouraging to see the authors promised to refresh the benchmark regularly. However, it is not clear how to maintain such benchmark in the long term. As the time goes by, some evolving benchmark is no longer most up-to-date, how to handle the benchmark data? How would this evolving benchmark interact with the known dataset?  \nQ3. How much does the new KoLA benchmark differ from existing LLM benchmark? There are so many available benchmark published, each focusing on one/multiple capabilities of the model. Why do we need KoLA? What if we combined existing ones? \nQ4. It is not clear the contribution/motivation with standardized overall scoring. As the score would depend on the evaluated models, it will change a lot as more evaluated models would be added in. Also why this is better than simple ranking?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "S1. The paper presents a new LLM benchmark with some innovations, including constructing benchmark with emerging corpora, evaluating model's knowledge creation capability.\nS2. The paper evaluated major SOTA LLMs, providing good comparison in model's capability from different perspective.\nS3. The paper reads well, easy to follow.", "weaknesses": "W1. Benchmark on emerging corpora is a great idea and it is quite encouraging to see the authors promised to refresh the benchmark regularly. However, it is not clear how to maintain such benchmark in the long term.  \nW2. It is not clear why we need another new LLM benchmark. Given all different benchmarks available publicly, I am not convinced KoLA is a must-have addition. \nW3. It is not clear why the standardized overall scoring can give better idea than simple ranking.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FsML81KwZb", "reviewer": "Reviewer_vPns", "summary": "The paper studies how to thoroughly evaluating LLMs on its knowledge capability.  Inspired by cognitive science, the authors establish an extensive benchmark that focuses on memorization, understanding, applying and creating respectively. For each capability, the authors have chosen/created new tasks for that capability and evaluate a significant amount of LLMs to draw insights conclusions on those experiments.\n\nThe author also introduces a new metric for knowledge creating, which in the experiments shows a notable correlation to faithfulness.", "questions": "The paper said that \"Comparing the second-season results with the first season, the rankings of most open-source models have declined\" but Table 2 and Table 3 seem to show results of four levels of the same season.\n\nWhy COPEN and 2wikiMultiQA are also considered as exclusive?\n\nHow the knowledge is designed in Figure 2 to apply self contrast metric please?", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper tackles a timely and important issues which evaluates the LLM capability instead of just evaluating on some tasks. To answer this question seems still hard and the paper has selected various datasets and made sensible grouping to evaluate the four aspect that it mentions. \n\nThe paper has run the experiments for several seasons for now and has shown some interesting trends that correlate with model size. The paper also proposes a novel metric for knowledge creation that seems interesting and notably correlated with faithfulness.", "weaknesses": "There are several questions that I think arise after reading the paper, I would consider these just missing some clarity:\nThe paper said that \"Comparing the second-season results with the first season, the rankings of most open-source models have declined\" but Table 2 and Table 3 seem to show results of four levels of the same season.\nWhy COPEN and 2wikiMultiQA are also considered as exclusive?\n\nOne of the potential strength of the paper is to analyse the results on the fresh data of each season that the paper claims; however, we don't find such results in the current version. Meanwhile, the paper draws some conclusions (the ones related to knowledge) that don't seem to be part of the contributions of this particular paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)