Exploring the Cognitive Knowledge Structure of Large Language Models: An Educational Diagnostic Assessment Approach
Authors: Zheyuan Zhang
Jifan Yu
Juanzi Li
Lei Hou
Venue: EMNLP 2023 Findings (2023)
Keywords: large language models
diagnostic assessment
knowledge structure

Abstract: Large Language Models (LLMs) have not only exhibited exceptional performance across various tasks, but also demonstrated sparks of intelligence. Recent studies have focused on assessing their capabilities on human exams and revealed their impressive competence in different domains. However, cognitive research on the overall knowledge structure of LLMs is still lacking. In this paper, based on educational diagnostic assessment method, we conduct an evaluation using MoocRadar, a meticulously annotated human test dataset based on Bloom Taxonomy. We aim to reveal the knowledge structures of LLMs and gain insights of their cognitive capabilities. This research emphasizes the significance of investigating LLMs' knowledge and understanding the disparate cognitive patterns of LLMs. By shedding light on models' knowledge, researchers can advance development and utilization of LLMs in a more informed and effective manner.

Exploring the Cognitive Knowledge Structure of Large Language Models: An Educational Diagnostic Assessment Approach
Zheyuan Zhang,, Jifan Yu,, Juanzi Li', Lei Hou Department of Computer Science and Technology Tsinghua University, Beijing, 100084, China (zheyuan-22, Jujf21]mmaill tsinghua. cn {lijuanzi, houlei Noulei)@tsinghus edu. cn
Abstract
benchmarks that comprise common standardized exams, such as AGIEval (Zhong et al., 2023), C- Eval (Huang et al., 2023). M3Exam (Zhang et al., 2023), and CMExam (Liu et al., 2023). However, although standardized exams contain diverse infor- mation, these works condense them into single overall score, lacking structured understanding of LLMs' knowledge and cognitive patterns.
Large Language Models (LLMs) have not only exhibited exceptional performance across var- ious tasks, but also demonstrated sparks of in- telligence. Recent studies have focused on as- sessing their capabilities on human exams and revealed their impressive competence in dif- ferent domains. However, cogmitive research on the overall knowledge structure of LLMs is still lacking. In this paper. based on educational diagnostic assessment method, conduct an evaluation using MoocRadar, meticulously annotated human test dataset based on Bloom Taxonomy. We aim reveal the knowledge structures of LLMs and gain insights of their cognitive capabilities. This research empha- sizes the significance of investigating LLMs' knowledge and understanding the disparate cognitive patterns of L.LMs. By shedding light on models' knowledge, researchers can ad- vance development and utilization of LLMs in more informed and effective manner.
For example. while LLMs demonstrate excep- tional performance on tasks challenging for hu- mans, they might still struggle with basic knowl- edge, as illustrated in Figure 1, which may lead to over-estimation of the validity of model generated contents. Therefore, there is pressing need for fur- ther research of models' knowledge and cognitive distribution in comparison humans.
Proposed Research. To investigate this prob- lem, we draw inspiration from psychometric meth- ods that use cognitive psychology theories to eval- uate LLMs. This topic has gained traction LLMs continue to demonstrate exceptional perfor- mances (Chollet, 2019; Singh et al., 2023; Bubeck et al., 2023). In this work, we adopt the Educa- tional Diagnostic Assessment approach and lever- age MooeRadar (Yu et al., 2023), a novel student exercise dataset annotated with Bloom's Taxon- omy (Anderson and Krathwohl, 2001), to assess the cognitive capability of LLMs. Specifically, we delve into three primary research questions: 1) Performance Analysis: the proficiency and robust- ness of LLMs across various question domains; 2) Deficit Assessment: the knowledge structure and the extent to which LLMs are similar with humans; and 3) Error Assessment: the error pattern of LLMs in answers and explanations. Our findings con- tribute to a deeper understanding of the knowledge structure of LLMS and insights for evaluation. Contributions. Our main contributions are:
1 Introduction
Large language models (LLMs), such GPT se- ries (Brown al., 2020). Flan (Wei et al., 2022), and PaLM (Chowdhery et al., 2022), have gained significant attention worldwide due to their remark- able ability. Given their unprecedented human-like performances, researchers have started to explore alternative evaluation metrics beyond traditional benchmarks like MMLU (Hendrycks et al., 2021) and Big-Bench Bench (Ghazal et al., 2017).
Existing Works on LLMs Evaluation with Ex- ams. Researchers have long sought models capable of passing human exams (Nilsson, 2005). Recently, a new approach simulates professional exams de- signed for humans to evaluate LLMs. For example. OpenAl (2023) reports the performance of GPT series on variety of exams, including AP exams, SAT, Leeteode, and so on. There are also emerging
(i) We introduce the topic of the cognitive knowl- edge structure of LLMs.
Equal contribution
Corresponding Author
ii We propose a method of Educational Diag-

---

Dataset
Instructions
Results
Diagnostic
Problem
Response
Performance
Knowledge Type: Cognitive Dimension: Question Type: TF [Question] Methane has center symmelry. [Options] True, False
Please determine truc taise and provide your reason.
(Chal3PP) True.
Discipiness
Contexta
Single single chokse quacilion? why?
Deficit
Problem
What's
Kroodlslge Cognttive
Knowledge Type: Cognitive Dimension: Question Type: [Question] The incorreci combinatio of quantum numbers for ouler atomic electrons among the following is:
Response
Multiple Choice
[ChalGPT)
coroect
[Options] A: n=3, |+2, m=0, ms=+1/2. n=2, 1-1, mm11 m1m1/2 C: n=4, [=1, m=2, ms=-112. D: n=3, 1-1, m= ms=+1/2
What's the answer of this multiple choice question? why?
Answers Explanations
Figure I: ChatGPT correctly answers question that is challenging with higher knowledge type and cognitive dimensions right (problem 2) but encounters difficulties in an casier one (problem 1). We design specific instructions evaluate LLMs and assess their performances in three aspects.
nostic Assessment to evaluate LLMs on their cognitive knowledge structure.
lyze, Evaluate, and Create. Table demonstrates a detailed description of Bloom's Taxonomy.
(iii) We assess LLMs' performance, deficits, and errors, gaining insights into their capabilities.
Cognitive Dimensions
Descriptions
Rememhering
Remember, Know, Identify,
2 Method
Understanding
Translate, Explain, Induce,
2.1 Educational Diagnostic Assessment
Applying
Prove, Estimate, Exccute,
Analyzing
Compare, Select, Organize,
In education scenarios, Diagnostic Assessment is a widely employed method to gauge students' knowledge structure (Falmagne et al., 2006), dis- covering their proficiency on certain subject mat- ters and learning styles (Vuong et al., 2021), typi- cally through sets of questions (Leighton and Gierl, 2007). Two main approaches of Diagnostic Assess- ment include deficit assessment, which focuses on identifying and addressing knowledge gaps in various domains and the degree knowledge mas- tery, and error assessment, which focuses on error patterns and strategies for correction (Bejar, 1984). Drawing inspiration from Diagnostic Assessment methods, in this work, based on Bloom's Taxon- omy, use deficit assessment to test the accuracy of models on wide range of exercises, and error assessment on their answers and explanations.
Evaluating
Evaluate, Judge, Criticise,
Creating
Design, Create, Program,
Knowledge Types
Deseriptions
Factual
Terminology and Details
Conceptual
Relationships and Theories
Procedural
Processes and Methods
Meta
Strategy and Self-knowledge
Table 1:
A detailed descriptions and examples of Bloom's 's Taxonomy in MoocRadar.
We carefully select 8453 questions appropriate for model evaluation, which fall into three types: single choice (SC), multiple choice (MC), and true or false (TF). Additionally, we exclude the dimen- sion of Create because of the scarcity of related exercises. We further classify these questions into four disciplines by their course information, includ- ing STEM, social science, humanity, and others. We test the performance of models on them and analyze the distribution of these features. More de- tails of MoocRadar are illustrated in the appendix. Model Selection. We carefully choose 3 ad- vanced models that have consistently demonstrated leading performance and are widely recognized in the field, including: Text-Davinci-003, Chat- GPT, and GPT-4, which represent a series of most acknowledged models. All experiments are per- formed using the APIs provided by OpenAL. Specif-
2.2 Experimental Setup
Dataset. In this section, we introduce the dataset utilized for assessment, MoocRadar, offering an extensive overview of its general information. MoocRadar is fine-gran mraiindd and multi-aspect ex- ercise repository designed for cognitive modeling and educational diagnostic assessment. According Bloom's Taxonomy, questions in MoocRadar are categorized into four Knowledge-Types: Factual- Procedural-
knowledge, Conceptual-knowlled knowledge, and Meta- knowledno and six cognitive dimensions: Remember, Understand, Apply, Ana-

---

ically, use the completion API for Text-Davinci- 003 and the chat completion API for ChatGPT and GPT-4. ensure consistency in the quality of the responses, we set the temperature to 0 to get greedy search responses generated by each model.
the models. Comparatively, STEM exercises are more challenging as illustrated in human results, while LLMs demonstrate impressive capability STEM knowledge. GPT-4 even outperforms hu- mans with context. However, is surprising that LLMs don't perform as effectively in social sci- ence and humanities exercises, even though these disciplines primarily involve natural language.
Experimental Design. As shown in Figure I, we design different prompts tailored to each type of exereises query LLMs for both answers and ex- planation. All tasks are conducted in zero-shot sce- nario. To simulate human-like behavior that solv- ing exercises with relevant knowledge, we leverage the BM25 algorithm to retrieve the two most related discussions from the subtitles in the corresponding courses in MOOCCubeX (Yu et al., 2021) and test their effect. Moreover, extract real student be- haviors on MoocRadar dataset from MOOCCubeX and calculate their average scores to serve refer- ence of humans. Based on both results from human and LLMs, this work provides a road map with investigation to the following research questions: (RQ1) Performance Analysis: What's the fea- tures of LLMs' basic performance different dis- ciplines and their robustness to these questions?
Models
Total
STEM
S.S.
Human.
Others
GPT-3.5
0.436
0.418
0.461
0.483
0.421
ccotexxt
0.508
0.468
0.554
0.614
0.504
ChalGPT
0.506
0.480
0.547
0.533
0,525
context
0.526
0.441
0.642
0.639
0.601
GPT-4
0,657
0,613
0.732
0.684
0.690
-context
0.687
0.629
0.765
0.774
0.733
Human
0.746
0.625
0.924
0.854
0.791
Table 2: Models and human performance. S.S. and Human. are short social science and humanity, and is the result of models with context. Bold figures denote best performance.
(RQ2) Deficit Assessment: According to Bloom Taxonomy, compared with humans, what knowledge distribution does LLMs demonstrate? Are they similar to humans in knowledge structure? (RQ3) Error Assessment: Based on answers and explanations, what's their pattern of errors?
Robustness, In single choice questions, ma- nipulate the order of the options by either placing the correct answer at the beginning or the end. This allows us to examine if such modifications affect the model's s accuracy. As shown in table 3, we find that 1) ChatGPT more robust to changing of options, while the other two exhibits cognitive bias as Primacy Effect (Deese and Kaufman, 1957) that early appearance aids performance; 2) if the correct answer appears later in GPT-3.5 and GPT-4, they mistakenly change their answers and explana- tions; 3) later appearance causes less consistency in answers and explanations in less robust models.
3 Experiment
this section, we conduct experiments and ana- lyze the results from three perspectives in the fol- lowing subsections. We assign score of to each question type. Following standardized exams, for multiple questions, models receive score of 0.5 if they fail to select all correct options. We then calculate the average score across questions.
Models
Answer
Explanation
Real
GPT-3.5
0.405
0.436
0.390
3.1 Performance Analysis
-first
0.554
0.476
-last
0,362
0.390
0.296
Firstly, we assess their performance both with and without contexts, compare their performance in different disciplines, and examine their robustness.
ChaiGPT
0.489
0.501
0.487
-fisst
0.440
0.438
0.438
-last
0.412
0.430
0.412
Disciplines and Context. We exhibit scores of model answers with or without context on the four disciplines (STEM, social science, humanity, and others). As shown in Table 2, the later versions of GPT significantly outperform previous models, with GPT-4 being the most advanced, but not bet- ter than humans' average. Additional knowledge from context indeed enhances the performance of
GPT-4
0.635
0.643
0.632
ffrst
0.727
0.728
0.727
-last
0.520
0.529
0.473
Table Single choice accuracy of three models. First and last indicate the place of the correct answers. Real is when answers and explanations are both correct. Bold figures denote the best performance.

---

3.2 Deficit Assessment
Models
GPT-3.5
ChatGPT
GPT-4
We utilize Bloom's Taxonomy in MoocRadar to demonstrate models' distribution in cognitive di- mensions and knowledge types and design a score to measure similarities of models to humans.
Likeness
0.262
0.396
0.474
Table 4: son Correlation Coefficient of knowledge structures.
Models' similarity human, measured by Pear-
Bloom Taxonomy Distribution. As shown in Figure 2, demonstrate the distribution based Bloom's taxonomy, where deeper colors represent better performance. The 0 and grids are due to the limited number of exercises, typically only one. Generally, both in knowledge types and cognitive dimensions, questions in the intermediate range are more challenging for models and humans. We design similarity score for deeper understanding.
3.3 Error Assessment
In this section, we analyze the error of each models, by delving into their explanation of their answers. Explanation Accuracy. Table 5 demonstrate the accuracy of answers in each type. We mainly find that: 1) Models perform best on TF and worst on MC. MC could be more difficult than SC and TF, because of more thinking steps (determine TF of each options, and select multiple ones). 2) Ex- planations and answers are more consistent in TF than in SC and MC for the same reason, as there are more chances to make errors. 3) Accuracy of expla- nations falls behind answers in MC, where models can select some of the correct options for the wrong reason. 4) Context does not necessary aid and even hurt explanation performances, but indeed aids an- swer accuracy. More advanced models are more consistent in their answers and explanations.
GPT3.5
ChatGPT
meba
proc.
0.36
0.5
0.41
0.43
0.32
0:45
0.31
0.34
0.36
0.32
0.46
0.47
0.53
0.39
FoLL
0:63
0.63
1.
0:33
0.57
0.63
GPT4
Student
mcD.
0:65
0.71
0.85
0.91
0.55
pro..
0.66
0.53
0.50
6.46
0:83
0.74
0.45
0.66
0.90
Krueledie
Models
SC
MC
0.61
6:63
0.71
0:42
0.4K
0.73
0.97
GPT-3.5
0.531 (0.531)
0.437 (0.405)
0.383 (0.483)
-context
0.617 (0.617)
0.376 (0.480)
0.280 (0.527)
fxct.
0.76
0.36
ChatGPT
0,617 (0.592)
0.501 (0.488)
0,298 (0.496)
unde
anai
ev3l.
evst.
<cotext
0.715 (0.706)
0.492 (0.482)
0.333 (0.541)
Omrendo1ss
Cogn Ormendions
GPT4
0.751 (0.746)
0.643 (0.6 (3.635)
0.555 (0.663)
Figure 2: The distributions of accuracy Bloom's 's Tax- onomy of different models and average of students.
-conexxt
0.775 (0.771)
0.605 (0.667)
0,578 (0.691)
Table Accuracy of explanations. TF, SC, and MC short for the three question types. The numbers in parentheses represent answer accuracy.
Similarity Score. According to the accuracy of models in various dimensions of knowledge and cognition, we develop metric to measure their similarity to humans, which primarily considers knowledge structure, beyond mere performance, and estimates the extent which their cognitive structure proportional that of humans. Specifi- cally, given a model M, the 4s5 vector of the model distribution in bloom's 's taxonomy x and human dis- tribution y, convert & and into 1 #20 vectors and ÿ. the similarity between M and human can be defined as: Likeness (M) p(ū, (), where p(ž, ū) represents the Pearson Correlation Coeffi- cient of 5 and y. We calculate the Likeness of the three models in Table 4. The likeness also exhibits a rising tendency as the models evolve. Models that follow human instructions better are also more similar to humans in knowledge structure.
3.4 Discussion
In this section, we discuss our findings the pro- posed three research questions:
Performance Analysis. We exhibit different models' performance. Comparing with humans, they are less proficient in disciplines primarily involve natural language, but better at STEM. Though with sufficient knowledge, they might have hallucination on specific long-tail concepts in hu- manity and social science. LLMs are not robust in option orders, and exhibit a cognitive bias Primacy Effect rather than Recency Effect.
Deficit Assessment. Models are less proficiency in the intermediate range Bloom's Taxonomy. The reason could be that application- based ques-

---

tions, such solving mathematical problems and making deductions using chemical theorems, are prone to errors and are inherently challenging for models. For analyzing and evaluating questions, the strong linguistic capabilities of models allow them to excel in these tasks, and perform even better than intermediate-level questions. More ad- vanced models demonstrate more similarity with humans in knowledge structure, which might be an additional effect of human alignment.
not conduct experiments at the parameter level. 2) though we have discovered some phenomena, fur- ther experiments and deeper analysis are not con- ducted. We include some of them in the case study section in the appendix.
Future Works. Future works include 1) more models for experiments, 2) further exploration robustness and similarity with humans, and 3) the next step of diagnostic assessment, investigate how to optimize the knowledge structure of LLMs.
Error Assessment. By comparing different kinds questions, we find that gap exists for mod- els between knowledge and answers. They perform worse in multiple choices, as there are more think- ing steps and error chances. Accuracy of explana- tions can be worse than answers: models were asked to generate answers first, their explanation could shift due to wrong answers and question or- ders, and cause their hallucinations. Due to the limitations of autoregressive architecture (Bubeck et al., 2023), their errors could snowball.
Ethics Statement
We foresee ethic concerns in this work. The MoocRadar dataset employed in our research is publicly available, and does not contain any per- sonal information.
Acknowledgements
This work is supported by grant from the Institute for Guo Qiang, Tsinghua University (2019GQB0003), and also supported by Tsinghua University Initiative Scientific Research Program.
4 Conclusion
this work, we introduce a new research ques- tion on LLMs analyzing, which calls for a deeper understanding of the knowledge structure of these models. We use Educational Diagnostic Assess- ment tool to test the performance of LLMs on various dimensions, and develop a metric to measure the similarity of their knowledge structure with humans. We provide findings and discussion for insight into research on the cognition of LLMs.
References
Lorin W Anderson and David R Krathwohl. 2001. A taxonomy for learning, leaching, and assessing: A re- vision of Bloom's 's taxonomy of educational objectives. Longman,
Isaac L Bejar. 1984. Educational diagnostic assessment. Journal of Educational Measurement, 21(2):175- 189.
Limitations
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu. Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin. Scott Gray, Benjamin Chess. Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
this section, we describe the limitations of this work in terms of the dataset and experiments. Dataset. We investigated the knowledge distri- bution of LLMs based on the MoocRadar dataset. MoocRadar is fine-grained, well-structured dataset that is distinct from commonly used bench- marks in terms of knowledge annotation. However, as a dataset for educational diagnostic assessment, it's still limited the following aspects: 1) Dif- ferent categories of exercises (e.g. question type. disciplines) have an unbalanced distribution; 2) As demonstrated in the Robustness section, the per- formance of the models vary due to different forms of exercises.
Sébastien Bubeck, Varun Chandrasekaran, Ronen EI- dan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lund- berg, et al. 2023. Sparks of artificial general intelli- gence: Early experiments with gpt-4. AcXiv preprint, 1bs/2303.12711
Experiment. Due to time and cost constrains, 1) we only included three LLMs by OpenAI, which models. Therefore, we did
are all

---

François Chollet. 2019. On the measure of intelligence. ArXiv preprint, abs/1911.0115..
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M. Dai, and Quoe V. Le. 2022. Finetuned language models are zero-shot learners. In Proc. of ICLR. OpenReview.net.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. ArXiv preprint, abs/2204.02311.
Jifan Yu, Mengying Lu, Qingyang Zhong, Zijun Yao, Shangqing Tu, Zhengshan Liao, Xiaoya Li, Manli Li, Lei Hou, Hai-Tac Zheng, fine- aineggrineed and multi-aspect knowledge repository for improving cognitive student modeling in moocs. ArXiv preprint, abs/2304.02200
2023. Moocradar: A
James Deese and Roger A Kaufman. 1957. Serial ef- fects in recall of unorganized and sequentially or- ganized verbal material. Journal of experimental psychology. 180.
Jifan Yu, Yuquan Wang, Qingyang Zhong, Gan Luo, Yiming Mao, Kai Sun, Wenzheng Feng, Wei Xu, Shulin Cao, Kaisheng Zeng. Zijun Yao, Lei Hou, Yankai Lin, Peng Li, Jie Zhou, Bin Xu, Juanzi Li, Jie Tang. and Maosong Sun. 2021. Mooccubex: A large knowledge-centered repository for adaptive learning in mooes. In Proceedings of the 30th ACM Interna- tional Conference on Information & Knowledge Man- agement, CIKM '21, page 4643-4652, New York, NY. USA. Association for Computing Machinery.
Jean-Claude Falmagne, Eric Cosyn, Jean-Paul Doignon, and Nicolas Thiéry, 2006. The assessment of knowl- edge, in theory and in practice. In Formal Concept Analyxis: 4th International Conference, ICFCA 2006, Dresden, Germany, February 13-17, 2006. Proceed- ings, pages 61-79. Springer.
Ahmad Ghazal, Todor Ivanov, Pekka Kostamaa, Alain Crolotte, Ryan Voong. Mohammed Al-Kateb, Waleed Ghazal, and Roberto V Zicari. 2017. Big- bench v2: the and improved bigbench. In Proc. of ICDE, pages
Wenxuan Zhang, Sharifah Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. 2023. M3exam: A multilingual, multimodal. multilevel benchmark for examining large language models.
IEEE.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language under- standing. In Proc. of ICLR. OpenReview.net.
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang. Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2023. Agieval: A human-centric benchmark for evaluating foundation models.
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. 2023. C-eval: multi-level multi multi-discipline chinese evaluation suite for foundation models.
Jacqueline Leighton and Mark Gierl. 2007. Cognitive diagnostic assessment for education: Theory and applications. Cambridge University Press.
Junling Liu, Peilin Zhou, Yining Hua, Dading Chong, Zhongyu Tian, Andrew Liu, Helin Wang, Chenyu You, Zhenhua Guo, Lei Zhu, and Michael Lingzhi Li. 2023. Benchmarking large language models emexam a comprehensive chinese medical exam dataset.
Nils J Nilsson. 2005. Human-level artificial intelli- gence? be serious! AI magazine, 26(4):68-68.
OpenAI. 2023. Gpt-4 technical report.
Manmeet Singh, Vaisakh SB, Neetiraj Malviya, et al. 2023. Mind meets machine: Unravelling gpt-4's cog- nitive psychology. ArXiv preprint, abs/2303.1 1436.
Quan-Hoang Vuong, Viet-Phuong La. Manh-Toam -oan Ho, Thanh-Haug Pham, Thu-Trang Vuong, Ha-My Vuong, and Minh-Hoang Nguyen. 2021. data col- lection secondary school students' stem perfor- mance and reading practices an emerging country. Data Intelligence, 3(2):336-356.

---

A Appendix
Querying Details. For context settings, we use the BM25 algorithm retrieve the two most related contexts from the subtitles of the corresponding class. As illustrated in Figure 3, for the question about the pioneer of mathematical logic, the BM25 algorithm retrieves context about the emergence and development of logic and the concept of math- ematical logic. The two contexts will be placed before instruction, along with the questions and options form the prompt, and fed into LLMs. non-context settings, the context position will sim- ply be empty. We also test different instructions to make sure that models will follow them to provide answers and explanations.
A.1 Details of Experiment
This subsection shows details of the dataset we use, and experiment for diagnostic assessment.
Problem Statistics. Table 6 demonstrates the details of the dataset we use, which is selected from the original MoocRadar. Generally, we in- clude three question types (single choice, multpple choice, and true or false), four knowledge types (factual knowledge, conceptual knowledge, proce- dural knowledge, and meta knowledge), and five cognitive dimensions (remember, understand, ap- ply, analyze, and evaluate), to form total dataset of 8430 questions.
Context
Prompt
Knowledge Types
Cognitive Dimensions
Context 1
[Context]
Ex.
N
N
N
Today we will introduce the emergence and development of logic
[Instruction]
[Question]
SC
5968
Factual
2020
Remember
1715
Who is the pioneer of
mathematical logic?
MC
1086
Conceptual
4032
Understund
4066
[Options]
TF
1376
Procedural
2268
Apply
1667
A: Leibniz,
The basic part of mathe- matical logic two calculations
B: Boole,
total
8430
Meta
110
Analyze
640
C: DeMorgan,
Evaluate
342
D: Frege
Table Categories data. Ex., SC, MC, and TF are short for Exercise, Single Choice, Multiple Choice and True or False questions.
Figure 3: Construction of prompts in experiments.
Annotation details. To accurately assess the models' performance both answers and explana- tions, we first invited educational experts filter questions, because a part of questions in the dataset have accompanying pictures, which can't be input into models. Then we asked human annotators to evaluate the answers and explanations gener- ated by the three models respectively, to prevent misjudgment in automatic evaluation tools. These annotators are familiar with MOOC exercises, and provided with the original questions and correct swers, and have access the internet determine accurately.
Types
Questions
Options
Which of the following works was created by Vincent Gogh?
Les Nymphéas Sunflowers
SC
C: Grunde Odalisque
Which the following rare earth elements?
Uranium
Lutecium
MC
C.C
C: Dysprosium
D: Samarium
Light only exhibits the properties of waves.
True
TF
False
False
Table 7: Question types examples: single choice (SC), multiple choice (MC), true false (TF). SC have only correct option, while MC have more than 2 correct options. TF should be determined as True or False.
A.2 Cases
In this subsection, we demonstrate some of the error cases we have seen during our experiments, and hope to provide insights into the model's 's error patterns.
Problem Examples. Table 7 demonstrates ex- amples for each type of questions. There two more than two options in single choices and only one correct options, while multiple choices have more than one correct options. True or false questions should be answered as True or False.

---

all birds fly. then ostriches fly.
Ostriches are birds. Therefore, ostriches can fly. Does this reasoning contain any formal errors?
Question
Correct
No.
Answer
Yes. While the premises state that every bird capable of flying. does mean that every individual bird, including the ostrich. capable of flying. Ostriches are actually flightless birds.
Model
Answer
(ChatGPT)
If all hirds can fly, then sparrows can fly. Sparrows are birds.
Question
The temm "logic" originally derives from: A: English. B: French, C: Latin, D: Greek.
Therefore, sparrows can fly.
Question
Does this reasoning contain any formal erros?
Correct Answer
Correct
Greek
No.
Answer
The term "logic" originally derives from the language: C.Latin. The term "logic" originated from the Latin word "logica" which itself was derived from the Greek word "logikë." Therefore....
Model
The reasnning an example valid deductive argument from purely logical standpoint, the argument itself is valid.
Model
Answer
Answer (ChatGPT)
(ChatGPT)
Table 8: Case I:
Changing the entities infects the answer. the question is consistent with its knowledge, ChatGPT indeed follows the task of focusing on the formal errors or purely logic. Otherwise, it fails to follow the question, as it refuses to override own knowledge. This phenomenon indicates ChatGPT might be less robust when there are knowledge conflicts.
The term "logic" originally derives from: A: English, B: French, C: Greek, D: Latin.
Question
Correct Answer
C: Greek
The term "logic" originally derives from: C: Greek, The Greek philosopher Aristotle is oflen credited as one of the founders formal logic and his work on reasoning and
Model
Answer
(ChatGPT)
Which language does the term 'logic' originally derives from?
Question
"When underguing uniform acceleration, direction of velicity always lies along the same line as the direction of acceleration." Is it correct? A. No. B. Yes
Correct Answer
Greek
Question
The term "logic" originally derives from the Greek word "logikě," " which derived from "logos." meuning "word," " "reason," "thought." The study of logic can be traced back to
Model
Correct Answer
Answer
A. No
(ChatGPT)
The answer is A: No. The reason is that uniform acceleration only requires acceleration to constant, the direction of velocity and acceleration can be different. For example, in uniform circular motion....
Model Answer (GPT-4)
Easier question forms aid per- formances. We have discussed the effect of different question types. ChatGPT have the correct knowledge about the origin of this word, but it cannot give correct answers, the first question demonstrates. The fol- lowing two questions types improve the performance of ChatGPT, by two different ways: moving the correct option forward which is consistent to Case ease the burden of models to answer by simplifying the question form. This case corroborates the findings that models better TF than SC or MC, because there are fewer thinking steps.
Table 10: Case 3:
"When undergoing uniformm acceleration, the direction of velocity always lies along the same line direction of acceleration." correct? A. Yes. B. No
Question
Correct
B. No
Answer
The answer is This is because uniform acceleration refers to In this case, the change in velucity aligned with the direction accelcration.
Model
Answer
(GPT-4)
Table 9: Case 2: the Primacy Effect of GPT-4. For the exactly same questions, switching options causes a direct change in GPT-4's answer, and both answers are accompanied with solid but conflict explanations, which proves the instability of its knowledge. Statisti- cally, models have a tendency to choose more forward contents, which influence their rationale more, just like the Primacy Effect in cognitive psychology. This phe- nomenon is also observed in GPT-3.5.


Reviews:
[{"review_id": "CR5JB4vEyj", "reviewer": "Reviewer_XMhD", "summary": "", "questions": "A. In Section 2.2 (Lines 140-145), the authors specified that \"all tasks are conducted in zero-shot scenario.\" However, in the same section, the authors also said, \"leverage the BM25 algorithm to retrieve the two most related discussions from the subtitles in the corresponding courses\". The authors in Appendix A (Lines 455-468) iterate that they tested in-context and non-context settings. These sounds contradict the first explanation of using \"zero-shot scenario.\" Can you clarify this?\n\nB. In Figure 2, do numbers represent students' performance/accuracy in corresponding questions?\n\nC. Can you add an extended version of Table 4 that includes ±context, ordering effect, and topic subjects? I think it will be very interesting.\n\nD. Can you add your insights in the Discussion section on why the LLMs perform worse in intermediate-level Bloom's Taxonomy questions and how to improve the models' performance?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the comprehensive and valuable suggestions and questions you've raised, as well as your recognition of the excitement of our work. We kindly respond to your concerns as follows.\n\n> Details about experiments: More details about the annotations for the dataset and models' answer/explanation accuracies would be important to ensure the credibility of the results. For example, how many annotators rated how many questions? What were the inter-rater scores? etc.\n\nThank you for pointing out this important issue. In particular, we initially engaged a group of experts to conduct two rounds of screening and annotations on each question in the dataset. The aim was to eliminate questions containing images to answer (which can't be models' input), unanswerable questions, and questions without answers (e.g., vote questions), reducing the total number of questions from 9328 to 8430. Subsequently, we enlisted a team familiar with the MOOC context. Each team member was responsible for either 600 or 1000 questions, contingent on the question type. We presented the standard answers and contexts for each question to the team members and encouraged them to verify the model's outputs for correctness using online resources. After one round of annotation, team members exchanged and performed mutual checks and corrections on each other's work.\n\nWe have exhibited part of the annotation details in the Appendix. Once the paper is accepted, we also intend to supplement additional content on the additional page to improve the paper further.\n\n> How the model generated explanations? Can you share the prompts?\n\nThank you for your reminder. We exhibit our instructions in Figure 1. For example, for True or False questions, we prompt the model with \"Please determine true or false of the following question and provide your reason.\" We further examine the effectiveness of our instruction, to ensure that models indeed provide their answers and explanations based on the prompts. We will also consider providing more detailed examples in the public code repository and appendix.\n\n> Question A: In Section 2.2 (Lines 140-145), the authors specified that \"all tasks are conducted in zero-shot scenario.\" However, in the same section, the authors also said, \"leverage the BM25 algorithm to retrieve the two most related discussions from the subtitles in the corresponding courses\". The authors in Appendix A (Lines 455-468) iterate that they tested in-context and non-context settings. These sounds contradict the first explanation of using \"zero-shot scenario.\" Can you clarify this?\n\nThis is a great question, and we will provide further clarification here. Regarding the term \"zero-shot\", we are referring to the fact that we did not incorporate an in-context learning method [1] when generating model outputs. Instead, we directly provided the model with the question without pre-exposing it to examples of single-choice or multiple-choice answers. On the other hand, within our contextual setup, we indeed introduced additional question-related knowledge to the model. As depicted in Figure 3 of the appendix, we augmented the question stem by including subtitles from the course, to simulate students' behaviors. \n\nTherefore, while both our context and in-context learning (ICL) employ the term \"context\", their implications are distinct, respectively pertaining to **supplementary knowledge** and **exemplar questions**. \n\n> Question B: In Figure 2, do numbers represent students' performance/accuracy in corresponding questions?\n\nYes, they do. The numbers represent the accuracy in corresponding questions from 0 to 1. \n\n> Question C: Can you add an extended version of Table 4 that includes ±context, ordering effect, and topic subjects? I think it will be very interesting.\n\nOf course, this is indeed a good suggestion. Below is the extended table we have supplemented.\n\n**Without context:**\n\n| Models    | Disciplines       | SC           | SC(first)   | SC(last)    | MC           | TF           |\n|-----------|-------------------|--------------|-------------|-------------|--------------|--------------|\n| GPT-3.5   | STEM              | 0.393 (0.396)| 0.542 (0.570)| 0.403 (0.357)| 0.348 (0.454)| 0.546 (0.548)|\n|           | Social Science    | 0.440 (0.423)| 0.571 (0.584)| 0.407 (0.410)| 0.385 (0.496)| 0.517 (0.515)|\n|           | Humanity          | 0.551 (0.453)| 0.553 (0.563)| 0.357 (0.357)| 0.465 (0.549)| 0.505 (0.505)|\n|           | Others            | 0.617 (0.385)| 0.596 (0.607)| 0.296 (0.298)| 0.357 (0.429)| 0.560 (0.560)|\n| ChatGPT   | STEM              | 0.467 (0.468)| 0.395 (0.397)| 0.418 (0.395)| 0.272 (0.479)| 0.611 (0.566)|\n|           | Social Science    | 0.552 (0.537)| 0.520 (0.520)| 0.448 (0.445)| 0.317 (0.506)| 0.614 (0.604)|\n|           | Humanity          | 0.602 (0.503)| 0.506 (0.506)| 0.463 (0.447)| 0.303 (0.527)| 0.634 (0.624)|\n|           | Others            | 0.530 (0.519)| 0.500 (0.500)| 0.437 (0.424)| 0.312 (0.464)| 0.627 (0.604)|\n| GPT-4     | STEM              | 0.605 (0.595)| 0.691 (0.691)| 0.494 (0.473)| 0.546 (0.650)| 0.718 (0.716)|\n|           | Social Science    | 0.730 (0.724)| 0.788 (0.788)| 0.585 (0.627)| 0.607 (0.706)| 0.782 (0.770)|\n|           | Humanity          | 0.670 (0.668)| 0.743 (0.743)| 0.558 (0.565)| 0.524 (0.657)| 0.757 (0.752)|\n|           | Others            | 0.698 (0.698)| 0.839 (0.839)| 0.622 (0.587)| 0.446 (0.554)| 0.769 (0.769)|\n\n**With context**\n\n| Models   | Disciplines      | SC           | MC           | TF           |\n|----------|------------------|--------------|--------------|--------------|\n| GPT-3.5  | STEM             | 0.341 (0.448)| 0.298 (0.484)| 0.595 (0.595)|\n|          | Social Science   | 0.363 (0.536)| 0.246 (0.544)| 0.604 (0.604)|\n|          | Humanity         | 0.503 (0.595)| 0.357 (0.619)| 0.663 (0.663)|\n|          | Others           | 0.509 (0.469)| 0.223 (0.460)| 0.687 (0.687)|\n| ChatGPT  | STEM             | 0.593 (0.581)| 0.298 (0.499)| 0.675 (0.656)|\n|          | Social Science   | 0.623 (0.624)| 0.376 (0.582)| 0.737 (0.735)|\n|          | Humanity         | 0.647 (0.615)| 0.351 (0.589)| 0.748 (0.752)|\n|          | Others           | 0.596 (0.598)| 0.268 (0.451)| 0.754 (0.739)|\n| GPT-4    | STEM             | 0.594 (0.608)| 0.570 (0.679)| 0.738 (0.734)|\n|          | Social Science   | 0.671 (0.759)| 0.637 (0.733)| 0.807 (0.803)|\n|          | Humanity         | 0.531 (0.797)| 0.541 (0.686)| 0.797 (0.792)|\n|          | Others           | 0.626 (0.754)| 0.455 (0.580)| 0.784 (0.776)|\n\nSince we did not conduct experiments on option ordering under the context condition, the second table lacks the additional columns for the other two single-choice settings. Thank you for your insightful suggestions. From this informative extended table, we can derive the following conclusions, which are consistent with the conclusions in the paper:\n\n* Models perform better with the earlier appearance of the correct answers (Primacy Effect).\n\n* Models achieved better scores in social science and humanity. However, without human results as references, we can not ascertain whether such outcomes stem from the difficulty of STEM questions, as shown in Table 1 in the paper.\n\n* Models perform best on SC and worst on TF, because of more thinking steps.\n\n> Question D: Can you add your insights in the Discussion section on why the LLMs perform worse in intermediate-level Bloom's Taxonomy questions and how to improve the models' performance?\n\nThank you for your valuable suggestions. Here, we have a few thoughts that you might find insightful. \n\n* Firstly, lower dimensions such as \"remembering\", \"understanding\" and \"factual knowledge\" tend to be more manageable for models. This is due to the auto-regressive training objectives of these models, which ensure strong memory capabilities. \n\n* Secondly, in the intermediate dimensions like \"applying\" and \"procedural knowledge\", application-based questions are included. These involve tasks like solving specific mathematical problems, making deductions using chemical theorems, engaging in logical reasoning, and more. Such questions are prone to errors and are inherently challenging for models.\n\n* Lastly, as for higher dimensions \"evaluating\" questions, the formidable linguistic capabilities of models indeed allow them to excel in these tasks, and perform even better than intermediate-level questions.\n\nThese are some of our conjectures. The reason for such a distribution stems from the models' proficiency in natural language. Future models should enhance their capability for reasoning and problem-solving. \n\n> Adding horizontal lines in Table 4 will improve the readability and consistency with other tables.\n\nThank you a lot for helping us improve our readability. We will proceed to revise the table accordingly. \n\n**[Final remark]** Once again, we extend our gratitude for your acknowledgment of the insights within our article and for the highly beneficial suggestions you provided. As many discussions and details are hidden due to the short paper's page limitation, we earnestly hope that our responses have addressed your concerns regarding the experiments, and we would appreciate it if you could consider raising the soundness score. Should you have further questions, we sincerely invite you to engage with us.\n\nRef:\n\n[1] Brown, Tom, et al. \"Language models are few-shot learners.\" Advances in neural information processing systems 33 (2020): 1877-1901.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- More details about the annotations for the dataset and models' answer/explanation accuracies would be important to ensure the credibility of the results. For example, how many annotators rated how many questions? What were the inter-rater scores? etc. \n- How the models generated explanations? Can you share the prompts?", "reasons_to_reject": "Although the results are interesting, and their usage of Bloom's taxonomy was really insightful, I think the paper missed essential details about the experiment settings and reproducibility.\n- More details about the annotations for the dataset and models' answer/explanation accuracies would be important to ensure the credibility of the results. For example, how many annotators rated how many questions? What were the inter-rater scores? etc. \n- How the models generated explanations? Can you share the prompts?", "questions_to_authors": "A. In Section 2.2 (Lines 140-145), the authors specified that \"all tasks are conducted in zero-shot scenario.\" However, in the same section, the authors also said, \"leverage the BM25 algorithm to retrieve the two most related discussions from the subtitles in the corresponding courses\". The authors in Appendix A (Lines 455-468) iterate that they tested in-context and non-context settings. These sounds contradict the first explanation of using \"zero-shot scenario.\" Can you clarify this?\n\nB. In Figure 2, do numbers represent students' performance/accuracy in corresponding questions?\n\nC. Can you add an extended version of Table 4 that includes ±context, ordering effect, and topic subjects? I think it will be very interesting.\n\nD. Can you add your insights in the Discussion section on why the LLMs perform worse in intermediate-level Bloom's Taxonomy questions and how to improve the models' performance?", "paper_topic_and_main_contributions": "This paper evaluated the OpenAI LLMs' performance in an educational assessment context, and how the models' outputs align with the cognitive psychology framework, like Bloom's Taxonomy. The authors showed that the models are better than humans in STEM but not in other topics, less proficient in questions located in the intermediate range of Bloom's Taxonomy, and performing less in multiple-choice questions than T/F or single-choice questions.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Adding horizontal lines in Table 4 will improve the readability and consistency with other tables."}, "extra_text": {}}, {"review_id": "ViVFtYDOxf", "reviewer": "Reviewer_Hzpw", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for acknowledging the novelty of our work and for providing constructive feedback on our method and experiments. We are deeply honored by the opportunity to enhance our work with your assistance and address some of your concerns.\n\n> The description of the method is too short, the description of diagnostic assessment should be added.\n\nWe appreciate your suggestion to fill in more details. Educational Diagnostic Assessment is a widely adopted method used in education to evaluate a student's knowledge and cognitive level, such as strengths, weaknesses, and learning styles [1]. In pursuit of a deeper comprehension of the knowledge and cognitive structure of LLMs, we drew inspiration from Educational Diagnostic Assessment to organize our experiments. To be precise, we adopt two traditional methods in Diagnostic Assessment, Deficit Assessment and Error Assessment [2], respectively focusing on the degree of knowledge mastery and the error patterns by examinations. We transpose them into the scenario of LLMs, devising experiments that respectively examine the knowledge structure and error types with explanations. This allows for a comprehensive assessment of the model's performance.\n\nShould this work be accepted, we intend to include more refined conceptual definitions on the additional page. We once again express our gratitude for your valuable suggestions.\n\n> The work needs to be evaluated against more models, resulting in more comprehensive analysis and discovery.\n\nWe perceive this as an excellent suggestion, especially in light of the current proliferation of powerful and diverse LLMs. We are indeed inclined to delve deeper into this realm. As a short paper, constrained by both length and resources, we employed three models that are currently at the forefront of capability and exhibit continuity in development. Our primary aspiration is to present a sharp insight that can attract more exceptional endeavors in this domain.\n\n**[Final Remark]** Thank you again for assisting us to improve our work. We hope our response has addressed your concerns about methods and models. We would really appreciate it if you could consider raising the score. If you have more questions, please feel invited to engage with us for further discussions.\n\n\nRef:\n\n[1] Falmagne, J-C., et al. \"The assessment of knowledge, in theory and in practice.\" IEMC'03 Proceedings. Managing Technologically Driven Organizations: The Human Side of Innovation and Change. IEEE, 2003.\n\n[2] Bejar, Isaac I. \"Educational diagnostic assessment.\" Journal of educational measurement, 1984.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This is an interesting study that reveals the knowledge structure of LLMs.\n\n2. This paper designs specific instructions to evaluate LLMs and assess their performances in three aspects.\n\n3. The authors carries out experimental design from multiple perspectives to evaluate LLMs more comprehensively and effectively.\n", "reasons_to_reject": "1. The description of the method is too short, the description of diagnostic assessment should be added.\n\n2. The work needs to be evaluated against more models, resulting in more comprehensive analysis and discovery.", "questions_to_authors": "", "paper_topic_and_main_contributions": "1. Based on educational diagnostic assessment method, this paper conducts an evaluation for LLMs. \n\n2. This paper aims to reveal the knowledge structures LLMs and gain insights of their cognitive capabilities. \n\n3. This research emphasizes the significance of investigating LLMs’ knowledge and understanding the disparate cogni- tive patterns of LLMs. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6WgQBeIdo6", "reviewer": "Reviewer_rWPM", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and constructive suggestions. We also genuinely appreciate your recognition of our experimental methodology, presentation, and readability. Here we would like to take this opportunity to address the concerns you've raised.\n\n> Weakness 1: In fact, different instructions have a significant effect on the results of LLMs. The authors need to add an analysis of the effect of the selected instructions on the current experimental results.\n\nThank you for pointing that out, and we believe this is a crucial question. This question encompasses two subsidiary questions: 1) Can our instructions be accurately comprehended by models? 2) How could our instructions affect the current experimental results?\n\n* Regarding the first question, before conducting the formal experiments, we assessed three candidate prospective prompts, by examining whether models can follow the instructions to generate answers and explanations. For example, in single-choice questions, both the choice and the explanation should be generated. We sampled 100 questions to examine three candidate prompts, and the results are shown below: \n\n|      | GPT3.5 | ChatGPT | GPT4  |\n|------|--------|---------|-------|\n| Prompt1 (Ours)                                    |\n| What's the answer to this single-choice question? Why? | 1.00 (0.99) | 1.00 (1.00) | 1.00 (1.00) |\n| What's the answer to this multiple-choice question? Why? | 1.00 (1.00) | 1.00 (1.00) | 1.00 (1.00) |\n| Please determine true or false and provide your reason. | 1.00 (1.00) | 1.00 (1.00) | 1.00 (1.00) |\n| Prompt2                                           |\n| Which of the following options in the given questions is the correct answer? The reason is?  | 1.00 (0.28) | 1.00 (1.00) | 1.00 (1.00) |\n| Which of the following options in the given questions are the correct answers? The reason is? | 1.00 (0.44) | 1.00 (1.00) | 1.00 (1.00) |\n| Is the statement of the following questions true or false? The reason is? | 1.00 (1.00) | 1.00 (1.00) | 1.00 (1.00) |\n| Prompt3                                           |\n| Please tell me the answer to this single-choice question and provide your reason. | 1.00 (0.99) | 1.00 (1.00) | 1.00 (1.00) |\n| Please tell me the answer to this multiple-choice question and provide your reason. | 1.00 (1.00) | 1.00 (1.00) | 1.00 (1.00) |\n| Please tell me the answer to this true or false question and provide your reason. | 1.00 (1.00) | 1.00 (1.00) | 1.00 (1.00) |\n\n\nThe numbers and the numbers within parentheses represent the probabilities assigned by the model to the provided answers and explanations, respectively. With our instructions, answers are correctly derived from the choices provided and reasons were given, which substantiates that models accurately comprehend them. \n\n* As for the second question, we also find this to be both interesting and significant. For example, in our selected instructions, we ask models to output answers before they give explanations. We have provided some insights in the Discussion section, where we conjecture models' explanations could shift due to wrong answers and question orders, as they have to give answers before rationales. This leads to a lower accuracy of explanations. Thank you for reminding us, and we intend to incorporate an extended analysis in the additional page if this paper could be accepted.\n\n> Weakness 2: Such an approach cannot be effectively extended to other datasets, which leads to limitations in exploring the structure of knowledge.\n\nThank you for your insightful question. In this work, our rationale for using the MoocRadar dataset stems from its detailed question annotations. It is its granularity of cognitive dimension annotating that has enabled us to achieve a deeper understanding of the cognitive capabilities of LLMs. As a short paper, we aimed to present insights, with the hope that in the future, more datasets with similarly detailed annotations on knowledge structure will emerge. This advancement will undoubtedly further propel the development of this field.\n\n> Weakness 3: I suggest that the author can describe the MoocRadar dataset more carefully (e.g., Knowledge-Types). This may be a barrier for readers.\n\nAs a brief supplement, the categorization according to the Bloom's Taxonomy is as follows:\n\n**Cognitive Dimensions** are: *Remembering* (Remember, Know, Identify), *Understanding* (Translate, Explain), *Applying* (Prove, Execute), *Analyzing* (Select, Compare, Summarize), *Evaluating* (Evaluate, Judge, Criticise), and *Creating* (Create, Design).\n\n**Knowledge Types** are: *Factual Knowledge* (knowledge of terminology and details), *Conceptual Knowledge* (knowledge of relationships and theories), *Procedural Knowledge* (knowledge of processes and methods), and *Metacognitive Knowledge* (knowledge of strategy, self-knowledge and \"thinking about thinking\"). [1, 2]\n\nThank you again for your suggestion. If this article is accepted, we will provide more comprehensive details about the dataset in the additional page to further improve our readability.\n\n**[Final Remark]** We would like to reiterate our gratitude. Thank you for your insightful recommendations that will contribute to the refinement of our work. We hope our supplementary information can address your concerns. If feasible, we would greatly appreciate your consideration in enhacing the paper's soundness score. If you have more questions, we are more than willing to engage in further discussions.\n\nRef:\n\n[1] Yu, Jifan, et al. \"MoocRadar: A Fine-grained and Multi-aspect Knowledge Repository for Improving Cognitive Student Modeling in MOOCs.\" in Proceedings of SIGIR, 2023.\n\n[2] Anderson, Lorin W., and David R. Krathwohl. A taxonomy for learning, teaching, and assessing: A revision of Bloom's taxonomy of educational objectives. Longman, 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strength 1: This paper is clearly organized and well-written. It does not create barriers for the readers and is very clear and logical in its presentation.\n\nStrength 2: The motivation is clear and the research question is valuable. This paper explores LLMs' knowledge structures and cognitive patterns and compares them with human cognition. This will help the progress of LLMs.\n\nStrength 3: The experimental methodology is presented very clearly, from three perspectives: Performance Analysis, Deficit Assessment and Error Assessment. The content is very detailed and the conclusion of the paper is relatively reasonable.\n\nStrength 4: The results of the experiment are very complete and illustrate the strengths and weaknesses of the current LLMs in answering questions and making explanations from different perspectives.\n", "reasons_to_reject": "Weakness 1: In fact, different instructions have a significant effect on the results of LLMs. The authors need to add an analysis of the effect of the selected instructions on the current experimental results.\n\nWeakness 2: Such an approach cannot be effectively extended to other datasets, which leads to limitations in exploring the structure of knowledge.\n\nWeakness 3: I suggest that the author can describe the MoocRadar dataset more carefully (e.g., Knowledge-Types). This may be a barrier for readers.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposed an educational diagnostic assessment approach to test the performance of LLMs on various dimensions. Current works lack a structured understanding of LLMs’ knowledge and cognitive patterns. This approach conducts an evaluation using MoocRadar, a meticulously annotated human test dataset based on Bloom Taxonomy, and delves into three primary research questions: Performance Analysis, Deficit Assessment and Error Assessment.\nThere are three main contributions: this work introduces the topic of the cognitive knowledge structure of LLMs; it proposes a method of Educational Diagnostic Assessment to evaluate LLMs on their cognitive knowledge structure; it assesses LLMs’ performance, deficits, and errors, gaining insights into their capabilities.\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper seeks to understand LLMs by applying a human test set based on Bloom's Taxonomy of Learning. Reviewers find that the approach has significant merit, but highlight weaknesses related to reporting information about the dataset, including annotator training and agreement, and robustness to instruction wording. These concerns appear to have been addressed in the discussion period.
Decision: Accept-Findings