RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
CogLM Benchmark Development
To comprehensively and accurately assess the cognitive abilities of LLMs, we undertake the following efforts:
(1) We revisit 12 cognitive abilities proposed by PTC, 10 of which are selected and redefined to construct CogLM according to the characteristics of LLMs (section 2.1 ).
(2) We create standardized data construction guidelines to ensure the quality of CogLM (section 2.2 ).
(3) We conduct extensive human testing to ensure the alignment between CogLM and PTC (section 2.3 ).
(4) We build a Calibrated Mapping Function to establish a reliable mapping between testing results on CogLM and cognitive age (section 2.4 ).
Definition of Cognitive Abilities
According to PTC, the development of human cognition is divided into four stages, which include 12 cognitive abilities. Considering that the interaction interface of most LMs is limited to text-based format, we exclude reflexes and sensorimotor aspects of multimodal interaction and build CogLM based on the remaining 10 cognitive abilities.

We strictly define the concept of each cognitive ability based on PTC and provide representative examples for explanation as shown in Table 1 .
Standardized Annotation Guidelines
To ensure that CogLM can accurately reflects the cognitive abilities of LLMs, we have established standardized annotation guidelines and strictly adhere to them during the annotation phase:

Data Format Although modern LLMs generally possess strong generation capabilities, early-aged LLMs (e.g., GPT-2) have limited generation abilities (similar to Human infants). Therefore, we have opted for multiple-choice questions as the assessment format. This approach avoids the influence of variations in generation capabilities on the accurate evaluation of cognitive abilities.

Number of Samples Abilities in the early stages are relatively simple and have a more concentrated form of expression, while abilities in the later stages are more comprehensive and have a more diverse form of expression. Based on this, we have set the number of samples to increase with each stage, as shown in the Table 2 .

Qualified Annotator We select adults with backgrounds in psychology or artificial intelligence as data annotators. Annotators are provided with detailed materials on PTC and required to study them carefully. We then assess annotators’ understanding of PTC through exams (see Appendix Table 10 for the examination paper). Finally, we provide annotators with at least 3 example samples for each cognitive ability. Each annotator is required to annotate no fewer than 30 questions and options for two specific cognitive abilities.

Annotation Quality Control After annotation, we conduct cross-checks among annotators to identify samples with quality issues. Quality issues include questions that cannot effectively assess the corresponding cognitive abilities, questions with ambiguities, and elements of bias or violence.
Consistency with Theory
After the dataset construction is completed, we consider conducting human tests to further ascertain whether CogLM is consistent with PTC and whether it can effectively reflect cognitive abilities. We randomly select 10 samples from each subset of CogLM to create questionnaires, which are then distributed to testers aged between 6 and 20. Out of the 207 completed questionnaires, 141 are deemed valid (based on the reasonableness of test duration 1 1 1 We deem the papers completed by the questionnaires in less than 10 minutes as invalid. ). We calculate the Spearman and Pearson correlation coefficients between the age of the participants and their questionnaire scores. It turns out that spearman correlation is 0.7169 and pearson correlation is 0.7362 (all the p-values <1e-10 ), indicating a strong correlation between them.
This statistical result validates the effectiveness of the Standardized Annotation Guidelines we have developed in ensuring the efficacy of CogLM for assessing cognitive abilities.
Calibrated Cognitive Age Mapping Function
After confirming the positive correlation between answer accuracy and cognitive age, we aim to further construct the mapping function between them. We first make adjustments to the method of calculating accuracy. The number of candidate options for questions in CogLM falls within the range [2,5] . Such a variability can impact the likelihood of providing a correct answer through guessing when participants are uncertain. Therefore, we calculate the calibrated accuracy on certain subset {\mathbb{S}} as follows:

Acc=\frac{1}{|{\mathbb{S}}|}\times\sum_{i=1}^{|{\mathbb{S}}|}\frac{\bm{1}_{\mathrm{predict_{i}=answer_{i}}}-1/|\mathrm{candidates_{i}}|}{1-1/|\mathrm{candidates_{i}}|} (1)

A negative calibrated accuracy (worse than random selecting) indicates a clear deficiency in the corresponding cognitive ability.
We further use 80% of the questionnaire results in Section 2.3 as the training set {\mathbb{S}}_{Q} to optimize the regression function f(\cdot) as follows:

\begin{gathered}\mathcal{L}_{regression}=\frac{1}{|{\mathbb{S}}_{Q}|}\times\sum_{i=1}^{|{\mathbb{S}}_{Q}|}(f(Acc_{i})-age_{i})^{2}\ f(Acc)=\sum_{i=1}^{4}w_{i}\times Acc_{\mathrm{stage}i}+b\end{gathered} (2)

The Spearman correlation between the age predicted by f(\cdot) and the real age on the remaining 20% samples is 0.9354, which signifies that f(\cdot) can precisely approximate the mapping from results on CogLM to cognitive age. We observe that w_{1}:w_{2}:w_{3}:w_{4}=1:2.6:1.4:2.5 , indicating that cognitive abilities in the second and fourth stages are better at reflecting cognitive age under the evaluation of CogLM .
Main Results
As shown in Table 3 , We run the model with the largest number of parameters in each series on CogLM , and report the adult human performance for comparison. Overall, the cognitive abilities of OPT, Llama-2-chat-70B, GPT-3.5-Turbo, and GPT-4 models successively increase, and the performance of each model gradually declines with the increase of stage, consistent with humans. Specifically, the latest state-of-the-art model, GPT-4, has demonstrated remarkable cognitive capabilities, achieving a level comparable to that of a 20-year-old human. It is also worth noting that both GPT-3.5-Turbo and GPT-4 surpass humans in empathy ability at stage 2, which is natural, as humans tend to have some degree of selfishness. Despite its superior performance, GPT-4’s performance on plan ability (59.4) is still barely satisfactory, far behind that of humans (95.6), which is consistent with the conclusion of Valmeekam et al. ( 2022 ) . Our results indicate that enhancing the ability of planning is the major direction for improving the overall cognitive abilities of LLMs in the future. For more detailed evaluation results, please refer to Appendix Table 8 .
What are the key factors affecting the cognitive abilities of LLMs?
We explore this question from two perspectives: the parameter size and the optimization objective of LLMs, as they are proven to have significant impact on other abilities. We leave the exploration of factors that require changes to the parameters of LLMs (e.g. fine-tuning on different types of datasets) for future work.
Can advanced technologies help enhance LLMs’ cognitive abilities?
To answer this question, we applied two representative techniques separately to measure whether cognitive abilities of LLMs could be improved.
How Cognitive Ability Affects the Performance of LLM
According to PTC, the development of human cognitive abilities is a gradual process, where the cognitive abilities of early stages can influence the advanced cognitive abilities. Additionally, cognitive abilities significantly determines the capacity to solve real-world problems. Therefore, we are very interested in whether these two aspects are similarly manifested in LLMs.
Potential applications of advanced LLM cognitive ability
Although there is still room for improvement, the cognitive abilities of advanced LLMs have approached levels close to that of adult humans as discussed in Section 3.2 . A natural question is, what are the potential applications for advanced LLMs’ cognitive abilities? When humans address cognitive questions, they deduce and provide answers based on their cognitive abilities. While we have demonstrated in Section 3.3.2 that the cognitive chain-of-thought (Chain-of-Cognition, COC) generated by LLMs barely help self-address cognitive questions, we are curious whether COC can assist early-aged LLMs in improving cognitive performance. On this basis, we use the question together with the COCs generated by advanced LLM (GPT-3.5-Turbo) as input to test the performance of early-aged LLM (Llama-2-chat-7B) on CogLM . As shown in Table 6 , in most cognitive abilities, COC can significantly improve the performance of Llama-2-chat-7B. We leave the research on using COC from advanced LLMs to guide the improvement of cognitive abilities in early-aged LLMs and even children for future exploration.
Conclusions
In this paper, we introduce Piaget’s Theory of Cognitive Development (PTC) as a tool to track the development of cognitive abilities of LLMs. We construct CogLM based on the scenerio experiments used in PTC, and conduct thorough human testing to ensure the alignment between CogLM and PTC. Through extensive experiments on multiple series of LLMs, we show that: (1) Human-like cognitive abilities have emerged in State-of-the-art LLMs (GPT-4), comparable to those of 20-year-old humans. (2) The parameter size and optimization objective are two key factors affecting the cognitive abilities of LLMs. (3) The ability of performing downstream tasks is positively correlated with the level of cognitive abilities.
We believe that our findings can provide a novel insight into the emergence of abilities in LLMs, and shed light on the future development advanced abilities of LLMs.
Limitations
Despite obtaining some valuable findings through CogLM, our current exploration does not consider the language model’s performance at different training stages to further provide insights for model training, and we will explore it in our future work.
Ethics Statement
Our dataset does not contain any harmful or offensive contents. Any personal or sensitive information is anonymized and treated with utmost confidentiality. We ensure the protection of participants’ privacy and obtain informed consent for data collection, annotation, and analysis. We incentivized all the annotators uniformly throughout the annotation process.
Figure 1: Average calibrated accuracy (%) of models with different parameter size and humans in different cognitive stage.; Average calibrated accuracy (%) of models with different parameter size and humans in different cognitive stage.
Figure 2: Comparison of the performance of Llama-2-70B and Llama-2-chat-70B at each stage on CogLM .; Comparison of the performance of Llama-2-70B and Llama-2-chat-70B at each stage on CogLM .
Figure 3: Cognitive ability interdependence matrix. The vertical axis represents cognitive abilities erased through prompts, and the color depth (calibrated accuracy) indicates the impact on the corresponding horizontal axis abilities after erasure.; Cognitive ability interdependence matrix. The vertical axis represents cognitive abilities erased through prompts, and the color depth (calibrated accuracy) indicates the impact on the corresponding horizontal axis abilities after erasure.
Definitions and examples of cognitive abilities included in CogLM .
Data statistics on all ability subsets of CogLM .
Calibrated accuracy (%) of largest model in evaluating series. Acc and Age refer to calibrated accuracy and the age of equivalent human performance. The value of Age is calculated according to Equation 2 . Bold indicates the best performance.
Calibrated accuracy of GPT-3.5-Turbo on CogLM with multiple settings. "Base" indicates settings where both COT and SC are not used.
Accuracy (%) of GPT-3.5-Turbo on GSM8K and StrategyQA datasets when different cognitive abilities are erased.
Calibrated accuracy of Llama-2-chat-7B on CogLM with and without Chain-of-Cognition from GPT-3.5-Turbo as input.
The statistics of considered language models.
Calibrated accuracy (%) of all models in evaluating series. Acc and Age refer to calibrated accuracy and the age of equivalent human performance. The value of Age is calculated according to Equation 2 and rounded to the nearest integer. Bold indicates the best performance.
Cognitive-ability-setting-prompts of different cognitive abilities.
Examination paper to ensure the annotators are qualified.