The Cognitive Capabilities of Generative AI: A Comparative Analysis with Human Benchmarks

## Abstract

Abstract There is increasing interest in tracking the capabilities of general intelligence foundation models. This study benchmarks leading large language models and vision language models against human performance on the Wechsler Adult Intelligence Scale (WAIS-IV), a comprehensive, population-normed assessment of underlying human cognition and intellectual abilities, with a focus on the domains of Verbal Comprehension (VCI), Working Memory (WMI), and Perceptual Reasoning (PRI). Most models demonstrated exceptional capabilities in the storage, retrieval, and manipulation of tokens such as arbitrary sequences of letters and numbers, with performance on the Working Memory Index (WMI) greater or equal to the 99.5 th percentile when compared to human population normative ability. Performance on the Verbal Comprehension Index (VCI) which measures retrieval of acquired information, and linguistic understanding about the meaning of words and their relationships to each other, also demonstrated consistent performance at or above the 98 th percentile.
Despite these broad strengths, we observed consistently poor performance on the Perceptual Reasoning Index (PRI; range 0.1-10 th percentile) from multimodal models indicating profound inability to interpret and reason on visual information. Smaller and older model versions consistently performed worse, indicating that training data, parameter count, and advances in tuning are resulting in significant advances in cognitive ability.

## 1 Introduction

Generative artificial intelligence (GenAI) refers to a class of models capable of creating new content, whether it is text, images, music, or even code (Team et al., ; Wu et al., ; Gardner et al., ; Ouyang et al., ). Unlike traditional AI systems designed for specific tasks, these models learn the underlying patterns and structures within vast datasets and then use this knowledge to generate novel outputs that often mimic human creativity (Zhao et al., ; Nath et al., ). The excitement surrounding GenAI stems from its potential to revolutionize numerous fields by performing human-like cognitive functions (Wang et al., ; Zhuang et al., ; Zhang et al., ). This ability to understand, learn, adapt, and create in a way that mirrors our own thought processes opens doors to groundbreaking applications across industries like art, design, research, and communication (Ko et al., ; Lu et al., ; Ge et al., ; Yang et al., ). However, human cognition encompasses a vast array of specialized abilities in the processing, storage, interpretation, and generation of information across auditory and visual channels to accomplish these unique human capabilities (Cao et al., ; Subramonyam et al., ).

Recent advances in generative AI have been driven by large parameter models (e.g., the GPT (Achiam et al., ), Gemini (Team et al., ) families) trained with a broad range of textual content. Scaling experiments have shown that testing losses often follow power law relationships with data, model size (number of parameters) and compute (Kaplan et al., ). This progress has extended to multimodal models that learn complex cross-modal relationships between representations such as language and visual media. The capabilities of these models has sparked debate about the potential impact of AI and proximity to artificial general intelligence (Bengio et al., ). The rapid progress increases the need for careful assessment of performance.

To assess their potential and limitations as models of human cognition, it is crucial to characterize the cognitive capabilities of generative models and understand how they compare directly to human ability (Ilić and Gignac, ). This will facilitate an understanding of pragmatic capabilities, limitations, and an ability to track progress just as is done with human normative cognitive tests of ability. It will also provide insight into unique streams of cognitive development of generative artificial intelligence under the assumption that models, like biological organisms, will develop unique capabilities based on their underlying neural architecture and environmental demands.

Figure: Figure 1: Benchmarking Models Against Human Performance on the Wechsler Adult Intelligence Scale. We perform a comprehensive, population-normed assessment of AI models against underlying human cognition and intellectual abilities, with a focus on the domains of Verbal Comprehension (VCI), Working Memory (WMI), andPerceptual Reasoning (PRI).
Refer to caption: extracted/5908197/Hero_Figure.png

General cognitive capabilities have been systematically researched and benchmarked in humans for over a century (Spearman, ; Sternberg et al., ; Weiss et al., ). Refined standardized tests of ability and performance that are normed in large representative population-based samples allow for individual performance scores to be compared directly to normative performance. Because tests such as the Wechsler Adult Intelligence Scale–Fourth Edition (WAIS-IV; (Wechsler, )) have population norms for comparison, individual scores can be used to make decisions about relative ability, and can be used to inform the allocation of special resources for education, housing, care, or competency. They can also be utilized to identify exceptional ability and gaps between areas of performance that are diagnostic of specific neurocognitive disorders (e.g., Autism Spectrum Disorder, Dyslexia; (Neisser et al., )).

Beyond the focus on comparison to human performance, there may be value in the development of specific tests of GenAI cognition across discernible domains that are analogous to human tests akin to the development of analogous tests of cognition in animal models. Indeed, models of cognitive constructs such as working memory, semantic understanding, and visual processing, have facilitated research to understand the underlying neural architecture and mechanisms of cognition. Similar to the relative capabilities of the visual cortex of an hawk compared to a human – both display impressive but highly specialized abilities – we may find that GenAI models, by virtue of their architecture and context, develop novel patterns in cognitive functioning that prove to be quite different from humans, where, for example, consistent non-negligible positive correlations have been demonstrated between many cognitive ability test scores (Detterman and Daniel, ; Walker et al., ). This phenomenon, which is known as the Positive Manifold (Jensen, ), has been demonstrated to hold for LLMs using acceptable, yet non-validated, subtest proxies of the VCI and WMI cognitive ability tests (Ilić and Gignac, ).

We compared the performance of GenAI models on tests of cognitive performance that are commonly used to index human intellectual ability. We focused on tests of verbal ability including linguistic-based knowledge and conceptual understanding, capabilities to temporarily store, retrieve and manipulate arbitrary information, and capabilities to understand, reason, and manipulate visual information. Both individual test scores and composite index scores provide nuanced and interpretable information about intellectual capabilities and both were used to evaluate and better understand the capabilities of generative models.

## 2 Methods

To facilitate the assessment of GenAI models using the Wechsler Adult Intelligence Scale, Fourth Edition (WAIS-IV), we implemented a series of methodological adaptations to accommodate the unique input and output modalities of these models. This involved converting traditional verbal and visual stimuli into text-based prompts (see Appendix B) and interpreting model-generated text outputs as responses to test items. Specific adaptations for each subtest, along with validation procedures, are detailed in 2.1. For this study, we selected a representative set of state-of-the-art (SOTA) large language models (LLMs) and vision language models (VLMs), encompassing a range of model sizes, architectures, and training datasets (See section 2.2).

### 2.1 Testing materials, administration, and scoring

Individual items from WAIS-IV subtests (Wechsler, ) were converted to prompts to be individually administered to the selected LLM (See Table 1 for a full list of tests with descriptions). Specifically, tasks comprising the verbal comprehension and working memory indices (VCI and WMI) were converted to language-based prompts and administered to all models including those with language-only capabilities as well as those with multimodal language and visual capabilities. Tests from the perceptual reasoning index (PRI) were only administered to multimodal models because they require both image recognition and language capabilities. We were unable to administer the Block Design subtest from the PRI, because it requires manual manipulation of real-world objects (cubes). However, valid PRI composite scores were still calculable substituting the alternate PRI subtest Figure Weights (Wechsler, ). Subtests from the Processing Speed Index (PSI) were not administered as there was no clear way to maintain fidelity to the WAIS-IV testing procedures that are required for valid comparison to human performance norms. A full-scale IQ score (FSIQ) could not be calculated for any model as PSI is a required component of FSIQ.

In each instance, the prompt presented to the model consisted of the instructions as outlined in the manual (Wechsler, ), including the example provided as part of the administration process. However, certain adaptations were necessary to accommodate the specific requirements of the model. Several tests involved reading the test content, while others utilized visual and physical aids (e.g., cards and boards). These tests were converted into prompts, and in some cases, the translation provided the GenAI models with an advantage due to their ability to access the full context while generating responses.
Additionally, phrases such as "You can only ask me to read the problem one more time" were omitted, as GenAI models are trained on instructions and often request repetition without responding to the actual assessment being conducted. Phrases such as "Just say what I say" or "Listen" were not removed and were left as they were originally presented, causing some models to mistakenly provide responses such as "I am a text-based chat assistant and thus I cannot hear or repeat the numbers." and "Okay, I’m listening!.". No further instructions or inquiries were provided.
No additional instructions or queries were provided.

Each subtest consists of progressively challenging questions whose difficulty-level is normed against the population. For example, the Digit Span test, in which subjects are requested to verbally repeat a sequence of digits read aloud to them, begins with a sequence of three digits and progresses to a sequence of eight or nine digits depending on the subtest. Primary WAIS-IV composite indices and related cognitive domains are described below, along with brief descriptions of each subtest and administration procedures (See Table 1).

#### Verbal Comprehension Index

These tests assess verbal knowledge and reasoning capacity. Subtests include: Similarities, in which the subject must state how two words are alike (e.g., "How are a kite and an airplane alike?"); Vocabulary, in which the subject provides definitions of words; Information in which the subject answers questions on a broad set of topics (e.g., "Who was the first president of the United States?"); and Comprehension in which subjects are asked to reason on common sense narrative scenarios.

#### Working Memory Index

These subtests probe the capacity to store, manipulate, and reorder information, including reasoning numerically. Digit Span requires subjects to repeat back random number strings according to varying rules (i.e., Forwards, Backwards, and Sequencing conditions). Arithmetic requires subjects to solve math word problems of increasing difficulty without pen and paper. Letter-Number Sequencing requires subjects to manipulate arbitrary letter-number strings into alphabetic and numerical order.

#### Perceptual Reasoning Index

Involves tests of visuospatial skills and nonverbal abstract reasoning. Matrix Reasoning (Ex: Fig. 2) requires subjects to choose the correct image from multiple options to successfully complete a visual pattern. Visual Puzzles (Ex: Fig. 3) requires subjects to select the correct tile to complete a picture, assessing mental rotation and spatial relations ability. In Figure Weights (Ex: Fig. 4), subjects are presented with an analogue scale depicting shapes of relative weight (e.g., two blue squares equals one red triangle). The subject is presented with a balanced scale demonstrating the weight relationship between objects, and an incomplete scale with shapes on only one side. They must then select from multiple choices to balance the incomplete scale. Picture Completion is a test in which subjects are presented with an image that is missing some key characteristic (e.g., an image of a woman without a shadow standing next to a sign that is casting a shadow). The subject must verbally indicate what is missing in the image.

All answers to prompts were scored by one of two clinical psychologists trained to administer and score the WAIS-IV. Any ambiguous results were reviewed by both psychologists to reach a consensus on scoring. Individual test scores were collated into raw index scores and then converted to age-normed scores with accompanying performance percentiles. As different norms exist for different age ranges, we selected the norms for age 25 years to 29 years and 11 months. In addition to the calculation of composite indices, we also compared individual subtests to domain-specific averages and composite scores in a series of discrepancy analyses, in order to probe the relative strengths and weaknesses of the models. The WAIS-IV provides standardized scoring for a variety of such comparisons, to determine if performance on an individual subtest might represent a statistically significant deviation from what would be expected in the population of those with similar performance profiles.

**Table 1: Summary of Tests. Indices, subtests, and cognitive dimensions measured by the WAIS-IV.**
|  | Index / Subtest | Cognitive Dimensions | Administration | Assessed |
| --- | --- | --- | --- | --- |
| Verbal Comprehension | Similarities | Verbal abstract reasoning, concept formation | Examinee verbally explains how two words are alike. | T<br>M |
| Vocabulary | Language comprehension, word knowledge | Examinee provides verbal definitions of words. | T<br>M |  |
| Information | General fund of knowledge, long-term memory recall | Examinee answers questions covering a broad range of general knowledge areas. | T<br>M |  |
| Comprehension | Social judgment, common sense, practical reasoning | Examinee answers questions about social situations and common sense problem-solving. | T<br>M |  |
| Perceptual Reasoning | Block Design | Visual construction, spatial reasoning | Examinee manipulates blocks to reproduce visual designs. | N/A |
| Matrix Reasoning | Nonverbal abstract reasoning, pattern recognition | Examinee selects the missing element that completes a visual pattern. | M |  |
| Visual Puzzles | Mental Rotation, Spatial reasoning, part-whole analysis | Examinee chooses pieces that fit together to create a complete visual image. | M |  |
| Figure Weights | Quantitative and analogical reasoning | Examinee views scales and selects the response option that balances the scale. | M |  |
| Picture Completion | Visual perception, attention to detail | Examinee identifies the missing element from a picture. | M |  |
| Working Mem. | Digit Span | Attention, working memory, mental manipulation | Examinee recalls a series of digits forward and backward. | T<br>M |
| Arithmetic | Calculation Ability | Examinee mentally solves arithmetic word problems. | T<br>M |  |
| Letter-Num. Seq. | Working memory, alphanumeric sequencing | Examinee performs alphanumeric sequencing of letter-number strings. | T<br>M |  |
| Process. Speed | Symbol Search | Visual scanning, target discrimination, processing speed | Examinee rapidly scans a search group and indicates whether a target symbol is present. | N/A |
| Coding | Graphomotor processing speed | Examinee copies symbols paired with geometric shapes as quickly as possible. | N/A |  |
| Cancellation | Selective attention, visual scanning | Examinee scans an array of visual stimuli and marks target items according to a rule. | N/A |  |

### 2.2 Models

Language only models included OpenAI’s GPT-3.5 Turbo, Google’s Gemini Nano, Gemini Pro, and Gemini Advanced. Multimodal models included OpenAI’s GPT-4 Turbo, GPT-4o, Google’s Gemini Flash, Gemini Goldfish and Anthropic’s Claude 3 Opus, Claude 3.5 Sonnet. While model characteristics including the underlying training data, number of parameters, or internal tuning approach in the model are not publically available, key characteristics of the models are known that can be informative about factors that influence performance. First, we are able to compare progressive versioning of models to qualitatively compare performance. As an example, GPT-3.5 Turbo, Gemini Pro, and Claude 3 Opus are all earlier versions when compared to GPT-4 Turbo, GPT-4o, Gemini Advanced, and Claude 3.5 Sonnet. Further, Gemini Flash and Gemini Nano are intentionally designed as smaller models to be hosted on hardware chips rather than in cloud computing where the models can be much larger. Finally, Claude models indicate that they optimize for advanced reasoning. While all of these pieces of information are limited due to the lack of public disclosure, they do allow us to determine if 1) larger parameter versus smaller parameter models perform better; 2) models intentionally optimized for reasoning do improve reasoning capabilities.

## 3 Results

Index level results (VCI, WMI, PRI) demonstrate both exceptional capabilities and significant deficits when compared to a normative population (See Table 2 for full results). All but one model demonstrated abilities on the VCI in the Very Superior range, scoring within the top percentile of human normative ability with the exception of Gemini Flash which fell in the Superior range (82^nd%ile) and Gemini Nano (23^rd %ile) which demonstrated ability in the Borderline range. The majority of models also performed in the Very Superior range on the WMI, with the exception of Gemini Nano (37^th%ile; Low Average). However, among the models with multimodal capabilities, performance on the PRI demonstrated a significant and consistent deficit with all models demonstrating capabilities in the Extremely Low range ($<$ 1^st%ile) with the lone exception of Claude Sonnet which demonstrated ability in the Low Average range (10^th%tile).

Next, we analyzed individual differences in index-level results to understand relative strengths and weaknesses within domains of intellectual ability (See Table 3 for full Results). First, we observed a consistent relative strength in WMI compared to other indexes representing a statistically significant difference compared to the normative population (p$<$.15 or p$<$.05 level). Overall,this indicates that models’ generally evidence stronger ability to process, store and manipulate information compared to any other ability, including language. Further, all models with visual/multimodal capability to perform tests comprising the PRI demonstrated significantly worse perceptual reasoning performance p$<$.05) when compared verbal comprehension and working memory indexes. This indicates a general relative weakness that is invariant across distinct developers and models in the ability to understand and reason on visual information.

**Table 2: Model Scores on the WAIS-IV. Composite index standard scores, individual subtest scaled scores, and percentile rankings for each generative AI model tested.**
|  | OpenAI | Google | Anthropic |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | GPT-3.5 | GPT-4 | GPT-4o | Gemini | Gemini | Gemini | Gemini | Gemini | Claude 3 | Claude 3.5 |  |
|  | Turbo | Turbo |  | Nano | Pro 1.0 | Adv. | Flash | Goldfish | Opus | Sonnet |  |
|  |  |  |  | (1.0 S) | (1.0 M) | (1.0 XL) | (1.5 S) | (1.5 M) |  |  |  |
| WAIS-IV Comp. Idx. | Scaled | Scaled | Scaled | Scaled | Scaled | Scaled | Scaled | Scaled | Scaled | Scaled |  |
| WAIS-IV Subtests | Score (%) | Score (%) | Score (%) | Score (%) | Score (%) | Score (%) | Score (%) | Score (%) | Score (%) | Score (%) |  |
| Verbal Comp. | 143 | 136 | 141 | 89 | 132 | 134 | 114 | 141 | 138 | 145 |  |
| Standard Score (%ile) | (99.8) | (99) | (99.7) | (23) | (98) | (99) | (82) | (99.7) | (99) | (99.9) |  |
| Similarities | 13 (84) | 14 (91) | 15 (95) | 1 (0.1) | 8 (25) | 13 (84) | 14 (91) | 14 (91) | 13 (84) | 14 (91) |  |
| Vocabulary | 19 (99.9) | 15 (95) | 16 (98) | 4 (2) | 19 (99.9) | 15 (95) | 6 (9) | 17 (99) | 17 (99) | 19 (99.9) |  |
| Information | 19 (99.9) | 19 (99.9) | 19 (99.9) | 19 (99.9) | 19 (99.9) | 18 (99.6) | 19 (99.9) | 19 (99.9) | 19 (99.9) | 19 (99.9) |  |
| Comprehension | 19 (99.9) | 18 (99.6) | 19 (99.9) | 12 (75) | 18 (99.6) | 19 (99.9) | 19 (99.9) | 19 (99.9) | 17 (99) | 19 (99.9) |  |
| Working Mem. Idx. | 139 | 150 | 150 | 95 | 139 | 150 | 139 | 145 | 150 | 145 |  |
| Standard Score (%ile) | (99.5) | (>99.9) | (>99.9) | (37) | (99.5) | (>99.9) | (99.5) | (99.9) | (>99.9) | (99.9) |  |
| Digit Span | 19 (99.9) | 19 (99.9) | 19 (99.9) | 11 (63) | 19 (99.9) | 19 (99.9) | 19 (99.9) | 19 (99.9) | 19 (99.9) | 19 (99.9) |  |
| DS Forward | 18 (99.6) | 18 (99.6) | 18 (99.6) | 18 (99.6) | 18 (99.6) | 18 (99.6) | 18 (99.6) | 18 (99.6) | 18 (99.6) | 18 (99.6) |  |
| DS Backwards | 18 (99.6) | 16 (98) | 19 (99.9) | 4 (2) | 19 (99.9) | 19 (99.9) | 19 (99.9) | 19 (99.9) | 19 (99.9) | 19 (99.9) |  |
| DS Sequencing | 19 (99.9) | 19 (99.9) | 19 (99.9) | 13 (84) | 19 (99.9) | 19 (99.9) | 18 (99.6) | 19 (99.9) | 19 (99.9) | 17 (99) |  |
| Arithmetic | 15 (95) | 19 (99.9) | 19 (99.9) | 7 (16) | 15 (95) | 19 (99.9) | 15 (95) | 17 (99) | 19 (99.9) | 17 (99) |  |
| LN Sequencing | 19 (99.9) | 19 (99.9) | 19 (99.9) | 1 (0.1) | 19 (99.9) | 19 (99.9) | 16 (98) | 19 (99.9) | 14 (91) | 16 (98) |  |
| Perceptual Reas. | – | 50 | 58 | – | – | – | 50 | 52 | 50 | 81 |  |
| Standard Score (%ile) | – | (<0.1) | (0.3) | – | – | – | (<0.1) | (<0.1) | (<0.1) | (10) |  |
| Matrix Reasoning | – | 1 (0.1) | 1 (0.1) | – | – | – | 1 (0.1) | 1 (0.1) | 1 (0.1) | 8 (25) |  |
| Visual Puzzles | – | 1 (0.1) | 2 (0.4) | – | – | – | 1 (0.1) | 1 (0.1) | 0 (<0.1) | 2 (0.4) |  |
| Figure Weights | – | 2 (0.4) | 6 (9) | – | – | – | 2 (0.4) | 4 (2) | 1 (0.1) | 10 (50) |  |
| Picture Completion | – | 1 (0.1) | 1 (0.1) | – | – | – | 1 (0.1) | 1 (0.1) | 1 (0.1) | 1 (0.1) |  |

Next, we examined performance on individual tests within each index (See Table 3 and Table 4). Within the WMI, a consistent relative weakness in Arithmetic, was observed in comparison to Digit Span, indicating a relative weakness in mathematical reasoning compared to the ability to encode, manipulate, and reorganize numeric values. Gemini Nano demonstrated further relative weaknesses in the ability to sequence and reverse numbers compared to the ability to encode and retrieve simple numeric lists. Analysis of the VCI demonstrated consistent relative strengths across models from independent developers, on the Information subtest compared to Similarities and Vocabulary (p$<$.05 and p$<$.15). This indicates that models are particularly strong in the storage and retrieval of natural language-encoded knowledge – often referred to as “crystallized knowledge” in human subjects – in comparison to the cognitive capabilities required for verbal analogic and verbal abstract reasoning and semantic understanding. This difference in relative performance persisted across generations of models indicating that models are consistently stronger in predicting the write answer than understanding and reasoning with language concepts. Finally, within the PRI, relative differences in opposite directions between best-in class models were observed with GPT-4o demonstrating stronger relative ability in decoding Visual Puzzles and Claude Sonnet demonstrating significantly stronger relative performance in Matrix Reasoning.

**Table 3: Discrepancy Score Comparison Table for WAIS-IV Composite Indices. Includes scale-level comparisons for Digit Span vs. Arithmetic and individual Digit Span component score comparisons.**
|  | OpenAI | Google | Anthropic |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Index Level Comparisons | GPT-3.5 | GPT-4 | GPT-4o | Gemini | Gemini | Gemini | Gemini | Gemini | Claude 3 | Claude 3.5 |  |
|  | Turbo | Turbo |  | Nano | Pro 1.0 | Adv. | Flash | Goldfish | Opus | Sonnet |  |
|  |  |  |  | (1.0 S) | (1.0 M) | (1.0 XL) | (1.5 S) | (1.5 M) |  |  |  |
| VCI - PRI (Difference) | – | 86** | 83** | – | – | – | 64** | 89** | 88** | 64** |  |
| VCI - PRI Base Rate | – | 0.2% | 0.2% | – | – | – | 0.2% | 0.2% | 0.2% | 0.2% |  |
| (Critical Value, Significance) |  | 8.32** | 8.32** |  |  |  | 8.32** | 8.32** | 8.32** | 8.32** |  |
| VCI - WMI (Difference) | 4 | -14** | -9** | -6 | -7* | -16** | -25** | -4 | -12** | 0 |  |
| VCI - WMI Base Rate | – | 14.1% | 25% | – | 29.7% | 10.5% | 3.0% | – | 18.1% | – |  |
| (Critical Value, Significance) |  | 8.81** | 8.81** |  | 6.48* | 8.81** | 8.81** |  | 8.81** | – |  |
| PRI - WMI (Difference) | – | -100** | -92** | – | – | – | -89** | -93** | -100** | -64** |  |
| PRI - WMI Base Rate | – | 0.1% | 0.1% | – | – | – | 0.1% | 0.1% | 0.1% | 0.1% |  |
| (Critical Value, Significance) |  | 8.81** | 8.81** |  |  |  | 8.81** | 8.81** | 8.81** | 8.81** |  |
| Scale Level Comparisons |  |  |  |  |  |  |  |  |  |  |  |
| Digit Span Total - Arithmetic | 4** | 0 | 0 | 4** | 4** | 0 | 4** | 2* | 0 | 2** |  |
| Base Rate | 9.4% | – | – | 9.4% | 9.4% | – | 9.4% | 29% | – | 29% |  |
| (Critical Value, Significance) | 2.57** |  |  | 2.57** | 2.57** |  | 2.57** | 1.89* |  | 1.89* |  |
| DS Forward-DS Backwards | 0 | 2 | -1 | 14** | -1 | -1 | -1 | -1 | -1 | -1 |  |
| Base Rate | – | – | – | $<$0.10% | – | – | – | – | – | – |  |
| (Critical Value, Significance) |  |  |  | 3.65** |  |  |  |  |  |  |  |
| DS Forward-DS Sequence | -1 | -1 | -1 | 5** | -1 | -1 | 0 | -1 | -1 | -1 |  |
| Base Rate | – | – | – | 8.5% | – | – | – | – | – | – |  |
| (Critical Value, Significance) |  |  |  | 3.60** |  |  |  |  |  |  |  |
| DS Backward-DS Sequence | -1 | -3* | 0 | -9** | 0 | 0 | 0 | 0 | 0 | 0 |  |
| Base Rate | – | 17.3% | – | 0.1% | – | – | – | – | – | – |  |
| (Critical Value, Significance) |  | 2.62* |  | 3.56** |  |  |  |  |  |  |  |

Wide variability in performance was observed on the VCI. While scores were highly skewed towards exceptional performance (99.5^th%ile to $>$99.9^th%ile, models that are known to be small parameter counts demonstrated the worst relative and absolute performance (Gemini Flash = 82^nd%ile; Gemini Nano = 23^rd%ile). Similarly, most models performed exceptionally on the WMI (Range, 95^th %ile - $>$99.9^th %ile) with the exception of Gemini Nano (37^th %ile). In contrast, all models tested demonstrated poor performance on the PRI (Range, $<$.01^th%ile - 10^th%ile), with skewed performance such that the majority of models fell below the 2^nd%ile. The exception is Claude Sonnet which demonstrates a significant increase over its predecessor Claude Opus indicating that while performance overall is poor, there is measurable progress in the ability for generative models to reason on visual concepts.

Test level analyses demonstrate statistically significant relative strengths and weaknesses within individual composite indices. First, all models demonstrated perfect or near perfect scores on Information indicating that prior encoding and ability to access crystalized knowledge is exceptional compared to a normative population (Range, 99.6^th%ile - 99.9^th%ile). Next, within the VCI, multiple models demonstrated relative weaknesses in similarities (Gemini Nano, GPT 3.5, Gemini Pro; range indicating a relative difficulty reasoning on linguistic concepts compared to correctly retrieving crystallized linguistic knowledge. Gemini Nano and Gemini Flash both demonstrated relative weaknesses in vocabulary indicating these models struggle to understand language relative to other models. Gemini Nano also demonstrates lower scores in Comprehension when compared to Information, reinforcing that while this model has encoded and can retrieve crystallized knowledge well, it struggles to demonstrate human-level reasoning with linguistic information.

With regard to the WMI, models again performed exceptionally well with all demonstrating perfect scores in Digit Span with the exception of Gemini Nano which demonstrated a relatively poorer performance that was consistent with average human performance. Both Arithmetic and Letter Number Sequencing represented relative weaknesses for Gemini Nano. Further, when comparing subtests of Digit Span, we observed a relative weakness in Digit Span Backwards when compared to Digit Span Forward. Once again, Nano performed inconsistently, demonstrating abilities in digit span forward that were in the 99.6^th%ile) while failing at manipulating similar numeric strings (Digit Span Backwards 2^nd%ile). Further, Nano was unable to correctly perform any Letter-Number sequencing tasks (see Table 5 for complete results). Taken together, results indicate that Nano is proficient in encoding and retrieving the correct information but performs poorly when tasked with manipulating information, a key attribute of working memory.

Finally the PRI was assessed in the subset of multimodal models (GPT-4 Turbo, GPT-4o, Gemini Flash, Gemini Goldfish, Claude 3 Opus, Claude 3.5 Sonnet) demonstrating consistent performance in the Extremely Low range ($<$ 0.2^nd% ile). The best performing model was Claude 3.5 Sonnet which fell in the 10^th%ile, consistent with Borderline performance in this domain. The Figure Weights test was a relative strength compared to other tests (Visual Puzzles, Matrix Reasoning) indicating a relative strength in analytical reasoning compared to pattern recognition or spatial reasoning. Of note, and despite overall poor performance, Claude 3.5 Sonnet demonstrated dramatic improvements over Claude 3 Opus in Matrix Reasoning and Figure Weights indicating that while performance in these domains lag behind language-based domains, models can be trained and optimized to acquire perceptual reasoning capabilities.

**Table 4: Verbal Comprehension vs Perceptual Reasoning. Relative strengths and weaknesses for WAIS-IV verbal comprehension and perceptual reasoning subtests.**
|  | OpenAI | Google | Anthropic |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Verbal Subtests | GPT-3.5 | GPT-4o | GPT-4 | Gemini | Gemini | Gemini | Gemini | Gemini | Claude 3 | Claude 3.5 |  |
|  | Turbo | Turbo |  | Nano | Pro 1.0 | Adv. | Flash | Goldfish | Opus | Sonnet |  |
|  |  |  |  | (1.0 S) | (1.0 M) | (1.0 XL) | (1.5 S) | (1.5 M) |  |  |  |
| Mean VCI Subtests | 17.00 | 16.00 | 16.67 | 8.00 | 15.33 | 15.67 | 12.67 | 16.67 | 16.33 | 17.33 |  |
| Similarities (subtest - mean) | -4.00** | -2.00** | -1.67* | -7.00** | -7.33** | -1.67* | 0.33 | -2.67** | -3.33** | -3.33** |  |
| Base Rate | 2% | 15% | 25% | 1% | 1% | 25% | – | 5% | 2% | 2% |  |
| (Critical Value, Significance) | 1.91** | 1.91** | 1.56* | 1.91** | 1.91** | 1.56* |  | 1.91** | 1.91** | 1.91** |  |
| Vocabulary (subtest - mean) | 2.00** | -1.00 | -0.67 | -4.00** | 3.67** | -0.67 | -6.67** | 0.33 | 0.67 | 1.67** |  |
| Base Rate | 10% | – | – | 1% | 1% | – | 1% | – | – | 15% |  |
| (Critical Value, Significance) | 1.58** |  |  | 1.58** | 1.58** |  | 1.58** |  |  | 1.58** |  |
| Information (subtest - mean) | 2.00** | 3.00** | 2.33** | 11.00** | 3.67** | 2.33** | 6.33** | 2.33** | 2.67** | 1.67** |  |
| Base Rate | 15% | 5% | 10% | 1% | 1% | 10% | 1% | 10% | 5% | 25% |  |
| (Critical Value, Significance) | 1.64** | 1.64** | 1.64** | 1.64** | 1.64** | 1.64** | 1.64** | 1.64** | 1.64** | 1.64** |  |
| Perceptual Reasoning |  |  |  |  |  |  |  |  |  |  |  |
| Mean PRI Subtests | – | 1.33 | 3.00 | – | – | – | 1.33 | 2.00 | 0.67 | 6.67 |  |
| Matrix Reas. (subtest-mean) | – | -0.33 | -2.00** | – | – | – | -0.33 | -1.00 | 0.33 | 1.33 |  |
| Base Rate | – | – | 25% | – | – | – | – | – | – | – |  |
| (Critical Value, Significance) |  |  | 1.92** |  |  |  |  |  |  |  |  |
| Visual Puzzles (subtest-mean) | – | -0.33 | -1.00 | – | – | – | -0.33 | -1.00 | -0.67 | -4.67** |  |
| Base Rate | – | – | – | – | – | – | – | – | – | 1% |  |
| (Critical Value, Significance) |  |  |  |  |  |  |  |  |  | 1.99** |  |

**Table 5: Longest Digit and Letter-Number Sequencing Spans with discrepancy analyses.**
|  | OpenAI | Google | Anthropic |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | GPT-3.5 | GPT-4 | GPT-4o | Gemini | Gemini | Gemini | Gemini | Gemini | Claude 3 | Claude 3.5 |  |
|  | Turbo | Turbo |  | Nano | Pro 1.0 | Adv. | Flash | Goldfish | Opus | Sonnet |  |
|  |  |  |  | (1.0 S) | (1.0 M) | (1.0 XL) | (1.5 S) | (1.5 M) |  |  |  |
| Longest Digit Span Forwards (LDSF) | 9 | 9 | 9 | 9 | 9 | 9 | 9 | 9 | 9 | 9 |  |
| LDSF Base Rate (25-29 yrs.) | (17.5%) | (17.5%) | (17.5%) | (17.5%) | (17.5%) | (17.5%) | (17.5%) | (17.5%) | (17.5%) | (17.5%) |  |
| Longest Digit Span Back. (LDSB) | 8 | 7 | 8 | 3 | 8 | 8 | 8 | 8 | 8 | 8 |  |
| LDSB Base Rate (25-29 yrs.) | (7%) | (18.5%) | (7%) | (97.9%) | (7%) | (7%) | (7%) | (7%) | (7%) | (7%) |  |
| Longest Digit Span Seq. (LDSS) | 9 | 9 | 9 | 9 | 9 | 9 | 9 | 9 | 9 | 9 |  |
| LDSS Base Rate (25-29 yrs.) | (2.5%) | (2.5%) | (2.5%) | (2.5%) | (2.5%) | (2.5%) | (2.5%) | (2.5%) | (2.5%) | (2.5%) |  |
| Longest Letter-Number Seq. (LLNS) | 8 | 8 | 8 | 0 | 8 | 8 | 8 | 8 | 8 | 8 |  |
| LLNS Base Rate (25-29 yrs.) | (6%) | (6%) | (6%) | (100%) | (6%) | (6%) | (6%) | (6%) | (6%) | (6%) |  |
| Longest Digit Span Comparisons |  |  |  |  |  |  |  |  |  |  |  |
| LDSF-LDSB (difference score) | 1 | 2 | 1 | 6 | 1 | 1 | 1 | 1 | 1 | 1 |  |
| Base Rate (All Ages) | (86%) | (64.5%) | (86%) | (1%) | (86%) | (86%) | (86%) | (86%) | (86%) | (86%) |  |
| LDSF-LDSS (difference score) | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |  |
| Base Rate (All Ages) | – | – | – | – | – | – | – | – | – | – |  |
| LDSB-LDSS (difference score) | -1 | -2 | -1 | -6 | -1 | -1 | -1 | -1 | -1 | -1 |  |
| Base Rate (All Ages) | (66.5%) | (40%) | (66.5%) | ($<$1%) | (66.5%) | (66.5%) | (66.5%) | (66.5%) | (66.5%) | (66.5%) |  |

## 4 Discussion

Results demonstrate that Generative AI models are capable of exceptional performance compared to normative human ability in key domains of cognition. First, and perhaps not surprising, all models demonstrate exceptional capabilities in storage, retrieval, and manipulation of arbitrary tokenized information such as sequences of numbers and letters while being significantly weaker in mathematical reasoning. While poor performance was most pronounced in small parameter models, the discrepancy between reasoning and information management persisted across model generations and developers indicating a generally stable discrepancy in ability.

Generative models also demonstrated exceptional verbal abilities, with variability in performance according to the size of the model. Once again, models were strongest in retrieval of stored information with consistent relative weaknesses in tasks that require understanding of linguistic concepts or the relationships between words and concepts. However, with the exception of Gemini Nano, all models demonstrated understanding as well as crystalized knowledge well above normative ability indicating that models generally excel in language-based tasks, even those requiring reasoning and understanding beyond simple regurgitation of acquired knowledge. In the case of small parameter models, they may be best for storing and retrieving information naturalistically (e.g., in the context of hardware constraints) but may not be capable of understanding or manipulating that information to solve a problem.

Finally, the dramatically poorer performance on visual processing tasks indicates that generative models, as they stand today, have profound deficits in the ability to understand the meaning or relationship in visual representations. Across developers and versions, models could not understand the meaning of objects, reason, problem-solve, or detect abnormal patterns in visual representations. There is some indication that current modeling approaches can lead to the acquisition of these abilities as Claude 3.5 Sonnet demonstrated profound increases over the previous generation (Claude 3 Opus). Claude 3.5 Sonnet showed advances over its predecessor in Matrix Reasoning (0.1^th%ile vs. 25^th%ile), which measures the ability to detect meaningful patterns in visual stimuli and Figure Weights which indexes the ability to understand and reason on mathematical relationships that are visually presented (0.1^th%ile vs. 50^th%ile). No such improvement was observed in visual puzzle solving or image completion tasks that test the ability to understand and make sense of visual information.

Given the relative complexity of visual information processing compared to auditory processing in both humans and other vertebrates that rely on complex visual capabilities, the gap between visual and auditory cognitive capabilities in generative AI might not be addressed through incremental improvements in architecture or model training but may require separate specialized architecture for visual and auditory processing with enhanced interaction capabilities, as is the case with vertebrates.

The above results clearly demonstrate that for current SoTA models the Positive Manifold, which posits positive correlations between different cognitive capabilities, holds when VCI and WMI are considered (as demonstrated in Ilić and Gignac (), and fails to hold for when including PRI as well.

The current work presents with the significant limitations that the model parameters themselves (underlying training data, parameter count, tuning approach) are all proprietary across all examples and thus not available for comparative analysis. While this does not limit the validity of the tests as these parameters are not known with human subjects either, it does limit the ability to draw definitive conclusions about factors that influence performance. This hampers scientific inquiry into GenAI models as model organisms to understand the acquisition of cognitive capabilities. The study is further limited by the inherently non-standard approach to WAIS-IV administration. While we attempted to copy key testing conditions and excluded tests that could not conform, there is an inherent limitation in the difference in testing setup from that which the scores were normed on. Despite these limitations, the current work represents the first of its kind approach to benchmark GenAI against human norms of intelligence. The results demonstrate the unique cognitive capabilities of current generative AI models as well as relative weaknesses, while providing a path to advance discrete domains of cognition that underlie wide sets of human cognitive abilities.

## Appendix A Broader Impact

The development of benchmarks help to assess the performance of foundation models. Capable models need to be developed responsibility and with attention to their strengths and flaws. Leveraging knowledge and tools from disciples such as clinical and neuro-psychology has allowed us to develop a set of grounded tasks that shed insight on the functioning of LLMs that are complementary to existing public benchmarks. However, we must acknowledge that these tasks were designed specifically for evaluating human cognitive functioning and therefore extrapolation of the results to performance on more mundane, real-world tasks and conclusions that compare language model abilities to human cognitive functioning need to be treated with care.

## Appendix B Prompt Examples

Note: Test is slightly altered to protect copy-written materials.

## Appendix C Visual Tasks Examples

Figure: Figure 2: Example of the Matrix Reasoning test in which subjects are requested to identify a pattern between different rows and columns of the matrix. In this example, the first row is composed of green square, followed by a red triangle (from left to right). The second row also starts with a green square meaning that the correct option in the question mark is a red triangle. Answer: 2
Refer to caption: extracted/5908197/matrix_reasoning_example.png

Figure: Figure 3: Example of the Visual Puzzles test in which subjects are requested to identify all the pieces that compose the provided design. In this example, the provided design is a three color rectangle. It can be composed by arranging rectangle (2), rectangle (1) and flipping rectangle (4), from left to right. Answer: 1,2,4.
Refer to caption: extracted/5908197/visual_p_example.png

Figure: Figure 4: Example of the Figure Weights test in which subjects are requested to determine the correct way to balance the scales by finding the missing pieces. In this example, the scale on the left holds a single red circle, while the scale on the right holds two red circles. Considering that each red circle carries the same weight, attaining balance necessitates the addition of one red circle to the scale on the left. Answer: 1.
Refer to caption: extracted/5908197/fw_example.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: none
soundness: 3
strengths: properly done study, well written paper
summary: In the paper, the authors test current LLMs by OpenAI, Google, and Anthropic for cognitive capabilities according to the WAIS benchmark. The study is properly done and the paper is written well.

However, the current version of the paper provides too little novelty and significance in insight to warrant a publication in ICLR.
weaknesses: The numerical results for storage, retrieval, and manipulation are indeed not surprising and logical regarding the hardware capabilities pattern completion capabilities and vast representation space of the LLMs. Here, the paper could have made a difference in 1) identifying details through quantitative analysis, and 2) testing with other, more recent benchmarks on intelligence.

Furthermore, shortcomings of LLMs in visual processing/reasoning are not surprising either, as the tested models were either explicitly trained on uni-modal (text data) only, or we have no insight into training (but reasons to believe that). Here, the paper could have contributed in 1) testing  Multimodal LLMs as well, and 2) comparing between the tested LLMs to identify where qualitative differences lay in detail.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: How do you know the test items (or very similar items) aren't in the models' pre-training data? 

"predicting the write answer than understanding and reasoning" -- write --> right

"the gap between visual and auditory cognitive capabilities in generative AI" -- do you mean language instead of auditory?
soundness: 2
strengths: The paper is well-written and the experiments are explained clearly.
summary: This paper tests several generative AI models on different parts of the Wechsler Adult Intelligence Scale test, including verbal comprehension, working memory, and perceptual reasoning.  The top models excel on verbal comprehension and working memory tests but perform much worse on perceptual reasoning tests.
weaknesses: The paper lacks critical discussion regarding using tests designed to test human intelligence on LLMs.  The only such discussion is in the appendix (in the "Broader Impacts" section).  I think that discussion needs to be at the very beginning, and the authors need to argue why this kind of evaluation for an LLM gives useful information about its more general capacities.   Given the many problems that have been identified with LLM benchmarking using tests designed for humans, what can we learn from this particular evaluation? As one part of this discussion, the authors should address the question of whether the test items they presented to LLMs are in those LLMs pre-training data.  

I was particularly confused by the "working memory" tasks.  For humans, these are given in an audio format, since it would not make sense to give such tests to them in a written format that they have access to while answering, since that would not test working memory.   But it seems here that the LLMs get these tests in their context window, and have access to them while answering?  I didn't understand what this is a test of, with respect to LLMs.  

At the end, the authors state that their results "demonstrate that Generative AI models are capable of exceptional performance compared to normative human ability in key domains of cognition."   This seems like a fallacy to me, since there is not yet evidence that performance on these tests for LLMs is indicative of broader performance in "key domains of cognition".  

The paper lacks citations to other work that evaluates LLMs on IQ tests, e.g., 

Wasilewski & Jablonski: Measuring the Perceived IQ of Multimodal Large Language Models Using Standardized IQ Tests

Huang & Li: Measuring the IQ of Mainstream Large Language Models in Chinese Using the Wechsler Adult Intelligence Scale

King: Administration of the text-based portions of a general IQ test to five different large language models

Lenhard & Lenhard: Beyond Human Boundaries: Exploring the Proficiency of AI Technology and its Potential in Psychometric Test Construction

(And there are probably other such papers in the recent literature.) 

The paper also lacks any reference to critical discussion of giving tests designed for humans to LLMs, e.g., 

https://www.aisnakeoil.com/p/gpt-4-and-professional-benchmarks

https://www.science.org/doi/10.1126/science.adj5957

---

confidence: 3
contribution: 1
initial_rating: 3
presentation: 3
questions: What do you see as the main contribution to the machine learning community? Could this work have implications for how we perform evaluations or how we can develop new models?

How large is the dataset? You mention that all answers were scored by clinical psychologists, so I would assume that it must be quite small to be feasible.
soundness: 3
strengths: * The study addresses a question that is of broad interest, namely, how state-of-the-art models compare in performance to humans on cognitive tasks. 
* The evaluation is original in the sense that it is grounded in an actual IQ test that is used on humans and it is performed by trained psychologists. 
* It is interesting and noteworthy that the models perform so poorly on the perceptual reasoning tasks. 
* The paper is clearly written and easy to follow.
summary: This study performs a benchmark evaluation on the cognitive abilities of state-of-the-art language models and vision language models. It uses the Wechsler Adult Intelligence Scale (WAIS-IV) which is a test developed for humans that incorporates several aspects of intelligence, such as working memory, perceptual capabilities, and verbal comprehension. The findings include that the models perform well on working memory tasks—greater or equal to the 99.5th percentile of human performance. The performance was also high on verbal comprehension, but surprisingly low on perceptual reasoning.
weaknesses: While this is an overall sound study of how state-of-the-art generative models compare against humans on intelligence tests, I am unsure of its value as a machine-learning paper. From what I can assess, it does not contribute any new methodology, evaluation framework, or insights that could significantly advance the state of knowledge in the field. It is definitely interesting to benchmark these models against human performance, but there exists much literature on this topic already (little of which is cited here). See, e.g., Mahowald et al. (2023) for a review. It is thus unclear what this paper contributes beyond what is already in the literature. Moreover, existing work I am aware of tends to focus on specific linguistic or cognitive phenomena, which gives more concrete insights into where models may or may not fail. These things are important to study so that we can develop better evaluations, understand how LLMs fare as cognitive models in various regards, possibly inject inductive biases etc. The approach here, on the other hand, is more general. That makes it difficult to understand the implications of the results, especially for someone unfamiliar with the WAIS-IV framework.

Another weakness I see is that several decisions are made without rationales or citations. For example l145 “as GenAI models … often request repetition without responding to the actual assessment being conducted” or l182 “we selected the norms for age 25 years to 29 years and 11 month”. 

In addition, the present study only uses closed-source models, with some details missing on the experimental setup, like the exact model snapshot used and the decoding method.

----

Kyle Mahowald et al. 2023. Dissociating language and thought in large language models.

---

confidence: 4
contribution: 1
initial_rating: 5
presentation: 3
questions: - Can you better characterize novelty of your work in light of similar existing works?
soundness: 3
strengths: - The use of the WAIS-IV provides a familiar and well-established framework for comparing AI capabilities to human performance. This offers a more relatable and interpretable assessment compared to typical machine learning metrics.
    
- The methodology is robust and well presented.
summary: The paper attempts to measure the cognitive abilities of various large language models (LLMs) and vision-language models (VLMs) using a modified version of the WAIS-IV, a standardized intelligence test for humans. The authors find that while models perform exceptionally well on tasks involving verbal comprehension and working memory, they exhibit significant deficits in perceptual reasoning.
weaknesses: - The main premise of the work is centred around testing LLMs using traditional IQ measures like WAIS test and there are several works which do similar analysis: 
https://www.techrxiv.org/doi/full/10.36227/techrxiv.171778886.6283965 https://www.techrxiv.org/doi/full/10.36227/techrxiv.171778886.62839657
https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0307097
https://www.techrxiv.org/doi/full/10.36227/techrxiv.22645561.v1
https://www.researchgate.net/profile/GuozhengRao/publication/334844361_How_Well_Do_Machines_Perform_on_IQ_tests_a_Comparison_Study_on_a_Large-Scale_Dataset/links/5dd279b94585156b351c1774/How-Well-Do-Machines-Perform-on-IQ-tests-a-Comparison-Study-on-a-Large-Scale-Dataset.pdf
Such works should be referred/ discussed and the novelty of the current work should be highlighted.
    
- One of the central findings of the paper that LLMs do well in language-based tasks while struggling with visual reasoning - is already well-established in the field.  For instance, Cao et al. (2024) directly address the "visual cognition gap" between humans and multimodal LLMs, a key finding echoed in this paper (https://arxiv.org/abs/2406.10424 )
    
- There also exists work on analysing working memory capacity of LLMs (https://ojs.aaai.org/index.php/AAAI/article/view/28868) and more discussion needs to be done to link to such works.
    
- The paper mostly uses all closed source models and it might be more interesting to use some open source models (Llama, Mistral etc) across different sizes for better understanding how trends of performance change.

Meta-review: This paper benchmarks modern language and vision-language models against human performance using the WAIS-IV intelligence test framework. While the study is well-executed and clearly presented, all reviewers identified significant limitations in novelty and contribution. The findings that models excel at verbal/memory tasks but struggle with visual reasoning largely confirm known capabilities. Critical weaknesses include: insufficient discussion of using human-designed tests for AI evaluation, limited citation of similar prior work, lack of open-source model comparisons, and absence of methodological innovations. The study provides incremental insights without advancing the field's understanding of AI capabilities or evaluation methods. Given these limitations and the authors' non-response during the rebuttal period, the paper falls below the acceptance threshold.

Decision: Reject