KiVA: Kid-inspired Visual Analogies for Testing Large Multimodal Models

## Abstract

Abstract This paper investigates visual analogical reasoning in large multimodal models (LMMs) compared to human adults and children. A “visual analogy” is an abstract rule inferred from one image and applied to another.
While benchmarks exist for testing visual reasoning in LMMs, they require advanced skills and omit basic visual analogies that even young children can make. Inspired by developmental psychology, we propose a new benchmark of 1,400 visual transformations of everyday objects to test LMMs on visual analogical reasoning and compare them to children and adults. We structure the evaluation into three stages: identifying what changed (e.g., color, number, etc.), how it changed (e.g., added one object), and applying the rule to new scenarios. Our findings show that while models like GPT-4V, LLaVA-1.5, and MANTIS identify the “what” effectively, they struggle with quantifying the “how” and extrapolating this rule to new objects. In contrast, children and adults exhibit much stronger analogical reasoning at all three stages. Additionally, the strongest tested model, GPT-4V, performs better in tasks involving simple visual attributes like color and size, correlating with quicker human adult response times. Conversely, more complex tasks such as number, rotation, and reflection, which necessitate extensive cognitive processing and understanding of the 3D physical world, present more significant challenges. Altogether, these findings highlight the limitations of training models on data that primarily consists of 2D images and text.

## 1 Introduction

Figure: (a) Visual analogy domains.
Refer to caption: x1.png

What is visual cognition? Humans make countless visual inferences daily from observing objects and scenes and quickly notice visual changes. We generalize common hypotheses about changes from different observations and use these insights to solve new problems. If one puts a wool sweater in the washing machine and it comes out smaller, one might infer that the wash shrinks sweaters and avoid washing wool coat in the future. If cookies disappear, one might infer that someone is eating our treats and and proceed to hide the chocolate elsewhere. This ability to draw parallels between situations and apply learned patterns to a new scenario is known as analogical reasoning. Analogical reasoning is a hallmark of human intelligence and learning [, , , ]. It is what enables us to be flexible, adaptive and robust learners across a wide variety of settings, making out-of-distribution generalizations and responding accordingly to meaningful patterns [, ].
Analogical reasoning is already available to young children [, , ], and is crucial for problem-solving in various contexts, from building scientific models to appreciating metaphors to formulating legal arguments.

Today, we witness much progress in large multimodal models, but these models remain data-hungry and still require substantial human involvement to adapt to new contexts [, ]. As analogical reasoning is instrumental for general-purpose and adaptive machines, it is crucial to examine whether current models have such capabilities. Critically, examining analogical capabilities does not rely on external prior knowledge, nor does it permit models to “cheat” by merely depending on their training data. Reasoning about analogies involves first recognizing relationships between object characteristics and quantifying the similarities and differences, then recognizing the same relationship in a new set of objects. In this paper, we focus on visual analogies, which evaluate the ability of models to engage in abstract reasoning based on visual observations. See Figure 1 for a summary of our visual analogical reasoning benchmark and results.

There is a growing body of work examining visual analogical reasoning capabilities in large multimodal models [, , , , ]. Existing benchmarks of visual analogies include (a) ConceptARC [, ], (b) variations of Raven’s Progressive Matrices [] and (c) abstract spatial reasoning [] (see prior benchmarks in Figure 2). These prior benchmarks all have several critical limitations. First, they rely on abstract shapes and grids, lacking real-world relevance. This abstraction of stimuli neither aligns with the training data of large multimodal models nor effectively mimics the complexity and variability found in everyday visual tasks, making it less suitable for assessing how well AI models can perform analogical reasoning in practical contexts. Second, the transformations examined involve conjunctions of visual concepts such as counting and ordering and extracting pixels according to some arbitrary rule, which does not tap into basic visual cognition. Humans do not require the ability to solve these specific tasks to function effectively in their daily lives nor to demonstrate their capacity for visual analogical reasoning. Third, while we know that models often perform poorly on these benchmarks, where they fail in the reasoning process needs to be clarified since existing evaluations focus solely on prediction accuracy rather than the reasoning approach or what is perceived.

Figure: (a) Prior benchmarks.
Refer to caption: x3.png

We propose a Kid-inspired Visual Analogies (KiVA) benchmark founded on developmental psychology (See Figure 1(left)). We focus our analysis on basic visual analogical capabilities that are present early in human development and are important for understanding the physical world. In particular, our benchmark isolates the following fundamental capabilities:
detecting changes in color  [, ] and size  [, ], changes that involve rotation and reflection  [, ], and changes in small numbers of objects  [, ]. Our benchmark stands out in the following ways:

First, our dataset utilizes real-world objects curated from established 3D object datasets [, ] and aligns closer to the training distribution of computer vision models and visual data of humans. Our benchmark includes physically grounded visual changes of these real-world objects. This approach differs from existing abstract reasoning datasets (see Figure  2 for comparison).

Our second key approach is inspired by developmental psychology, specifically how children learn to perform analogical reasoning not abstractly, but from simple objects in grounded contexts [, , ]. We propose a similar approach for large multimodal models, investigating if they can perform similarly to children on basic visual analogical reasoning capabilities related to color, size, orientation, and number as reported in []. This approach, starting with simple, real-world relevant tasks in child development, allows models to develop robust reasoning abilities before tackling more advanced tasks, providing a clearer pathway for evaluating and improving cognitive functions in AI.

Third, we break down our evaluation to examine the different steps involved in analogical reasoning to pinpoint which steps a model can perform and where it may fail: 1) detecting if there is a general change in an object for a particular visual domain, 2) inferring the specific change rule within the correctly identified size, color, etc.  and 3) extrapolating the inferred rule to a new example. This three-stage evaluation (Fig. 3) gives us insights into models’ reasoning processes beyond simply selecting a correct or incorrect response at the end.

Results from KiVA demonstrate that visual analogies are not entirely solved by the large multimodal models we tested, GPT-4V [], LLaVA-1.5 [] and MANTIS []. These models do not match even the capabilities of a three-year-old child (See Figure 1(right)). While LMMs can detect some forms of object transformations, they cannot make extrapolations about those transformations on new objects. In particular, GPT-4V outperforms LLaVA-1.5 and MANTIS but also demonstrates weaker performance in orientation and number changes than in size and color changes which are also processed more quickly by humans, processed at an earlier age (e.g., [, ]), and at a more primary region of the visual cortex (e.g., [, ]).

Finally, we include a more challenging benchmark, KiVA-adults (dataset and results described in Appendix C), which includes 2,900 transformations requiring deeper generalization from training examples. These transformations are solvable by adults but not by children under seven years old, providing the next benchmark for models to surpass after KiVA.

## 2 Related Work

Evaluating human visual analogical reasoning. There are a variety of tasks designed in Developmental Psychology to examine human visual analogical reasoning early on in life. Children are asked to compare simple object and relational matches [, , ] along dimensions such as color [, ], number [, ], size [, ] and spatial orientation [, ]. Older children and adults are evaluated on Raven’s Progressive Matrices (RPMs) [, , ] and Bongard Problems [, ]. Even though they tend to be the most representative and largest testbeds for testing advanced visual analogical reasoning, RPMs and Bongard problems utilize abstract geometric shapes and test recognition of arbitrary patterns that (1) cannot be solved by young children before the age of 6 and (2) are not critically relevant to visual processing in everyday life.
We are the first to provide a visual analogical reasoning benchmark that includes common real-world objects and more natural visual cognition skills such as counting and spatial transformations (reflections and rotations)—a task that even a young child can handle []. We also examine where people and models
may fail by breaking evaluation down into finer steps.

Evaluating visuo-linguistic reasoning in AI models.
Several proposals for evaluating modern AI systems’ visuo-linguistic reasoning capabilities followed the recent successes of large multimodal models. Many concentrate on a narrow, isolated set of tasks for detecting object properties and transformations. These tasks include size estimation [, ], color perception [, ], counting objects [, ], spatial reasoning in terms of object viewpoint/pose and chirality [, , ] and visuo-linguistic compositionality [, , ]. Typically, the objective of these tasks is to evaluate models’ ability to report a correct property about objects in an image, but they lack the depth to probe the pattern abstraction and generalization involved in visual analogical reasoning. They focus on only detecting narrow and superficial object properties, and fall short in testing robust understanding.

Broader benchmarks, such as visual question answering setups [, ], attempt to investigate the models’ understanding of various visual concepts. One approach taken by [, ] was to try and push the envelope on various tasks to capture anecdotal and qualitative observations regarding the performance of GPT-4. Perception Test [] proposed a second approach: a visual video-based benchmark including developmentally-inspired tasks such as object permanence, object tracking, spatial relations, etc. Recently, the BLINK benchmark was introduced to show that core visual perception tasks, easily solvable by humans "within a blink," remain challenging for large multimodal models due to their resistance to language-based mediation  []. However, all these benchmarks fall short in evaluating the deeper, more complex aspects of visual analogical reasoning and generalization.

Another specific class of benchmarks tests generalization and reasoning within 2D puzzle grids. These include Abstraction and Reasoning Corpus (ARC) [, , ]; a direct translation of RPMs-based human evaluation has previously been applied to models by [] and [] (also see prior benchmarks (b) and (c) in Figure 2). However, the stimuli are simple, monotonic shapes like squares and circles, lacking real-world complexity and variability. Moreover, they primarily focus on abstract pattern recognition and logical sequencing without incorporating real-world contexts, thereby neglecting more fundamental visual cognition domains that three-year-olds are capable of.

In summary, although numerous benchmarks assess advanced visual capabilities in large multimodal models, none evaluate abilities that are demonstrably present in young children, such as drawing analogies about transformations of simple real objects. Developing a curriculum that gradually increases in complexity and abstraction, starting with benchmarks that young children can solve, ensures that models build foundational visual reasoning skills before tackling abstract and complex tasks, mirroring the natural progression of human cognitive development.

## 3 The KiVA Benchmark for Visual Analogical Reasoning

We introduce the kid-inspired visual analogies benchmark, wherein real-world objects undergo common transformations necessary for everyday visual cognition. We focus on isolating and testing basic visual transformations that even a three-year-old child understands []. As we show in Figure 1, we examine noticing color changes [, ], size changes [, ], rotation, reflection [, ], and number changes such as addition and subtraction of a small number of objects [, ]. The benchmark is available at [https://github.com/ey242/KiVA](https://github.com/ey242/KiVA).

Figure: Figure 3: An example of a trial in KiVA. A transformation $T_{given}$ is first given to models and humans to observe. We then evaluate them on cross-domain change detection (left). Suppose the correct domain of transformation $D_{cross}$ is identified (as indicated by the green arrow). In that case, humans and models are further evaluated on their within-domain change detection $D_{within}$ (middle) and then on visual extrapolation $E$ (right). Otherwise, for incorrect cross-domain detection (yellow arrow), humans and models skip to make a visual extrapolation.
Refer to caption: x5.png

### 3.1 A Three-Stage Experimental Paradigm

We use our proposed dataset to benchmark computational models’ and human subjects’ visual analogical reasoning capabilities. Critically, we utilize the same testing paradigm for both kinds of subjects. In each trial, we start by presenting a given transformation $T_{given}$ of an object that changes by a specific rule, following the experimental paradigm of other analogical reasoning benchmarks for humans and computational models [, , ]. We then evaluate the subject’s ability to determine what changed, or the domain of transformation $D_{cross}$, how it changed, or the specific transformation within the correct domain $D_{within}$, and the outcome of a new object that undergoes the same transformation—i.e., an extrapolation $E$. We break the question down into these three steps to test the different cognitive processes involved in analogical reasoning. The last step represents the core visual analogy task, while the first two assess the necessary prerequisites for accurate analogical reasoning. We pose all questions in a multiple-choice format, which enables automatic scoring. All multiple-choice options include “Doesn’t apply” to accommodate responses that the provided choices may not cover. Please refer to Figure 3 for the three-stage query pipeline and Appendix A.2 for more details of the specific prompts used.

Cross-domain change detection (“what”). We first evaluate if the subject can detect what changed in the given transformation $T_{given}$ and attribute these changes to the relevant visual domain across all domains $D_{cross}$, such as identifying a change in size as opposed to number and orientation (see Figure 3). We randomly sample incorrect multiple-choice options from other possible change domains. “No change” and “Doesn’t apply” are also included as options to accommodate for alternative model reasoning. Suppose the model fails to identify basic changes, such as distinguishing a change in number from a color change. In that case, it is pointless to advance to predicting how new objects will change based on the given transformations. This limitation highlights the inadequacy of existing visual analogical reasoning benchmarks [, , , ], which focus solely on advanced predictions without ensuring fundamental change detection capabilities.

Within-domain change detection (“how”). If a model or human detects the correct cross-domain change $D_{cross}$, we ask them further to identify the specific change within the correct visual domain $D_{within}$ (see green arrow in 3). This step is crucial because it ensures the model or human can accurately specify the rule governing the change before extrapolating to a new object. If they fail to identify the specific change, any attempt at extrapolation would likely be incorrect. By pinpointing where reasoning fails, we can better understand models’ and humans’ limitations and improve analogical reasoning capabilities in models. We randomly sample the incorrect multiple-choice options from other possible specific changes within $D_{cross}$.

Visual Extrapolation. Finally, we proceed to the step captured by other benchmarks: presenting a new image and asking the model to extrapolate how it will change based on the previously identified transformation (see Figure 3 and other extrapolation examples of other visual domains in Appendix A.1). We ask models to visually extrapolate independent of their performance in verbal change identification to account for the possibility that models may engage in visual analogical reasoning separately from verbal reasoning and can, therefore, perform well in visual tasks even if they struggle with the prior verbal descriptions. This approach helps us determine if a model’s visual reasoning can function independently of its verbal reasoning skills. It provides a more nuanced evaluation of its cognitive capabilities and identifies specific areas for improvement.

### 3.2 A Dataset of Visual Analogies

We create a dataset of stimuli using naturalistic everyday objects that better represent the characteristics of real-world visual data and better match the training data of computer vision models (and humans). We take 3D models of household objects from [] and objects commonly encountered by infants and children from []. To set up the dataset, we perform five basic visual transformation domains: changing the size, color, and number of objects, rotating and reflecting the objects along different axes (see Figure 1 for the transformation domains examined). We select these five types of object transformations because they are crucial for object and scene recognition (e.g., [, ]), scene segmentation (e.g., []), and detecting significant changes in the environment [, ]. Other visual properties, such as depth [], spatial compositionality [, ], and physical affordances [, ] are also crucial for such purposes; however, the five transformations in our benchmark have been shown to be developmentally plausible for analogical reasoning in young children, as examined in [], and are therefore prioritized. Below, we outline the five visual transformation domains, each consisting of subdomains that specify an object transformation. There are 100 object transformations for each subdomain of transformation, totaling 1,400 object transformations for the 14 subdomains.

Color changes.
Noticing color changes can signal alterations in an object’s state or presence, which is essential for tasks like identifying ripe fruit or detecting hazards []. In KiVA, the general transformation rule for color is that input objects change to a single color as in []. The subdomains of color examined are red, green, and blue.

Size changes.
Size perception allows individuals to understand and interact with their environment accurately, guiding tasks like identifying objects, planning actions, navigating spaces, and avoiding obstacles  []. In KiVA, objects undergo transformations in two subdomains: they turn bigger or smaller (in both height and width) as in [] by a factor of 2.

Rotation.
Mental rotation is the ability to recognize and map different views of the same object  []. This is essential for identifying objects despite changes in perspective, which is vital for navigation, object manipulation, and spatial orientation []. KiVA takes inspiration from psychometric studies probing mental rotation in humans (e.g., [, ]), featuring object rotation in 2D space by the subdomains of $90$ degrees (clockwise or counterclockwise) or $180$ degrees.

Reflection. Like rotation, recognizing reflections is essential for spatial orientation and precise object identification. This ability aids in understanding object symmetry and chirality, essential for distinguishing left and right shoes or gloves, and recognizing molecular isomers in organic chemistry, for example []. Chiral objects cannot be rotated and translated into alignment with their reflection, so there is a visual difference between the object and its reflection []. We reflect chiral objects in two subdomains: along the x-axis or the y-axis as in [].

Number changes.
Accurately monitoring and comparing quantities is essential in various fields like economics and science; it is also important in daily life activities like shopping, cooking, and managing resources effectively, as in caching and rationing [, ]. Transformations in this domain reflect basic mathematical operations over the number of objects in an image. Subdomains of number transformations include addition $(+1,+2)$ and subtraction $(-1,-2)$. We restrict the number of objects in an input or output image to under 8.

## 4 Comparing Analogical Reasoning in LMMs and Humans

Evaluating Large Multimodal Models. We test several LMMs: 1) GPT4-V (OpenAI API model: gpt-4-vision-preview) []: an extension of the language-only GPT-4 [] incorporating computer vision capabilities, 2) LLaVA-1.5 []: an open-source model that integrates a vision encoder with a language model, specifically designed to enhance general-purpose visual and language understanding, 3) MANTIS [] which builds on modified architectures from notable models like LLaVA to support interleaved multi-image input. We combine the given transformation with the choices of new object transformations at the extrapolation step into a single composite input image for LLaVA-1.5, which is limited to processing a single image, but present these as separate images to MANTIS, which is fine-tuned to manage interleaved multi-image conversations. We evaluate GPT-4V under both multi-image and single-image presentations.

We randomize each experiment over three seeds and run each trial (Figure 3) on a model three times. We score correct choices as 1 and incorrect choices as 0. We calculate the mean score across its three seeds. Subsequently, to evaluate the performance for each transformation domain, we calculate the overall mean and standard error for the average scores of all averaged trials within each domain. GPT-4V, LLaVA-1.5, and MANTIS complete the entire benchmark, featuring 1,400 transformations. Open-source models ran on an A6000 48 GB single GPU for under 12 hours.

Evaluating Humans. A corresponding visual analogies task, developed using JsPsych [], was administered to two groups of human participants. All methods were approved by IRB (protocol 2020-10-13755) prior to testing both child and adult participants.
We recruited 40 adults (aged 21 to 35 years) on Prolific [] to complete a random subset of 14 trials (we randomly sampled 1 trial from each of the 14 transformation subdomains), leading to a total of 560 responses. We recruited 20 children (aged 3 to 7 years, $mean$ = 4.6 years, $se$ = 0.21 years) from local early childhood centers and ChildrenHelpingScience [] to complete a random subset of 10 trials (we randomly sampled 2 trials for each of the 5 transformation domains), leading to a total of 200 responses. Human participants received the same text and image prompts as the models without feedback, but we informed them that rewards were contingent on their performance. Adults were paid at least $19/hour with a bonus of $0.01 per correct response, while children received stickers based on their performance.

Mirroring model evaluation, we calculate humans’ the average scores and standard errors across trials within each transformation domain. The standard error reflects the variation in performance across trials completed by all participants, not variations between human participants.

## 5 Results

Figure: Figure 4: LMM Performance. Overall, LMMs can detect transformations and identify the general visual domain of the transformations (“Cross Domain” in blue). However, performance generally declines when the models are asked to pinpoint the exact transformation observed within the correctly identified visual domain (“Within Domain” in orange). Performance for visual extrapolation declines even more, as illustrated by the green bars labeled “Extrapolation.” Additionally, the models’ performance is domain-dependent. Error bars represent standard errors of performance across object variations within the same transformation.
Refer to caption: extracted/5753910/figures/SimpleModels1.png

Figure: Figure 5: Human adults (left) and children (right) performance. Both children and adults show robust performance across “Cross Domain” (blue) and “Within Domain” (orange) change detection as well as “Extrapolation” (green). Humans perform equally well across domains, with rotation proving slightly more difficult for children. Error bars represent standard errors of performance across object variations within the same transformation.
Refer to caption: extracted/5753910/figures/SimpleHumans1.png

Models get worse with increasing reasoning complexity, unlike humans. Overall, LMMs can detect transformations and identify the general visual domain of the transformations (e.g., color vs. size), as indicated by the blue bars labeled “Cross Domain” in Figure 4, with GPT-4V and MANTIS generally outperforming LLaVA-1.5. GPT-4V and MANTIS outperform human children in recognizing when an object changes color or spatial orientation (rotation and reflection). However, performance generally declines when the models are asked to pinpoint the exact transformation observed within the correctly identified visual domain (e.g., becoming bigger or smaller if size is the correctly identified domain), as reflected by the orange bars labeled “Within Domain”. Performance for visual extrapolation declines even more, as illustrated by the green bars labeled “Extrapolation”. In other words, models’ success in verbally describing transformations does not guarantee their success in extrapolation. We may attribute part of the models’ failure in analogical reasoning to an inability to correctly recognize the given transformation $T_{given}$. However, another part of the model’s failure lies in extrapolating the correctly identified transformation to a novel object and predicting the corresponding outcome. By contrast, children and adults show robust performance across cross-domain and within-domain change detection and extrapolation, as shown in Figure 5. They can correctly describe the transformations involved in $T_{given}$ and extrapolate the visual transformations to new objects of novel input values. Specific numerical results can be found in Table 1 in Appendix B. We also observe a similar pattern of results for KiVA-adults (see results of adults and models in Appendix C.2).

Model performance depends on the visual domain. Overall, models are better at detecting and describing color transformations than transformations in other domains. GPT-4V performs better at detecting and extrapolating color and size transformations, which involve more discrete and local processing than the other domains [, ]. However, LLaVA-1.5 and MANTIS do not perform well on generalizing size transformations relative to other types of transformations, even when all the possible relative sizes of objects are presented to LLaVA in a single image. Overall, all three models are less able to tell what changed within the visual domains of rotation, reflection, and number, as demonstrated by the orange bars labeled “Within Domain” in Figure 4, and consequently also did not perform well in extrapolations for those domains. In contrast, children and adults generally perform well across all visual domains in Figure 5, with children performing slightly worse on rotation, suggesting greater difficulty in appreciation of spatial orientation compared to other domains.

Figure: Figure 6: Mean error score of GPT-4V is positively correlated with mean response time of adults in within-domain transformation identification (left) and visual extrapolation (right). Each dot is color-coded by transformation domain and labeled by subdomain). In subdomains where adults took longer to respond correctly, GPT-4V also tended to make more errors, $r$ = .78, $p$ = .0011 for within-domain and $r$ = .63, $p$ = .017 for extrapolation responses.
Refer to caption: x6.png

Interestingly, even though human adults perform about equally well in all visual domains, their response time differed by the domain in a pattern that corroborates well with GPT-4V’s performance pattern across the domains, as demonstrated in Figure  6. Adults take longer to identify the within-domain transformation and make visual extrapolations about rotations and reflections; GPT-4V likewise shows higher error scores in those domains. This suggests that what might be cognitively demanding to humans is perhaps more challenging to process computationally for GPT-4V.

In-context learning and prompt engineering did not improve model performance.
We explore whether we can attain better model performance through careful prompt engineering, which has shown promising results on various tasks  [, ]. We consider four different prompt engineering methods: 1) Reasoning through code []: We first prompt the model to generate code snippets describing each transformation in the task, then rephrase the task question to incorporate the generated code. 2) Reasoning after Reflection: We ask the model to reflect on its answers two times for each question in the task. 3) Reasoning through instruction: inspired by [], which shows that chain-of-thought reasoning is more effective on several benchmarks, we prompt the model to generate step-by-step instructions on how to answer each question, then use the instructions to generate an answer. 4) In-Context Learning []: We prompt the model with two randomly sampled examples with solutions for each concept before displaying the task. Apart from text prompt engineering, we experiment with different visual prompting for LLaVA-1.5. Recent works [, , ] show that visual model performance is sensitive to the altering of color and size of the visual input. We apply two visual prompting approaches: 1) Color: we alter the image background color (initially transparent) into black and white []. 2) Size: We apply a center crop to the images, varying the image size between 0.9 and 1. Surprisingly, we see that performance with these approaches remains the same, which points to the challenging nature of our benchmark. Refer to Appendix A for details.

## 6 Discussion

Despite the scale of image and text data used to train GPT-4V, LLaVA-1.5, and MANTIS, these models fail to reason about visual analogies in the way humans do from a young age. GPT-4V outperforms the other models but still does not approach child performance. It performs better in transformations related to color and size than those related to number and spatial orientation, but even then, it shows a substantial decline in performance with increasing reasoning complexity leading up to visual extrapolation, whereas humans do not show much decline from generalizing changes to making extrapolations about new objects.

Unlike simply observing discrete changes such as color or size, perceiving reflection, rotation, and numerical changes requires actively engaging with the environment: determining number changes involves sequentially keeping track of numbers, whereas reflection and rotation necessitate visualizing and mentally manipulating objects in space. These tasks are not only more cognitively complex but are also more deeply grounded in interactions with the 3D physical world rather than relying on 2D image-text correlations, which are often used in training LMMs. Furthermore, compared to spatial and numerical transformations, changes in size and color tend to be processed much earlier in the visual pathway of the brain [, ] and in human development [, , , ]. Adults take longer to process visual analogies related to changes in number and orientation, suggesting that it may be more cognitively challenging. We do not find a correlation with children’s reaction times as there is great variability in response times given the age range that we recruited (3 to 7 years old) [], but we also observe that children are less accurate in reasoning about rotation relative to other domains.

Limitations. Our benchmark is designed to assess basic visual change detection and analogical reasoning capabilities commonly studied in children by cognitive psychologists []. It aligns with the documented capabilities of young children, as evidenced by the successful completion of our task by individuals as young as three years old through randomized sampling from the larger set of trials. We observe that LMMs perform worse than human participants and that there is no marked improvement through in-context learning and visual or textual prompting techniques. However, this outcome should not be hastily categorized as a null result. Further research is necessary to explore practical strategies for improving model performance. Furthermore, our benchmark is far from capturing the full spectrum of visual cognition. However, it represents a foundational effort to systematically evaluate visual analogical reasoning fundamentally and groundedly using everyday real-world objects and established principles from developmental and cognitive psychology.

## 7 Conclusion

Our findings suggest that large pretrained multimodal models are still less capable than humans in visual analogical reasoning. While these models can identify changes in images, their performance diminishes when tasked with verbally identifying domain-specific changes and further declines when asked to extrapolate these identified changes to a new, unseen object. Among the three models evaluated, GPT-4V performs the best, particularly in tasks involving straightforward visual analogies, such as changes in color and size. However, it struggles with more challenging analogies that require a greater understanding of the 3D physical world, such as numerical and spatial changes. These rely less on simple image-text correlations and more on deeper spatial and contextual understanding. In contrast, humans, including young children, can recognize and interpret a wide range of object relations and transformations, as noted in existing studies [, ].

## Appendix A Visual Analogical Reasoning Prompts

### A.1 Stitched visual extrapolation examples for each domain

Visual Extrapolation. As the final step of the querying process, we presented an image of a new object and ask the model to predict what the object will look like if it goes through the same change as the given transformation.

Figure: Figure 7: Example of a visual extrapolation trial involving a reflection.
Refer to caption: extracted/5753910/figures/StitchedExtrapolationEgReflect.jpg

Figure: Figure 8: Example of a visual extrapolation trial involving an angular rotation.
Refer to caption: extracted/5753910/figures/StitchedExtrapolationEg2DRotation.jpg

Figure: Figure 9: Example of a visual extrapolation trial involving a size change.
Refer to caption: extracted/5753910/figures/StitchedExtrapolationEg.jpg

Figure: Figure 10: Example of a visual extrapolation trial involving a number change.
Refer to caption: extracted/5753910/figures/StitchedExtrapolationEgCounting.jpg

Figure: Figure 11: Example of a visual extrapolation trial involving a color change.
Refer to caption: extracted/5753910/figures/StitchedExtrapolationEgColour.jpg

### A.2 Prompts for the three-step evaluation of the analogical reasoning process

We first include a system prompt to orient the models for visual analogical reasoning.
You are an excellent visual puzzle solver! You will be given a visual puzzle that requires using visual analogical reasoning. You will think "step-by-step" and carefully examine the visual evidence before providing an answer.
Then we provide an initial instruction prompt:
You are given a visual puzzle. The puzzle features a left-to-right transformation of an object on top and three left-to-right transformations of a different object on the bottom marked by (A) or (B) or (C). The transformations involve a change of either the size, orientation, number, or color of an object.

- 1.
Cross-domain change detection.
“Which one of the following rules best describes the left-to-right transformation on top of the puzzle where the picture on the left transforms to the picture on the right? Answer with the correct rule number. Surrounded by parentheses, then provide a "step-by-step" reasoning for your choice."
- 2.
Within-domain change detection.
“Which one of the following rules best describes the left-to-right transformation in the top of the puzzle where the picture on the left transforms to the picture on the right?. Answer with the correct rule number surrounded by parentheses. Then provide a "step-by-step" reasoning for your choice."
- 3.
Visual Extrapolation.
“Which one of the three left-to-right object transformations (marked by either (A), (B) or (C) ) on the bottom of the puzzle is the same as the left-to-right transformation on the top of the puzzle? Answer with the correct letter surrounded by parentheses (or (D) if none of the options apply), then provide a a "step-by-step" reasoning for your choice."

### A.3 Prompting through reflection and self-critique

- 1.
Cross-domain change detection.
“Which one of the following rules best describes the left-to-right transformation on top of the puzzle where the picture on the left transforms to the picture on the right? Answer with the correct rule number surrounded by parentheses, then provide a "step-by-step" reasoning for your choice."
- 2.
Within-domain change detection.
“Which one of the following rules best describes the left-to-right transformation in the top of the puzzle where the picture on the left transforms to the picture on the right?. Answer with the correct rule number surrounded by parentheses, then provide a "step-by-step" reasoning for your choice."
- 3.
Visual Extrapolation.
“Which one of three left-to-right object transformations (marked by either (A), (B) or (C)) on the bottom of the puzzle is the same as the left-to-right transformation on the top of the puzzle? Answer with the correct letter surrounded by parentheses (or (D) if none of the options apply), then provide a "step-by-step" reasoning for your choice."

### A.4 Prompting through instructions

- 1.
Cross-domain change detection.
“Which one of the following rules best describes the left-to-right transformation on top of the puzzle where the picture on the left transforms to the picture on the right? Answer with the correct rule number surrounded by parentheses, then provide a “step-by-step” reasoning for your choice."
- 2.
Within-domain change detection.
“Provide brief instructions on how to establish if a transformation involves an object rotates 90 degrees or 180 degrees. Use the instructions form before to answer the following question: Which one of the following rules best describes the transformation in the top of the puzzle where the picture on the left transforms to the picture on the right?. Answer with the correct rule number surrounded by parentheses, then provide a “step-by-step” reasoning for your choice."
- 3.
Visual Extrapolation.
“Provide brief instructions on how to determine which one of three left-to-right object transformations (marked by either (A), (B) or (C) ) on the bottom of the puzzle is the same as the left-to-right transformation on the top of the puzzle? Use the instructions from before to determine which one of three left-to-right object transformations (marked by either (A), (B) or (C) ) on the bottom of the puzzle is the same as the left-to-right transformation on the top of the puzzle? Answer with the correct letter surrounded by parentheses (or (D) if none of the options apply), then provide a step-by-step reasoning for your choice."

### A.5 Prompting through code

- 1.
Cross-domain change detection.
“Which one of the following rules best describes the left-to-right transformation on top of the puzzle where the picture on the left transforms to the picture on the right? Answer with the correct rule number surrounded by parentheses, then provide a "step-by-step" reasoning for your choice."
- 2.
Within-domain change detection.
“Generate python code using the package pillow that takes in the left image in the left-to-right transformation on top and outputs the right image. Denote this snippet as training snippet using the insights from the training code snippet, which one of the following rules best describes the left-to-right transformation in the top of the puzzle where the picture on the left transforms to the picture on the right?. Answer with the correct rule number surrounded by parentheses, then provide a "step-by-step" reasoning for your choice."
- 3.
Visual Extrapolation.
“Generate a brief code snippet using python and the pillow package for each left-to-right transformation in the bottom. Each snippet takes in the left picture of the transformation and outputs the right one. Now Which one of three code snippets is the same as the training code snippet you have produced before. Answer with the correct snippet letter ((A) or (B) or (C)) surrounded by parentheses (or (D) if none of the options apply), then provide a "step-by-step" reasoning for your choice."

## Appendix B Additional details for models’ and human adults’ performance on KiVA

Table: Table 1: Mean performance of humans and models in solving each stage of KiVA, sorted by transformation domain. Standard errors are included in parentheses.

## Appendix C KiVA-adults

### C.1 KiVA-adults compared to KiVA

KiVA preserves the input and output values between the given transformation $T_{given}$ and test transformation $T_{test}$ for extrapolation. As shown in Table 2, only the object undergoing transformation varies between training (left column) and test (middle column). For example, in a numerical transformation, if the given transformation involves turning three bowls into two bowls, test asks for the result of transforming three apples (answer: two apples). KiVA-adults, on the other hand, changes the input value at test (right column). For instance, if the given transformation shows three bowls converting into two bowls again, the test asks for the transformation of four apples (answer: three apples). This requires not only generalizing a transformation across different objects but also abstracting a function to generalize over different input values.

**Table 2: Examples of transformation input values tested on KiVA vs. KiVA-adults. For KiVA, the extrapolation input matches the given transformation input in relevant domain values (e.g., both use three objects in the number domain). For KiVA-adults, the extrapolation input differs from the given transformation input (e.g., the given transformation has an input of three objects, but the test starts from four objects in the number domain).**
|  | Given Transformation | a | KiVA Extrapolation | a | KiVA-adults Extrapolation |
| --- | --- | --- | --- | --- | --- |
| Color |  |  |  |  |  |
| Size |  |  |  |  |  |
| Rotation |  |  |  |  |  |
| Reflection |  |  |  |  |  |
| Number |  |  |  |  |  |

**Table 3: Visual subdomains involved in KiVA vs. KiVA-adults. Color: KiVA examines transformations of three primary colors (red, green, blue); KiVA-adults includes two additional colors (yellow, grey). Size: KiVA enlarges or shrinks objects in both height and width by a factor of 2; KiVA-adults additionally involves non-uniform scaling, independently halving or doubling either width or height. Rotation: KiVA includes rotating objects by $\pm 90^{\circ}$ or $\pm 180^{\circ}$; KiVA-adults includes two additional angular values ($\pm 45^{\circ}$ and $\pm 135^{\circ}$). Reflection KiVA reflects objects either along the x- or y-axes; KiVA-adults includes rotating objects along both axes in KiVA-adults. Number changes: KiVA includes addition and subtraction of objects by 1 or 2; KiVA-adults includes division and multiplication of objects by 2 or 3.**
|  | KiVA |  | KiVA-adults |
| --- | --- | --- | --- |
| Color | Red, Green, Blue |  | Red, Green, Blue, Yellow, Grey |
| Size | $1/2\times$ and $2\times$ height & width |  | $1/2\times$ and $2\times$ height & width, or separately |
| Rotation | $\pm 90^{\circ}$, $180^{\circ}$ |  | $\pm 45^{\circ}$, $\pm 90^{\circ}$, $\pm 135^{\circ}$, $180^{\circ}$ |
| Reflection | reflect along the $x$ or $y$ axes |  | reflect along $x$ or $y$ axes, or both axes together |
| Number | $+1,+2,-1,-2$ |  | $+1,+2,-1,-2,\times 2,\times 3,\div 2,\div 3$ |

Furthermore, KiVA-adults includes more subdomains than the original task. The subdomains in KiVA are more distinguishable from each other (see Table  3 for the subdomains evaluated in the two tasks respectively). In KiVA-adults, we also have to add additional constraints in the sampling for the transformation domain involving number changes to avoid ambiguity between multiplication and addition (both involving an increase in number), or between subtraction and division (both involving a decrease in number).

We run GPT-4V, LLaVA-1.5 and MANTIS on KiVA-adults which consists of 2,900 transformations. We also test 10 children and 40 adults (of the same age ranges as those recruited for KiVA) on KiVA-adults. Each adult completes 19 trials that are randomly sampled from the 29 subdomains, totaling 760 responses. Each child is assigned to complete 10 randomly sampled trials, but none of them succeed on completing them with passing accuracy.

### C.2 Models’ and human adults’ performance on KiVA-adults

Figure: Figure 12: Performance of GPT-4V, LLaVA-1.5 and MANTIS on KiVA-adults. Performance is organized by question type: cross-domain change detection, within-domain change detection (which is only presented if cross-domain change detection is correct), and extrapolation of change to a new picture. Error bars represent standard errors of performance across object variations within the same transformation.
Refer to caption: extracted/5753910/figures/HardModels1.png

Figure: Figure 13: Performance of human adults on KiVA-adults. Performance is organized by the reasoning step: cross-domain change detection, within-domain change detection (which is only presented if cross-domain change detection is correct), and extrapolation of change to a new picture. Error bars represent standard errors of performance across object variations within the same transformation (not human subject variations).
Refer to caption: extracted/5753910/figures/HardAdults1.png

Table: Table 4: Mean performance of humans and models in solving each stage of KiVA-adults, sorted by transformation domain. Standard errors are included in parentheses.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Did the authors experiment with other visual analogy domains? If so, what were the results?

2. Could LLMs perform better on these tasks if the image contents were translated into text descriptions? Would textual encoding support LMMs in achieving higher accuracy in detecting “how” changes occurred and extrapolating rules?
soundness: 3
strengths: 1. The dataset, inspired by developmental psychology, is unique in its simplicity, enabling assessments that even young children can complete. Its three-stage structure offers a clear breakdown of different analogical reasoning abilities in LMMs versus humans.
   
2. Extensive experimentation demonstrates specific strengths and weaknesses of LMMs, providing critical insights. For example, while models can recognize "what" changed in an image, they struggle to quantify "how" it changed and to generalize this rule to new objects (e.g., recognizing transformations in rotation or reflection).
summary: KiVA is a new benchmark for assessing visual analogical reasoning in large multimodal models (LMMs) by comparing their performance to human adults and children. Inspired by developmental psychology, KiVA includes 1,400 visual transformations of everyday objects and tests models on identifying changes, quantifying them, and applying inferred rules to new scenarios. Experiments with models like GPT-4V, LLaVA-1.5, and MANTIS reveal that while LMMs can recognize "what" has changed, they struggle with quantifying "how" it changed and generalizing these rules, especially for complex tasks like rotation and reflection, highlighting a significant gap between LMMs and human reasoning abilities.
weaknesses: 1. The selection of visual analogy domains, while simple and fundamental, lacks sufficient justification regarding why these specific transformations were chosen over others. Intuitively, additional characteristics—such as edibility, danger, sharpness, and liveliness—are also essential features humans consider. For more complex natural scenes, it’s unclear whether the selected features are more significant than others.  The authors can provide further rationale for choosing these five factors or discuss the broader context of feature selection in visual analogy.

2. The discussion on how to improve LMM performance on these tasks is limited. It’s challenging to determine whether the low performance is due to limitations in analogical reasoning or to information loss during the initial perception stage. I’m curious whether translating the visual information into text would improve LMM performance on the task, as models might process textual representations more effectively. Investigating or discussing whether such an experiment might clarify whether perceptual or reasoning stages primarily limit LMM performance could help.

3. **Typographical errors**: Line 39 is missing a period after "reasoning"; text in Figure 1 is obscured by images, particularly in the percentage labels; Figure 8 is not referenced in the corresponding paragraph.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 4
questions: - I am not sure that the descriptions of the instructions / prompts in the appendix are accurate. Specifically, the instructions in A2 (prompting models and human adults) read the same as the instructions in A4 (prompting models through reflection and self-critique).
- line 168 "three=year=old" should be hyphenated with "-"
- "No Change" was given as an option. Were no-change trials also included?
- "If they fail to specify the change, any extrapolation would likely be incorrect". --> is this true? conditional data?
- line 279 "handpicked by Developmental Psychology experts..." what does this mean? How could we evaluate the veracity of this statement?
- did human participants practice the tasks (with feedback)? I could imagine this could particularly matter for children.
- related work: A recent relevant paper on Bongard problems may also point towards possible ways to improve analogical reasoning: Depeweg, S., Rothkopf, C. A., & Jäkel, F. (2024). Solving Bongard Problems With a Visual Language and Pragmatic Constraints. Cognitive Science, 48(5), e13432. https://doi.org/10.1111/cogs.13432
soundness: 4
strengths: - Breaking down visual analogies into these reasoning steps helps to highlight exactly where humans and models fail
- Presenting both adult and child data on the benchmark questions is valuable. The human studies appear well conducted.
- Various additional common steps are evaluated to improve model performance, which interestingly do not seem to change model performance greatly.
- Examination of model response consistency helps to unpack where model decisions go wrong.
summary: Overall this is a strong paper. The contribution of the benchmark is interesting and well-designed, and both adult and child data is provided as baseline for comparison. The paper is well written.
weaknesses: 1. The rotation task does not appear to assess 3D rotation, which is the main focus of studies of mental rotation from cognitive psychology. As far as I can tell, these rotation tasks could in principle be solved by rotating the image plane (e.g. pixels, monitor, or the participant's head). Since you have 3D objects, why not add real 3D rotations (where a "hidden" part of the object due to self-occlusion is revealed)? This task would further strengthen the challenge of the benchmark.
2. Additional analysis of conditional performance would add further understanding to model performance. For example, line 262 states "If they fail to identify the specific change, any attempt at extrapolation would likely be incorrect". Is that true? I think the authors have data to assess this: is the accuracy on extrapolation different depending whether you condition on specification correctness?
3. Similarly to point (2), unless I have missed it, the authors do not appear to evaluate model performance on specification and extrapolation if the model is first given the answer to the previous step. For example: tell the model that the object increased in size, then instruct the model to apply the same transformation to a new image. Can the models at least do this? If not, this indicates and even more fundamental problem.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 4
questions: Were there any noticeable patterns in tasks that LMMs failed but humans succeeded, outside of category (e.g. any specific objects?) when the models are consistent? Likewise for humans?

How were outliers handled in your human studies, if at all?

The child participants span a wide age range that covers important developmental milestones relevant to this task. Despite the small number of subjects, do you see correlations with age, especially for your specific test categories?
soundness: 3
strengths: The writing is clear and well-structured. Figures also clearly demonstrate the test tasks and their results. The authors introduce a novel and well-motivated benchmark for studying LMM capabilities. The test is grounded by using real-world objects, and draws inspiration from developmental psychology. The three stages introduced by the authors help clarify where LMMs have shortcomings. The experimental design is rigorous, and validated with human studies of both children and adults. The analysis is detailed and includes both error and consistency patterns. The results reveal important limitations in LMM visual reasoning capabilities that have been previously underexplored.
summary: The authors describe KiVA, a benchmark for evaluating visual analogical reasoning in LMMs, using image pairs of common household objects before and after several different transformations. They test several LMM models as well as children and adults on three different tasks: what changed, how it changed, and applying the rule to a new object. Five transformations are used: color changes, size changes, rotation, reflection, and number changes. They find that each progressive task is more difficult and less consistent for LMMs, and strategies like prompt engineering do not help.
weaknesses: The discussion could be expanded with discussion of why models tend to fail at certain transformations, outside of investigating their consistency. While the paper mentions objects were "handpicked by developmental psychology experts", it doesn't detail the selection criteria or validation process. There's no reported validation that the transformations are equally discriminable across categories. For instance, an example image shows a die face with five dots - almost completely symmetric under 90 degree rotations, one of the allowed transformations. Would the dataset include that difficult a question, or nearly-as-difficult ones with minor visual changes across transformations? Lacking space in the 10-page limit, several experiments are described extraordinarily briefly along with their results, with real methods left to the appendix. One such description - “verbal questions facilitate” -  lacks useful data, only reporting p values without actual quantities of the results.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1.I still have questions on why in the 3-stage evaluation procedure, even if the first question is answered incorrectly, the third question can be answered correctly. The authors did not provide an explanation on how this could occur, and what it could mean.
soundness: 2
strengths: 1.KiVA introduces realistic, grounded visual tasks inspired by developmental psychology, aimed at evaluating analogical reasoning in ways similar to early human cognitive development.
2.The authors broke down their evaluation and proposed a 3-stage evaluation procedure to examine the different steps involved in analogical reasoning to determine which steps a model can perform and where it may fail.
3.Results from KiVA demonstrate that state-of-the-art large multimodal models, i.e., GPT-4V OpenAI (2023a), LLaVA-1.5 (Liu et al., 2024) and MANTIS (Jiang et al., 2024), cannot solve visual analogies like humans can. The authors discovered that While LMMs can detect some object transformations, they cannot make extrapolations about those transformations to new objects.
summary: This work explores visual analogical reasoning in large multimodal models (LMMs) and compares their performance to that of human adults and children. Inspired by developmental psychology, the authors developed a benchmark, KiVA, with 1,400 visual transformations of everyday objects to test and assess models' capabilities in visual analogical reasoning.
weaknesses: 1.The KiVA test only include changes like orientation, changes in numbers, changes in size of the objects, and reflection while neglecting other basic transforms common in daily life like stretching(which can also be solved by young children).
2.LMMs today still rely heavily on the language model backbone. In other words their ability to assess the images heavily rely on what information the image encoder can provide to the backbone. Some encoders tend to lose information like size or rotation. Blaming the poor performance of the inference only on the analogy reasoning ability of the models might not be fair.

Meta-review: Inspired from developmental psychology, the paper introduces a new benchmark to assess visual analogical reasoning in multimodal models and compare it with adults and children. The task involves comparing two images of common household objects (before and after transformation), and test the model's ability to predict what changed, how it changed, and applying the rule to a new object. The paper finds that current models find reasoning about visual analogies difficult even when compared to young children, and are less consistent compared to humans (adults and children). The authors also observe no marked improvement through in-context learning techniques like prompting. 

The paper received positive reviews. The reviewers appreciated the simplicity of the benchmark while still being informative of limitations in large multimodal models. Reviewers found the experiments to be well conducted which reveals important and understudied limitations in visual reasoning capabilities of existing models. I concur with the reviewers that the benchmark is well motivated, novel, focused and well-constructed. The benchmark focuses on three important questions (what changed, how it changed, and extrapolating the rule to a new object). Posing these questions as multiple-choice is also well thought out. 

While the reviewers agreed on the usefulness of the benchmark, there were some clarification questions asked around the design of the benchmark (specially about choice of tasks, validating task correctness) that the authors responded to. It will be useful to incorporate these discussions into the final manuscript.

Decision: Accept (Poster)