The ConceptARC Benchmark: Evaluating Understanding and Generalization in the ARC Domain
Authors: Arsenii Kirillovich Moskvichev
Victor Vikram Odouard
Melanie Mitchell
Venue: Accepted by TMLR ()

Abstract: The abilities to form and abstract concepts is key to human intelligence, but such abilities remain lacking in state-of-the-art AI systems. There has been substantial research on conceptual abstraction in AI, particularly using idealized domains such as Raven’s Progressive Matrices and Bongard problems, but even when AI systems succeed on such problems, the systems are rarely evaluated in depth to see if they have actually grasped the concepts they are meant to capture.

In this paper we describe an in-depth evaluation benchmark for the Abstraction and Reasoning Corpus (ARC), a collection of few-shot abstraction and analogy problems developed by Chollet (2019). In particular, we describe ConceptARC, a new, publicly available bench- mark in the ARC domain that systematically assesses abstraction and generalization abilities on a number of basic spatial and semantic concepts. ConceptARC differs from the original ARC dataset in that it is specifically organized around “concept groups”—sets of problems that focus on specific concepts and that are vary in complexity and level of abstraction. We report results on testing humans on this benchmark as well as three machine solvers: the top two programs from a 2021 ARC competition and OpenAI’s GPT-4. Our results show that humans substantially outperform the machine solvers on this benchmark, showing abilities to abstract and generalize concepts that are not yet captured by AI systems. We believe that this benchmark will spur improvements in the development of AI systems for conceptual abstraction and in the effective evaluation of such systems.


Published in Transactions on Machine Learning Research (08/2023)
The ConceptARC Benchmark: Evaluating Understanding and Generalization in the ARC Domain
Arseny Moskvichev Santa Fe Institute
arsenymookvichene@@aalloom
Victor Vikram Odouard
vicviod@@gmail.comm com
Santa Pe Instibute
Melanie Mitchell
mm mm@ edu
Santa Fe institute
Reviewed on OpenReview: https: // openreview. net/ forum? id- 8ykyCb t2g
Abstract
The abilities to form and abstract concepts are key to human intelligence, but such abilities remain lacking in state-of-the-arl AI systems. There has been substantial research con- ceptual abstraction in AI, particularly using idealized domains such as Raven's 's Progressive Matrices and Bongard problems, but even when AI systems succeed on such problems, the systems are rarely evaluated in depth to see they have actually grasped the concepts they are meant to capture.
In this paper we describe an in-depth evaluation benchmark for the Abstraction and Reason- ing Corpus (ARC), a collection few-shot abstraction and analogy problems developed by Chollet (2019). In particular, describe ConceptA ARC, new, publicly available benchmark in the ARC domain that systematically assesses abstraction and generalization abilities number of basic spatial and semantic concepts. ConCept differs from the original ARC dataset in that it is specifically organized around "concept groups" of problems that focus on specific concepts and that vary in complexity and level abstraction. We report results on testing humans on this benchmark as well as three machine solvers: the top two programs from a 2021 ARC competition and OpenAI's GPT-4. Our results show that hu- mans substantially outperform the machine solvers on this benchmark, showing abilities to abstract and generalize concepts that not yet captured by AI systems. We believe that this benchmark will spur improvements in the development of Al systems for conceptual abstraction and in the effective evaluation of such systems.
1 Introduction
Forming and abstracting concepts is at the heart of human intelligence (Carey 2011 Hofstadter 1995
Lake
et al. 2017)- These abilities enable humans to understand and create internal models of the world, to use these models to make sense of new information, often via analogy, and to docide how to behave in novel situations. Giving machines such abilities was one of the key goals of McCarthy all's 1955 Dartmouth Al workshop, but these precisely the capabilities that are still largely lacking in today's 's Al systems [Mitchel] 2021 Mitchell and Krakauer) 2023
In AI, research on concept formation and abstraction often utilizes idealized domains that caplure some of the essential aspects of abstraction and analogy in the real world. In such domains one can be explicit about the assumed prior knowledge without requiring the open-ended knowledge involved in real-world language and imagery. Examples of idcalized domains that require abstraction and analogy abilities include Raven's 's Progressive Matrices (Carpenter et al. 1990 Zhang et al. 2019), Copycat letter-string analogies (Hofstadter

---

Published in Transactions on Machine Learning Research (08/2023)
and Mitchell 1994). Bongard problems [Bongard 1970 Foundalis 2023). and the Abstraction and Reasoning Corpus (ARC) (Chollet, 2019). The latter three especially, which require the solver to generale answers to problems (rather than selecting from candidate answers), remain open challenges for Al systems.
There has been substantial research on developing AI systems to solve problems in each of these domains (Malkiński and Mańdziuk] (022b). Symbolic AT systems have been developed to solve many of the original Raven's problems (Lovett and Forbus, (017) and deep learning methods have surpassed human accuracy automatically generated Raven's-like problems (Malkiński and Mañdziuk 2022a). Particular subsets of Copycat letter-string problems have been solved by early "active symbol" methods (Hofstadter and Mitchell, 1994) and more recently by large language models (Webb et al., 2022). Simple Bongard problems have been recently addressed by program induction methods (Sonwane et al. 2021,, and automatically generated Bongard-like problems have been tackled by deep learning systems (Nie et al. 2020). ARC problems were the subject of a 2020 Kaggle challenge (Kaggle.com) 2020) and a limited number were solved by program- synthesis approaches (Alford et al.) 2022; Banburski et al. 2020 de Miquel Bleier) 2020; Wind 2020a). However, few of these ellorts have probed the extent to which AT systems have actually grasped the abstract concepts that these various problems solve a problem involving a specific concept, to what extent will it be able to solve other problems that target the same concept, including problems that instantiate the concept in a quite different manner? Such generalization abilities would be crucial to any AT system operating in the real world.
meant to capture. More specifically, if an Al system is able to
In this paper, we examine how to evaluate the degree to which an Al system has learned or understood a concept in generalizable way. Machine learning systems are typically developed by randomly splitting a set of examples into training and test sets. However, this kind of evaluation does not systematically test for the kind of learning and understanding that is needed for "out of distribution" generalization. Indeed, it has been shown many times that machine learning systems can learn "shortcuts" that produce high accuracy on the test set but that do not generalize more broadly (Geirhos et al. 2020 Lapuschkin et al., 2019). To evaluate AI systems, in particular systems that are claimed to perform abstract reasoning, new evaluation methods and benchmarks are needed that specifically test that the system has grasped the relevant abstract concepts.
We proposc systematic concept- based evaluation method, in which test examples are designed to instantiate variations on chosen concepts. If a system performss well on a range of such examples that vary in complexity and degree of abstraction, that performance provides strong evidence that the system has understood the concept in a generalizable way. In previous work we applied such an evaluation method to programs that exceeded human accuracy on the RAVEN corpus (Odouard and Mitchell, 2022). Our evaluation provided evidence that, while attaining high accuracy on the test set, these programs had not actually learned gen- eralizable abstract concepts. In this paper we propose concept-bass based evaluation benchmark for the ARC domain. We discuss why ARC is excellent domain for studying concept formation and abstraction in both humans and Al systems, but argue that the original ARC test examples do not systematically evaluate concept understanding.
Our contributions in this paper are (1) the creation of a new concept-based evaluation benchmark for the ARC domain and (2) results from our studies using this benchmark to evaluate state-of programs that solve ARC problems, as well as human performance on this benchmark. Our results show that humans exhibit strong conceptual generalization abilities in the ARC domain, as compared with much weaker abilities in current AT programs, both those designed for this domain and more purprae large language models. We believe that our benchmark, and future extensions of it, will spur improvements in the development of AI systems for conceptual abstraction and in the effective evaluation of such systems.
2 The Abstraction and Reasoning Corpus
Chollet (2019) proposed the Abstraction and Reasoning Corpus (ARC) as a domain for evaluating rast concept understanding and reasoning abilities in both humans and Al systems. ARC consists of a set of analogy problems, exemplified by those given in Figure 1 In particular, each problem consists of a sot of inniiaall
demonstrations initial and transformed grids and one or more test input grids. In Chollet's terminology, the demonstrations coupled with the test inputs form a lask to be solved. To solve task, an agent needs
2

---

Published in Transactions on Machine Learning Research (08/2023)
Demonstrations
Demonstrations
Demonstrations
Test Input
Test Input
Test Input
?
?
(a)
(b)
(c)
Figure 1: Three sample ARC tasks from Chollet (023 Each task consists of set task demonstrations transformations between colored grids that follow the same abstract rule and (here) single test input, The job of the solver is to generate new grid that results from applying the abstract rule to the test input. (Best viewed in color.)
infer the abstract rule governing the demonstrations and apply that rule each test input produce correct output grid.
The ARC domain was inspired by the hypothesis that humans possess innate (or early learned) "core knowl- edge systems" on which all further learning and knowledge is based. According to Spelke and Kinzler (2007)
core knowledge systems include:
(1) Objectness: knowledge that the world can be parsed into objects that have certain physical properties, such traveling in continuous trajectories, being preserved through time, and interacting upon contact;
(2) Numerosity:
knowledge of small quantities and notions of "smallest," "largest," " "greater than," *less
than," etc.;
(3) Basic geometry and topology: knowledge of lines, simple shapes, symmetries, containment, etc.;
(4) Agents and goal-directed behavior: knowledge that some entities are agents who have their own intentions and act achieve goals.
In creating ARC tasks, Chollet assumed the first three priors that is, the only knowledge that should be necessary to solve these tasks. For example, Figure T(a) requires spatial notions of extending line diagonally from an object to a boundary; Figure 1(b) requires parsing connected sets of pixels into objects and recognizing shapes across different rotations and symmetries; and Figure requires notions of counting and comparisons among quantities.
The tasks in Figure are sampled from the 1,000-task corpus created by Chollet. Eight-hundred tasks were made public online [Chollet] (023) and as a challenge on the Kaggle platform (Kaggle.com) (020)). The remaining 200 tasks kept "hidden" test set for evaluating Al systems; 100 of these used to
3

---

Published in Transactions on Machine Learning Research (08/2023)
evaluate submissions to the Kaggle challenge. Each program in the Kaggle challenge was allowed to generate three candidate solutions for each test input in the hidden evaluation set; if one of the three was correct, the test input was considered solved. A task is considered to be solved if all of its test inputs are solved. The first-place program in the Kaggle challenge solved 21% of the 100 hidden tasks; an ensemble of the first- and second-place programs solved about 31%
ARC remains challenging for Al systems, even for enormous pre-trained language models (see Section 7 for several reasons. ARC tasks involve few-shot learning-inferringg abstract concept from just a few examples. Moreover, the "core knowledge" required is enormously open-ended (e.g., even recognizing an "object" in this domain can require taking context into account), and solving the tasks requires applying core knowledge concepts with a flexibility that is key to human coguition but has not yet been achieved in AL
One limitation on ARC's usefulness for Al rescarch is that might be too challenging. Many of the tasks in Chollet's corpus are diffieult even for humans, and the corpus as a whole might be sufficiently difficult for machines that it does not reveal real progress machine acquisition of core knowledge. Another limitation is that the current corpus does not systematically test generalization of the concepts underlying individual tasks. For example, if an ARC solver correctly answers the test input in Figure [cc, one cannot conclude that the solver can generalize the concepts of "counting" and "greater than" system might have employed another strategy to solve this specific instance. Only by systematically evaluating a system on many variants of a given concept can we gain evidence that the system grasps that concept in a way that predicts corresponding generalization abilities.
We address those limitations by developing a new benchmark set of tasks in the ARC domain that (1) are designed to rely straightforward instances of core concepts (and thus be relatively easy for humans), and (2) systematically evaluate the degree to which a task solver has sufficient understanding of a particular concept so as to be able to generalize. Furthermore, we test three programs- the first- and second-place programs from the ARC-Kaggle challenge, as well as OpenAl's GPT-4 pre-trained language model on our tasks, and compare their performance to humans tested on these same tasks.
3 The ConceptARC Benchmark
As first step in developing new benchmarks for concept understanding in the ARC domain, we created Concept.ARC 7 We studied all publicly available ARC tasks and manually identified 16 concepts used in them (listed in the left column of Table 1). Each of these concepts is central in one or more tasks in Chollet's published ARC "training" and "evaluation" sets, though those sets were not organized around specific concepts, nor does our set of 16 concepts by any means cover all the different concepts used in ARC tasks. For each concept, we created 10 new ARC tasks that are different instantiations of the concept. This set of tasks is termed the concept group for a given concept. Fach of our tasks has three different test inputs. The purpose of creating multiple test inputs for a given task was to control for possible "shortcuts" that might enable solver to solve particular test input without correctly grasping the underlying rule, whercas the purpose of creating multiple lasks within a concept group was to ensure that a solver is able to generalize across different instantiations of the concept.
As an example, Figure shows thrce tasks from ConceptARC that are variations on the concept Sameness. Figure 2/a) focuses on sameness between shapes (in each transformation, only objects with the same shape are retained); in Figure 2(b) lines with the same orientation are retained, and in Figure 2(c) each grid is divided (by gray line) into two subgrids; if the two subgrids are identical, both are copied, and if not, only the lefthand subgrid is copied. These sample tasks illustrate the range of variation among test inputs for a given task and among tasks in a given concept group. This range is meant to be sufficiently broad that an agent that correctly solves most or all of the tasks in a group is likely to possess a rich understanding of the concept. (Examples of problems from each concept group are given in Appendix A
NNaaly all of the 800 published tasks had only test input; the number of test inputs per task in the hidden test set was not revealed.
2 All ConceptARC: tasks can be downloaded from https://githhub http::://ggtttaa...///cttrroriii http://ttw...eeeeeeererrrrrrr/ae
4

---

Published in Transactions on Machine Learning Research (08/2023)
Demonstrations
Demonstrations
Demonstrations
Test Inputs
Test Inputs
?
?
Test Inputs
?
?
?
?
?
?
aa)
(b)
(c)
Figure 2: Three sample tasks from ConceptARC, each of which has set of demonstrations and three test inputs. Each task is variation the concept Sameness. (Best viewed in color.)
We constructed the tasks in the ConceptARC benchmark manually We do not believe that interesting, diverse task variations on a particular concept could be constructed automatically, unless we were able to create an automated system that understands the concept a general way (and the challenge of developing such a system is what inspired the benchmark in the first place). Given that the goal of the ARC benchmark is evaluate humanlike concept abstraction, (following Chollet (2019)) constructed the tasks using our intuitions about human
concepts.
It is imporlant to note that "concept" does not have a rigorous definition in cognitive science. Here use the term informally to refer to a facet of core knowledge that plays important roles in the original ARC tasks, and that play more obviously contral roles in the ConceptARC tasks in a given concopt group.
4 Human and Machine Performance on ConceptARC
In this section present results from studies of human and machine performance ConceptARC.. We recruited human participants using the Amazon Mechanical Turk and Prolific platforms and tested them on tasks in our corpus, using a visual interface. We also obtained code for the first- and second-place ARC-Kaggle winning programs and ran them on the same tasks, in the same text-based format used in the ARC-Kaggle competition. Finally we used OpenAT's APT to test GPT-4 on these tasks, using a text prompt similar to the format given to the ARC-Kaggle programs. Details of each study are given in the next sections.
Recall that each of our 16 concept groups contains 10 tasks, each of which includes three unique test inputs, for total of 30 test inputs per concept. Both humans and machine solvers are allowed three guesses for
It. should noted that tasks, following human conventions, color black plays special role, signifying background "unfilled" grid squares.

---

Published in Transactions on Machine Learning Research (08/2023)
each test input, and solver (human machine) is considered correct on test input if one of the three guesses is correct.
Table gives, for each concept, the accuracies over the 30 test inputs in the concept group. These results provide an assessment of how well solvers can generalize over the range of different tasks associated with each concept. The human accuracy reported for each concept is the average accuracy over the 30 test inputs in that concept-group group, where the accuracy on given test input is the fraction of participants who correctly solved that test input. The accuracies reported for each Kaggle-ARC program and for GPT-4 are simply the fraction of test imputs in each concept group that were correctly solved by the program. We discuss these results in detail in Section 8
Table 1: Accuracies of humans, the two top-scoring ARC-Kaggle programs, and GPT-4 (with temperatures O and 0.5) on test inputs in each concept group in CoCCeptARC.
Concept
Humans
ARC-Kaggle First Place
ARC-Kaggle Second Place
GPT-1 Temp 0 GPT-4 Temp 0.5
Above and Helow
0.90
0.70
0.33
0.23
0.37
Center
0.94
0.50
0.20
0.33
0.33
Clean Up
0.97
0.50
0.20
0.20
0.27
Complete Shape
0.85
0.47
0.30
0.23
0.23
Copy
0.94
0.23
0.27
0.23
0.27
Count
0.88
0.60
0.40
0.13
0.17
Extend To Boundary
0.93
0.77
0.47
0.07
0.1
Extract Objects
0.86
0.43
0.43
0.03
0.07
Filled and Not Filled
0.73
Horizoutal and Vertical
0.43
0.10
0.27
0.33
Inside Outside
0.91
0.57
0.10
0.16
Move To Boundary
0.91
0.37
0.30
0.20
0.20
Order
0.83
0.27
0.27
0.27
Same Different
0.88
0.53
0.17
0.17
0.27
Top and Bottom 2D
0.95
0.60
0.57
0.23
0.37
Top and Bolton 3D
0.93
0.50
0.03
0.20
0.27
5 Details of Human Studies
To evaluate human accuracy in our tasks, we ran an online study, recruiting participants from the Amazon Mechanical Turk and Prolific platforms. This section provides additional details our procodure.
5.1 Procedure
Participants were presented with visual interface for solving ARC tasks, adapted from the original ARC viewer (Chollet 2023), and programmed for online data collection using the psiTurr framework Each
participant was presented with a random selection of tasks (17 for most participants, although see Appendix C for a discussion of a few exceptions). Fach task had three test inputs, but these were randomly split among participants, with each participant seeing only one test input of a given task. Similar to the ARC-Kaggle programs, participants were given three attempts to solve cach test input. If a participant managed to solve the test input correctly, they were asked to verbally describe their solution before moving on to the next task. We will report on the analysis of this natural language data in future work.
Note that among the 17 tasks given to a participant, the first two were extremely simple training tasks, for which the participants were allowed unlimited attempts. These training tasks were included to give the participants time to familiarize themselves with the study interface. Additionally, among the remaining tasks, three were "minimal," " that is, the simplest concept instantiations we could create. These minimal tasks served as "attention checks," " helping to identify individuals who did not try solve the tasks or follow instructions (sec Section 5.2 We give examples of minimal tasks in Appendix B Because they were used to
*Results for human participants and machines on all 480 test inputs be downlonded from https://github.com victorvikran/Concepaatt htttps:/ppatuukk.org
6

---

Published in Transactions on Machine Learning Research (08/2023)
determine which participants to exclude, did not include performance the minimal tasks in the results given in Secctoon 4
5.2 Exclusion criteria
We used two criteria to exclude participants. A participant was excluded from further analysis if 1) they failed at solving two more minimal tasks; or 2) they provided empty nonsensical explanations for their solutions (such as "Nice," "Solve task is good," " and so on). Failing the first criterion suggests that the person was not paying attention to the task, while failing the second shows lack of ability or motivation to follow the task instructions. Since was always faster for a participant to pretend to fail any given problem rather than to try to solve it, excluding unmotivated, inattentive participants was crucial to avoid skewing the results.
In total, 55 out of 482 initial participants were excluded based on inadequate explanations (all from Amazon Mechanical Turk). An additional 12 participants were excluded based on failing to solve two more minimal tasks (8 from Amazon Mechanical Turk, 4 from Prolific).
5.3 Participants
The final sample comprised 415 participants 204 from Amazon Mechanical Turk and 211 through Prolific. To ensure linguistic fluency in English for the purpose of collecting natural language descriptions, only U.S.- based Amazon Mechanical Turk workers were invited to participate in the study, and only U.S. or U.K. participants were recruited from Prolific
Since test inputs were randomly assigned to different participants, and since the psiTurk platform does not naturally have a mechanism to monitor how many participant answers were collected for each test input, there is some variation in the amount of data collected for different test inputs. Overall, each test input was given to at least 8 participants (with exception, which given to only 7 participants). The detailed wttp////cccooo.rccaa///ore...t.t
results available at https://gi httpa/////////taoooooooo///////a tested on each test input in the corpus.
provide the number of participants
6 Details of Testing Winning Programs From the Kag-le Challenge
As described in Section 2. in 2020 the Kaggle platform hosted three-month competition on ARC tasks (Kaggle.com 2020). Competing programs were scored on 100 hidden tasks. Programs were allowed to make three predictions for each test input, and if one of the predictions was correct, the test input was considered to be solved. Using this metric, the first and second place programs attained accuracies of 21% and 19% respectively. An ensemble of the two winning programs attained an accuracy of about 31%, and as of this writing, this the -tate-of-the-art accuracy on this hidden evaluation set To our knowledge, there have been no published largo-scale experiments to date evaluating humans tasks in the ARC corpus (though, we describe in Section 8 small-scall studies were performed by Acquaviva et al. (2022) and Johnson et al. 2021
We obtained the source code for the first- and second-place ARC-Kaggle winners on GitHub 0 and tested each them on all of the CouceptARC tasks.
The first and second place programs in the ARC-Kaggle challenge both work by performing a heuristic search over fixed, manually defined set of grid opcrations to generate pipcline of these operations that, when applied to inputs from the task demonstrations, correctly generales the corresponding outputs.
'All of the minimal tasks are included in the corpus provided at https://gitthb. www.///ctoriiirrra//oooeeeoooooe
TW cannot exclude the possibility that person from another country might register U.S.-based account Prolific or Amazon Mechanical Turk. However, we have manually checked the verbal answers provided by the study participants. All participants in the final sample demonstrated high fluency in English, which least should ensure that they fully understood the study instructions.
8F. Chollet, Persoual Communication, April 7, 2023.
(first-place alejandrodemmiqull/RRgg Kaggle second-place ARggle winner).
"https://github.. https/tttuu.cmmottttteeeeeeeee wwwp///tppparee//RRC---ee-
ARC-Kaggle
winner);
https://gsthub.com
7

---

Published in Transactions on Machine Learning Research (08/2023)
In particular, the first-place program WWind 2020ag constructed its solutions using a manually created set of 142 grid operations, such operation that splits a given grid into multiple grids consisting of "background color" and "objects" consisting of color-connected pixels, operations that perform rotations, reflections, and other variations on a given grid, and an operation that extracts the "object" with the most non-black pixels. The second-place program (de Miquel Bleier] 2020) constructed its programs from a set of 50 manually created grid operations, and used a genetic algorithm to search for successful pipelines of operations.
Both programs were able to increase their success by augmenting the given task demonstrations, for example, by flipping the demonstration input and output grids along the diagonal, by remapping colors, and other heuristic transformations.
Given the open-ended nature of ARC, we doubt that similar heuristic search methods, even over a much larger number of grid operations, will achieve anything like human performance on ARC tasks. Even the authors of the winning programs scem to agrec that a wholly different kind of method is needed. The author of the first-place program wrole, "Unfortunatell, I don't feel like my solution itsell brings us closer to AGI" (Wind 2020b) and one of the authors of the second-ples placee program noted that "No team out of the 914 [ARC- Kaggle competition] participants found satisfying, Al-focused solution for this problem" (de Miquel Bleier
2020
7 Details of Testing GPT-4
GPT-4 (OpenAl 2023) is a large-scale multimodal Al system created by OpenAl. Webb et al. (2022) showed that the publicly available languago-only version of GPT-4 (as well as its prodecessor GPT-3) was able to match or exceed human performance on several idealized analogy tasks, in a zero-shot manner (i.e., without any fine-tuning on these tasks). To test the generality of these findings, we assess GPT-4's zero-shot performance on the tasks in ConceptARC, which have some resemblance to the tasks used by Webb et al.
To test this language-only version of GPT-4 on the tasks in ConceptARC, we used the API provided by Ope- nAI GPT-4 API prompts have "system" " and "user" components, with the "system" component intended to provide general instructions, priming the model towards certain behaviors, and the "user" component for dialogue inputs. We used the prompt structure (similar the one used by Webb et al. (2022)) illustrated in Figure
Within each row of grid, the colors of each pixel were numerically coded as in the original ARC data files given by Chollei (2023) (these were the inputs to the ARC-Kaggle competitors) and space-separated. For example, [2 2101 would encode a row with four pixels: red, blue, black, and blue again.
We tested GPT-4 on all the tasks in our corpus, first with temperature 0 (following Webb et al. (002 and then with temperature 0.5. In every case, GPT-4's response was in the correct format for an output that is, the errors that the model made were "true" mistakes, rather than improperly formatted correct answers. In the case where temperature In the case where temperature was set to 0.5, repeated each task prompt three times, and if at least one of outputs was correct, we considered the task to be solved correctly IT
set to 0, the outputs are deterministic, so only one output was considered.
8 Discussion
8.1 Human and Machine Performance
Table shows that the human participants achieved substantially higher accuracy than the machine solvers on tasks in our ConceptARC benchmark (for a more detailed report with uncertainty estimates, see Ap- pendix D.. Recall that the average accuracy across test inputs in concept group measures how well solvers
htttpps//oppeaii com/ con/utt In experiments the model name was set to "gpt-4", the temperature was set to 0 0.5, and other parameters were left at their default values.
11 Following (Wbbb et al. 22022) also tested CPT-4 using the same prompt format as in Figure 3 but with "nn inserted after each grid row, to indicate new lines. The results were not substantially different from testing GPT-4 without inserting i!
8

---

Published in Transactions on Machine Learning Research (08/2023)
Demonstrations
"You are helpful assistant that solves unalogy making puzzles. Only give answer, no other words or text."
System:
User: "Let's try to complete the pattem::
1: 0 |101
[0 (3
input 2: 00000000000010000000000000000000
Test Input
00000000000000000000000000000000
output
3: |1000
[0000000010010000000000000000000
?
output
(a)
(b)
Figure 3: (a) ConceptARC task. (b) Corresponding prompt given to GPT-4. (Best viewed in color.)
can generalize over different tasks representing a given concopt. The average difference in per-concept ac- curacy between humans and the first-place ARC-Kaggle program was 40 percentage points. The human participants exhibited over 90% average accuracy on 11 of the 16 concepts, and over 80% accuracy on each of the remaining 5 concepts. In contrast, the first-place program never scored above 80% accuracy any concept, and for 11 out of 16 concepts, its accuracy was below 60%. The second-place program's accuracy never reached 60% and was below 50% on 15 out of 16 concepts. On each concept, GPT-4 performed better with temperature set to 0.5 (and given three guesses) than it did with temperature set to But even in the former case GPT-4's performance was relatively weak: its accuracy was below 30% on 12 out of 16 concepts and its maximum accuracy was only 37%.
GPT-4's weak performance here contrasts with its much better performance other idcalized domains for analogy-making found by Webb el al. (2022). We speculate that there are two potential ial reasons for this discrepancy. First, to successfully solve ConceptARC problems, system must identify the relevant higher-level features (c.g., objects, shapes, number, orientation) in cach grid and determine how they were transformed into a new grid. In the grid-based tasks discussed in Webb et al., which were inspired by Raven's Progressive Matrices, the relevant. features were pre-extracted and given as inputs. This likely makes the grid-based tasks created by Webb et al. considerably less challenging than those in ConceptARC. Second, we speculate that the patterns in Raven-based tasks tested by Webb et al. might be easier to recognize for large language models: each grid row had a constant. number of features (which might aid in finding analogies using attention mechanisms) and the concepts in these tasks were generally of more sequential rather than spatial nature. Testing these speculations is topic for future work.
The generally high accuracies of humans on each concept indicates successful generalization over the different variations cach given concept group. In contrast, the much lower accuracies of programs we tested indicates a lack of ability to generalize over the variations in a concept group, and thus failure to develop the abstractions that ARC is meant to test.
While the first- and accond-place ARC-Kaggle programs did not reach human-level accuracy, it is interesting to note that these programs have siguificantly higher accuracy on the tasks in ConceptARC than they did on the original tasks in the ARC-Kaggle competition, where their respective accuracies were 21% and 19%. This is likely due to our intentional design of the tasks ConceptARC be casier than those in the original ARC set. Providing an casier benchmark also gives more insight into differences between the two programs: while their scores on the original set. quite close, in results the first-place winner has substantially
9

---

Published in Transactions on Machine Learning Research (08/2023)
Demonstrations
Demonstrations
Test Input
Correct Solution
Human
Program
Test Input Correct Solution
Human
Program
(a)
(b)
Figure 4: Two examples illustrating human "near-miss" errors, compared with errors made by the first-place ARC-Kaggle program on the same test input. (a) A task in the Copy concept group. The human correctly copied the green and red object into the blue rectangle, but incorrectly deleted the original object. The first-place program ("Program") did not seem to grasp the notion of copying an object. (b) A task in the Extend To Boundary concept group. The human correctly extended a line to the boundary, but modified the original object to make it a solid rectangle rather than a single line. The first-place program did not seem to grasp the notion of extending a line from given object a boundary. (Best viewed in color.)
higher accuracy than the second-place winner the average difference in per-concept accuracy between the two programs is about 23 percentage points.
We ran simple x test to see whether there were statistically significant differences in accuracy across concepts. For humans, as well as for Kaggle First- and Second- place algorithms, the data strongly suggest such a difference (p < 0.001 in each of the three hypothesis tests). the same time, GPT-4 results did not provide sufficient evidence that there were true differences in its mastery of those concepts (p 0.27 and p 0.23 for 0 and 0.5 temperature regimes respectively). In other words, the observed variation in GPT-4's performance across concepts might. have occurred by chance. At the same time, GPT-4's generally low accuracy across all concepts, together with the relatively low-data regime, limits the statistical power of this test, hence this negative result should be interpreted with caution.
8.2 Comparing Human and Machine Errors
It enlightening to compare the kinds errors made by humans and those made by programs on these tasks. In analyzing a sample of errors made by the human participants in our study, we found that many errors included obvious carcless mistakes (c.g., off-by-onc crrors in the size of the output grid), wrong answers due "giving up" (simply copying the input grid or creating blank grid), and "near misses," " in which it is obvious that the person grasped the underlying concept but made error in applying it. In Figure 4 we show two examples of these types of near-missos by humans, and corresponding incorrect answers to the same task by the first-place ARC-Kaggle program.
As illustrated in Figure 4 the errors made by the winning ARC-Kaggle programs and by GPT-4 are harder categorize. As we described in (Section 6 abbov, the ARC-Kaggle winning programs were not designed capture abstract concepts, but instead heuristically construct pipelines of pre-designed grid transformations, it not surprising that their errors typically less interpretable than those of humans. GPT-4 of course
10

---

Published in Transactions on Machine Learning Research (08/2023)
was not designed for this domain at all, although Webb et al. (2022) demonstrated the pattern-recounition abilities of large language models in other analogy domains.
8.3 Limitations of Our Studies
There are several limitations of the studies we report here in using the ConceptARC corpus to assess con- ceptual abstraction abilities in humans and machines.
Because the tasks in ConceptARC were created mamually, the corpus is relatively small: 16 concept groups, with 10 tasks per concept-group and three test inputs per task, for a total of 480 test inputs. We plan to substantially extend this corpus in the future, adding additional concept groups, tasks, and test inputs order to more thoroughly explore abstraction and generalization abilities in the ARC domain.
Our human studies, using Amazon Mechanical Turk and Prolific, included 415 participants, each being tested on approximately 17 test inputs (from different tasks) in an approxinately 45-minute session. As we described in Section 5) this yielded typically 8 to 14 people solving each test input. Our results, showing high human accuracy on these tasks, are based on these relatively small sets of people, whose numbers were limited by the funds we had available for these studies. In future work we will extend these studies to determine if they generalize over larger populations of human solvers.
One additional limitation for both Concept and the original ARC dataset- is that there might be more than one reasonable solution for given test input. Like Chollet in the original ARC set, tried to design tasks that have only one clear solution for cach test input. However, there is always the possibility of other solutions that people would find reasonable, but that would be counted as incorrect in our study. By closely examining incorrect solutions submitted by humans in our studies, we found six (out of 480 total) test inputs that we considered to be ambiguous in this way. There might be additional such test inputs that we did not identify. Indeed, a small number of ambiguous test inputs are likely inevitable in any corpus. However, the effects of such ambiguity on our results are mitigated, for both humans and machines, by (1) allowing three solutions to be submitted for each test input; (2) having multiple test inputs per task; and (3) having multiple tasks per concept group. These factors help make the aggregate results robust in spite of an inevitable small subset of ambiguous test inputs.
There are also a small number of tasks that allow for "shortcut solutions": for example, tasks in which the correct solution to a lest input simply to copy it, which can be a default stralegy for the ARC-Kaggle winners and an easy pattern for GPT-4 to recognize. However, our purpose in creating numerous tasks that are variations particular concept is to make it highly unlikely that any program could use shortcuts to solve most or all of the tasks in a given concept group.
9 Related Work
In similar spirit to our work on the ConceptARC benchmark, Kim et al. (2022) created the "Mini-ARC" dataset, in which grids are fixed at 5 x 5 in order to simplify the domain, and 150 tasks (each containing one test input) are organized around six broad categories (movement, color, object, number, geometry, and "common sense"). This set can complement our ConceptARC benchmark, which allows any grid dimensions and systematically explores 16 more specific spatial and semantic concepts.
Johnson et al. (2021) carried out a study of humans solving ARC tasks. They chose 40 tasks from the public ARC dataset and tested each of 95 participants on a random subset of 10 out of the 40 tasks. On average the participants' per-task accuracy was about 84%, though with substantial variance. Johnson et al. also recorded the average time to complete each task, well as participants' action sequences while generating responses, and analyzed the errors people made. Similar to the results of our study, the authors found that human errors generally were near-misses, whereas the errors made by the first-place ARC-Kaggle program indicated that it did not grasp the underlying abstract rule. The human study we report in this paper can be seen as a follow-up to Johnson et al's study, but with a larger set of (newly created) ARC problems that are variations on specific concepts (rather than a randomly chosen subset of the original ARC tasks) and with a considerably larger population of participants.
11

---

Published in Transactions on Machine Learning Research (08/2023)
In developing AI systems to solve ARC problems, the predominant approach is automated program synthesis that is, automatically generating a series of operations on grids or or other representations that yields a solution. The primitive operations are typically created mamually, and heuristic search is used to find a combination of operations that solves a given task. For the two winning ARC-Kaggle programs, the primitive operations were sets of hand-designed grid transformations, and the synthesized programs were piplelines of transformations resulting from houristic search methods.
Since the end of the Kaggle competition, several new program-synthesis approaches to ARC have been explored. For example, Banburski et al. (2020) used a program-syuthesis algorithm called "DreamCoder" (Ellis et al. 2020) that, given a domain-speeific primitive operation, can generate more abstract operation that can be added to the set of available primitives. Banburski et al. manually defined a small set of grid- transformation operations, and used these as the basis for training an agent based on DreamCoder to generate programs that could solve a small set of ARC tasks that focused on symmetry transformations. In follow- up work, Alford et al. (2022) explored a similar method using a neural-network-guiddd program-synthesis approach.
In contrast to using grid- triiftraasfoooation primitives, Xu et al. (0022) proposed "object-centric" approach to solving ARC tasks. In their system, grids are mapped to graph representations, and the system searches for pipelines of operations on these graphs rather than on the grids themselves. The nodes in a graph correspond to objects in a grid and links between nodes correspond to relationships between objects. Which sets of pixels are grouped as an "object" in a graph is decided heuristically, as is what relationships are included in the graph. In their experiments, Xu et al. focused on a set of 160 "object-centric" tasks from the public ARC dataset, and showed that their system was able to solve about third of them.
In an interesting on ARC tasks may be due to their ability to generate descriptions of abstract concepts in natural language. The authors carried out a study, like ours, in which they asked human participants to both solve ARC problems and generate natural- natural-language instructions that would enable another human to produce the correct output, given only the test input (i.e., not including the dernonstrations). The authors then tested these instructions on other human participants and found that the instructions were sufficient for solving the task about 88% of the time. The authors relcased the LARC 354 original ARC tasks with human-generated language instructions. They used this dalaset to train and evaluate selected program-synthesit methods, to see if these systems could utilize language the way humans do. The results were quite poor the best system was able to solve only about 12% of the tasks it was tested on.
weweeee--eeieeeeseeeeeeeeeeeeeee study, Acquaviva et al. (2022) posited that the advantage of humans
ARC) dataset, which couples
Any of these program-synthesis systems might be improved by adopting more expressive domain-specific languages and by improving their program-search methods. While these and other approaches to ARC-like tasks (Ainooson et al.) 2023 Assouel et al.) 2022, Ferré 2021 Fischer et al. 2020) have produced some promising results, as of this writing the first-place ARC-Kaggle program remains the most successful single approach (though as we described above, an ensemble of the top two winning programs attained higher accuracy). As yet, there is no AI system that is close to reaching human accuracy and generalization abilities on ARC tasks. The ARC domain remains a wide-open challenge for AL.
10 Conclusions and Future Work
In this paper we have described ConceptARC, a new benchmark sct of tasks in the ARC domain. The tasks in ConceptARC are designed to systematically test. conceptual understanding and generalization while remaining relatively easy for humans. Our purpose in designing benchmark with these attributes is threefold: first, to promote the development of AI systems that grasp generalizable core concepts and are able to use them in new situations; second, to fairly evaluate systems that are claimed to have such abilities; and third, to provide an ion set that is not overly diffieult, and that would thus mask real progress in developing such systems.
In addition to describing and publishing the ConceptARC benchmark, we have reported results of testing humans and machine solvers on these tasks. Our results show that humans substantially outperform state-
12

---

Published in Transactions on Machine Learning Research (08/2023)
of-the-art programs on all the concepts in our benchmark; morcover, when humans make errors, they often still exhibit a grasp of the underlying concept, unlike the programs. We also showed that our benchmark is able to reveal differences in performance among machine solvers that were masked by the difficulty of the original ARC dataset. In addition, we showed that GPT-4's performance, while impressive given that in many cases it exceeded the second-place ARC Kaggle program even though it was not designed for or trained on these tasks, is dramatically below that of humans, which contrasts with the results of Webb et al. (0022) in testing GPT-4 on other idealized analogy domains.
As described in [Section 5) in addition to asking participants to solve tasks, also asked them to write natural language instructions for solving a given test input. In the near future we will perform a new study with human participants to test the viability of these instructions, by giving a test input (without accompanying demonstrations) along with the corresponding instructions, to see people can arrive at the correct solution by following these instructions. Following Acquaviva et al. (2022), ; we will take the viable instructions and use them as part of a training set for a new machine ARC solver, to see augmenting training with language inputs will improve performance.
In the future we also plan to extend the ConceptARC benchmark to encompass additional tasks and concept groups, and to further evaluate humans and machine solvers on these new tasks, as well as to more thoroughly analyze the different types of errors made by humans and machines. In particular, in addition to the tasks that we make publicly available, we will create a "hidden" " evaluation set that can be used in future ARC competitions.
When solving task in the ARC domain, humans bring to bear not only their core knowledge about the world but also highly evolved visual system that is not present in any of the proposed machinc solvers or in GPT-4. While the grids in an ARC task are visually simple, it may be that incorporating routines inspired by the visual system (Ulmann 1987) into program-sy synthesss approaches could be a way to make progress on these tasks. We plan to explore this hypothesis in future work. We also plan to test the multimodal version of GPTT4 on ARC tasks, once it made publicly available.
Acknowledgments
This material is based in part upon work supported by the National Science Foundation under Grant No. 2139983. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation. This work has also been supported by the Templeton World Charity Foundation, Inc. (funder DOT 501 10001 1730) under the grant httt::://oii.orrg/0.555////////
References
S. Acquaviva, Y. Pu, M. Kryven, T. Sechopoulos, C. Wong, G. Ecanow, M. Nye, M. Tessler, and J. Tenen- baum. Communicating natural programs to humans and machines. Advances in Neural Information Processing Systems, 35:3731-377 -3743,, 2022.
J. Ainooson, D. Sanyal, J. P. Michelson, Y. Yang, and M. Kunda. An approach for solving tasks on the Abstract Reasoning Corpus, 2023. 0.425
S. Alford, A. Gandhi, A. Rangamani, A. Banburski, T. Wang, S. Dandekar, J. Chin, T. Poggio, and P. Chin. Neural-guided, bidirectional program search for abstraction and reasoning. In Proceedings of the Tenth International Conference on Complex Networks and Their Applications, pages 657 668. Springer, 2022.
R. Assouel, P. Rodriguez, P. Taslakian, D. Vazquez, and Y. Bengio. Object-centric compositional imagination for visual abstract reasoning. In ICLR Workshop on the Elements of Reasoning: Objects, Structure and Causality, 2022.
A. Banburski, A. Chandi, S. Alford, S. Dandekar, P. Chin, and T. Poggio. Dreaming with ARC. Technical report, Center for Brains, Minds and Machines (CBMM), 2020.
M. M. Bongard. Pattern Recognition. Spartan Books, 1970.
13

---

Published in Transactions on Machine Learning Research (08/2023)
S. Carey. The Origin of Concepts. MIT Press, Cambridge, MA, 2011.
P. A. Carpenter, M. A. Just, and P. Shell. What one intelligence test measures: A theoretical account of the processing in the Raven progressive matrices test. Psychological Review, 97(3):404 431, 1990.
F. Chollct. On the measure of intelligence, 2019. arXiv:1911.0154..
F. Chollet. The Abstraction and Reasoning Corpus (ARC), 2023. http/httpss/gtttuu..c/////clllld Online; last accessed May 4, 2023.
A. de Miquel Bleier. Finishing 2nd in Kaggle's Abstraction and Reasoning Challenge, 2020. URL https:// abstraction blog. blog.jovian com f wwfffinssiiie wwwiinniiinnnnin 2nd-in 2ndgie- - and hha.aaaanccaaleleee-eeeeee0////
K. Ellis, C. Wong, M. Nye, M. Sablé-Meyer, L. Cary, L. Morales, L. Hewitt, A. Solar-Lezama, and J. B.
Tenenbaum. DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learning, 2020. arXiv:2006.0888/
S. Ferré. First steps of an approach to the ARC challenge based on descriptive grid models and the minimum description length principle, 2021. 12.00848.
R. Fischer, M. Jakobs, S. Mücke, and K. Morik. Solving abstract reasoning lasks with grammatical evolution. In Proceedings of the Conference Lernen, Wissen, Daten, Analysen, LWDA, pages 6-10, 2020.
H. E. Foundalis. Index of Bongard Problems, 2023. URL http://t/////////oaaaaaaaa////// htm Online; last accessed May 4, 2023.
R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665-677 2020.
D. R. Hofstadter. Fluid Concepts and Creative Analogies: Computer Models of the Fundamental Mechanisms of Thought. Basic Books, New York, 1995.
D. R. Hofstadter and M. Mitchell. The Copycat project: model of mental fluidity and analogy-making. In K. Holyoak and J. A. Barnden, editors, Advances in Connectionist and Neurol Computation Theory, volume 2, pages 31-112. Ablex Publishing Corporation, 1994.
A. Johnson, W. K. Vong, B. M. Lake, and T. M. Gurockis. Fast and flexible: Human program induction abstract reasoning lasks, 2021. arXiv:2103.05823
Kaggle.com. Kaggle Abstraction and Reasoning Challenge, 2020. URL https://www.kagg///
alstaccioo--rddrrasssanaa rnd-ressnnnng---eee-eee--en--n caanessonnnn-calleegge challenge.
S. Kim, P. Phunyaphibarn, D. Ahn, and S. Kim. Playgrounds for abstraction and reasoning. In NewrIPS Workshop on Neuro, Causal, and Symbolic Al (NCSI), 2022.
B. M. Lake, T. D. Ullman, J. B. Tenenbaum, and S. J. Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40:c253, 2017.
S. Lapuschkin, S. Wäldchen, A. Binder, G. Montavon, W. Samek, and K.-R. Müller. Unmasking Clever Hans predictors and assessing what machines really learn. Nalure Communications, 10(1):1096, 2019.
A. Lovett and K. D. Forbus. Modeling visual problem solving analogical reasoning, Psychological Review, 124(1), 2017.
M. Malkińíski and J. Mańdziuk. Deep learning methods for abstract visual reasoning: A survey Raven's Progressive Matrices, 2022a. arXiv:2201.12382.
M. Malkiński and J. Maúdziuk. A review of emerging research directions in abstract visual reasoning. Information Fusion, 2022b.
J. McCarthy, M. L. Minsky, N. Rochester, and C. E. Shannon. A Proposal for the Dartmouth Summer Research Project in Artificial Intelligence, 1955. Reprinted in AT Magazine, 27, no. 4, 2006, 12-14.
14

---

Published in Transactions on Machine Learning Research (08/2023)
M. Mitchell. Abstraction and analogy-makinn making in artificial intelligence. Annals of the New York Academy of Sciences, 1505:79-100 100, 2021.
M. Mitchell and D. C. Krakauer. The debate over understanding in Al's large language models. Proceedings of the National Academy of Sciences, 120(13):e221500772) 2023.
W. Nie, Z. Yu, L.. Mao, A. B. Patel, Y. Zhu, and A. Anandkumar. Bongard-T.OGO: A new benchmark for human-level concept learning and reasoning. Advances in Neural Information Processing Systems, 33: 16468-16480,, 2020.
V. V. Odouard and M. Mitchell. Evaluating understanding on conceptual abstraction benchmarks, 2022. arXiv:2206.14187
OpenAL GPT-4 Technical Report, 2023. arXiv:2303.08774.
Sonwane, S. Chitlangia, T. Dash, L. Vig, G. Shroff, and A. Srinivasan. Using program synthesis and inductive logic programming to solve Bongard problems, 2021. arXiv:2110.09947.
E. S. Spelke and K. D. Kinzler. Core knowledge. Developmental Science, 10(1):89-96, 2007.
S. Ullman. Visual routines. In Readings in Computer Vision, pages 298 328. Elsevier, 1987.
T. Webb, K. J. Holyoak, and Н. Lu. Emergent analogical reasoning in large language models, 2022. arX1v:2212.09196.
J. S. Wind. DSL Solution to the ARC Challenge, 2020a. URL http:://gitnub..oo/boooog/
ARC
www.:ootttoccmmttiiionnmoo pdf
J. S. Wind. 1st place solution code and official documentation, 2020b. URL https://ww/ wttp:///wwwkkggl..... waggle. com/
Y. Xu, E. B. Khalil, and S. Sanner. Graphs, constraints, and search for the Abstraction and Reasoning Corpus, 2022. 100.802200.00880.
C. Zhang, F. Gao, B. Jia Y. Zhu, and S.CC. Zhu. RAVEN: A dataset for relational and analogical visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pages 5312-5322, 2019.
15

---

Published in Transactions on Machine Learning Research (08/2023)
Appendix A Examples of Tasks From Each Concept Group
Above and Below
Center
Clean Up
Demonstrations
Demonstrations
Demonstrations
Test Input
Test Input
Test Input
Complete Shape
Copy
Count
Demonstrations
Demonstrations
Demonstrations
Test Input
Test Input
Test Input
?
16

---

Published in Transactions on Machine Learning Research (08/2023)
Extend to Boundary
Extract Objects
Filled and Not Filled
Demonstrations
Demonstrations
Demonstrations
Test Input
Test Input
Test Input
?
Horizontal and Vertical
Inside and Outside
Move Boundary
Demonstrations
Demonstrations
Demonstrations
2
2
-
H
Test Input
Test Input
Test Input
?
?
17

---

Published in Transactions on Machine Learning Research (08/2023)
Order
Same and Different
Top and Bottom 2D
Demonstrations
Demonstrations
Demonstrations
Test Input
Test Input
Test Input
Top and Bottom 3D
Demonstrations
Test Input
?
18

---

Published in Transactions on Machine Learning Research (08/2023)
Appendix B
Examples of Minimal Tasks
Above and Below Minimal
Clean Up Minimal
Inside and Outside Minimal
Demonstrations
Demonstrations
Demonstrations
Test Input
?
Test Input
Test Input
?
Appendix C Participant Recruitment Details in Our Human Study
In order to establish the data collection regime that yields the highest quality data, we introduced minor recruitment and procedure adjustments after the start of data collection.
In the first batch of participants collected via Amazon Mechanical Turk, each received 11 problems (this batch also only had two "minimal Problems," as opposed to three such problems for everyone else). However, preliminary data examination showed that some participants did not fully follow the study instructions and had to be excluded (see [Section 5.2 In response, we made the screening criteria more strict (requiring Master Worker qualification, 99% of HITs approved with at least 2000 HIT history, as opposed 95% approval requirement in the first batch). Participants in all but the first batch were paid $10 upon completing the experiment. Participants in the first batch were paid $5. In all batches, the median pay-per-hovd hour exceeded the U.S. minimal wage.
Additionally, since participant quality was very "bimodal" (i.e. each participant either diligently followed instructions on all tasks, or ignored instructions on all tasks and were thus excluded), we increased the number of tasks per participant, so that non-excluded participants provided us with more data.
Lastly, after the first large batch of participants, we transitioned the study another crowdsourcing platform: Prolific.org. This done both due to technical reasons and to diversify the data source.
19

---

Published in Transactions on Machine Learning Research (08/2023)
Appendix D Accuracies With Binomial Confidence Intervals
Table 2: Accuracies of humans, the two top-scoring ARC-Kaggle programs, and GPT-4 (with temperatures and 0.5) on test inputs in each concept group in ConceptARC (same Table 11, with binomial confidence intervals added.
Concept
Humans
ARG-Kaggle First Place
ARC-Kaggle Second Place
GPT-4 Temp
GPT-4 Temp 0.5
Above and Below
0.90 (0.86, 0.93)
0.70 (0.52, 0.83)
0.33 (0.19, 0,32)
0.23 (0.12, 0.11)
0.37 (0.22. 0.54]
Center
0.94 (0.90, 0.96)
0.50 (0.33, 0.67)
0.20 (0.1, 0.37)
0.33 (0. 19, 0.51)
0.33 (0.19. 0.51)
Clean Up
0.97 (0.94, 0.99)
0.50 (0.33, 0.67)
0.20 (0.1, 0.37)
0.20 (0.1, 0.37)
0.27 (0.14, 0.44)
Complete Shape
0.85 (0.8, 0.89)
0.17 (0.3, 0.64)
0.30 (0.17, 0.48)
0.23 (0.12, 0.41)
0.23 (0.12, 0.41)
Copy
(0.9, 0.96)
(0.12, 0.41)
0.27 (0.14, 0.44)
(0.12, 0.41)
0:27 (0.14, 0.44)
Count
0.88 (0.83, 0.91)
0.60 (0.42, 0.75)
0.40 (0.24,
0.13 (0.06, 0.3)
0.17 10.07. 0.34)
Extend To Boundary
0.93 (0.89, 0,96)
0.77 (0.39, 0.88)
0.47 (0.3, 0.64)
0.07 (0.2. 0.21)
0.1 (0.03, 0.26)
Extract Objects
0.86 (0.81, 0.9)
0.43 (0.27, 0.61)
0.43 (0.27, 0.61)
0.03 10.00, 0. 17)
0.07 (0.02, 0.21)
Filled and Not Filled
0.96 (0.93, 0.98)
0.73 (0.56, 0.86)
0.43 (0.27, 0.61)
0.17 (0.07, 0.34)
0.27 (0.14, 0.44)
Horizontal and Vertieal
0.91 (0.91, 0.94)
0.43 (0.27, 0.61)
0.10 (0.03, 0.26)
0.27 (0.14, 0.43)
0.33 (0.19. 0.51)
and Outside
0.91 (0.91, 0.94)
0.57 (0.39, 0.73)
0.10 (0.03, 0.26)
0.10 (0.03, 0:26)
D.17 (0.07, 0.34)
Move To Boundary
0.91 (0.91, 0.94)
0.37 (0.22, 0.54)
0.30 (0.17,
0.20 (0.1, 0.37)
0.20 (0.1, 0.37)
Order
0.83 (0.78, 0.87)
0.27 (0.14, 0.44)
0.23 (0.12, 0.41)
0.27 (0.14, 0.44)
0.21 (0.14, 0. 44)
Same and Different
0.88 (0.83, 0.91)
0.53 (0.36,, 0.7)
0.17 (0.07, 0.33)
0.17 (0.07, 0.34)
0.27 (0.14, 0. 44)
Top and Bottom 2D)
0.95 (0.91, 0.97)
0.60 (0.42, 0.75)
0.57 (0.39, 0.73)
0.23 (0. 0.41)
0.37 (0.21, 0.54)
Top and Bottom
(0.89, 0.96)
0.50 (0.33, 0.67)
0.03 (0.00, 0.17)
(0.1, 0.37]
(0.14, 44)
20


Reviews:
[{"review_id": "Ny2GgLo1Rk", "reviewer": "Reviewer_5wLx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Neat and timely study that moves beyond traditional benchmarking tasks to settings that are not only more realistic but also more difficult for existing AI systems (e.g., GPT-4). \n- The paper is very well-written and is effectively ready for publication as is.\n\nWeaknesses\n- It's unclear how narrow models optimized for the task itself would perform. Instead of using general purpose learners (e.g. GPT-esque systems). How would a standard supervised learning model (e.g., finetuned ResNet perhaps) perform on each of the concept groups? This would simply provide another baseline to compare against.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an evaluation task that extends the Abstraction and Reasoning Corpus (ARC) to cover conceptual \"wrappers\" to the above benchmark (e.g., above, center, inside/outside). The authors then benchmark ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Sorry if I missed it but did you discuss why/how each concept group in ConceptARC was selected? It would be good to provide some methodological description for why these were chosen.\n- It would good to understand how easily others can leverage this dataset, perhaps including a code snippet or release information.", "broader_impact_concerns": "Can you please include a brief discussion about manipulation and deception as it pertains to high-level abstract reasoning?", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mdJuOBZ8y7", "reviewer": "Reviewer_yaNs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This work is a valuable contribution that will be very useful in the continued effort to develop AI systems with human-like abstract reasoning capabilities. The proposed benchmark makes some important improvements relative to the original ARC benchmark, and the human behavioral experiment and evaluation of GPT-4 / heuristic programs are thorough and well documented.\n\nI have a number of comments (detailed below), but I think the paper is already in good shape, so these are really just suggestions (some of which I imagine won't be feasible to address in this particular study).\n\n#### Comments:\n\n- It would be helpful to add a bit more discussion concerning which features of the conceptARC problems make them more difficult for GPT-4 than the problems in Webb et al. (2022). There are a few places in the paper where the problems from Webb et al. are described as more 'idealized' but there is not much discussion otherwise. One essential difference seems to be that the conceptARC problems require object segmentation from pixel-level inputs, whereas the problems in Webb et al. (e.g. digit matrices) only involve analogies applied to individual tokens. This also relates to the first element of core knowledge listed on pg. 3, 'objectness'.\n- The evaluation of error types (e.g. near misses vs. failing to grasp the concept altogether) was somewhat unsystematic. First, one of the descriptions of a common human error type involved 'giving up', which doesn't necessarily sound like a careless mistake -- it seems like this could just as easily result from a failure to grasp the underlying concept. Second, there is no description of how errors were categorized, and no quantification of the frequency of different error types. But this does seem like a potentially important aspect of the evaluation, especially if it reveals a systematic difference between humans vs. programs. One possible way to improve this section is to have blind raters judge whether errors are more likely the result of careless mistakes vs. failure to grasp the underlying concept.\n- In Webb et al., analogy problems were presented using line breaks ('\\n') to indicate the spatial layout of a grid, whereas it appears that GPT-4 was evaluated on conceptARC without this formatting. This seems like it could potentially affect performance. Insofar as LLMs have exposure to 2D pseudo-visual arrays, it seems likely that it often involves explicit line breaks of this sort.\n- Relatedly, I wonder how human participants would perform if given the problems in the same format as GPT-4. It does seem that it might be difficult in some cases to identify objects when the problems are formatted in this way. \n- Were human participants given feedback when they got problems correct? If so, an interesting comparison would be to see how GPT-4 performs when given similar feedback. In principle it could benefit from in-context learning based on this feedback.\n- It would be helpful to include supplementary figures depicting some of the 'minimal' problems used to determine subject exclusion in the human study. There is a concern that these are not sufficiently minimally difficult to be treated as mere attention checks.\n- It would be informative to include some measure of error for the performance estimates (in both humans and programs). Binomial confidence intervals seem like a natural choice. It might also be helpful to plot the results, either instead of or in addition to providing a table.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes ConceptARC, an extension to the Abstraction and Reasoning Corpus (ARC) benchmark, that is intended both to be more reliably solvable by humans, and to systematically explore the role of specific concepts. The study also presents results on this new benchmark for human participants, GPT-4, and two heuristic program designed to solve ARC problems. The basic finding is that human participants reliably perform well, significantly outperforming both the heuristic programs and GPT-4.\n\n\n\n\n\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I do not have any requested changes, but the authors may wish to address some of the concerns detailed above.", "broader_impact_concerns": "I do not believe there are any ethical implications that would necessitate such a statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gATgIcUHSD", "reviewer": "Reviewer_v6Vo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n* The paper is very well written. The prose is clear and persuasive. The organization is good. The motivation, experiments, and results are well-explained.\n* I particularly appreciated the comparison between human and machine errors in Section 8.2.\n* I also appreciated the thorough discussion of limitations of the work in Section 8.3.\n* The experimental design seems sound. I did not find any issues related to the experiments or results.\n* The connections to related work are also good.\n\n## Weaknesses\n* The paper suggests that all of the tasks in a concept group are related by a shared concept. There are two possible interpretations of this, each with potential issues. (1) Tasks within a concept group are designed to evaluate _only_ that concept. (2) Each task within a group involves _many_ concepts, but the _intersection_ of concepts within a group has just one shared concept.\n   * The issue with (1) is that I don’t think it is actually possible to evaluate just a single concept. For example, the tasks in the Sameness group shown in Figure 2 also involve objects, shapes, and colors. One could say that objects, shapes, and colors are not “concepts”, but that would require a careful (and probably contentious) argument.\n   * The issue with (2) is that we might as well take the original ARC tasks and organize them by concept, rather than making new tasks.\n* The notion of a “concept” is not well-defined in this work. The “core knowledge” that ARC uses is backed by cognitive science research. Is there any criteria that one could use to determine whether something is _not_ a “concept” for the purposes of ConceptARC? Nonetheless, I am willing to look past this issue because there is a long line of work on concept learning in AI, and it is always hard to precisely define the meaning of “concept”.\n* I was also sometimes confused about the word “generalization” as used in this work. For example: “the much lower accuracies of programs we tested indicates a lack of ability to generalize.” Usually, generalization means either (1) a model trained on A performs well on B; or (2) a finding that we observed in setting A can also be observed in setting B. When generalization is mentioned in this work, which kind is it, and what are A and B?\n* Relatedly, I think there is a missed opportunity to examine within- and between- concept group performance, for both humans and machines. In the absence of further results, it would be good to hint at possibilities in this direction.\n* The performance of GPT-4 is said to be “impressive given that it was not designed or trained for such tasks.” It is difficult to know how impressed we should be though, given that it’s the lowest performing method, and this is a new benchmark. It would be great if it were possible to design some simple baseline approach to give further context for these results. Admittedly, I can’t think of a baseline that is simple enough to easily understand but not so simple that it gets zero on all tasks.\n* I found it strange that each task has three test inputs, but human participants were each only given one of the three. Why is this better than just having one test input, like most of the original ARC tasks do?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes a new benchmark in the style of Chollet’s Abstraction and Reasoning Corpus (ARC) that differs in two ways. First, rather than presenting all tasks in one large collection, the new tasks are organized into 16 concept groups. Second, the tasks are designed to be easier than those in the original ARC, since the latter may be too difficult to measure AI progress in the near term. Human solvers and three machine solvers are evaluated. The machine solvers include the first and second place winners of the ARC Kaggle competition and GPT-4. Humans substantially outperform all machine solvers, the first place winner outperforms the second place winner, and GPT-4 performs worst overall.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Overall, I think the paper is already in very good shape. Here are a few small things:\n\n* The related works that develop easier ARC tasks (Kim et al. 2022 in particular) are not mentioned until Section 9. These works seem extremely related, but since this is TMLR, novelty doesn’t matter. Nonetheless, it would be good to mention these works in the introduction and/or where the motivation to simplify ARC is first mentioned.\n* Can you clarify how 3 different guesses per test input are collected from GPT-4? Since the temperature is set to be 0, I would have thought that the 3 guesses are often identical. I also would expect that a higher temperature might be better, though I do see that the Webb et al. work also used a temperature of 0.\n* In the limitations section, it says: ““In addition to these limitations, our studies revealed that there are a small number of tasks in the ConceptARC corpus that are ambiguous—that is, for which test inputs have more than one reasonable solution.” Can you elaborate? Which tasks are these, and how did you make this determination?\n\n### Minor\n* Typo: “that are vary in complexity”\n* “Pretrained” vs. “pre-trained” both used\n* Typo: “each of these concepts is central to in one or more tasks”\n* “Psiturk” vs. “psiTurk” both used", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission proposes an extension to the original Abstraction and Reasoning Corpus (ARC) by incorporating concept groups and systematic human studies. All three reviewers find the submission addressing a timely topic, the studies thorough and rigorous, and the findings useful to the community.  Eventually, all reviewers recommended acceptance.  The AE agrees. 
Decision: Accept (poster)