LLMs and the Abstraction and Reasoning Corpus: Successes, Failures, and the Importance of Object-based Representations
Authors: Yudong Xu
Wenhao Li
Pashootan Vaezipoor
Scott Sanner
Elias Boutros Khalil
Venue: Accepted by TMLR ()

Abstract: Can a Large Language Model (LLM) solve simple abstract reasoning problems? We explore this broad question through a systematic analysis of GPT on the Abstraction and Reasoning Corpus (ARC), a representative benchmark of abstract reasoning ability from limited examples in which solutions require some "core knowledge" of concepts such as objects, goal states, counting, and basic geometry. GPT-4 solves only 13/50 of the most straightforward ARC tasks when using textual encodings for their two-dimensional input-output grids. Our failure analysis reveals that GPT-4's capacity to identify objects and reason about them is significantly influenced by the sequential nature of the text that represents an object within a text encoding of a task. To test this hypothesis, we design a new benchmark, the 1D-ARC, which consists of one-dimensional (array-like) tasks that are more conducive to GPT-based reasoning, and where it indeed performs better than on the (2D) ARC. To alleviate this issue, we propose an object-based representation that is obtained through an external tool, resulting in nearly doubling the performance on solved ARC tasks and near-perfect scores on the easier 1D-ARC. Although the state-of-the-art GPT-4 is unable to "reason" perfectly within non-language domains such as the 1D-ARC or a simple ARC subset, our study reveals that the use of object-based representations can significantly improve its reasoning ability. Visualizations, GPT logs, and data are available at https://khalil-research.github.io/LLM4ARC.

Published in Transactions on Machine Learning Research (02/2024)
LLMs and the Abstraction and Reasoning Corpus: Successes, Failures, and the Importance of Object-based Representations
Yudong Xu Deportment of Mechanical & Industrial Engineering, University of Toronto
wil.nu@mail wtw.emmmoouuuoooooooooo
Wenhao Li
Department of Mechanical & Industrial Engineering, University of Toronto
Pashootan Vaezipoor
pashantan@Gcs.oronto edu
Deportment of Computer Science, University of Toronto Vector Instiutte for Artificial Intelligence
Scott Sanner
Department of Mechanical & Industrial Engineering, University of Toronto
Elias B. Khalil
Deportment of Mechanical & Industrial Engineering, University of Toronto
Scale AI Research Chair in Data- Data-Driven Algorithms for Modern Supply Chains
Reviewed on OpenReview: https: openreview. net/ forum? id= E8m8oySuPJ
Abstract
Can a Large Language Model (LLM) solve simple abstract reasoning problems? We explore this broad question through a systematic analysis of GPT on the Abstraction and Reasoning Corpus (ARC) (Chollet] 2019), representative benchmark of abstract reasoning ability from limited examples which solutions require some "core knowledge" of concepts such objects, goal states, counting, and basic geometry. GPT-4 solves only 13/50 of the most straightforward ARC tasks when using textual encodings for their two-dimensional input-output grids. Our failure analysis reveals that GPT-4's capacity to identify objects and reason about them is significantly influenced by the sequential nature of the text that represents an object within a text encoding of a task. To test this hypothesis, we design a new benchmark, the 1D-ARC, which consists one-dimensional (array-like) tasks that are more conducive GPT-based reasoning, and where it indeed performs better than on the (2D) ARC. To alleviate this issue, we propose an object-based representation that is obtained through external tool, resulting in nearly doubling the performance solved ARC tasks and near-perfect scores the easier 1D-ARC. Although the state-of-the-art GPT-4 is unable to "reason" perfectly within non- language domains such as the ID-ARC or a simple ARC subset, our study reveals that the use of object-basod reprosentations can significantly improve its reasoning ability. Visualizations, GPT logs, and data are available at htttp::/khaalll-rreeeses wetieeearrhhtigthh.h github. 1o/LLM4ARC,
1 Introduction
It has been recently claimed that Large Language Models (LLMs) such as GPT-4 OOpeAAl 2023a exhibit
"sparks of artificial general intelligence" (Bubeck et al. 2023 As result, the impressive question-answe wnwweeing and text generation abilities of pre-trained LLMs are already being deployed in rather consequential e- commorce and educational setting [ LLMs are to be used to reliably solve complex, noisy, real-world
http:://gpenas.cc///b////tttnnne chatgigt------cttt-t---t---eeeee aadbatgt--and-viiee-------

---

Published in Transactions on Machine Learning Research (02/2024)
problems, one would expect them to be capable of reasoning in simple, unambiguous, idcalized settings. By "reasoning", we here mean "using evidence, arguments, and logic to arrive at conclusions or make judgments", as defined in (Huang and Chang 2022). While the performance of LLMs on arithmetic and language-based commonsense reasoning benchmarks has been the subject of recent analyses (sce for example Section 4.1 of (Huang and Chang 2022) for a brief survey); it is unclear whether LLMs exhibit the ability to generate abstract concepts based on a handful of "training" samples (Odouard and Mitchell, 2022).
To quantitatively mcasure the gap between machine and human learning, the Abstraction and Reasoning Corpus (ARC) was introduced in (Chollet 2019 The author advocates for leveraging human-level intelligence as frame of reference for evaluating general intelligence. To that end, he draws upon the work of developmental psychologists Spelke and Kinzler (2007) on the theory of Core Knowledge to determine axes along which human-like intelligence should be measured. Core Knowledge identifies four broad categories of innate assumptions that form the foundation of human cognition:
Objectness: The ability to perceive the surroundings consisting of cohesive, persistent, and non-interpenetrating objects.
Agentness and goal-directedness: as intentional agents with goals, capable of contingent and reciprocal actions, while distinguishing them from inanimate objects.
The tendency to perceive certain objects in the environment
Numerical knowledge: Innate knowledge of abstract number representations for small numbers and the concepts of addition, subtraction and comparison between those numbers.
Elementary geometry and topology: Knowledge of distance and basic 2D and 3D shapes.
The ARC, benchmark of 1,000 image-based reasoning (asks, is thus proposed as test of the above four core knowledge systems in humans or AI systems. Each task requires the production of output image given a specific input, with 2 to 5 input-output image pairs provided training instances to "learn" the underlying procedure (Figure 2b. The training inputs are different from the actual test input, though they are solvable using the same (unspecified) procedure. Crucially, no acquired knowledge outside of the aforementioned priors is required to solve these tasks. Note that although these priors are explicitly described, the ARC tasks remain completely open-ended: objects can have different shapes and colors and form various relations with one another, and the grid size can also vary between tasks. This feature makes these problems not amenable to solving through search. This is in contrast to games like Go and Chess (Silver et al. 2018 2017] where the search space is large but the set of moves is finite and fixed. In fact, so far the approaches that have employed heuristic search via a predetermined set of transformations have all fallen short of generalizing to the hidden tasks; see Section 5
Given the vast corpus of hurman knowledge that LLMs are typically trained on, one might wonder whether they could have acquired the priors listed above. To answer this question, we conducted a comprehensive study of GPT-3.5 and GPT-4 on the ARC. We contribute three high-level findings that we believe shed light some intrinsic limitations of the LLM framework and, to some extent, how to resolve them:
GPT fails on simple ARC tasks: Using pure text encodings (Figure 3) of tasks such as those in Figure 2 GPT-4 can only solve 13/50 of the simplest ARC tasks (Section We conduct a failure analysis which revcals that the culprit is the LLM's inability to maintain "object cohesion" " across the lines of text that represent the ARC image grids.
2. GPT does better when objects can be easily detected in text: We hypothesize that the issue raised in point 1 is tied to the two-dimensional nature of the ARC grids. In Section 3 we introduce the ID-ARC benchmark, a set of ARC-like tasks that can be represented as a single line of text. Relative to the 50 ARC tasks, GPT performs better on 1D-ARC but is far from perfect.
An object-based representation boosts GPT's performance significantly: Leveraging the object-centric graph abstractions of the ARC solver Abstract. Reasoning with Graph A Abstractions
2

---

Published in Transactions on Machine Learning Research (02/2024)
ARC Task
Direct-grid (Section
GPT
GPT Output
ARC Solution
Based on the demonstrations, the transformation rule seems to creating a new object (Object
Text Encoder
Prompt
(Figure 4)
3) with color green and size 5, positioned the midpoint between the two existing objects (Object I and Object and forming cross shape.
Object-based (Section 4)
Instructions
Examples
ARGA
?
Text Encoding (Figure 3.8)
Output Parser
Here's the output image for the test input:
Figure 1: Example of solving an ARC task with GPT. An "ARC Task" consists of a set of training input-output pairs followed by a lest imput for which GPT should produce a correct output. To do so, a prompt created. It includes high-level instructions on what GPT should do, optionally with additional in-context examples. A text encoding of the ARC task of interest is also included in the prompt. The encoding may be direct represenlation of the 2-dimensional grids or object representation produced by external ARC solver, ARGA. GPT must then "reason" about the prompt to produce answer. The output is then parsed and checked for correctness.
Task 6e82a1ae
Task 25ff71a
Task 3906de3d
?
?
Figure/ 2: Sample ARC Tasks. Three tasks (separated by dashed lines) are shown. For a given task, each column contains example input-output pair. The first two columns contain the "training' instances and the third column contains the "test" instance. The goal is to the training instances to solve the test instance. The left task ("Recolor by size") requires recoloring the grey objects to green, red, blue based on if their size is 2, 3, or 4. The middle task ("Static movement") requires moving the non-black object down pixel. The right task (*Dynamic movement.") requires moving the red objects up towards the blue objects until they make contact.
(ARGA) (Xu et al. 2023). we provide the LLMs with more structured object-based representation of the onput-output ARC grids (Section 4p. This results in significant jump in performance, where GPT-4 produces the correct output for 23 instead of 13/50 tasks and achieves near-perfect scores on many 1D-ARC task types. Further experiments on additional LLM models such as LLAMA- 2 (Touvron et al. 2023) and datasets such as the full ARC training set and the Mini-ARC (Kim et al. 2022). although not the focus this paper, exhibit a similar pattern.
Given the increasing interest from the artificial intelligence community in the reasoning capabilities of pre-trained LLMs and the unique characteristics the ARC (and ID-ARC), we believe that our work contributes to research on imbuing LLMs with such capabilities. We demonstrate that the use of an external tool that produces appropriate representations crucial. We hope that our experimental design on the ARC, the new ADC dataset, and the integration of domain-specific external tool for improved representation will be useful in generating new ideas at the intersection of LLMs and reasoning.
3

---

Published in Transactions on Machine Learning Research (02/2024)
00000000001n
0000000000'n
0000000000ln
0000300000'n
0103330100kn
0000300000kn
0000000000In
0000000000hn
0000000000hn
0000000000
Figure 3: Visualization of direct-grid encodings. Left: each pixel is represented with a number corresponding to the pixel color, with no delimiters. Right: each pixel is represented with the color descriptor, separated by the delimiter "1º. The lext string has been formatted for easier reading.
2 A first attempt at solving ARC tasks with an LLM
The task of solving the ARC with an LLM necessitates the encoding of two-dimensional (2D) input-output images using a textual representation A text-cncoded ARC task is incorporated into an LLM prompt, which then generates the solution. This section proposes a straightforward pipeline and evaluates it before mining the results to understand failure modes.
2.1 Textual encoding
A 2D grid with colored pixels can be directly encoded into text by representing each pixel's color either numerically (using values from to 9, each representing one of the ten colors) or with color descriptors (c.g., "blue", "green", "black"). delimiter delineates between adjacent pixels and "newline" characters were used to separate the rows in an image. We assessed the impact of different delimiters (",", "I", or no delimiter) on LLM performance. Figure 3 provides two visual examples of this direct-grid encoding.
2.2 Prompting and strategy
After encoding ARC images into text, we incorporate the latter into prompts that instruct the LLM to solve the task at hand. We explored two single-stage strategies for prompting the LLM.
Few-shot learning: By design, an ARC task is a few-shot learning task, providing handful of training examples for generating a solution for a test examplc. Therefore, adopting the few-shot learning stratogy is the most straightforward and intuitive initial approach for leveraging the inherent structure of ARC tasks. The prompt created using this strategy has two main sections: "instructions" and "task". The "instructions" section outlines the nature of an ARC task and the expected behavior of the LLM; it is the same across all tasks. The "task" section provides information about the single ARC task of interest, including its few-shot examples. This approach, in line with the classic few-shot learning concept, encourages the LLM to leverage the provided examples to solve the task. An example of this prompting strategy can be found in Figure 4
In-context few-shot learning with chain-of-thought: Building on the few-shot learning strategy and drawing inspiration from chain-of-thought (CoT) prompting introduced in (Wei et al. 2022), we investigate a natural combination thereof. This approach enriches the learning context for the LLM by augmenting the original prompt with an "examples" section, which includes two simple ARC-like tasks differrent from the actual task of interest-and their step-by-step solutions. This strategy not only leverages the inherent task examples but also provides a stable learning base of in-conteat ezamples for the LLM, encouraging a CoT response. As such, it assesses the I.M''s capacity to generalize and apply knowledge acquired from limited set of contextual examples to solve similar task. An example prompt using this approach can be found in Figure
At the time of our research, GPT-4's vision API, which enables the model to accept image inputs, not yet available. Consequently, this paper does focus on multimodal capabilities of GPT-4. However, provide glimpses into these capabilities and their palential Section 0 and Appendix
4

---

Published in Transactions on Machine Learning Research (02/2024)
Few-shot Learning
In-context Few-shot Learning with CoT
Model Input
Model Input
You are chatbot with human-like reasoning and inference capabilities, adept at solving tasks concisely.
You'll tasked with solving tasks involving input grids objects that transform output grids. Your goal is discem the transfornations applied to the input achieve the corresponding output.
Instructions
Instructions
Let's engage reasoning and logic-based lasks. Each task demonstrate a transformation from an input grid to an output grid. At the end, you'll receive new input grid. Your task is to determine its corresponding output grid and describe the transformation steps from input grid.
Task 1: Input grid 1: 0000
Becomes output grid 1:
0000
0620
0020
0020
Examples
0000
0000
Answer:
Transformation applied:
Demonstrations:
1.Moee object color object until they touch.
Input grid
111
Becomes output grid 1:
000
000
111
Task 2:
Input grid 1:
111
Becomes output grid 1:
000
000
000
000
111
Task
000
000
Input grid 2:
022
Becomes output grid 2:
000
002
022
Task
Input grid 2:
022
Becomes output grid 2:
000
000
002
002
022
000
002
Input grid 3:
200
200
Input grid 3:
200
Becomes output grid
000
200
What input grid become?
000
Model Output
Model Output
000
Based the demonstrations, the transformation rule seems to be moving the object down one
200
200
pixel. Here's s output grid 3: 000
200
Answer: Transformation applied:
200
1. the non-0 object down by one pixel.
Figure 4: Example prompts. Left: Few-shot Learning. Right: In-context Few-shot Learning with CoT. The prompt texts have been formatted for easier reading (for example, the "Task" string on the left is provided to the LLM as "Demonstrations:)/llae
grid 1: 1000000000.0..
Table Direct-grid variants, performance comparison. Each row corresponds to a variant of a direct-grid encoding. Each column corresponds to combination prompting method with either GPT-3.5 GPT-4. GPT solutions were obtained through OpenAT's APT with temperature set to 0. The values correspond to the number of tasks, out of 50, solved by each method; higher is better and top-performers are bolded.
Direct-grid encoding
Few-shot
In-context Few-shot w/ CoT
Pixel
Delimeter
GPT-3.5
GPT-4
GPT-3.5
GPT-4
Number
Number n/a
12
Word
3
12
13
Word
4
13
2.3 Results
Given that the most advanced ARC solvers achieve only a 30% accuracy on the hidden Lest set of the ARC, we strategically selected a subset 50 ARC tasks. These tasks were among the "easiest", allowing our resources be more efficiently allocated for in-depth experimentation. Our goal was upper-bound the LLM's performance on the ARC as whole; note that we do discuss results on all 400 ARC training tasks in the following paragraph and Section 4.2 but restrict much of the analysis in this paper to the selected 50

---

Published in Transactions on Machine Learning Research (02/2024)
tasks. We define "easy" tasks those that have been previously addressed using the symbolic search-based method, ARGA (Xu et al. 2023). This designation stems from the fact that ARGA's implementation confines its solution space to 15 functions, in contrast to seate-of-the-rr models which can have a more expansive function set, with the Kaggle first-place solution, for example, encompassing 42 functions. Thus, studying this subset provides clearer understanding how a purely search-based solver, with a restricted function set, tackles ARC tasks.
The top-performing pixel representation and delimiter combination (Word + "I") solves only 13 out of the 50 tasks, as shown in Table I Using the top-performing representation, assessed the complete ARC training set and GPT-4 managed to solve 81 out of the 400 tasks. Furthermore, on the Mini-ARC datasct-a simplified version of ARC detailed in Section 5 this approach solved 35 out of the 149 tasks. These accuracy rates are consistent with the performance observed on the subset 50 tasks. In the following section, we will delve into the reasons why the LLM struggled with these tasks given that they are easy for a non-LLM method.
2.4 Analysis
We started our analysis by extracting key attributes such as pixel and color counts from each ARC task. We then appliod logistic regression to explore potential relationships between these features and the performance of the LLM.
An intriguing finding from analysis is that the number of colored pixels in a test image is associated with a notable negative coefficient, indicating a potential inverse relationship with the LLM's ability to solve lasks. Since a set of adjacent colored pixels often corresponds to an object in the ARC, this finding suggests that tasks with fewer objects are more likely to be solved by the LLM. Conversely, we find a positive coefficient associated with the average number colored pixels in training images, implying a possible positive correlation with task solvability. This could suggest that more colored pixels in training images provide more learning material for the LLM, potentially improving performance. The full set of features studied can be found in Appendix
A closer examination of the tasks that GPT-4 solved correctly using the direct-grid approach reveals some interesting patterns in the reasoning provided by the model. Out of the 13 tasks that were correctly solved, only three tasks were accompanied by the correct reasoning steps. Surprisingly, for some tasks, GPT did not provide any reasoning at all, despite the presence of reasoning examples within the In-context Few-shot Learning with CoT prompts. This inconsistency in the application of reasoning illustrates possible gap in GPT's understanding and application of the reasoning process, which further complicates the task of solving ARC problems. An example of task where the reasoning provided by the model was incorrect despite achieving the correct output is illustrated in Figure 5 Further examples can be found in Appendix D
Direct-grid approach: Move color green object I pixel down and color yellow object pixel up.
Object-based approach: Move color object pixel towards color 4 object.
?
Figure 5: Reasoning provided by GPT-4 for an example task. Both approaches produced the correct output grid. The direct-grid approach produced the wrong reasoning while the object-based approach produced the correct one.

---

Published in Transactions on Machine Learning Research (02/2024)
C
C*
A
B
B
D
D
A
01100010011111000000000000000000
0000mm10022211100h110002221
A
D
Figure 6: Object cohesion analysis: two example tasks and their textual representations. Generated based on the ARC task seen in Figure (Right), the two tasks are identical modulo the 90-degree rotation. Left: objects A and B are vertical and become non-sequential when represented in text. Right: objects C and D are horizontal and become sequential when represented in text.
2.5 Object cohesion
To further understand the limitations of GPT on ARC tasks, we explored the concept of object cohesion in text, defined the "ability to parse grids into 'objects' based continuity criteria including color continustyy spatiol contiguity, and the ability to parse grids into zones, partitions" (Chollet 2019) Object cohesion is integral part of human cognition (SpeIke and Kinzler 2007 and is assumed to be significant part of the Core Knowledge priors required for ARC solving (Chollet 2019h.
Our objective was to investigate how the textual representation of objects influences GPT's problem- solving capacity. Given that the initial identification and abstraction of objects are pivotal resolving the ARC (Acquaviva, et al. 2022). understanding the impact of textual object depiction the performance of language models is critical. We discovered that GPT's performance deteriorates significantly when objects are not sequentially represented within the text. To further demonstrate this, selected tasks with clear horizontal or vertical objects and manipulated them to adopt the opposite orientation. A visualization of the difference between sequential and non-sequential object representation can be found Figure G
For each "horizontal" or "vertical" original ARC task, we generated a rotated version and compared performance in both the horizontal and vertical configurations. An example visualized in Figure 0 with examples shown in Appendix C The results in the leftmost, part of Table significant performance drop in the vertical case, reinforcing hypothesis. It is evident that GPT struggles with object cohesion when objects are not sequentially arranged within the text. This insight not only deepens our understanding of the model's limitations but also guides us toward potential solutions. In the following section, we delve further into this finding by generating a new dataset that guarantees object soquentialness, and assossing the performance of LLMs on this dataset.
3 Does reduced task dimensionality improve LLM performance?
We introduce ID-ARC, a novel variation on the original ARC that reduces its dimensionality to facilitate future research and provide a more approachable benchmark for LLMs. The 1D-ARC maintains the same Core Knowledge priors as the ARC but restricts the dimensionality of the input and output images one dimension. Consequently, the images comprise only a single row of pixels, significantly reducing the complexity of tasks and enabling all objects to be represented within single sequence. This modification effectively removes the challenge of maintaining object cohesion in non-sequential text. We visualize some example tasks in Figure but the visualization for the full dataset is included in Appendix A
The ID-ARC dataset was strategically designed to adapt transformation types from the original ARC dataset ono-dimensional format. This methodology effectively preserves the core knowledge priors inherent the original ARC. An example of the design and generation process for a ID-ARC task is illustrated in A Appendix B
7

---

Published in Transactions on Machine Learning Research (02/2024)
Recolor by size
Static movement (3 pixels)
Dynamic movement
Figure 7: Example tasks from the 1D-ARC dataset. Each task is inspired by an ARC task; see Figure 2 From left to right: Recolor by size, Static movement by 3 pixels, Dynamic movement (move the block on the left until it touches the green pixel).
Table 2: Results for direct-grid approach. The number of solved tasks is out of 50. The first column is for the 50 tasks from the ARC. The second block of columns for some 1D-ARC task types; results on the full 1D-ARC can be found in Appendix A
The third block is for three task types (Fill, Move, Pile) with horizontal (H) and vertical (V) variants. All models were provided the same prompts for each task.
ARC
Move
Move
Move
Recolor
Fill
Move
Pile
LLM
Denoise
Subset
1 Pixel
Pixels
Dynamic
by Size
II
II
V
II
V
GPT-3.5
2
13
2
0
2
GPT-4
13
33
30
46
12
32
LLAMA2-13B
0
8
4
0
0
0
LLAMA2-70B
2
11
1
0
0
0
Our data generators have been developed to be capable of creating a variety 1D-ARC tasks. They rely on task-specific parameters such the maximum width of the ID sequence, the maximum number of objects, and the maximum size of the objects. This parametric approach ensures tasks can originate from the same foundational concept transformation yet manifest with varying complexities.
These generators can be seen as "1-concept" in design, each tailored for specific transformation. However, their modular construction allows for the combination of multiple "1-concept" generators sequentially, leading to multi-concept tasks. As an example, a task could first necessitate the relocation objects, then require a color transformation of these objects. This layered approach augments the scope and intricacy of the tasks, serving as comprehensive evaluation of LLM capabilities.
3.1 Results
We used the best-performing prompts from Table [ for the direct-grid approach and documented the results in the rightmost part Table 2 Notably, the direct- strect-grid encoding shows a relative improvement in performance on the 1D-ARC as compared to the original ARC. This implies that reducing both task space complexity and the spatial dimensionality of the input-output pairs enhances the LLM's ability parse and reason with the encoded information.
Even with the significantly simpler 1D-ARC, there is still much room for improvement in performance. While GPT-4 is able to solve some tasks more effectively, it still falls short on others. This finding suggests that providing a sequential representation of objects in text alone may not be sufficient for GPT to elfectively solve ARC tasks.
In light of these findings, we next explored the benefits of employing an external tool to perform object abstraction for GPT, thereby completely removing the challenge of object cohesion in text.
8

---

Published in Transactions on Machine Learning Research (02/2024)
4 Enhancing LLM performance with an object-based representation
To address the challenges we have identified thus far and to enhance LLM performance, we propose the integration of an external tool to aid in producing an object representation of a task. More specifically, we leverage the ARGA algorithm [Xu et al. 2023) to exccute object abstraction before prompting LLMs for the solution. We note that a task is deemed to be solved correctly when the LLM produces the correct textual representation of the output, given the imput produced by the external tool. As such, whenever an LLM is coupled with ARGA, one should think of the integrated "ARGA+ -LM" system tackling the task, rather than the LLM alone.
4.1 Object- Object-based textual representation
ARGA is non-learning approach that aims to solve ARC by first abstracting the images into graph representations and then conducting scarch within Domain-Specific Language (DSL) defining possible changes to the graphs to identify the solution. We leverage the first component of ARGA to acquire graph representation of the images. These graph representations, in which each node (or vertex) corresponds to an object in the image grid and each edge represents relationships between the objects, are subsequently encoded into object-oriented text representations. It is worth noting that ARGA provides a suite of hand-designed abstraction methods cater different tasks. In our case, we apply the "best-fit" abstraction, utilizing the abstraction that ARGA deems optimal for generating the solution for each task. Essentially, our objoctive is to evaluate the effectiveness of an abstraction mechanism that excels at object abstraction. After transforming the images into graphs, examine two textual representations:
Object Descriptors: This encoding technique presents a list of objects, each correspouding to a node in the graph and its associated attributes; see Figure g (Top). offers a clear and intuitive representation of the image set of distinct objects, each carrying its properties.
On the other hand, the Object JSON encoding method provides a more structured representation of the graph; see Figure 8 (Bottom). This approach involves constructing a JSON list that encapsulates nodes and their corresponding attributes from the graph. The inherent organization of this format simplifies parsing and processing for the LLM, facilitating efficient extraction of pertinent information and relationships between the nodes.
Object JSON:
Each encoding approach is further explored with an additional variant that includes edge information from the graph. In ARGA, edge information is utilized to identify relations between objects, as certain transformations applied to objects depend on other objects. For instance, an operation might involve recoloring an object to match the color of its neighbor. In the context our study, our aim to investigate whether the inclusion of edge information in the textual representation augments GPT's ability to solve ARC tasks.
Image Size:(10, 10jin
Objects:in
Object 1: coordinates=((4, 1)], colore "blue", size=1in
Object coordinates=(14, 7)], colors "blue", size=tin
Object 3:
coordinates=(44 4), (3, 4), (4, 3), (5, 4), (4, 5)],
color= "green", sizen5
Image Size:(10, 10)in Objects:In
["coordinates": [[4, 1]], "color": "blue", "size": 1),
("coordinates": [[4, 7J], "color": "blue", "size": 1),
("coordinates". [[3, 4]. [4, 3). [4, 4), [4, 5), (5, 4]].
"color": "green". "size": 5)]
Figure 8: Visualization of object-based textual encodings. The 2D grid image is first transformed into graph representation using ARGA. Then, the graph is encoded using the object descriptors representation (Top) or the object JSON representation (Bottom).
9

---

Published in Transactions on Machine Learning Research (02/2024)
Table 3: Object-based variants, performance comparison. The values correspond to the number of tasks solved by each method, out of 50; higher better and the top-performer bolded.
Few-shot
In-context Few-shot w/ CoTT
Object-basod encoding
GPT-3.5
GPT-4
GPT-3.5
GPT-4
Object Descriptors
20
Object Descriptors with Edge
Object JSON
21
23
Object JSON with Edge
19
16
Table 4: Results for object-based approach. In addition to the caption of Table 2 the numbers in parenthesos are the increase in the percentage of tasks solved with the object-basod approach comparod to the direct-grid approach of Table 2 The number of lasks in each column is 50, so a 2% increase means that given method has solved one more task.
ARC
Move
Move
Move Dynamic
Recolor by Size
Fill
Move
Pile
LLM
Denoise
Subeet
1 Pixel
3 Pixels
H
V
H
H
V
0
39
14
7
21
36
17
15
1
10
GPT-3.5
(+5%)
(+58%)
(+14%)
(+2%)
(+38%)
(+46%)
(+30%)
(+30%)
(+2%)
(-)
(+10%)
(+20%)
23
50
49
37
40
50
49
37
GPT-4
(+20%)
(+34%)
(+74%)
(+52%)
(+52%)
(+20%)
(+4%)
(+969)
(+18%)
(+40%)
(+20%)
(+74%)
6
29
9
7
4
17
2
3
0
0
1LAMA2-13B
(+1%%)
(+12%)
(+12%)
(+6%)
(+8%)
(+26%)
(+1%)
(+6%)
(-)
(-)
(+2%)
(-)
39
17
8
5
LLAM22-00B
(+10%)
(+56%)
(+26%)
(+2%)
(+16%)
(+36%)
(+12%)
(+10%)
(-)
(-)
(+4%)
(-)
4.2 Results
We leveraged the prompting methods outlined in Section 2.2 in combination with our proposed object-based textual representations, replacing the direct-gric encoding. The results, presented in Table 3 show a marked improvement, with the success rate increasing from 13/50 tasks to 23/50 tasks on the ARC subset. Table 4 also shows that the previously observed performance gap between horizontal and vertical tasks in Table 2 is eliminated with the object abstraction, confirming our hypothesis that GPT's challenges with object cohesion in non-sequential text were the root cause. The orientation of objects becomes inconsequential, desired. An even bigger performance boost is observed for the 1D-ARC, where GPT-4 achieves 50/50 on some task types. LLAMA-2 Touvron el al. 2023 models also exhibit an increase in performance for the object-based method.
On the complete ARC training set, the new object-based method allowed GPT-4 to solve 97 out of the 400 tasks, up from 81 previously. Likewise, on the Mini-ARC datasct, performance improved to 45 out of 149 tasks, up from 35. These results underscore the value augmenting the LLM with an external tool that provides an appropriate representation, particularly when comes ARC tasks.
4.3 Analysis
We conducted the same solvability regression analysis from Section 2.4 observing the same correlations between task complexity attributes and solvability. Intriguingly, the models' performance was observed to decline when edge information was integrated into the representation. This unexpected result suggests that the influx of excessive information might overwhelm GPT, resulting in diminished performance. This discovery underscores the need for future research to find an optimal balance ween supplying adequate contextual information and avoiding information overload.
Furthering our analysis, we performed a similar examination of the tasks correctly solved by GPT under the based approach. Out of the 23 tasks that produced the correct output, impressive 20 tasks
10

---

Published in Transactions on Machine Learning Research (02/2024)
exhibited correct reasoning (See Appendix Dj. This significantly improved reasoning performance underscores the impact of effective object abstraction GPT's reasoning abilities. Figure 5 additionally showcases GPT's reasoning when prompted using the beject-based approach for a task where the direct-grid approach initially fell short in providing accurate reasoning.
5 Related work
Prompting methods for LLMs This is a very active area of development (Qiao el al.] 2022 Huang and Chang 2022). CoT prompting is introduced in (Wei et al. 2022), providing LLMs with intermediate reasoning steps leading to improved performance some complex reasoning tasks. Extending this, (Kojima 2022) demonstrated LLMs' potential as zero-shot reasoners by incorporating "Let's think step by step" phrase in the prompt. This approach notably enhanced accuracy on various reasoning tasks, thus hinting al untapped zero-shot capabilities within LLMs that can be leveraged through simple prompting techniques.
et al.
The survey by Mialon et al. 2023 emphasizes the potential of augmentation in overcoming limitations of a pure LLM approach. "Toolformer" self-learns to use external tools via APIs, significantly improving zero-shot performance across various tasks (Schick et al., 2023). Program-Aide Aided Language models (PAL) (Gao et al. 2022) combines the strengths of LLMs with Python interpreter to accurately solve some reasoning tasks.
Augmented LLMs
Solvers for the ARC Since the introduction of the ARC in 2019, various mods have been proposed to address it. A powerful DSL coupled with an efficient program synthesis algorithm has the potential to solve the
ARC, as initially proposed (Chollet] 2019). Notable examples include the Kaggle challenge's (Kaggic 2020)
winning solution, which utilized a manually-created DSL and DAG-based search for program synthesis (top
quarks
2020). Other high-ranking Kaggle participants followed similar strategies (de Miquel Bleier 2020
Golubev 2020 Liukis 2020 Penrose, (0220). Fischer et al. (0220) employed Grammatical Evolution
algorithm within their chosen DSL, while (Alford et al.) 2021 utilized the DreamCoder program synthesis system EElis et al. 2020) to derive abstractions from a basic DSL and compose solutions for new tasks through neural-guided synthesis. More recently, ARGA (Xu et al. 2023 was proposed as an object-centric framework that represents images using graphs and tree search for a correct program in a DSL based on the abstracted graph space. Alternative approaches for the ARC challenge have also been explored. The Neural Abstract Reasoner, deep learning method, achieved success a subset of ARC tasks (Kolev et al., 2020) Assouel et al. (022)) devised compositional imagination technique to generate unseen tasks for enhanced generalization. Ferré 2021) focused on an approach based on descriptive grids. However, these alternatives have not yet surpassed state-of-the-ars results.
ARC-like datasets have been introdeeed to lackle the ARC's complexity. The Mini-ARC (Kim et al. 2022), a 5 x 5 compact version of the ARC, was generated manually to maintain the original's level of difficulty. The Sort-of-ARC (Assouel et al. 2022) shares ARC's input space but presents simpler problems with 20x20 images containing three distinct 3x3 objects. The ConceptARC dataset presents set of manually
crafted tasks, grouped and categorized by 16 distinct core concepts (Moskvichev et al. 2023). The PCFG benchmark (Mirchandaní et al. 2023) is similar to our ID-ARC and was generated using the probabilistic context-free grammar
LLM for the ARC In a recent study (Moskvichev et al. 2023 the capabilities both automated methods and human cognition were explored with respect to the ARC. Their research employed state-of-the-art ARC solvers ftop quarks 2020 de Miquel Bleier 2020) and GPT-4 to tackle tasks originating from the ConceptARC dataset, comparing these solutions with those produced by humans. The approach to prompt GPT-4 was comparable to our few-shot direct-grid encoding method outlined in Scction 2.2 This study revealed that GPT-4 lags significantly behind both the leading ARC solver and human performance, finding that aligns with our own. Other recent studies also echo this finding (Mirchandani et al. 2023 Camposampiero et al., 2023). However, is critical to highlight that, basod on our investigations, the proficiency of GPT-4 on the ConceptARC dataset could potentially be enhanced by adopting an abject-based representation in the prompting process.
11

---

Published in Transactions on Machine Learning Research (02/2024)
While our paper focused on the effects of representation, recent study has shown improvements in the performance of L.Ms on the ARC by incorporating hypothesis search Waag et. al. 2022) Additionally, language instructions from LARC (Acquaviva et al. 2022) were utilized to assist LLMs in solving the ARC,
as demonstrated in another recent study (Pu and Alford 2023). Although the improvement was notable, it
was not exceptional, suggesting the presence of additional challenges which we discuss Section G
6 Conclusion and Future work
We have explored the capabilities and limitations of the GPT LLM in solving ARC tasks seen as representatives of certain kind of human-like intelligence. Our exploration started with straightforward, grid-based textual encoding approach, which revealed that GPT struggles due to the non-sequential ial represenlation of complex objects in text. We then introduced the ID-ARC, simplified, single-dimensional version of the ARC. By reducing the task complexity and dimensionality, we aimed to make ARC tasks more approachable for LLMs. Our evaluations on the 1D-ARC indicated improvements in performance but also highlighted that simplification alone could not bridge all the gaps in GPT's reasoning processes. Tn the third phase of our exploration, we adopted an object-based baseetd- approach, integrating an external tool, the ARGA framework, to assist in object abstraction. This led to significant improvements in GPT's problem-solving abilitics, realfirming the importance of structured, object-based representations in complex reasoning tasks.
Our research uncovers potential avenues for future exploration. For instance, edge information was not fully utilized by the LLM, suggesting that GPT-4 may not be capable of dealing with graphs when represented in text form. As we delve deeper into the possibilities of structured representations, we might consider introducing a "language" of transformations for LLMs to use in solving ARC tasks.
Our rescarch has predominantly focused on the LLM's ability to solve ARC tasks comprehensively. However, it's important to recognize that solving an ARC task can be a multi-step process, encompassing distinct sub-abilities. These include the LLM's capacity to abstract and understand objects from given inputs (Ability to Abstract), to formulate reasoning based on the understood input (Ability to Reason), and to generate accurate outputs using the reasoning and input (Ability to Execute). Each step presents unique challenges for LLMs. By this definition, our study can be viewed as assisting LLMs with the "Ability to Abstract", The recent study by Pu and Alford introduced in Section 5 can be seen as helping LLMs with the "Ability to Reason", since the reasoning is provided to LLMs directly. In both of these cases, LLMs still face difficulties in completely solving ARC tasks, which suggests that all three sub-abilities require further improvements.
Sub-abilitics of LLMs
Therefore, we believe that further experimentation aimed at dissecting and examining these sub-abilities could provide deeper insights into LLMs and pave the way for more sophisticated and capable language models in the future.
Visual Inputs for LLMs Recently, advancements have been made in developing LLMs capable of processing image inputs. GPT-4V is one such example (OpenAl 2023b). At the time of our research, the GPT-4V API was not available. We thus conducted experiments by mamually prompting ChatGPT-4 with image attachments. Evaluating on the set of 50 tasks described in Section 2.3 we found that GPT-4V, at its current stage, is only able to solve 2 out of the 50 tasks; in contrast, GPT-4 solves 23 tasks and even LLAMA2-13B solves 6 tasks. An example of solving an ARC task with GPT-4V is depicted in Figure g More details about this experiment are provided in Appendix G These findings demonstrate that GPT-4V with a multimodal visual and language input almost fails completely on the selected ARC tasks and significantly underperforms pure language- based LLMs, even pure language LLMs that are much smaller than GPT-4V. Hence, it appears there is room for significant improvement for multinodal LLMs to correctly reason about combined visual and language inputs in the context of ARC tasks, which should serve as a future challenge for L.LSs like GPT-4V.
12

---

Published in Transactions on Machine Learning Research (02/2024)
Language Instructions
Based on the observed transformations from the demonstration grids, it appears that when there's a single blue square, three green squares appear in the direction where there's the most space.
GPT-4V Reasoning
GPT-4V Prompt
##### Applying the Transformation to the Test Input Grid: Given the "Test Input Grid" has two blue squares:
1. the blue square the left:
There's more space its right,
add three green squares to its right.
2. For the blue square on the right:
There's more space to its left, so we add three green squares to its left.
GPT-4V Output Reconstructed
Expected Solution
Figure 9: GPT-4V solving an ARC task. Language instructions as well as an image attachment were used as prompts the model. GPT-4V describes the output text. Subsequently, a corresponding grid was manually reconstructed visually represent the text-based output described by GPT-4V.
References
Sam Acquaviva, Yewen Pu, Marta Kryven, Thcodoros Sechopoulos, Catherine Wong, Gabriclle Ecanow, Maxwell Nye, Michael Tessler, and Josh Tenenbaum. Communicating natural programs to humans and machines. Advances in Neural Information Processing Systems, 35:3731-383 37733,,,, 2022.
Simon Alford, Anshula Gandhi, Akshay Rangamani, Andrzej Banburski, Tony Wang, Sylee Dandekar, John Chin, Tomaso Poggio, and Peter Chin. Neural-guided, bidirectional program search for abstraction and reasoning. In International Conference on Compler Networks and Their Applications, pages 657-668. Springer, 2021.
Rim Assouel, Pau Rodriguez, Perouz Taslakian, David Vazquez, and Yoshua Bengio. Object-centric composi- tional imagination for visual abstract reasoning. In Objects, Structure and Causalily, 2022. URL ICLR2022 Workshop the Elements of Reasoning:
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4, arXiv preprint arXiv:2303.127 2023.
Giacomo Camposampiero, Loic Houmard, Benjamin Estermann, Joël Mathys, and Roger Wattenhofer. Abstract visual reasoning enabled by language. In Proceedings of the IEEE/CVF Conference Computer Vision and Pottern Recognition, pages 2642-2646 2023.
François Chollet. On the measure of intelligence. arXiv preprint arXiv:1911.015/77 2019. Kaggr//dddeeeeeeeee///Kageeggee 2020.
Alejandro de Miquel Bleier. Arc kaggle. Accessod: 2022-08-01.
Kevin Ellis, Catherine Wong, Maxwell Nye, Mathias Sable-Meycr, Luc Cary, Lucas Morales, Luke Hewitt, Armando Solar-Lezama, and Joshua B Tenenbaum. Dreamcoder: Growing generalizable, interpretable knowledge with wake-sleop bayesian program learning. arXiv preprint ar XrXin:000. 08381, 2020.
Sébastien Ferré First steps of an approach to the are challenge based on descriptive grid models and the minimum description length principle. arXiv preprint ar xrXiv:2112.00848, 2021.
Raphael Fischer, Matthias Jakobs, Sascha Mücke, and Katharina Morik. Solving abstract reasoning tasks with grammatical evolution. Tn LWDA, pages 6-10, 2020.
13

---

Published in Transactions on Machine Learning Research (02/2024)
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang. Jamie Callan, and Graham
Neubig. Pal: Program-aided language models. arXiv preprint vrXie:2211. 1002211....4 10435, 2022.
Vlad
Aro-kaggle-3rd-plaace abstraction wwwpaactiooo-ad-dreeaaeeeeeeeeee www.rrasiiiinggaallllggggggeeeee
Golubev.
https: / http:://www..ggleee... com/c/
2020.
Accessed: 2022-08-01.
Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large languago modols: A survey. arXiv preprinl Xiv:22212, 10403, 2022.
Kaggle. Arc-kaggle-mai Accossod: 2022-08-01.
2020.
Subin Kim, Prin Phunyaphibarn, Donghyun Ahn, and Sundong Kim. Playgrounds for abstraction and reasoning. In NewrIPS 2022 Workshop on Newro Causul and Symbolic AI (nCSI), 2022. URL https:
//openreview.nete www.nnn/foruuriid-AAPyyyye
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Twasawa. Large language models are zero-shot reasoners. In Alice H. Oh, Alckh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Newral Information Processing Systems, 2022. URL htttps://operrevier. net/ Forum?1d==2TBbbyyyEH
arXic preprint Victor Kolev, Bogdan Georgiev, and Svetlin Penkov. Neural abstract reasoner. arXiv:2011.09860, 2020.
Agnis Liukis.
Arc-kaggle-5h--plce. abstraction-and www.rraaaainoolalllllgaaaaaaaaaa
https://www kaggle.com/c/
2020. Accessed: 2022-08-01.
Grégoire Mialon, Roberto Dess), Maria Lomeli, Christoforos Nalmpantis, Ram Pasumuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, et al. Augmented language models: a survey. arXie preprint Xiw:2332.07842. 2023.
Suvir Mirchandani, Fei Xia, Pete Florence, Brian Ichter, Danny Driess, Montserrat Gonzalez Arenas, Kanishka Rao, Dorsa Sadigh, and Andy Zeng. Large language models general pattern machines. orXin preprint WrXi:2337.0478.1 2023.
Arseny Moskvichev, Victor Vikram Odouard, and Melanie Mitchell. The conceptare benchmark: Evaluating understanding and generalization in the are domain. arXiv preprint arXiv:2305.0714. 2023.
Victor Vikram Odouard and Melanie Mitchell. Evaluating understanding on conceptual abstraction bench- marks. arXiv preprint arXiv:2206.144 14187, 2022.
OpenAl. Gpt-4 technical report. arXie preprint urXi::2303.08774, 2023a.
AI Gpt-4v(ision) system card. 2023b. URL http://wwr/gggaaaacccc//oc/c/cca
Andy Penrose.
Arc-kaggleeSthipllac.
https://wwv waggle com/c/
2020. Accessed: 2022-08-01.
Yewen Pu and Simon Alford. Larc solving with gpt4.
2023. Accessod: 2023-12-20.
Shuofei Qiao, Yixin Ou, Ningyu Zhang, Xiang Chen, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang, and Huajun Chen. Reasoning with language model prompting: survey. arXio preprint ar 2217 2022.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themsolves to use tools. arXiv preprint Xiv:23300. 761, 2023.
14

---

Published in Transactions on Machine Learning Research (02/2024)
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354 359, 2017.
David Silver, Thomas Hubert, Julian Schrittwieser, Toannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Gracpel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go (hrough self-play. Science, 362(6419):1140 1144, 2018.
Elizabeth S Spelke and Katherine D Kinzler. Core knowledge. Developmental science, 10(1):89-96, 2007.
top quarks. Are-solution.
2020. Accessed: 2022-08-01.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. Xiv preprinl or wxxi:2337..09288 2023.
Ruocheng Wang, Eric Zelikman, Gabriel Poesia, Yewen Pu, Nick Haber, and Noah Goodman. Hypothesis search: Inductive reasoning with language models. arXiv preprint XrXiv:2309. 260.2009.0 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning large language models. In Advances in Neural Information Processing Systems, 2022.
Yudong Xu, Elias B. Khalil, and Scott Sanner. Graphs, constraints, and scarch for the abstraction and reasoning corpus. In Proceedings of the 37th AAAI Conference on Artificial Intelligence (AAAI-23), Washington D.C., USA, 2023.
15

---

Published in Transactions on Machine Learning Research (02/2024)
A 1D-ARC full dataset
Move Pixel
Mirror
Move Pixels
Denoise
Move Pixels
Denoise Multicolor
Move Dynamic
Pattern Copy
Move 2 Pixels Towards
Pattern Copy Mutticolor
Fill
Recolor by Odd Even
Padded Fill
Recolar by Size
Hollow
Recolor by Size Comparison
Flip
Scaling
Figure 10: 1D ARC: visualizations of selected sample tasks. For each task type, one pair of input-
output images is shown.
16

---

Published in Transactions on Machine Learning Research (02/2024)
Table Full results of direct-grid and object-based approach on the 1D-ARC dataset. Each row displays results for one generated task type using different methods. The values correspond to the number of tasks out of 50; higher better and the top-performer is bolded for each task type.
Direct-grid
Object-based
Task
GPT-3.5 GPT-4
GPT-3.5
GPT-4
Move
39
50
Move
22
50
Move
14
49
Move Dynamic
37
Move Towards
17
50
Fill
44
49
Padded Fill
37
44
Hollow
40
48
Flip
20
50
Mirror
13
Denoise
48
48
Denoise Multicolor
36
50
Pattern Copy
31
45
Pattern Copy Multicolor
21
47
Recolor by Odd Even
15
13
Recolor by Size
21
40
Recolor by Size Comparison
17
28
Scaling
46
B 1D-ARC Design Process
Identify ARC task that requires core prior knowledge.
See Figure 11 Left).
2. Determine the solution and the inherent knowledge priors.
Solution: Recolor objects based on their sizes.
Knowledge priors:
Objectness (Object cohesion)
Numbers and Counting priors (Understanding of object size through counting)
Develop the 1D-ARC task using the identified solution and knowledge priors
See Figure II Right).
4. Reiterate steps 1-3 to create comprehensive 1D-ARC dataset.
17

---

Published in Transactions on Machine Learning Research (02/2024)
?
Figure 11: Example of ARC task (Left) and 1D-ARC task adapted from it (Right)
18

---

Published in Transactions on Machine Learning Research (02/2024)
C
Vertical and horizontal datasets
Task Fill (a699fb00)
Task Move (3906de3d)
Task Pile (1e0a9b12)
Original ARC Task
Example Generated Horizontal Variant
?
?
?
Example Generated Vertical Variant
?
?
?
Figure 12: Visualization for generated vertical and horizontal variants for three ARC tasks. Each column displays the original ARC task on the top, one generated horizontal variant in the middle, and one generated vertical variant at the bottom.
D GPT reasonings for solved tasks
Table 6: GPT Reasoning for correctly solved ARC tasks using best performing direct-grid approach. The correctness of the reasoning provided is determined manually. If reasoning was not provided, it is reported as N/A.
Task
Reasoning
Correctness
Transformation applied:
15/
1. Move color objects (blue or red) row down.
aabf363d
N/A
19

---

Published in Transactions on Machine Learning Research (02/2024)
Transformation applied:
d5d6de2d
1. Replace color red object with color green object
Remove the outer border of the color green object Transformation applied:
dc433765
1.
Move color green object 1 pixel down and color yellow object pixel up.
X
b1948b0a
N/A
74dd1130
N/A
X
Transformation applied:
Fill the empty spaces inside the gray objects with the following colors:
c0176784
Cyan for the top object
X
Orange for the middle object
Purple for the bottom object
6150a2bd
N/A
X
c8101002
N/A
X
67385a82
N/A
X
67a3c6ac
N/A
Transformation applied:
a5313dff
1. Replace the black cells inside the red shape with blue cells.
Transformation applied:
a699fb00
Add a red object the right of every blue object,
X
except when there is already red object to the right of the blue object.
Table 7: GPT Reasoning for correctly solved ARC tasks using best performing object-based approach. The correctness of the reasoning provided is determined manually. If a reasoning was not provided, it is reported as N/A.
Task
Reasoning
Correctness
Transformation applied:
a79310a0
1. Move color object 1 pixel to the right
Recolor color 8 object color 2
Transformation applied:
25H77 la9
1. Move the object 1 pixel to the right.
Transformation applied:
aabf363d
/
1. Recolor the larger object with the color of the smaller object.
Transformation applied:
d5d6de2d
Remove color-2 objects
2. Recolor color-0 objects to color-3
Transformation applied:
6c434453
For each color-1 object with size 8, remove the corners to create two color-2 objects with size 5.
20

---

Published in Transactions on Machine Learning Research (02/2024)
Transformation applied:
deldf850
1. Surround cach color-2 object with color-1 objects in a 3x3 square, excluding the center where the color-2 object is located.
Transformation applied:
aedd82e4
1. Recolor all color-2 objects that are not touching the largest color-2 object to color-1.
Transformation appliod:
dc433765
1. Move color 3 object 1 pixel towards color 4 object.
Transformation applied:
Recolor color 5 objects to color 1 if size is 4
6e82alae
2. Recolor color 5 objects to color 2 if size is 3
3. Recolor color 5 objocts to color 3 if size is 2
Transformation applied:
b1948b0a
1.Recolor color 6 objects to color 2
Transformation applied:
4258a5f9
For each color-5 object, create a color-1 object
with 8 pixels surrounding the color-5 object.
Transformation applied:
ded97339
Connect color-8 objects with the same color a straight line.
Transformation applied:
7f4411dc
1.Remove all color objects with size 1.
Transformation applicd:
1.Find the midpoint between the color-1 objects.
e9614598
2.Create a color-3 object of size 5 in cross shape centered at the midpoint.
Transformation applicd:
1. Recolor the largest color-5 object to color-1
ca.2ff37
Recolor the smallest color-5 object to color-2
Recolor the remaining color-5 object to color-4
Translormation applied:
6150a2bd
x
1.Rotate the colors the objects 90 degrees clockwisc.
Transformation appliod:
1. For each object, create new object with
913fb3ed
8 coordinates surrounding the original object.
Assign colors the new objects based on the original object's color:
color 2 -> color 1, color 3 -> color 6, color 8 -> color 4.
Transformation applicd:
c8f0f002
R.Recolor color objects color
Transformation applied:
67385a82
R.Recol color 3 objects with size > 1 to color 8. Transformation applied:
c522a75a.
1. Surround each color-5 object with color-1 objects in 3x3 square, keeping the color-5 object in the center.
x
21

---

Published in Transactions on Machine Learning Research (02/2024)
Transformation applied:
67a3c6ac
1.Rotate the colors clockwise within the objects.
Transformation applied:
a5313dff
L.Recolor color 0 objects color
Transformation applied:
a699fb00
1. Recolor color 0 objects to color 2
E
Logistic regression analysis
Coefficient
Feature
Direct-grid
Object-based
Number of colored pixels in test input image
-0.151312
-0.365261
Average number of colored pixels in training input images
0.215891
0.326572
Number of unique colors in test input image
-0.282226
0.346230
Average number of unique colors in training input images Number of pixels changed in test instance
0.192485
-1.186780
0.110529
0.142800
Average number of pixels changed in training instances
-0.152656
-0.090327
Test input image size
-0.004665
0.001771
Training input images average size Number of training instances
-0.013070
-0.005959
0.297392
0.158643
Score
Performance Metric
Direct-grid
Object-based
Precision (unsolved)
0.78
0.83
Precision (solved)
0.44
0.73
Recall (unsolved)
0.86
0.74
Recall (solved)
0.31
0.83
Table 8: Results of logistic regression analysis. Top: Comparison of feature coefficients for the best- performing direct-grid and object-based approaches, demonstrating the impact of each feature on an ARC task's solvability. Bottom: Precision and recall scores of logistic regression model for solved and unsolved tasks.
22

---

Published in Transactions on Machine Learning Research (02/2024)
F
Visualization of GPT solutions on example ARC tasks
Task dctelleso
"Tas de1di1i50 Expected Outbut
Directigrid GPT4 Output
Object basee GPT4 Output
Task Seafii7a Input
Saa6ib7e Expected Outout
OPT Oulput
Osject-beser GPT-4 Oulput
Task odttetibb Test input
Task 6d75etbb Expected Test Output
Diractigrd GPT4 Dutput
Object bases GPT,4 Output
Input
Expacted Output
Dirac-grid Output
Object Cutput
Task 03133765 Test Inout
Task dc433763 Expected Test Cutsut
Directiond GPT4 Output
Object-baze GPT:4 Cutout
Figure 13: Visualization of example ARC Tasks. Each row showcases an ARC task. The first column displays the test input for that task, while the second column shows the expected test output. The third and fourth columns present the predicted outputs using the best-performing direct-grid approach and object-based approach, respectively.
23

---

Published in Transactions on Machine Learning Research (02/2024)
G Experiments with GPT-4V
This section briefly introduces the preliminary experiments conducted for GPT-4V. The GPT-4V API was not available at the time of our research, therefore, the experiments were conducted by manually prompting ChatGPT-4 with image attachments. First, the prompting method is described and visualized, followed by a display of the results. Examples GPT's output will also be shown.
G.1 Prompting
The prompting style follows the few-shot approach introduced in Scction 2.2 consists of "instructions" " and "task" sections. The "instructions" section is similar to before where it outlines the nature of an ARC task and the expected behavior of the LLM. The "task" section provides information about the ARC task of interest, however, instead of using textual representations, an image containing the few-shot examples as well the test imput is provided. Figure 14 shows example of the prompts.
G.2 Results
GPT-4V cannot currently produce images as outputs, so it produces texts that describe the output grid. To evaluate the results, manually examined GPT-V4's outputs and reconstructed the output images based the descriptions it produced. We found that GPT-4V was only able to correctly describe the output image for 2 lasks out of the 50 tasks we defined earlier, even more so, only further 3 out of 50 tasks had the correct logic before producing the wrong output. 2 examples of ARC tasks as well as GPT-4V outputs for them can be seen in Figure 15
24

---

Published in Transactions on Machine Learning Research (02/2024)
GPT-4V Input
Let's play a puzzle that focus on reasoning and logic. In this puzzle, you will be provided with an image file containing few demonstrations of how an "Input Grid" gets transformed into corresponding "Output Grid". In the end of image, you will get a brand new "Test Input Grid", then you must answer the corresponding "Test Output Grid" as an output image and describe the transformations used step by step starting from the "Test Input Grid".
Instructions
Pazzle #d5144900
Task
Figure 14: Visualization for prompting GPT-4V with an ARC task.
25

---

Published in Transactions on Machine Learning Research (02/2024)
Task e9614598
Based on the observed transformations from the demonstration grids, appears that when there's single blue square, three green squares appear in the direction where there's the most space.
Examples
GPT-4V Reasoning
#### Applying the Transformation to the Test Input Grid: Given the "Test Input Grid" has two blue squares:
1. For the blue square the left: There's more space to its right, so we add three green squares to its right.
Test Solution
For the blue square on the right:
There's more space to its left, add three green squares to its left.
GPT-4V Output Reconstructed
X
Task 913fb3ed
From these examples, it seems that: Cyan square transforms into a Yellow square surrounding it.
A Green square transforms into Purple square
GPT-4V Reasoning
surrounding it.
-A Red square transforms into Blue square surrounding it.
Examples
The size of the surrounding square can vary.
### Applying these rules to the "Test Input Grid":
1. *Cann square**: Transform into a Yellow square surrounding it.
2. **Grenn square**: Transform into Purple
square surrounding it.
3. *Red square**: Transform into Blue square surrounding it.
GPT-4V Output Reconstructed
Test Solution
Figure 15: Visualization of GPT-4V outputs for ARC tasks.
26


Reviews:
[{"review_id": "M9KI8nTcUf", "reviewer": "Reviewer_xsGU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The 1D-ARC corpus is a good contribution. I am curious how it differs from existing string manipulation tasks, though, like the dataset introduced by Andreas et al. 2018 (and used in more recent work, e.g., Grand et al. 2023).\n* Investigating the representation of task instances that can be used as input to a language-only model is interesting.\n* The analysis of failures, e.g., in Section 2.4, seems interesting (though I would like to see a lot more detail about how this analysis was performed).\n* The experiment with rotation is interesting\n\nWeaknesses:\n* I disagree with some of the framing: when we cater to an LLM by providing it a better input representation, is it that the LLM is now able to complete the task and solve the corpus, or have we just constructed some broader system that can do it?\n* I struggle to understand some of the background motivation on evaluating LLMs on a task like ARC which is fundamentally a visual reasoning task. Specifically, the final sentence of the paper: \"the necessity for substantial advancements to enable reliable reasoning by LLMs\". Why do we think in particular that solving ARC is necessary for building \"reasoning\" LLMs?\n* Evaluation is only performed on GPT-4 and 3.5. Why not evaluate on the many other publicly available language models?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper evaluates GPT-4 on the Abstraction and Reasoning Corpus (ARC). It finds that GPT-4 struggles on a text-based linearized representation of the tasks in the corpus, analyzes why it fails, and proposes a new version of ARC (1D-ARC) that uses 1D tasks instead of the 2D grid originally included in ARC. Finally, the paper applies ARGA (Xu et al. 2023) to compute graph-based representations of the task instances on which they evaluate GPT-4's reasoning, and find that with this representation of the task GPT-4 performs better.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Summary of weaknesses to address:\n* Some of the framing around the capabilities of LLMs here is not completely accurate. Providing GPT-4 a more usable representation of the task as computed by ARGA doesn't mean GPT-4 is solving the task; it means a broader system is proposed that improves performance on the task.\n* Clearer motivation of the evaluation \n* There should be more details about how the analysis was performed in Sections 2.4 and 4.3.\n* Evaluation on more models\n\nQuestions:\n* In the results, there are 50 tasks. What does it mean to \"solve\" a task? Are there multiple problem instances per task, and \"solving\" one means successfully predicting the 6th frame for every instance of the task? Or is there only one instance per task?\n* Shouldn't the goal of selecting 50 examples to evaluate serve as a _lower_ bound rather than an upper bound on ARC as a whole?\n* I realize that this paper was only recently published, but Mirchandani et al. 2023 also perform evaluation on ARC. How does this evaluation compare to the one proposed in this paper?\n* How might tokenization act as a confounder even in the 1D case? E.g., did you evaluate with multiple kinds of pixel representations (entire words, numbers, random tokens as used in Mirchandani et al. 2023) for the 1D case and see how robust performance is over them?\n* Why is Table 4 showing factor improvements rather than absolute improvements? It seems much less meaningful when the numbers (and dataset size in general) are already really small\n\nMinor points:\n* Evaluation on \"GPT\" is ambiguous as there are tons of different models fitting under the general category of \"GPT\". These instances should be replaced with GPT-4", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "APQ1wWXBzJ", "reviewer": "Reviewer_RCQt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Srengths: \n\n1. The paper is clearly written, with helpful figures illustrating the task setups. \n\n2. The investigation is well-motivated, and I particularly like the use of the heuristic parsing algorithm.\n\nWeaknesses: \n\n1. Only two closed-source models have been studied. Could any of the recent open-source models be used, such as LLAMA 2? \n\n2. Using open source models could also enable various attention visualizations, which may shed light on the observed findings. \n\n3. A baseline of how well the model understands the object representation should be conducted. In this baseline, the ground truth reasoning pattern is given. If the model performs well in generating the new output according to the pattern, then it suggests an issue of pattern finding. However, if the model still couldn't perform well, then it seems to suggest that the model is having trouble understanding the textual input as a \"grid\". \n\n4. The benchmark is by definition quite synthetic. I am wondering if there could be real-world extensions/adaptations that highlight the practical use of such reasoning ability.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper looks at the abstraction and reasoning capability of large language models, largely based on the Abstraction and Reasoning Corpus (ARC) and the author-proposed derivatives. The benchmarks require the model to identify patterns among input-output pairs, which are specified as textual characters. The authors conducted a suite of investigations, studying the effect of various factors, such as \"object\" representation, input dimensionality, and few-shot prompting. Significant gaps have been observed between GPT performance and ideal performance, and the object representation has been found to be most influential in ablation studies.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the weaknesses section.", "broader_impact_concerns": "No apparent ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kIO0wuMOPD", "reviewer": "Reviewer_uAxr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "In terms of pros, I should first say that although I am no expert in\nthe field of generative AI, I find the paper to be pretty easy to\nfollow. There are no technicalities that would represent a challenge\nfor an average reader. The topic and the discussion are quite simple\nto understand. Second, the submission is accompanied by an archive\ncontaining the experimental data, which is also a plus. Finally, the\nconclusions drawn by the authors make perfect sense to me.\n\nAs for cons, I have to say they result from (some of) the same aspects\nI identified as pros. Namely, the insights offered by the manuscript\nare somewhat shallow, which affects the observations and conclusions\nmade. I would say the submission lacks some \"essence juice\" that would\nmake me think it is valuable. Namely, it is really unsurprising that\nLLMs have a hard time with 2-d data as they are designed to predict\nthe next token in a one-dimensional context *sequence*. And it seems\nplain that a rigorous textual/symbolic summary of a puzzle, which the\nauthors apply as the final ammunition, should provide an LLM with\nhigher chances of success in solving the puzzle.\n\nIn general, the trend of looking for magical (reasoning) properties of\ngenerative AI and applying it all over the place for the tasks it does\nnot fit looks weird to me. Granted I come from a different area in AI,\nI will not say this particular criticism of mine justifies rejection.\nThe work seems to have merits: it studies quite a fun problem and\napplies uncomplicated but sensible mechanisms to analyze and address\nit.\n\nHaving said that, I find it strange that the authors say they\nperformed only *preliminary* experimentation with GPT-4V despite the\nexistence of the API. And I feel the article will definitely benefit\nfrom a full-blown comparison of the upsides and the downsides of the\nproposed ideas versus GPT-4V. Otherwise, the findings look incomplete\nto me.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This article offers an analysis of the poor abstract reasoning\ncapabilities demonstrated by the widely acclaimed large language\nmodels (LLMs) GPT-3.5 and GPT-4, based on the existing set of\nbenchmarks called ARC (stands for Abstraction and Reasoning Corpus).\n\nThe article claims that the lack of the LLMs' reasoning power for the\nARC puzzles stems from their lack of spacial understanding of these\npuzzles. Namely, an ARC puzzles represent a 2-d grid demonstrating\ncolored shapes while LLMs are designed to generate text in the context\nof 1-d streams of information. This basically results in the model's\nfailing to understand that some of the bits of information are related\nto each other and represent the same object in the grid. The authors\npropose a test confirming their hypothesis, which consists in\ntransposing the stream and showing the performance of both versions of\nGPT drops down even more on the transposed benchmarks.\n\nFurthermore, the article proposes a simplified set of benchmarks\ncalled 1D-ARC, where all the puzzles are described by a 1-d image, to\ncheck LLMs' abstract reasoning capabilities on benchmarks where the\ndata is stream-like. The authors show that both versions of GPT\nexhibit better performance on this set of benchmarks as the models do\nnot have to reason about 2-d information.\n\nFinally, the authors adapt the existing technology for translating the\nARC grids into a DSL-like symbolic representation of objects appearing\nin the puzzles, which helps GPT-* models further increase their\nperformance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please incorporate the use of GPT-4V in the text, the experimental\nresult and their analysis.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers were generally happy with the final version of the manuscript. I think the work makes a valuable point when it comes to evaluation on ARC, and the results are appropriately couched with respect to where the gains come from (essentially an external preprocessing system). It serves as food for thought for others trying to make claims (positive or negative) about LLMs' fundamental abilities, since these are typically conflated with the task format as in ARC.

The main weaknesses I see are this work lacking breadth of LMs and being out of date potentially soon (although that's true of any paper in this fast-moving area) and that the results might be unsurprising (uAxr). However, it clears the bar for TMLR's review criteria.
Decision: Accept (poster)