Combining Induction and Transduction for Abstract Reasoning
Authors: Wen-Ding Li
Keya Hu
Carter Larsen
Yuqing Wu
Simon Alford
Caleb Woo
Spencer M. Dunn
Hao Tang
Wei-Long Zheng
Yewen Pu
Kevin Ellis
Venue: ICLR 2025 Poster (2025)
Keywords: Abstract Reasoning
Visual Reasoning
Program Synthesis
Induction
Transduction
Primary area: neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
TL;DR: We designed a system that combines induction and transduction for abstract reasoning tasks

Abstract: When learning an input-output mapping from very few examples, is it better to first infer a latent function that explains the examples, or is it better to directly predict new test outputs, e.g. using a neural network? We study this question on ARC by training neural models for \emph{induction} (inferring latent functions) and \emph{transduction} (directly predicting the test output for a given test input). We train 
on synthetically generated variations of Python programs that solve ARC training tasks. We find inductive and transductive models solve different kinds of test problems, despite having the same training problems and sharing the same neural architecture: Inductive program synthesis excels at precise computations, and at composing multiple concepts, while transduction succeeds on fuzzier perceptual concepts. Ensembling them approaches human-level performance on ARC.

Published as a conference paper at ICLR 2025
COMBINING INDUCTION AND TRANSDUCTION FOR ABSTRACT REASONING
Wen-Ding Li" Keya Hu Carter Larsen' Yuqing Wu' Simon Alford Caleb Woo Spencer M. Dunn' Hao Tang' Wei-Long Zheng* Yewen Pu "4 Kevin Ellis?!
'Comell SShangaii Jiao Tong University *Autodesk "co-leads "co-advising correspondence: {wl678, ke kellisy@cornell.ed., hu mu_keya@sjtu eau.edu.co
Please refer to the updated Xiv version which has improved results with more authors.
ABSTRACT
When learning an input-output mapping from very few examples, is it better to first infer latent function that explains the examples, is it better directly predict new test outputs, e.g. using a neural network? We study this question ARC by training neural models for induction (inferring latent functions) and transduction (directly predicting the test output for given test input). We train on synthetically generated variations of Python programs that solve ARC training tasks. We find inductive and transductive models solve different kinds of test problems, despite having the same training problems and sharing the same neural architecture: Inductive program synthesis excels at precise computations, and at composing multiple concepts, while transduction succeeds on fuzzier perceptual concepts. Ensembling them approaches human-level performance on ARC.
INTRODUCTION
Robust generalization from few examples remains one of the most important ways in which human intelligence surpasses Much recent work views this generalization as a form of abstract reason- ing: Given just a few training input-outputs Frain Vrain together with test input Tiest, the idea is
to predict the corresponding test output yhest using reasoning strategies such as analogical reasoning, chain-of-thought, inductive program synthesis, transductive prediction (Thoms [023 Wang
et al.
2024 Witt et al. 2023 Lee et al. 2024; Tang et al.) [202aa Hocquette Cropper [2024] Butt
al.
2024 The Abstraction and Reasoning Corpus (Chollet 2019), henceforth ARC) is few-
shot learning benchmark that tests the ability to rapidly learn a diverse range of new skills, and apply
them situations. Fach ARC task is presented input-outputs over colored grids, but gage concepts such as occlusion, pathfinding, collision, symmetry, gravity, bouncing, counting, etc., making ARC essentially a composite of many reasoning datasets, and one of the more interesting unsolved benchmarks that stresses broad-coverage few-shot learning (Figure
object contact
gravity
symmetry, occlusion
jumping
A
A
bouncing
mazes
pathfinding
growth
Figure 1: Few-shot learning tasks from the Abstraction and Reasoning Corpus (ARC). Each task typically has 2-5 input-output examples. Here we show just one input-output example per task.
Here we study neural methods for induction and transduction, using few-shot learning problems from ARC our testhed. Induction means first finding function f where f (aimin) Yrains and

---

Published as a conference paper at ICLR 2025
then predicting Yest f (Zwaa) Transduction instead outputs Yest without explicit construction of an intermediate function f. Intuitively, induction captures the notion that a learner should first explain the training data, then use that explanation to make predictions. Inductive learners can perform better by spending more time optimizing or searching for better explanations, using the training examples Train: Vtrain score candidate functions. Transduction instead captures the intuition that the training examples themselves should play a direct role in generating new predictions, and that successful prediction need not require explicit explanation. See Figure
induction
transduction
Insin
Inain
Python progrom.lly
Infer function
cat tranctoraa aup.t.r Lii
run function fix lent
Best
Fine the yetlow ranusetall
direct
mog upet sgaart
Jest
tus Salar darr1
rest
exdur aisa squarmo sand pratinsk) calor
?
reture aiputtatc
Figure 2: Induction generates an intermediate function f to explain training input-outputs. Trans- duction directly predicts the test output, for example using a neural network.
We train neural networks for both induction and transduction by generating large corpus of syn- thetic problems. We discover that neural models for induction and transduction are strongly comple- mentary. We believe this surprising: Although any pair of models would generally solve somewhat different problems, usually this can be attributed to different priors, data, or architecture. Instead, we find that, even controlling for priors, data, and architecture, most problems solved by induc- tion were not solved by transduction, and vice versa. Moreover, induction and transduction can be trivially ensembled by using induction to generate candidate functions f until either a satisfactory function is found (e.g. (Xrain) 3/rain) until test-time compute budget is reached, at which point, transduction kicks in as fallback: That they are complementary has practical implications.
Our study is tightly linked to program synthesis. We represent functions f as Python code, mean- ing that induction synthesizes programs. We train transduction models on LLM-produced Python scripts, meaning that transduction is trained on the input-outputs of symbolic code. Although pro- gram learning has long been a popular vision of how general Al could work (Solomonoff 1964] Schmidhuber 20044) [Hutter] 2004), the dominant theory has always been of explicit code gen- eration (induction), rather than implicitly teaching neural networks to imitate code (transduction). Our work puts this assumption to the test.
Testing these neural methods requires a large dataset of function-learning problems, which is chal- lenging to generate because not only must we make novel functions, but also good inputs those functions. Consider the transformations in Figure [T: What counts as a good input for one function unlikely to work for another. To address this challenge, we first generate a deterministic Python function for J' and then a probabilistic program for sampling inputs to f. finally executing those pro- grams to produce input-outputs. This helps generate inputs that are appropriate for the underlying transformation, and forces Frrain Mtrain to be explainable by deterministic mapping.
We contribute the following:
1. A study finding that neural models for induction and transduction are strongly complementary, even when trained on the same problems. This contradicts seminal neural program synthesis work (Devlin et al. (2017). which found induction superior), and contradicts the findings of the leading ARC team (Cole al.) (2024), which advocates transduction with test-tine test-time training).
2. An automated data generation methodology that starts with 100-160 program solutions for ARC training tasks, and expands them to make 400k new problems paired with Python solutions.
3. A study of how these methods scale. We find performance saturates quickly when increasing labeled data, but scales with compute, both at training and testing time.
4. Analysis of families of problems solved by each approach, and how they compare to humans.
2

---

Published as a conference paper at ICLR 2025
2 NEURAL MODELS FOR INDUCTION AND TRANSDUCTION
We consider few-shot supervised learning problems where the learner is trained to map members of an input space X' to output space V/ For K-shot learning, we receive K training input-outputs (Imain Ytran)) E X x ydk, together with single test input Trest E X, and predict yeest E V. Our neural models for K-shot learning are meta-learned (Mishra et al. 2017 inter alia.) using meta- learning data further annotated with a ground-truth function j d y. which supervises the induction model. Below we define the training and use of these models.
Neural networks for induction and transduction. A neural network for transduction a function that maps (Frrinn Drsin: Trea) to a distribution over Yhea. and which has learnable pa- rameters 0. In other words, to X x x)* x X' - A()),, where the notation (SS) means the set of distributions over S. We can also write this as a conditional distribution, to [raain Yrrain Teest). A neural network for induction is function i that maps (Frrinn Yrrain, Ties) to a distribution over functions/ ff that map X to y. with learnable parameters 9. In other words, 1 : X x y X X -
Definition:
A(X' + ))), which we can write conditional distribution ( Tuains Vurinn Feest).
Training induction and transduction. Both types of models are trained via meta-learning. We assume meta-learning dataset D of few-shot learning problems, each equipped with ground- truth function f such that fx) = y for every I, in (Frain: Ytsin) and (Irest Bles)) Inductive and transductive models are meta-trained to minimize the following losses:
-iee-Mtaa TRANSDUCTION Loss E, (uii Memm Sice Miea DrD |- log to (Jtes Timin Vtrain; Trest)
(1)
INDUCTION Loss (muiee DuaisFFees-eee S D-D
[ log i [f] Puain Fiest)
(2)
After meta-learning the models encounter a test-time few- shot learning task (Fuain: Yrain Twss). Transductive models predict their most likely output for
Testing induction and transduction.
Yeest (approximated via beam search). Inductive models sample a test-time budget B functions
fi /.... J'n which are filtered by (Derain; Yumin,, and finally used to predict ylest [[Fess) Writing
Vrest for the predicted test output:
TRANSDUCTION: Ptest = arg max to G3/ Pirain Yttrain Trest)
(3)
yEY
INDUCTION: Drest Uniform (F)
(4)
where F (Free)))) : for 1 B fo (Irrai)) Y
fo ~ ia [imann Uraiin Fress)
Combining induction and transduction. Induction allows checking candidate hypotheses against the training examples. Therefore, we know when induction has found plausible solution-but sometimes fails to find any solution. Transduction has the opposite property: We can't check if its predictions match the training examples, but it always offers candidate answer. Therefore we ensemble by attempting induction first, then transduction none of the candidate hypotheses explained the examples:
ENSEMBLE: Grest ~ Uniform (F) Ff 2
Uress arg max to(y|Furain Yisain Prea) if F
(5)
yEy'
Every input from X' and output from Y is a 2D grid ranging from 1-30 pixels per side, with each pixel containing one of ten colors. Because ARC tasks are highly diverse yet typically have an abstract program-like structure, we represent the underlying function f as Python code, which is computationally universal, and possible in principle of solv- ing any ARC task. Therefore the induction model must generate Python code, so we initialize our models with Llama3. 1-8--intruut (Dubey et al. 2024)) because it was pretrained
Instantiating the framework for ARC.
source code !T] We encode 2D colored grids as strings using token per pixel, and use newlines to delimit rows (Appendix B.I. We then meta-learn by further fine-tuning Jlama3.1-8B 1-8B-instruct for induction or transduction using synthetically-geet -egerrate corpus of problems, described next.
!Our preliminary experiments suggested Ilama3. 18B--instttttttt was better than -B-vral-7--v0.3, Qwen2- 7B- Instuuttttt, and Salo---oooorr-..---tttee
3

---

Published as a conference paper at ICLR 2025
3 GENERATING DATASETS FOR INDUCTION AND TRANSDUCTION
Generating ARC-style tasks is challenging because of the diversity of concepts that can occur in ARC. It is also challenging because need to generate not just function, and also inputs that serve as good examples for that function.
At a high level, our dataset grows out of 100 manually-written Python programs, each of which both solves a given task (function f). and also randomly generates new input grids. We call these 100 martally-written programs seeds. Each seed is commented with natural language describing the core intuitions behind the problem. We then use large language model to mutate and recombine the seeds, producing many thousands of programs (Figure
100 seed solutions
100k seed problems with solutions
100 seed problems
language
language
concepts: description:
description:
the insul you wi bee a sing's coloed snaps, anound
code
Boating
def generate, Input():
def transform wtt.rfooomoriddiuttgrriia.
solve
our the srape so the axppu has the same dimarakore the stape.
make
baokground
remix
runtime
escarm plas
def generate, geserate__nnutt)) transform irput gridi:
manual problem solving
synthetic augmentation
Figure 3: Synthetic data generation pipeline, starting with human-written programs (seeds).
The structure of seeds. Each seed consists of three parts:
1. A natural language description of its specific ARC task task-including how to solve that task represeeredeed Python comment the top of the seed.
2. A Python function trans grid corresponding the function f in the manuscript, which maps each input grid of specific ARC task to its corresponding output grid.
3. A Python function generate_ _nptu,, which takes no arguments, and which randomly generates new inputs to f (new inputs to transform_grids
Prior knowledge. The seeds impart a prior upon the system by demonstrating good programs for solving training tasks. We further codified much of this prior into Python library containing code that we found useful across many seeds, such as subroutines for generating random sprites, detecting symmetries, extracting objects (Appendix A.2 Synthetic problems can that same library.
However, this prior knowledge different from previous Domain Specific Languages for ARC (Butt al. 2024 Wind 2020 Ainooson et al.) 2023 Domain Specific Languages restrict the class of allowed programs by only allowing stereotyped combinations of domain-specific primitives. We still allow arbitrary Python code, which helps cover the long tail of diverse tasks.
Remixing the seeds. To generate larger synthetic dataset we "remix" the seeds using LLMs. Each new synthetic ARC problem is generated by a three stage pipeline (Figure II):
1. A new natural language description is sampled by prompting LLM with seed natural lan- guage descriptions, effectively using in-context learning to recombine and mutate elements of different problems, in the spirit of self-instruct (Wang al. 2023).
2. Code is generated for that new description via Retrieval Augmented Generation (RAG: [ewis al. (2020)). Our RAG pipeline retrieves seeds with similar descriptions, and prompts an LLM to generate code for the new description, given the retrieved seeds.
3. The newly created generate_ innpnnttttt is executed to make inputs, which are passed to transform_grin to produce input-output grids.
4

---

Published as a conference paper at ICLR 2025
Figure illustrates example problems generated by pipeline. Unless otherwise mentioned, create synthetic datasets with GPT40-mini and ada-002.
concepts:
concepts:
concepts: pattern detection, copying,
shape detection, contour extraction, bounding box
color blending, expansion
positioning
Figure 4: Example synthetic ARC problems generated by our pipeline. Concepts are generated in comment near the top of the Python script as part of the natural language description of the seed.
4 EMPIRICAL STUDY OF INDUCTION AND TRANSDUCTION
We train inductive and transductive models with the goal of understanding (1) how the methods compare; (2) how performance scales with train-time effort; and (3) how performance scales with test-time compute (for induction only, as it allows drawing more samples at test time to improve performance). We report performance on the 400-pro (00-problem public validation split of ARC, which is harder than the training split. The systems described in this section learn from 100-pn problem subset of the training split, specifically problems for which we created seeds.
Induction and Transduction are strongly complementary. Despite training the exact same problems, inductive and transductive models solve different tasks, and neither approach dramat- ically more effective than the other. And although these methods have similar overall solve rate, most problems solved by induction are not solved by transduction, and vice versa (FigureFA))
An alternative explanation is that induction and transduction are not actually complementary. but instead that, having trained two neural networks with different random initializations, they simply solved different problems due randomness train test time. To test this alternative explanation, we trained many models with different random initializations. We find that the problems solved by induetionn/ransduution are surprisingly stable across these different runs (Figure5B), In other words, some problems are friendlier to induction, and others friendlier to transduction (Figure
(A) complementary methods induction transduction
(B) finding is stable across random seeds, and (C) statistically significant transduction induction
1.0
induction transduction
frequency
p<.004
correlation (r)
26
35
0.25
0.50
0.75
correlation
problem solved
0.0
Figure 5: (A) Induction and transduction solve different problems, where solve means predicting right output given tries. Venn diagram for models trained 100k synthetic problems generated using gpt4o-mini.
(B) Training many models with different random seeds, and then measuring the correlation between solved tasks by different models. Solved tasks strongly correlates with other models of the same class but not the other class. (C) Statistical significance test evaluating the null hypothesis that correlation is independent of whether model is inductive/transduetive.
5

---

Published as a conference paper at ICLR 2025
solved only by induction
Place the puzzle pieces into the hole with the same shape
Find the shortest path between two red pixels
Place sprites position indicated by pixels blue square.
Draw squares on diagonal using border as guide
solved only by transduction
pattern maps differently colored 3x3 square
Reconstruct occluded parts the pattern
Arrange sprites four quadrants
majority surrounding color
solved by both
Shoot lasers in all directions, using the 3x3 object for color
Expand pixels to the closest boundaries
Swap the colors in region marked by gray
XOR of left and right
Figure 6: Example tasks solved by induecion/transdacttioooohh See also Appendix C
Performance scales with dataset size, but quickly saturates with increasing number of seeds. We trained models while systematically varying the number of human-created seeds use, and varying the amount of synthetic data generated from those seeds (Figure 7 Performance improves with increasing training data for fine-tuning (increasing synthetic data), but saturates for increas- ing quantity of numan-created seeds. We conjecture that this saturation occurs because each seed serves introduce few core concepts, and that after enough seeds, essentially all of the important concepts have been demonstrated. This suggests that, beyond a critical threshold number of seeds, the method can scale with increasing amounts of compute without demanding further human data labeling. Looking beyond ARC, this means that methodology could probably be applied to other few-shot function-learning problems using a modest amount of manual data labeling.
increasing synthetic data, seeds
increasing seeds, synthetic problems
Validation Accuracy
0.150
Induction
Induction
0.125
Transduction
Transduction
0.100
0.075
0.050
20
40
60
80
100
25
50
100
num synthetic problems (10*3)
num seeds
Figure 7: Increased manual human effort (# seeds) does not significantly increase performance, but increasing compute spent generating synthetic data increases performance of the final model.

---

Published as a conference paper at ICLR 2025
Induction performance scales with test-time compute. We vary the test-time sampling budget for induction, finding an almost monotonic increase solve rate (Figure g left). In principle, drawing additional samples runs the risk of discovering solutions that are "false posiitvvsss,ves"" meaning they fit the training examples without correctly predicting the test output. In practice, about 9% of samples that fit the training examples are false-positives. Figure g (right) shows that about half of this 9% corresponds to problems where the majority of the probability mass is still placed on the correct output, meaning that a simple majority vote scheme would squash any false positives (e.g. clustering in AlphaCode Li et al. (022)) Appendix shows example false positives.
ARC Val Accuracy
20%
15%
# validation problems
Solve Rate
60
10%
Sample Oracle
5%
Sample + Filter
20
Codelt
0%
500
1500 number of samples
1000
2000
0.2 0.4 0.6
False Positive Rate
Figure 8: Left: Sample+Oracle assumes an oracle that selects one of the sampled programs. It upper-bounds the accuracy of randomly selecting one program consistent with the training exam- ples (Sample+Filter). Induction model trained with 100k gp14-desrrption-gnt4oom
data. Codelt (Butt et al., 2024) a recent neural program induction model for ARC, Right: Histogram of false positive rate. Of the problems that have false positives, about half have a false positive rate less than 0.5, meaning that most (filtered) samples predict the right test output.
Stronger LLMs make better synthetic data, and induction is more sensitive to data quality. To save costs, the previous results used GPT4o-mini to generate synthetic data. To understand the value of stronger models we generated 100k synthetic problems using GPT4 to generate natural language problem descriptions (but GPT4o-min still generated the code). The richer and more diverse synthetic problems elicited from GPT4 significantly improved performance, but primarily for induction, while transduction was less sensitive to data quality (Table T.
Table I: Val Ace: validation tasks that are correctly solved tries.
System
Val Acc.
Finetuning Data
Ensemble
26.50%
Induction, 2048 samples
18.78%
GPT-4 for generating descriptions, GPT-4o-mini for code
Transduction, beam size 20
15.25%
Ensemble Induction, 2048 samples
19.50%
11.07%
GPT-4o-mini for generating descriptions and code
Transduction, beam size 20
13.50%
SCALING OUR METHOD
Motivated by findings so far, we scaled up our method by producing two larger datasets:
ARC-Heavy: 200k problems from 160 seeds. The purpose of ARC-Heavy is to scale our method in an easily reproducible way, while also filling any gaps in its mastery of the training split. We first ran models from Section on the training split to identify 60 problems that they still struggled with, for which produced 60 new seeds, giving 160 seeds in total. From those seeds produced 200k synthetic problems, with GPT4 generating natural language descriptions and GPT4o-min generat- ing the corresponding Python code.
7

---

Published as a conference paper at ICLR 2025
ARC-Potpourri: 400k problems from heterogeneous sources. The purpose of ARC-Potpourii is to assemble the biggest dataset that we could, even if it comes from a messy mixture sources. Starting with ARC-Heavy added all synthetic data from Section We further added 100k transduction-only training examples from ReARC (Hodel 2024).
Test-time improvements. We improve transduction with test-time training (Sun al. (2020), ab- breviated TTT; Appendix E}. We expand our sampling budget to 20k programs.
We call our resulting systems BARC. Table the performance of various BARC models. Both transduction and induction are effective, with induction solving slightly more problems, until adding test-time training, after which transduction does slightly better. An ensemble scores 54.375%, passing previously published methods. Our best model performs nearly as well as the average human (60.2%) but much worse than the best humans (98%).
Table 2: % validation tasks correctly solved in 2 tries. Human results from LeGris et al. 2024
System
Val Acc.
Fair comparison?
ARC-Heavy: BARC models
Induction, 10k samples, majority vote
30.50%
Transduction (no TTT)
19.25%
Ensemble (no TTT)
37.50%
Transduction (TTT)
29.75%
Ensemble (TTT)
43.25%
ARC-Potpourri: BARC models
Induction, 20k samples, majority vote
38.00%
Transduction (no TTT)
29.125%
Ensemble (no TTT)
47.875%
Transduction (TTT)
39.25%
Ensemble (TTT)
54.375%
Codelt (Butt al.
2024
15%
Yes, only trains training
Claude-3.5
Greenblatt (0044)
21% / /42
Yes, but closed LLMs time
Wind 2020
39%
No, designed by looking at val set
Avg/Best Human 60.2% / 99.8%
Yes
WHICH PROBLEMS ARE EASIER FOR THE MODELS, AND FOR HUMANS?
Do problems that challenge humans also challenge the model, and vice-versa? We sort ARC validation problems into 5 equally- sized difficulty classes using data from LeGris iet al. (2024). Figure illustrates peculiar relationship between human and model accu- racy: All models surpass human performance on the hardest problems, but underperform on the easiest problems. Because our models train on simple Python programs, this suggests some problems are simple code and learnable by transformers, but very hard for people-ane and conversely that people possess priors allowing effortless solution of problems beyond what our Python program generator produces. For prob- lems of typical difficulty, the model roughly tracks human performance, and across all dif- ficulty levels, transduction and induction serve complementary roles, even when augmented with test time training.
transduction+ Induction ensemble
easiest 20%
model % solved
0.6
hardest
0.4
human % solved
Figure 9: Human vs model performance across difficulty levels. The easiest difficulty level con- tains problems in the top 20% of human accuracy, and the hardest difficulty level contains the 20% of problems with the lowest human accuracy.
8

---

Published as a conference paper at ICLR 2025
Which concepts easier for the models? We test ConceptARC (Moskvichev al. 2023 an alternative ARC-style test-set which classifies its tasks into "concept groups' each "above below." We models trained on ARCCPPtpourr' finding that specific concept categories are easier for induction or transduction (Figure TO). We find an intuitive division of labor between the approaches: Concept groups such as counting are best solved with symbolic code, while transduction better handles perceptual processes such judging whether shape is more horizontal more vertical, more top/bottom.
single isolated high-level concept such as "sameness"
ConceptARC reveals another dimension along which transduction and induction differ: Because ConceptARC illustrates one concept per problem, there is no need to compose many concepts gether. Therefore the induction model, which is uniquely equipped for symbolic composition, loses a key advantage. Transduction has more limited composition capabilities can instantiate indi- vidual concepts in flexible subsymbolic ways, which could explain why it excels ConceptARC.
100
Transduction
Ensernble
GPT4
66.7
IceCube
Accuracy
51.2
41.
19.1
0
Estractobjects
Center
Move MoverBBoundary
Count
CompleteShape
Order
Bauundryyy
SameDifferent
Cleantip
Rep8ottom20
FillaNoofitled
Copy
TopBottomaD
InsideeCutsida
HortzontaVertical
AboveBolow
All tasks pass@3
Induction
pass@1
transduction
Figure 10: ConceptARC accuracy by concept group. Concept groups sorted left-to-right by ratio of inductive transductive performance. IceCube is the original ARC Kaggle winner (Wind 2020).
We report pass@3 @3 because Moskvichev al. (202)) report accuracy given 3 attempts.
RELATED WORK
ARC was originally designed to challenge conventional deep learning and spur progress on alter- native paradigms (Chollet 2019). The first wave of successful approaches used discrete program search over domain-specifer programming languages, including the original Kaggle winner (Wind) 2020). These symbolic approaches held their own against GPT-4 (Wang al., 2024), but have cently been surpassed by transductive architectures using test-time training (Cole et al. (2024), and by LLM-guided program generation (Greenblaut) 2024). ARC has so far resisted conventional neu- ral and symbolic approaches, but is solvable for adult humans, and to some extent, children (LeGris)
al. 2024 Opielka al. 2024).
Code generation via LLMs is done in many recent works (Li et al. 2022 Gao et al. 2023 Chen
[02]] Austin et al. (022).. We most directly build on & Ellis (204 Greenblatt (2024). The former fine-tunes LLMs for inductive program synthesis using ILM-generater variations of human-written programs. While there are many technical differences, key factor is that we gen- erate function inputs by synthesizing an input_generatee ator function, rather than have an LLM directly generate possible inputs. This matters because an LLM alone could not generate complex, precisely-correct inputs such ARC grids. This potentially makes our work applicable other few-shot generalization problems with complex input-spaces such as webpages, robot planning, etc. Greenblatt] (2024) samples many Python programs from GPT4o: Comparable to induction model. but instead of fine-tune fine-tuning, it uses prompting. Fine-tuning forced us to create a dataset of new problems, which created the opportunity for exploring transductive models.
al., [202]]

---

Published as a conference paper at ICLR 2025
Classic work in neural program synthesis has previously compared induction and transduction (Ro- bustFill: Devlin et al. (2017)) We explore here a richer space of functions, reaching a qualitatively different conclusion than RobustFill: Instead of finding transduction inferior to induction, them complementary. More broadly, the transductive-iiductii divide lies near the heart of super- vised learning. Inductive approaches, such as linear regression, first construct a function where f(rrii)) & yrrain and then predict Mest = f (Trest. Transductive approaches, such as Support Vector Machines and In-Context Learning, instead output their predictions by performing direct compar- isons with the training data. We use the same neural network architecture and dataset to perform both tasks, allowing a controlled comparison between these paradigms.
find
8 DISCUSSION
What we learn about robust sample-efficient generalization. explicit symbolic hypothe- ses nor implicit neural representations suffice to solve all problems: each has their own domain of applicability, and simply ensembling models specialized in each does not cover all cases. Engi- neering a more clever neural program search, or training transductive predictors on more data, is unlikely to be fruitful. Instead we need representations irreducible to a purely neural or symbolic form, which intertwine inductive and transductive reasoning. One way of implementing this idea is to do program synthesis within language whose atomic primitives are non-symbolic, and to pre- train those primitives to encapsulate the basic atoms of core knowledge. While work has taken steps
in related directions (Reed & De Freitas 2015 Alet et al. 2018 Tang & Ellis 2023
Li Liet al. 2024).
how to engineer and scale this idea remains open.
To what extent is this methodology applicable beyond ARC? Few-shot function learning is very flexible framework, but our particular method is most applicable when the target generalization can be described in symbolic code. As an immediately tangible example, web scraping and other forms of data- munging could fit within our framework. As more ambitious goal, symbolic code is an especially good medium for defining precise models of how the world works. This true both within the natural sciences (Schmidt & Lipson, 2009) and also within AI, with examples such as robotic policies (Liang et 2023). planners (Wong et al., 2023). and world models more broadly (Das) eet al., 2023 Tang et al.) 2024b Evans et al. 2021 Liang al. 2024). These are not the kinds of programs that occur often in LLM pretraining data sO merely prompting is unlikely to perform well but nonetheless feasible to curate around 100 seeds demonstrating what the system should learn.
Theoretically, induction and transduction should not be so complementary. Equivalences be- tween induction and transduction are well-know, such as the 'kernel trick' which allows translating parametric function fitting into transductive problem. Our metalearning models, given infinite metatraining data, should similarly converge because transformers are universal function approxi- mators. That there remains a difference interesting precisely because it deviates from what one would expect theoretically.
Limitations. Our system does not grow more competent few-shot learning by solving new prob- lems: Instead, it bootstraps from manually encoded knowledge in the seeds, which is transformed into a few-shot learner via an LLM training/inference pipeline. A more compelling approach would be to have the system discover for itself the knowledge that we compiled for it within the seeds, for instance by practicing on training tasks, without supervising on ground truth solutions.
Our work is only evaluated on ARC. However, ARC is designed to contain many concepts and problems embedded within it, so can be viewed as an open-ended composite of different learning problems. Owing to this diversity, it is also notoriously challenging. and has resisted solution de- spite a series of profile competitions. We therefore believe that although evaluating on multiple benchmarks is desirable, ARC is appropriate benchmark to as the centerpiece of an experi- mental evaluation.
Code & Data Availability. Our code, data, and model weights are freely available https: //gi com/ xu3kev/BARC
Acknowledgements. We are grateful for advice from Robert Hawkins regarding the statistical anal- ysis in Figure 5C and for discussions with Weinan Sun about biological learning. This work was partly supported by NSF CAREER award to K.E.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
James Ainooson, Deepayan Sanyal, Joel P. Michelson, Yuan Yang, and Maithilee Kunda. A neurodiversityinnpiin solver for the abstraction & reasoning corpus (arc) using visual imagery and program synthesis, 2023, URL https: http:///axxvv.orro. www.oorg///////ooeeeeeeeeee.eese att/aaas.../...e./ 2362 69425
Ferran Alet, Tomás Lozano-Pérez, and Leslie P Kaelbling. Modular meta-learning, In Conference robot learning, pp. 856-868. PMLR, 2018.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Temr. Quoc Le, al. Program synthesis with large language models. Xiviv preprint arXiv:2 108.07732, 2021.
Natasha Butt, Blazej Manezak, Auke Wiggers, Corrado Rainone, David W Zhang, Michaël Deffer- rard, and Taco Cohen. Codeit: Self-improving language models with prioritized hindsight replay. In International Conference on Machine Learning, 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger. Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc- Grew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained code. arXiv preprint arXiv:2107.03374 2021.
François Chollet. On the measure of intelligence, 2019.
Jack Cole, Mohamed Osman, Michael Hodel, Keith Duggar, and Tim Scarfe. Machine learning street talk, June 2024.
Ria Das, Joshua B Tenenbaum, Armando Solar-Lezama, and Zenna Tavares. Combining functional and automata synthesis to discover causal reactive programs. Proceedings of the ACM on Pro- gramming Languages, 76PPLL) 1658, 2023.
Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Pushmeet Kohli. Robustfill: Neural program learning under noisy i/o. ICML, 2017.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshom, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez. Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song. Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Ander- son, Graeme Nail, Gregoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Ko- revaar, Hu Xu, Hugo Touvron, lliyan Zarov, Imanol Arrieta Ibarra. Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong. Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang. Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Rantala-Yeary, Lawrence Chen, Liang Tan. Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh,
Kshitiz Malik, Kuenley Chiu. Kunal Bhalla, Lauren Rantala- Yeary, Laurens van der Maaten,

---

Published as a conference paper at ICLR 2025
Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kam- badur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Olivier Duchenne, Onur AGelebi, Patrick Alrassy, Pengchuan Zhang. Pengwei Li, Petar Vasic, Peter Weng, Prajiwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong. Raga- van Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang. Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim. Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Ra- parthy, Sheng Shen. Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende. Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodin- sky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vig- nesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whimey Meers, Xavier Martinet, Xiaodong Wang, Xi- aoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang. Yaelle Goldschlag. Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho. Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti. Bran- don Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang. Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai,, Chris Tindal, Christoph Feichtenhoferr Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Ar- caute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco GuzmAan, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hamid Shojamazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk. Henry Aspegren, Hunter Gold- man, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Irina-Flena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Jean- Baan-Baptstt Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang. Joe Cummings, Jon Carvill Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang. Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Ke- neally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mo- hammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navy- ata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani. Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Sa- tadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lind- say, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang. Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen
12

---

Published as a conference paper at ICLR 2025
Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Tim- othy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vumtor Albiero, Vlad lonescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li. Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Con- stable, Xiaocheng Tang, Xiaofang Wang. Xiaojian Wu, Xiaolan Wang, Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary Ve Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang. and Zhiwei Zhao. The llama 3 herd of models, 2024. URL https: / org/ abs/ 2407 21783
Richard Evans, Matko Bodanjak, Lars Buesing, Kevin Ellis, David Pfau, Pushmeet Kohli, and Marek Sergot. Making sense of raw input. Artificial Intelligence, 299:103521, 2021. ISSN 0004-3702. doi: htts::/doi.oogg ottp:///...0011...rtt...00000000 103521. URL https: www.. sciencedirect.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang. Jamie Callan, and Graham Neubig. Pal: Program-aided language models. In International Conference Machine Learning, pp. 10799. PMLR, 2023.
Ryan Greenblatt. Draw more samples. httpss//reewworeeeerrcassssbssss compstttcccccm/// getting-50- 10tt-500ssttt-oo sota on arc--rrc-aap -acc--agi--ttii wegi-with--ee gpt 2024. Accuracy from ARCPrize Leader- board.
Céline Hocquette and Andrew Cropper. Relational decomposition for program synthesis. arXiv preprint arXiv:2408. 12212, 2024.
Michael Hodel. Reare. htttpss//gfftt...ccm//mcccocllle GitHub repository].
2024. [Online
Marcus Hutter. Universal artificial intelligence: Sequential decisions based on algorithmic proba- bility. Springer Science & Business Media, 2004.
Seungpil Lee, Woochang Sim, Donghyeon Shin, Wongyu Seo, Jiwon Park, Seokki Lee, Sanha Hwang, Sejin Kim, and Sundong Kim. Reasoning abilities of large language models: In-depth analysis on the abstraction and reasoning corpus, 2024. URL https:tt////mm /tttp:://axxv.oor//aa
Solim LeGris, Wai Keen Vong, Brenden M. Lake, and Todd M. Gureckis. H-are: A robust estimate of human performance on the abstraction and reasoning corpus benchmark, 2024. URL https:
// //arxiv.org/aat 2409 01374
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-auge augmented genera- tion for mowleege--ntennive nlp tasks. Advances in Neural Information Processing Systems, 33: 9459-9474, 2020.
Chengshu Li, Jacky Liang Andy Zeng, Xinyun Chen, Karol Hausman, Dorsa Sadigh, Sergey Levine, Li Fei-Fei, Fei Xia, and Brian Ichter. Chain of code: Reasoning with a language model- augmented code emulator. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st Interna- tional Conference Machine Learning, volume 235 of Proceedings of Muchine Learning Re- search, pp. 18259-28227, PMLR, 21-27 Jul 2024. URL https: htttp::/procceeting..o press v235/ 1025//i24ar..0 httm]
Wen-Ding Li and Kevin Ellis. Is programming by example solved by llms?, 2024.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser. Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science, 37866624 2022.
13

---

Published as a conference paper at ICLR 2025
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference Robotics and Automation (ICRA), pp.
IEEE, 2023.
Yichao Liang, Nishanth Kumar, Hao Tang, Adrian Weller, Joshua B. Tenenbaum, Tom Silver, JoAčo F. Henriques, and Kevin Ellis. Visualpredicator: Learning abstract world models with
neuro-symbolic predicates for robot planning, 2024. URL.
2410.23155
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- learner. arXiv preprint arXiv:/707.033 2017.
Arsenii Kirillovich Moskvichev, Victor Vikram Odouard, and Melanie Mitchell. The conceptARC: benchmark: Evaluating understanding and generalization in the ARC domain. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL
net/
ffrummid=8yk Bd=yykybttyyyyyyg
Gustaw Opielka, Hannes Rosenbusch, Veerle Vijverberg, and Claire E. Stevenson. Do large lan- guage models solve are visual analogies like people do?, 2024. URLhttps://arxiv.oo abs 09734
Scott Reed and Nando De Freitas. arXiv:1511.0627,, 2015.
Neural programmer-interpreeess.
arXiv preprint
Jürgen Schmidhuber. Optimal ordered problem solver. Machine Learning, 54()):1112544 2004.
Michael Schmidt and Hod Lipson. Distilling free-form natural laws from experimental data. science, 324(5923):81-55, 2009.
Ray J Solomonoff. A formal theory of inductive inference. Information and control, 7(1):1-22, 1964.
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time train- ing with supersuupo for generalization under distribution shifts. In International conference on machine learning, pp. 9229-92244 9248. PMLR, 2020.
Hao Tang and Kevin Ellis. From perception programs: Regularize, overparameterize, and amor- tize. In Andreas Krause, Emma Brunskill, Kyunghyun Cho. Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of 40th International Conference Machine Learn- ing. volume 202 of Proceedings of Machine Learning Research, pp. 33616-33631. PMLR, 23-29 Jul 2023. URL http::tpp///oomm http::://pocceednngg mlr press v200//s//ve///ee/// 2taz22222/an22c.htmm
Hao Tang, Keya Hu, Jin Peng Zhou, Sicheng Zhong, Wei-Long Zheng, Xujie Si, and Kevin Ellis. Code repair with llms gives an exploration-exploittiio tradeoff. NeurIPS, 2024a.
Hao Tang, Darren Key, and Kevin Ellis. Worldcoder, a model-based Ilm agent: Building world models by writing code and interacting with the environment. NeurIPS, 2024b.
Luca H. Thoms, Karel A. Veldkamp, Hannes Rosenbusch, and Claire E. Stevenson. Solv- ing are visual analogies with neural embeddings and vector arithmetic: generalized method. ArXiv, abs/2311.08088 2023. URL https:: //api. seeanniicccoolaa org/ Corpus1D:265158111
Ruocheng Wang, Eric Zelikman, Gabriel Poesia, Yewen Pu, Nick Haber, and Noah D Goodman. Hypothesis search: Inductive reasoning with language models. ICLR, 2024.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi. and Hannanch Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In The 61 Annual Meeting The Association For Computational I. Linguistics, 2023.
Johan Sokrates Wind. 1st place 2020 are kaggle. http:://github..cmm/pppp ARC- PRC-solution 2020. [Online GitHub repository).
14

---

Published as a conference paper at ICLR 2025
Jonas Witt, Stef Rasing, Sebastijan CommanAmAGo Tias Guns, and Claus Chrssthhis Carbon. A divide-align-conquer strategy for program synthesis, 2023. URL https://arxiv.orrs abs/ 0DSS/231.000 03094
Lionel Wong, Jiayuan Mao, Pratyusha Sharma, Zachary S Siegel, Jiahai Feng, Noa Korneev, Joshua Tenenbaum, and Jacob Andreas. Learning adaptive planning representations with natu- ral language guidance. ICIR, 2023.
A DATA GENERATION TECHNICAL DETAILS
Step1: Random remix to generate language descriptions
Prompt example
randomily sample
new language
problems
concepto:
language
new concapts
new description:
Step2: RAG generate codes
Prompt example
calculate cosine similarity with
get four codes similar language new description
language
osnoepts:
language part
desoription:
new language
code pistrat Legatlll transto no g106:
LIM
new code
propate_irputti www.foww.rrdtiwwwc.....
100 seed problems
now deseriptire
new language
concepti.... new descrittinn..
Figure 11: A new natural language description is sampled by prompting an LLM with seed natural language descriptions, effectively using in-context learning to recombine and mutate elements of different problems. Code generated for that new description via Retrieval Augmented Generation (RAG). Our RAG pipeline retrieves seeds with similar descriptions, and prompts an LLM to generate code for the new description, given retrieved seeds.
The prompt template for generating natural language descriptions by randomly sampling language descriptions from seed problems is as follows:
You've generated these on previous requests:
(examples)
Brainstorm (num_generations) more, using samilar thinking:
"python
concepts:
<concepts in your generation
description:
<description of your new generation>
The prompt template for generating Python code from the natural-language descriptions (and similar example code retrieved from the seeds, via RAG) is as follows:
15

---

Published as a conference paper at ICLR 2025
You puzzle maker designing geometric, physical, and topological puzzles for curious middle-schoolers.
Each puzzle consists of uncovering deterministic rule, pattern, procedure, algor thm, transformation law that maps inputs to outputs. Both the inputs outputs are grids colored pixels. There are 10 colors, but the order of the colors is never relevant to the puzzle.
The middle schoolers transformation, which be implemented Designing a puzzle involves also creating example inputs, which can be implemented as Python function called 'generate_inge generaae_iinpute function should be stochastic, that every time you run it, you get another good example of what the transformation be applied to.
trying discover this deterministic
Python function called 'main',
Unlike 'main', the
Here is a averview of the puzzle you are designing:
[description]
Please implement the puzzle by writing code containing the 'generate input' and 'main' functions. Use the following standard library ('common. ))
" ppython {common_ling _lbb}
Here examples from puzzles with similar descriptions show you how to use functions 'common.py's
(examples)
task implement the puzzle, following these steps:
1. Inspect the example puzzle implementations, making note of the functions details
used and the
2. Inspect the puzzle's description the new puzzle
3. Brainstorm a possible implementation
4. Generate a code block formatted like the earlier examples with a comment starting '# concepts:" listing the concepts and '# description:* describing the inputs and transformation from the given description.
Be sure to make the transformation 'main' deterministic. Follow the description closely.
Execution and Filtering of Generated Problems We heuristically filter problems to improve the quality of data based on the following criteria:
The generator and transformation functions can be executed, producing at least 4 input- output grids examples.
Transformation being deterministic:
We check for consistency by running the functions with different random seeds and filter out those with non-deterministic outputs.
Appropriate grid sizes: We remove input-output grids with height or width larger than 30, aligning with grid sizes in ARC
Color permutation check: Since we use numpy arrays with integers 0-9 to represent colors, we want ensure transformations don't don't rely arithmetic operations of these integers. We filter this by checking if input-output remains consistent when permuting the underlying color-number mapping.
Removal of problems with all trivial identity input-output examples.
16

---

Published as a conference paper at ICLR 2025
A.1
SEED EXAMPLES
Example seed 0d3d703e
Example seed 1b2d62fb
Example seed 0dfd9992
Figure 12: Three seed examples
*MMasnsasssaee: problem id: 0d3d703e from common import
import numpy np from typing import
# concepts:
color mapping
description:
The input is grid where each column is of the same color. # To make the output, change each color occording to the following mapping:
green yellow, blue gray, red pink, teal -> maroon, yellow green, gray -> blue, pink -> red, maroon -> teal
def transform_grri((innt...ggg Initialize output grid
output_grid input_grid.copy ()
Performs color mopping
output_gric (lambdaa color: color_man color_map.ee color_map.get get(color, color)
(output_grid)
return output_grid
Constructing the color map [Color.GREEN Color. YELLOW,
color_map
BBLE Color Golor,GGRAY,
Color. Color.PINK,
Color. : Color.MAROOS
Color Color. Color.GREEN,
Color.GRAY BLUE,
Color, Color
MMROOON Color. TEAL
def inperateeinpuueee
grid np.full((3, 3), Color.BLACK for x range(grid.c shape [e]):
grid[x, :] random. choice (1 (lisoocolor_mpp) ()))
return grid
P problem id: 1b2d62fb
import numpy as np from typing import
from common import
17

---

Published as a conference paper at ICLR 2025
concepts:
boolean logical operations, bitmasks with separator
description:
In the input you will two maroon bitmosks separated by blue vertical bar To make the output, color teal the pixels that are not set in either bitmasks (logical NOR)
def mennsformmgriddinnptt....
np.ndarray) np.ndarray: Find the blue vertical bar. Vertical means constant for x x_bar in sinnputgriddshapp.......... :] == Color.BLUE)
if ap.aal(inut__grid(___ break
left_mask :x :] right_mask input_grii(xx_arr11., :]
output_grid up.zzero__iikeeleff_mmee
output_grid[(leett Color.MAROON (right_mas Color.MARON
)] Color.TEAL
return output_grid
def generate_input() -> nd.nrarryy:
create pair of equally sized maroon bitmasks width, height
10), randint(2, 10)
gridl zeros ( (width, height), dtype=int)
grid2 = np. (width, height), dtype=int)
for x in range(width):
for y range (height):
gridl[x, y] np. random. choice [Color. MAROON, Color.BLACK] )
CMRROON
grid2[x, = random. choice [CColor.MAROON, Color.BLACEE
create a blue vertical bar bar np. (1, height), dtype=int) bar :] Color. BLUE
grid np. www.onntttteeegrrrir.ll bar, grid2), axis=0)
return grid
"
problem id: 0dfd9992
from common import
import numpy np from typing import
concepts:
# occlusion, translational symmetry
description:
In the input you will see a translotionolly symmetric pottern randomly occluded by black pixels. To make the output, remove the occluding black pixels to reveal the translationally symmetric pattern.
def transforr_grid(innut..eg # Plan:
# 1. Find the tronslational symmetries
18

---

Published as a conference paper at ICLR 2025
# 2. Reconstruct the sprite by ignoring the black pixels ond exploiting the symmetry
W, input_grid.shhap
Identify the translational symmetries translations stw.transllaaiooaaosettttttttty
ignore_colorse (oooorrrrsscorrrrooooooooooooo BLACK])
assert len(translations)
0, "No translational symmetry found"
Reconstruct the occluded black pixels by replacing them with colors found the orbit of the symmetries
output_grys = np. copy (input_grid) for in range
for y range(h):
if output_grid y] Color.BLACK:
# Use the translationol symmetry to fill in the occluded
pixels
# do this compute the ORBIT of the current pixel under the translations
# and take the most common non-black color in the orbit
Compute the orbit into the output orbit_pixels orbit(output_grid) x, y, translations) transformed,
orbit_colors = finput_grid(ttrasffrorr for transformed_x, transformed_y in orbit_pixels)
# occluded by black, so whotever color it is, black doesn 't count
orbit_colors orbit_colors {Color.
Copy the color
assert Len(orbit_color: == 1, "Ambiguity: multiple colors in the orbit"
output_griddx, y] orbit_colors.popt
return output_grid
def generate_input():
Make random large canvas
grid (n..ful(((p..rndnn randint (15, 30), np. random. randint 30))
Color. (BACKK
Make the basic sprite w, random.randiite 8), random. randint sprite random_spp Londom_sprite((e (w, h, density=1, color_pa color_paleette:Colrr.woo
Place the sprite in the canvas
for x range(0, shide w): for y in range grid. h) blit_sprite(grid, sprite, x, y)
Create random occluders pooccluders random. randint(1, 5)
for in range(n_occludersg) grid. Fandom.randint(C grid.
x, = random. randint
shape [1])
w, random. randint 7), random, randint 7)
occluder_ ciirrrspptteeee np. fo.full((w, h), Color. BLACK)
olit_sprite(grian accluder_sprite, x, y)
19

---

Published as a conference paper at ICLR 2025
return grid
A.2 COMMON LIBRARY
20

---

Published as a conference paper at ICLR 2025
"""Common library for ARC
import numpy as np import random
class Color:
Enum for colors
Color. CoLor.BLACK, Color.BLUE, Color. RED, Color. GREEN, Color. YELLOW, Color. GREY, Color. Color.ORANGE Color. TEAL, MAROON
Use Color.ALL_ COLORS for 'set' of all possible colors Use Color.NOT_BLACK for 'set' of oll colors except black
Colors are strings (NOT integers), so you CAN'T do http///rttmmeic/indeeeeet (The exception is Color.BLACK which is 0) ann
them.
def flood_ (igd_f y, color, connectivity=4): unK
Fill the connected region that contains the point (x, y) with the specified color.
connectivity: or 8, for 4-way or 8-way connectivity. 8-way counts diagonals connected, 4-way only counts cardinal directions as connected.
unn
def draw_ (ineidd x, y, end_x=None end_y=None, length=None, direction=
None,
color=None, stop_at_color::))
Draws Line starting at (x, y) extending to (end_x, x, end_y) of the apecified length in the specified direction
Direction should be a vector with elements -1, 0, or 1.
If length is None, then the line will continue until it hits the edge of the grid.
stop_ar at_ atlorloo: optional list colors that the line should stop at. If the line hits a pixel of one of these colors, it will stop.
Example:
blue diagonal line from (0, 0) (2, 2)
draw Line (grid, 0, 0, length=3, color=blue, direction= (1, 1)) draw_line 0, 0, end_x=2, end_y=2, color=blue)
def finn__coneetee__cooopnnntttt/rr background-color.Buu BLACK, connectivity
4, monochromatii-=Truus
Find the connected components in the grid.
Returns a list of connected
components, where each connected component is a numpy arroy.
connectivity: or 8, for 4-way or 8-way connectivity.
monochromatic: if True, each connected component is assumed have only one color. If False, each connected component con include multiple colors.
21

---

Published as a conference paper at ICLR 2025
def random_scatter_ponnts
color, density=9.5, background-Color- BLACK)
Randomly scatter points of the specified color in the grid with specified density.
def Scale_pattern(pattere scale_factor).
Scales the pottern by the specified foctor.
def blit_ (bject obj, background-color.BBLkkk) unn
Draws an object onto grid using its current location.
Example usage:
bl blit_object (output_grid, an_object, bockgroundtacckgrrndd.dooo)/)/ nnn
def blit_sprite(grid, sprite, x, y, background-color.BBLAG
":"
Draws sprite onto the grid at the specified location.
Example usage:
blit_sprite (outputtgttuuu the sprite, x=x, y=y, backgroundd-ackggrondd..oooo
unK
def bo background-Color.BBAGC
Find the bounding box of the non-background pixels in the grid, Returns a tuple (x, y, width, height) of the bounding box.
Example usage:
objects = find_ cot.:ccoeeeedccopooeeeeeee (inputt_riitttt monochromotic=:rue,,
backgrouun-ccllor.BLCCC BLACK, connectivity=9)
teal
teal_objectee object=[obj for obj in objects if np. any (obj Color. TEAL)] teal_x, teal_y, teal_w, teal_h bounding_ box (teetooottttobjjcct) .nn
def object_positionnobi, background-Color. BLACK, anchor="upper left"):
.""
(x,y) position of the provided object. By default, the upper left corner.
anchor: "upper left", "upper right", "lower left", "lower right", "center", "upper center", "lower center", "left center", "right center"
Example usage:
X, object_positionnoee (obj, backkgrounttsockgronnd..oooo anchor= left")
middle_x, middle_y = object_position(bb), backerrount-bckkrrunddoooood
onchor="center"")
def crop(grid, background unn
Crop the grid the smallest bounding box that contains oll non-background pixels.
Example usage:
22

---

Published as a conference paper at ICLR 2025
# Extract sprite from object sprite crop backgrrounttbokkgronddd.ooo//// .."
def translate x, y, background-ColorrBBBAG. ana
Translate by the vector (x, y). Fills the new pixels with the background color.
Example usage:
red_ object ... extract object shifted_red recis_erd__jeceee translate (red_ object, x=1, y=1)
redjjjjet, object, bockgrounclbcckgromdd.ooooo
nnn
def collision(_=Nore objectI=None, object2=None, x1=0, y1=0, x2=0, o Minkerround-Cooo.BLA.G...
Check if object1 and object2 collide when object1 is at (xI, y1) and object2 is at (x2, y2).
Example usage:
Check if sprite can be placed onto a grid at (X,Y) (abiisson((jjeettooutttptttt
sppicte= x2=X, y2=V)
Check if two objects collide object2=object2, xi=XI, y1=Y1, x2=X2, y2=Y2)
def contact(_-None, object1-None object2=None, x1=0, y1=0, x2=0, y2=0, background-Color. BLACK, connectivity=4,))
Check if objectl and object2 touch each other (have contact)
when objectl is at (x1, yl) and object2 is at (x2, y2).
They are touching each other if they share a border, or if they overlap.
Collision implies contact, but contact does not imply collision.
connectivity: or 8, for 4-way 8-way connectivity.
(8-way counts diagonals touching, 4-way only counts cardinal directions as touching)
Example usage:
Check if sprite touches anything if it were be ploced at (X object2=o_sprite, x2=X, y2=V)
contact (ootttttooottt....uuuutgggo
# Check if two objects touch each other (obtacttobjetttlobbeee object2=object2) ...
def generate_position_has hntpe_raaasnttttaaaaaaaa (max len, position_num, if if_padding= False):
"""
Generate the position of the lines with random intervol.
def rranoo..fre__oaattoooooooorrrrrr sprite, background-Color.ce BLACK, border_size=0, padding=0, padding_conecttiityy--.
Find a random free location for the sprite in the grid
23

---

Published as a conference paper at ICLR 2025
Returns tuple (x, of the top-left corner of the sprite in the grid, which can be passed 'blit_sprite'
border_size: minimum distance from the edge of the grid background: color treated as transparent
padding: if non-zero, the sprite will be padded with non-bockground
color before checking for collision
padding connectivity.. or 8, for 4-way or 8-way connectivity when padding the sprite
Example usage:
x, = random_ free_
sprite, podding=1,
padeeing_connecttve connectivity-=8 border_size=1, background=Color BLACK) # find the location, using generous podding
assert not collision (objecti=grid, object2-sprite, x2=x, y2=y) blit_sprite(grid, sprite, x, y) Inn
def penect_iineeroorgrriie
Computes the interior the object (including edges)
returns a new grid of 'bool' where True indicates that the pixel is part of the object's 's interior.
Example usage:
interior = object_interione ioterior bockground=Color..BAACC for in np. orgwher (interior):
x,y is either inside the object or at least on its edge
unn
def bereec_boundary(gritee background-Colon BLACK)
Computes the boundary of the object (excluding interior)
returns a new grid of 'bool' where True indicates that the pixel is part of the object's boundary.
Example usage:
boundary object boundary bockground-Color..LAkk)
assert np. al [ob][bondddy) != Color.
def objectt,eigtboos(@grid,, background=Color. BLACK, connectivity=4) nn"
Computes a mask of the points that neighbor or border the object, but are not part of the object.
returns a new grid of 'bool' where True indicates that the pixel is part of the object's border neighbors5.
Example usage:
neighbors oboectt_eeeieeeeeee objectt_nighbooss(ttj] bockground=Color.c BLACk) assert np. all (obj Color. BLACK)
class Symmetry:
Symmetry tronsformations, which tronsformed the 2D grid in ways that preserve visual structure.
Returned by 'detect_
'detect_transe peetect_rotationnl_symeee .""
'detect_.trror_symmet
def apply(self, x, y, iters=1):
24

---

Published as a conference paper at ICLR 2025
Inn
Apply the symmetry transformation to the point (x, y) 'iters'
times.
Returns the tronsformed point (x',y²)
def orbit(grid, x, y, symmetries):
":"
Compute the orbit of the point (x, y) under the symmetry transformations 'symmetries'.
The orbit is the set of points that the point (x, y) mops to ofter applying the symmetry trans transformat formatioos different numbers times. Returns a list of points in the orbit.
Example:
symmetries detect, connttionnlsymmerrryyooo ((nputtggrtgtggrr for x, y in np.ar arghere (input_gria != Color. BLACK)
# Compute orbit on to the target grid, which is typically the output
cymmetric_poineee
orbit
x, y, symmetries)
now we do something with them like copy colors or infer
missing colors
def detect_ sestetttrannlattionasssseeettt .nn
(gnoreccoloss
Finds translational symmetries in grid.
Satisfies: grid[x, y] grid[x + translate_x, y translote_y] for all x, y, long neither pixel is in "ignore_colors'.
Returns a list of Symmetry objects, each representing a different translational symmetry.
Example:
symmetries
ool.:reeidoiigooooooooooooooo ignore_colors-t occluder_color)
for in np. wp..arghere (grid != occluder_color):
Compute orbit on to the target grid
When copying to an output, this is usually the output grid symmetric_ pepetriccpppisssscc = orbit(grid, x, y, symmetries) for
in symmetric_points:
assert grid[x, y] == grid[x, y] or grid[x, y] == occluder_color
def detect_mirror, htt.:mrroorssmttyyerrrrr.r (cro
Returns list of mirror symmetries.
Satisfies: grid[x, y] grid[2=mirror.. X, 2+mirror_y y] for all x, y, as long as neither pixel is in "ignore_colors'
Example:
symmetries detect_minee waaismiirrrr_ssmeee serrrsytmtttyyriiie wewiiiinnooorssaeeeeeeeooeeeeeee colors=[Color..deeeoooooooc BLACK)) # ignore__olle color: In case parts of the object have been removed and # occluded by black
for x, y in np. orgwhere (grid != Color.BLACK) for sym in symmetries: symmetric_x, symmetric_y sym, apply (x, y)
assert wriddssymetric... symmetric == grid[x, y]
or grid[synmettrcc. symmetric_y] == Color.
If the grid has both horizontal and vertical mirror symmetries, the returned list will contain two elements.
25

---

Published as a conference paper at ICLR 2025
def (ereett.rotafooollssyytttttrrrr ignore_colors= www.coolooosoooooorooooooooo
Finds rotational symmetry in grid, or returns None if no symmetry
is possible.
Satisfies: gridíx, yJ gridly
Sitote_cente__. -x rotate_ centerry rotate_center_x]
# clockwise
grid[x, y] = grid[-y potate_center_e + Petote_cente__11
rotateetteeeeeeeee centery rotate_ccnter.a
# counterclockwise
for all x,y, as long as neither pixel is ignore_colors
Example:
sym detect_rotations syyerroooo_smmmttyyggrri,, ignore_ cowoooooooooooo.olloeo BLACK]) @gnore_color, In case parts of the object have been removed and # occluded by black
for x, in ap.argwhere
!=
rotated_x, rototed_y sym.apply(x, y, iters=1) +1 clockwise, -1 counterclockwise assert grid[rotated_ rotated_y] grid[x, y] or grid[rototed_x, rotated_y] == BLACK
print center_x, cymterner_y) # In case these are needed, they are floats
def js_contiguous (bi (bitmask, Background-Color.BLAG connectivity-4)
Check if array is contiguous.
Color that counts as tronsporent (default: Color.BLACH connectivity: or 8, for 4-way (only cardinal directions) or 8-way connectivity (also diagonals) (default: 4)
background:
Returns True/False Inn
def randon_ random_sprite(n, m, density=0.5, symmetry=None, cowlor_paleteeeeeeeeeeeeeeeeeeee connectivity=4, background-Color. BLACK)
Generate a sprite (an object), represented as numpy arroy.
n, m: dimensions of the sprite. If these are lists, then random value will be chosen from the list.
symmetry: optional type of symmetry to opply to the sprite. Con be 'horizontal', 'vertical', 'diagonal', 'radial', 'not_symmetric'. If None, random symmetry type will be chosen.
color_paletter optional list of colors to use in the sprite. If None, a random color palette will be chosen.
Returns an (n,m) NumPy array representing the sprite. InK
def detect_objects _Nooee, predicate=None, background-Color. BLACK, monochromatic=Fal connectivity-Noni allowed dimensions=None, colors=None, lan_overlap=Fallse
I
Detects and extracts objects from the grid that satisfy custom specification.
predicate:
a function that takes candidate object as input and returns True if it counts as on object
background:
26

---

Published as a conference paper at ICLR 2025
color treated transparent monochromatic:
if True, each object assumed to have only one color If False, each object can include multiple colors. connectivity:
4 8, for 4-way 8-way connectivity. If None, the connectivity is determined automatically. al lowed_ dimensions:
a list of tuples (n, m) specifying the allowed dimensions of the objects.
If None, objects of any size al ollowed. colors:
a list of colors that the objects are allowed to hove. If None, objects of any color are allowed.
can overlap:: if True, objects can overlap. If False, objects cannot overlop.
Returns a list of objects, where each object is numpy orray.
27

---

Published as a conference paper at ICLR 2025
concepts:
concepts:
concepts:
color inversion,
pixel sorting. frequency counting. color arrangement
shape completion, filling
objecxt transformation
concepts:
concepts:
concepts:
symmetry detection, reflection
color blending, radial expansion
object detection, shape expansion, color filling
concepts:
concepts:
concepts:
pattern detection, copying, positioning
inverse patterns, scaling
color blending. shape merging
Figure 13: Nine example problems generated automatically by our pipeline.
28

---

Published as a conference paper at ICLR 2025
A.3 GENERATED ARC EXAMPLES
B
FINE TUNING TRAINING DETAILS
B. PROMPTING THE MODELS
We must include in our prompts for our fine-tuned models the input/output 2D colored grids of each problem. To do this we represent the problem textually by naming the colors one-by-one, We renamed certain colors which were more than one token (e.g., maroon->brown saves token/pixel), and presented the grid as a whitespace-delimited 2D) array with newlines delimiting rows. Please see below.
examples Exmppee 1
example
Eiampi 2
Example 3
Figure 14: Prompt example illustration
Transduction example:
Role: system
You are world-class puzzle solver with exceptional pattern recogni skills. Your task is analyze puzzles, spot patterns, provide direct solutions.
Role: user
Given input- grid pairs reference examples, carefully observe the patterns to predict the output grid for test input. Each pair follows the same transformation rule. Grids are 2D arrays represented as strings, with cells (colors) separated by spaces and rows by newlines. Here are the input and output grids for the reference examples: Example
Input:
Black Gray Black
Gray Gray Gray
Black Gray Black
Output:
Black Black Blue Red Black Black
Black Black Red Blue Black Black Blue Red Blue Red Blue Red
Red Blue Red Blue Red Blue
Black Black Blue Red Black Black Black Black Red Blue Black Black
Example
Input:
Gray Black Black
Black Gray Black
Black Black Gray
Output:
Blue Red Black Black Black Black
Red Blue Black Black Black Black
Black Black Blue Red Black Black
Black Black Red Blue
Black Black
29

---

Published as a conference paper at ICLR 2025
Black Black Black Black Blue Red Black Black Black Black Red Blue
Example
Input:
Black Gray Black
Black Gray Gray
Gray Gray Black
Output:
Black Black Blue Red Black Black
Black Black Red Blue Black Black
Black Black Blue Red Blue Red
Black Black Red Blue
Red Blue
Blue Red Blue Red Black Black
Red Blue Red Blue Black Black
Here is the input grid for the test example: Input:
Black Black Black
Black Gray Black
Gray Gray Gray
Directly provide the output grids corresponding to the given test input grids, based on the patterns observed in the reference examples.
Role: assistant
The output grid for the test input grid is: <c.
Black Black Black Black Black Black
Black Black Black Black Black Black
Black Black Blue Red Black Black
Black Black Red Blue Black Black
Blue Red Blue Blue Red
Red Blue Red Blue Red Blue
Induction example:
Role: system
You are world-class puzzle solver with exceptional pattern recogni tion skills and expertise in Python programming. Your task is to analyze puzzles and provide Python solutions.
Role: user
Given input-output grid pairs reference examples, carefully observe patterns to predict the output grid for new test input. Each pair follows the same transformation rule. Grids are 2D arrays represented as strings, with cells (colors) separated by spaces and rows by newlines.
Here the input and output grids for the reference examples: Example
Input:
Black Gray Black
Gray Gray Gray
Black Gray Black
Output:
Black Black Blue Red Black Black
Black Black Red Blue Black Black
Blue Red Blue Blue Red
Red Blue Red Blue Red Blue
30

---

Published as a conference paper at ICLR 2025
Black Black Blue Red Black Black Black Black Red Blue Black Black
Example
Input:
Gray Black Black
Black Gray Black
Black Black Gray
Output:
Blue Red Black Black Black Black
Red Blue Black Black Black Black Black Black Blue Red Black Black
Black Black Red Blue Black Black
Black Black Black Black Blue Red
Black Black Black Black Red Blue
Example
Input:
Black Gray Black
Black Gray Gray
Gray Gray Black
Output:
Black Black Blue Red Black Black
Black Black Red Blue Black Black
Black Black Blue Red Blue Red
Black Black Red Blue Red Blue
Blue Blue Black Black
Red Blue Red Blue Black Black
Here the input grid for test example: Input:
Black Black Black
Black Gray Black
Gray Gray Gray
Write Python function 'transform' that can convert any given input grid to its corresponding output grid based on the pattern observed in the reference examples.
Role: assistant
Let's solve this puzzle using Python code with the common library functions. We'll first reason about the problem and then write the code to solve it. The 'transform' function will take the input grid and return the output grid Here is the Python code with the comments describing how solve the problem:
"python
from common import
import numpy as np from typing import
# concepts:
repetition, diagonal lines, color change
description:
# In the input you will see 3x3 grid with specific diagonal pattern of gray pixels.
The output should be the result repeating this diagonal pattern horizontally and vertically
31

---

Published as a conference paper at ICLR 2025
to fill 6x6 grid, changing the color from gray to blue and red in an alternating manner.
def transform(input_grr np.ndarray) -> np.ndarray: Create new 6x6 output grid filled with the background color (black) output_grid np. ful 6), Color.BLACH
Check for the diagonal pattern in the input grid for in range(3):
for in range(3):
if input_grid jl
Fill same position the output grid with alternating
colors
output_grid[i 2, 1 2] = Color.Blu BLUE output_ggid{1 2,
1] Color. RED
output_grid(1 +
Color. RED
output_grid(1 + 1, 1] Color. BLUE
return output_grid
C EXAMPLES OF INDUCTION SOLUTIONS
C.I ARC PROBLEM F3CDC58F
from common import
import numpy np from typing import
# concepts:
color counting, grid transformation
description:
In the input, you will see a grid with several colored squares.
To make the output, count the number of squares of each color.
Fill the output grid with bars of corresponding colors. The height of each bar corresponds the number of squares of that color.
def transform nd.rray)yy) -> np. adarray: Create a blank output grid output_grid np. likeos___kkette (input_grid) ((nppuug_))))())uuttgrrid))
# Count the number of squares of each color color_counts = {color: for color in Color.NOT BLL_KBACK]
for row in input_gria
for cell in row:
if cell in color_ color_counts[ccll]
Define the height of the bars based on the counts max nax_height 10 # Maximum height of the bars for color, count in color_counts. items(): if count 0:
for h range (cout,,,, max max_height)) output_grid[-(1 + 1), (color 1) % len NOT ] = color # Fill the output grid with
32

---

Published as a conference paper at ICLR 2025
colors in bar representation
return output_grid
C.2
ARC PROBLEM BR03B35F
from common import
import numpy as np from typing import
concepts:
color detection, pottern extraction, rotation transformat in
description:
the input, you will see 5x5 grid consisting of a colored pattern in the top left corner and a 2x2 square of red pixels in the bottom right corner.
Depending on the color of the top
left corner (blue or green), rotate the pattern by 90 degrees clockwise or counterclockwise.
The output grid should be the trans formed pottern placed in the same position the original top left pattern.
def transform(fnput__rri
# Extract the top left 5x5 pattern pattern input_grid(0:5, 0:5]
# Identify the color in the top left corner corner_color = pattern[e, 0]
Define rotation based on the corner color
if corner_color Color.BLUE
transformed_patten = op.r k=-1) # Rotote 90 degrees counterclockwise
elif corner_color Color. transformed_pattern np. rot3 (pattern, k=1) # Rotote 90
degrees clockwise
else:
raise ValueError ("The corner color must be either BLUE GREEN.
# Create the output grid
output_grid np. fullu(( 2), Color.BLACK, # Initialize small grid output_grid blit_sprite (output_grid, frams_orrmd_patee x=0, y=0, background-Color. BLACK)
return output_grid
C.3
ARC PROBLEM 2072ABA6
from common import
import numpy np from typing import
concepts:
color mapping, pottern repl icaticatio
33

---

Published as a conference paper at ICLR 2025
description:
In the input you will see a 3x3 grid of colored pixels.
The colors are either black or gray, The output should be o grid where:
# I. If the pixel is black, it remains black in the output.
# 2. If the pixel is gray, it should be replaced with a 2x2 block of blue and red pixels in checkerboard pattern.
The blue pixel should occupy the top-left and bottom-right of the 2x2 block, while the red pixel occupies the top-right ond lettom-left.
def transform(input_grrd)):
# Create an output grid that is larger than the input grid output_grid app.zeros(fnput__rrii) (input_grid. [hape[0]] 2, soout_grid.shapee 2), dtype=int)
np.zeros
Fill the output grid based on the input grid for x in (nneettnttt_grdddshapeeeeee
for y in range (ta.eennnnngrrr.ssspeeeeeee if input_grid y] == Color BLACK:
Block stays black in the output continue
else:
# Replace gray with checkerboard pattern of blue and red
output_grid x:2 + 2, y:2 y 21 [ [Color. BLUE, Color. RED], [Color.RED] RED, Color.BLUE.
return output_grid
C.4 ARC PROBLEM EF26CBF6
from common import
import numpy as np from typing import
concepts:
color transformation, grid sections, boundary detection
description:
In the input, you will grid with pattern of yellow ond blue
pixels with a black background, and multiple colored circles (not yellow or blue) placed rondomly within the grid.
# The goal is to tronsform the output grid by replacing all the blue pixels with the color of the closest circle
and keeping the yellow pixels unchanged.
def lransformiinut__gridt Create a copy of the input grid modify output_grid np. copy (eapy(inpun (input_grid)
ndarrry) -> ndarray:::
# Find the coordinates of the colored circles dircle_coordinates np.argwhere (iiput_grid I=
( input_grid Color. YELLOW) & input_grid I= Color.BLUE).
# Iterate over the grid and replace blue pixels with the neorest circle color
34

---

Published as a conference paper at ICLR 2025
for in Taage(innuu__ge shape [1]):
for y in
if input_grid y] BLUE:
# Find the closest circle
[losest_color None
closes_distaace float('inf') for circle_x, circle_y y in circle_cee circle_coordinates coordiiates::
distance abs acircclee__ x) abs cbs y) #
Manhattan distance
if distance < closest_distance closest_distance distance closest_color input_grid[circles circle_yl # Replace the blue pixel with the closest circle color output_grid(xx yl closest_color
return output_grid
C.5
ARC PROBLEM E7639916
from common import
import numpy as np from typing import
# concepts:
boundary detection, color filling
description:
# In the input grid, you will see block background with several purple pixels forming boundary.
The task is fill the area enclosed by the purple boundory with blue pixels.
def transform(tnput_rrine
# Create output grid that starts copy of the input grid output_grid copy (input_grid)
# Find the coordinates of the purple pixels purple_coords ar
== Color.
If no purple pixels are found, return the original grid
if len (eepppllee_coore return output_grid 0:
Get the boundories of the purple pixels min_x, min_y np. pp.min(purplee_cordss axis=0) max_x, max_y = mp. (purple_coords, axis=0)
Fill the area enclosed by the purple boundary for x in range (min_,min_x, max_x + 1):
for y range(min_y, max_y + 1):
# Check if the current position is outside the purple boundary
if (x min_x
max_x y == min_y y max_y) and Color.BLACK y]
output_grid(x, y] Color. BLUE
return output_grid
C.6 ARC PROBLEM C074846D
35

---

Published as a conference paper at ICLR 2025
from common
import
import numpy as np from typing import
concepts:
rotation, color change, symmetry
description:
the input, you will see colored object with a single groy pixel. To make the output, rotate the object 90 degrees clockwise oround the gray pixel,
and color the newly exposed pixels green.
def (oansforn(tnput_grid) # Find the gray pixel location gray_pixel_locass
= np. argwwhree(inputgrrie == Color. GRAY) ==
assert gray_x, gray_y gray_pixxe_Locctionss]]
Create an output grid
output_grid np. (flt(innttgrrid.shaa Color.BLACK,
Rotate the object around the groy pixel for x in (inguti
for y in range grid. shape [0])
[1]): (1)p
if input_grid[x, y] != Color BLACK and input_gridie y] !=
GRAY:
Calculate new position ofter 90 degrees clockwise rotation
new_x x gray_x (y gray_y) new_y gray_y (x gray_x)
# Check if the new position is within bounds if 0 <= new_x
ceapputggrddshapeeeeeee and 0 <= new_y output_grid.shape [1]
Place the rotated pixel the output grid output_grid new_y] input_grid y]
# Color newly exposed pixels green if output_grid y] Color BLACK: output_grid(x, yl Color
# Place the groy pixel back in the center output_griddggryyy gray_y] = CoLor.GRAY
return output_grid
C.7 ARC PROBLEM AE58858E
from common import
import numpy np
from typing import
# concepts:
object detection, color change, size comparison
description:
In the input, you will see grid with red objects of various sizes.
# To make the output, change all objects larger than certoin size (3 pixels) to pink.
36

---

Published as a conference paper at ICLR 2025
def transform(input_grri)) # Create a copy of the input grid to produce the output output_grid copy (input_grid)
Find all connected components (red objects) in the input grid finput_grid, background-Color. BLACK, monochromatic-Trug,
red_objects
Iterate through the red objects and change their color if they are larger than 3 pixels
for obj in red_ :ed_objects:
size = spbsumoobj = Color. RED)
if size > 3:
output_griddobb == Color. RED] Color.
return output_grid
C.8 ARC PROBLEM F1
from common import
import numpy np from typing import
concepts:
# vertical and horizontal lines, intersection
description:
In the input, you will see two red pixels.
To make the output, draw blue square with the red pixels os corners. Additionally, drow a vertical and horizontal line that intersect at each red pixel.
def transform bransform(input.egg
# Copy the input grid to the output grid output_grid copy (input_grid)
Find the positions of the red pixels
red_pixels ap. ar.hrgweere (inputrr(inptgggri == Color. RED)
Ensure there are exactly two red pixels
assert len (ee(re_pii))ls (x1, y1), (x2, y2) red_pixels
Sort the red pixels' positions to determine which is top-left and bottom-right
top_left (min(x1, x2), min y2))
jestoom_right_o (max (x1, x2), max y2))
Draw blue square from the top-left to the bottom-right corner
popputt_rrdltoo_leetteeeo bbottomm 0] +1, top top_ [e] :bottom_right/
1+1] Color.
Draw horizontal and vertical lines through the red pixels for x, in [(x1, yl), (x2, y2)]: output_grid[X, :] Color RED Horizontal line through red
pixels
output_grid[; y] Color.REE RED # Vertical line through red pixels
return output_grid
37

---

Published as a conference paper at ICLR 2025
C.9 ARC PROBLEM 33B52DE3
from common import
import numpy as np from typing import
concepts:
object detection, color mapping, grid transformation
description:
In the input, you will 20x20 grid containing 4x4 pottern made of colored pixels (not necessarily contiguous) and a series of 3x3 gray squares.
The task is to extract the colors from the 4x4 pattern and color the corresponding 3x3 gray squares with those colors in a specific order. The first gray square should be colored with the first color from the pattern, the second gray square with the second color, and so on.
def transform(innut_griin
# Step 1: Detect the 4x4 color pottern in the input grid. color_pattern peteet_objectsstnppts (input_grid, monochromatic- connectivity=8)
Step 2: Extract the colors from the detected pattern. colors []
for obj in
cropped obj crop (gridzobo background-color..BAAk)) colors. eotosseeeteedcroopeed.oo (ttpipeeeeoj.llltennn)
Step 3: Identify the locations of the gray squares.
gray_squares detect_objects (input_grid, colors= [Color GRAYY)
monochromatic-Truee connectivity =8)
Step 4: Color the gray squares with the corresponding colors from
the pattern.
output_grat np. copy (input_gne) (input_grid) grid) for i, gray_square in (aaumerrategray_squurre if Ten (coccol))))
# Color the gray square with the corresponding color x, y np. www.ee square Color. Color.GRAY) GRAY) flood_fill fill(output_grid, x[0], y[0], colors(i)
return output_grid
C.10
ARC PROBLEM 332EFDB3
from
common
import
import numpy np from typing import
# concepts:
borders, alternating patterns
description:
# In the input, you will see an empty block grid. To make the output, draw a blue border around the grid, but only the edges that are
38

---

Published as a conference paper at ICLR 2025
multiples of 2 (i.e., the top and bottom edges, and the left and right edges at every even column/index).
def (rannsorm((ippu__rrid)
Get the dimensions of the input grid height, width input_grid.shape
# Create the output grid, initially a copy of the input grid output_grid = input_grid.copy()
Draw the blue border on edges that are multiples of 2 for x in range(0, height, 2): # Vertical edges: multiples of 2 x, 0, direction= 1), color=Color.Bue BLUE) # Top edge draw wraw_Line(output__rrid x, width 1, direction= (0, 1), color= Color.BLUE) # Bottom edge
for y in range width, 2) Horizontal edges: multiples of 2 (outpuutuutggrrd 0, y, direction= (1, 0), color=Color. BLUE))
# Left edge wrawlline height 1, y, direction=(( (1, 0), color= Color. # Right edge
return output_grid
D INDUCTION FALSE POSITIVE SOLUTIONS
D.1 ARC PROBLEM 25094A63
from common import
import numpy as np from typing import
concepts:
grid tramsformation, color replacement, pottern detection
description:
In the input, you will grid with colored regions. If 5x5 square in the grid consists entirely of single color,
replace that square with 5x5 yellow square in the output grid, If the square is not fully filled with single color, leave it as is
def (onnsformiiput_griid
# Plan:
1. Iterate the grid to find 5x5 squares.
2. For each 5x5 square, check if it is monochromatic (oll pixels
are the same color). # 3. IF it is, replace it with a 5x5 yellow square in the output grid
output_grid np. np. coppputtgit
# Dimensions
square square_size size
for x in range range(lenntnput_grion for y in range (wen(inuut_gridee # Extract the 5x5 region region input_grid(xxx + square_size, y:y + square_size]
square_size 1):
square_size 1):
# Check if the region is monochromotic
39

---

Published as a conference paper at ICLR 2025
if np. aregiinn region[O, 0]) # All pixels should be the same color
Replace the region with a yellow square output_grid[xxx + square_size, y:y + [imaar_size]] = Color
.YELLOW
return output_grid
D.2 ARC PROBLEM 009D5c81
from common import
import numpy as np from typing import
concepts:
pattern recognition, color transformation, grid manipulation
description:
In the input, you will 15x15 grid with pattern of purple pixels that forms a shope and a blue circle in the center of the grid, # To create the output, you should color the shape formed by the purple pixels with a specific color based the number of purple pixels: there are 1-10 purple pixels, color the shape yellow. If there are 11-20 purple pixels, color the shape green.
# If there are 21-30 purple pixels, color the shape red.
If there more than 30 purple pixels, color the shope orange. The blue circle should be replaced by black.
def transform(fnpptt.rrdd)
# Create an output grid initialized to block output_grid op.zeros_like (input_grid)
Count the number of purple pixels purple_count = np. www.oonounneerreeee (input_grid == PURPLE)
Determine the color based on the number of purple pixels if <= purple_count
10:
shape_color Color.YELLOW YELLOW
elif 11 <= purple_counts <= 20:
shape_color = Color. GREEN
elif purple_coume
30:
shape_color Color. RED
else:
shape_color ORANGE
Find the purple shape and color it accordingly output_grrid == Color. Rator.PUREE PURPLE] shape_color
Replace the blue circle with black outpput_grid(innut.grii
BLUE] BLACK
return output_grid
D.3 ARC PROBLEM B95E3D8E
from common import
import numpy as np from typing import
40

---

Published as a conference paper at ICLR 2025
concepts:
occlusion, symmetry detection, reconstruction
description:
In the input you will grid containing repeated pottern that has been partially occluded by black squares.
# To make the output, remove the occluding black squares and restore the pattern based the detected symmetry.
def Transform(inputerrr
# Plan:
1. Extract the occluded pattern from the input grid.
2. Detect the symmetries in the pottern.
# 3. Fill the missing parts of the pattern using the detected symmetries.
Step 1: Identify non-black pixels to find the pattern occlusion_colod Color. BLACK
non con_black_mast input_grid != occlusion color
cacuussiioooolrrroo backgrround-oceluusino.ooom
objects = cetect_objecce (input_grid, monochromatiic-False,
connectivity=8)
Step 2: Find the symmetries in the detected objects symmetries
(ingutrrddntttgr ignore_colors= [occlusiion_colr]]]
Step 3: Fill in the missing parts using the symmetries output_grid = syppu_grrd.copy () for x, in np. argaarer (non (non_blackkmaskk)
for symmetric_x, symmetric_y in orbit(input_grid,, x, y,
symmetries):
if output_grid [ymretrrccc symmetric_yl == occlusion_color
output_gria [symmetric_k, symmetric_yl = input_grid[x,
return output_grid
D.4 ARC PROBLEM 73cCF9c2
from common import import numpy np
from typing import
concepts:
# pottern extraction, rotation, centering
description:
In the input, you will see a grid with a pattern repeated several times but with random rotations.
To make the output, extract one instonce of the pattern and rotate it back to its original orientation.
def Transform(input_rrie nd.rraryyy) -> nda # Create a copy of the input grid for the output output_grid np. copy (input_grid)
Find connected components to extroct patterns components find_ find_coomected_comoneeee (output_gric background-Color. BLACK, connectivity-s)
41

---

Published as a conference paper at ICLR 2025
# For each component, check if it's volid pattern
for component in components:
# Crop the component to get the pottern
pattern ccompcomn,,,nt, background BLACK)
Check if the pattern can be rototed back to its originol orientation # We can rotate the pattern and compare with the original original_pattens = pattern.copy
Rotate the pattern in 90-degree increments and check if the
original pattern matches
rotated_patterns = [mp.roos@((orgginaaaataee
k) for k in range
4)
assume the first valid pottern the original orientation for rotated in potated_patterre if np. ap.array_equa (rotated, original_pattern):
# If it matches, we've 've found the original orientation output_grid pattern break
return output_gris
D.5 ARC PROBLEM D56F2372
from common import import numpy as np from typing import
concepts:
object detection, symmetry detection, cropping
description:
In the input you will grid filled with different colored shapes. The output should be the smallest symmetric shape (with respect to the center) that can fit all the others inside it.
If there are no symmetric shapes, the output should be grid of the same size filled with the background color.
def coonsfforminputgeee. nd.rrayyyy) -> np. nonnddara::
# Find all connected components (shopes) in the grid finput_rrid, background-color. components find__comeected_cooooon BLACK, connectivity-s, monochromatic=FFlse)
Initialize variable to track the largest symmetric shope found max_area
None
# Check each component for symmetry and area
for component in components:
# Crop the component to get its shape
cropped_shape crop (component, background:color.BB..
Check for horizontal and verticol mirror symmetry is_symmetric np. lerrayyeeuaaacppppeeeee shape,, np. flipp
cropped_shape, axis=0)) or \
np. apraaraa_equal (croppeddoopppppeeepe, np.flip(
@ropped_shape axis=1))
42

---

Published as a conference paper at ICLR 2025
if is rs_symmetric: area np. www.con_nonneooorrrooooooo coont__oozerr(rroppee_shpa Color Color.BLACK) if area > max_area:
max_area = area
= cropped_shape
If we found 0 symmetric shape, return it; otherwise return a grid of the some size filled with the background color
if largest_s srreesssyymmtric_hhheee is not None: return
else:
return np. fp.fuul((hnut_griddshaap Color.BLACK, BLACK)
E TEST-TIME TRAINING
Test time training an approach for updating model parameters test time, which we apply to transduction model. We assume that we given test problems D comprising triples (Imain: Ytrain (esss and data augmentation procedure AUG(æ, y) which constructs variations of an ARC problem, for example by permuting colors and rotating grids. Then model parameters optimized to maximize the likelihood of augmented test tasks where random training input-output selected to serve fake test example:
E
[t (3/tain [k]
Traiin [: k]. Terrin [k: 1 :], Mtrain k). Y/train [k: + 1 1:D]
(6)
-D
kow1. k.w1...eem (Timi))
(#i 3 )ms AUG (Ktrenn Minie)
where len( is the number of example training inputs, and is the index of the training input- output which is randomly selected to serve fake testcase. Note that this procedure does not rely on access to ground truth test predictions.
We choose each training example bined data augmentation for each fake task, which gives us 12k pseudo training dataset from the evaluation dataset. We also randomly include 5k RE-ARC examples and 5k ARC-Heavy examples, which we speculated would have regularizing effect. The dataset size for test time training 22k total problems.
fake test example and do 10 times different randomly com-
F EXPERIMENT PARAMETERS
INDUCTION
Fine-tuning Hyperparameters:
training type
lora rank
lora alpha
learning rate
gradient accumulate steps
lora finetune
64
64
2e-4
2
per device batch size
device
epcoh
weight decay
learning rate scheduler type
8
8xA100
3
cosine
For the last 230k data finetune we used full finetuning instead of LoRA:
training type
learning rate
weight decay
gradient accumulate steps
full finetune
lc-5
0.05
epoch
per device train batch
learning rate scheduler type
devices
16
cosine
3xA100
43

---

Published as a conference paper at ICLR 2025
Inference Hyperparameters:
temperature: 0.8 (1.0 for the full-data fine-tuned model)
top-p: 1.0
Output selection: For experiments in section when allowing one or two attempts, filter the sample programs using train input-output examples and then randomly select one two distinct programs uniformly. We report the expected value in our results. For experiments in section we take the exccution results of test output from the programs that pass all the train examples and use majority vote to select the top 2, and the case of concept arc, the top 3 test outputs.
F.2 TRANSDUCTION
Fine-tune Hyperparameters:
training type
learning rate
weight decay le-2
gradient accumulate steps
device 8xA100
full finetune
le-5
engineer epoch
other epoch
per device train batch
learning rate scheduler type
3
cosine
For the final engineering results, train for epochs. For all other experiment results, train for 2 epochs.
Inference Hyperparameters:
temperature: 0
use beam search: True
beam width:
engineer results: 40
100k data scale: 20
all other experiment results: 3
top-p: 1.0
Test-time Fine-tuning Hyperparameters
training type
lora rank
lora alpha
learning rate
gradient accumulate steps
lora finetune
64
2c-4
device batch size
device
epcoh
weight decay
learning rate scheduler type
2
4xA6000
cosine
44


Reviews:
[{"review_id": "7N3vaA8DQ6", "reviewer": "Reviewer_y4PU", "summary": "This paper studies the differences between inductive approaches and transductive approaches on ARC tasks.  They generate a synthetic dataset of ARC problems for training.  They demonstrate that induction and transduction solve different problems, and that neither is obviously superior to the other, and rather ensembling both approaches leads to the best performance.  They achieve impressive performance (40%) using LLama 3.1.", "questions": "* In the authors definition of induction, the distribution over the hypothesis $f$ depends on $x_{test}$, i.e. $f \\sim i_{\\theta}(x_{train}, y_{train}, x_{test})$.  In pure induction you would believe that the hypothesis only depends on the train set, i.e. $f \\sim i_{\\theta}(x_{train}, y_{train})$, and only at test time $x_{test}$ is used.  The authors definition seems to blur the line between induction and transduction, as their definition of induction forms the hypothesis based on the train examples and the test example, which clasically would be considered transduction, see e.g. the definition of Vapnik [1].\n\n* How are the grids that are inputs to the ARC problem encoded.  It would be helpful to explicitly reference how in Python code the visual examples are represented, as in my understanding LLama 3.1 is a pure text model (not a multimodal model)\n\n[1] Alex Gammerman, Volodya Vovk, and Vladimir Vapnik. Learning by transduction. preprint arXiv:1301.7375, 2013.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* The study of the different types of problems that induction and transduction solve is of great conceptual interest\n* The synthetic dataset and it's generation pipeline has value that can be of independent interest\n* They achieve compelling performance (40%) with LLama 3.1, matching or beating stronger models GPT-4o, Claude 3.5\n* The illustrations are informative", "weaknesses": "* A more principled understanding of the difference between induction and transduction, or a categorization of the problems they can solve  seems to be missing.  Adding some minimal insight into this could enhance the papers impact.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NrsGBzZXaz", "reviewer": "Reviewer_xinS", "summary": "This work validated the Transduction and Induction methods on ARC tasks and found them to be suitable for different tasks. By combining these two, a methodology is proposed to gain performance improvement in ARC.", "questions": "1. Are there guidelines for determining what tasks are appropriate for induction and what tasks are appropriate for transduction? \n2. How is the ensemble method proposed in Section 5 designed?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. The authors have developed a diverse set of experiments around transduction and induction with respect to the ARC task.  \n2. The authors improve the performance on ARC tasks with an ensemble that includes both transduction and induction.", "weaknesses": "1. The overall presentation could be improved. The presentation about the main contributions is confusing. The paper's title combines induction and transduction, but the abstract claims that inductive models and transductive models perform differently. The abstract does not summarize how the authors combine induction and transduction, nor does it provide a specific statement on the differences. \n2. The paper does not provide a theoretical analysis, and the experimental validation is somewhat narrow. Specifically, the authors only conducted verification on the ARC benchmark. Although the authors argue that ARC includes different types of tasks, they are all tasks in a grid-world environment, rather than real-world tasks. This casts doubt on the reliability of the proposed conclusions in the real world. \n3. A more specific discussion is needed for the conclusions obtained from the authors' experimental analyses. it's reasonable that induction and transduction methods are applicable to different problems. But what exactly are the types? Figure 6 provides some examples of ARC problems, which are insufficient to give the reader valid conclusions. What we are more interested in knowing is whether there are some guidelines for determining what tasks are more suitable for induction and what tasks are more suitable for transduction.  \n4. The author's statement of combining induction and transduction in the title is not sufficiently elaborated in the manuscript. In Section 5, the author simply states that an ensemble method is proposed, but exactly how the two are combined is important to the current title and should be elaborated. \n\nOverall, I think this work is more of a technical report around the ARC challenge and doesn't bring significant insights to the community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eFfM5HQbVo", "reviewer": "Reviewer_22bP", "summary": "The paper expands ARC dataset with LLM generated synthetic data, and investigate induction and transduction approaches on this dataset.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper contributes a expanded ARC dataset generated by LLM combined with manual efforts. This is valuable to the community.\n2. The paper performs a thorough evaluation of induction-based and transduction-based approach.\n3. The paper is written clearly and easy to follow.", "weaknesses": "1. As we know LLM often hallucinates. How do the authors quality check the generated dataset?\n2. While the authors concluded that induction and transduction approaches are complementary, I hoped to see more in-depth analysis of why induction works better for some while transduction works better for the others.\n3. While the authors also discussed the limitation that such method is only tested on ARC, I still think this is a weakness. I wonder the conclusion would be different for some other real-world datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LSz78gHl1u", "reviewer": "Reviewer_aaZL", "summary": "The paper presents an interesting approach to combining induction and transduction for solving reasoning tasks. Induction methods predict functions that explain the training data and use them to forecast test inputs, while transduction methods directly predict test outputs based on the training and test inputs. The paper proposes a way to effectively integrate both approaches by utilizing LLMs to generate synthetic datasets, which are then used to train neural networks for induction and transduction, respectively. Experiments conducted on the ARC benchmark demonstrate that induction and transduction complement each other effectively.", "questions": "Regarding the ensemble approach:\n> line135-136: Transduction and induction can be ensembled by predicting according to induction whenever at least one of the B functions fits ..., and otherwise using transduction ...\n\nInstead of switching sequentially between these two approaches, would it be possible to develop a learning method to blend induction and transduction? A meta-function could predict whether induction or transduction should be applied based on the input. This resulting function may provide insights into what differentiates induction-friendly tasks from transduction-friendly ones.\n\nPlease refer to the weakness section for other questions.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Combining induction and transduction to tackle challenging reasoning tasks such as ARC appears to be a novel approach. The successful use of LLMs to generate additional examples for training the proposed model is a noteworthy demonstration. While the idea may seem straightforward, its simplicity allows for broad applicability to other tasks.\n\nThe paper is well-written and easy to follow. The technical sections are clearly explained with well-defined formulas, ensuring high-quality presentation throughout.\n\nThe discovery that induction and transduction are complementary is particularly intriguing and is likely to spur further research in the field of abstract reasoning.", "weaknesses": "My primary concern is that the intuition behind the proposed ensemble method isn't clearly articulated. It's not immediately evident why the approach outlined in Equations (3)-(5) is expected to be effective. When ensemble, the method starts with induction and then applies transduction if the induction doesn't find a solution—does this suggest that induction takes precedence? For example, does the method presume that any problem should be addressed with induction whenever possible? Offering more insight into the reasoning behind these choices would be beneficial. Please refer to the first question in the subsequent section as well.\n\nWhile the finding regarding the complementary is interesting, it is not well-investigated why this is happening:\n> line 244; most problems solved by induction are not solved by transduction, and vice versa\n\nIt would be interesting to explore why this separation occurs. What distinguishes problems suited for induction from those better addressed by transduction? Any insights into these questions would be valuable.\n\nMinors:\n\nAs the authors noted in the limitations section, the method has only been demonstrated using the ARC benchmark. Evaluating it against other benchmarks would enhance the robustness of the paper. This raises several questions, such as whether the complementary nature of induction and transduction will apply to other reasoning benchmarks. However, I'd agree with the authors that the ARC is an unsolved, challenging task, and the experiments reasonably underpin the paper's claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)