Product of Experts with LLMs: Boosting Performance on ARC Is a Matter of Perspective
Authors: Daniel Franzen
Jan Disselhoff
David Hartmann
Venue: ICML 2025 poster (2025)
Keywords: Abstraction and Reasoning Corpus (ARC)
ARC-AGI
Large Language Models
Product-of-Experts
Search-based Methods
Test-Time Training
Data Augmentation
Abstract Reasoning
Puzzle Solving
Primary area: deep_learning->large_language_models
TL;DR: We achieve SOTA open source performance on the ARC-AGI challenge by leveraging LLMs combined with algorithmic supported sampling and product-of-experts scoring.

Abstract: The Abstraction and Reasoning Corpus (ARC-AGI) poses a significant challenge for large language models (LLMs), exposing limitations in their abstract reasoning abilities. In this work, we leverage task-specific data augmentations throughout the training, generation, and scoring phases, and employ a depth-first search algorithm to generate diverse, high-probability candidate solutions. Furthermore, we utilize the LLM not only as a generator but also as a scorer, using its output probabilities to select the most promising solutions. Our method achieves a score of 71.6% (286.5/400 solved tasks) on the public ARC-AGI evaluation set, demonstrating state-of-the-art performance among publicly available approaches. While concurrent closed-source work has reported higher scores, our method distinguishes itself through its transparency, reproducibility, and remarkably low inference cost, averaging only around 2ct per task on readily available hardware.

Product of Experts with LLMs:
Boosting Performance on ARC Is a Matter of Perspective
Daniel Franzen 12 Jan Disselhoff" 12 David Hartmann *32
Abstract
The Abstraction and Reasoning Corpus (ARC- AGI) poses significant challenge for large lan- guage models (LLMs), exposing limitations in their abstract reasoning abilities. In this work. we leverage task-specific data augmentations throughout the training, generation, and scoring phases, and employ a depth-first search algo- rithm to generate diverse, high-probability can- didate solutions. Furthermore, we utilize the LLM only generator but also as scorer, using its output probabilities select the most promising solutions. Our method achieves score of 71.6% (286.5/400 solved tasks) on the public ARC-AGI evaluation set, demonstrating state-of-the-art performance among publicly avail- able approaches. While concurrent closed-source work has reported higher scores, our method dis- tinguishes itself through its transpareney, repro- ducibility, and remarkably low inference cost, av- eraging only around 2ct per task on readily avail- able hardware.'
LLM generates Solution Candidates
Use uncertainty in predictions to select submission
Figure 1. Example of a typical ARC-AGI task.
illustrated in Figure 1) may appear straightforward to hu- mans, both traditional algorithmic approaches (Wind, 2020) and contemporary neural architectures (Li et al., 2024) have struggled to achieve significant success ARC-AGI, high- lighting potential limitations in current artificial reasoning methods.
1. Introduction
Large Language Models (LLMs) have demonstrated traordinary capabilities across diverse tasks, from natural language processing to code generation. Even so, evaluating the extent to which these systems possess abstract reasoning abilities continues to pose a major challenge in the artificial intelligence community. The Abstraction and Reasoning Corpus (ARC-AGI), introduced by Chollet (2019) and de- signed assess core knowledge and the ability to generalize in AI, exemplifies this difficulty. Although these tasks (as
Although scaling up models has undoubtedly yielded sub- stantial performance gains on many tasks, size alone does not fully address the core limitations evident in challenges like ARC-AGI. Indeed, the rapid evolution of apen-source systems such I.L.MA332--3 (Dubey et al., 2024) and Nvidia NeMoo-Minttro- (Sreenivas et al., 2024) demon- strates that significant capabilities can emerge even at more modest scales. This aligns with mounting evidence that many perceived shortcomings stem from implementation details or suboptimal data repre- sentations rather than from fundamental reasoning deficits (Singh & Strouse, 2024; Bostrom & Durrett, 2020; Sun et al., 2023). For instance, Allen-Zhh -hh & Li (2025) observe that models may be aware of their mistakes without being able to correct them, while Allen-Zhu & Li (2024) highlight how subtle data modeling choices can impede fine-tuning progress. Collectively, these insights suggest that models of- ten possess the latent capacities needed tackle ARC-AGI;
large language models
Eqqual contribution 'Johannes Gutenberg University Mainz Memberr of "the ARChitects" Kaggle team. 'Lambda, Inc.. Correspondence to: Daniel Franzen <dfrannen. gmmil.com>> Jan Disselhoff <JanDisscl.tt@maal @em..iimmmmmim...... David Hartmann <davidh @ambbda.i>>
Proceedings of the n2nd International Conference on Machine I Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
We assume a price of 36ct/hour Nvidia 4090 GPU

---

Product of Experts with LLMs: Boosting Performance on ARC Is Matter of Perspective
Table Performance comparison of related work. We distinguish between solutions where the underlying model weights are open- source or proprietary.
Visual Representation of a Task Instance:
Input
Output
Model
Public Eval Accuracy [%]
Open
Name
Source
Compact String Format of same Instance:
ol-preview (Kamradt, 2024)
X
Ryan Greenblatt (Greenblatt, 2024)
42
X
<bos>
A
Z
LF
Jeremy Berman (Berman, 2024)
58.5
X
2
LF
GPT o3 (arcprize.org, 2025)
82.8
X
0
Avg. Human (LeGris et al., 2024)
60.2
?
TTT (Akyürek et al., 2024)
53.5
/
BARC (Li et al., 2025)
56.75
V
LE
TTT+BARC (Akyürek al., 2024)
62.8
V
<eos>
Ours
71.6
Figure 2. Our standard tokenization approach. Note that we use one token per cell instead of compressing the problem more. We also try to not include any unnecessary delimiters. The Pre-pre Pre-prompt (the alphabet in upper then lower case, i.e. "A...Za..zz) is only in- cluded for the first example. Depending on the model and run there might be some small changes to the pre-prompt and input/output prefix tokens.
the real challenge is creating the conditions under which these capacities be reliably expressed.
Building on these insights, we developed an approach specif- ically tailored to the ARC dataset. Our method achieves SOTA performance for open source models of 71.6% (or points) on the public ARC-AGI evaluation set and surpasses average human performance of 60.2%, as measured by LeGris al. (2024).
The Original ARC Dataset:
The Abstraction and Rea- soning Corpus (ARC-AGI) introduced by Chollet (2019) challenges the idea that language models can efficiently generalize from small number of examples, often referred to as few-shot prompting. The original ARC-AGI dataset consists of 900 reasoning tasks, divided into 400 training tasks, 400 public evaluation tasks, and 100 private and thus unpublished evaluation tasks. Each task involves input and output grids of varying sizes, ranging from Ix1 to 30x30 and utilize a palette of ten distinct colors.
In particular, we employ depth-first search (DFS) al- gorithm on LLM predictions to generate diverse, high- probability solutions, and re-use the same LLM also as a product of experts (see Section 4.1) to select the best can- didate. This dual role allows us to rank candidate solutions via augmented likelihood estimates, effectively amplifying the model's latent reasoning abilities. Compared to more heavily scaled closed-source systems, our method stands out for its transparency, reproducibility, and low inference cost of around 0.02$ per task, in stark comparison to 17$ per task for o3 (arcprize.on) org, 2025). This demonstrates that abstract reasoning on ARC-AGI is not exclusively the do- main of massive proprietary models.
The objective of each individual ARC task is discern the transformation rule from input to output from the examples and apply it to new input grids to generate the correct out- puts. A task is considered successfully solved when the model produces the accurate output within a maximum of two attempts. Designed to be straightforward for humans yet challenging for machine learning systems, the tasks high- light the current limitations of AI in abstract reasoning. In study by LeGris et al. (2024), the average human was able to correctly solve 60.2% of the evaluation tasks, while 97.8% of the tasks were solved by at least one participant using two guesses.
In the sections that follow, we detail data modeling and training strategies, describe DFS-based solution exploration, and provide comprehensive results and ablation studies.
Our final model, along with the training and inference code, publicly available on GitHub.
Competition-driver Progresses: Since ARC's introduc- tion in 2019, several competitions with hundreds of partici- pants have sought develop solutions with strong perfor- mance on the dataset. Approaches up 2024 frequently employed program search over domain-specific languages (DSLs), and have yielded a score of 39% using Top-3 scor- ing (Wind, 2020).
2. Related Work
The Abstraction and Reasoning Corpus (ARC) has played a central role in advancing research on abstract reasoning in artificial intelligence, inspiring a wide range of studies focused on its dataset, competitive benchmarks, and the development of solutions driven by resource constraints.
In 2024, ARC-AGI hosted another Kaggle competition, where for the first time large language model (LLM) ap-
2

---

Product of Experts with LLMs: Boosting Performance on ARC a Matter of Perspective
proaches dominated the leaderboard. One popular method was test-fime training (TTT). This approach was first in- troduced in Sun et al. (2020), first suggested for ARC by Cole (2024) and later popularized by Akyürek et al. (2024). Test-time training leverages the few examples provided in each challenge as a small dataset. By fine-tuning on these examples before generating an answer, LLMs can achieve a substantial increase in performance. In Akyürek et al. (2024), the authors demonstrate that TTT more than doubles their performance on ARC-AGI. TTT is particularly effec- tive in competition settings like ARC, as it allows models to extract additional training data from the limited examples available, enhancing their ability to generalize and solve new tasks.
In contrast, TTT+BARC is fully open-source and notably the first public approach surpass the average human per- formance on ARC, showcasing the benefits of transparent methodology in advancing abstract reasoning research.
3. Notations and Setup
To ground our approach formally, we adopt a Bayesian per- spective on puzzle-solving, treating each puzzle as a partial observation from an underlying distribution of solutions.
We consider a collection of tasks (for example drawn from the ARC benchmark), where each task is denoted by p E P, and P represents the space of all possible tasks. For each task p. there exists an associated solution space Sp-
Problem Representation. Throughout this paper, we use the terms task, puzzle, and problem interchangeably. all referring specification given by a small set of l input- output examples and single test input. Concretely, we write
Notable Mentions: Other approaches explored various strategies for utilizing LLMs. In Li et al. (2025), the au- thors classify two different avenues: Induction, where a LLM infers a function that can solve the problem which is then applied (often using python or DSL), and Transduc- tion, where the LLM directly generates the solution using a tokenized description of the problem (see Figure 2). The authors argue that these approaches solve different kinds of problems, despite using the same underlying architecture. In their experiments, induction and transduction solve roughly the same amount of problems (38% and 43% respectively), which can be increased to 75% by employing ensembles. Additionally, they use the induction network to generate large set of novel challenges, dubbed ARC-Heavy. Some approaches make use of alternative ARC datasets, such as ConceptARC (Moskvichev et al., 2023). The most notable- and only additional dataset we use- the well-known RE- ARC dataset. Hodel (2024) introduced this dataset, which implements generators for all 400 tasks of the public train- ing dataset. Their code can be used to produce an arbitrary amount of training data for these tasks, but does not intro- duce novel challenges. All other datasets might include challenges that mimic the evaluation challenges- thereby re- ducing the difficulty of those challenges immensely. By only using the RE-ARC dataset, we still increase our training data immensely, but stay close solving the ARC challenge as intended.
p = ((1,, (.). a),,
where (z,, Ma) indicates the ith input-output example pair and è is the test input for which we seek the correct output. Although not explicitly observed, each problem p admits at least one correct solution 8, E Sp.
We assume the existence of a true probability distribution
P(s ({s1p) p)
over candidate solutions s E Sp- If exactly one possi- ble valid answer exists the distribution P(- p) would be sharply peaked at sg. While this is the case for most chal- lenges, we will keep our theory more general, assuming multiple valid answers might exist. This can also arise from insufficient information in the given example pairs, which in the worst case prevents us from uniquely inferring the correct solution based solely on the provided data. Exam- ples for this are sometimes found in ARC-AGI, which fre- quently results in an update of the dataset (Neoneye, 2024; RubenKelevra, 2024).
Hence, P( p) may be spread out over several plausible hypotheses. Identifying s from Sp typically requires lever- aging priors or additional constraints (e.g.. knowledge of how ARC tasks are designed). Formally, one may write a posterior
Data augmentation has been a common approach in previ- ous ARC-AGI competitions (Akyürek et al., 2024; Li et al., 2025). However, our method extends beyond traditional dataset augmentation, applying transformations throughout our approach, during training (initial finetuning as well as test time training), inference and selection.
P(s (p) P(p s) P(s) P(p)
where P(s) encodes how we believe solutions are structured a priori, and P(p s) measures how well s explains the limited observed examples. The goal is to select
Table compares recent ARC approaches, revealing that OpenAl-o3, a closed-soure, method, currently reports the highest score but lacks reproducible details. Further, o3 uses an immense amount of computation for each task, using 17 17$ of compute for a single challenge (arcprize.org, 2025).
S/ argmaax ((s p), eES,
3

---

Product of Experts with LLMs: Boosting Performance ARC Is Matter of Perspective
Search Performance Comparison
Accuracy for Different Selection Methods
Accuracy (Proportion Solved)
0.8
0.8
Solutions Found
0.7
0.6
Generation Accuracy
0.6
DFS T-20% (ours)
0.4
Product-of- II, P3 (ours)
DFS 9=9% (ours)
Mean Probability x Σ, P
DFS T-0.% (ours)
Max Generation Probubility
0.5
Beam search (2x)
0.2
Beam search (4x)
Selection Accuracy
Greedy sampling
Overall Accuracy
0.4
10
102
0
Runtime per Task (Seconds)
Confidence Threshold (-log(prob))
Figure 3. Number solutions found by various sampling algo- rithms function of runtime. The different values for cach sampling variant are calculated using (identity), 2 (reflections), 4 (rotation), & (reflections-rroationn and 16 augmentations. Addi- tionally, colors and the order or examples are randomly permuted in each augmented version of task. For almost any runtime budget, we find that a DFS variant discovers the most solutions.
Figure 4. Top-2 accuracy and coverage of different selection meth- ods function of the confidence threshold T'. Solid colored lines denote the fraction of tasks solved using a specific selection algo- rithm. The solid black line shows the fraction of tasks where the correct solution was among the sampled candidates, and thereby provides an upper bound rithms. The dotted lines evaluate the performance of our selection algorithms. compared to this upper bound: What percentage correct candidates are actually selected when they are present? It shows that even when using low DFS probabilities and therefore sampling high number of candidates PoF is able select the correct solution among all candidates with high specificity.
performance of the selection algo-
but in practice we do not have direct access P. Instead, train model to approximate yielding P stand-in for the true distribution.
Finally we define family of problem transformations ("aug- mentations"),
(1) Augmentations. While naive multinomial sampling from P can already produce favorable candidate solutions, we enhance the model's robustness further by leveraging augmented training data.
= {6)1,...... bmh.
where each dy transforms both problem p and its solutions 8 such that
These augmentations diversify the training distribution without altering correct solutions, effectively shaping the model's learned prior.
p)) = (+) (y)) for all (p, s).
The trained LLM then provides us with a probability dis- tribution P(( p) over solutions Sp- Using multinomial sampling, this allows us to sample s ~ P(. p) However, sampling repeatedly from P may be expensive and does not ensure coverage of high high-probability solutions. In con- trast, enumerating Sp in full would provide us with full knowledge of P(s) but is impractical. Instead, we rely on more systematic procedure to select promising candidates by deriving candidate set of high-probbbilityy solutions via threshold-basee search. Subsequently, we refine their probabilities by aggregating over multiple problem augmen- tations.
For the ARC puzzles, such augmentations include rotations and reflections of each task, shuffling of the example order and permutation of colors. The augmentations in & define parts of the prior P(s) by encoding invariances that are expected to hold for all valid solutions.
4. Methods
Our approach trains large language model (LLM) to ap- proximate the true solution distribution P(- p). Given task p and some solution candidate s, tokenize both and use the trained LLM to calculate probabilities for each token. By aggregating the probabilities of the solution tokens, we can define a probability function P(s p) that describes the probability of sampling given as a prefix, setting the stage for subsequent sampling and tearch-based refinement.
(2) Candidate Generation. To address the mentioned challenges of multinomial sampling, we propose a threshold- based search mechanism. Instead of mere random sampling,

---

Product of Experts with LLMs: Boosting Performance ARC Is Matter of Perspective
since subsequent predictions build the incorrect founda- tion. The model lacks training for stability under prediction errors, causing cascading mistakes with unexpectedly high confidence scores. This explains why the highest probability sequence from a single forward pass may not correspond to the globally optimal solution.
Correct Solution Rank Improvement
Gen. Rank
200
100
We can mitigate this issue, and even benefit from it, by re-augmenting each candidate under every Φj E and computing its likelihood using P provided by the LLM. Unlike in the previous step, this phase does not rely on generative sampling; instead, it directly evaluates the log- likelihood of s's tokens for each augmented input Øy (p). Using these re-augmented candidates, we form single aggregate score by taking the product of probabilities across all augmentations:
Worse Rank
Equal Rank
Better Rank
Sel. Rank
20
10
0
50
100
150
200
250
300
350
Problem Index (sorted by Inference Rank)
Figure 5. Comparing the rank of the correct solution using the gen- erative model P and the ensemble selection P among candidates C. If possible, our ensemble almost always improves the rank of the correct solution, increasing the chance of selecting it. For readability we clip the lower plot a rank 25.
SCOTe agg (s)
II (())(s) @,)()). E.
This product-based approach is sensitive to outliers, filtering solutions that seem unlikely from different augmentation perspective. As result, this approach, on average, outper- forms randomly selected augmentation, as we prove in the following section. Finally, we select the solution
we systematically explore the space of solutions via depth- first search (DFS) algorithm.
Given test problem p. we derive a set of candidate so- lutions by sampling under all valid augmentations $5 (p)) Concretely, we define
argmax score
P
sEC,.r
Cp.T = {s S E Sp 3o, E T P())(s) (y(p)) > T..
as the final answer for problem p.
This two-step approach; (1) single-augmentatinn pruning and (2) post-hoc multi- augmentation scoring, ensures that we systematically ex- plore high-probability solutions and then refine their rank- ings, accounting for LLM inconsistencies across problem representations. In practice, even if several solutions enter Cp.r. their final ranks can vary greatly. By consolidating evidence from multiple perspectives, the correct solution often stands out and becomes casier to pinpoint.
generation with
where > is a threshold on the LLM's probability es- timates. In practice, we run a Depth-First Search over the space of potential solutions, pruning any partial path whose accumulated probability falls below T. If multiple augmen- tations yield the same solution (up to augmentation), we merge them into a single candidate. By caching intermediate computations during inference, this DFS-based approach can rapidly pinpoint all likely solutions above the thresh- old T. This guarantees that solutions with sufficiently high P(s p) are not overlooked and solutions with low P(s p) are never considered.
4.1. Product of Expert Augmentations
We next analyze the performance of our ensemble method in terms of the KL divergence of the ensemble distribution P compared to the true distribution P. Let each valid aug- mentation Φ, induce approximations of the true augmented distributions (s),(s) (p())) by the LLM, denoted as
(3) Candidate Ranking via Product of Experts How- ever, once we have generated the set Cp.r, the highest- probability solution according to a single augmentation is not always correct.
This limitation partially caused by the autoregressive architecture, as models can only attend to previously gener- ated tokens when predicting the next token. This constraint means optimal decisions sometimes require information that becomes available only in later predictions. In the Sudoku experiment (Section 5.6). for instance, the model may need to solve the entire puzzle internally before predicting the first cell, potentially leading to confident but incorrect early predictions. Once an error occurs, the model cannot recover
Ps(() P()s(s) dy(p))
Since the LLM may be inconsistent across augmentations (in contrast to the true distribution P), our approach de- scribed in the last section combines them by the geometric- mean ensemble:
1 m
P(s)
[[[ j=1
5

---

Product of Experts with LLMs: Boosting Performance ARC Is Matter of Perspective
where Z is the normalization constant. A value of Z represents the case that the LLM is consistent across all augmentations, P. P3 for all (i,j). Intuitively, P places low probability on those s for which even few (s) are low probability.
can prune out plausible-but-incoreett candidates, correct ones accumulate strength across viewpoints. This synergy typically yields more reliable predictions than relying single representation of the problem alone.
5. Experiments
We aim to show that if each Pj is close to the Irue dis- tribution P in terms of KL divergence, then P provides in expectation- a better estimate of than any randomly chosen Py. We formalize this idea in the following well- established theorem known from literature (Hinton, 1999; 2002).
Our approach to solving ARC-AGI combines data expan- sion, multi-stage fine-tuning of language models, and spe- cialized solution evaluation. Below, we explain how these components work together to improve the model's perfor- mance while keeping computational costs manageable.
Theorem 4.1 (Error Bound for Log-Pooles Augmentations), Suppose we have m valid augmentations {di,..., 2m in the sense of preserving solution distribution, and define
5.1. Data Modeling
In order to apply LLMs to ARC-AGI puzzles, we need to tokenize the data in a manner suitable for our model. This process requires careful consideration of two main challenges:
(s))) P(0)(s) dy(p)), for each j -I,...,m m.
Assume euch single-aune muggentaummnnttiinn predictor P3 has bounded KL divergence from P. i.e.,
First, due the limited context size in typical LLM archi- tectures, increase of inference time and decline in perfor- mance long context tasks (Liu et al., 2024), require representation that minimizes the number of tokens the model needs to process. Secondly, it is widely recognized that numerous common failure modes in Large Language Models (LLMMs) stem from tokenization (Singh & Strouse, 2024; Bostrom Durrett, 2020; Sun et al., 2023). For instance, standard tokenization techniques group numbers (some but not all combinations) of one, two three suc- ceeding digits into dedicated "grouped-diglt tokens" (Singh & Strouse, 2024). These kinds of merges would complicate the puzzles unnecessarily.
Dj == = KL((1P))))) S 83.
Now define the "geometric-mean" ensemble
P(s) == z
1
[[:=
j=1
where
== Σ uES,
II (+)} # 1=1
Then the KL divergence between P and P is given by the average of the linlleaagmmentation divergences and Z:
To address this, we opted to simplify the token set available to the model. particular, we reduced the number of tokens available from over 120.000 64 tokens (see Table 5 in the Appendix).
kL(P||P)
Σ (1() + log Z
This reduction offers key benefits. It significantly decreases the model size, as we can remove the majority of rows from the embedding layer. Further, token merges that typically occur during text tokenization are no longer possible. This ensures that the model can focus precisely on the data with- out the interference of digit separators.
With log Z < 0, and equality iff P. = =P for all i.
See Appendix C for proof of Theorem 4.1. The key take- away is that log Z < O becomes smaller whenever aug- mentations disagree, which can improve the ensemble in expectation relative to any random single-augmentation pre- dictor. As a result, this approach performs especially well when different experts disagree- state which naturally arises in our case, due to the causal autoregressive nature of the L.MMs.
As illustrated in Figure 2, add a small number of ex- tra tokens to the start of task. Surprisingly, this addition slightly improves the model's performance. We believe that during fine-tuning (where the embedding layers are also trained), the model learns to use these extra tokens as a form of computational buffer, which influences every subsequent token, thereby enhaneing overall performance.
Practical Implications In practice, a product of experts approach often shines when different augmentations catch different errors. As long as the true solution does not get zero probability under any single augmentation, it remains viable. Hence, while disagreements between augmentations
5.2. Training the models
Choosing a suitable large language model (LLM) was es- sential for achieving strong performance. After evaluating
6

---

Product of Experts with LLMs: Boosting Performance on ARC Is Matter of Perspective
Table 2. www-guess-accuuracyy the ARC-AGI public evaluation when adding parts of our method. Baseline score shows performance of our network after initial fine-tuning, generating two samples with stochastic sampling. TTT adds tist-tmme training. 16xAug samples one solution candidate for each of 16 random augmentations of each task, choosing the two with highest sampling probability as guesses. PoE uses the product of experts to select the two best of the 16 sampled candidates, again using 16 (different) random augmentations to calculate the PoE score. Finally, DFS leverages our custom seet--hisstteearch sampling scheme with 9% for candidate generation.
Model
Baseline
TTT
16xAug
PoE
+ DFS
Llama-3.2-30
14.9%
40.9%
52.9%
59.5%
61.4%
NeMo-MMnitron-SB
18.3%
44.5%
62.5 %
67.6%
71.6%
various models, we identified Mistral-NeMo--Minttroon Base (Sreenivas et al., 2024) as exhibiting the strongest performance in our experiments. Given the model's 's size, efficient fine-tuning methods were necessary for effective utilization.
The initial fine-tuning took 98 GPU hours on Nvidia H100, while test-time training takes (on average) 51 seconds for single task on Nvidia RTX 4090 GPU. For an overview of our training parameters, see Table 4 in the Appendix.
Therefore, we used Low-Rank Adaptation (LoRA) (Hu et al., 2022). 4-bit quantization and gradient checkpointing, all supported by the unsloth library. We applied the LoRA adaptations to all layers of the network, including the input and output embeddings.
5.3. Solution Inference
As introduced in Section 4, we generate potential solution candidates using DFS-based sampling to produce the set Cp.r. The goal here is to generate a small set of candi- dates with a high chance of containing the correct solution and doing so quickly. Our set of augmentations & in- cludes 16 functions per task each Dş symmetry is used twice but with different, randomly chosen color permuta- tions and example re-orderings. Note that this is the same class of augmentations used in training, but each color permutation and example ordering is newly randomized. Table 3 provides a comparison between DFS, Beam-search, multinomial and greedy sampling. DFS sampling is able to quickly and efficiently find a high quality set of candi- dates, while having low computational overhead compared to stochastic sampling for generating multiple solutions and using substantially less VRAM than beam search. In addi- tion, it exhibits lower false positive rate. While DFS with T = 9% finds less correct solutions than 4x Stochastic sam- pling (76.0% vs 77.3%), it still results in a better selection score, as it, on average, only returns about half as many false positives. Moreover, DFS accomplishes this using only a fourth of the inference time (9:32h vs 39:47h).
For each task
p = ((12, $):
with solution sy. we computed gradients only on the out- puts y for é >> 1 and sy. This approach ensures that the model is never tasked with predicting an input grid, and acknowledges that correctly predicting the first output grid is impossible without at least one example. To increase the amount of training data, and to better align the LLM with the data prior, we train on augmented data, adding all D3 symmetries of any given task as well as color permutations and re-ordering of the examples.
Initial fine-tuning: The initial fine-tuning used a LoRA rank of 256 and was done on a single H100 GPU. While several ARC-like datasets exist, such as ConceptARC (Moskvichev et al., 2023) and ARC-Heavy (Li et al., 2025), we elect to only use RE-ARC (Hodel, 2024) for training. This is done to minimize " "conceptual leakage", where a par- ticular type of problem might be present in the training data, reducing the difficulty of the evaluation tasks substantially in a way that was not intended. Instead, we train only on replications of the training examples of the offical ARC- AGI training set (i.e. RE-ARC), minimizing this effect and making sure that our results are robust.
Comparison to beam search: While beam search with 4 beams can achieve the same accuracy as DFS with T = 9%, it requires roughly twice the amount of VRAM (7.3GB vs 14GB), as it explores four paths simultaneously, while DES only needs to keep a single path in memory at any time. also takes four times as long (37:36h vs 9:32h) for the can- didate generation step. The speed advantage of DFS comes mostly from early pruning of low probabilty paths. In Beam Search, the same amount of paths is explored each time, regardless of their cumulative sampling probability, while DFS stops when the cumulative sampling probability falls beyond the chosen threshold, thereby reducing unnecessary computations. Additionally, for all augmentations after the first one, we pass the most promising solution candidate
Test-time training: Secondary training was time- constrained and focused solely the evaluation set, us- ing LoRA rank of 32 and running for 64 training steps with batch size 1. Just using test-time training increases the percentage of correctly solved tasks significantly, as can be seen in Table 2. Varying training parameters only had marginal effects.
7

---

Product of Experts with LLMs: Boosting Performance ARC Is Matter of Perspective
Table Comparison of sampling and selection strategies on the 400 tasks of the ARC-AGI public evaluation set: Under "Candidate generation", we list the percentage of correct solutions sampled with different strategies using 16 augmented versions (reflections, rotations, and randomly permuted colors and examples) each task. We also list the average number of candidates generated per task, the runtime of the sampling process on the full dataset and the maximum video memory consumption. Under "Selection", we compare the accuracy of various selections strategies, performed on the scores calculated in subsequent scoring process on 16 additional random augmentations. Total runtime includes the test-time fine-tuning on task's 's examples (see Table 4 in the appendix), as well as the candidate generation and selection process. All experiments were performed with the NeMo-Minitron-88 model on a Nvidia RTX 4090 GPU.
Sampling
Candidate generation
Selection (2-guess accuracy)
Total runtime [hh:m]]
method
solutions found
avg. cand. per task
runtime [hh:mm]
max.
using 16 augmentations
VRAM
max P,
min P,
Σ P
I P
Greedy
70.8%
6.7
9:39
GB
63.3%
65.8%
66. 16.1%
67.6%
18:52
Stochastic (2x)
74.5%
11.2
19:53
7.0GB
64.5%
67.6%
66.9%
69.9%
34:08
Stochastic (4x)
77.3%
17.6
39:47
7.0
63.5%
68.8%
67.1%
70.8%
58:55
Beam search (2x)
75.0%
15.9
29:33
9B6
63.1%
65.9%
65.0%
69.9%
47:27
Beam search (4x)
79.0%
34.7
37:36
GB
61.9%
67.9%
65.0%
71.6%
71:39
DFS T=20% (ours)
73.5%
4.9
5:58
GB
63.5%
68. 68%
66.4%
70.0%
14:12
DFS T=9% (ours)
76.0%
9.3
9:32
GB
63.5%
68.8%
66.6%
71.6%
20:50
DFS T=0.5% (ours)
83.5%
84.7
80:56
7.3 GB
63.3%
69.1% 1%
66.9%
71.8%
134:43
found so far as an inital guess to the DFS and process it in a single forward pass before starting backtracking, which is much faster than token-by-token generation. Note that these comparisons should be interpreted with caution, the beam search algorithm is not implemented in the unsloth library used for the other experiments, which might provide some time savings. However, even when accounting for those savings, beam search still requires far more time overall, as it returns significant amount of false positives, which increase the runtime required in the subsequent selection process, where each candidate is evaluated under different augmentations.
5.4. Selection Strategies
Up to this point, our method generates candidates likely to include the correct solution. However, solving the task requires identifying it among the candidates, using at most two guesses.
As introduced in Section 4.1, we again use a set of aug- mentations & to calculate the results of a product of expert ensemble P. A candidate s E Cp.r is selected for one of the two guesses if it has the (second-)highest probability according to P. Figure 5, we compare the rank of the correct solution before and after using this augmentation procedure. In most cases where the correct solution does not start at rank 1, this augmentation leads to a better rank for the correct solution, increasing our chance to solve a given task. In Theorem 1.1, we proved that our product of experts approach is superior to selecting one augmentation at random, which can clearly be seen in Table Here, we compare different sampling methods and different aggrega- tion methods. In all cases, using the product of probabilities leads to an increase in score, with min P. taking second place for most sampling methods and max P. performing the worst. For our T = 9% DFS inference, PoE increases the final score by 5% compared to averaging the probabilites (66.6% vs 71.6%).
As we do not know the sampling probability of the correct solution beforehand, we have to treat the probability bound as a pyrreeearameter. We found that values between T 5% to T 20% provided reasonable compromise between inference time and number of correct solutions, but the exact parameter depends the model and training procedure used. Similarly, due to the way probability mass is distributed on the solution tree, DFS is faster when the model has a higher degree of certainty in its predictions.
We compare the number of candidate sets that contain the correct solution for different values of T in Figure 4. This function is monotonically increasing in T, but are infer- ence costs and the size of the set Cp.r, making the selection of the correct candidate harder. Our final results are calcu- lated using T = 9%, as it uses roughly the same amount of inference time as greedy sampling.
5.5. ConceptARC
To make sure we do not overfit on the original ARC data, we further evaluate our method on ConceptARC (Moskvichev et al., 2023) an ARC-like dataset containing tasks sorted into specific conceptual categories. Our method achieves
We provide pseudo-code for our DFS sampling algorithm in Algorithm 1 in the Appendix.
8

---

Product of Experts with LLMs: Boosting Performance on ARC Is a Matter of Perspective
6. Discussion
Accuracy for Different Selection Methods for the 3M Sudoku Dataset (1000 samples)
Our method builds on familiar techniques - data augmenta- tion. Bayesian modeling, and product of experts scoring- but tailors them specifically for ARC-like puzzles.
Accuracy (Proportion Solved)
At our methods core, we use a single fine-tuned LLM in two roles: as generator, it proposes solutions for each puzzle augmentation: as scorer, it re-scores each generated candidate across all augmentations by taking the product (geometric mean) of likelihoods. The benefit is twofold. First, a candidate solution must be jointly plausible under every valid transformation to rank preferably, making harder for the model to latch onto spurious correlations found in just one representation. Second, this log-linear pooling approach naturally acts as an ensemble method, as we show in Section 4.1.
0.8
Generation Accuracy
0.6
Product-of- Saducttof-Expeerts/ II, P3 (ours)
Mean Probability, & Σ, Pj
0.4
Max Generation Probability
Selection Accuracy
0.2
Overall Accuracy
0
3
4
Despite ARC's reputation for complexity, our two-phase "generate- theetee--ncorree routine achieves SOTA results among open models. While only single closed-source solution (arcprize.org 2025) posts higher absolute score at $17 per task, our fully open-source process stands out for its transparency, reproducibility and, above all, its cost- effectiveness of only 0.028 per task.
Confidence Threshold (-log(prob))
Figure 6. Results the Sudoku experiments (plot equivalent to Figure 4, but showing top-1 accuracy instead of top-2). We can see that our product experts approach increases the accuracy substantially to 53% solved Sudoku puzzles over simply selecting the generated solution with the highest sampling probability. Note that generation accuracy completely coincides with product of experts probability, showing that a correct solution is sampled, approach consistently selects it.
By applying these ideas to ARC, we underline a broader principle: when dealing with structured or abstract rea- soning tasks, the key factor is to exploit valid semantic- preserving transformations, forcing a model to remain con- sistent across multiple views of the same problem. This allows us to use a single model as an ensemble of experts. We believe this perspective can generalize to more complex symbolic reasoning challenges, wherever such transforma- tions can be defined. Our results demonstrate that large lan- guage models, properly steered in inference and supported by prior aware scoring, can go beyond default sampling approaches to capture deeper structures in abstract domains.
73.3% 2- 2-guess accuracy on ConceptARC (using the exact same hyperparameters as DFS T-9%), showing that we gen- eralize well to other ARC-like datasets of similar difficulty.
5.6. Sudoku
We further test our approach on the Sudoku 3M dataset (Radcliffe, 2020) to evaluate generalizability of the method to different domains. Since the underlying "rules" of Su- doku remain consistent between tasks, we do not use any test-time training in this case. Instead, we start out with our Llama 3B model pre-trained on ARC, which we then finetune again on 128000 Sudoku tasks. As the Sudoku tasks never have any ambiguity, we report top-1 accuracy rather than 2op-2. To handle the increased complexity for the LLM compared to ARC, we use DFS with threshold of T=1%, which provides a good trade-off between accuracy and runtime (see Figure 6). This setup reaches 53% accu- racy on 1000 randomly chosen unseen Sudoku puzzles, far better than state-of-the the-the-art LLMs, which have a solve-rate less than 3% on comparable benchmarks (Seely et al., 2025). Notably, if the correct solution of puzzle is sampled, we select it in 100% of cases. This is caused by the fact that Su- doku correctness is simple to evaluate. Using our standard augmentations described in Section 5.3 on the predictions, the model can identify errors more frequently, thereby sig- nificantly reducing the likelihood of false positives.
6.1. Future Work
Building upon our insights, several promising directions emerge for future investigation. First, it would be valuable to further explore the generalizability of using single large language model as a Product-of of-Experts through augmenta- tions beyond ARC-specific transformations. In particular, text-based augmentations such as linguistic reformulations or stylistic variations present possible paths extend our method to a broader array of natural language reasoning tasks. Second, the effectiveness of our depth-first search (DFS) candidate-generation strategy warrants evaluation beyond ARC-like puzzles; exploring tasks such as logical reasoning, program synthesis, or mathematical problem- solving could yield insights into its broader applicability and effectiveness in structured problem-solving domains.
9

---

Product of Experts with LLMs: Boosting Performance ARC Is Matter of Perspective
Acknowledgement
Bostrom, K. and Durrett, G. Byte pair encod- ing is suboptimal for language model pretraining, 2020. URL https: wtw.oooogg/...115333131 eatpsseemml1..1....
would like to express our sincere gratitude to Lambda, for providing computational resources essential for optimiz- ing our pipeline. Specifically, they supplied with a server equipped with 8xH100 GPUs, enabling rapid iteration our ideas. Their support was instrumental in winning the ARC Kaggle Competition 2024 using the approach shown in this paper.
2020
Chollet, F. On the measure of intelligence. CoRR, abs/1911.01547, 2019. URL http::/aaxxii.rr//// abs/1911. 01111911..011.7.
This work has been supported by the "Research Center for Algorithmic Intelligence as an Emergent Phenomenon" (funded by the Carl-Zeiis-Stiftung) and by the Deutsche Forschungsgemeinschalll (DFG, German Research Founda- tion), project 233630050 (Collaborative Research Center TRR 146).
Cole, J. Community Interview Jack Cole Lab42 lab42.global. https://lab42.g obal/ wew.iiiie-ttteiiieeeeeeee jackiiiaaacccooleee.. 2024. [Accessed 29-01-2025].
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A.,
Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A.,
Generative AI language tools were used for text editing; all Al-generated output was subsequently reviewed, revised, and validated by the authors, who assume full responsibility for the final revision.
Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravanku- mar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Rozière, B., Biron, B., Tang, B., Chern, B., Caucheteux, C., Nayak, C., Bi, C., Marra, C., McConnell, C., Keller, C., Touret, C.. Wu, C., Wong. C., Ferrer, C. C., Nikolaidis, C.,, Allonsius, D., Song, D., Pintz, D., Livshits, D., Esiobu, D., Choud- hary, D., Mahajan, D., Garcia-Olano, D., Perino, D., Hup- kes, D., Lakomkin, E., AlBadawy, E., Lobanova, E., Di- nan, E., Smith, E. M., Radenovic, F., Zhang. F., Synnaeve, G., Lee, G., Anderson, L.., Nail, G., Mialon, G., Pang, G., Cucurell, G., Nguyen, H., Korevaar, H., Xu, H., Tou- vron, H., Zarov, I., Ibarra, I. A., Kloumann, M., Misra, I., Evtimov, I., Copet, J., Lee, J., Geffert, J., Vranes, J., Park, J., Mahadeokar, J., Shah, J., van der Linde, J., Bil- lock, J., Hong, J., Lee, J., Fu, J., Chi, J., Huang, J., Liu, J., Wang. J., Yu, J., Bitton, J., Spisak, J., Park, J., Rocca, J., Johnstun, J., Saxe, J., Jia, J., Alwala,
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
References
Akyürek, E., Damani, M., Qiu, L.. Guo, H., Kim, Y., and Andreas, J. The surprising effectiveness of test-time training for abstract reasoning. 2024. URL https:// arxiv.org abs /21...211...000 07279.
V., Upasani, K., Plawiak, K., Li, K., Heafield, K., Stone, K., and et al. The llama herd of models. CoRR, abs/2407.21788. 2024. doi: 10.48550/RRXIV.2000... 10. 48550 /0t/500//riiiv 2407 2407.21787. 21783.
Allen-Zhu, Z. and Li, Y. Physics of language mod- els: Part 3.1, knowledge storage and extraction, 2024. URL https: //tp://///eeeiiieeeee ner/forum? id-5x788rqbej.
URL https:
Allen-Zhu, Z. and Li, Physics of language models: Part 3.2, knowledge manipulation, 2025. URL https:// openreview net: (tt.ormmmmdddoDii001......
Greenblatt, R.
Getting 50% (SoTA) on ARC-AGI with
GPT-4o - www.worresarccc..ssstee...o https:
/ //redvoodresearcch. http://eaaaaaa.......a....aaaaaa swbstack. com/p/
arcprize.org. OpenAl o3 Breakthrough High Score on ARC-AGI-Pub - arcprize.org
2024. [Accessed 25-01-2025].
2025.
Hinton, G. E. Products of experts. In 9th International Conference Artificial Neural Networks: ICANN '99. IEE, 1999.
[Accessed 25-01-2025).
Berman, J. How I came in first ARC-AGI- Pub using Sonnet 3.5 with Evolutionary Test-
Hinton, G. E. Training products of experts by minimizing contrastive divergence. Neural Com- put., 14(8):1711-800,, 2002. doi: 10.1162/
time Compute - jeremyberman.uusttak.cok httpss//jjereybbee http://rmmybemma.uuusssaaaaa substack. subpaaaccoosccccccoo//// com/p/
1tw-----ott---roo---55-66- -at-eeoorr---------------------- (at-:36------------ arc-aigg
089976602760128801. URL ht htttps://doi.org/////
2024. [Accessed 25-01-2025).
1162/03997660276122228
10

---

Product of Experts with LLMs: Boosting Performance ARC Is Matter of Perspective
Hodel, M. Addressing the abstraction and reasoning corpus via procedural example generation, 2024. URL https: //////aaaa /arxiiv. org/ abs/ / 2404. 07353.
Seely, J., Imajuku, Y., Zhao, T., Cetin, E., and Jones, L, Sudoku-bench: Evaluating creative reasoning with su- doku variants, 2025. URL
abs Abs/2505.16138
Hu, J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adapta- tion of large language models, 2022. URL https:
Singh, A. K. and Strouse, D. Tokenization counts: the
impact of tokenization arithmetic in frontier Ilms. CoRR, abs/2402.14903. 2024. doi: 10.48550/ARXIV, 2402.14903. URL https: http://d....rr.1..93339 arXiv.
//openreview.no wtt.reeww.ne.eoumiiiiieeee... ttt.fforrmmiiineeeeeeee
Kamradt, G. Testing ol ol-preview ARC-
AGI, 2024.
URL attps://wwy
kaggle com/ code/ code/greggamratt/ wtwp:ii------peeeeeeoo-a-------- 2024-11-10.
Sreenivas, S. T., Muralidharan, S., Joshi, R., Chochowski, M., Patwary, M.. Shoeybi, M., Catanzaro, B.. Kautz, J., and Molchanov, P. Lim pruning and distillation in practice: The minitron approach, 2024. URL https:
Accessed:
LeGris, S., Vong, W. K., Lake, B. M., and Gureckis, T. M. H-arc: A robust estimate of human performance the
/ //aaiiiv.co org/ abs 2400 11796.
abstraction and reasoning corpus benchmark, 2024. URL https:/ / arxiv. org/ /bs/24/s//22/ 01374.
Sun, K., Qi. P., Zhang, Y., Liu, L.,, Wang. W. Y., and Huang, Z. Tokenization consistency matters for generative mod- els on extractive NLP tasks. In Bouamor, H., Pino, J., and Bali, K. (eds.), Findings of the Association for Compu- tational Linguistics: EMNLP 2023, Singapore, Decem- ber 6-10, 2023, pp. 1:300-133100 Association for Com- putational Linguistics, 2023. doi: 10.18653/V1/2023, URL https://ddi.oomm 10. N0.18653/v1/2023 http:////2222.......... findings-emmlp.c0
Li, W., Xu, Y., Sanner, S., and Khalil, E. B. Tackling the abstraction and reasoning corpus with vision transform- ers: the importance of 2d representation, positions, and org/ abs/
objects, 2024. URL https: 2410. 06405.
Li, W. Hu, K., Larsen, C., Wu, Y., Alford, S., Woo, C., Dunn, S. M., Tang, H., Zheng, W., Pu, Y., and Ellis, K. Combining induction and transduction for abstract rea- soning, 2025. URL https: http://pppeeeeeeeeeeeeee Forum?id- Dorrum?dd=mmootAVVD
Sun, Y., Wang. X., Liu, Z., Miller, J., Efros, A. A., and Hardt, M. Test-time training with self-supervision for generalization under distribution shifts. In Proceedings of the 37th International Conference on Machine Learning,
Liu, F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M.. Petroni, F., and Liang, P. Lost the middle: How language models long contexts. Trans. Assoc. Comput. Linguistics, 12:157-173, 2024. doi: 10.1162/TACL 00638. URLhttps::0 //doii::///...o// http://ti.orrg///////o 62/tacl 0_603
ICML 2020, 13-18 18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 9229- 9248. PMLR, 2020. URL http: / htttt:/prrooeediing http//pooeeenng... mir. mir.press/vile wttpsss//11///un2.b.bb. swm00bnn.b.// html.
Wind, J. S. ing Challenge - / //www. www.:kaggge....coopppo.e. wwm wwwp//ommeiiiinns////o abstractiion-add-reaso chag-challn challenge/ di discussion/154597, 2020.
Abstraction and Reason- kaggle.com.
https:
Moskvichev, A., Odouard, V. V., and Mitchell, M. The con- ceptare benchmark: Evaluating understanding and gener- alization in the ARC domain. Trans. Mach. Learn. Res., 2023, 2023. URL https:// htttt:::///eenrviee...tt forum? dit.miii-dyyykttt2222
2020. [Accessed 30-01-2025).
Neoneye. 79fb03F4 test is unsolvable, water flow Issue #100 fcholle/ARC-AGI - gimh https: //gpt/bb.tbbb/g www//chhlleee//CC-cc issues/100 00, 2024. [Accessed 30-01-2025).
Radcliffe, D. G. 3 million sudoku puzzles with rat- ings, 2020. URL https://wwe http::/wwwwwkglll.e. kaggle.gge.... com/
dsv/1495975.
RubenKelevra. Puzzle id: 0d87d2a6 result is am- biguous. /ttpss//githie com/ fchollet/
ARC-AGI/issues/144 2024. [Accessed 30-01- 2025].

---

Appendix for Product of Experts with LLMs: Boosting Performance on ARC is Matter of Perspective
A. Appendix: DFS Algorithm
The algorithm presented here assumes that the model supports internal caching for already seen sequences and only needs to process the newly added tokens. Note that we use negative log probabilities avoid numerical issues, while the main paper uses percentage values for clarity.
Our actual implementation differs from this simple variant, and requires to prune the key-value-cuche of the transformer ourselves.
using unsloth, which does not support dynamic caching
Furthermore, we use various performance optimizations, like simultancous initial forward pass of the best known sequence including prompt and prediction (which is much faster than token-by-tonn boekeb generation) as well as aggregating the sequences during backtracking to avoid the unnecessary processing of sequences that would be discarded later
Algorithm Depth-First Probability-Guidee Sampling for LLMs.
function _aSpsc {model, prompt, threshold, mar_len, eos_id}
Input: model is the language model
Input: prompl is the prompt that should be completed
Input: threshold is the maximum negative log probability allowed
Input: mar Jen the maximum length (including the prompt)
Input: cos_id is the index of the end of sentence token function Explore {lokens, score) if tokens Eos id or [tokens] 2 mar len then return (score, tokens)
end if
next token logits + model-predict (ogiis(to next_token prob + llo.- weg.sooftmaxxeeee wotesommmaaeetttttooooooooo
valid sequences t-
for each possible next token t. do nert & score + nert token prob[[r
if nexl-_ccrre < threshold then next.ooenss + current.tokens + [t]
continuations < cecooreenextttokeeeeee nert_score)
valii_sequencess+ velidcss continuations
end if end for return valid_scquences end function
return Explore(prompt, 0.0) end function
12

---

Appendix for Product of Experts with LLMs: Boosting Performance on ARC is Matter of Perspective
B. Appendix: Training Parameters
Table Training parameters and times for the initial and the test-time fine-tuning processes. Test-time fine- tunint is performed separately for each task, each time starting from the initially fine-tuned base model.
Initial Fine-Tuning
Test-Time Fine- Tunnnnuning
Batch size
4
Gradient acc. steps
2
LoRA rank
256
32
LoRA
24
16
LoRA bias
off
off
rank-stabilized LoRA
LR (LoRA adapters)
le-4
le-4
LR (embeddings)
le-5
1c-5
LR schedule
cosine
cosine
LR warmup phase
25%
50%
Weight decay
off
off
Optimizer
adamw_8bi
(elamw_Sbit
Base model quantization
4 bit
4 bit
Data type
bfloat16
b6loat16
Trained tokens
outputs only
outputs only single task examples
Training dataset
RE-ARC
368 [Llama]
Number of Epochs
64
1 1200 [NeMo]
Training performed on
1x Nvidia H100
1x Nvidia RTX 4090
15 hrs. [Llama]
12 sec./task [Llama]
Training time
98 hrs. [NeMo]
51 sec./task [NeMo]
Table 5. Reduced Token Set for ARC-AGI-specific L.M Model
Token Category
Tokens
Purpose
Alphabet
A-2, a-z (exel. 1,0,
Learned pre-prompt tokens
Numbers
0-9
Encoding the 10 colors
Newline token
in
Signals end of each grid line
Input/Output
I,
Signals start of problem input/output
Begin token
(bos)
Inserted once at the beginning
End token
(cos)
Inserted after each output
Padding token
(pad)
Internal usage (e.g. batching)
13

---

Appendix for Product of Experts with LLMs: Boosting Performance on ARC is Matter of Perspective
C. Appendix: Product of Experts Proof
Theorem C.1 (Error Bound for Log-Pooled Augmentations), Suppose we have m valid augmentations { Qm the sense of preserving solution distribution, and define
Pÿ(s) = P(o)(s) I oz(r)), for each j .,.... m.
Assume each single-augmentation predictor Pj has a bounded KL divergence from P. i.e.
Dj = KL(PII I 63.
Now define the "geometric-mean" ensemble
P(s) = z II P((0)
1=1
where
Z = E [P)]
wES, J=1
Then the KL divergence between P and P is bounded by the average of the single-augmentationn divergences:
KT
P
Σ (1(P)1
1=1
Proof.
Let us write
Dj = KT[(P]] P3) = Eewr [- log (s)] - EeaP log P(s)]
By assumption, Dj S 8, for each j.
Step 1: Expressing (I.(P P).
By definition of KI. divergence,
KL(PIP) = Σ P(s) log
P(a) P(s))
sES,
Eg- log P(s) - Es~ -Es-p-loog P(s)]-
Since we can rewrite P
P(s)
exp
1
log (s())
f=1
get
m Σ
log P(s)
1
log P3({) + log Z.
j=1
Thus,
1
Eg- - log P(s)
Σ E - log [s)] + log Z.
Subtracting E.-pp|-ee log Ps))] then yields
KL PP
1
Ea log Ps()) - Ea-p(- log P(s))
m
ΣTh (I(P)(())
log Z.
Hence to complete the bound, we need only to show that log Z S 0, i.e. that < 1.
14

---

Appendix for Product of Experts with LLMs: Boosting Performance on ARC is Matter of Perspective
Step 2: Bounding log Z. Recall that
m Z
Σ SESp
(s)
==1
Since the geometric mean is always smaller than the arithmetic mean for positive numbers, it follows that:
7 ΣΣ IΣ
[P((]]
sES, j=1
with equality exactly when all Py are equal. Further, as all P3 are probability distributions we find that:
Σ Σ
[+((}]]
1
sESp j=1
eE
Putting it all together. From Step of the proof, we have the decomposition
KL(PIP) = 1 Σ KKPP)))) + log Z.
average excess NLL
Combining with the bound Z < yields
m Σ (((1)))))
(P s m
1=1
15


Reviews:
[{"review_id": "8gHwiapRiZ", "reviewer": "Reviewer_Bm9D", "summary": "Authors propose a new approach to solve ARC-AGI challenge. In particular, authors train an LLMs to generate diverse, high probability solutions using augmented training data. Authors define an augmentation transformation for ARC-AGI dataset, which include rotations\nand reflections of each task, shuffling of the example order and permutation of colors. Next, authors systematically explore the space of solutions via a depth first search (DFS) algorithm, pruning any partial path whose accumulated probability falls below dynamically updated threshold. Finally, the solution is selected based on a product of probabilities across all augmentations. Authors test their approach of two models, and ablate on each component of their algorithm, showing the effectiveness of the proposed approach.\n\n## update after rebuttal\nI appreciate detailed response provided by authors. The paper has merits, but my main concern rating generalizability is still open. I will keep my score as weak accept", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and valuable feedback. We appreciate the detailed comments, and we address each of the points raised below.\n\nFirst of all, we appreciate the reviewer’s careful observations regarding formatting issues. We will revise the manuscript to address inconsistencies such as inconsistent paragraph spacing and other typographic details by following the organizers’ template.\n\nWe would like to clarify that the threshold used for candidate generation was not dynamically updated during candidate selection. Instead, it was chosen as a trade-off between computational speed and solution quality. We agree that a dynamically tuned threshold could potentially improve performance and view this as an interesting direction for future work.\n\nWe agree that assessing the generalizability of our approach is crucial. The focus on the ARC-AGI benchmark was intentional, as it emphasizes generalization on novel tasks using minimal example pairs. Nonetheless, a more systematic evaluation, namely extending to other tasks and benchmarks, is very important. In our revised manuscript, we will elaborate on potential future work, including: (1) discussing the idea of using a single LLM as a Product-of-Experts via augmentations on language tasks, which could also be explored using text-based augmentations such as reformulations or style changes. (2) testing the DFS component in generating high-quality candidates for tasks beyond ARC-like problems.\n\nIn response to concerns regarding the systematic evaluation of generalization, we have prepared a more detailed analysis of the Sudoku experiment (we uploaded Fig. 4 adapted for Sudoku to https://imgur.com/a/SY4g2OQ). This task, which shares characteristics with ARC in terms of example pairs, 2D data, and logical reasoning, provides further insights into the potential and limitations of our approach. Here, we achieve 53% solve rate for Sudokus using a fine-tuned Llama 3B model. (Please refer to our answer to Reviewer 2b2p for more details.) Finally, we also apply our method to the ConceptARC dataset, achieving similar results (73.3% accuracy with DFS-9% and otherwise same hyperparameters). While this dataset has a lot of similarity to the ARC dataset, it still provides some indication of generalization.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes, authors focus on a single benchmark, and test their approach with two opensource models.", "theoretical_claims": "No", "experimental_designs_or_analyses": "Yes, experiments described in Sec 5.\nGeneralizability of the proposed approach remains unexplored. Although authors provide preliminary results in Appendix A1 for Sudoku task, more systematic experiments are needed.", "relation_to_broader_scientific_literature": "Authors focus on ARC-AGI benchmark, that seeks to measure generalization on novel tasks, as opposed to other popular benchmarks that skill at tasks that can be prepared for in advance. Authors elaborate on test-time-training approach, utilizing task-specific augmentation transformation to synthesize more training data, and evaluate candidate solutions.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Paper is clear and well-written, although with some formatting issues. Proposed approach is novel and intricate. However, generalizability of the proposed approach remains unexplored. Although authors provide preliminary results in Appendix A1 for Sudoku task, more systematic experiments are needed.", "other_comments_or_suggestions": "Lines 085-085: \"The objective of the individual ARC tasks is to discern this rule\" - which \"this rule\"?\nLine 192: \"The trained language model (LLM)\" - LLM already defined in line 203\nLine 226: \"This two-step approach—(1) DFS-base\" - spaces missing around \"-\"?\n\nOverall, formatting seems off: some paragraphs have space between each other - which is expected - but some not. For example, lines 283-300, right column.  Line numbering on page 5 is suddenly cursive. Appendices are not named as such in sections after references, just some tables and pictures are dumped together. Should be fixable by following the template proposed by organizers: https://icml.cc/Conferences/2025/CallForPapers", "questions_for_authors": "See \"Other Strengths And Weaknesses\"", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4s0T4AVvX7", "reviewer": "Reviewer_LUdW", "summary": "The paper describes a system for the ARC challenge, based on using data augmentations.\nThe augmentations are basic transformations of the images (rotation, reflection, shuffling of the example order and permutation of colors) that can be applied to both the input problem and the solution, such that the transformed solution corresponds to the transformed problem.\nThese augmentations are then used for:\n1) Generating additional data for regular supervised training of the model.\n2) Generating alternative solution candidates. A given problem is transformed with different augmentations to produce alternative outputs, which can then be un-transformed to get alternative solutions to the original problem.\n3) Reranking of the candidate solutions. The probability of each candidate solution is calculated under each possible transformation, then combined to get an overall score for that solution.\n\nTest-time training from previous work is also applied. \nThe results show top open-source results (71.6%) on the public ARC evaluation set, with only GPT 03 outperforming it.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for their careful evaluation, interesting questions and positive feedback. We appreciate the detailed questions and suggestions, which will help us improve both the clarity and presentation of our work. Below, we address each of the reviewer’s points:\n\nBayesian Framework: We employed a Bayesian formulation to rigorously detach the uncertainty of the problem from the uncertainty of the erroneous LLM prediction for a single augmentation. Although this formulation did not alter our pipeline, it provides a formal grounding for ARC-like challenges and helps understand the theoretical value that the PoE approach has and shows that it is guaranteed to perform well under reasonable assumptions.\n\nWe agree that Figure 4 needs a more detailed explanation. We will revise the caption and discussion to clearly describe what is being illustrated. To clear any potential misunderstandings:\nSolid colored lines denote the number of tasks solved using a specific selection algorithm. The solid black line shows the number of tasks where the correct solution was amongst the sampled candidates, and thereby provides an upper bound for the performance of the selection algorithms. The dotted lines evaluate the performance of our selection algorithms, compared to this upper bound: What percentage of correct candidates are actually selected when they are present? It shows that even when using low DFS probabilities -  and therefore sampling a high number of candidates - PoE is able to select the correct solution among all candidates with high specificity.\n\nRegarding augmentations during Test-Time training and beam search comparisons: The same type of augmentations is consistently applied, however, we switch color and example permutations randomly whenever a task is drawn. We will clarify this in the revised version to avoid any ambiguity. \n\nTo clarify the use of augmentations in the ablation study without DFS: In this case we simply use stochastic sampling for each of the 16 different augmentations, thus generating up to 16 different candidates, which we will also clarify in the revised version.\n\nThe extra tokens mentioned (depicted as “A-Za-z” in Figure 2) are tokens representing single letters (‘A’, ‘B’, ‘C’...). As we also train the embeddings during fine-tuning, they take the role of a soft-prompt and are trained along with the model parameters. We will provide a more detailed explanation of their content and function and provide an additional ablation study regarding their usage in the next revision.\n\nRegarding the dfs threshold as used in the algorithm: We will change this to be more clear. We decided to use percentages in the main paper as they are easier to understand, but algorithmically we add (negative) logprobs instead to mitigate floating-point errors. This also answers the second point regarding threshold comparison. \n\nOn normalization by length, we have considered it, but surprisingly, this was rarely an issue for the ARC tasks. Investigating this, we found that the LLM has very high certainty for almost all positions, only being uncertain at some key pixels. Further, we found that the generated candidates rarely differ in length, as the model is very good at predicting the correct output shape, mitigating the effect of length normalization.\n\nWe will incorporate the noted typesetting-related suggestions in the next revision.\n\n**Reasons why DFS is faster than beam search**:\n\nThe speed advantage of the DFS algorithm during the candidate generation step comes mostly from the early pruning of low-probability paths. In Beam Search, the same amount of paths is explored each time, regardless of their cumulative sampling probability. As DFS stops when the cumulative sampling probability falls beyond the threshold, low-probability paths will not be explored by DFS, thereby reducing unnecessary computations. For example, in cases when the correct solution has a sampling probability higher than (1 - threshold), which happens frequently for the simpler tasks, DFS is guaranteed to follow only a single path and explores no side-branches at all, making it as fast as standard sampling of a single candidate.\n\nAdditionally, for all subsequent augmentations after the first one, we pass the most promising candidate solution as an initial guess to the DFS, and change the DFS exploration order to process this sequence first in a single forward pass of the model, which is much faster than token-by-token generation. Note that this does not change the result of the DFS algorithm, as all paths are still being explored.\n\nAs the DFS prunes low-probability solutions, the average number of candidates generated per task is also much lower for DFS with T=9% than with 4x Beam search, thus also making the subsequent scoring process faster as fewer candidates have to be scored. Note that the advantage during scoring could be mitigated by post-filtering the beam search results with a similar probability threshold.", "claims_and_evidence": "The work is interesting. \nThe ideas are good, showing creative ways of how task-specific augmentations can be used to improve performance and search the candidate space.\nThe results are good, setting a new state-of-the-art for open-source ARC models.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": ".", "experimental_designs_or_analyses": "Beam search is shown to perform just as well as DFS, while the advantage of DFS is speed. \nThese approaches are fundamentally doing the same thing - generating alternative candidates while pruning possible paths.\nPlease provide more of an explanation about where the speed advantage of DFS is originating from (or performance advantage, when considering comparable speed).", "relation_to_broader_scientific_literature": "SOTA compared to other open-source solutions to this problem", "essential_references_not_discussed": ".", "other_strengths_and_weaknesses": "The clarity of the paper and some details could be improved.\nCurrently, different graphs and tables are provided but only mentioned very briefly much later in the paper, without much analysis.\nThe problem is formulated in a bayesian framework but it is unclear why that is necessary or what benefit it provides.\nThere are various other open questions that need clarification (below).\n\nIt is difficult to understand what Figure 4 is showing exactly. It could use some more explanation.\n\nOn page 4 there is a large section of sampling under the \"Training\" sub-section. Not sure why that is there, it should probably belong to the \"Candidate Generation\" subsection.\n\nAre the augmentations used during test-time training as well?\n\n\"As illustrated in Figure 2, we add a small number of extra tokens to the start of a task.\"\nIt is not clear what these tokens are. Do you mean the \"A...Za...z\" tokens in Figure 2? If so, it is not clear what the actual contents (text?) of those tokens is.\n\nDFS generates candidates that can then be ranked using PoE. In Table 2, it is unclear how you rank candidates with PoE without the candidate generation step (DFS).\n\nWhen using beam search instead of DFS, it is unclear whether beam search also makes use of augmentations or is it just searching alternative outputs for a single input.\n\nBeam search is shown to perform just as well as DFS, while the advantage of DFS is speed. \nThese approaches are fundamentally doing the same thing - generating alternative candidates while pruning possible paths.\nPlease provide more of an explanation about where the speed advantage of DFS is originating from (or performance advantage, when considering comparable speed).\n\nAlgorithm 1 in the appendix seems confusing:\n\n1. The DFS threshold in the main paper is defined as a percentage (e.g. 9%). It is not clarified what this represents or how it is applied.\nIn Algorithm 1 the threshold seems to be an absolute logprob threshold instead.\n\n2. The algorithm continues as long as the current score is smaller than the threshold. While adding logprob values, the score can only get smaller. Should this be greater than the threshold instead?\n\n3. Assuming that this is greater-than the threshold instead, the score still doesn't seem to be normalised by length anywhere and it will continuously get smaller as more tokens are generated. Is that not an issue?\n\n\n####### update after author response\nThank you for your responses. I am already quite positive with the score and I will stick to it.", "other_comments_or_suggestions": "Typo: \n\"make us of alternative\" -> \"make use of alternative\"", "questions_for_authors": ".", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WkECvoiETq", "reviewer": "Reviewer_SUoh", "summary": "This paper presents a novel approach to solving the Abstraction and Reasoning Corpus (ARC-AGI) challenge, which tests abstract reasoning abilities in AI systems. The authors achieve SOTA performance for open-source models with a 71.6% accuracy rate (286.5/400 solved tasks) on the public ARC-AGI evaluation set.\nKey Contributions:\n\n- DFS-based sampling approach to generate diverse, high-probability candidate solutions.\n- Product of Experts (PoE) Scoring: The model serves as both a generator and scorer, using its output probabilities across different augmentations to select the most promising solutions.\n- Data augmentation approach: task-specific data augmentations throughout training, generation, and scoring phases, including rotations, reflections, color permutations, and example reordering.\n\n## Update after rebuttal\n\nThe authors answered my questions with great detail and provided convincing elements to address my concerns. I have therefore updated my rating to accept.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive review. \n\n\n**Regarding DFS vs BFS**:\n\nThe output of our generation process would be the same if we use DFS or BFS, as only the paths with sufficiently high sampling probability are kept. However, using BFS would make our optimizations harder. During generation, transformers use a key-value cache of previously generated tokens to avoid recalculation of this data. DFS has the advantage that we need to keep this cache for only one path in memory, thereby requiring only about as much memory as standard sampling. For BFS, on the other hand, we would need to keep this cache in memory concurrently for all parallel paths that are being explored, which would require a lot of memory (though on very large systems, one might be able to parallelize the inference better when using BFS). In our experiments, we chose to evaluate Beam Search instead, which is essentially a width-constrained version of BFS, and follows only the most promising paths to reduce the memory footprint.\n\n\n**On ReARC**:\n\nWe have seen no signs of overfitting on ReARC, and our ReARC trained model also performs well on other datasets besides the ARC eval set. For our largest model, we trained for 1200 epochs, using 6 examples (+1 challenge) in each prompt. As we never re-used any of the generated re-arc examples, this required 8400 examples for each or the 400 tasks, which we generated with the re-arc github code. We also evaluated the model using only test-time training on the ConceptARC dataset and achieved similar results to the ARC evaluation set (reaching 73.3% top 2 accuracy with DFS-9%), indicating that there is no overfitting specifically to the official ARC datasets. Additionally, we ran our method on Sudoku tasks, which have so far proven very hard for LLMs to solve. Our method allows a fine-tuned LLM to solve 52% of 1000 random Sudokus drawn from the 3M Sudoku dataset. We uploaded Fig. 4 adapted for Sudoku to https://imgur.com/a/SY4g2OQ.", "claims_and_evidence": "Nothing to report.", "methods_and_evaluation_criteria": "ARC-AGI is the program synthesis benchmark reference to measure reasoning abilities.", "theoretical_claims": "Nothing to report.", "experimental_designs_or_analyses": "Nothing to report.", "relation_to_broader_scientific_literature": "Nothing to report.", "essential_references_not_discussed": "Nothing to report.", "other_strengths_and_weaknesses": "- Why BFS vs DFS? Ablation on that?\n- Any sign of overfitting to re-arc? How much data was used? Were new tasks randomly generated at each epoch or was a fixed dataset used?", "other_comments_or_suggestions": "None", "questions_for_authors": "1. Why DFS and not BFS? Are there any ablations on that?\n2. Is there any sign of overfitting to re-arc? How much re-arc data was used? Were new tasks randomly generated at each epoch or was training done on a fixed dataset sampled from re-arc once and for all the training?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xMg1HYNk1X", "reviewer": "Reviewer_2b2p", "summary": "This paper proposes a new way to solve the ARC tasks. It first employs a depth-first search algorithm to generate diverse, high-probability candidate solutions for the ARC tasks, then applies an LLM to not only act as a generator but also as a scorer, using its output probabilities to select the most promising solutions. Experimental results show that their proposed method can enhance LLM's performance on ARC.\n\n## update after rebuttal\n\nIn the rebuttal, the authors have addressed my previous concerns and thus i have updated my score correspondingly.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and review, and we address each of the points raised below and improve the clarity of the manuscript in the next revision.\n\nRegarding why the autoregressive nature leads to the mentioned problem: (from the paper: “However [...] the highest probability solution according to a single augmentation is not always correct, partly due to autoregressive inconsistencies.”)\nThe claim in question was that in autoregressive models, the highest probability sentence is not necessarily the correct one. This becomes qualitatively more evident in the additional Sudoku experiment that we have expanded for this discussion (see below), which might help explain this more clearly: The autoregressive LLM can only attend to tokens in the past to reason what the next one should be. Because of this, it is sometimes necessary to make decisions that depend on information that becomes available later, even if the problem might be simple. For example, consider a Sudoku where the first cell is empty. In some cases the LLM would need to solve the full Sudoku internally before making the first prediction. As a result, errors can be introduced with relatively high confidence. After this error has been predicted, at some point there is no possible correct prediction that can be made. At this point the LLM behaviour is effectively undefined, as it is not trained to be stable under prediction errors. In other words, follow-up errors do not necessarily have a low probability, and the first error can be caused by sheer complexity of the necessary prediction due to the autoregressive nature. We will aim to explain this more clearly in the next revision.\n\nRegarding limiting our tokens:\nLimiting the tokens indeed limits the trained model's generalizability to other tasks, but the methodological approach is not tied to the small vocabulary. Instead we reduced the vocabulary for pragmatic reasons (to reduce the memory consumption in the spirit of the ARC Challenge) as the input and output embedding layers contain a very large number of weights and tokenization merges can substantially inhibit reasoning on numbers. Of course, instead of replacing the tokenizer, additional tokens could be used instead.\n\nRegarding generalization:\nWe agree that extending our approach and demonstrating its generalizability beyond ARC-like problems is important, and this is an active area of work for us. While many reasoning datasets pose challenges due to reliance on language output (where multiple correct answers can dilute probability), Sudoku offers a strong test case for structured reasoning. To provide evidence for broader applicability, we enhanced our preliminary Sudoku experiments significantly. \n\n**Extended Sudoku Experiments**:\n\nWe fine-tuned a Llama 3B model on 1 million puzzles from the 3M Sudoku dataset and applied our method to solve hard instances (requiring 19 to 31 clues). Using 8 augmentations and a 1% DFS threshold (with consistent hyperparameters otherwise), our pipeline correctly solved 52% of 1000 test Sudokus using a single guess. (Figure 4 adapted for Sudoku, illustrating this experiment, is available at https://imgur.com/a/SY4g2OQ). This is a notable success on a task known to be very difficult for LLMs. Importantly, the Product-of-Experts component was highly reliable, identifying the correct solution in 100% of the cases, where the correct solution was among the candidates generated by DFS (blue dotted line). We believe this result is complementary to our main results and shows that augmentation scoring using a single LLM can substantially increase the performance of an LLM on reasoning tasks.", "claims_and_evidence": "Please see the weakness section.", "methods_and_evaluation_criteria": "Yes, the proposed methods make sense for the problem.", "theoretical_claims": "Yes, I have checked the claims.", "experimental_designs_or_analyses": "Yes, the experiments are generally sound.", "relation_to_broader_scientific_literature": "The key contributions of the paper are related to the broader scientific literature.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Weaknesses:\n1. Line 252-254: why autoregressive will lead to this problem? please explain it.\n2. in section 5.1, the authors mention that they limit the number of tokens in the vocabulary to 64, which may limit the generalizability of the fine-tuned models from ARC to other tasks.", "other_comments_or_suggestions": "Please see the weaknesses above.", "questions_for_authors": "1. Line 252-254: why autoregressive will lead to this problem? please explain it.\n2. I understand that ARC is an important (inductive) reasoning task. However, have the authors tried their methods on other inductive reasoning tasks? I believe that this will make the proposed method more useful for the whole community.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors propose to sovle the ARC-AGI challenge through training an LLM to generate diverse set of candidate solutions searched over through DSF. Almost sota scores are reached. Reviewers agree on the paper being well written, significant novelty and good empirical results. The rebuttals are acknowledged. We urge the authors to take into account the minor writing criticism and improve their paper. Other than that the paper is a good contribution and most suitable for publication at ICML.
Decision: Accept (poster)