Searching Latent Program Spaces
Authors: Matthew Macfarlane
Clément Bonnet
Venue: NeurIPS 2025 spotlight ()
Keywords: Test-Time Compute
Latent Search
Deep Learning
Meta-Learning
Primary area: deep_learning
TL;DR: We introduce the Latent Program Network (LPN), a new neural architecture that builds test-time search directly inside its inference, by learning and searching through a latent space of implicit programs before generating outputs.

Abstract: General intelligence requires systems that acquire new skills efficiently and generalize beyond their training distributions.
Although program synthesis approaches have strong generalization power, they face scaling issues due to large combinatorial spaces that quickly make them impractical and require human-generated DSLs or pre-trained priors to narrow this search space.
On the other hand, deep learning methods have had high successes, but they lack structured test-time adaptation and rely on heavy stochastic sampling or expensive gradient updates for fine-tuning.
In this work, we propose the Latent Program Network (LPN), a new architecture that builds in test-time search directly into neural models.
LPN learns a latent space of implicit programs---neurally mapping inputs to outputs---through which it can search using gradients at test time.
LPN combines the adaptability of symbolic approaches and the scalability of neural methods.
It searches through a compact latent space at test time and bypasses the need for pre-defined domain-specific languages.
On a range of programming-by-examples tasks, LPN either outperforms or matches performance compared to in-context learning and test-time training methods.
Tested on the ARC-AGI benchmark, we demonstrate that LPN can both learn a compact program space and search through it at test time to adapt to novel tasks.
LPN doubles its performance on out-of-distribution tasks when test-time search is switched on.

Searching Latent Program Spaces
Matthew V. Macfarlane"
Clément Bonnet"
UUiiversity of Amsterdam
Equal contribution
Abstract
General intelligence requires systems that acquire new skills efficiently and gener- alize beyond their training distributions. Although program synthesis approaches have strong generalization power, they face scaling issues due the large combi- natorial spaces that quickly render them impractical, requiring human-genn -emannggeeeatedd DSLs or pre-trained priors to narrow this search space. On the other hand, deep learning methods have had high successes, but they lack structured test-time adap- tation and rely on heavy stochastic sampling or expensive gradient updates for fine-tuning. In this work, we propose the Latent Program Network (LPN), novel architecture that builds in test-time search directly into neural models. LPN learns a latent space of implicit prograls-nneurllyy mapping inputs to outputs through which can search using gradients at test time. LPN combines the adaptability symbolic approaches and the scalability of neural methods. It searches through compact latent space at test time and bypasses the need for pre-defined domain- specific languages. On range of geggraammin-----xxaaaane outperforms or matches performance compared to in-context learning and test-time training methods. Tested the ARC-AGI benchmark, demonstrate that LPN can both learn a compact program space and search through it at test time to adapt novel tasks. LPN doubles its performance out-of-die out-of-distribution tasks when test-time search is switched on.
tasks, LPN either
Introduction
The central goal of artificial intelligence has long been to create generally intelligent systems with human-like cognitive capabilities. While recent years have seen remarkable achievements in narrow AI domains, with systems achieving superhuman performance in games (Campbell al. 2002 Silver et al. 2017] Vinyals et al. 2019) and specialized tasks (He et al. [015) Jumper et al. [021]] we face fundamental challenge:
systems struggle generalize beyond their training distribution Yu
et al. 2024a] or to effectively adapt to novelty (Zhang et al.) 2021). This highlights a critical gap between optimizing for task-specific performance and achieving true general intelligence.
In On the Measure of Intelligence, Cealiigenne,/hhollet Chollet (2019) argues that traditional benchmarks that measure skills alone are insufficient for developing generally intelligent systems. Such benchmarks can be "gamed" through either unlimited prior knowledge of the task encoded by developers or massive amounts of training data, masking system's true ability to generalize and adapt. Instead, it is argued we need to measure skill-acquisition efficiency- how effeetively system can learn new tasks with minimal experience and evaluate generalization capability in a way that controls for both prior knowledge and training data.
By controlling for prior knowledge and experience, the Abstraction and Reasoning Corpus (ARC-
AGI) [Chollet, 2019 is benchmark that measures skill acquisition efficiency rather than pure skills.
Correspondence to <natthew. www.ivveee.omm co. uk>, <clement. bonnet160gmalt m0mmmaml...mmmm...........mm.mm. com>
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Its few-shot learning setup requires systems to adapt to novel tasks with very limited data, highlighting learning efficiency. Importantly, ARC-AGI is designed to evaluate artificial intelligence systems while calibrated to human performance. Therefore, it serves as a useful compass when comparing general intelligence between humans and machines.
Current approaches to solving ARC-AGI and program synthesis benchmarks generally fall into two categories: inductive and transductive methods. (1) Inductive approaches [Parisotto et al. 2016 Devlin et al., [2017b Butt et al. 2024) infer underlying programs from examples by generating
explicit programs using a domain- speeifiic language (DSL). However, handcrafting problem-spp ceoblem-specifie
DSLs is inherently unscalable for real-world applications and relies on the unrealistic assumption that a wwhaam-gggeeeee -ennaangeeertted DSL is always provided. Progress on Al-generated DSLs et al., 202T] has also been limited. In this work, we investigate methods that solve problems using only input-output example data. (2) In-context learning- also called
-ethhoss (Devlin et al.) 701 Kolev et al. 2020 Li et al.] [024a] train neural models to condition on few examples and produce the desired output for a new input, implicitly inferring the program within the model's activations. These methods benefit from greater scalability by removing the need for predefined DSL. However, in-context learning struggles with consistency [Devlin et al., 2017b], i.e., being able to map the inputs to the outputs of the specification itself. Performing fine-tuning at test time is powerful way to resolve consistency, but happens to be extremely costly on large models and is also prone to over- fitting given limited data (Li et al., 2021]. Additionally, in-context learning in transformers [Vaswani ct al. [017] suffers from the quadratic complexity of self attentioo (Hübotter et al. 2024 which hinders the practicality of scaling specification size.
As an attempt to get the best of both inductive and transductive worlds, we introduce the Latent Program Network (LPN), which integrates the benefits of scalable neural architecture with program induction. By representing implicit programs in a continuous latent space, LPN allows for efficient test-time program search. LPN is more efficient than test-time fine-tuning methods like in Devlin ct al. 2017a] because it performs backpropagation only through a fraction of the total parameters. Also, by performing latent program aggregation and recombination in latent space, LPN removes the quadratic cost of attention when scaling specification size.
Our contributions are as follows. (I) We introduce a new architecture named the Latent Program Network (I.PN) that builds in test-time adaptation by learning latent space of programs and searching for the best latent representation given new data. (2) We show that gradient-basee latent search during training optimizes the latent space for effective test-time adaptation, yielding significant performance improvements. (3) We demonstrate that LPN generalizes specification sizes beyond those seen during training, even improving performance, unlike in-context learning, which fails to generalize without parameter fine-tuning at large specification sizes.
2
Related Work
The challenge of navigating vast program spaces has driven diverse approaches to program synthesis. Early researchers focused on deductive methods, using eeoorem-proving techniques to construct provably correct programs based on formal specifications [Manna and Waldinger 1980]. How- ever, the difficulty of obtaining formal specifications led to shift toward inductive approaches [Solomonofi]] [964]. These methods instead infer programs from input- examples (Shaw et al.,
[77merss 1977. Biermann
1978), making program synthesis more practical for real-world
applications.
However, for real-world problems, the exponential search space is significant challenge for inductive program synthesis methods [Lee et al.) 2018). Three distinct paradigms have emerged to narrow down the search space, all leveraging learning. (1) Researchers have developed differentiable programming languages that integrate symbolic reasoning with continuous optimization [Feser et al. 2016 2017, Gaunt et al.) 2017]. (2) Neural networks have been trained as priors to constrain the search space and enhance the efficiency of program search techniques [Balog et al.. 2016 Devlin et al.] 2017b). In this paradigm of leveraging neural priors for search, large language models (LLMs) have been demonstrated to be valuable for navigating complex search spaces, including
discrete program synthesis [Wang et al. 2024 Li et al., 2024d Barke et al., 2024]. Codelt [Butt jet al. 2024] leverages the pre-trained CodeT5 model [Wang et al. [023] to guide discrete program
search. (3) in-contexttransductie approaches bypass intermediate program representations entirely

---

by (meta-) training neural networks to directly map input to output examples and a new input to its corresponding output [Devlin al. 2017ab Kolev et al. 2020]. Neural networks offer a
fully differentiable program representation, allowing program induction through gradient descent
in parameter space, where network weights encode the program structure [Graves] [014 /aremba
and Sutskever, 2014, Neelakantan et al., 2015 Kaiser and Sutskever 2015 Kurach et al. 2015].
While early research focused on learning individual programs, more recent works have broadened
the scope to learning multiple programs [Devlin et al. 2017a Kolev et al. 2020 Li et al. 2024a]
Pre-trained LLMs can also be used to perform transductive reasoning, where they directly generate outputs from specifications included in the prompt- process known in-context learning Gendroon
2023 Mitchell et al. 2023 Li et al., 2024a] Brown et al., 2020]. Recent works by Hendel et al.
et al.
2023 Yang al. [022] demonstrate the emergence of task vectors within LLM activations.
LPN advances this approach by explicitly learning and disentangling the program space from other activation information. This separation yields two benefits: enhanced program representations and more efficient program space exploration. While such neural approaches eliminate the need for formal programming languages or DSLs, they sacrifice both interpretability and the ability to perform systematic search during inference. AsDevvin et al. [2017b) demonstrated, neural approaches struggle to maintain consistency with given specifications and underperform compared to neural-symbolic methods Parrisotto et al. 2016].
Recent work has explored fine-tuning model parameters at test time test-time training (TTT) to resolve this inability to be consistent with the specification [Devlin et al. 2017a] Hottung et al.; 2021b
Hübotter et al.; 2024, Li et al., 2024a, Akyürek et al., 2024]. TTT treats the network parameters
as a program, where test-time fine-tuning becomes a search through program space via parameter optimization. Since neural networks universal function approximators (Hornik et al. [989]. the target program is likely to exist somewhere in this parameter space. Meta-learning approaches such MAML [Finn al. 2017]. perform gradient updates on few-shot data in the full parameter space. LEO [Rusu et al. 2018 similar to our work explore a version of MAML, introducing a bottleneck, such that the inner loop optimisation is performed in latent space. Their decoder is then a hypernetwork (Ha et al. 2016 which conditions on the latent to generate the raw parameters for inference. LPN removes the need for such hypernetwork leveraging the in-context learning abilities of transformers to directly perform generation conditioned on the latent variable.
Conditioning neural models on latent spaces has emerged powerful approach across diverse do- mains. Latent space optimization has been applied to molecule generation [Gömez- Bombarlll et al. 2018) and combinatorial optimization challenges, including the traveling salesman problem [Hot tung et al. 2021a la Chalumeau et al., 2023). Recent work has extended -atent-based approaches to black-box optimization through energy-basee models Yuu et al. 2024b], and symbolic mathematics, where latent optimization helps balance equation complexity with accuracy [Meidani et al. 2023]. In
program synthesis, LEAPS [Trivedi et al. 2021 demonstrated how reinforcement learning could
search latent program spaces for Karel programs [Pattis] 1994).
LPN can be interpreted through the lens of cemitteootieed variational inference (SVI) Kim et al. 2018 Marino et al.) 2018). SVI operates in two phases: first, it performs amortized variational inference using an encoder trained on the complete dataset. Then, for each data point, it performs ad- ditional updates to minimize the amortization gap [Gershman and Goodman) [0014] the discrepancy
between the log-likelihood and the evidence lower bound (ELBO) [Krishnan al. 2018 Cremer
ee al. 2018). The LPN equivalent of these updates for test examples is performing gradient descent in the latent space.
3 Background
Program Synthesis aims to generate deterministic programs in a target language, such that outputs generated from inputs are consistent with the given specification. Typically, the problem space Y consists of programs formulated within a domain-specific language (DSL). Each task is defined by a specification set X. where each specification, X,, E X, described by a set of input/output (I/O) examples:
X, {(aT' yf ,r),, (T ".
A program I E Y is considered to solve the task associated with X,, if it satisfies: Vjj E [1, [(x)) = y"" This definition requires that the program replicates the output for each input in the specification. We denote F., to represent the true function that generates the I/O pairs.
3

---

Program Synthesis Generalization. We consider the problem of applying learned program to new imput rather than explaining the specification. Given a set of input-output examples generated by program Fm. (not necessarily constrained to DSL., along with an additional input 2n11
Pm {(x v1" ................................ (x yn 1
The objective is to generalize from the provided examples and predict the corresponding output for x.+11 This can be done via induction or transduction [Li al., 2024a]. If do not limit the Kolmogorov complexity of programs [Kolmogorov] 1965 Solomonotti 1964), we can find an explanation for any given specification, whether or it corresponds the true program that underlies the specification. Evaluating generalization performance not only tests the ability to explain specification but also whether it can successfully infer the underlying program in its generality and apply it to new input. This problem bears resemblance to few-shot learning, with the difference of having only one task. The ARC-AGI benchmark [Chollet 2019) falls under this program synthesis formulation. Each of its tasks is composed of input-output pairs represented as 2D grid of shape up 30x30, whose cells can take any of 10 colors.
4 Latent Program Network (LPN)
Latent Optimization
Encoder
+
Decoder
Decoder log x.
Figure I: Inference of the Latent Program Network (LPN) model. (Left): the encoder maps I/O pairs to latent space of encoded programs. (Middle): the latent program is refined during an optimization process to best explain the given I/O pairs (figure detailed in the appendix at Figure 20). (Right): the decoder executes the latent program to generate the desired output for newly given input.
Prior work tackling programming by example with a neural approach has focused on directly training models to maximize the likelihood of decoding the correct output given a specification (in context) (Kolev al. 2020). However, we diverge from such transduction-bar based methods, as they cannot inherently adapt at test time or ensure specification consistency. Instead, we explicitly factorize inference into three core components visualized in Figure] [ First, we introduce a bottleneck that encourages the network to learn explicit representation of programs via compact latent space, an architecture also used Neural Processes [Garnelo al., 2018). Secondly, we introduce a method for searching this latent space to explain the given data effectively. Lastly, conditioned on the latent program and new input, we predict the output. Despite this added structure, the Latent Program Network (LPN) remains fully differentiable end-to-end. This structure provides several key benefits. Firstly, explicitly learning program representations acts as a good neural network prior for generalization. Secondly,
use this program encoding to verify that our initial guess for latent program explains the given data. If not, the program latent can be refined at test-time to best explain the given data. Lastly. LPN removes issues faced by transductive approaches that overfit to specification sizes (number of input-output pairs) seen during training.
4.1 Latent Program Inference
LPN is composed of three core architectural components to perform inference. A neural encoder, neural decoder, and latent space optimization. In this section, we discuss each component and then outline how to train the full system in later sections.

---

Encoder. The probabilistic encoder is trained to approximate the Bayesian posterior over program latents. Specifically, maps an onputt pair (x, y) to a distribution the latent space go (2|2; y). representing possible programs that could explain the given input-output mapping. Using variational approach is important because, for any given input-output pair, there exists a broad range of possible programs that map the input to the output, even when restricting to, e.g., programs of low Kolmogorov complexity [Solomonofi 1964 Kolmogorov 1965]. We discuss LPN as semi-amortised variational inference in Section D Intuitively, the encoder is trained to learn an abstract representation of programs in a continuous latent space by implicitly encoding input-output pair examples. In practice, use multivariate normal distribution whose mean ju and diagonal covariance E parameters inferred by the encoder. To take advantage of hardware parallelization, the encoder can process all the 1/O pairs in given specification in parallel. By encoding each pair independently and aggregating using the mean LPN is permutation invariant to the specification order, as opposed to a naive sequence model that would concatenate I/O pairs.
Decoder. The probabilistic decoder is responsible for mapping latent program and an input to its expected corresponding output, directly predicting the output pixel-by-pixx instead of via DSL. models the distribution of possible outputs y given an input x and latent 2. Note that even if the underlying I/O mappings are deterministic, we still use a probabilistic decoding framework Po(y]æ, =) to be compatible with maximum likelihood learning. Figure 20 the decoder generating different outputs by keeping the input fixed but varying the latent program, which in this figure represents a specific grid pattern to reproduce. In real task, the aim of this encoder-decodes system is to learn a compressed representation of the space of possible programs we care about (e.g., in the case of ARC-AGI, this would correspond to programs that use the Core Knowledge priors (Chollet 2019
Latent Optimization. The encoder is trained to approximate the posterior over programs and may not encode the right abstraction given an I/O pair. Especially if the task is very novel, the encoder may fail at producing the right latent program, which, fed to the decoder, would generate the wrong output. Therefore, we include middle stage of latent optimization where, starting from the encoder's prediction 2, we search for a better latent program z', one that would better explain the observed data according to the decoder pa. The search process is generally denoted 2' = [(po, 2, 2. y)) and can be implemented in several ways (c.f. section 4.2) Analogous to system / /system 2 thinking (Kahneman 20 we can think of the encoder generating an intuitive first guess as to what the observed program may be (system 1). and the latent optimization process executing a search for hypotheses that would better explain the observations (system 2). See Section C test-time inference pseudo-code.
Encoder
Latent Optimization
Decoder
(3)
2 ()) v)
z' = f(pe 2, f(p,,2,,, v,
i ~ po 21)
4.2 Search Methods for Latent Optimization
Given n ouput- pairs ((1,, W=) the search process z' [(po z, x.y) attempts to find a 2' that satisfies:
z' E arg max log po (o(
This means we search for the latent that would most likely make the decoder generate the right outputs given the corresponding inputs. By finding a latent that can explain all the input-output pairs, the latent solution to the optimization problem is more likely generalize to new input-output pair. We describe here two instantiations of the search process, namely a sampling and a gradient ascent algorithm, both acting in the latent space of programs. We leave for future work the exploration of other search methods like evolutionary strategies [Hansen and Ostermeier 2001 Chalumeau et al. 2023 that could better trade-off exploration and exploitation of the latent space.
Sampling, A naive version of the latent search process is to sample a batch of latents from either the prior distribution p(=) or around the approximate Bayesian posterior qo (a .i, ys) and select the latent that gives the highest log likelihood of decoding the given input-output pairs. Specifically, for all E [1, K]. 2 ~ p(=), and we select z' E arg max., Σ log po [,, zs). Sampling asymptotically converges to the true likeeioo latent (equation 4 and can prove useful
5

---

when the function to optimize (here, the decoder log- (egliekliiood) is not differentiable. However, the efficiency of sampling- based search decreases exponentially with the dimension of the latent space, which makes it impractical for most applications.
Gradient Ascent. Since the decoder is differentiable neural network, its log-likelihood (oy(o( 2) is also differentiable with respect to z and one can use first-order methods like gradient- ascent to efficiently search through the latent space for a solution to the latent optimization problem (cquation 4 see Figure 20 for a visualization of a 2D latent space trained on grid patterns). This visualization highlights that only small portion of the latent space can explain all the input-output pairs, corresponding to high decoding likelihood. Notably, poor initialization can lead the search to converge different local minima, highlighting the importance of amortized inference from the encoder. We initialize =6 as the mean of all the pair latents sampled from the encoder, and refine iteratively follows:
vk K], = a V.
Σ log po n4, 2=1
The series (e,,)_( should exhibit increasing decoding likelihood if the step size a is small enough. In practice, we generate the output with the best latent found during the gradient ascent algorithm, which may not always be the one that is obtained after taking the last gradient step (2'g))
4.3 Training
To train the LPN system end-to-end, assume have dataset of tasks, where task is defined as n input-output pairs (z4, y:) generated by the same program. To simulate the test conditions of predicting new input from given specification, each of the outputs y; from their inputs I; and all the n 1 other pairs (zzz V3 j We emphasize that we do not use the specific pair (Iz; y4) to reconstruct yıs which would lead to the encoder directly compressing the output yi as a shortcut without learning mrogram-relatee abstractions.
design the training procedure to reconstruct
When reconstructing output yi. we first sample latents from the encoder 4s (2| for all j * i. We then aggregate them by computing their mean n-1 ΣjØs 2j, then we perform latent optimization using c.g., gradient ascent to obtain z/. Finally, we compute the negative log-likelihood of the right output y, using its corresponding input x4 and the refined latent 2/-- In practice, we compute the cross-entropy loss of the decoder logits pa (1i [z;, z and the labels which is derived from maximizing the likelihood of categorical distribution. The full training pipeline is detailed in Section C Specifically, we compute the reconstruction loss Live and the KL loss LxL between the approximate posterior and the prior:
Liee (g. -/
2 i=1 - loggoo 21)
(Ø) Em DKL (go(2/x1, BR) (O.
(6)
The dependence of the reconstruction loss Lree 0) in ei arises from using the reparameterization trick [Kingmai 2013) when sampling each latent 2y. Indeed, we first sample a normal random vector ej ~ N (1, I), then we infer the mean (4 and diagonal covariance Σ, using the encoder and recompose the latent 2, = My + Σy. Then, 2 is used by the decoder to reconstruct the output. Note that we can decide whether to let the decoder gradient flow through the latent update. Indeed,
it is more computationally efficient to stop the gradient through the update, by changing line 9 of algorithm 2 with 2! = 2/ + a g/, where g = V. > j log po po(y)la,. 2)|2= with gf notating a stop-gradient on g'-
We denote 3 as the weighting factor that balances the reconstruction and KL terms [Burgess et al. 2018), which gives the combined training objective: (ouu( 0) Lee (ő. 0) + (LkL This training procedure offers some freedom in the latent optimization, i.c., how to compute 2 from z; Training with gradient ascent latent optimization (as detailed in Algorithm 2 incurs a compute overhead due to the cost of the latent gradient computation through the decoder. Although we may use high compute budget test-time, we propose to use small number of gradient ascent steps during training, ranging from 0 to steps.

---

Experiments
Inference
Training
Grad 0
Grad
Grad 5
Grad 20
Grad 100
Sample 250
Grad 0
3.2 (2.7)
3.6 (3.0)
18.8 (14.4)
52.5 (25.0)
67.5 (20.0)
3.2 (2.7)
Grad 1
8.6 (4.4)
44.6 (10.9)
85.4 (7.6)
98.4 (1.4)
99.5 (0.5)
10.2 (5.3)
Grad 1
0.6 (0.1)
13.7 (3.0)
60.2 (7.5)
88.9 (6.0)
94.1 (3.8)
0.7 (0.2)
Grad 5
0.0 (0.0)
0.4 (0.3)
31.9 (11.2)
88.5 (11.9)
98.1 (2.1)
0.5 (0.4)
Sample 5
6.1 (4.4)
8.2 (6.5)
27.7 (21.6)
56.3 (27.5)
72.2 (21.2)
6.1 (4.4)
Table 1: Ablation of LPN training and inference methods the Pattern task, reporting exact match accuracy (%). Rows/columns represent different trainingfinference methods, differing only in the latent optimization. Grad [N] stands for N gradient ascent steps, Sample [N] for N samples from the encoder distribution without leveraging any gradients, and Grad means that the decoder parameter gradient flows through the latent optimization update. Training was performed for 20k steps with 3 seeds, aggregating performance as mean (and standard deviation in brackets) over the runs. Bold values indicate the best training method for each inference regime. See expanded table in Section B.T
5.1
Setup
We consider the ARC-AGI 2024 challenge [Chollet et al. 2024] as the testing domain for our method. This program synthesis benchmark encompasses diverse tasks designed to adaptation and out- of-distribution generalization, rather than memorization. We implement both the LPN encoder and decoder as small transformers [Vaswani et al. 2017 see Section G for full architecture details. We introduce the simpler Pattern task to investigate LPN's dynamics before large-scale ARC-AGI training, see Section A It generates 10x 10 black input grids with blue pixel indicating where a 4x4 program-specific pattern should be pasted. This task is sufficient to demonstrate weaknesses in deep learning models that do not leverage test-time computation (see Section 5.5).
Baselines. We compare LPN an in-context learning method Kolev et al. 2020 Li et al. 2024aa]
which encodes each of the input-output pairs and then concatenates these embeddings to condition output prediction, notably never producing any intermediate program embeddings. We discuss the motivation for this design in Section Then, we also compare to test-time fine-tuning [Devlin al. 2017a Akyürek et al. 2024], which, given a test-time specification, performs parameter-based gradient updates on the in-context model. See Section H for details on baseline implementations and Section for hyperparameters.
5.2 Pattern Task
We compare a variety of LPN training and inference methods in Table [ to better understand the dynamics of LPN. We aim to first answer whether LPN can self-improve test-time by searching its latent space, and then how the training inference strategy affects performance. For each training method, train small IM-parameter model for 20k steps with batch size of 128 and evaluate with different inference modes. We find that inference using no test-time adaptation performs poorly across all instances of LPN (and in-context baselines, Section B.2.. This shows that amortizing inference is difficult for models of this size and training time.
All variations of LPN training show strong scaling in performance as the number of gradient steps at test time is scaled. This demonstrates that LPN is capable of test-time adaptation to improve pre-trained amortization performance. We visualise the decoded outputs at many points in the latent space of LPN on a simple pattern task in Section B.4 showing structure in the latent program representations, and Section B.4 shows the likelihood of the specification is smooth across the space, enabling gradient based test time search.
When comparing different LPN training inference strategies, gradient ascent step of latent op- timization shows higher returns than training with mean pooling, with the difference particularly pronounced when scaling the inference budget. With 100 steps of gradient ascent inference time,
7

---

LPN training with Grad 0 gets an accuracy of 67.5%, whereas training with one gradient step (Grad reaches 99.5%. This demonstrates the benefits of training the latent space with the awareness that gradient ascent will be performed at test-time, an important inductive bias for the LPN architecture. We also observe that gradient ascent vastly outperforms sampling- -aage search, validating that search without any gradient signal is highly inefficient. Lastly, we perform training with and without the stop gradient on the gradient update itself. We found that using the stop gradient actually leads to higher test time performance while also being more computationally efficient.
5.3 String Manipulation Task
To investigate the generalizability of our results beyond environments with significant spatial structure, we perform an ablation on a synthetic sequence task and replicate the analysis previously conducted on the pattern dataset.
This synthetic dataset features vast program space, with over 100 mil- lion unique programs, each defined by composing 3 to 5 parameterized rules that transform sequences of num- bers (ranging from 0 to 4), see Sec- tion B.7 for further details. Our ex- periments demonstrate that Test-Time Training (TTT) exhibits overfitting, as evidenced by its performance degra- dation when gradient tin-ttnning is performed. In contrast, LPN main- tains robust performance without over- fitting. Furthermore, incorporating single gradient step during training en- hances LPN performance marginally at higher inference gradients.
Inference
Training In-Context
Grad 0
Grad
Grad 100
77.7 (1.2)
78.1 (1.1)
72.8 (0.8)
11.8 (1.9)
LPN Grad
75.6 (1.7)
83.3 (0.8)
81.3 (0.5)
LPN Grad
71.3 (0.4)
86.1 (1.5)
81.8 (1.8)
LPN Grad
**
67.9 (1.3)
85.0 (0.4)
81.0 (1.3)
Table 2: Exact match accuracy (%) on the test set for the string manipulation task. The comparison includes In- Context baseline, Test-Time Training (TTT) (Ir = 1e 4), and three LPN variants (with Ir le 1): Grad 0, Grad I, and Grad ** (decoder gradient flows through latent optimization). Performance is averaged over 3 runs, with standard deviations in parentheses.
5.4 Starting test-time search from the encoder
We experiment on the Pattern task again, this time assess the benefits of amor- tizing inference. We ablate the impact of initializing latent optimization with the en- coder versus using the prior.
Inference Method
Gnd encoder
Training Method
Grad
Grad
We specifically compare no latent search (blue) to doing latent optimization in two ways: sampling from the prior = p(=) (orange) and sampling from the encoder 2 (2),,,,,)) (green).
Grad
Sample
20
Accuracy (%)
Initializing latent search with the encoder is critical for performance across all train- ing methods. This validates the intuition that LPN perform fast system 1-like reasoning using the encoder and then nar- down the search space during latent optimization, simulating system reason- ing.
Figure 2: Ablation on the role of the encoder and latent optimization. Latents are initialized from the mean of the encoder latents (except for orange). Grad N stands for doing latent optimization with N gradient steps. Both the encoder initialization and the latent optimization matter for LPN.
8

---

5.5 Adapting Out-Of-Distribation
A significant challenge to deep learning meth- ods is generalizing out-of-disrribttion (OOD) tasks. We study the OOD behavior of LPN, in- context learning, and parameter fine-tuning the Pattern task in [ In-context learning has no test time adaptation performance as has no inherent mechanism to scale compute beyond single forward pass. We include full experiments on range of varying OOD settings in Section B.2
Inference
Training
Grad O
Grad 10
Grad 100
In-Context
0.0 (0.0)
TTT
0.0 (0.0)
1.8 (0.7)
0.3 (0.1)
LPN Grad
(0.5)
18.8 (14.5)
41.1 (29.6)
LPN Grad
0.0 (0.0) 59.9 (11.6)
88.0 (5.3)
I.NN Grad
0.0 (0.0) 38.5 (13.0)
81.8 (10.9)
Table 3: Out-of-distribution (OOD) performance on the Pattern task, measured as exact match accu- racy (%). Performance averaged over 3 training runs with different seeds, with standard deviation in parentheses.
When trained with 1 step of gradient ascent, LPN recovers high accuracy (88%) using 100 steps of gradient ascent at inference time. TTT and in-context learning incapable of recover- ing the right OOD pattern at inference time. We perform similar experiment in Section B.7 on a synthetic integer sequence task with less spatial structure, with the same conclusions.
5.6 Scaling Specification
For any learning method that can handle vary- ing amounts of data, an important question is how the method scales to handling more data than it was trained on. In Figure 3 we evalu- in-context learning, TTT, and LPN with and without gradient search in the OOD setting of the pattern task, with varying specification sizes. LPN smoothly generalizes to different specifica- tion sizes, with only a small drop in performance with less data, and keeps improving with more data. In-context learning overfits to the specifica- tion size it is trained on, and TTT only performs well once given a sufficiently large specification and high inference compute. We ablate this in Section B.3 repeating the experiment for vary- ing training specification sizes.
0.6
0.6
0.4
LPN LPN
TTT grad 200 Innontt
0.2
0.0
Specific www.iiaiiooon.. 512
Figure 3: Exact match accuracy (% on the out-of- distribution (OOD) pattern task as the specification size is scaled from input output pair 19 pairs, for different inference methods.
5.7 ARC-AGI 2024
Setup We train on the re-arc dataset [Hodel 2024), designed to be distribution relative to the ARC training set (which we don't 't train on) et al. [002bb]. The evaluation set is significantly OOD, representing a challenging generalization experiment. We train a 178M-parameter LPN with 256-dim latent space for 100k steps for 2 days a TPU v4-32, see Section E Leveraging insights from pattern task experiments, we use gradient ascent in LPN during training. To remain efficient, we train LPN in Grad 0 mode for 95k steps and then fine-tune in Grad I for further 5k steps, which we find gives a marginal boost in performance at lower levels test time compute Section|B.. B.8
Results Table 4 shows top-2 accuracy on ARC-AGI (standard to ensure resolvability of all prob- lems). Even after significant amortization of inference, we observe large performance gains from scaling gradient search at test time. LPN significantly outperforms TTT in-distribution. We also see strong evidence that the LPN has learned a structured latent space for representing programs, demon- strated by a t-SNE plot of the embeddings of different programs in the training set, see Section F Out of distribution, LPN doubles its performance by using latent space search at test-time (scaling FLOPs from 2e11 to 2e15). Using from 2F+11 to 2F+ 13 FLOPs, LPN outperforms TTT, until we scale test-time compute to much higher levels, where TTT achieves higher performance. This finding is likely due to the lack of diversity in programs seen by LPN during training, relative to experiments

---

D--Distriuttio
Out-of-Distribure
FLOPs
LPN
TTT
Codelt Mirchandani
LPN
TTT
Codelt Mirchandani
2E+11
68.75
45.75
7.75
5.85
2B+12
75.95
51.75
10.25
7.35
2E+13
80.00
58.50
15.25
13.50
2E+14
76.25
58.75
15.50
16.00
2E+15
78.50
57.00
15.50
15.25
2E+15
14.00
14.75
6.75
Table 4: Results on ARC-AGI, measured by exact match accuracy (%), for varying values of FLOPs. In In-Distribution refers to puzzles in the ARC train set not included in the training set, but in-distribution relative to re-ARC. disttr-iiitiiittoo refers to puzzles from the eval set. LPN and TTT stand for Latent Program Network (ours), and Test- Test-Time Training. Codelt and Mirchandani baselines from prior work.
other datasets, limiting the ability have smooth expressive latent space. See Section B.1 11 for an ablation on the performance contribution to LPN from encoder initialization and gradient search.
Relative the pattern task, TTT is less susceptible to overfitting on ARC-AGL, possibly due to the use of larger model sizes. Both methods outperform two ARC-specific LLM approaches that leverage far larger pre-trained models, including CodeT5 [Butt et al. 2024] (220M parameters) and text-davinci MMrrchandani et al. 2023] (175B parameters). This work represents an improvement over such methods due significantly lower inference and training cost, and greater generalization from a limited training distribution. At the time of writing, the highest performing model on ARC-AGI o3- 13-preview(low (LLM) which achieves 75.7% a cost of $200 per task. Both the training and inference costs are far beyond this work, there is an opportunity for future work to investigate scaling up this work such scale. To further contextualise the results compared to other methods applied to ARC-AGI we outline in Section | benchmark highlighting the training assumptions and compute for all methods where the information is public.
complete table of leading and diverse approaches to the
In section Section B.9 investigate whether the problems solved, the ARC-AGI evaluation test set, by TTT and LPN are different. We find that indeed different subsets of the test set are solved by each method and we visualize 3 examples of solutions only solved by LPN and TTT respectively.
Lastly, in Section B.10 we explore L.NN's capacity for compositional generalization test time. We find that the latent search process can compose programs seen during training into novel combinations, which avoids the overfitting seen in models without search, that tend to execute only a single program. However, this capability is not robust; our analysis identifies both successful examples where the search enables composition and clear failure cases where does not.
Conclusion
We introduced Latent Program Network (LPN), novel approach to inductive program synthesis leveraging continuous latent spaces for efficient test-time adaptation. LPN integrates adaptation directly into its architecture, refining latent representations through gradient-based search, which we identify as an efficient adaptation method. We show that LPN generalizes beyond the training distribution, relying on test-time adaptation rather than expanded synthetic datasets.
Limitations and Future Work. A limitation of this work is the limited diversity of programs on which LPN is trained. While augmentations are used during training, the distribution of programs remains narrow and restricts the potential for learning an expressive complex latent space. We limit bradient-based search standard optimizers. Future work could explore alternative optimization methods, such as evolution strategies, and explore discrete program representations to enhance compositional generalization.
10

---

Acknowledgments and Disclosure of Funding
We thank Google's TPU Research Cloud (TRC) for supporting this research. We are also grateful to Nathan Grinsztajn, Natasha Butt, Levi Lelis, and Jessica Hu for their feedback the early versions of the paper. Matthew Macfarlane is supported by the LIFT-project 019.011, which partly financed by the Dutch Research Council (NWO). We also thank reviewers for their detailed feedback, which we have integrated into the final version of the paper.
References
Ekin Akyürek, Mehul Damani, Linlu Qiu, Han Guo. Yoon Kim, and Jacob Andreas, The surprising effectiveness test-time training for abstract reasoning. arXiv preprint arXiv:2411.07279, 2024.
Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. arXiv preprint arXiv:1809.108: 2018.
Matej Balog, Alexander L Gaunt, Mare Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. Deepcoder: Learning to write programs. arXiv preprint arXiv: 1.07989, 2016.
Shraddha Barke, Emmanuel Anaya Gonzalez, Saketh Ram Kasibatla, Taylor Berg- Prrg-KKikpattrck, and Nadia Polikarpova. Hysynth: Context-free lIm approximation for guiding program synthesis. arXiv preprint arXiv:2405. 15880, 2024. httpp:/tp////riiiooorr abs 2405. 15880
Alan W Biermann. The inference of regular lisp programs from examples. IEEE transactions on Systems, Man, and Cybernetics, 8(8):585-600) 1978.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Moosertt-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877-1901. 2020.
Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in 3-vae. arXiv preprint (TXiv:/804.03599, 2018.
Natasha Butt, Blazej Manczak, Auke Wiggers, Corrado Rainone, David W Zhang, Michaël Defferrard, and Taco Cohen. Codeit: Self-improving language models with prioritized hindsight replay. arXiv preprint arXiv:2402.048855 2024.
Murray Campbell, A Joseph Hoane Jr, and Feng-hsiung Hsu. Deep blue. Artificial intelligence, 134 (1-2):57-883 83, 2002.
Felix Chalumeau, Shikha Surana, Clément Bonnet, Nathan Grinsztajn, Arnu Pretorius, Alexandre Laterre, and Tom Barrett. Combinatorial optimization with policy adaptation using latent space search. Advances in Neural Information Processing Systems, 36:7947-7959, 2023.
Francois Chollet, Mike Knoop, Bryan Landers, Greg Kamradt, Hansueli Jud, Walter Reade, and wttee/ee......ooooooeeeeeeoeeeoo http://tiiinnsstrceoeoeeeeeaeeee prizzriee-ppeeeeeeee 2024
Addison Howard. Are prize 2024. https://kaggle, 2024. Kaggle.
François Chollet. On the measure of intelligence, 2019. URL 01547
Chris Cremer, Xuechen Li, and David Duvenaud. Inference suboptimality variational autoencoders. In International conference machine learning, pages 1078-1086. PMLR, 2018.
Jacob Devlin, Rudy Bunel, Rishabh Singh, Matthew Hausknecht, and Pushmeet Kohli. Neural program iacaainnduction Advances in Neural Information Processing Systems, 30, 2017a.

---

Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Pushmeet Kohli. Robustfill: Neural program learning under noisy i/o. In International conference on machine learning, pages 990-998. PMLR, 2017b.
Kevin Ellis, Catherine Wong, Maxwell Nye, Mathias Sablé-Meyer, Lucas Morales, Luke Hewitt, Luc Cary, Armando Solar-Lezama, and Joshua B Tenenbaum. Dreamcoder: Bootstrapping inductive program synthesis with wake-sleep library learning. In Proceedings of the 42nd acm sigplan international conference on programming language design and implementation, pages 835-850, 2021.
John K Feser, Marc Brockschmidt, Alexander I. Gaunt, and Daniel Tarlow. Differentiable functional program interpreters. arXiv preprint arXiv:1611.0190 2016.
John K Feser, Mare Brockschmidt, Alexander L Gaunt, and Daniel Tarlow. Neural functional
programming. 2017.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference machine learning, pages 1126-1135. PMLR, 2017.
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprini arXiv:1807.01662, 2018.
Alexander L Gaunt, Marc Brockschmidt, Nate Kushman, and Daniel Tarlow. Differentiable programs with neural libraries. In International Conference on Machine Learning, pages 1213- 1222. PMLR, 2017.
Gaël Gendron, Qiming Bao, Michael Witbrock, and Gillian Dobbie. Large language models are not strong abstract reasoners. arXiv preprint arXiv:2305. 230.i.9555..5 2023.
Samuel Gershman and Noah Goodman. Amortized inference in probabilistic reasoning. In Proceed- ings of the annual meeting of the cognitive science society, volume 36, 2014.
Rafael Gómez-Bombarelli, Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lobato, Benjamín Jänchez-Lengelinn Dennis Sheberla, Jorge Ryan P Adams, and Alán Aspuru-Guzik, Automatic chemical design using data-driven continuous representation of molecules. ACS central science, 4(2):268 276, 2018.
Timothy D Hirzel,
Alex Graves. Neural turing machines. arXiv preprint arXiv:1410.5401, 2014.
David Ha, Andrew Dai, and Quoe V Le. Hypernetworks. arXiv preprint arXiv: 2016.
Nikolaus Hansen and Andreas Ostermeier. Completely derandomized self-adaptation in evolution
strategies. Evolutionary Computation, 9(2):159-195, 2001. doi: 10. 11111/16666660175000000000 90398.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034, 2015.
Roee Hendel, Mor Geva, and Amir Globerson. In-context leaminn learning creates task vectors, arXiv preprint 10001:2310.159/6, Oct 2023. URL
15916
Michael Hodel. Addressing the abstraction and reasoning corpus via procedural example generation, 2024. htttttt//aaxxiii. www.aiiwi....aaaaaeoooooeoooeaee http://bb/2444.0000 07353
Kurt Horiik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks universal approximators. Neural networks, 255):359-366, 1989.
André Hottung, Bhanu Bhandari, and Kevin Tiemney. Learning latent search space for routing problems using variational autoencoders. In International Conference on Learning Representations, 2021a.
André Hottung, Yeong-Dae Kwon, and Kevin Tierney Efficient active search for combinatorial optimization problems. arXiv preprint 0rX1v:2106.05126, 2021b.
12

---

Jonas Hübotter, Sascha Bongni, Ido Hakimi, and Andreas Krause. Efficiently learning at test-time: Active fine-tuning of 1lms, arXiv preprint arXiv:2410.0800 080000000 2024.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko. et al. Highly accurate protein structure prediction with alphafold. nature, 59677873)5883--
2021.
Daniel Kahneman. Thinking, fast and slow. 1st ed. New York : Farrar, Straus and Giroux, [2011] 2011, URL URL.tttps /ttp:://searcde h/t.s/ce..cc.. library wisc www...du/ccc.............. etth//cetaallgg99910114499111111 In- cludes bibliographical references (pages 447-448) and index.
Eukasz Kaiser and Ilya Sutskever. Neural gpus learn algorithms. arXiv preprint arXiv: 2015.
Yoon Kim, Sam Wiseman, Andrew Miller, David Sontag, and Alexander Rush. Semi-amortized variational autoencoders. In International Conference Machine Learning, pages 26877. PMLR, 2018.
Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint arXiv: 1312.6131, 2013.
Victor Kolev, Bogdan Georgiev, and Svetlin Penkov. Neural abstract reasoner. arXiv preprint arXiv:2011.09860, 2020.
Andrei Nikolaevich Kolmogorov.
Three approaches to the definition of the concept "quantity of information.. Problemy peredachi informatsii, 1(1):-111, 1965.
Rahul Krishnan, Dawen Liang, and Matthew Hoffman. On the challenges of learning with inference networks on sparse, dimensiimnni data. In International conference on artificial intelligence and statistics, pages 143-151. PMLR, 2018.
Karol Kurach, Marcin Andrychowicz, and Iyya Sutskever. Neural random-access machines. arXiv preprint arXiv:1511.0633 2015.
Woosuk Lee, Kihong Heo, Rajeev Alur, and Mayur Naik. Accelerating search-based program synthesis using learned probabilistic models. ACM SIGPLAN Notices, 53(4):336 2018.
Shaohua Li, Xiuchao Sui, Jie Fu, Huazhu Fu, Xiangde Luo, Yangqin Feng, Xinxing Xu, Yong Liu, Daniel SW Ting, and Rick Siow Mong Goh. Few-shot domain adaptation with polymorphic transformers. In Medical Image Computing and Computer Assisted Intervention-MICC.. 2021: 24th International Conference, Strashourg. France, September 27-October 1, 2021, Proceedings, Part II 24, pages 330-340. Springer, 2021.
Wen-Ding Li, Keya Hu, Carter Larsen, Yuqing Wu, Simon Alford, Caleb Woo, Spencer M Dunn, Hao Tang, Michelangelo Naim, Dat Nguyen, et al. Combining induction and transduction for abstract reasoning. arXiv preprint 2rXiv:2411.02272..0 2024a.
Wenhao Li. Yudong Xu, Scott Sanner, and Elias Boutros Khalil. Tackling the abstraction and reasoning corpus with vision transformers: the importance of 2d representation, positions, and objects. arXiv preprint arXiv:2410.06400 2024b.
Wenhao Li, Yudong Xu, Scott Sanner, and Elias Boutros Khalil. Tackling the abstraction and reasoning corpus with vision transformers: the importance of 2d representation, positions, and objects, 2024c. httss:s:: http://aawii...rra///////// 06405
Yixuan Li, Julian Parsert, and Elizabeth Polgreen. Guiding enumerative program synthesis with large language models. In International Conference on Computer Aided Verification, pages 280-301. Springer, 2024d.
Zohar Manna and Richard Waldinger. A deductive approcch to program synthesis. ACM Transactions Programming Languages and Systems (TOPLAS), (1):900121, 1980.
Joe Marino, Yisong Yue, and Stephan Mandt. Iterative amortized inference. International Conference Machine Learning, pages 3403-3412. PMLR, 2018.
13

---

Kazem Meidani, Parshin Shojaee, Chandan K Reddy, and Amir Barati Farimani. Snip: Bridging math- ematical symbolic and numeric realms with unified pre-training. arXiv preprint arXiv:3310002227, 2023.
Suvir Mirchandani, Fei Xia, Pete Florence, Brian Ichter, Danny Driess, Montserrat Gonzalez Arenas, Kanishka Rao, Dorsa Sadigh, and Andy Zeng. Large language models as general pattern machines. arXiv preprint arXiv:2307.0477
2023.
Melanie Mitchell, Alessandro B Palmarini, and Arseny Moskvichev. Comparing humans, gpt-4, and gpt-4v on abstraction and reasoning tasks. arXiv preprint arXiv:2311.09244 0222277,,,, 2023.
Arvind Neelakantan, Quoc V Le, and llya Sutskever. Neural programmer: Inducing latent programs with gradient descent. arXiv preprint arXiv: 1,1444434 2015.
Emilio Parisotto, Abdel-rahman Mohamed, Rishabh Singh, Lihong Li, Dengyong Zhou, and Pushmeet Kohli. Neuro-symbolic program synthesis. arXiv preprint 1rXiv:1611.01830 2016.
Richard E Pattis. Karel the robot: gentle introduction the art of programming. John Wiley & Sons, 1994.
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osin- dero, and Raia Hadsell. Meta-learning with latent embedding optimization. arXiv preprint arXiv:1807.05960, 2018.
David E Shaw, William R Swartout, and C Cordell Green. Inferring lisp programs from examples. In IJCAI, volume 75, pages 260-267, 1975.
David Silver, Thomas Hubert, Julian Schritwiessrr, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, al. Mastering chess and shogi by self-play with general reinforcement learning algorithm. arXiv preprint 1rXiv:1712.01885. 2017.
Ray J Solomonoff. A formal theory of inductive inference, part Information and control, 7(1):1-22. 1964.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputiing 568: 127063, 2024.
Phillip D Summers. A methodology for lisp program construction from examples. Journal of the ACM (JACM), 24(1):161-175, 1977.
Dweep Trivedi, Jesse Zhang, Shao-Hua Sun, and Joseph J Lim. Learning to synthesize programs as interpretable and generalizable policies. Advances in neural information processing systems, 34: 25146-2511 25663, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung. David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. nature, 575(7782)3350-35 2019.
Evan Wang, Federico Cassano, Catherine Wu, Yunfeng Bai, Will Song, Vaskar Nath, Ziwen Han, Sean Hendryx, Summer Yue, and Hugh Zhang. Planning in natural language improves 11m search for code generation. arXiv preprint arXiv:2409.03333, 2024. URLhtttpa bubttps:/aante //arxiv..ivvv/// orgoorg///ao. /
2409. 03733
Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi DQ Bui, Junnan Li, and Steven CH Hoi. Codet5+: Open code large language models for code understanding and generation. arXiv preprint arX1v:2305.07922, 2023.
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie- Yan Liu. On layer normalization in the transformer architecture, 2020. https::s::/ //arxiv org/ wabb///002.04440
14

---

Liu Yang, Ziqian Lin, Kangwook Lee. Dimitris Papailiopoulos, and Robert D. Nowak. Task vectors in in-context learning: Emergence, formation, and benefits. arXiv preprint arXiv:2501002240, Jan 2024. URLpttpps: patttps///arivv org/ abss//00.00944
Han Yu, Jiashuo Liu, Xingxuan Zhang, Jiayun Wu, and Peng Cui. A survey on evaluation of distr--ditt generalization. arXiv preprint arXiv:2003001874, 2024a.
Peiyu Yu, Dinghuai Zhang, Hengzhi He, Xiaojian Ma, Ruiyao Miao, Yifan Lu, Yasi Zhang. Deqian Kong, Ruiqi Gao, Jianwen Xie, al. Latent energy-based odyssey: Black-box optimization via expanded exploration in the energy-based latent space. arXiv preprint arXiv:2405. 16730, 2024b.
Wojciech Zaremba and Ilya Sutskever. Learning to execute. arXiv preprint arXiv:1410.4611: 2014.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):007-115, 2021.
15

---

A Datasets
A.1 Pattern Task
The ARC-AGI challenge contains many diverse programs leveraging different knowledge priors. Injecting these priors into LPN by training the model master ARC-like tasks requires significant compute resources when training from scratch, without an LLM-based initialization. Therefore, investigate the training dynamics and properties of LPN before such a large-scale training, we develop a simpler task called Pattern task (see figure 4 within the same domain, but using a narrow distribution of pattern-lik programs. This specific task always generates fully-black 10x10 inputs with a single blue pixel at random location that defines where the output pastes 4x4 pattern sampled from uniform distribution. The pattern program-specific, meaning the same across different pairs but varies from specification to specification. This task demonstrates how deep learning methods without test-time computation may still make errors on such tasks. We then extend this task to study an out-of-distribution setting in section 5.5
Figure 4: Example of input (top row) and output (bottom row) pairs of specification sampled from the Pattern task. Each sample a batch of pairs that share the same pattern.
A.2 ARC-AGI
We evaluate LPN on the ARC-AGI benchmark by training it on the re-arc dataset [Hodel 2024] a dataset designed to be in distribution relative to the ARC training set, previously used for training large language models and ARC specific transformers to solve ARC-AGI et al. [202bb] Akyürek al. 2024 Most previous works however expand beyond this dataset, usually synthetically adding additional programs [Butt et al.) 2024). or crafted datasets. We restrict training to tasks from only this dataset with the goal of assessing generalization instead of closing performance gap with more data. This also ensures LPN solely learns from core knowledge priors without data leakage from the evaluation set. The evaluation set is known be significantly out of distribution and therefore represents a challenging generalization experiment. We train 178M 778-mmarmmtte LPN with a 256-dim latent space for 100k steps (batch size 256) for 2 days on TPU v4-32, see Section for further details, in terms of data, this amounts to 51M I/O pairs. Leveraging insights from smaller scale experiments we make use gradient in LPN during training. We train LPN Grad mode for 95k steps and then fine tune in Grad 1 for further 5k steps. We outline our hyperparameter procedure for both LPN and TTT for ARC-AGI in Section
16

---

B Expanded Experiments
B.1 LPN Training Extended
We provide an expanded version of Table inference axes.
! with additional ablations for both the training and
Inference
Training
Grad 0
Grad
Grad
Grad 20
Grad 100
Sample 250
Grad 0
(2.7)
(3.0)
18.8 (14.4)
52.5 (25.0)
67.5 (20.0)
3.2 (2.7)
Grad
8.6 (4.4)
44.6 (10.9)
85.4 (7.6)
98.4 (1.4)
99.5 (0.5)
10.2 (5.3)
Grad
(0.1)
13.7 (3.0)
(7.5)
88.9 (6.0)
94.1 (3.8)
0.7 (0.2)
Grad 5
0.0 (0.0)
0.4 (0.3)
31.9 (11.2)
88.5 (11.9)
98.1 (2.1)
0.5 (0.4)
Grad 5 **
(0.0)
0.0 (0.0)
9.9 (2.9)
87.1 (6.0)
95.1 (3.3)
0.0 (0.0)
Sample
6.1 (4.4)
8.2 (6.5)
27.7 (21.6)
56.3 (27.5)
72.2 (21.2)
6.1 (4.4)
Sample
10.8 (8.0)
13.3 (10.1)
39,9 (21.4)
72.3 (18.5)
87.9 (9.2)
10.8 (8.0)
Table Ablation study of LPN training and inference methods on the Pattern task. Rows represent training methods: LPN Grad [N] for N gradient steps, LPN Grad [N] with gradient flow through latent optimization (analogous to veta-learning), and Sample [N] for random samples. Columns represent inference methods: Grad [N] for N gradient ascent steps and Sample [N] for random search with N samples. Training was performed for 20k steps with seeds, with performance reported mean (standard deviation) over 3 runs. Bold values indicate the best performance per inference method.
B.2 Adapting Out-Of-Distribution
We provide expanded results for the We include results for varying levels OOD starting with in-distribution, weak OOD, and strong OOD.
(OOD) experiments outlined in Section 5.5
17

---

Inference
Training
Grad 0
Grad 10
Grad 100
In-Context
15.3 (0.6)
Test Time Training
15.3 (0.6)
(1.7)
0.78 (0.69)
LPN Grad 0
30.2 (15.7)
72.8 (29.5)
82.2 (21.3)
LPN Grad
26.6 (7.4)
98.0 (0.6)
99.2 (0.6)
LPN Grad 2
14.4 (3.5)
97.2 (1.6)
98.9 (1.2)
LPN Grad 3
1.0 (0.7)
85.7 (6.3)
98.3 (1.9)
LPN Grad "*
10.6 (6.4)
93.8 (5.1)
97.4 (1.9)
In-distribution
Inference Grad 10
Training
Grad 0
Grad 100
In Contett
(0.9)
Test Time Training
(0.9)
2.1 (1.1)
0.00 (0.00)
LPN Grad
7.6 (5.6)
(35.7)
62.8 (38.3)
LPN Grad
7.4 (4.4)
93.1 (4.3)
97.7 (2.2)
LPN Grad
3.0 (2.6)
86.1 (6.0)
95.9 (2.1)
I.NN Grad
0.0 (0.0)
55.0 (7.8)
93.9 (3.8)
Grad
1.3 (1.0)
81.7 (13.3)
91.5 (5.5)
Weakly out-of distriiuttiii
Inference
Training
Grad 0
Grad
Grad
In Do-Context
0.0 (0.0)
Test Time Training
0.0 (0.0)
1.8 (0.7)
0,3 (0.1)
LPN Grad
0.3 (0.5)
18.8 (14.5)
41.1 (29.6)
LPN Grad
0.0 (0.0)
59.9 (11.6)
88.0 (5.3)
LPN Grad 2
0.0 (0.0)
(13.0)
81.8 (10.9)
LPN Grad 3
0.0 (0.0)
11.3 (9.3)
72.0 (14.0)
LPN Grad A*
0.0 (0.0)
40.9 (19.8)
71.1 (14.3)
Strongly
Table 6: Study of the out-of-distribution (OOD) performance on the Pattern task. Models are trained on patterns that have density of 50% (half black. half colored), then evaluated on the same distribution, on a density of 75% (weakly OOD) and 100% (strongly OOD), Performance is averaged training runs with different seeds, with standard deviation parentheses.
18

---

B.3 Validating the Decoder
Training deep networks from scratch solve ARC-like tasks has been challenging Li al. 2024c If it is the case that neural networks struggle even to learn to execute single programs this represents significant bottleneck training models from scratch on broad distribution of programs. Therefore, before training LPN end-to-end, we conclusively show that our decoder architecture does not suffer from such a bottleneck, and can learn individual programs.
We show of the 400 total tasks from the ARC-AGI training set, and for each of these tasks, we train a small LPN architecture of 800k parameters (except for the last task which required bigger model with 8.7M parameters) on the corresponding task generator from re-arc [Hodel 2024). Specifically, we select the first five tasks from the arc-agi_ swraiiing_challee file (007bbfb7, 00d62c1b, 017c7c7b, 025d127b, 045e512c) shown in figure Figure S
json
+
007bbfb7
00d62c1b
017c7c7b
045e512c
025d127b
Figure 5: Overfit training the first 5 ARC-AGI training tasks. The captions correspond to task IDs. For each task, the top row contains the input grids, and the bottom row the output grids. Each task consists of observing all pairs but the first and inferring the output of the leftmost pair given its input. Each curve corresponds to a separate training run.
We evaluate both the distribution of re-arc generated tasks on which it is trained and on the true task from the ARC-AGI training set in figure Figure We show that for each task, the small LPN decoder-only model successfully learns individual programs and manages to solve the corresponding ARC-AGI task. Therefore, model outperforms previously reported results in Li et al. [2024c), and concludes that our architecture does not suffer from decoder bottleneck. Note that the encoder is not helpful in this experiment since the task is always the same. Our later results on ARC-AGI Section 5.7 take this step further and show that we can learn a single transformer architecture capable of executing over 270 programs in the ARC training dataset.
Training Loss
Pixel Accuracy
1.0
Task ID
Task
re-arc
ARC-AGI
0.8
00462c1b
007bbεb7
100
100
017c7c7b
Loss
025d127b
00d62c1b
96.5
100
045c512c
017c7c7b
99.7
100
0254127b
100
100
0.0
045e512c
98.3
100
10000
Training Steps
(a) Training loss
(b) Performance after 10k steps of training
Figure 6: Training loss and performance of LPN training on 5 of the re-arc distributions. For each task, only samples from the re-arc generators are used for training. The corresponding ARC-AGI tasks are never seen during training.
19

---

B.4 Analyzing the Latent Space
Figure 7: (Left) A 2D pattern task with inputs containing marker points where patterns should be placed, with patterns varying for each program. (Right) The latent traversal visualizes the effect of traversing the latent space, on the predicted pattern by the decoder marker points.
To validate that the encoder is learning programs in its latent space, design with small 4x4 grids that have 2x2 patterns. We train small L.PN model until convergence with latent space of dimension 2 to easily visualize it in figure Figure 7 Due the simplicity of the task we train the model with mean training, i.c. no latent optimization. Because we are using a 2D Gaussian prior for the latent space, we can convert R² to the unit square using the normal cumulative distribution function (CDF), and then plot on the unit square at coordinates (æ, y) the decoder's output when conditioned by the latent CDF(=) (xr, v). or equivalently, percent- point function (PPF). These results demonstrate the diversity and smoothness of the latent space, showing structure terms color patterns, which motivates performing gradient ascent latent optimization in more complex tasks. This shows that the latent space can encode wide range of diversity its latent space which is especially important for adapting to unseen patterns.
simpler task
(PPF(x:), PPF(y)) using the
The figures the left show random initialization in the latent space and corresponding latent trajectory following the gradient as- cent algorithm. The log-likelihood for each latent is shown during op- timization. The decoded pattern traced until convergence in the la- tent space, giving insights into how LPN performs latent optimization at test time find the optimal latent program.
12
20

---

B.5 Scaling Specification Size
1.0
0.8
Accorecy
0.6
0.6
0.4
LN good
0.4
grod
LNN god 10
grad tos TTT gred 400
grde 100
0.2
02
0.0
0.0
13 Spocification Size
16
17
10 12
12
13
14
37
18
Spocification Size
Figure 8: Accuracy of Latent Program Network (LPN) and Test-Time Training (TTT) models as we scale specification size. Higher specification sizes improve accuracy across different training methods. The dashed vertical line at specification size 3 indicates the fixed size used during training.
To analyze the effect of scaling specification size, we conduct an ablation using models trained on the pattern task for single seed. We compare both in-context learning and LPN. We then evaluate different inference strategies on the in-distribution dataset and analyze their performance as the test specification size increases.
Our results indicate that LPN generalizes effectively across varying specification sizes. This robust- ness is primarily due the use of mean pooling, which prevents overfitting to specific context lengths and enables the model to gracefully handle both slightly larger and significantly larger specifica- tion sizes. Furthermore, structurally similar, improving generalization.
pooling encourages representations of the same program to remain
In contrast, in-context learning exhibits strong overfitting to the training specification size, with noticeable drop in performance as specification size deviates from the training distribution. How- we observe that fine-tuning-basee inference can scale effectively with increasing specification sizes, provided that sufficient amount of data available. The effectiveness of fine-tuning highly dependent the choice of hyperparameters, requiring substantial amounts of data to maintain strong performance as the specification size grows.
B.5.1 Performance Across Training Sizes
To further explore the impact of training specification size model performance, present results for training sizes 7, and 11, evaluated across test sizes 1, 3, 7, 11, 15, and 19 on the pattern task. The following tables report accuracy for LPN with and without gradient ascent, In-Context Learning, and Test-Time Training (TTT). Notably, In-context learning consistently overfits to specific test sizes that align closely with the training size. In contrast, LPN, particularly with 100 gradient ascent steps, demonstrates robust generalization, maintaining high accuracy across all test sizes and training sizes. TTT shows improved performance test sizes increase, approaching L.NN's accuracy at larger specification sizes when also trained on large specification sizes.
Method
Test Size
15
LPN grad 0
38,9
44.1
47.5
44.7
48.8
45.9
LPN grad 100
89.8
92.1
94.1
94.1
95.3
96.3
In-Context
2.1
0.0
0.0
0.0
0.0
0.0
TTT grad 100
7.8
19.9
50.9
72.4
81.8
90.4
Table Performance (top-2 accuracy, percentage) for training size on the pattern task.
21

---

Method
Test Size
11
15
19
LPN grad 0
14.6
27.7
32.8
33.4
33.2
35.1
LPN grad 100
92.7
91.0
92.6
93.6
In-Context
0.0
0.0
78.3
0.0
0.0
0.0
TTT grad 100
11.3
34.7
77.9
90.6
94.7
93.2
Table Performance (top-2 accuracy, percentage) for training size 7 the pattern task.
Method
Test Size
11
15
19
LPN grad 0
11.1
26.6
40.8
43.2
50.0
48.8
LPN grad 100
96.9
98.0
97.9
In-Context
0.0
0.0
0.1
94.1
0.1
0.0
TTT grad
0.1
16.2
56.1
78.3
93.7
92.2
Table 9: Performance (top-2 accuracy, percentage) for training size 11 on the pattern task.
22

---

B.6 Measuring Floating Point Operations
To evaluate the computational efficiency different methods, we measure the number of floating- point operations (FLOPs) required for performing inference on a single task. The reported FLOP measurements correspond the cost of generating an entire 30x30 grid given input-output pairs (specification size).
While gradient-based approaches do not require additional computational budget in terms of trial attempts, their inference mechanisms involve performing gradient updates, either in parameter space or latent space, which adds computational overhead. A key distinction between TTT and LPN is their computational cost: LPN updates only the latent space by backpropagating through the decoder, whereas TTT requires backpropagation through all model parameters. This makes TTT significantly more expensive and less efficient for real-time applications.
10
1014
FLOPs
UN grad
In-tonted
LPN Sampling Uk grad TTT grad
Sampling 10
grad 10 grad 10
Lay Sampting too LPN grad 1n0 TTT grod 180
Figure 9: Floating-point operations (FLOPs) required for test-time inference across different methods.
The LPN Sampling results can be understood as ablation of LPN without gradient- based search, but only sampling multiple latents from the encoder computing the decoder likelihood for each, avoiding the computational cost backpropagating through the decoder. However, this efficiency gain comes at significant cost in performance, shown in the pattern task results Table
23

---

B.7 Sequence Ablation
B.7.1 Dataset
To investigate the generalizability of results beyond environments with significant spatial structure. we perform an ablation synthetic sequence task and replicate the analysis previously conducted on the pattern dataset. This synthetic dataset features vast program space, with over 100 million unique programs, each defined by composing 3 to 5 parameterized rules that transform sequences of numbers (ranging from 4). Programs are composed of to rules with specific integer-based parameters (e.g., thresholds or operation values). These rules are then applied in their chosen order: for each rule, we process the sequence from left to right, transforming each position according to the rule's 's condition and operation, producing intermediate sequence that serves as input to the next rule. This sequential, rule-by-rule application, with each rule scanning left to right, enables complex transformations and vast numbers of programs with meaningfully different outputs. The core rules are defined as follows:
a number greater than its right neighbor by a threshold k; decrease by value m, modulo 5.
If' a number has identical neighbors on both sides, multiply its value by factor n and clip the result to 4.
If number is greater than its left neighbor by threshold k add value m, modulo 5.
If number is less than its right neighbor by threshold k, subtract a value m, modulo 5.
Replace the number with function of its neighbors (c.g., sum, average, maximum, or minimum), modulo 5.
These parameterized rules may interact at the same position across their sequential applications, creating cascading effects that result in complex transformations. While single input-output pair may uniquely identify the underlying program, multiple pairs typically provide sufficient information to determine it. The table below presents accuracy metrics for models evaluated on this task.
24

---

B.8 LPN fine tuning
We also ablate LPN training by training in full Grad 0 mode for 100k steps. We show the results compared to fine tuning for 5k steps in Grad 1 mode.
FLOPs
LPN Grad Tune
LPN Grad Tune
2E+11
7.75
8.25
2E+12
10.25
10.25
13+13
15.25
13.60
2E+14
15.50
15.10
2F+15
15.50
15.10
Table 10: Performance of L.PN with and without gradient tuning on ARC-AGI for 10k steps
We find marginal performance gain at higher computational budgets from fine-tuning with one gradient step, so we adopt this approach in the main method. However, we note slight perfor- mance drop in Grad 0 inference, which is expected as the network begins to optimize with gradient adaptation.
25

---

B.9 ARC-AGI Solution Analysis
In this section we investigate whether there are differences between the types tasks in ARC-AGI evaluation dataset that LPN solves vs Test-Time Training. We run both methods at the budget of 2E+14 and analyze the problems solved. We first analyze the overlap between problems solved between the two methods. We see that there is spread of the different problems being solved by the different architectures, a roughly even split between problems only solved by TTT, problems solved only by LPN and problems solved by both.
35
30
Number of tasks
25
20
15
10
5
0
TTT only
LPN only
Both LPN & TTT
Figure 10: Bar Chart showing Problems Uniquely solved by LPN and TTT and tasks solved when at least solves the problem
If we measure the accuracy when either LPN or TTT solves an ARC puzzle. We achieve a combined score of 22%. We also show examples of problems solved only by LPN and examples only solved by Test- Time Training to understand the differences between the problems each method solved.
26

---

Train Input 1
Train Input
Train Input 3
Train Output 1
Train Output
Train Output 3
Test Input
Test Output (Solution)
Figure 11: LPN Example
27

---

Train Input
Train Input
Train Input
Train Output
Train Output 3
Train Output
Test Input
Test Output (Solution)
Figure 12: LPN Example 2
28

---

Train Input 1
Train Input 2
Train Input 3
Train Output 1
Train Output 2
Train Output
Test Input
Test Output (Solution)
Figure 13: LPN Example 3
29

---

Train Input 1
Train Input 2
Train Output 1
Train Output 2
Test Input
Test Output (Solution)
Figure 14: TTT Example 1
30

---

Train Input 1
Train Input 2
Train Input
Train Output 1
Train Output 2
Train Output 3
Test Input
Test Output (Solution)
Figure 15: TTT Example 2
31

---

Train Input 1
Train Input
Train Input
Train Output
Train Output
Train Output
Test Input
Test Output (Solution)
Figure 16: TTT Example 3
32

---

B.10 Investigating Composition
To evaluate the generalization capabilities of LPN novel compositions of learned primitives, we construct simple test cases by combining two distinct operations from the training set that the model has only encountered separately during training. This setup probes whether LPN can flexibly integrate these primitives to produce the desired composite effect, a key indicator of its ability to handle systematic generalization beyond rote memorization.
train input
train output 1
train input
train output 2
train input
train output 3
train input 4
train output 4
test input
test output grad: 0 steps
test output grad 1600 steps
Figure 17: Testing LPN on the composition of two operations previously only seen separately,
In Figure we examine specific instance where the first operation extracts the pattern within bounding box. while the second replaces all black pixels with pink. The direct output of the
33

---

encoder reveals overfitting to isolated training patterns: it correctly extracts and the pattern within the bounding box, but fails convert black pixels black. However, by performing gradient ascent in the latent space to refine the encoding, we can mitigate these errors. This optimization aligns the latent representation more closely with the composite objective, enabling the decoder to accurately render the full pink conversion across the extracted pattern. This demonstrates LPN's potential for compositional reasoning, where latent-space adjustments unlock emergent behaviors not explicitly trained for.
train input
train output 1
train input
train output 2
input
output 3
train input
train output 4
test output grad: steps
test output grad 1600 steps
test input
Figure 18: Testing LPN on the composition of three operations.
In Figure 18 we examine the combination of three primitives: a gravity-type operation, fill operation, and the use of checkerboard pattern for the fill. While LPN can directly predict the
34

---

gravity and fill components, it overfits to the fill operations seen during training. As a result, even with gradient-based search, it is not able resolve this error and apply the checkerboard pattern instead.
35

---

B.11 LPN ARC-AGI Inference Ablations
To dissect the contributions of key components to LPN's performance on ARC-AGI, we performed ablations on various test-time inference strategies. These experiments isolate the roles of the program prior and gradient-based search, using the ARC-AGI evaluation (out-of off--siiiiiitttionn relative to Re-ARC). We fix the number of sampled latents during inference and vary the search budgets across {10, 50, 100, 200, 400) steps, evaluating on a single seed.
We compare the following inference strategies:
Encoder Sampling. Samples multiple latents from the encoder distribution, evaluates the specification likelihood for each, and selects the highest-scoring one. This assesses structured gradient-basee based search against brute-force sampling near the encoder output.
Gaussian Init Gradient Search. Employs the core LPN gradient-based search but initializes from sample drawn from the Gaussian prior. This isolates the encoder's contributions in providing strong initializations and enhancing overall performance.
Encoder Init Local Search. Initializes from the encoder output but substitutes gradient updates with a mutation operator (continuous noise perturbations). This evaluates gradient- based optimization against unstructured local search, while permitting exploration beyond the encoder distribution.
Gaussian Init + Local Search. Pairs prior-based initialization with local search. This baseline quantifies the joint necessity of encoder-guided initialization and gradient direction.
Encoder Init + Gradient Search (LPN). The full LPN inference procedure.
Table 11 reports the performance (in %) across these strategies and budgets.
Table 11: Ablation results (%) on ARC-AGI evaluation for inference strategies under varying search
budgets.
Inference Method
10
50
100
200
400
Encoder Sampling
9.12%
10.75%
10.13%
10.50%
10.62%
Gaussian Init + Gradient Search
1.38%
2.75%
5.37%
6.88%
10.75%
Encoder Init Local Search
8.38%
9.50%
10.37%
10.75%
10.75%
Gaussian Init Local Search
0.25%
0.25%
0.25%
0.75%
0.25%
Encoder Init + Gradient Search (LPN)
9.10%
13.25% %
15.00%
15.50%
15.50 %
These results reveal that repeated encoder sampling yields only marginal gains over a single sample (7.75%), underscoring the limitations of brute-force approaches. The gap between Gaussian Init Gradient Search and Encoder Init + Gradient Search emphasizes the encoder's dual role: seeding effective starting points for low budgets and amplifying gains high budgets (c.g.. +4.75% 400 steps). Encoder Init + Local Search matches gradient search at low budgets (10 steps) but plateaus thereafter, highlighting gradient guidance's superiority for deeper exploration. Gaussian Init Local Search yields near-zero performance, confirming the indispensability of encoder initialization paired with directed optimization. In aggregate, Encoder Init + Gradient Search (LPN) dominates all baselines, validating the incoder-ifooman initialization and gradient-based refinement as pivotal to LPN's s efficacy on ARC-AGI.
36

---

C LPN Algorithm
Below we outline two algorithms: first, LPN test-time inference (Algorithm TD and its mechanism for performing inductive inference. Second, we provide the full algorithm for LPN during training (Algorithm 2
Algorithm LPN Test-Time Inference with Gradient Ascent Latent Optimization
Require: n input-output pairs (ay, v:), test input In+1, number of gradient steps K
1: for i= = -...,,,1 . do
Can be done in parallel
2: Sample 2, ~ 20(2|x5; Vi)
3: end for
4: Initialize latent t -==== 2
5: for 1...... K do
Perform gradient ascent
6: z' z' a V Ei= log ( =>
7: end for
8: return Vn+1 ~ (0 (211,
Algorithm 2 LPN Training with Gradient Ascent Latent Optimization
Require: Encoder parameters o, decoder parameters 0
1: for wort=1. 1,.., num wum_ttumiiinnn traininggsteeeeee do
2: Sample input-output pairs (æs; ys) from the same program
3: for 1,..., do
>> Can be done in parallel Using the reparameterization trick
4:
Sample 2; go (2/20,, 3t) 5: end for
6: for i
do 2/ + n-1 Σ 2j
Can be done in parallel
7:
jti
8:
for I,.... do
b Perform gradient ascent the latent space log po [x,, =)(== Optional stop-grad on the 2nd term
9:
2 244 a V. -
8#1
10:
end for
11:
L; log po (y) (,, z/) (ss ((),
12:
end for
C 1 Σ C,
Total loss for all pairs
14:
Update Φ and 0 via gradient descent on C
end for
37

---

D
Variational Inference
Latent Program Networks (LPNs), when operated in grad mode, perform form of variational inference. Updating the latent representation using gradients from the encoder constitutes semi- amortized variational inference [Kim
2018 Marino al. 2018
In prior work, such as LEAPs Trivedi et al. 2021]. it is assumed during training that the full representation of program f observable. In this setting, variational inference on programs be conducted using a standard Variational Autoencoder (VAE) restricted to the program space. LEAPs introduce form of variational inference via Execution Evidence Lower Bound (ELBO), where function reconstruction is based on correctly executing the function for given input rather than reconstructing the full program representation. The Execution ELBO is defined as:
(=I))(=I Bx-p((a) Pog f(z) x =]]] (L(g)(=If) p(z))
This formulation is adnatageous because it enables learning a function that directly executes the program without requiring explicit reconstruction of the program followed by execution. In LPNs, this is critical, as the differentiable parameterization of the function executor allows backpropagation through the executor to perform program search in the latent space.
In this work, we assume that the functions generating the underlying input-output pairs are not fully observable, aligning with real-world scenarios where the data-generating functions are rarely fully known. Instead, we observe only partial data for each function, represented as a dataset Xa = {II,,d )t,a)) J where each dataset corresponds to set of input-ou(put pairs generated by a particular unseen function. Across D such datasets, the following objective serves practical approximation to the Execution ELBO:
D
Na E (=)x(())Xa) [log po (Br.a Li.d, 2)] - (L(gs(2(Xa) p(2)),
where D is the number of datasets, each representing input-output pairs from distinct unseen function, and Na is the number of samples in dataset d. However, this standard objective is flawed because the encoder (= (=)Xa) can "cheat" by encoding specific details of the pair (Ia V),a) it needs to reconstruct, leading to memorization rather than capturing the general function f. To address this, we propose the Leave-One One-Out (LOO) objective:
ΣΣ
Ex-go(=)X [log (o Ti,d 2)] (L(go(:|Xa) p(z)).
where X-idd = Xa (( Vi.a)) denotes the dataset Xa excluding the i-th input-output pair. This objective prevents memorization by denying the encoder access to (Ia,d> Vi.a) when producing the latent representation x used for its reconstruction. Consequently, 9s must infer the underlying function from the remaining data X-1,d, making the LOO objective a better functional approximation of the Execution ELBO's intent by directly promoting generalization. Training optimizes the encoder (x(2)Xa) and decoder Po x, 2) across multiple datasets, resulting in amortized inference model ds that efficiently proposes an approximate posterior for any given dataset X
For optimal performance a specific test dataset X we employ semi-amortized variational inference. First, the trained amortized encoder (=]= rapidly generates an initial latent rep- resentation zo. Then, with the encoder parameters and decoder parameters 0 fixed, we perform instance-specifer optimization starting from 2p to find a refined latent representation =". This is achieved by directly maximizing the ELBO via latent This combination of an efficient amortized proposal and instance-specific ELBO refinement yields a superior latent program representation 2* tailored to the test problem. This procedure, known semi-amortized variational inference, balances computational efficiency with high-quality inference.
38

---

E Hyperparameters
In this section, we outline our approach to hyperparameter search and provide full documentation of all the hyperparameters used in all reported experiments.
Hy Hyperparameter Search To ensure a fair comparison between LPN and in-context learning, which share the same core architecture for embedding inputs and generating outputs, we kept all architectural parameters identical across methods. We conducted hyperparameter testing to determine whether to use rotational embeddings. We performed testing by repeating the decoder validation experiment with and without rotational embeddings. Rotational embeddings improved performance across all individual tasks and so was selected to be used in both the encoder and decoder of LPN.
For the ARC-AGI results, we performed hyperparameter search over the learning rate for test time adaptation for both test-time tuning and LPN. Since these methods operate in different spaces, they are unlikely to require similar parameters, making it fairer to tune this parameter independently for each baseline. We validated on held-out test set of unseen problems from the RE-ARC dataset. We searched over learning rates ranging from 10 to 10 7, Performance was evaluated by measuring average accuracy across five FLOP measures (ranging from 2E+11 to 2E+15). For the baseline pattern task Table [ we choose a learning rate of 0.1 without performing hyperparameter optimisation as the experiment is simply to understand the behavior of LPN and not to optimize performance. For pattern OOD task we perform grid search for LPN and TTT over a dataset of from the strongly OOD task for 10 gradient steps, filtering out learning rates that do not decrease the test-time loss by at least 1.0%. We use these learning rates for the scaling specification size ablation also.
Validating the Decoder In section B.3 we train an LPN model on each of the re- are generators corresponding the first tasks from the training set. For each task, we train the model for 10k gradient steps with a batch size of 128 and 4 pairs per specification, resulting in 5, 5,120,000 input- output pairs. We gather all hyperparameters table common to all the tasks except 045e512c which had 4 encoder layers, 8 heads, 32 embedding dimensions per head, an MLP factor of 2.0, for total of 8.7M parameters.
Component
Hyperparameter
Value
Number of Layers
Number of Heads
Encoder Transformer
Embedding Dimension per Head
16
Latent Dimension
32
RoPE
False
Number of Layers
Number of Heads
Decoder Transformer
Embedding Dimension per Head
16
MLP Dimension Factor RoPE
1.0
False
Number of Parameters
829k
Training Steps
10k
Training
Batch Size
128
Optimizer
Adam
Gradient Clipping Norm
1.0
Learning Rate
4e-4
Number of Rows & Columns
30, 30
Table 12: Hyperparameters for the experiments from section B.3
validating the decoder.
39

---

Pattern Task In section 5.2 we train an LPN model on 10x10 task with 4x4 patterns. We train each method (mean, gradient ascent, etc) for total of 20k steps with batch size of 128 and pairs specification, resulting in total of 10M input-output pairs.
Component
Hyperparameter
Value
Number of Layers
Number of Heads
Encoder Transformer
Embedding Dimension per Head
16
MLP Dimension Factor
1.0
Latent Dimension
32
RoPE
False
Number of Layers
Number of Heads
Decoder Transformer
Embedding Dimension per Head
16
MLP Dimension Factor RoPE
1.0
False
Number of Parameters
973k
Training Steps
20k
Training
Batch Size
128
Prior KL Coeff
lc-4
Optimizer
Adam W
Gradient Clipping Norm
Learning Rate
4e-4
Number of Rows Columns
10, 10
TTT Learning Rate
le-4
LPN Learning Rate Optimizer
le-1
Testing
Adam
Table 13: Hyperparameters for the experiments from section 5.2 i.c. the pattern task.
40

---

Analyzing the Latent Space In section B.4 we train small LPN model on reduced version of the Pattern task with grids of size 4x4 and patterns of size 2x2. We used the following hyperparameters for training in table 14
Component
Hyperparameter
Value
Number of Layers
Number of Heads
Encoder Transformer
Embedding Dimension per Head
12
MLP Dimension Factor
4.0
Latent Dimension
RoPE
False
Number of Layers
Number of Heads
Decoder Transformer
Embedding Dimension per Head
12
MLP Dimension Factor RoPE
4.0
False
Number of Parameters
1M
Training Steps
200k
Training
Batch Size
128
Prior KL Coeff
le-3
Optimizer
Adam W
Gradient Clipping Norm
Learning Rate
4e-4
Number of Rows Columns
TTT Learning Rate
le-4
LPN Learning Rate Optimizer
le-1
Testing
Adam
Table 14: Hyperparameters for the experiment in section B.4 i.c. analyzing the latent space.
41

---

Out-Of-Distribution In section 5.5 we train LPN models similar to those above in the Pattern task
and evaluate them on different distributions. We gather hyperparameters used for training in table 13
Component
Hyperparameter
Value
Number of Layers
4
Number of Heads
Encoder Transformer
Embedding Dimension per Head
MLP Dimension Factor
2.0
Latent Dimension RoPE
32
False
Number of Layers
Number of Heads
Decoder Transformer
Embedding Dimension per Head
MLP Dimension Factor RoPE
1.0
False
Number of Parameters
1M
Training Steps
100k
Training
Batch Size
128
Prior KL Coeff
le-4
Optimizer
Adam W
Gradient Clipping Norm
1.0
Learning Rate
4e-4
Number of Rows & Columns
10, 10
TTT Learning Rate
le-5
LPN Learning Rate Optimizer
le-1
Testing
Adam
Table 15: Hyperparameters for the experiments from section 5.5 i.c. the study of out-of-distribution
performance of LPN the Pattern task.
42

---

ARC-AGI
In table 16
we finally present the hyperparameters used for experiments on ARC-AGI
(section 5.7
Component
Hyperparameter
Value
Number of Layers
Number of Heads
Encoder Transformer
Embedding Dimension per Head
64
MLP Dimension Factor
4.0
Latent Dimension RoPE
256
True
RoPE max freq
10
Number of Layers
Number of Heads
Decoder Transformer
Embedding Dimension per Head
64
ML. Dimension Factor RoPE
4.0
True
RoPE max freq
10
Number of Parameters
178M
Training Steps
100k
Training
Batch Size
256
Prior KL Coeff
le-4
Optimizer
Adam W
Gradient Clipping Norm
1.0
Learning Rate
3c-4
Number of Rows & Columns
30, 30
TTT Learning Rate
lc-4
LPN Learning Rate
5e-2
Testing
Optimizer
Adam
Table 16: Hyperparameters for the experiment section 5.7 i.e. training LPN to solve the ARC-AGI benchmark.
43

---

Additional Charts
Latent Program Embeddings
40
43
1-5NE
Figure 19: T-SNE visualization of the latent space of input-output pairs sampled from the re-arc generators. We see strong evidence that the latent embeddings encode information about programs with significant clustering in the latent space for the same programs across different input output pairs.
44

---

F.2 Decoder Gradient Field
Figure 20 visualizes the likelihood landscape of decoding the correct output conditioned on the given input for a single pattern task, while varying the latent input. The gradient contours overlaid on the plot illustrate the optimization dynamics when performing gradient- based updates in this space. The trajectories depict how gradient ascent seeks to maximize the decoding likelihood, revealing the structure of the landscape. Certain regions form basins of attraction that lead to valid solutions, while others correspond to local optima where the likelihood of decoding the correct output stagnates. This visualization highlights both the feasibility of optimizing the latent representation and the potential challenges of escaping suboptimal regions in the latent space.
-10
-15
20
25
-35
Figure 20: Visualization of the likelihood landscape for decoding the correct output conditioned on the input, as a function of the latent space.
45

---

G
Architecture
In all our experiments, programs are defined the input-output space of ARC-AGI, i.e., 2D grids whose cells can take 10 different values and have shapes (n,, m) with n, E [1,30]. We implement both the encoder and decoder as small transformers [Vaswani et al. 2017] specifically designed for this benchmark, in contrast to the more general large language models (LLMs) typically used [aang et al. 2023
Columns
Padding
Columns
Padding tokens
3
4
3
4
Input
Output
Encoder
Input
Output
3x4
3x4
Latent
Decoder
4x3
4x3
3
4
Latent
Rows
Columns
Padding tokens
Rows Columns
Padding tokens
LPN architecture for ARC-AGL. Both the encoder and decoder small transformers that take flattened padded grids as inputs. The actual number of rows and columns is prefixed to each sequence.
Figure 21:
We model the input and output images as 2D grids, which we pad and flatten in a raster-scan fashion form sequences of pixel values, each of size 30 30 900 (see Figure 21). Each grid sequence prefixed with shape information, namely two extra values for the number of rows and columns, resulting in sequences of 902 values. For both the Encoder and Decoder grid positions encoded using RoPE
[Su et al. 2024 for both row and column indices.
G.1 Encoder
The encoder processes both the input and output grids from a given pair and returns distribution of inferred program latents underlying the task. Specifically, it outputs the mean and diagonal covariance of multivariate normal distribution from which latents can be sampled. Each grid sequence contains 902 values, and we add an extra CLS token for the output embedding, resulting in total sequence length of 1 1805 for the encoder transformer. The encoder is implemented standard transformer [Vaswani et al. 2017) with pre-layer normalization [Baevski and Auli 20188) Xiong al. 2020) and muad attention. To incorporate identify the input from the output add an embedding emb(c), c E {0, 1} is the channel index (0 for input, for output). All 1800 color values (0 9), four shape values (1 to 30), and the CLS token separately embedded into R# using lookup tables. Padded tokens, determined by the shape values, are masked, and the sequence is fed into multiple transformer blocks, attention mask is non-causal, allowing all non condepdded tokens to attend to each other during encoding. The CLS embedding is passed through layer normalization and parallel dense layers output the mean and diagonal log- log-covariance of the multivariate normal distribution over latents. Sampled program latents have dimension d, which may differ from the embedding dimension H.
Section E for hyperparameter details. In the encoder the
46

---

G.2 Decoder
The decoder takes an input grid and latent program and autoregressively generates an output grid. Its design is similar to the encoder, with key differences. First, the flattened sequence is prefixed with a projection of the latent embedding. Since the decoder generates the output autoregressively. the attention mask is causal on the output grid portion of the sequence (the second half). The attention mask also dynamically accounts for padding tokens based on the predicted output shapes. The sequence embeddings corresponding to the output are extracted and projected to either shape logits for the first two embeddings or grid logits for the 900 output grid embeddings. Each output token embedding maps to logits for the next token in raster-scan fashion. However, due to padding at each row, the last embedding of each row mapped to the first token the next row.
47

---

H
Baselines
H.1 Transductive Baseline
We compare LPN against transductive bascline that directly conditions on the specification without explicitly constructing latent program representation. This approach, similar to Kolev al. (2020).
Li ct al. 2024a], processes the specification by encoding and concatenating each input-ouiput pair separately.
Encoder:
The encoder maps each input-output pair to an encoding vector:
2, = c c, 9i) Vi E [1,
where Eg is neural network parameterized by that processes individual input-output pairs.
Concatenation: Unlike LPN which searches for single latent program, the transductive baseline uses each encoding as token embedding in the input sequence to the transformer. Specifically, the encodings are concatenated as:
[=1; .2: zn]
(10)
where |;] denotes sequence concatenation and each z; serves as token embedding in the transformer's input sequence.
Decoder: The transformer decoder processes this sequence embeddings along with the new input to generate the output:
in+1 ~ pe (yl An+++ Zeat)
(11)
where the decoder attends both the new input Ix+1 and the sequence of specification embeddings Zeat.
By concatenating per-pair embeddings, allowing the decoder to access information from all grids during inference. Processing input-output pairs independently in the encoder serves strong prior for capturing high-level program features, reducing the risk of learning spurious correlations between pixels of different pairs. In contrast, methods that process all pairs jointly increase the encoder's computational demands. Additionally, feeding all raw pairs directly to the decoder would complicate positional encodings, requiring simultancous modeling of both within- Sathin-pairraa and across-pair positions. Our approach mitigates this by encoding positional information separately within each pair at the encoder stage. The decoder then receives each pair's embedding in a distinct position within the concatenated sequence, enabling clear differentiation between examples.
joint representation of all input-output pairs,
H.2 Test-Time Fine-Tuning
We implement the following test-time parameter tuning approach where the transductive model's parameters are fine-tuned on the specification itself. Given a specification of n input-output pairs, we perform gradient updates the model parameters to better predict each output given its input and the remaining pairs.
Update Process: Starting from the pre-trained parameters 0 and (decoder and encoder respectivll)) for each pair (z,, y4) in the specification, we compute the loss:
Lim (0, 0) - log po (yi Mi, >m))
(12)
where أशाी = (11 B..;... . (2 11-1): es (2141, 1141): (a vn)] represents the con- catenated embeddings of all pairs except the i-th.
We then update the parameters using gradient descent:
= aV..
(13)
-
V
Σ Cim
(14)
where a is the learning rate for test-time adaptation.
48

---

Inference: After K steps of parameter updates, we use the tuned parameters d' and 6' to make predictions on new inputs. The prediction process remains the same as the transductive baseline but uses the adapted parameters:
Dat1 ~ Por (yl le (21, V1 Di............. cor (2 vn)))
(15)
49

---

I ARC-AGI Approaches Summary
Comparing methodologies on the ARC-AGI benchmark requires careful consideration of the diverse variables at play. High-performing methods, such as the o3- Ses-previewwlor model and the ARC-AGI 2024 winners, often differ substantially in their training data, test-time procedures, and computational assumptions. Our work is specifically designed to control for these variables, aiming to isolate and understand the dynamics of test-time adaptation strategies rather than maximize performance through increased compute or specific methodological biases. To provide comprehensive overview and situate our findings within this broader research landscape, we present a summary table outlining various prominent methodologies. The table details their model sizes, training strategies, test-time adaptation approaches, and reported performances.
Table 17: Key for Table Column Abbreviations
Shorthand Column
Description
Lang. Pre-Train
Indicates whether model underwent large-scale pre-training on a general language corpus.
ARC-AGI FT
Specifies the model was fine-tuned on official ARC-AGI training set. Denotes fine-tuning on the Re-ARC dataset, procedurally generated dataset for ARC.
Re-ARC FT
ARC Heavy FT
Refers fine-tuning on ARC Heavy, an augmented and more difficult version of ARC dataset.
CoT Infer.
Shows the model uses Chain-of- Chain-of-Thought reasoning during inference. Indicates whether the model generates a program uses a Domain Domain-Specific Lan- guage (DSL) to solve the task at inference time.
Prog. Infer.
Model Size
number parameters in model (e.g.. million, for billion).
ARC-AGI Comps.
Notes whether the method includes components strategies specifically designed for the ARC-AGI benchmark.
ARC-AGI Perf.
reported accuracy on ARC-AGI evaluation set.
Table 18: Comparison of Methods on ARC-AGI Benchmark (Part of 2): Training and Inference Strategies
Lang. Tre-Train
ARC-AGI
Re-ARC
ARC Heavy
CoT Infer.
Prog. Infer.
Method
FT
FT
FT
o3- o3-preview-low
Yes
Yes
Not public
Not public
Yes
No
ARChitects
Yes
Yes
Yes
No
Grok4 thinking
Not public
Not public
Not public
No
Grok 3 mini low
Yes
Not public
Not public
Not public
Yes
No
Qwen3-256b instruct
Yes
Not public
Not public
Not public
Yes
No
GPT4.5
Yes
Not public
Not public
Not public
Yes
No
Codeit
Yes
No
Yes
No
No
Yes
Mirchandani
Yes
No
No
No
No
No
LPN
No
Yes
No
No
LPN Latent Search
No
Yes
No
Inductive Li
Yes
No
Yes
Yes
No
Yes
Transductive Li al.
Yes
No
Yes
Yes
No
No
Llama 4 Maverick
Yes
Not public
Not public
Not public
Yes
No
50

---

Table 19: Comparison of Methods on ARC-AGI Benchmark (Part 2 of 2): Model Details and Performance
Model Size
ARC-AGI Comps.
ARC-AGI vl Perf.
Method
Model
o3-preview-low
13-prvieew-low
Not public
75.7%
ARChitects
-BBaae
8B
I
56.0%
Grok4 thinking
Grok-4
Not public
66.7%
Grok 3 mini low
Grok-3-mini
Not public
16.5%
Qwen3-256b instruct
Qwcn-3-256b
256B
11.0%
GPT4.5
GPT4.5
Not public
10.3%
Codeit
CodeT5
220M
14.75% [2]
Mirchandani
text-davinci-003
175B
6.75%
LPN
Vanilla Transformer
178M
7.75%
LPN Latent Search
Vanilla Transformer
178M
15.5%
Inductive Li et al.
Llama3.1 1-8...1-BBinne 1-8B-instruct
8B
[2]
38.0%
Transductive Li et al.
Llama3.1 1--.3..----iiseiii 1-8B-instruct
8B
43.0%
Llama Maverick
Llama Maverick
400B
4.4%
ARC-AGI Comps.: Whether the methodology directly targets the ARC-AGI benchmark with specific
components.
ARC-specific data augmentations for training and test time, ARC-specific tokenization scheme. ARC-specific DSL.
51

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We include limitations section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
52

---

Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We disclose all hyperparameters and full describe the method and architecture
used.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
53

---

Answer: [Yes]
Justification:
Code is provided with scripts for all experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
wwt.Subbmiiiioooioooooooooii for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide full details on datasets used and hyperparameters for all experi- ments.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Multiple seeds always performed for experiments with 1-sigma error bars given and stated.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
54

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide information on the exact hardware our experiments were trained on and the expected runtime.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp://neurips htt.ti/ggi////////.....////////a
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: This work foundational research exploring the adaptation test time in neural networks with no direct societal impacts from this work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
55

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
56

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
57

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
58


Reviews:
[{"review_id": "ceI8MUQYYa", "reviewer": "Reviewer_cyKr", "summary": "This paper introduces the Latent Program Network (LPN), a neural architecture that learns a continuous space of implicit programs. Specifically, it adopts a VAE-like architecture and additionally uses latent representation optimization before decoding. LPN uses gradient-based search at test time to adapt to new tasks, a process that was shown to double its performance on out-of-distribution problems on the ARC-AGI benchmark. By building in this adaptation mechanism, LPN outperforms or matches other methods like in-context learning and test-time training in various settings.", "questions": "Please see my comments above. \n\nHere I have to first apologize for not being able to provide many valuable suggestions. Because I am not familiar with the domain at all. I remember clearly that I clicked 'highly irrelevant' for this paper during bidding, but confused why I am still assigned this paper. Maybe this is because of the keyword \"program synthesis\"? As a result, I put my confidence score at only 2, and let's see how other reviewers review this paper.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength:\n1. The code is open-sourced, which helps replication.\n2. After reading the paper, I have a feeling that the LPN is a kind of VAE, yet we don't just decode the latent representations directly, but optimize the latent representation by either sampling or gradient ascent, which I think is valuable and interesting.\n3. The performance on OOD setting significantly outperforms baseline methods.\n\nWeakness:\n1. All the evaluations are conducted based on a narrow set of visual, grid-based reasoning tasks, which are not comprehensive enough to demonstrate the superiority of LPN.\n2. The paper's definition of a \"program\" is implicitly tied to transformations on 2D grids. The claim of creating a general approach to program synthesis is not fully verified. Still related to Weakness 1, since the authors only evaluate LPN on grid-based reasoning tasks, I doubt the claim of \"program\" in this paper.", "was_revised": true, "final_justification": "After discussing with the authors and other reviewers, I find that I have some misunderstandings about the paper.", "rebuttal": "We thank the reviewer for their detailed review, for highlighting that our code is fully open-source for reproducibility, and for noting the strength of the out-of-distribution (OOD) performance of LPN.\n\nOne highlighted strength was the recognition of LPN as a type of variational autoencoder (VAE), a point with which we fully agree. We clarify this further in Appendix D, explicitly framing LPN as a VAE and describing the latent optimization step as semi-amortised variational inference (SVI). Specifically, at test time, SVI refines the aggregate posterior for individual data points to close the amortization gap remaining after training. While we reference this discussion in line 182, we agree that explicitly stating in the main text that LPN performs variational inference would enhance clarity.\n\nTo address the reviewer's concern about our experiments focusing narrowly on visual grid-like tasks, we highlight an additional experiment in Appendix B.7. Here, we explicitly compare LPN and test-time-training (TTT) on sequence-based programs without grid-based features, confirming that our method is not limited to grids. This experiment further demonstrates TTT's tendency to overfit and supports our claim that LPN effectively improves performance through latent-space gradient updates at test time.\n\nRegarding the comment that our definition of \"program\" implicitly focuses on transformations of 2D grids and thus may not fully support a general claim, we agree that our experiments do not show LPN learning a fully general space of all programs. Indeed, LPN intentionally leverages biases from its training distribution to efficiently represent a specific class of programs tied to a particular input-output distribution.\nWe believe LPN is general in the sense that it makes minimal assumptions beyond having data in the form of N input-output pairs. Our experiments demonstrate that test-time adaptation improves performance consistently across all three tested environments and provides robust generalization to out-of-distribution tasks.\nHowever, we acknowledge the reviewer's concern. We will update the manuscript, in both the introduction and discussion, to clarify that our method targets program synthesis specifically within programming-by-example tasks, exploring grid-based and integer-sequence domains. Training LPN on a truly general program space would require extensive pre-training on diverse datasets and remains an exciting direction for future research, given sufficient compute resources.\n\nRegarding the reviewer’s concern about the claim that LPN represents a \"program space,\" we agree that there are important distinctions between LPN's neural programs (latent vector plus neural decoder) and traditional code-based programs. Although both represent valid programs that map inputs to outputs, they differ notably in terms of compositionality, differentiability, and expressivity.\nSpecifically, the space learned by LPN covers only a small subset of the broader space expressible through traditional code. While LPN sacrifices compositionality, it gains differentiability, enabling efficient test-time search, which is challenging for methods that generate code.\nWe will explicitly clarify this trade-off in the final paper, emphasizing that LPN learns a narrow, implicit program space optimized for efficient search and strong performance within the training distribution.\n\n**Additional Ablation**\n\nWe also draw the reviewers’ attention to an additional ablation conducted for reviewer qA5h. This evaluates different LPN test-time inference strategies on ARC-AGI, providing further insight into the roles of the program prior and gradient-based search in driving performance."}, {"review_id": "sZMVvGtD36", "reviewer": "Reviewer_qA5h", "summary": "This paper presents a novel approach for program search and applies it to the ARC-AGI benchmark. The approach uses a VAE to encode program synthesis tasks into a patent program-like space, then optimizes the latents using test time gradient updates. This is like transduction models with TTT but instead of optimizing the full transduction network, you just optimize the latent, which could be more efficient. Compared to the transduction baseline, the proposed model works much better. Performance on ARC is comparable to prior approaches, although it falls short of recent transductive models, possibly due to reduced training time and dataset size and quality. The authors provide thorough analysis of their method.", "questions": "Q1. any theories why more grad steps doesn't work?\nQ2. on page 4, you say \"we diverge from such transduction-based methods, as they cannot inherently adapt at test-time or ensure specification consistency\" — isn't this exactly what TTT does, which transduction-based methods are compatible with? \nQ3. for table 3 is \"in distribution\" the re-arc dataset?  is it the training set, or a validation set from RE-ARC? and then ood is ARC-AGI validation? maybe I missed this in the main text, but if it's not there it should be described there (and even better in the table too).", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- The approach is really novel! It is also principled, and focused on a clear innovation. There have been many approaches tried on ARC-AGI in light of the competition last year and previous years, and this is one of the more innovative approaches.\n- The paper is written well, clear, and easy to follow. The figures are beautiful (e.g. figure 1) \n- The paper includes informative ablations.\n\nWeaknesses:\n- On ARC-AGI, as compute scales, the proposed approach does not work better than TTT. However, the authors do a decent job acknowledging this, and provide reasonable hypotheses for why this might be the case. \n- Some of the details of how the baselines work are hard to understand in the main text. The appendix does a good job thoroughly explaining the different approaches, but I think a few more details of the baselines and how they are implemented could be useful for the main text.\n- Other papers achieve much higher validation scores on ARC-AGI. Upon closer inspection, most of those higher scores come from bells and whistles and scaling, but perhaps the paper could do a stronger job explaining why its final scores aren't competitive on ARC. To this end, a table with validation scores from an increased number of approaches could be useful, just to put this paper more in context with the surrounding ARC literature (so readers don't have to do that work ourselves).\n- The empirical experiments to test the benefits of the latent approach compared to TTT are not very strong. For example, in Figure 9, LPN + grad isn't that much computationally more efficient than TTT. And it's not clear to me how much the other proposed benefit, the program prior, is playing a role. Is there an experiment for this?", "was_revised": true, "final_justification": "I will maintain my score. The other reviewers have largely overlapping views on the paper, except for reviewer cyKr who had low confidence in their review.", "rebuttal": "We thank the reviewer for their thorough evaluation, highlighting the method's novelty, clear writing, and detailed ablations. Below, we address each question and comment provided.\n\nWe note the reviewer’s suggestion to improve the explanation of baselines in the main text. We acknowledge this and will revise the main text to include sufficient details about the baselines.\n\n## Contextualizing ARC-AGI Results\n\nThe reviewer accurately notes that certain approaches achieve higher scores on the ARC-AGI benchmark but typically involve substantial, often ARC-specific additions (e.g., pre-trained and fine-tuned language models, synthetic data generation at test time). Our primary goal was to fairly compare test-time adaptation methods (In-context, TTT, and LPN) while holding other factors constant. Hence, direct comparisons with methods employing ARC-specific enhancements or additional pre-trained models would not be equitable.\n\nHowever, we agree that contextualizing our method's performance relative to various approaches is valuable, regardless of differing computational requirements or assumptions. We have thus compiled a comprehensive table summarizing a range of methods, pre-training and post-training strategies, and ARC-AGI scores. This provides a clearer context regarding LPN's performance relative to existing methods.\n\n[See Review H6z6 for table]\n\n\n## Answers to questions\n\n**Q1. “any theories why more grad steps doesn't work?”**\n\nOne theory is that using more gradient steps during training reduces the pressure on the encoder to produce a strong initialization for the decoder since extra gradient steps can compensate for a weaker starting point. Thus, beyond a certain threshold, the encoder may receive a less effective training signal, resulting in poorer initialization given the same computational budget. Our experiment (Figure 2) underscores the encoder's importance, indicating that weaker encoder performance might negatively impact test-time search.\nTable 1 supports this observation: training with 5 gradient steps (Grad 5) results in 0.0% encoder-only performance, whereas training with 0 steps (Grad 0) achieves 3.2%. However, with at least 5 test-time gradient steps, the Grad 5 model performs better. Grad 1 may strike the optimal balance, training a strong encoder for good initialization while still allowing effective latent space refinement through gradient steps at test time.\nTo mitigate this issue, we applied progressive training, from 0 steps gradually increasing to N steps (see line 366), which ensures the encoder initially receives strong training signals. Gradient steps are only increased once the model converges, preventing encoder underfitting.\nAdditionally, we hypothesize that doing more gradient steps in the latent space at test-time might eventually overfit the latent to the specification (train input-output pairs) and fail to generalize to the test input. For instance, in Table 3, there is some evidence that suggests minor overfitting occurs on the in-distribution dataset when going from 2e13 to 2e15 flops, with additional search marginally lowering performance. In the future, it would be valuable to study the extent to which specification overfitting occurs in LPN.\n\n**Q2. on page 4, you say \"we diverge from such transduction-based methods, as they cannot inherently adapt at test-time or ensure specification consistency\" — isn't this exactly what TTT does, which transduction-based methods are compatible with.**\n\nThe reviewer makes a valid point. Our original wording intended to emphasize that transduction-based methods (e.g., TTT) typically lack an inherent mechanism for adapting to new data without modifying the model's parameters directly (unlike in-context learning and LPN). We acknowledge this was unclear and will clarify this distinction in the final paper, explicitly highlighting the benefits of methods capable of adapting to new data and maintaining consistency without relying on parameter-based fine-tuning.\n\n**Q3. for table 3 is \"in distribution\" the re-arc dataset? is it the training set, or a validation set from RE-ARC? and then ood is ARC-AGI validation? maybe I missed this in the main text, but if it's not there it should be described there (and even better in the table too).**\n\nThe in-distribution dataset refers to the ARC-AGI training set, and the out-of-distribution (OOD) dataset is the ARC-AGI evaluation set. Neither dataset is used directly for training; instead, we use data generated by RE-ARC during training. However, RE-ARC tasks are built upon primitives from the ARC training set, making them effectively in-distribution relative to the ARC training set. While we attempt to explain this distinction in the description of Table 3, we acknowledge that it can be clearer and will provide additional clarification in the final version of the manuscript.\n\n\n\n## Additional LPN experiments on ARC-AGI\nThe reviewer notes the lack of analysis regarding the specific components contributing to LPN's performance on ARC-AGI. To address this, we conducted additional ablations exploring different test-time inference strategies to better understand the impacts of the program prior and gradient-based search. We evaluated performance on a single seed on ARC-AGI eval dataset (out of distribution relative to re-arc), keeping constant the number of sampled latents during inference, varying budgets in [10,50,100,200,400].\n\nSpecifically, we compared the following inference strategies.\n\n**Encoder Sampling**:  Samples multiple latents from the encoder, evaluates the likelihood of the specification given each latent, and selects the best. This tests the value of structured, gradient-based search versus brute-force sampling near the encoder output. \n\n**Gaussian Init + Gradient Search**: Uses the same gradient-based search as core LPN but initializes from a sample drawn from the prior (Gaussian). This isolates the encoder’s role in both suggesting good initial points and improving overall performance.\n\n**Encoder Init + Local Search**: Starts from the encoder output but replaces gradient updates with a mutation operator (continuous noise perturbation). This examines the benefit of gradient-based updates versus unstructured local search, while still allowing movement beyond the encoder distribution.\n\n**Gaussian Init + Local Search**: Combines prior-based initialization with local search. This baseline helps evaluate the importance of both encoder initialization and gradient guidance.\n\n**Encoder Init + Gradient Search (LPN)**: The standard LPN inference method.\n\nBelow is the table comparing these different inference strategies with varying search budgets.\n\n\n| Inference Method                   |    10 |    50 |   100 |   200 |   400 |\n|--------------------------------------|-------|-------|--------|--------|--------|\n| Encoder Sampling                     |  9.12 | 10.75 | 10.13 | 10.50 | 10.62 |\n| Gaussian Init + Gradient Search |  1.38 |  2.75 |  5.37 |  6.88 | 10.75 |\n| Encoder Init + Local Search     |  8.38 |  9.50 | 10.37 | 10.75 | 10.75 |\n| Gaussian Init + Local Search           |  0.25 |  0.25 |  0.25 |  0.75 |  0.25 |\n|  Encoder Init + Gradient Search (LPN)  |  9.10 | 13.25 | 15.00 | 15.50 | 15.50 |\n\n### Summary of Results\n\nRepeated sampling from the encoder offers slight improvements over a single sample (7.75%), but the gains are minimal. Comparing _Gaussian Init + Gradient Search_ to _Encoder Init + Gradient Search_ highlights the encoder's importance: it not only boosts performance with few gradient steps but also significantly improves results at high budgets (e.g., 400 steps).\n_Encoder Init + Local Search_ performs similarly to gradient-based search at low budgets (10 steps), but diverges sharply as the budget increases, demonstrating the effectiveness of gradient guidance.\n_Gaussian Init + Local Search_ performs poorly, with results near 0%, showing the combined importance of good initialization (via the encoder) and gradient-based search.\nOverall, _Encoder Init + Gradient Search (LPN)_ consistently outperforms all other methods, confirming that LPN’s core components, i.e., encoder-informed initialization and gradient-based optimization, are essential for strong performance."}, {"review_id": "hzLcJUCGFc", "reviewer": "Reviewer_H6z6", "summary": "The Latent Program Network (LPN) is neural architecture that combines symbolic program synthesis with neural networks by learning a latent space of implicit programs, which are then searched using gradient-based optimization at test time. This approach addresses the limitations of traditional program synthesis and deep learning methods (lack of consistency, adaptability, and generalization), as well as the expense and overfitting issues of Test-Time Training (TTT). LPN's key contributions include its ability to adapt to new tasks through gradient-based search in learned latent space, demonstrating improved generalization and consistency over in-context learning or parameter fine-tuning. Empirical results on the ARC-AGI benchmark and a synthetic \"Pattern\" task show LPN's superior performance in out-of-distribution scenarios and with varying specification sizes, outperforming both in-context learning and TTT.", "questions": "A few questions and improvements for the paper are:\n\nQ: The core experiments use small models which raise question about how LPN scales to the size of state-of-the-art LLMs or more complex program spaces. Demonstrating consistent LPN benefits with larger models would validate scalability and robustness claims.\n\nQ: How LPN methods would compare with the results presented in [1]. \n[1] \"Combining Induction and Transduction for Abstract Reasoning\" Li et al. 2024", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths are:\n-The paper tackles program synthesis and reasoning tasks by proposing a hybrid neural-symbolic method with practical test-time adaptability\n-The paper shows a combination of latent space modeling and test-time optimization for program synthesis and it cuts computation cost at test time \n\nWeaknesses are:\n-It seems the evaluations are mostly for ARC-AGI which restricts conclusions on broader generalization\n-The paper doesn't compare results with winners of ARC-AGI 2024.", "was_revised": true, "final_justification": "The summary table and additional information adds to the paper score, since it clarifies major points in clarity and quality. The authors added clarifying table and highlighted the difference from the previous work", "rebuttal": "We thank the reviewer for their detailed feedback and address each comment below.\n\nWe acknowledge that most evaluations focus on grid-based reasoning tasks. However, we direct the reviewer’s attention to the sequence-based program tasks discussed in Section B.7, which do not rely on a 2D grid structure. These tasks further illustrate the tendency of TTT to overfit and demonstrate the consistent improvement of LPN with additional gradient steps.\n\n## Contextualizing ARC-AGI Results\n\nRegarding comparisons with other methods on ARC-AGI, we emphasize the importance of controlling variables carefully when evaluating selected test-time adaptation methods (i.e., In-context, LPN, TTT). High-performing methods such as the o3-preview-low model or ARC-AGI 2024 winners differ significantly in training, test-time data, and computational assumptions. Our selection of the ARC-AGI benchmark for evaluation was specifically aimed at isolating and understanding the dynamics of test-time adaptation strategies, rather than maximizing performance through specific methodological biases or increased compute.\n\nHowever, we agree that providing context about the diverse performance levels and computational assumptions across different methods is essential. To enhance clarity, we will include a summary table outlining various methodologies, their model sizes, training strategies, test-time adaptation approaches, and respective performances.\n\n| Method                | Language Pre-Training | ARC-AGI Fine-Tune | Re-ARC Fine-Tune | ARC Heavy Fine-Tune | Chain of Thought Inference | Program-Based Inference | Model Size | ARC-AGI Components | Model                         | ARC-AGI v1 eval Performance |\n| --------------------- | --------------------- | ----------------- | ---------------- | ------------------- | -------------------------- | ----------------------- | ---------- | ------------------ | ----------------------------- | --------------------------- |\n| 03-preview-low        | Yes                   | Yes               | Not public       | Not public          | Yes                        | No                      | Not public |                    | 03-preview-low                | 75.7%                       |\n| ARChitects            | Yes                   | Yes               | Yes              | Yes                 | No                         | No                      | 8B         | [1]            | Mistral-NeMo-Minitron-8B-Base | 56.0%                       |\n| Grok4 thinking        | Yes                   | Not public        | Not public       | Not public          | Yes                        | No                      | Not public |                    | Grok-4                        | 66.7%                       |\n| Grok 3 mini low       | Yes                   | Not public        | Not public       | Not public          | Yes                        | No                      | Not public |                    | Grok-3-mini                   | 16.5%                       |\n| Qwen3-256b instruct   | Yes                   | Not public        | Not public       | Not public          | Yes                        | No                      | 256B       |                    | Qwen-3-256b                   | 11.0%                       |\n| GPT4.5                | Yes                   | Not public        | Not public       | Not public          | Yes                        | No                      | Not public |                    | GPT4.5                        | 10.3%                       |\n| Codeit                | Yes                   | No                | Yes              | No                  | No                         | Yes                     | 220M       |  [2]             | Code T5                       | 14.75%                      |\n| Mirchandani           | Yes                   | No                | No               | No                  | No                         | No                      | 175B       |                    | text-davinci-003              | 6.75%                       |\n| LPN                   | No                    | No                | Yes              | No                  | No                         | No                      | 178M       |                    | LPN Encoder-Decoder  Transformer          | 7.75%                       |\n| LPN + Latent Search   | No                    | No                | Yes              | No                  | No                         | No                      | 178M       |                    | LPN Encoder-Decoder  Transformer           | 15.5%                       |\n| Inductive Li et al    | Yes                   | No                | Yes              | Yes                 | No                         | Yes                     | 8B         | [2]             | Llama3.1-8B-instruct          | 38.0%                       |\n| Transductive Li et al | Yes                   | No                | Yes              | Yes                 | No                         | No                      | 8B         |               | Llama3.1-8B-instruct          | 43.0%                       |\n| Llama 4 Maverick      | Yes                   | Not public        | Not public       | Not public          | Yes                        | No                      | 400B       |                    | LLama 4 Maverick              | 4.4%                        |\n\n(ARC-AGI Components): whether the methodology directly targets the ARC-AGI benchmark\n\n[1] ARC-specific data augmentations for training and test time, ARC-specific tokenization scheme\n\n[2] ARC-specific DSL \n\n\n\n\n## Answers to Questions\n\n**Q1. The core experiments use small models which raise question about how LPN scales to the size of state-of-the-art LLMs or more complex program spaces. Demonstrating consistent LPN benefits with larger models would validate scalability and robustness claims.**\n\nWe first note that, although the LPN models trained for ARC-AGI might be small compared to large language models, they are still relatively large, with approximately 178M parameters. These models require pixel-perfect execution on 900-pixel grids across hundreds of programs, demanding significant capability from the neural decoder.\nRegarding scaling LPN to state-of-the-art LLMs, we agree that this is an exciting direction for future research. However, state-of-the-art LLMs typically have around 200 billion parameters, and training such models exceeds the scope of this initial study, for engineering and financial reasons. Nevertheless, our experiments demonstrate LPN's consistent advantages across different scales, from 1M parameters (pattern tasks) to 178M parameters (ARC-AGI).\n\n**Q2. How LPN methods would compare with the results presented in [1]. [1] \"Combining Induction and Transduction for Abstract Reasoning\" Li et al. 2024**\n\nWe appreciate the reviewer’s request for comparison with Li et al. (2024), which also evaluates methods on the ARC-AGI benchmark. We have included their models in the summary table provided above, alongside different methods and assumptions. It's important to note that their work uses significantly different compute and data assumptions than ours. Specifically, their experiments rely on a pre-trained language model (LLaMA 3.1-8B-Instruct) for both induction and transduction strategies.\nMethodologically, their transduction approach aligns with our transduction baseline in that both use similar inference strategies: they do not explicitly represent programs but instead predict outputs directly from input-output examples. However, our work differs in its assumptions regarding model size and pre-training data. These distinctions are critical to contextualizing performance and understanding the broader implications of test-time adaptation methods like LPN."}, {"review_id": "RI3FB6DLZJ", "reviewer": "Reviewer_6Xrm", "summary": "This paper introduces a new way of performing test-time adaptation for learning algorithms from few examples. The general idea is to optimize a latent \"function vector\" in a variational inference framework. They evaluate primarily on ARC, finding that although it is not state-of-the-art, that it is very compute efficient and do many experiments to get insight on exactly what is important for it to work.", "questions": "I get how your method could adapt somewhat out-of-sample because you optimize $z$ at test time, but I'm skeptical that it could compositionally generalize, because fundamentally the \"program\" is being \"executed\" by a neural network, and has to be represented by a fixed length vector. Did you ever see evidence of compositional generalization?\n\nDid you ever try also also optimizing the parameters of the encoder/decoder during test-time optimization?\n\nDid you ever try having the encoder take multiple examples as input?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Creative method that is very compute efficient relative to TTT\n2. I like the connection to variational inference. Unlike a lot of work on ARC, this paper tries to connect to general ideas in machine learning, rather than engage in benchmark-hacking.\n3. Rich analyses on synthetic tasks which give insight on the details of what makes the method work\n4. Good description of prior work\n\nWeaknesses:\n1. There are other natural ablations and variations of your system to consider; see Questions\n2. Mostly only evaluated on ARC. I strongly believe the paper should not be rejected on this basis, because ARC is rich and challenging and we still do not have open-source solutions, but it is a weakness. Note Appendix B7 has experiments on a very different synthetic dataset, which also helps address this weakness. (If accepted you should probably cover these other experiments in the main text.)\n3. Nitpick: What do the performance numbers for ARC mean? Is that the number of tasks solved or the percentage of tasks solved?", "was_revised": true, "final_justification": "The authors explained a further interesting result relating to compositionality. This makes me want to raise the score even more, so I'm thinking of this as a `5.5/6`. I'm unconcerned by our sole negative reviewer, cyKr, saying that ARC is a \"narrow set of visual, grid-based reasoning tasks, which are not comprehensive enough\"--ARC is a rich testbed, and essentially a composite of many diverse reasoning puzzles. (I would take the plunge and upgrade to a score of `6/6` if they showed their method working on another challenging dataset, however)", "rebuttal": "We thank the reviewer for their detailed review and appreciate their recognition of the creativity and efficiency of LPN relative to TTT.\n\nTable 3 reports ARC-AGI performance as the percentage of tasks solved. We will update the table caption to clarify this.\n\nBelow, we respond to the other questions posed.\n\n**Q1. “... I'm skeptical that it could compositionally generalize, because fundamentally the \"program\" is being \"executed\" by a neural network, and has to be represented by a fixed length vector. Did you ever see evidence of compositional generalization?”**\n\nWe agree that since the latent program is ultimately executed by a neural network, there are inherent limitations to the compositionality achievable in this representation. LPN’s ability to perform compositionally generalization depends on the nature of the considered generalization. We investigated a variety of types of compositional generalization and found that the search component of LPN enables generalization of type Compose-Different-Concepts and Switch-Concept-Order outlined in [1]. \n\nWe ran experiments combining programs from the training set in novel ways not seen during training. For example, we tested a task that required both filling black cells within objects with a specific color and cropping a grid based on a rectangle, which are two operations that were only ever seen separately during training. While the encoder alone failed to generalize (only executing the cropping task), LPN with gradient-based search was able to successfully compose the two after 500 gradient steps. This demonstrates that the smooth, learned latent space supports compositional generalization beyond what was explicitly trained.\nWe agree with the reviewer that illustrating the model's generalization capabilities adds clarity, and we will include this analysis in the appendix.\n\nHowever, we find no evidence in our experiments to suggest that LPN is capable of generalising for length-based recurrence or applying concepts in structurally different ways (e.g., chaining programs longer than those seen during training). We will also include such negative examples in the appendix.\nAs noted on line 394, future work could explore discrete latent representations, potentially enabling more robust compositionality. However, learning such representations is non-trivial and must be weighed against increased training complexity.\n\n[1] Shi, Kensen, et al. \"Compositional generalization and decomposition in neural program synthesis.\" arXiv preprint arXiv:2204.03758 (2022).\n\n**Q2. Did you ever try also also optimizing the parameters of the encoder/decoder during test-time optimization?**\n\nWe did consider the possibility of hybridizing TTT and LPN by combining LPN’s latent space search with decoder parameter updates at test time. This could help when the decoder cannot fully execute a discovered latent program.\nWhile this may improve performance for complex programs, it would also introduce parameter updates, diverging from LPN's core advantages of efficiency and reduced overfitting. In both the pattern and systematic sequence tasks, parameter fine-tuning degraded performance. Only on the ARC-AGI benchmark have we observed increased performance, likely due to decoder capacity limitations.\nWe chose not to include this hybrid in our initial work to maintain focus on LPN's unique strength: efficiently adapting without weight updates. We hypothesize that as the diversity of programs during training increases, the need for parameter updates will diminish, reinforcing the value of search-based adaptation.\n\n**Q3. Did you ever try having the encoder take multiple examples as input?**\n\nWe did explore feeding multiple input-output pairs to the encoder in early versions of LPN. However, this approach had key downsides:\nIt led the encoder to overfit to the number of examples seen during training, reducing its ability to generalize across varying specification sizes (as seen in Figure 3).\nIt broke the permutation invariance of the I/O pairs within the specification, which is a desirable property.\nEncoding pairs independently allows for parallel computation and faster training.\nThat said, the reviewer raises a valid point: ideally, representations of individual pairs could inform one another. A promising direction for future work would be to, e.g., encode each pair independently, then apply attention-based aggregation to allow cross-pair interaction and refinement.\n\n**Additional Ablation**\n\nWe also draw the reviewers’ attention to an additional ablation conducted for reviewer qA5h. This evaluates different LPN test-time inference strategies on ARC-AGI, providing further insight into the roles of the program prior and gradient-based search in driving performance."}]

Author rebuttal:

Meta-review: (a) Summary of Scientific Claims and Findings

The paper introduces the Latent Program Network (LPN), a novel architecture that learns a latent space of implicit programs. This allows the model to use gradient-based search at test time to adapt to new tasks, aiming to combine the adaptability of program synthesis with the scalability of deep learning. The key finding is that LPN's performance on out-of-distribution tasks doubles when this test-time search is enabled.

(b) Strengths

1. The core concept of a searchable latent program space is considered novel and interesting.

2. The paper provides strong evidence of the model's effectiveness on out-of-distribution tasks.

3. The paper is well-written and easy to follow.

(c) Weaknesses

1. The experiments are confined to a narrow set of visual, grid-based tasks, which raises questions about the method's general applicability.

2. The term "program" seems implicitly tied to the specific grid-based tasks used in the evaluation, weakening the claim of a general program synthesis approach.

(d) Most Important Reasons for Decision

The novelty of the approach and its strong out-of-distribution performance are compelling reasons to accept.

(e) Summary of Discussion and Rebuttal

The discussion centered on the paper's limited evaluation. Reviewers questioned whether the LPN could generalize to non-visual or text-based tasks. One reviewer defended the use of the ARC benchmark as "best-in-class" for visual inductive reasoning, but the core issue of the method's generalizability beyond this domain remained the primary point of contention.
Decision: Accept (spotlight)