Structure-Aware Graph Hypernetworks for Neural Program Synthesis
Authors: Wenhao Li
Yudong Xu
Elias Boutros Khalil
Scott Sanner
Venue: ICLR 2026 Poster (2026)
Keywords: neural program synthesis
weight-space learning
meta-learning
permutation-equivariant graph networks
zero-shot generalization
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We show that a structure-aware hypernetwork can directly generate full neural-network weights—treating weights as a continuous program modality—and outperform baselines with strong zero-shot generalization across unseen tasks.

Abstract: We study the neural program synthesis of $\textit{parameterized}$ function families through the lens of meta-learning with hypernetworks. Given a user intent $U$, a meta-learner $M_{\phi}$ produces a full weight set $\hat{\theta}=M_{\phi}(U)$ for a target neural network with fixed architecture $S$, and the instantiated network $m_{S,\hat{\theta}}(X)\to Y$ realizes the behavior intended for $U$. Classical hypernetworks typically $\textit{ignore the target network’s structure}$ and emit a flat list of weights; as a consequence, they fail to account for $\textit{neuron-permutation symmetry}$—many distinct parameterizations of $S$ implement the same function—so equivalent solutions are treated as different targets, fragmenting supervision and hurting out-of-distribution generalization. To address this, we propose $\textit{Meta-GNN}$, a hypernetwork that constructs a $\textit{neural graph}$ from the target architecture $S$ and applies $\textbf{structure-aware}$ message passing with parameter-tied encoders and decoders. This design reduces the search space during learning by collapsing equivalent classes of target networks, without loss of expressivity. Empirically, across modular arithmetic ($\textit{AddMod}$-$p$), array operations ($\textit{SumFirst}$-$n$), and inverse-rule tasks from 1D-ARC, $\textit{Meta-GNN}$ substantially improves learning and $\textbf{out-of-distribution generalization}$ compared to classic hypernetworks and direct $(U,X)\to Y$ baselines. Mechanistic analyses reveal $\textit{what is learned}$: on $\textit{AddMod}$-$p$ the synthesized Transformers recover the canonical clock representation and admit a compact closed-form map $U\mapsto\theta$. These results demonstrate that structure-aware Meta-GNNs enable reliable generalization to $\textit{unseen program parameterizations}$, providing a critical advance for the nascent field of neural program synthesis.

Published as a conference paper at ICLR 2026
STRUCTURE-AWAR GRAPH HYPERNETWORKS FOR NEURAL PROGRAM S SYNTHESIS
Wenhao Li'
Yudong Xu Elias B. Khalil'
Scott Sanner
DDppartment of Mechanical & Industrial Engineering, University of Toronto
Vector Institute for Artificial Intelligence
chriswenhao 1i @ma www.lll.uuoooooooooooo ca, wil. wil.xu@mail www.aalll.uorootoooooo
e elias. w @etteeroooot..co ssannerêmie
ct s
ABSTRACT
We study the neural program synthesis of parameterized function families through the lens of meta-learning with hypernetworks. Given a user intent U, meta- learner Ma produces full weight set Ô = MU(U) for a target neural network with fixed architecture S, and the instantiated network mgg the behavior intended for U. Classical hypernetworks typically ignore the target network's 's structure and emit a flat list of weights; as a consequence, they fail to account for neuron-pent pruron-permuttaiion symmetry- many distinct parameterizations of S implement the same function-ss so equivalent solutions are treated as different tar- gets, fragmenting supervision and hurting out-of distriibiiiiitionnnn generalization. To address this, we propose Meta-GNN, hypernetwork that constructs neural graph from the target architecture S and applies structure-aware message pass- ing with parameter-tied encoders and decoders. This design reduces the search space during learning by collapsing equivalent classes target networks, without loss of expressivity. Empirically, across modular arithmetic (ADDMOD-p), array operations (SUMFIRST-n). and inverse-rule tasks from ID-ARC, Meta-GNN sub- stantially improves learning and outti--distribution generalization compared classic hypernetworks and direct (U, X) - Y baselines. Mechanistic analyses reveal what is learned: ADDMOD-p the synthesized Transformers recover the canonical clock representation and admit a compact closed-form map U + O. These results demonstrate that structure-aware Meta-GNNs enable reliable gen- eralization to unseen program parameterizations, providing critical advance for the nascent field of neural program synthesis.
(X) (x,,(X)
Y realizes
INTRODUCTION
Program synthesis is traditionally framed as a combinatorial search over symbolic artifacts such abstract syntax trees domain-specifie languages (DSLs) (Solar-Lezama) 2008 Alur et al., 2013). A user intent-given by I/O pairs, contracts, or natural language- must be realized by token sequence that is both syntactically valid and semantically correct. This setting exposes two long-standing bottlenecks. (i) Search: navigating the discrete program space requires expensive enumeration, even small edits simultaneously affect syntax and semantics. (ii) Verification: correctness is hard True/False property, offering no graded signal about how far a candidate pro- gram is from meeting the specification. These factors make scaling beyond small DSLs short snippets extremely challenging.
We propose continuous relaxation of program synthesis. A neural network with structure S and weights 0 can be viewed as an executable program, so we define a neural program (NeuroP) as the pair (S, 0) Goodfellow et al. 2016). To make synthesis tractable, we fix a template architecture and learn to generate its parameters 0 directly from user intent U. In this setting, syntax is guar- anteed by construction, while semantics ccatured by exec(S* 8, X) are differentiable whenever smooth loss be defined. Neural programs best understood parameterized forward-pan equations that approximate the desired behavior (c.g., a classifier with high but not perfect accuracy). This perspective replaces the two classical bottlenecks: instead of navigating discrete search space,

---

Published as a conference paper at ICLR 2026
Meta GNN (penes Pmsg q dec)
Target Network (w, b)
iii.
Initialize
Message Passing Decode
+
W11
W
re-lotid by
b
b
rC (loss?
bb
p
+
r6 1o
with
W12
w
w12
b2 1k-1
by
TC: (Poaa))
E
W
6
by
ri: lo
W222
v.
Softmax
Unembedding Layer
Add
Feed-Forward Layer
Add
Multi-head Self-Attentio
Embedding Layer
Posmout
sdp_oneeone algnn enszore norma 5200_thi
Attr
Wen
83
det product
a, b
A
Attr,
PC
Figure 1: Synthesizing a NeuroP. We use ADDMOD-p to illustrate the pipeline. Given 2 1,
the goal is to generate the weights of one-layer Transformer (the Target Network) that computes = (a + b) mod p for any integers a, b. A Meta-GGN is trained on pairs {((p, a.p.a,bb b). (i) At each step the meta-learner receives only the user intent U {p}. (v) The Transformer to be produced is first converted into a neural graph (iv) with nodes (biases) and edges (weights). As the meta-network, the only learnable parameters &p live the encoders Sene, message func- tion Smsg. and decoders fdee: all tied within permutation-equiiee runs & rounds of message passing parameterized by fmag with producing node/edge embeddings; (iii) group-tied decoders faee map these embeddings scalar parameters that instantiate target network weights w and biases b/. The instantiated Transformer (v) then maps (a, b) to ÿ. and gradients of L(ū. y) backpropagate through (v)->iiii->de (ii) (i) to up- date sp. We inject positional encodings only initialization (i) to break symmetry; these codes not used inside fencs mssg or fdee, so learning remains permutation-invarian This preserves target network expressivity and reduces the effective search space, which empirically improves learning.
groups. (ii) The Meta-GNN aggregation,
we optimize in continuous weight space: and instead of a hard True/False specification, we rely on graded losses that quantify semantic error. The trade-off is that we give up exact correctness guar- antees, but in return enable gradient-based optimization and rich learning signals.
Within this framework, two central questions arise: (1) can complete neural programs be synthe- sized in one pass from intent alone, and (2) can the synthesizer learn reliable mapping U - 0 that extrapolates to intents never seen during training? The latter especially significant, as it amounts to true out-of-distribution generalization (OOD), long-standing challenge in both program syn- thesis and machine learning. Under mild realizability and continuity assumptions, existence and stability results for intent-to- weights synthesis follow from standard extensions of universal ap- proximation and weight- pertuurrrttrration bounds; we provide formal statements in App. [ Given these existence results, it may seem that classical hypernetorks (Ha et al. 2016) may be able address
2

---

Published as a conference paper at ICLR 2026
target network program synthesis; unfortunately, such hypernetworks ignore the target network's structure and neuron paturonneemutatiion symmetry (Navon et al. 2023 Zhou et al. 2024), where many weight assignments implement the same function, fragmenting supervision and hurting OOD gen- eralization. To overcome this, we introduce a structure-aware Meta-GNN that decodes to the target network's computation graph (weights as edges, biases as nodes), uses DeepSet-style (Zaheer al.) 2017) parameter tying for the encoders and decoders within permutable neuron groups, and performs structure-aware message passing aligned with the graph's structure, thereby reducing the effective search space. We refer to Fig. for an overview of the methodology and $4.3 for additional details.
Empirically, we evaluate on three classic modeling setups with standard loss functions- an MLP trained with MSE for prefix summation (SUMFIRST-n), a Transformer trained with cross-entropy for modular addition (ADDMOD-p). and a seq2seq Transformer for mixtures of program families in 1D-ARC (Xu et al. 2024). This design highlights that our pipeline is architecture- and loss-agnostic, while the TD-ARC setting particular provides more realistic testhed with diverse tasks. Across all tasks, meta-learners which mimic the conventional supervised learning setup. Among our models, the structure-aware Meta-GNN achieves the strongest and most reliable generalization to unseen intents. On ADDMOD- p, it not only recovers the modular "clock" algorithm but also allows us to empirically extract the learned mapping U 0, providing direct evidence that the meta-learner captures the underlying algorithm rather than memorizing instances.
weight space substantially outperform direct XU) + Y baselines,
In summary, contributions are threefold. First, we establish NeuroP a program modality that trades exact symbolic semantics for continuous, differentiable representations, enabling gradient- based optimization in place of enumeration-basee search. Second, we propose structure-aware graph hypernetwork (Meta-GNN) that represents the target architecture as a neural graph, ties en- coders/decoders across
-etitaaiiioneuivleeent neurons, and generates the entire weight set in a single forward pass. We further provide empirical evidence that weeight synthesis is feasi- ble, and structure-aware meta-learners achieve strong OOD generalization across diverse program families. Third, we release complete code and datasets to support reproducibility and future research (see App.
RELATED WORK
Symbolic and classical program synthesis. Traditional synthesis operates over discrete artefacts such as abstract syntax trees or DSLs, using constraint solving to find programs that satisfy user intent. Pioneering systems like SKETCH (Solar-Lezamal 2013) and SyGuS (Alur et al. (013) frame the task as search within template space, yielding strong correctness guarantees but facing com- binatorial blow-ups as the template grows. Subsequent work improves scalability through smarter search: CEGIS (Gulwani et al.) 2017) iteratively alternates between candidate generation and coun- terexample finding; evc4sy (Reynolds et al., 2019) combines SMT enumeration with divide-and- conquer; and hybrid approaches integrate learning-based based heuristics or symmetry reduction to prune the search (Huang et al. 2020 Abate et al. 2023 Kim al. 2021). Yet all remain tethered dis- crete syntax and handcrafted templates. By contrast, we recast synthesis as continuous optimization in weight space: a fixed neural template guarantees syntactic validity, while differentiability enables efficient gradient-driven search without explicit symbolic enumeration.
Neuro-symbolic approaches. Neuro-symbolic methods blend discrete program representations with differentiable neural modules, aiming to merge symbolic structure with the adaptability of learning-based models. Techniques such as Neuro-Symbolic Program Synthesis (NSPS) (Parisotto et al. 2017) and Learning to Infer Program Sketches (Nye et al. 2019) augment partial symbolic sketches with neural modules, typically relying on handcrafted DSLs or differentiable logic. Recent
systems like NEAR (Shah et al. 2020). NeuroSketcher (Emireddy 2023), and SYNAPSE (Modak) ctt al. 2025) extend this idea by treating sketches as partially differentiable computation graphs. impiicii-eprresentttion work such as NeuroSAT (Selsam et al. 2019) and dPads (Cui & Zhu 2021) also uses continuous spaces, but ultimately decodes back to discrete code. In contrast, our method dispenses with DSLs and intermediate symbolic sketches altogether and treats the entire weight vec- for of a fixed architecture as the synthesized program, so the hypernetwork directly solves program synthesis in weight space rather than decoding back into discrete language.
3

---

Published as a conference paper at ICLR 2026
Hypernetworks and meta-learning. Meta-leaming and hypemnetwrk approaches attempt to learn mappings from task conditions to full model parameters. HyperNetworks (Ha et al. 2016) pioneered the idea of using one network to generate weights for another. Similarly, MAML (Finn et al. 2017)
and its variants (Rusu et al. 2019) enable fast adaptation to new tasks using shared priors. Beyond the original HyperNetworks paper, many later works explore task-conditioned hypernetworks and meta-lcarning for continual learning and scientific ML (von Oswald et al. 2020 Zhao et al.) [020) Beck et al. 2023 Cho et al. 2023). However, most hypernetwork or meta-learning settings still rely on some form of test- time supervision, such as gradient adaptation on labeled examples from the new task. In contrast, our synthesis framework requires complete zero-shot generation: at deployment the meta-learner receives only the user intent U, with no access to test input-output pairs (X, Y), and must emit correct and executable weights 0.
Weights as a modality and permutation symmetries. A growing body of work treats weights themselves first-class learning signal (Schürholt et al. (02T). (Ainsworth et al. (2023) demon- strate that aligned neuron permutations can "re-basin" independently trained models, while Navon networks that enforce cuuuoonpperutationn equiv- ariance for MLPs. Subsequent studies extend this principle: Zhou et al. (2024) apply permutation- equivariant functionals to CNNs, and et al. (2024); Kotinas al. (2024) represent diverse archi- tectures as neural graphs, using GNN updates that naturally respect permutation symmetry, albeit only for MLPs and CNNs. In contrast to methods that ingest trained networks for analysis or editing, our approach is generative: meta-learner maps user intent to a complete set of weights for fixed template, producing an executable network. Moreover, we extend the neural-graph formulation and symmetry-awar message passing to full nrunsformers-oot only MLPs or CNNs multi- head attention, residual pathways, and layer normalization.
al. (023) introduce DWWNNts--wight--spe
PROBLEM SETUP
Programs as (S, We model any executable program as pair (S. of a fixed structure S (architecture or grammar) and parameters 0. A valid program must satisfy
syntax: valid syntax(S, true,
semanties: V(X,Y) Du exec (, Y.
Symbolic programs: Verification. With slot budget T. parameters are token sequences 0 E Oaym over a finite vocabulary. The execution map exce(S, 0, is evaluated via compiler or interpreter and returns discrete pass/fail outcomes. This signal is non-graded and non-local: it reports what failed, not how to adjust 0. Search. Because tokens are discrete and syntax entangles with semantics, small edits can alter either or both; the objective landscape is piecewise constant.
Neural programs: Verification. Let T be the number of scalar parameters in S: neural parameters are / E O neu R.. The execution map is the forward pass ms,9. Let U denote the intent and Dū the corresponding data distribution over (X,Y) The semantic risk is
Csem 0) = Ex ( (m),,
is differentiable in 8, providing dense, graded signals that convey both direction and magnitude of improvement. Optimization. Fixing S yields valid syntax(S, 8) for all 0 RT,", removing syntax as a constraint during learning and disentangling it from semantics. Cast synthesis as continuous weight-space optimization with informative gradients, rather than discrete token search.
Trade-offs and scalability. We explicitly acknowledge two trade-offs.
Approximate semanties. By moving from discrete code to real-valued weights, correctness becomes approximate (e.g., a high-accuracy classifier rather than a Boolean proof). In return, we gain true scalability: gradients provide dense supervision, training is efficient, and we avoid DSL engineerin fixiggg S defines the space, and synthesis reduces to learning 0 from intent.
Human interpretability. Neural programs are not line-by-line human-readable like symbolic code. However, they are still deterministic, closed-form computations, and a substantial toolbox exists to analyze them. Attribution and feature-bast based methods (e.g.. LIME, SHAP, Grad-CAM), concept- based analyses (TCAV), and mechanistic studies of circuits and weight-space structure provide mul- tiple lenses for inspecting mss0 and relating parameters to behavior (Ribeiro et al.] 2016 Lundberg
& Lee 2017
Selvaraju al. 2017 Kim et al. 2018, Olah et al. 2020) Nanda et
2023)
4

---

Published as a conference paper at ICLR 2026
4 LEARNING TO SYNTHESIZE NEURAL PROGRAMS
Meta Learning Setting. Let U C Rd be the encoded intent space and ( C IRT the weight space for S. A meta-learner (hypernetwork) Ms uu - (SS) maps 0 = Ma(u), yielding the composed system F(u,x) x) = mss (M)
Assumption (realizable template). A task task family is realizable by a fixed, over over-parameterized net- work template S*
V exec (S", (S", X) -Y (X,YY) Dy.
With S* fixed, synthesis reduces minimizing the expected semantic loss over weights,
(0) = E(X,Y) [(exec(5* ((eexe((",,,)) ,0, y)]:
min Lxem (0), BERZ
where L is differentiable loss (e.g., mean squared error) be minimized by gradient descent.
We further show that, under common conditions on the task family, the learnability of an approxima- and weight-perturbation results; we summarize these guarantees and give formal statements in App. A While of interest to some readers, our emphasis here is practical-the design of symmetry-aware hypemneworr (Meta-GNN) and its evaluation on increasingly challenging synthesis suites suites-which we develop next.
tion for intent- 0 follows from standard universal-approxxmattio
4.1 AMBIGUITY IN WEIGHT SPACE DESIGN STRATEGIES
Even with fixed architecture S*, many distinct parameterizations implement the same function. Two ubiquitous sources of ambiguity are:
Multiple minima. Gradient-based training can converge to different that attain (near-) semantic loss; denote the set of such solutions by O.
Neuron permutations. Layerwise permutations of hidden units map any valid 0 to another nθ with identical network function; let be the permutation group.
We adopt two complementary design principles:
End-to-end training on semantie loss. Because the execution map is differentiable in weights, backpropagating the task loss through Md) å leverages continuity to learn stable weeehtt mapping without any post-hoc alignment.
Symmetry-aware inductive bias. Respecting permutation symmetries collapses equivalent param- eterizations and reduces the effective search space. Concretely, tying parameters across permutation- equivalent groups and using structure-aware updates substantially reduces the redundancy in super- vision.
4.2 PERMUTATION EQUIVARIANCE VIA NEURAL GRAPHS
We operationalize symmetry citreness-indedd broader structure awareness-by casting S as neural graph and deriving an update rule equivariant its automorphisms.
Equivariance and invariance. Following Section of Navon et al. (2023), we recall the basic notation and definitions. Let G be group and let V, W be vector spaces. The general linear group GL(VV denotes all invertible linear maps V - V. A representation of G on V is homomorphism P1 : > GL(V), where GI.(V) denotes the group invertible linear maps V - (i.e., invertible matrices in coordinates). Analogously, p2 G + GL(W) describes the action of G on W. A map
L: V W is said be G-equivarian
v) = p2(g)
Vg G, VEV.
Intuitively, applying a group symmetry before or after L yields the same result. special case is G-imvariance, where P2(g) is the identity for all g (e.g.. W IR), so L(p (g)v) I(v).
5

---

Published as a conference paper at ICLR 2026
represent the fixed template S* as a directed multigraph G* = (V, E), with nodes for biases and edges for weights. A neural-graph automorphism à V + V is node relabeling that (i) preserves adjacency, (ïi) respects node types (inputhidden/ourtuu/bbaaa) and (iii) preserves weight-sharing constraints. Each automorphism induces consistent permutation of parameters via (e(i, (Φ(i), ((j)), generalizing hidden- unit permutations in MLPs and channel permutations in CNNs. Crucially, automorphisms leave the instantiated network function unchanged, and message-passinng GNNs defined G* are provably equivariant to these induced parameter permutations (Lim et al. 2024).
Neural graphs and automorphisms. To unify these
4.3 STRUCTURE-AWARE WEIGHT SYNTHESIS
Motivated by the structure-aware view developed considerations we present three meta-architectures with increasingly strong structure awareness:
graph formulation and symmetry
Notation. Let U denote the encoded user intent embedding. All maps f(-) are MLPs.
Meta-MLP (structure-agnostio)) Let the target network be (S*, 0), and let T be the total number of scalar parameters in A single meta-MLP parameterized by yp predicts the weights, given intent embedding U, as flat vector:
fMMP(U::p) ER..
Meta-gMLP (group-aware), Index parameters by slots & {l,..., T) with group map g(t) E g and a fixed per-weight positional encoding p.. Use group-tied encoder parameters fsg gggg and a global mixer with parameters Pmizer:
he =(U pr Po()),
-= -[h (miier)
Meta-GNN (group- and structure-aware), Represent the template S parameter graph G+ EV) with neighborhood (ii)) == {j V (i.j) E E or (j. E E Nodes (bias slots) and edges (weight slots) have per-slot positional encodings Pv(e), PE(A,)) and belong to groups (i). (E(L)) as in Meta-gMLP. Let (t), (i, E Rd embeddings after steps. Use group-tied encoder parameters (oee and decoder parameters (ee together with global parameters for messaging and updates, denoted (maag "Pppa and Pupdd See Fig, | for a visual overview; the update modules Supd are omitted there for space.
Initialization (shared within groups).
(1) [(U |I ))
hi.j).) = S(U (PE)) gg
Message passing (permutation invariant aggregation). For = I....,k.
mf AGGGeNn)) f (b (i), hi) (i.j)s Pmsgg, hi (i) (h/ (i), m/; plpd)
(i, = (b) by (i), bz (i, (1.J): (ppd).
Decoding (shared within groups).
6, = (b (
iig = f P (E
Structure awareness in learning. Without positional encodings, Meta-GNN is permuta- tion-equivariant under automorphisms of the neural graph. Let Aut(G") be the automorphism group of G* and (e()) the induced permutation of the flattened parameter vector. If (i) all Pence Pmagy Pupd; Pdee are shared within groups and (ii) AGG is order-invariant, then for any à E Aut(G*),
@(U + (olo))(U
Ō(U:6. G*).
Unlike prior work that encodes trained networks (Navon et al. 2023 Lim et al. 2024 Zhou et al., 2024). we generative: the meta-learner must emit a concrete 0 that is written into fixed tensor slots of S*. In this setting, strict permutation permuration-qquivariannc equivariance incompatible with deployment: hidden units remain exchangeable, there is no permutation-camonte
way to assign scalars slots, just as training a vanilla MLP requires asymmetric initialization to avoid neuron exchangeability. Following the Transformer's design (Vaswani 2017), we break symmetry only at initialization by

---

Published as a conference paper at ICLR 2026
adding positional encodings to the initial intent embedding (cf. Fig. upper-left). The Meta-GNNY encoding. message passsng.. and decoding modules share parameters within groups and are other- wise permutation germutationiivvariant. Thus, the practical gain
classical (flat) hypernetwork (Meta-MLPP is structure awareness: (i) permutation-an -awaee -waree learning via invariant aggregation and group tying, and (ii) locality through message passing over graph neighborhoods. Together, these choices explic- itly exploit the computation graph, preserve target-network expressivity (Navon et al. [2023] Lim al. 2024), and collapse symmetry-induced redundancy-tthreeb shrinking the effective search space that flat Meta-MLP cannot model.
EXPERIMENTS
5.1 BENCHMARKS
We evaluate three settings (Table chosen to pair classical target architectures and losses MLP+MSE, Transforme++CE, seq2seq+CE with progressively richer intent structure. SUMFIRST-n and ADDMOD-D each form single program family with single varying parameter (the full spaces are cumfirst_n, E [1, 10), and addmod_p, primes p < 100), yielding clean, low-entropy program domains. By contrast, the ID-ARC subset is multi-family: intents (family, parameter) pairs spanning transformations such as Flip, Hollow, Fill, RecolorBar( 9), and ght 9) operations easy specify yet non-trivial to implement robustly. Repre- sentative families are illustrated in Fig.
(Fill__)
(Flip,
(Hollow,
(MoveRight, 2)
(RecolorBar, 3
Figure 2: ID-ARC subset (illustration). Example families (e.g., RecolorBar, MoveRigh, Flip) with intents specified as (family, parameter) pairs referenced in Table |
Field
SUMFIRST-R
ADDMOD.P
ID-ARC (mixture)
Program space
V
Σt-r La
y (a + mod
Task-specifio ID transformations
Input/ output
List 20 y
Pair (a, b) + y. a, layer Transformer
grid fixed length
Target network
hidden layer ReI.U ML.P
Seq2seq Transformer
# Woights
193
6048
17 17.568
Loss lunction
Mean squared eruor (MSE)
Cross-entrupy (classification)
Crus-entropy (Seq2Seq)
User intent
Prefix length n.
10)
Prime modulus p 100
(family, parameter)
Table 1: We deliberately select three representative combinations of architecture and loss demon- strate that our pipeline is agnostic to both model architecture and loss. For ID-ARC, the user intent specified by program family (e.g., Recoleereeee (erol) together with parameter (e.g.. 2).
Positioning w.r.f. symbolic DSL benchmarks: our setup is DSL-free, mapping intents directly to neural weights, so results are not directly comparable to systems whose performance hinges on the chosen DSL. Unlike large combinatorial DSL spaces, family with one (or few) intent variables (n, p. or (family, parameter)) target OOD over intent and permit mechanistic analysis. Still, SUMFIRST-n and ADD are spirit-close to canonical SyGuS tasks, and ID-ARC extends to more realistic sequence transforms.
compact program spaces-on
Data splits. For each benchmark, intents (e.g., or partitioned into rain held-out. Within each intent we form four partitions of 1/O pairs: train (used to update the meta-learner), test (new (X, Y) for the same train intents, measuring lnput-level generalization at fixed intent), validation
7

---

Published as a conference paper at ICLR 2026
SUMFIRST-n
ADDMOD-P
ID-ARC (MIXTURE)
Model
Train
Test
OOD Test
Train
Test
OOD Test
Train
Test
OOD Test
direct MLP
1,000
1.000
0.140
0.928
0.916
0.699
direct Transformer
0.849
0.850
0.682
0.994
0.996
0.603
Meta-MLP
1.000
1.000
0.145
0,996
0,994
0.947
0.953
0.956
0.625
Meta MMP
0.998
0.996
0.344
0.935
0.923
0,883
0.932
0,942
0.610
Meta-GNN
1.000
1.000
0.866
0.995
0.992
0.943
0.976
0.978
0.713
Table 2: Best-seed accuracies. Bold indicates the highest OOD Test accuracy; underline the second highest. A direct Transformer baseline is omitted on SUMFIRST-n (raw-integer inputs are ill-suited to sequence models). A direct MLP baseline is omitted on ID-ARC (seq2seq loss incompatible with naive MLPs). For the full seed distributions and variance analyses see App.
(unseen intents used only for model selection), and OOD test (unseen intents held out for final reporting, measuring zero-shot generalization across intents).
In ID-ARC (Fig. 2). some families are not parametric (Fill, Hollow, Flip) while others parametric (RecolorBas MoveRight) For the latter, training covers parameters {1 6, 8,9) and the held-out parameter is used exclusively for OOD test.
Primary metric: percentage of solved instances. For SUMFIRST-n we round to the nearest integer and check equality; for ADDMOD- and ID-ARC we exact match. We have trained each model with ten random seeds to assess variability to weight initialization as well as a means of increasing the odds of converging weights that perform well on OOD test instances; both of these uses of the random seeds are common (Bethard 2022).
Baselines and capacity matching. Direct prediction trains XU, + Y on train only (no meta component). Meta-MLP is the classical hypernetwork baseline: a MI.P that maps U + 0 as a flat list of parameters. Meta-gMLi (group-aware) and Meta-GNN (neural-graph, equivariant) add progressively stronger structural bias while sharing the same target ms,e. All models are capacity- matched to comparable trainable parameter counts (details in App. C
5.2 RESULTS
Is neural synthesis even possible? First, note that all meta models generate semantically correct programs for in-distribution test set in a single forward pass across benchmarks, including ~ 1-7 parameter Seq2Seq Transformer for 1D-ARC. Table 2 that test accuracies are close to 100% across models (rows) and benchmarks ("Test" columns). The real challenge, however, is in out-of- distribution generalization, as we will discuss next.
Benefit over direct prediction. Across all three benchmarks, meta-learners substantially outper- form direct models on OOD test (unseen intents) under capacity matching. This supports our claim that predicting programs (weights) yields more reliable intent generalization than predicting out- puts. In ADDMOD-p. we further show that the meta-learner recovers a closed-form + mapping (Sec. 5.3). rather than memorizing instances.
Role of structural bias. Meta-GNN consistently outperforms other hypernetwork variants. By viewing the target neural graph and using structure-aware message passing, Meta-GNN reduces weight-space redundancy so that supervision less fragmented. This structural bias yields more consistent learning signals and, empirically, stronger OOD performance.
Clarification on Meta-gMI.P's ranking. Although Meta-gMIPP incorporates grouping, does not consistently rank second on ADDMOD-p and ID-ARC. Under fixed meta-network budget, its global mixer fmixer processes [hr]]= E RT which forces dmeta to shrink as T grows and creates capacity bottleneck width (independent of T), and Meta-GNN avoids global concatenation by local message passing with group- group-shared shared modules.
larger S*. By contrast, Meta-MLP's capacity scales with its
Multi-family synthesis (ID-ARC), Beyond single-famin arithmetic, the ID-ARC mixture tests more "rral-world" regime with diverse families and intents (Fig. 2 Table 3 shows that Meta-
8

---

Published as a conference paper at ICLR 2026
Fill
Flip
Hollow
MoveRight
RecolorBar
Model
Test
0OD
Test
00D
Test
0OD
Test
0OD
Test
OOD
Direct Transformer
1.000
1.000
1,000
1.000
1.000
0.999
0,999
0.000
1.000
0.016
Mcta-MLP
0.995
0.992
0.138
0.127
0.996
0.997
0.999
0.996
1.000
0.000
Meta-gMLP
0.979
0.967
0.150
0.165
0.965
0.962
0.986
0.981
0,987
0.000
Meta- Meta-GNN
0.994
0.992
0.905
0.898
0.789
0.774
0.990
0.921
1.000
0.000
Table 3: Per-family test and OOD test success 1D-ARC (mixture) for the best run.
Per-intent PCA at p B3 (tokens 0-97)
Global PCA of flattened input-embedding matrix
15
10
Train primes
Test primes.
PC (34. 7% var)
PC (4.6% var)
40
-2
50
-2
-1
0
-20
-10
10
20
PC (43.1%
PC (95.1% var)
Figure 3: Clock geometry of the synthesized Transformer: Left: per-intent PCA for the predicted input-embedding matrix We of unseen p=83. Right: global PCA across primes using the W (p))).
GNN attains high OOD accuracy across most families. Notably, RecolorBar (color) is un- learnable for all meta-learners because the intent is categorical (no ordinal structure for unseen colors), whereas direct Transformer can in principle match the color token via attention; in MoveRight (dist), direct Transformers fail to extrapolate while all meta-learners succeed; and in Flip, only Meta-GNN reaches high accuracy thanks to structure-aware learning that reduces weight-space redundancy.
Overall, the empirical results align with our research questions: (i) neural programs in weight space can be synthesized end- andtto-end; (ii) continuous intent- A mappings are learnable and semantically stable, and (iii) structure-aware inductive bias (Meta-GNN) is key to strong OOD generalization across both single- and multi-family settings. Additional distributional analyses appear in App. $D
5.3
MECHANISTIC ANALYSIS
For ADDMOD-p, the target network admits a known closed-form ("clock") solution (Nanda al. 2023 Zhong et al. 2024): the embedding maps (a, b) (cos 0, sin 0) with 0 + yo: the core (atn++MPP) performs an angle-sum of the two embeddings; and the logit head outputs cos (w(a+b-c)) Because downstream blocks are crrquenncyagnooste a perfect meta-learner need only learn p (WE(p), (p())), leaving the angle-sum circuitry unchanged.
Per-intent geometry (unseen p). For an unseen modulus (p=83), PCA of token embeddings shows the expected clock geometry (Fig. 3 left), consistent with the analytic construction.
Across-intent geometry + closed form. Flattening W Wg(p) across primes and running a global PCA reveals a thin ID trajectory in PC2) (Fig. 3 right), indicating that the synthesized weights are governed by a single latent angle of the form + y. Fitting this trajectory yields the closed-form dependence with coefficients and goodness-of (see full derivations in App. $
112
cos (5.667//pp-.622
B+
11))
(p) 23,43x
PC2(p) b+ B Σ
Ba. 2k +
Σ
a 2j+1
(overall R² > 0.94).

---

Published as a conference paper at ICLR 2026
SCOPE AND LIMITATIONS
Our benchmarks are intentionally small in symbolic form: three synthetic but structured families (SUMFIRST-n, ADDMOD-p, and 1D-ARC) with fixed architectures and standard losses. This design lets us isolate the feasibility of strict zero-shot intent- 0 synthesis and perform mechanistic analyses (e.g., recovering the clock circuit on ADDMOD-p), rather than confounding the results with large, noisy domains. Although the task descriptions are simple, each intent corresponds to point in high-dimensional continuous weight space, and the meta-learner must discover smooth structure there without any hand-crafted DSL.
second limitation is architectural: each task currently uses a single, relatively shallow target tem- plate (1 (1-hidden-laye) MLP, 1-layer Transformer, or I-layer seq2seg Transformer). We chose these templates to keep the synthesized programs interpretable and leverage known ground-truth mech- anisms. In our framework, scaling corresponds to enlarging the neural graph G* (more nodes and edges), so Meta-GNN's cost grows roughly linearly V |+|/; applying the same synthesis prin- ciple to deeper mixed architectures is left to future work.
Finally, we not compare directly to classical or neuro-symbolic program-synthesis systems. Those methods operate over hand-crafted DSL template language, whereas our setting is DSL- free: the program is the weight vector. Introducing a bespoke DSL solely to run such baselines would make performance depend largely on the DSL design and obscure the weight-space mecha- nisms we study, so we instead use capacity-matchet direct neural models as baselines.
7 CONCLUSION
We asked whether program synthesis can operate directly in the continuous weight space of neural networks. Our findings answer in the affirmative. Treating fixed architecture and its weights neural program (NeuroP), we showed that (i) From intent, we synthesize MLPs and Transformers that are semantically correct in single forward pass. To the best of knowledge, this is the first demonstration of this possibility: and (ii) incorporating structural bias-via -vi structure- SuN-substaytially improves OOD accuracy on unseen intents over direct intent + imput baselines, with mechanistic evidence ADDMOD-p that the learner internalizes stable higher-order map- ping > 8. Taken together, the results position neural weights as an expressive, differentiable, and learnable program modality, and NeuroP principled route to scalable synthesis. Our future work will focus on extending the proposed framework to more complex program synthesis applica- tions. In particular, realistic user intents could be expressed in natural or other structured languages. Alternatively, user intent could be expressed through a few example input-output pairs, as is the case in the Abstraction and Reasoning Corpus (Chollet 2019). These complex modalities require more sophisticated hypernetwork.
Meta-
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by the Institute Information & Communications Technology Planning & Evaluation (ITTP) grant funded by the Korean Govermment (MSTT) (No. RS-2024-0045788,, National Al Research Lab Project).
REFERENCES
Alessandro Abate, Haniel Barbosa, Clark Barrett, Cristina David, Pascal Kesseli, Daniel Kroening, Elizabeth Polgreen, Andrew Reynolds, and Cesare Tinelli. Synthesising programs non-trivial constants. Journal of Automated Reasoning, 67(2):19, May 2023. ISSN 1573-0670. doi: 10.1007/510817-023 09664-4, URL
https://dci... http:///dcc..rg/1//// 02000772110700000066-------
Samuel K. Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo
permutation symmetries, 2023. httpt:h/paaaaaae owt. wwtp///g/////////o// 04836
Rajeev Alur, Rastislav Bodík, Garvit Juniwal, Milo M. K. Martin, Mukund Raghothaman, Sanjit A. Seshia, Rishabh Singh, Armando Solar- Lezama, Emina Torlak, and Abhishek Udupa. Syntax- synthesis. In Formal Methods Computer-Aided Design, FMCAD 2013, Portland, OR, USA. October 20-23, 2013, pp. 1-8, 2013. B.R.hhcp:://eeeee
leee. wwg. dww.ment.tt/n//oooooo/ /66eennt//66999 679887
Jacob Beck, Risto Vuorio, Zheng Xiong, and Shimon Whiteson. Recurrent hyperneworks are surprisingly strong in meta-rl. Advances in Neural Information Processing Systems, 36:62121-62138, 2023.
Steven Bethard. We need to talk about random seeds. arXiv preprint arXiv:22 13393, 2022.
Woojin Cho, Kookjin Lee, Donsub Rim, Noseong Park. Hypernetwork-basee meta-learning low-rank physics-informed neural networks. Advances in Neural Information Processing Systems, 36:1 1219-11231, 2023.
François Chollct. On the measure of intelligence. arXiv preprint arXiv: /rXiv/191.001547,200
Guofeng Cui and He Zhu. Differentiable synthesis of program architectures. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.). Advances in Neural Information Processing Systems, 2021. URL http:///pppneereeeeeeettttttuoee
Teja R Emireddy. Neurosketcher: Synthesizing non-differentiable programs with non-differentiable loss func- tions. University of Alberta Thesis, 2023.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp. 1126 1135. PMLR, 2017.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016.
Sumit Gulwani, Oleksandr Polozov, and Rishabh Singh. 2017, doi: 10.1561/250000010.
David Ha, Andrew Dai, and Quoe Lc. Hypemetworks. arXiv preprint arXiv: 160.009100 2016.
Kangjing Huang, Xiaokang Qiu, Peiyuan Shen, and Yanjun Wang. Reconciling enumerative and deduc- tive program synthesis. In Proceedings of the 41st ACM SIGPLAN Conference on Programming guage Design and Implementation. PLDI 2020, pp. 1159-11 1174, New York, NY, USA, 2020. Associa- tion for Computing Machinery. ISBN 978145037613 7613666666 36.
7/doi .aw.iorg///i 10.5/ 1 33855/3384411 3386007
doi: 10.1 00.1145/33854113333661
URL https:
Been Kim, Martin Wattenberg. Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pp. 2668 2677. PMLR, 2018.
Jinwoo Kim, Qinheping Hu, Loris D' Antoni, and Thomas Reps. Semantics-guided synthesis. Proc. ACM Program. Lang.. S(POPL), January 2021. doi: 10. 145/3433333311 URL /tll:tttss//ddd.rg///1/
3434311
Miltiadis Kofinas, Boris Knyazev, Yan Zhang. Yunlu Chen, Gertjan J. Burghouts, Efstratios Gavves, Cees G. M. Snoek. and David W. Zhang. Graph neural networks for learning equivariant representations of neural networks. In The Twelfth International Conference Learning Representations, 2024. URL https:
//openrevis net. forumm2 d-o06= sMyDHE

---

Published as a conference paper at ICLR 2026
Moshe Leshno, Vladimir Ya. Lin, Allan Pinkus, and Shimon Schocken. Multilayer feedforward networks with nonpolynomial activation function can approximate any function. Neural Networks, 6(6):861-888 -867, 1993. doi: 10. 101600///8993-11 10011///0990000000888033333 6080105)80131-5.
Derek Lim, Haggai Maron, Mare T. Law, Jonathan Lorraine, and James Lucas. Graph metanetworks for pro- cessing diverse neural architectures. The Twelfth International Conference on Learning Representations. 2024.
/ttp:/::////eeeeeeeeeeee net/ forum? d-i - ]K5hyxsnn
Scott Lundberg and Su-In Lee. unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017.
Sadanand Modak, Noah Tobias Patton, Isil Dillig, and Joydeep Biswas. Synapse: Symbolic neural-aided preference synthesis engine. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 07529927737,, 2025.
Neel Nanda, Lawrence Chan, Tom Lieherum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechamistic interpretabillity. arXiv preprint mrXiv:2301.05//1 2023.
Aviv Navon, Aviv Shamsian, Idan Achituve, Ethan Fetaya, Gal Chechik, and Haggai Maron. Equivariant architectures for learning in deep weight spaces. In International Conference on Machine Learning, pp- 25790-25816. PMLR, 2023.
Maxwell Nye. Luke Hewitt, Joshua Tenenbaum, and Armando Solar-Lezama. Learning to infer program sketches. In International Conference on Machine Learning, pp. 4861-4870, PMLR, 2019.
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 5(3):e00024-000 2020.
Emilio Parisotto, Abdel rahman Mohamed. Rishabh Singh, Lihong Li, Dengyong Zhou, and Pushmeet Kohli. Neuro-symbolic program synthesis. In International Conference on Learning Representations, 2017. URI
htttps://ppenrs ew net forum? i d-r W...FF......
Andrew Reynolds, Haniel Barbosa, Andres Nötzli, Clark Barrett, and Cesare Tinelli. cvc4sy: Smart and fast term enumeration for syntax-guided synthesis. In Isil Dillig and Serdar Tasiran (eds.). Computer Aided Verification, pp. 74-83, Cham, 2019. Springer International Publishing. ISBN 978-3-030-22543-5.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. " why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference knowledge discovery and data mining, pp. 1135- 144, 2016.
Andrei A. Rusu, Dushyant Rao. Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-lcarning with latent embedding optimization. In International Conference on Learning Rep- resentations, 2019. URL httptt:///::::tneeeeet
net wwre//foormmicmm Bagkk
Konstantin Schürholt, Dimche Kostadinov, and Damian Borth. Self-supervised representation learning on neural network weights for model characteristic prediction. Advances Neural Information Processing Systems, 34:16481-16492 2021.
Daniel Selsam, Matthew Lamm, Benedikt Bünz, Percy Liang, Leonardo de Moura, and David L. Dill. Learning a SAT solver from single-bit supervision. In International Conference on Learning Representations, 2019.
http://wwwwwwwtteeeeeeeeeteteete Lor Lort:/oouummmddduu _
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradicnt- based localization. In Proceedings of the IEEE international conference computer vision, pp. 618-632 18-626, 2017.
Ameesh Shah, Eric Zhan, Jennifer J. Sun, Abhinav Verma, Yisong Yue, and Swarat Chaudhuri. Learning dif- ferentiable programs with admissible neural heuristics. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
Armando Lolamm Program TrroorramSymthssiiii Sketching PhD thesis, University of California, Berkeley, 2008
5777, (50-))):5111))7:777 hecpaarece.aaaanPooaaaccccceennn URL http://///////.o..////0//0000000
Armando Sww.Soolaa--aaaaaa Program
2013. doi: 10.1007711009-012--00000
Sho Sonoda and Noboru Murata. Neural network with unbounded activation functions is universal approxima-
Applied and Computational Harmonic Analysis, 43(2):233-263 2017. doi: 10. 10166/jacha.200 wtt...011......0 12.00...
12

---

Published as a conference paper at ICLR 2026
Yu-Lin Tsai, Chia-Yi Wei, Hung-Yi Yi Tseng, and Hsuan-Tien Lee. Formalizing generalization and adversar- ial robustness of neural networks weight perturbations. Advances in Neural Information Process- ing Systems (NeurIPS), 2021. URL http:ttt////rrcceddlgggg/
Vaswani. Attention you need, Advances in Neural Information Processing Systems, 2017.
Johannes von Oswald, Christian Henning, Benjamin Grewe, and João Sacramento. Continual learning with hypemetworks. In International Conference on Learning Representations, 2020. URL https:
//openreview net net forte//rummttem Ld-rrnnekk--
Tsui- Wei Weng, Huan Zhang, Pin- Yu Chen, Wei Ye, Luca Daniel, Cho-Jui Lin, and Cho-Jui Hsich. Towards certificated model robustness against weight perturbations. In AAAI Conference on Artificial Intelligence (AAAI), 2020. URL httpss/s://// aaai org/ index pwtpppph/paapaaaeeeeetep /tta////aaatic///l..e.e...e 6105
Kaidi Xu. Huan Li, Sijia Zhang. Yihan Wang, Suman Jana, and Cho-Jui Hsieh.
Automatic perturbation analysis for scalable certified robustness of neural networks. In Advances in Neural Information Process- ing Systems (NeurIPS), 2020. URL httphttt//porceediiiieeee ngs. neurips... htt.:///......ooooooo/////////// cctppappr//2222/ff1ee//// Dcbc5671ae26f67871 cb9 1eeeeeebe1eefefff111e--eee-e Papeec
Yudong Xu, Wenhao Li, Pashootan Vaezipoor, Scott Sanner, and Elias Boutros Khalil. LLMs and the ab- straction and reasoning corpus: Successes, failures, and the importance of object-based representations. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL htt.:/tts://ppeeeeeeee net htt//erum?? i EdEE888ySSPP
Dmitry Yarotsky. Error bounds for approximations with deep relu networks. Neural Networks, 94:103-114, 2017. doi: 10.101 10.e0t66..euuee.2217 7.07.002.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. Advances in neural information processing systems, 30, 2017.
Dominic Zhao, Seijin Kohayashi, João Sacramento, and Johannes von Oswald. Meta-learning via hypernet- works. In 4th Workshop on Meta-leearning at NeurlPS 2020 (Metalearn 2020). NeurIPS, 2020.
Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas. The clock and the pizza: Two stories in mecha- nistic explanation of neural networks. Advances in Neural Information Processing Systems, 36, 2024.
Allan Zhou, Chelsea Finn, and James Harrison, Universal neural functionals. arXiv preprint orXiv: 2rXi:24022.05232. 2024.
13

---

Published as a conference paper at ICLR 2026
A FORMAL GUARANTEES FOR WEIGHT-SPACE SYNTHESIS
A.1 FORMAL GUARANTEES
Meta Learning Setting, Let C 1e be the encoded intent space and ((S) C R² the weight space for
S. A meta-learner (hypernetwork) Mr - ((S) maps u - 0 Ma(u), yielding composed system
Fu, (us
A.2 ASSUMPTIONS
A1 (Realizability). For u E V there exists 0 ((S) such that ms,o(X)
(X) = Y for all Y) Du.
A2 (Continuity). There exists a continuous target mapping w - @(S) on compact U.
A.3 UNIVERSAL APPROXIMATION FOR INTENT WEIGHTS
By universal approximation for ReLU networks (Leshno et al. 1993 Sonoda Murata 2017 Yarotsky 2017): Theorem (UAT for hypemetworks), For any > there exists a (ReLU) hypereework M, + ((S)
such that
sup (M) w(u)ll < wEd
Thus, under A1 A1-A2, any continuous intent-> 0 map is uniformly learnable by hypernetwork.
A.4 LIPSCHITZ SEMANTICS: FROM PARAMETER TO FUNCTION ERROR
For common architectures, certified pertutt-ettubttioon bounds can be obtained from layerwise operator norms
LiRPA certificates (Weng et al. [020 Tsai et al. [202]] Xu et al. 2020). We summarize these
Lemma (Parameter-Lipschiee semantics). There exists T.g <. Do (on a parameter region O and bounded imput domain X') such that
sup [1) mss,rr (x)]l S La 110 0'11 v0,0' E
zex
Corollary 1 (Uniform control of semantic error). IfsuPueu (M - w(u)ll Eu. then
sup sup (M) ms (1)] T.g Eg-
well ze
For classification, if MSsu(u) has margin (u) 0 X.. any 7u)) Le preserves the predicted labels on X..
Together, Theorem and Corollary esstablish two formal pillars: (i) appreximabiliiy of continuous intent-+0 mappings by ReLU hypernetworks, and (ii) semantic stability under small parameter errors- errors-reeiselly the regime targeted by end-to-end gradient-based training. While these results do not address sample complexity statistical rates, empirical sections demonstrate that such mappings be learned effectively in practice. Full proofs are deferred Appendix A.5
A.5 PROOFS
This appendix provides fully formal, self-contained arguments with explicit and codomain handling. use only standard facts about (i) universal approximation on compact sets and (ii) layerwise Lipschitz composition.
Standing assumptions, norms, and notation. Let 24 C Rd be compact and X C dd be bounded. Fix feedforward ReLU architecture
ha(x) = x,
he(x) = p(Wehe- be), -- M,
with Lp-Lisschizz (ReLU: L, 1). The model is mss hxs (:; 0) with parameters 0 = ((Wr, bx) M flatten per-layer tensors through fixed linear bijection vec (ueser() + R² and write again ( C R² for the admissible set in vectorized form. All proofs invariant this reshape.
Norms. Vectors use ||- la2 matrices the induced operator norm ||2- On parameters we the block
M
1001pax pax
([]Wal + [[bell2).
14

---

Published as a conference paper at ICLR 2026
that E. |F ||Abel12) 110 0'l par. When measuring uniform approximation of vector-valued maps 4 Ie² we use ol K² unless stated otherwise. Since all norms on Ie are equivalent, there exists Opart-o E (0, oc) with | S Opare for E
Assumptions (restated), (A1) (Realizabilitt)) For each a E 2 there exists E ((S) such that ms.e( y for (X,Y) Y) ~ Du- (A2) (Continuity) There exists a continuous target mapping w u + ((S) on compact U4
We also the compact image == (u) @(S) and, for å > 0, the closed 6-thickening K {0 R" K.) s 6).
A.6 PROOF OF THEOREM (UAT FOR HYPERNETWORKS)
Theorem (Theorem restated with codomain). Let U C Rd be compact and w : U - (SS) S R be continuous. For any 0:
(a) (Unconstrained parameters) (f((S)) = IRT, there exists a ReLU hypernework M u - R such that sup [[Ma(u) w(u)ll <. weld
(b) (Box-constraince parameters) ( = IT la,, by] is product box, there exists ReLU hypernetwork Ma U + (SS) with the same uniform approximation guarantee.
More generally, for any closed convex (SS) and any 1) there exists ReLU hypernetwork M :u RD such that sup,, d (u), O(S)) & and sup,, [I (u) w( < <Etn
Proof. Write (wu) (wi
with each We E C(U)..
Step I (coordinatewise approximation). Let (x)) max =) be ReLU. Since a is non non- polynomial and we allow biases, single-hidden-han ReI.U networks dense in C(2)) under the uniform metric (Leshho et al. Thm. 1). Thus, for each & there exists ReL.U network N₂ IR with
1993
sup NN wx wx)))) <.
wel
Step 2 (stacking). Define M(u) (N), Nr(u)) 24 Rº vector-valued ReL.U network). Then sup,, [l) w(u)l mo <.
Step 3 (codomain enforcement). (a) If (S) R² take M3 == M.
(b) (S) = II (ae, bi], post-compose with the ReLU clamp
clampp/..))())
+ a(r a) o( b)
coordinatewise: (TT())): = clamp (ar is I-Lipschitz and fixes ((S), hence
(.) (2.). Then IT (shallow) ReL.U network, IT M :4 - O(S), and
sup [[M(u)) w(x))l> I sup (MM w())l>> <
Set M4 = 11 M.
For general closed convex (S),, let The be the Fuclidean projection (non-expansive and identity on ((S)). Then sup,, [[T])(())) (u)ll> e and TTe M maps into @(S), although The need itself be exactly
representable by a finite ReLU network unless (S)) is polyhedral. For any compact set 3 ((4)) and any
y > 0, approximate 1l uniformly on by ReLU network P,, within v obtain ( = P., with
stated properties.
Context beyond existence. for pol-polynomial activations with biases. Two later works provide stronger, complementary statements for context:
The proof above only Leshno al.'s necessary and sufficient density
Sonoda & Murata (2017). Give constructive ridgelet/Radon representation f(u) = a b) o0 b) du(0, b) and quantify uniform approximation via quadrature, including explicit control of weight norms. This strengthens classical UAT with integral formula and error accounting.
15

---

Published as a conference paper at ICLR 2026
Yarotsky (2017). Provides rales for deep ReLU approximation Sobolev classes: for f Wn, 1] there exist networks with depth ([log(]/e) and size (e=-d/n -d/n log(L/e)) achieving || fll S e, and proves fixed-depth lower hounds (i - (D(s--/(a(L-22 1/))( for nonlinear C² targets. We do not need these rates for the theorem, but they clarify depth/width-error trade-offs.
A.7 PROOF OF LEMMA D
SERRAMMTERR-LPSCHHEEE SEMANTICS)
Lemma (Lemma restated with norms), Let @ c RD be compact and X C B be bounded. Then there such that
exists Lg = (S x))
x² <
sup [ ms (x)ll2 & La 110 0'1lpar aEx
V0,0'
Prouf. Let = (|aa(a;)) (o00=;;00001110 AW; W, W/. Ab. be -ú Using p's Lp-Lisschiszness
and submuliipicaatiityy of the operator norm,
Ac(a) z Lp (llw (1)-1(2;;)) 0) + |Will2 Ar-1 11bbol2) Since X and θ are compact and is Lipschitz, there exist finite bounds (by induction)
He 2 sup sup (<) 0)1l. Be 2 sup |Wele.
(2)
6E8
Unrolling equation ! with equation[yvellds
M
M II (L, joE+1
(M(x)
Σ 2-1
(LollAbella b,
All factors finite θ, hence there is Cs = Cs(, with
(x) S Cs >>>
< Cs 110
8 paar.
Taking supagx gives the claim with T.g Cs.
Architectural variants. For residual blocks he he-1 + p(Wehe-1 + be), equation becomes
Ae(x) < (1 + L,
L, [|Wa/l + Lp
so the product factor in equation changes to II (1 + B3)- Convolutions are linear maps; replace
|/Wlll by the corresponding operator norm.
Certified alternative (LiRPA). Neight-perturbatie
LiRPA
yields
computable
constant
LiPLarr((0.. (for a chosen parameter-spac norm) such that
sup zex
m (r) mss.6 (=)) I LiP (9, 110 0'|l.
see Weng et
2020
2021) 1 et 2020). This can tighten Ly not needed existence.
A.8
PROOF OF COROLLARY (UNIFORM CONTROL SEMANTIC ERROR)
Corollary (Corollary restated), Fix 8 >> 0 and ser = k which compact. Suppose
sup (M) w(u) I 6. NEM
Then
sup sup [Ims.M,(o)(()) (2)01))))112 I Loce. well x
where Lo = (,, is from Lemmal)
For multi-class classification with logit outputs in R and Eox output norm,
7(1) = inf (fu(zw((u)) - max (x) 0, ZER,
#'
then any e < (2L) guanantees that arg max, fo (u)) equals arg max, fo(sw(u)) X..
Proof. By construction U C e, so Lemma 1 applies pointwise u and yields the uniform bound
after taking suprema. For the margin statement, let duguu (z) = Mo()) [[:w[u))]] If (zgi(()) < 8. then the margin shrink by at most 28 (top logit down by 6, runner-up up by 6). With 5 = Tooo, requiring Logg 72 preserves the argmax.
16

---

Published as a conference paper at ICLR 2026
Using UAT with the parameter norm. If Theorem is invoked with || |loe but Corollary is stated in
|| parr use norm equivalence: choose the UAT accuracy E0/ /Cparre-oo that [|Ma( w(u) || par Eg uniformly in u.
17

---

Published as a conference paper at ICLR 2026
B NEURAL-GRAPH CONSTRUCTION
We convert any PyTorch network into a directed qulti--raph G (V, E) whose nodes carry bias scalars and whose edges carry weight parameters. The rules below are vechtecture-agnostie (sB. and are then specialised Transformers B.2). All code provided in neura aag.aph..pph....... py (Appendix)
B.1 CORE CONSTRUCTION RULES
Linear layers. Each weight matrix W Rmxx becomes directed edges (i j, wyy); each row-bias by a node.
Residual (skip) connections. As in Lim et al. (2024); Kofinas et al. (2024), skip edges are dashed, have fixed weight and do participate in message passing so that information can bypass intermediate layers.
Normalization layers. Batch/LLyerroom be rewritten as diag(r) x hence add one-to-one edges weighted by 7 and bias nodes that hold fs. This treats normalization exactly like diagonal linear layer and preserves permutation equivariance.
B.2 ATTENTION BLOCKS
A self-attention module is expanded into the sub-graph in Figure 6
Per-head parameter nodes. For each head h we create Qs, Ka, Va nodes that store the corresponding slices W2/K/V All heads share a single output-projeatio node O.
Score nodes and pass_thru edges. A dummy AttneS node is inserted to link Q₁ and K,. The dotted red pass-thru edges contain parameters; they exist solely tell the Meta-GNV that Qi and KA must be shuffled together. In other words they allow arbitrary head permutations.
align one2one edges. Inside a head, any the dread feature dimensions may be permuted. We express this by blue dashed al align_one2on to K,, row i). Like pass thru, they have weights and are masked out during message aggregation; their sole purpose is to inform the permutation-invarint update that within-head dimensions be shuffled independently other heads.
edges that connect corresponding neurons of two layers (c.g. Qn row i
Remark. Skip-connection and normalization handling is identical to the graph Meta-learnrr recipes of Lim al. (2024); [Kofinas al. (2024); only the attention-speciff pass_thru/al gn_one2one mechanism new, enahling our graphs remain IT-qquivariant for full multi-head Transformers.
18

---

Published as a conference paper at ICLR 2026
B.3 ILLUSTRATIVE GRAPHS
bar 6.36
Max 6:33
0.47
4.1
0.4
0.39
0.07
0.33
0.33
0.15
bras 0.22
0.1
0.66
9
Input normal
Figure 4: Illustrative neuron-level graph of toy 1-hidden layer MLP. The structure- nodes, weight edges, and their grouping-is is identical to the subnet used in SUMFIRST-n. but the layer sizes are reduced for elarity.
Input
normal
Figure 5: Adding residual (skip) connections to the MLP graph. Dashed edges have fixed weight 1. Layer sizes are again reduced for visual clarity.
19

---

Published as a conference paper at ICLR 2026
Attn,
Attn
Altn, Score
FC
Posinput
dot product
normal
stip one
align one2one
pass
Figure 6: Neuron-graph of single-layer, two-head Transformer (reduced dimensions). Red dotted pass_thru edges allow head permutations; blue dashed a iiniinooeeoooo one edges allow within- head dimensional shuffling. Both edge types are parameter-free and excluded from message updates.
20

---

Published as a conference paper at ICLR 2026
C DATASET, ARCHITECTURE AND PARAMETER DETAILS
This appendix records the dataset sizes, exact hyppe--praamttenn and trainable parameter budgets of every model used in our experiments. All use ADAM with learning-rate 10-3, batch size 256 and 50000 optimizer steps; only the ceattecctur-speciffe choices differ.
C.I DATASET DETAILS
Dataset
Train
Test
Val
OOD Test
ADDMOD-p
50000
112
1997
12.000
SUMFIRST-N
160000
8000
10000
10000
ID-ARC
00
16000
5000
5000
Table 4: Number of examples in each split for the three benchmarks.
ID-ARC meta-split - per-family/parameter composition. We include five families: three paramless (F1L1, 11,,, Plip, Ho ow) and two paramfuul (MoveRight RecolorBar), The generator holds out parameter 7 for OOD evaluation in paramful familics.
C.2 NEURAL PROGRAM TEMPLATES (mg)
SUMFIRST-n: 1-hidden layer ReLU MLP (10, 16, 1)= 193 weights.
ADDMOD-p: I-layer causal Transformer (dmod==16, Theads fex mex=2) + 6 048 weights.
The templates are fixed at Meta-train and Meta-test time; only their weights synthesized.
C.3 META-GNN COMPONENT ABLATIONS
To complement the theoretical discussion of structure awareness in $4.3 and address concerns about isolating Meta-GNN's design choices, conducted targeted ablation study ADDMOD-P with smaller Meta-GNN (fewer than 50k meta-parameters) for faster sweeps. The goal understand how sensitive performance to (i) initialization strategy, (ii) message aggregation, and (iii) edge directionality and parameter tying, rather than to exhaustively tune every component.
Ablation dimensions.
We vary four discrete hyperparameters:
init_mode {all, per-group, global): per-node, per- wei-teight. global initial embeddings for nodes/cdges.
altention-based, mean, or sum aggregation over incoming messages, combined with use_norm E (True, False) to toggle layer normalization in the prssage-passing block.
aggr E (attn, avg, sum]:
bidirectional (True, False}: whether to include reverse edges the neural graph.
rev_edge_man mode E {shared, separate, - --): when bidi rectiooonn wowoooaTruun reverse edges either share parameters with forward edges (shared) have their own parameters (separate); denotes the purely unidirectional case.
rums use the same training protocol and target template as main ADDMOD--pepprimenti results are reported as mean + standard deviation over five seeds.
Takeaways. Three trends emerge from Table
Group-awer initialization helps, but is not critical. All three initialization modes (all / per-group / global) with attention aggregation and hidirectional edges achieve similar high accuracies, with per-group slightly ahead. This supports choice of per-group emheddings good trade-off hetween performance and parameter efficiency.
Attention aggregation is consistently strong. Average and sum aggregation (avg/sum) remain competitive, but trail altention-based aggregation attention as the default aggregator the main experiments.
settings, especially without normalization. This justifies the
Bidirectionality and reverse-edge parameterization give small but consistent boost. Adding reverse edges helps over the unidirectional variant, and giving reverse edges separate parameters (separate) yields the best overall performance.
21

---

Published as a conference paper at ICLR 2026
init_mode
aggr
use_norm
bidirectional rev_odge_mode
best_test_qry.ace
all
False
True
shared
0.964 0.016
group
False
True shared
0.976 0.029
global
att
False
True
shared
0.969 0.021
per group avg
False
True
shared
0.949 0.032
per group avg
True
True
shared
0.964 0.013
per-group sum
False
True
shared
0.947 + 0.015
per-group
True
True
shared
0.954 + 0.012
per group attn
False
False
0.965 + 0.023
per-group
False
True
separate
0.979 0.014
Table 5: Meta-GNN ablations on ADDMOD-p with smaller meta-network. All configurations achieve strong performance; the configuration used in the main paper corresponds init per group, aggr = attn, bidirectiona = True, rev rev_edge mode separate.
Overall, the differences across configurations are modest: all achieve best_test.qry_ace in the 0.95-0.98 range ADDMOD-p with compact Meta-GNN, This suggests that the structure-aware design (ncural graph representation, permutation-eqqiiariin message passing, and group-tied encoders/decodders) is robust, and that our final configuration in the main paper reasonable, but not finely tuned, operating point. the main results we scale Meta-GNN up to roughly match other meta-architectures in parameter count under the same training budget, which slightly lowers absolute accuracies but preserves the same qualitative trends.
C.4 META-LEARNERS
The Meta-Icarner generates all weights of the above template; its own trainable parameters are listed below (they receive gradients during Meta-training).
Mis Miar be Mag Wggu Mao Pog
U
Meta MLP
MLP
Meta wMLP
MLP
May Mizi Mag Miss Mag- Pa-
U, Pos
MLP
MLP
Mau Mas Maai Mas Mags Mas. Bas
MLP
Meta GNN
Decode
U, Pos
Meta Networks
Target Network
Figure 7: Meta-learner architectures. From top to bottom: META-MLP (no structural input), META-gMLL (group & position tags), and META-GNN (full neural-graph connectivity). Boxes denote MLP blocks.
SUMFIRST-n
Model
Hidden configuration
trainable params
META MLP
(32, 128, 512, 256, 128)
259 650
META.gMMP
groups (32, 64, 128, 128) + mixer (128)
235 235
META.GNN
de 32, & 4 d, 226, 6
189 603
direct MLP (baseline)
259 449
22

---

Published as a conference paper at ICLR 2026
ADDMOD-p
Model
Hidden configuration
# trainable params
META_MIA
(32,3))
META gMLP
groups (32, 32) per group + mixer (16)
237 648
META GNN
dn 32, h =
284 256
direct MLP (baseline)
dp 140, =
218 240
direct Transformer
dm 64, ny 4, n
212966
ID-ARC
Model
Hidden configuration
trainable params
META MLP
(64, 32, 32)
589 408
META META.gMLP
groups (32, 4) per group + mixer (8)
755 562
META.GNN
dy = 32, k = 4
010018
direct Transformer
111, n2 =4, n
649 230
Remarks.
Parameter counts refer to the Meta-network or haseline itself; the weights produced for my are not trainable and therefore do not appear in the optimizer state.
Budgets for direct baselines were tuned so the largest baseline matches, within +5%, the smallest competing Meta-learner the same task.
Compared to naive fully- group- sied encoders/decoders and local message passing the neural graph. This yields fewer trainable meta-paran pata-paraeeters for a given target G" rather than total number of scalar weights. practice, this makes Meta-GNN more parameter- efficient while preserving or improving OOD performance relative to Meta-MLP.
connected hypernetwork that maps U directly to flat list of weights, Mcta-GNN
compute cost that scales with the number of nodes and edges
All GNNs employ bidirectional edges, attention aggregation, and group-shared MI.Ps.
Measured cost one NVIDIA V100 (32 GB).
SumFirst-n weight MLP): Meta-MLP& 15 min, Meta-GNN z 40 min. VRAM z 12 GB.
AddMod-p (6048- Transformer): Meta-MLP 30 min, Meta-GNN 90 min, VRAM 20 GB.
ID-ARC weightt Seq2Seq Transformer, on H100-40GB ): Meta-MLP 1hr, Meta-GNN x hrs, VRAM z 30 GB.
Runtime is dominated by executing the synthesized target net each step. Although the Transformer has ~ 30x more weights than the MIP, the wall-clock gap is only ~ 2x because kemels for the very small tensors in SumFirst latency-hound, whereas larger tensors in AddMod compute-hound.
23

---

Published as a conference paper at ICLR 2026
D DETAILED PERFORMANCE ANALYSIS
D.1 RESULTS ON ADDMOD-P
1.0
direcL_nlp mmlp
1.0
direct transformer
accuracy
0.8
META-MEP
0.8
META-gMLP
0.6
ME META_G GNGN
Accuracy
0.4
0.4
Train
0.2
0.2
Test
mip direct_transformer
META-gMLP
META_GNN
Test accuracy
Training mode
Figure 8: ADDMOD-p. Left: per-seed query (x) test (y) accuracy. Right: distribution over ten seeds. Meta-learners generalize far beyond direct baselines; Meta-GNN combines high accuracy with low variance.
Figure plots query versus test accuracy and the distribution over seeds. All three Meta-learners achieve > 90% accuracy on query examples, demonstrating that they can synthesize complete 6 parameeterr Transformers in one forward pass. generalizing to new moduli is harder: the best direct baselines reach only 0.65 test accuracy, while Meta-MI.P achieves median 0.89, Meta-gMI.P 0.78, and the structure-aware Meta-GNN 0.88 with the lowest variance.
Mechanistic validation. one-layer Transformer that solves modular addition is expected discover the clock representation (Nanda et al. 2023 Zhong et al. 20024). To test whether our synthesized net-networks do likewise, we project their token embeddings onto the first two principal components (Figure 99. For an unseen modulus 83, Mcta-MI.P still exhibits several distinct circular patterns, showing that it has not collapsed its internal states unique representation of the computation. Meta-GNN, instead, its aructure-awareness. indiectiionllatteetionn architecture to canonicalise the geometry into a single clear circle evidence that abstracts the task into one consistent internal coordinate system. Such canonicalization is crucial for learning generalizable abstractions and is desirable property for program synthesis systems. Clock Clock-specific metrics from /Zhong al. (2024) confirm that all META models implement valid clock solutions, with Meta-GNN closest the theoretical optimum.
24

---

Published as a conference paper at ICLR 2026
05
PC2
ace ma
9.0
-15
-2
-1
-1
-10
-0.5
0.0
0.5
1.0
1.5
PC1 (69.150
PG3 (9 (9.0%)
PCS
(3.9%)
(a) Meta-MLP, unseen modulus p 83. First principal components form several concentric circles circles-multiple permutation orbits of the clock solution.
J0n
1.00
3123
0.75
05
0.50
PC2 (264%)
0.0
-0.5
Pl
24
-1.0
21
-1.5- 2130
0.25
(b) Meta-GNN, unseen modulus p = 83. Bidirectional attention collapses embeddings to single canonical circle, consistent with Meta-GNN's superior zero-shot accuracy.
Figure 9: PCA geometry of the Transformer subnet synthesized for an unseen modulus. Meta-MLP (a) retains multiple permutation orbits; the structure-awar Meta-GNN (b) reduces the search space.
D.2
RESULTS ON SUMFIRST-n
1.0
direct, mlp
META_MLP
OOD Test Test accuracy
0.8
META-GMLP
META_ GNN
0.6
METT-MMLP-mixer
Accuracy
0.6
0.4
0.4
0.2
Train
Test
OOD Test
0.0
0.0
0.2
0.6 Test accuracy
0.4
0.8
1.0
direct_mlp
MEPA.MLP MLP
META
META-gMLP
META_GNN META-gMLP-miser
Training mode
Figure 10: SUMFIRST-n. Left: per-seed query (x-axis) versus test (y-axis) accuracy. Right: test- accuracy distribution across ten random seeds. Stracture-agnostic Meta-MLP behaves like the direct MLP baseline. Introducing weight-group tags (GMLP) occasionally discovers rule that extrapo- lates, while the structure-awar Meta-GNN yields both the highest best-case and the most consistent performance. The mixer-fir free variant GMLP-M- P-MX illustrates the effect of an even stronger inductive bias.
25

---

Published as a conference paper at ICLR 2026
compares the four weight generators task where many functionally distinct parameter settings solve the training data. Meta-MLP, devoid of structural cues, mirrors the direct MLP: fits the support/queyy splits yet rarely extrapolates to unseen n.
Figure 10
Adding layer-type grouping plus a low-rank mixer (Meta-gMLP) helps only sporadically- the median barely moves-but the few runs that do generalize indicate that even coarse positional tags can nudge optimization loward useful pattern.
Structure-ave aware message passing in Meta-GNN makes this nudge systematic: a much larger fraction of seeds exceed the gMLP ceiling and the best models approach 90% test accuracy. Thus progression MLP + gMLP - GNN shows that richer structural inductive bias raises the likelihood of converging to a rule that transfers, rather than memorising dataset-specific weight assignment.
Stronger bias via GMLP-MX. Eliminating the mixer freezes each weight group isolation. This ex- treme assumption forces the Meta-learner onto a single, casily-inne interrrtaateee Ainaarmasking strategy and lifts average test accuracy to > 0.98-yet the same rigidity makes the variant fail completely ADDMOD-p. The contrast underlines recurring theme: inductive bias must be strong enough to disambiguate weight space, but not so restrictive that precludes solutions in other domains.
D.3 RESULTS ON ID-ARC
1.0
direct transformer
1,0
META, MLP
0OD Test accuracy
0.B
META-gMLP
+
META, GNN
Accuracy
0.6
0.4
0.2
0.2
Train
Test
OOD Test
0.6 Test accuracy
0.4
direct trannformer
META,MLU META-gMLP
META_GNV
Training mode
Figure 11: ID-ARC (mixture). Left: per-seed query (x-axis) versus test (y-axis) accuracy across families. Right: distribution of test accuracy across ten random seeds. Direct Transformers either solve fail entire families, leading highly bimodal distribution. Meta-MLP and Meta-gMLP achieve more stable performance but often collapse on categorical intents such as Recol oar.. The structure-awe aware Meta-GNN attains the most consistent cross-family success, with many seeds generalizing well on Fl Flip and MoveRight.
Figure shows how different weight generators behave the multi-family ID-ARC mixture. Here the challenge more "rrll=world": diverse intents, some categorical, others ordinal, all intermixed within same training distribution.
Direct Transformers exhibit brittle behavior. They memorize input-output mappings for casy families such as Fill, Flip, or Hollow, but fail completely MoveRight ght (no extrapolation to larger distances) and on RecolorBar (no mechanism for categorical generalization). This leads to bimodal seed outcomes-either near-perfect on seen families or near-zero on harder ones.
Meta-MLP and Meta-gMLP stabilize training relative to direct Transformers, but still inherit the limitations of their architectures: they extrapolate reasonably on MoveRight thanks to the explicit distance parameter, yet collapse RecolorBa where the categorical intent provides no ordinal structure.
Meta-GNN again demonstrates the strongest inductive bias. Permutation-awan message passing converts sright-space symmetries into consistent supervision. which pays dividends on Flip-a a task where collaps- ing equivalent neuron permutations critical. Across seeds, Meta-GNN is the only model to consistently achieve high OOD accuracy on both Flip and MoveRight. while matching or exceeding others on Fill and Hol lo. The sole weak spot remains Reco 1 orBar, confirming that categorical intents lacking structure intrinsically resistant sentaagenealiztiion..
26

---

Published as a conference paper at ICLR 2026
Overall, the distribution ID-ARC illustrates sharper version of the theme in SUMFIRST-n: stronger structural bias increases not just the best-case accuracy but the likelihood of converging generalizable rules across diverse families.
D.4 SUMMARY OF FINDINGS
Across ADDMOD-p and SUMFIRST-n the three Meta-lcamers consistently
Fidelity. Produce fully functional sub-networks in single forward pass, with best-seed query accuracy or above 99% and mean query accuracy close to that mark for every model.
2. generalization benefit. Exceed direct intent- basclines unseen intents by 20 50 percentage points in test accuracy.
Role of structure. Show a clear, monotonic link between stronger structural bias and higher zero-shot performance, in line with the analysis of Section
Robust extrapolation. Reach, for example, 0.95 test accuracy on moduli never encountered during training (ADDMOD-p) and & 0.87 on list lengths beyond the training range (SUMFIRST-n).
27

---

Published as a conference paper at ICLR 2026
E EMPIRICAL RECOVERY OF THE CLOSED-FORM MAP P + WU
This appendix documents- detail analysis pipeline reverse-engineer the exact func- tion the meta-learner uses turn a prime into embedding matrices WE and Wy-
E.1 THEORETICAL IDEAL: THE "CLOCK" SOLUTION REVISITED
For ADDMODP the target network has known closed form (the "clock" algorithm):
1. Embedding We (a, b) >-> (cos 0, sin 0).
Core (attn+MLP) performs an angle-sum on two embeddings.
3. Logit head outputs w(a + 6 c)..
4. With + 4. p
Because every downstream block is frequency-agnostic, perfect meta-network should learn only
P +
(We(p), (r))),
leaving "angle-sum" section unchanged. This the hypothesis we set out to verify.
E.2 WHERE DOES THE PRIME INFORMATION LIVE?
Per-index variance. For every scalar weight we compute its standard deviation across all training primes and plot the result bar chart (Fig. 12
Nearly the entire parameter vector stays low variance, yet the same handfuul of indices protrudes sharply for every prime group examined. This repeated pattern indicates that a fixed subset parameters is responsible for carrying the prime signal; the rest of network effectively invariant.
1: (23. 53, 67,
2: (41, 28, 59,
1e+00
1e+00
8c-01
std-dev
8e-01
3d-dey
4e-01
2e-01
2e:01
0c+00
0c+00
2000
4000
6000
2000
4000
6000
Index
Index
group 1:137 33, 8.1
group 4: 13M, 12,
2e+00
2e+00
std-deu
1e+00
sid dd-dev
1e+00
5e-01
0c+00
0c+00
index
index
Figure 12: Per-index standard deviation of weights over primes. Only small and consistent set of indices shows large variance across groups of implying that prime- cemeedeeendeent information is funneled through a tiny portion of the parameter space.
Neural-Graph heat-map. To locate those hiri-variance indices the actual architecture we render the checkpoint as a NeuralGraph and color each edge (weight matrix W) and node (bias vector b) by the same standard deviation (Fig. 13 Apart from few isolated pixels inside the MLP blocks, only two contiguous regions blaze red:
28

---

Published as a conference paper at ICLR 2026
the input embedding matrix WE.
the output projection matrix Wu.
This visual confirms the closed-fore "clock" hypothesis: all prime- specific variation is concentrated in the mapping p (WE, Wo), while the intermediate angle-sum circuitry remains essentially constant.
Neural-Graph red shade sta-dev
6.55
625
0.15
0455
Prime-sensitivity heat-map on the NeuralGraph. color encodes the across-prime standard deviation: deep red indicates large change, very light red means almost no change. Edges carry weight matrices W and nodes carry bias vectors Aside from few isolated pixels inside the MLP, only two contiguous regions blaze ted-the input embedding We (left) and the output pro- jection Wy (ightt-echoing the closed-form clock solution. All intermediate edges are essentially prime-agnostic.
Figure 13:
E.3 FLATTEN-THEN-PCA. TURNING TENSORS INTO POINTS
Flatten step. For every prime query the meta learner, obtain (E(p) E R Vxd flatten it to vector in Rd d and create a data matrix of shape [P] x (Vd).
Why PCA? If each scalar weight obeys
(p) A, cus 0(p) B3 sin ((p) Cj. 0(p) + sp.
then every flattened lives in the 2-D sub-space spanned by the global vectors A (A₂) and B (B,)- PCA should return exactly two non-trivial components.
Why FFT? The angle function 0(p) carries a single frequency p' If PCA is faithful, the two principal components should share that frequency. FFT each PC therefore sanity-check.
E.4
EMPIRICAL GEOMETRY OF THE PCA PLANE
Parabolic curve. Figure 14 plots PC₁ PC2 for all 20 primes. The points lie almost perfect parabola- visual evidence one-dimensional latent path.
Shared frequency profile. The FFT spectra of PC; and PCa (Table 6 share not only the dominant peak (index 1) but also the 2d and 3rd peaks. Higher harmonics are two orders of magnitude smaller. This alignment shows that both principal components driven by exactly the same latent angle and its low-order harmonics,
mirroring the multi-frequency "clack" hehaviour reported in Nanda (2023 Zhong (024
29

---

Published as a conference paper at ICLR 2026
PCA of flattened input-embedding matrix
Train primes
13
Test primes
PC 2 (4.6% var)
2
37
61
-2
* 43
-2
-10
0
10
20
PC 1 (95.1% var)
Figure 14: Global PCA of flattened Wg. Each dot one prime, star marks the test prime. The trajectory is a thin parabola, signalling 1-D latent variable.
Table 6: Top FFT peaks of PC₁ and PC2. The first three peaks coincide, indicating a shared latent frequency spectrum.
index
frequency
amplitude
PC₁
0.048
149.97
0.095
68.70
0.143
46.99
PC2
0.048
39,56
0.095
6.37
0.143
2.78
E.5 FROM GUESSES THE FINAL CLOSED FORM
Step 1: fit for PCI. A fingle-freueenc cosine already explains almost all variance:
(C) 23.43 cos 5.072 + +3.631)
Rfc, 0.945.
Step 2: searching for PC2.
Trial 1 PC2 = sin 8
R² = 0.79
Trial 2 independent frequency for PC2
R² = 0.79
Trial 3 PC₂ = po (cos) + odd tweaks
R² = 0.964 V
PC:(p) 23.43 cos (5.63.672 3.631),
R 0.945,
PC2(p) = b+ B
BEr 2k
023+12 25+1
Rhc. 0.964,
== cos (5.672 3.631), Ba = (1/2) (-1)a
E.6 WHY AN POLYNOMIAL? BINOMIAL SERIES & ODD TERMS Even powers reproduce the binomial series.
For x: cos
22
30

---

Published as a conference paper at ICLR 2026
-all even powers. On angle range (0.06-0.52 rad) truncating after x² keeps the approximation error below 0.2%. Constraining PC₂ this form and fitting a single scale B yields the B-ratios
31 = 0.500, 32 = -0.125, 83 = 0.062, 84 = 0.039,
in excellent agreement with the binomial coefficients 1
Why do odd powers appear? PCA enforces zero mean and orthogonality to PC1. These linear operations transform v و² into [vT- const] az; which inevitably introduces odd powers when re-expanded in Their magnitudes satisfy |02j+1/ B 2 10-2, three orders smaller than the even block, confirming they statistical artefacts rather than part of the analytic clock function.
Why the meta learner settles on this form. ReLU/tanh networks approximate smooth polynomials far more easily than raw periodic functions. Once the network has produced the cosine feature (PC1). the cheapest way to supply the complementary sine information is to reuse that feature and pass it through layers that realise the truncated even-degree binomial series, plus the small odd correction required by PCA. This gives near-perfect fit (R² 0.964) without introducing any new latent frequency-exacte what the analytic clock solution predicts.
E.7 CONCLUSION AND OUTLOOK
this appendix we have manually recovered closed-form, symbolic description meta-learner's haviour:
(We(p), (o()),, where We and Wy follow an explicit cosine and polyyooial law. This shows that the network has indeed learned general analytic mapping, not table of memorised weights.
Next frontier. Our recovery was hand-crafted; the obvious next step is to automate it. A procedure that systematically extracts symbolic expressions from trained neural weights would let us translate neural programs back symbolic programs. Coupled with existing program-synne synthesis search, such tool would enable hybrid inference-plus-seen pipeline:
1. Inference stage A meta-network quickly proposes neural solution.
Symbolic extraction An automated analyser converts the weights into closed-form, form, human-readable pro- gram sketch.
3. Search / verification A symbolic synthesiser refines or proves extracted sketch.
Developing this socural-o-symmoun bridge" is, we believe, compelling direction for future work.
F CODE
Code repo: http://ghhhaaccmmm/mmsss2/1eeenn
G THIRD-PARTY ASSETS AND LICENCES
H USE OF AI ASSISTANCE
Portions of the manuscript underwent minor language polishing with LLMs (copyediting only). The authors generated all technical content, code, data, and analyses; model suggestions were reviewed and edited by the authors.
31

---

Published as a conference paper at ICLR 2026
Package
Version
Licence
URL Citation
PoTcc
2.2.1
BSD-style/ Apache 2.0
hctps:
https://pytorch.
torchmetrics
1.3.2
MIT
ht:ps::/ggttub..com
Lightning-Al/netrics
NumPy
1.26.4
BSD-3-Clause
hccps://numpy.org
NetworkX
3.2.1
SSD-3-Clause
a::ps://networkk.er
scikit-learn
1.4.1
BSD-3-Clauss
bE:pst//cikkteeenn..or
matplodib
3.8.4
PSF-based MIT
sssps//natt.otilb.org
tqdm
4.66.4
MPI.-200
nc:pss///3om.ggtnnu..11
Table 7: External code dependencies used in our experiments. All are open-source with permissive
licences compatible with redistribution of our research code. No external datasets were used; all data are synthetically generated as described in $5)
32


Reviews:
[{"review_id": "W7sHgX2Qwp", "reviewer": "Reviewer_fYXg", "summary": "The paper considers the problem of program synthesis. The authors refer to a neural network that approximates a symbolic program as a “neural program” (“NeuroP”) and explore whether a hypernetwork can be effective at converting a program specification (given by “user intent”) into a neural program. They note that a neural program has many neuron-permutation symmetries that can hinder a hypernetwork’s performance. To address this, the neural program’s structure is represented as a graph, that is divided into weight subsets. The hypernetwork then outputs a neural program in 3 steps: 1) A subset-specific encoder transforms the user intent to a corresponding latent vector. 2) The graph structure is utilized, along with message passing, in order to ensure the permutation equivariance inside the necessary weight subsets. 3) A subset-specific decoder is used to generate the weights of a neural program.\n\nThe paper claims three contributions: establishing NeuroP as an alternative to symbolic program synthesis; providing empirical evidence that using structure-aware hypernetworks can lead to strong OOD generalization across diverse program families; releasing the code and datasets.", "questions": "The approach appears to be applicable to any problem where hypernetwork are useful. Why was program synthesis (of neural programs) chosen as the target domain?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Appears to provide a novel approach (in the program synthesis literature) to generating neural programs using a hypernetwork.\n\nProvide an interesting comparison between the performance of group-aware and group-and-structure aware hypernetworks.\n\nTheir experiments cover an MLP as well as the transformer architecture, demonstrating that the method is effective for both.", "weaknesses": "Limited evaluation - experiments and baselines. It’d be good to see how it compares to other neural program synthesis methods.\n\nLimited novelty. From the three listed contribution, I would argue that the first (introducing the concept of a neural program) is not a contribution, while the third (releasing the code and benchmark) is of limited significance. The second listed contribution is “empirical evidence that intent-to-weight synthesis is feasible, and structure-aware meta-learners achieve strong OOD generalization across diverse program families.” However, in my opinion, this statement is not supported by the limited evaluation. It is possible that the way the hypernetwork is constructed is novel, but if so it should be explicitly states as a contribution.\n\nRelated work section appears to be lacking. The hypernetwork section discusses the original hypernetworks paper and then goes into MAML, while I imagine there are other important works that are related to this one (could be mistaken). Also, this section needs to incorporate other neural program synthesis approaches that represent the target program as a neural network (I don’t have a reference off the top of my head, please correct me in case all such references are already covered).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tx9WOAN43C", "reviewer": "Reviewer_rkyC", "summary": "This paper is in the field of hypernetworks. Given a user intent, it generate specific weights for another (target) neural network to complete the task. The main contribution is that the proposed architecture considered neuron-permutation symmetry (that is, for example, the order of neurons in a layer does not influence the functionality of the network). The architecture of the target network is transformed to a graph in which nodes are biases and edges are weights. While swapable nodes/edges are assigned different positional encodings to differentiate them, they are processed by the same encode function. $K$ message-passing processes are done on the graph, which include a permutation-invariant aggregation to merge all the informations from neighbors. Finally, every node and edge obtains an embedding which is then decoded as a scalar.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- As far as I know, this is the first hypernetwork that considered neuron-permutation symmetry, and the idea do make sense\n- Three variants (Meta-MLP/gMLP/GNN) are proposed and experimented, which help illustrate the effectiveness of the proposed apporach\n- A mechanistic analysis is provided, which shows strong evidence that the proposed architecture learns a generalizable solution", "weaknesses": "- While the tasks in the experimental section are illustrative, all of them are rather simple without significant practical usage", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zynHGXUc0j", "reviewer": "Reviewer_D2KG", "summary": "Traditional program synthesis methods that search within a template space rely heavily on handcrafted templates and incur substantial computational costs as template size increases. To bridge these gaps, the authors introduce the lens of meta-learning with hypernetworks, defining a neural program (NeuroP) as a differentiable program modality that enables continuous, gradient-based optimization. The authors then propose Meta-GNN, a structure-aware hypernetwork capable of generating the complete set of weights for a target architecture given a user intent U. Experimental results show that Meta-GNN demonstrates strong out-of-distribution (OOD) generalization compared to all baselines. It is notable that on the ADDMOD-p task, Meta-GNN learns a canonical clock representation, indicating that it captures underlying, generalizable computational regularities.", "questions": "1. What other tasks are commonly studied in traditional program synthesis, and could this method be compared directly with classical or symbolic approaches on those tasks?\n  \n2. In Tables 2 and 3, Meta-GNN does not achieve the best generalization across all tasks. What might explain these cases?\n  \n3. Beyond demonstrating strong OOD generalization, does the structure-aware design offer additional advantages, such as smaller model size, reduced training cost, or improved efficiency?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method makes meaningful contributions across multiple prior research directions and demonstrates clear originality:\n\n   - Leveraging hypernetworks for program synthesis is an innovative idea that mitigates the dependence on handcrafted DSLs present in neuro-symbolic approaches.\n\n   - The proposed Meta-GNN employs message passing and group-tied encoders and decoders to address the permutation symmetry problem in traditional hypernetworks.\n\n   - Out-of-distribution (OOD) generalization is a critical challenge in both meta-learning and hypernetwork research, and the paper shows that Meta-GNN achieves strong OOD generalization performance.\n\n2. The paper is well-structured and comprehensive, with clear formal definitions and thorough explanations of the key design components of Meta-GNN.", "weaknesses": "1. The claimed benefits of structure-awareness in improving generalization and performance are supported only by theoretical explaination, lacking ablation studies. Specifically, while the selected baselines partially demonstrate that greater structure-awareness correlates with better generalization, Meta-GNN, as a graph neural network, differs topologically from the MLP-based baselines. It would be valuable to further investigate the contribution of Meta-GNN’s own components.\n  \n2. In the experimental section, each task domain employs only a single target network configuration. As shown in Table 1, all target networks are shallow (mostly one-layer) , which limits evidence for scalability and cross-architecture generalization.\n  \n3. The paper lacks direct empirical comparisons with traditional methods, such as neuro-symbolic baselines, which would strengthen the claim of superiority over existing approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)