Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers
Authors: Zeyuan Allen-Zhu
Venue: NeurIPS 2025 poster ()
Keywords: language model
architecture design
Primary area: deep_learning

Abstract: Understanding architectural differences in language models is challenging, especially at academic-scale pretraining (e.g., 1.3B parameters, 100B tokens), where results are often dominated by noise and randomness. To overcome this, we introduce controlled synthetic pretraining tasks that isolate and evaluate core model capabilities. Within this framework, we discover \emph{Canon layers}: lightweight architectural components—named after the musical term ``canon''—that promote horizontal information flow across neighboring tokens. Canon layers compute weighted sums of nearby token representations and integrate seamlessly into Transformers, linear attention, state-space models, or any sequence architecture.

We present 12 key results. This includes how Canon layers enhance reasoning depth (e.g., by $2\times$), reasoning breadth, knowledge manipulation, etc. They lift weak architectures like NoPE to match RoPE, and linear attention to rival SOTA linear models like Mamba2/GDN—validated both through synthetic tasks and real-world academic-scale pretraining.
This synthetic playground offers an \emph{economical, principled path} to isolate core model capabilities often obscured at academic scales. Equipped with infinite high-quality data, it may even \emph{predict} how future architectures will behave as training pipelines improve—e.g., through better data curation or RL-based post-training—unlocking deeper reasoning and hierarchical inference.

Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers
[extended abstract]"
Zeyuan Allen-Zhu
FAIR Meta physics. allen-zhu.com
Abstract
Understanding architectural differences in language models is challenging, espe- cially at academic-scale pretraining (e.g.. 1.3B parameters, 100B tokens), where results often dominated by noise and randomness. To overcome this, in- troduce controlled synthetic pretraining tasks that isolate and evaluate core model capabilities. Within this framework, we discover Canon layers: lightweight ar- after the musical term "canon"- promote
chitectural components-namen horizontal information flow across neighboring tokens. Canon layers compute weighted sums of nearby token representations and integrate seamlessly into Transformers, linear attention, state-space models, or any sequence architecture.
We present 12 key results. This includes how Canon layers enhance reason- ing depth (e.g., by 2x), reasoning breadth, knowledge manipulation, etc. They lift weak architectures like NoPE to match RoPE, and linear attention to rival SOTA linear models like Mamba2/GDN- vallidteed both through synthetic tasks and real-world academic-scale pretraining. This synthetic playground offers an economical, principled path to isolate model capabilities often obscured academic scales. Equipped with infinite high-quality data, it may even predict how future architectures will behave as training pipelines improve-e.g. through better data curation or RI--based post-training uunloinng deeper reasoning and hierarchical inference.
1 Introduction
Recent advances in large language models (LLMs) have sparked transformative progress across numerous tasks, including question answering, summarization, translation, code generation [14. 16. 39 61]. Despite rapid progress, systematic understanding of effective neural architecture design has remained elusive, fundamentally hindered by some major challenges.
Challenge 1: Pretraining loss unreliable proxy for intelligence. Architectural comparisons often rely on perplexity or cross-entropy loss, but these metrics do not reliably reflect real-world capabilities especially since natural data is skills-mixed. For example, state-space architectures like Mamba [19 26] frequently achieve lower perplexity early in training due to rapid memorization,
"Following theory community tradition, we defer the full and future editions of paper to our project page physics allen-zhi com and sarn. waar...omaaalaaeeeaaeaaaaeeeaaaa htmmaabbrractt220330
The full V1.1 paper underwent NeurlPS 2025 review: due to result density, recommend consulting the full version for readability. Synthetic GatedDeltaNet (GDN) experiments were added V2.0; results on I- 8B Canon-layer pretraind models with wellworld data appear the follow-up Part 4.2 [2]: these were not included in the original NeurlPS 2025 submission, and we reserve the right to submit them elsewhere.
We provide a 3-video tutorial on YouTube: Part 4. (methoooooll & synthetic playground design). Part 4.1b (architecture principles from the playground), and Part 4.2 (when the playground reshapes real-life pretraining).
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Real-life Pretraining Obscures Architecture Differences
Synthetic Pretraining Enables Reliable Comparison
academic scale (1.36 parms, 1008 tokens):
Our solution: synthetic pretraining playground
30
reasoning (Brewe)
reasoeing (Dxpa)
knowdeddge capacity (Cape)
knowweddge man manipulation [Mano]
3 2.13 300
hieranchical structures [Lano]
Clear & controlled autcomes
Mini scaling- laws reveal model limits
Sharply reveal model differences (e.g., reasaning depth)
across models: 1-2%
architectural d differnnces lost in noise
diffiailltyy
this paner
ocross rondom seeds: 22-4%
4.2%
assom pusoos catKs sanEa
insufficient to see architectural strengths
X.mm 1970. Ysome dirth
real-world data: too skill-mixed, delays "emergent" skills
models fail simplest 2-hop reesoning:
your
Yhom?
Early emergence advanced skills
model size
GPT2: small size
Low cost supports rigorous studies
Beyond this scale:
controlled study
computationally infeasible
High quality data predicts future architectures
Figure I: Architecture search in noisy real-life pretrain (good luck!) vs. synthetic playground (scientific rigor).
yet perform poorly on complex reasoning tasks. Reliance on early stopping via perplexity is thus problematic: it may lead to comparing models that have merely internalized surface-level linguistic patterns without developing deeper reasoning or factual understanding [31].
Noise below emergence thresholds. Emergent abilities complee skills that only arise in large-scale models (e.g.. 7B parameters, 10T tokens -o)pplicct architectural compar- isons at smaller, academic scales (e.g.. 1.3B parameters, 100B tokens [10, 25 70)). At these scales, small benehmark gains (c.g.. 2%) often result from random initialization or data -ariiiinnnvviiieeeee that can cause 2-4% swings in accuracy (see Figure 1). More fundamentally, models fail even the simplest 2-hop reasoning tasks, performing no better than random guessing This basic reasoning floor masks architectural differences in more advanced cognitive skills, making evaluation at this scale deeply unreliable. While large-scale industry training might reveal these differences, its pro- hibitive cost blocks systematic ablations, impeding academic contributions to rigorous architecture science-and and often reducing design choices to heuristics and guesswork.
Challenge 2:
Challenge 3: Grokking, Data Quality, and Curriculum Learning. Failures in complex rea- soning tasks typically stem from deficiencies in training data, not architectural limitations. Too few challenging samples and a lack of mntermediate-compleee unstable grokking behavior- -hhee generalization only emerges after unnecessarily long pretrain- ing [43] and disrupt curriculum learning [11]. For instance, models lacking 2-hop reasoning data may unpredictably learn 3-hop tasks after extensive exposure to 1 1-hop and 3-hop examples. This makes training highly sensitive randomness, further complicating architectural comparisons. Re- inforcement learning (RL)-based post-training methods, such as GRPO [53] and PPO [52], aim to address this by delivering tailored data at optimal difficulty levels. While effective, these methods introduce new experimental confounds-it it becomes unclear whether performance gains stem from pretraining, RL fine-tuning, stochastic training dynamics, or architectural strength.
data often force models to rely on
Our approach: Atomic decomposition of intelligence. To overcome the noise and cost of real- world pretraining especially at academic scales where even 2-hop reasoning fails to emerge we decompose intelligence into core (ideally atomic!) components, such as reasoning depth and breadth, and design synthetie, controllable pretruin tasks to isolate and evaluate them independently. This framework sharply characterizes architectural strengths and scalability under clean, idealized conditions (see Figure 1), offering a principled and economical path for architecture design.
This approach directly addresses Challenge 1 by enabling single-skill evaluations, minimizing the confounding factors prevalent in real-world pretraining data. For example, it allows rigorous com- parisons of whether architecture A outperforms architecture B in reasoning depth. while ensuring modifications do not degrade other capabilities. By isolating intrinsic architectural biases, synthetic pretrain tasks reveal properties often obscured by noise and mixed signals in typical real-life setups.
Challenge is mitigated by lowering resource needs for rigorous comparisons. Synthetic bench- marks yield infinite high-quality data, enabling meaningful pretraining even for smaller models (c.g.. GPT2-small) where complex skills might otherwise not emerge. In these controlled environments, capabilities like deep multi-hop reasoning emerge clearly and reliably, allowing rapid identification of architectural limitations, investigation of mini scaling-laws, and uncover trends that real- pretrained models often fail to reveal due noise or insufficient signal despite extensive training.
In our simplest 2-hop reasoning tasks, birth years for 3 individuals are presented, followed by 3 "[name2] was born in the same year as []a equivalences. The model is prompted to infer the second group's birth years. Academic-seat scale models only guess. See Result 12

---

For Challenge 3, we manage data difficulty distributions to ensure adequate representation of pntemmdiate-compleiit samples, smoothing learning curves and enabling the early and consis- tent emergence of advanced skills-unlike less predictable real-world data prone to grokking-driven instability. As training pipelines improve-via better data curation or RI.-based continued pretrain- ing-synthetic pretrain benchmarks may provide predictive insight into which architectures best support scaling to more advanced tasks in the future.
We draw inspiration from physics, where idealized settings-such such as frictionless planes or vac- uum Seummerrs-reveed first principles by removing confounding factors. Similarly, synthetic tasks eliminate the noise, randomness, and data contamination of real-world datasets, enabling clean, controlled, apples-to-apples architectural comparisons, much like Galileo's Pisa tower experiment. This paper's key contributions are summarized below:
Result 0: Building the Synthetic Playground (Section 2+3). We introduce five synthetic pre- training tasks DEsks (reasoning depth), BREVO (reasoning breadth), CAPO (knowledge capacity), MANO (knowledge manipulation), and LANO (hierarchical language structure). This controlled en- vironment can reveal clear, commonsense capability trends at smaller scales: linear attention (e.g., GLA [69]) consistently underperforms; state-space models like Mamba2 [19] excel at memory but struggle with reasoning; and full Transformers dominate complex reasoning tasks.
Result 1: Canon Layers Add Horizontal Information Flow (see full paper) Transformers lack horizontal information flow within layers, leading inefficiencies even on simple tasks like associative recall. Drawing on the musical canon (overlapping repetition), we introduce Canon layers, horizontal "residual links" across neighboring tokens that can be flexibly inserted at multiple points - before attention (Canon-A), inside attention (Canon-Bi) -B), before MLP (Canon-CC ()) inside MI.P (Canon-D). While Canon layers can be implemented in many ways- even simple random averaging is highly effective this paper focuses on trainable 1-d linear convolutions of kernel size This lightweight and integrates seamlessly into any sequence model with minimal code.
Results 2-5: When Transformer Meets Canon (see full paper).
BoosT PERFORMANCE. In our playground, Canon layers improve reasoning depth (200-400%) reasoning breadth (30%), knowledge manipulation length (30%), and more. These stem from enhanced hierarchical learning dynamics and come with minimal compu- tational overhead.
REVIVING NOPE. Integrating Canon layers transforms NoPE models into strong performers, often matching or surpassing RoPE(+Canon). Canon layers outperform positional fixes like ALiBi [44] or H-Alibi [30], and reducing/removing RoPE usage improves length generaliza- tion.
ABLATION STUDY. Canon layers contribute cumulatively across sublayer positions (Canon- A/B/C/D), independently of attention or MLP components. Residual links improve training efficiency; minimal parameter tuning required without compromising stability.
MLP AND MoE. Canon layers can recover some knowledge capacity lost in gated MLP or mixture-of-expert (MoE) architectures, via improved training efficiency and stability.
Results 6-7: When Linear Attention Meets Canon (see full paper).
BooST PERFORMANCE. Canon layers elevate Gated I Linear Attention (GLA (69]) from 1-hop to 4-hop reasoning depth, double its reasoning breadth and knowledge manipulation length, making it comparable to Mamba2 and even surpassing it on tasks like BREVO.
ABLATION STUDY. Residual links and full Canon (A/B/C/D) are essential for maximizing effectiveness for linear-attention models, partial inplementations may underperform.
Results 8-9: When Mamba Meets Canon (see full paper).
SECRET OF SUCCESS. Mamba2's performance is driven by its built-in convid mechanism, which acts non-linear Canon-B layer applied to selective coordinates. Removing convid drops performance to match GLA, while replacing with full Canon layers further boosts results, highlighting the importance of horizontal information flow over SSM design.

---

Design Criteria Synthetic Pretrain Tasks
system reasoning (CoT)
Challenge architectural depth:
mental reasoning
Ensure real-world relevance
avoid shallow tasks (e.g., associative recall
avoid tasks solvable by external tools
Test mental reasoning (system-1):
*452352 + 547647 999999"
mental depth 4 x 8 CoT steps = 32 total steps.
Focus short (e.g,, 4096) context length
our focus for architecture design
summanizaiion
(Col)
long context often summorized to short windows for deep reasoning
long context (e.g, 1M tokens)
Figure 2: Our design criteria for synthetic pretrain tasks.
ABLATION STUDY. Canon suoices-such as integration points and residual links-can influ- ence Mamba2's performance. Mimetic initialization [63], while optimized for length gener- alization, harms shorter-context tasks, underscoring the need for diverse pretraining environ- ments.
Results 10-11: Comparing Architectures (see full paper)
CONTROLLED COMPARISONS. Applying full Canon layers consistently across RoPE, NoPE, Mamba2, and GLA allows controlled comparisons, revealing that full transformers outperform linear models in hierarchical reasoning tasks, achieving twice the reasoning depth.
REASONING DEPTH CHALLENGES. In GLA and Mamba2, limited reasoning depth stems from accumulated compression and retrieval errors-not memory aapacity-pinpointigg a key focus for future research linear models. Until this is resolved, hybrid designs (e.g., sliding-window Transformers with linear backbones) remain the most scalable path to deeper reasoning.
Result 12: Academic-Scale Real-World Pretraining (see full paper). Training 1.3B 1.3B-paramn -araameeet models on 100B tokens (context length 4096) reveals high noise and limited resolution, making many architectural comparisons statistically unreliable. Still, several consistent patterns emerge. Canon layers significantly improve NoPE and GLA elevating them to match RoPE and Mamba2, respectively- while removing convid weakens Mamba2 to GLA level. Linear models lag behind full Transformers on retrieval-heavy tasks, even with Canon layers. All models fail 2-hop reasoning, in short contexts (e.g.. 100 tokens), underscoring the limitations of academic-scale pretraining. Reducing or removing RoPE improves long-context generalization when Canon layers are present. These results align with our synthetic findings (Results 3, 6, 8, 10 I1).
In summary, Canon layers fundamentally improve horizontal information flow across diverse archi- tectures, enabling deeper reasoning and efficient scalability. Combined with synthetic benchmarks, they provide systematic insights into future opportunities in model design.
Future research. We plan to explore applications of Canon layers beyond academic scale, whose preliminary findings (w..t. 1-8B models pretrained using 1-2T tokens) align closely with those in this paper. Code is available on GitHub, models on HuggingFace, and all links are provided at physics. allen-zhu. com
Synthetic Tasks for Decomposing Intelligence
We design synthetic tasks to systematically evaluate specific capabilities of language model archi- tectures under controlled conditions, minimizing confounds and enabling clean comparisons, Task selection is guided by four criteria:
Criterion 1: Tasks must not be shallow. Shallow dasks-like associative recall copying-and casily solvable by small and shallow models, and do not mcaningfully test architectural strength. Deep learning relies on stacked layers to progressively learn abstract features [4], so tasks involving hierarchical reasoning better evaluate architectural scalability and efficiency.
Criterion Emphasis on mental thinking. Tasks should assess a model's ability to reason ternally without of-in-oo--Toughhe (CoT). While CoT helps decompose problems, it does not reflect intrinsic "system 1" reasoning [74]. For example, a model reasoning 4 steps internally and via CoT achieves 32 steps, but only internal ones reflect architectural strength. Current models like o3/R1

---

Five Synthetic Tasks Isolating Atomic Skills
(MAno): Knowledge manipulation
(DEPO): Mental reasoning depth
What's the -hh successor A??
What's answer mod
multi-hop reasoning on knowledge
(Ainected patt puen rondam order)
(BREVO): Mental reasoning breadth
What does A
(ie., 23 lookup tables)
[LANG]: Hierarchical language structure
depend on, list in topological order?
Structural reasoning: resolving ambiguity via global
[DAE giece ronngsn orderj
dynamic programming an CFG languages
(CAPo): Knowledge capacity
parse tree
parse tree
(nans) uas born (pear) homeinn (etyy worls for fromponyl.
many bet-perrporammteer model store?
Figure 3: Overview of our five synthetic tasks, each isolating an atomic skill for rigorous architectural compar- ison.
produce verbose reasoning traces even for trivial prompts (e.g.. ""tello")~evvelllii inefficiencies in system 1. To guide architectural progress, tasks must target mental reasoning.
Criterion 3: Avoid emphasis on length generalization. Length generalization is often unsta- ble-sensitive to random seeds and training order [n9] thus unreliable for comparing architec- tures. While length generalization is important, models over-optimized for long contexts (e.g.. 100k tokens) may exhibit reduced performance on standard lengths like 4096 tokens' 3 In practice, long inputs are typically summarized into shorter windows before reasoning. we prioritize evaluating architectures on dense, 4096-token contexts, where critical reasoning unfolds.
Criterion 4: Relevance to real- world skills. Tasks should prioritize broadly applicable skills while avoiding capabilities better suited to external tools. For example, large-number arithmetic (e.g., adding 10-digit numbers) is theoretically interesting but can be delegated to Python interpreters: failures in this area typically reflect limited data exposure rather than architectural weaknesses (e.g.. Llama3 70B miscaleulates 452352 + 547647). Synthetic tasks should focus universally relevant skills, aligned with real-world upplications, to ensure meaningful assessments.
2.1 Our First Set of Five Synthetic Pretrain Tasks
To operationalize the criteria above, we design five synthetic tasks-each targeting distinct dimen- sion of language model capability. We name them DEPO, BREO, CAPO, MANO, and LANO.
Task DEPO: Mental reasoning depth. Reasoning depth represents fundamental capability for LLMs. requiring models to retrieve information through multi-step computation. Task DEPO evalu- ates reasoning depth as k-hop traversal over directed permutations, where models compute the k-th successor for each query entirely internally, without intermediate steps like Chain-of-Thoughh (CoT) Each instance is formatted as:
<bos> x1 y1 x2 y2
xn yn Squery_kis q1 al <query_k2> q2 a2
<eos>
Here, 2n tokens encode n directed edges 2; - yi. forming a random permutation of n nodes.
The dataset is controlled by two parameters: N, the maximum permutation size, and K, the maxi- mum reasoning depth. During training, is sampled from [3, N]. while k: E K.]. Context lengths are fixed 2048 tokens. We employ two variants DEPO:
DEPO1: Each node spans 1-2 tokens from vocab size 50, with N 225, 300, 375 and K 8.
DEPO2: Each node spans 5-7 tokens from vocab size 4, with N 75, 100, 125 and K 16.
Evaluation focuses on both the hardest cases (n = N, A = K) and intermediate difficulty (k: K/2). For weaker models, we utilize reduced training setups with and DEPO2(K = 4). The full methodological details are provided in Appendix A.1
denoted DEPO (K 4)
Task BREVO: Mental reasoning breadth. This evaluates a model's ability to process multiple dependencies simultaneously, required in tasks involving tree-like traversal or dependency graphs. For example, solving queries like "Who are Alice's nephews?" or GSM-like examples requires parallel reasoning across branches of graph process relationships bottom-up [72]. Task BREVO isolates this capability using recursive traversal of directed acyclic graphs (DAGs), abstracting away natural language or arithmetic complexities. Each task instance is formatted as:
3 'This observed in methods like ALiBi [44], Halibi [30], and Mimetic initialization [63], whose perfor- mance degrades on shorter contexts, as we show in this paper.
Using CoT would reduce the k-hop task to simpler 1 1-hop associative recall.
5

---

<bos> x1 y1 x2 y2
Xm ym <query> q <ans> a1 a2 ap <eos>
Here, 2m tokens define m edges 2; + yr, representing dependencies where y, depends on zs. Upon receiving query vertex q. the model outputs all vertices recursively reachable from g, sorted in topological order starting from the leaves (e.g., u + + + q yields output u followed by v).
The dataset is parameterized by N, the maximum graph size, with DAGs created using n S N nodes, each of degree at most 4. Pretraining data is sampled by varying graph sizes, while testing focuses the hardest graphs (n N). We employ two variants of BREVO:
BREVOl:
Each vertex name spans single token, with N = 70/90/110, fit within 1024
tokens.
Name spans 2-4 tokens of vocab size 4, with N 30/40/50, fit within 1536 tokens.
BREVO2:
A key discovery from [72] revealed that, due to the non-uniqueness of valid outputs, language mod- els must preprocess the entire topological order of the DAG mentally before generating the first token d₁. This insight confirms that our synthetic data rigorously evaluates reasoning breadth by requiring models to globally process the underlying graph structure before producing outputs.
Task CAPO: Knowledge capacity. Task CAPO evaluates a model's efficiency in encoding factual knowledge directly within its parameters, quantified as bits per parameter, which measures reli- able storage capacity. Following the framework in [8], synthetic datasets of (fake) biographies constructed to test knowledge retention. Each biography includes several attributes (c.g.. birthdate, university, employer, etc.) and presented in diverse paraphrased formats to reduce surface-level memorization 7I. Capacity is measured using the next-token prediction distribution, accounting for both exact correctness and partial accuracy.
To highlight architectural differences, we adopt an undertrained regime where each biography is exposed only 100 times during pretraining The dataset includes N 50K to 2M biographies, encoding 2 x 10 to 10 total bits of information. Models of varying sizes are tested, and results are visualized via "bit vs. model size" plots. Additional details are provided in Appendix A.3
Task MANO:
Knowledge manipulation. Task MANO evaluates a distinct form of reasoning: the ability to manipulate stored knowledge internally, contrasting with in-context reasoning tasks like DEPO or BREVO. While those tasks focus on reasoning over external tokens, MANO requires models to retrieve factual knowledge embedded in their parameters and perform hierarchical compu- tation entirely mentally. This combination of retrieval and reasoning makes knowledge manipulation uniquely challenging and a skill that must be learned during pretraining 6
To test this capability, MANO employs synthetic modular arithmetic expressions inspired by human mental computation, particularly small-number arithmetic like the 9 x multiplication table. Models solve multi-step arithmetic problems without intermediate steps like Chain-of-Thouggh For exam- ple, given: <bos> + * a b с d <ans> the task requires evaluating ((a x b) + (c-d) d)) mod 23 for f 3, where operands a, b, c, d are sampled uniformly from [0, 22]. Modular arithmetic provides the foundational factual knowledge (23 x 23 operation tables), while the task challenges hicrarchical reasoning by recursively composing operations. Additional details are provided in Appendix A.4
The dataset is parameterized by maximum expression length L, with & sampled uniformly from [1, L]. We prepare three MANO datasets across difficulty levels: L 10, 13, and 16.
Task LANO: Hierarchical language structure. Task LANO evaluates structural reasoning over hierarchical relationships and long-ram rangrrnnge dependencies. Unlike DEPO, BREVO, and MANO, which rely on explicit valuve pairs (in-context or knowledge). LANO challenges models to infer implicit recursive structures across sequences and resolve global ambiguities within them.
To test this, LANO leverages synthetic datasets built from context-free grammars (CFGs). Training sequences consist CFG-valid sentences separated by <bos> tokens. For example:
'Exposing each biography 1000 times during pretraining diminishes architectural differences, as even trans- formers without MLP layers can achieve similar storage efficiency [8]. Uniform exposure ensures clean sys- tematic comparisons while avoiding confounding effects tied to rare outliers and junk data [8].
'For instance, questions like "Was [name] born in an even or odd month?" or derived 2-hop queries such as "What is mame]'s sister's birthdate?" demand reasoning layers over stored knowledge. These skills cannot reli- ably emerge through supervised mie-ttning alone [7] and require development during pretraining or continued pretraining.

---

<bos> 3 3 2 1 3 3 1 2 <bos> 1 2 3
2 2 1 <bos>
CFGs are designed with token-level ambiguity, where local tokens (e.g., 1, 2, provide insufficient information to directly infer their mapping to CFG rules. Resolving this requires dynamic program- ming to globally map the entire sequence to valid recursive application of CFG rules, which must also be learned during training. This reasoning grows worst-case complexity (O(n²)) as sequence lengths increase. Details are in Appendix A.5.
Building upon cfg3f [6], which includes sequences of lengths 100-500, we introduce extended datasets cjg3j and cfg3k, with sequences ranging up 200-1000 tokens increase recursive depth and test models on more nested rules and longer dependencies. Training uses context lengths of 1536 for cfg3j and cfg3k, compared to 512 for cfg3f. Evaluation prompts models with <bos> to generate CFG-valid sentences, validated via a dynamic programming parser. KI. divergence is also used compare token distributions against ground truth.
In summary, this set of five synthetic tasks covers non-overlapping skills and distinct aspects of accuracy-token--ee (DEPO, MANO), generative (Brevo, LANO), and distributional (CAPO, LaNo). While this pool be further enriched, it serves strong starting point for deriving meaningful architectural insights, as demonstrated in the following sections.
3 Initial Comparison on Well-Known Architectures
Language model architectures have evolved significantly since Transformers [64], resulting in three major families distinguished by computational mechanisms.
Quadratic- time attention models, pioneered by the original Transformer, include prominent archi- tectures such as BERT [35] and GPT2 [46]. Recent refinements include Rotary Position Embeddings (RoPE) [13, 59] and gated MLP layers [54]. We use the Huggingface implementation of Llama, denoted as Llama(RoPE), incorporating RoPE and gated MIP, and variant without positional em- beddings, Llama(NoPE).. We refer to these as RoPE and NoPE respectively when clear from the context. We exclude relative positional embeddings due to limited empirical benefits but additional computational costs [6].
RoPE models often generalize poorly beyond training context lengths. In contrast, NoPE general- izes better but suffers from lower overall performance. Recent attention-scor modifications (c.g., ALiBi [44] and Hard-Alibi [30)) partially address this trade-off: we discuss later sections.
Linear-time attention reduces computation by compressing sequences into fixed-length represen- tations. Examples include Linformer [65], Performer [15], Linear Transformer [34]. We focus on more recent Gated Linear Attention (GLA) [69], known for computational efficiency and scalability.
Recurrent and state-space models process long sequences using evolving hidden states instead of attending over all tokens. Mamba [19, 26] exemplifies this category: analyze its second gen- eration (Mamba2). Other prominent models include $4 [56]. S5 [56], RetNet [60], RWKV [42], HGRN [45] GSA (77), DeltaNet [71], and GatedDDelaaNet [70].
Avoidance of bybrid architectures. We exclude models integrating attention with linear or state- space mewhods e.g. Griffin [20]. Samba [48), GatedDeltaNet-HII H1/H2 [70] or sliding- wiiddwww atten- tion-to maintain clarity. Such hybrid approaches excel in extremely long contexts (e.g.. million tokens), but our analysis focuses explicitly on precision within standard context windows (4096 to- kens). In practice, long contexts are often compressed to shorter segments (e.g., via CoTss) for final detailed processing. making precise local reasoning essential.
Hybrid models can obscure architectural trade-offs; aggregated results may not reflect individual component contributions clearly. For instance, Mamba2 strong in memory tasks yet weaker in structured reasoning. Hybrids blending linear/state-space modules with attention mask these distinctions. Thus, for transparency, this study focuses entirely on isolated architectures to clearly analyze their inherent strengths and weaknesses.
Architecture Size Standardization. To ensure fair comparisons, we standardize model sizes and evaluate Llama, GLA, and Mamba2 as representative architectures from each family.
For all tasks except CAPO, we experiment with four architecture sizes. Llama models have 12 or layers, with hidden dimensions of 768 or 512 (and 12 heads), denoted as 12L768D, 8L512D, etc. (12L768D matches GPT2-small). We translate these configurations into GLA, Mamba2,
7

---

Task Depo
k=4/2)
Task (epo k=4/2)
Task Depol(K=4, k=4/2)
Liam(RoPP) original
GLA (l
(amba2(mlp) original (convid)
N-225
43/99%
73/92%
54,45%
93/100%
N-225
7/46%
2/14%
14/55%
19/62%
N-225
12/65%
2486
24/74%
43/54%
N=300
100/100%
85/98%
52/100%
67/100%
N=0
2/11%
1/9%
1/22%
4/2
N-000
4/35%
12/46%
23/62%
13/61%
N-375
57/99%
09/90%
39/9%
33/94%
0.0%
0/0%
0/6%
1116%
N75
1/22%
1358%
7/33%
10/42%
BL5120
12L5120
8L7caD
12L7600
BL5120
12L5120
BL768D
12L76RD
BL5120
12L5120
8L768D
12L7600
Task Dapo2 k-4/2)
Task
2KK42
kc-4/2)
k-4/2)
Llama(RoPE) original
orig
Mamba2(mlp)
original (convld)
N-75
100/100%
96/100% 100/100% 100/100%
N-75
8/22%
1/5%
5/30%
3/12 %
N-75
18/61%
80,95%
30/86%
69/89%
N-100-
98/100%
100/100% 99/100% 99/100%
6/34%
1/23%
3/25%
166%
17/63%
47/83%
17/53%
42/82%
N-125
100000001001000000000000000
89/100%
N-125
1/3%
2/26%
N-125
5/39%
4.685%
20/4196
24/75%
aL5120
1255120
aLFenD 12178RD
AL512D
5205120
AL76RD
7207
AL512D
1215120
AL76nD
1217880
Task Brevol
Task Brevol GLA original
Task Brevol
LIama original
Mampa original (convld)
N-70
41.9%
88.6%
64.5%
91.9%
N-70
33.7%
36.5%
46.1%
42. 42.2%
N=70-
80.1%
50.1%
72.
N-90
57.7%
71.1%
24.8%
79.1%
N-90
1.7%
2.8%
6.2%
11.9%
N-30-
0.3%
0.5%
3.8%
4.8%
N-110
12.1%
39.9%
9.8%
5.32%
N=110
1.2%
10.7%
2.9%
15 15.2%
110
0.1%
0.0%
1.2%
1.2%
8L512D
12L5120
BL768D 12L768D
8L512D
12L5120 BL760D 12L760D
8L512D
12L5120 0L768D 12L768D
Task Brevo2
Task Brevo2 GLA original
Task B Brevo2
LRoPP( original
Mambaz(mlp) -rrginaal (convid)
N-30-
71.5%
94.0%
87.2%
96.6%
N-30
45.5%
21.5%
33.2%
N-30-
50.8%
95.6%
68.1%
3.4%
N-40
30.6%
83.1%
67.2%
88.1%
N-40
0.7%
1.0%
1.8%
8.8%
N-40
12.5%
67.0%
14.5%
0.5%
N-50
19.7%
75.7%
42.1%
82.1%
N-50
0.1%
0.7%
1.0%
1.6%
N-50
3.3%
12.4%
4.0%
0.5%
8L512D 12L5120
8L768D
12L768D
8L512D 12L512D 8L768D
12L768D
8L512D 121512D 8L76RD
12L768D
Task Capo Llama(RoPE) original
Task Capo - GLA original
Task Capo (amba2(mip) - origi naliconvad
pureer
b4 (porem puree bi: 0.25 Ui Jaater
2 ppree 3/ppprr
paron
25 bi: paraer
0.25 kit par
1-2930930
102933930
1-193000
Task Mano
Task Mano GLA original
Task Mano Mamba2(mlp) original (convid)
LRoPP original
L-10
55.3%
17.8%
63.0%
66.0%
L-10
59.8%
56.0%
76.2%
56. 5%
L-10
96.5%
95.1%
95.2%
95.7%
L-13
19.4%
30.6%
36.7%
L-13
36.5%
31.1%
69.5%
44.4%
L-13
79.9%
84.8%
88.0%
91.8%
L-16
8.3%
7.7%
15.8%
32.6%
L-16
16.8%
24.7%
24.8%
22.4%
L-16
74.4%
90.1%
72.3%
87.4%
8L512D
12L5120
BL768D
12L768D
8L512D
12L5120
BL768D
12L768D
8L512D
12L5120
8L768D
12L768D
Task Lano
Lano GLA original
Task Lano
LIama(RoPE) original
Mamba2(mlp) original (convid)
c1g3t
91.1%
96.3%
93.4%
97.6%
dtgat
37,5%
61.1%
41.8%
54.9%
drgar 83.8%
92.2%
86.8%
92.2%
cfa3]
74.1%
91.4%
82.33 3%
90.3%
cfa3j
2.7%
17.1%
9.1%
35.7%
cfa3) 45.5%
72.0%
54 2%
74. 74
df33k
64.0%
75.1%
79.1%
dfg3k
13.0%
11.6%
12.9%
19.8%
cf33k
32.7%
50.0%
35.3%
46. 1%
8L512D
12L5120 8L768D
12L768D
12L5120
8L768D
12L768D
8L512D
12L5120
8L768D
12L768D
Figure 4: Initial comparison of RoPE, Mamba2, and GLA on five synthetic tasks. GLA performs poorly everywhere except knowledge capacity (CAPO); Mamba2 excels at knowledge (CAPO. MANO); Llama(RoPE)) is hest at reasoning (DEPo, BREVO, LANO). This confirms our synthetic playground as effective for architectural comparisons, but introducing Canon layers (see rest of the paper) will build a Pisa tower for more controlled and fair comparisons, where the landscape shifts drastically and reasoning depth improves 2-4x.
Mamba2(mlp) and Gated DeltaNet (GDN) to ensure comparable parameter counts? 7
For CAPO pbttperr-arrameter knowledge capacity), we vary model and data sizes more widely. Fol- lowing [8], we denote model scale by f-h: for Llama, this means layers, hidden size 64h, and heads. We extend this notation consistently to GLA and Mamba2.
Training. We use identical training settings (batch size, training steps, learning rates, etc.) across architectures to ensure fair comparisons. Complete details are provided in Appendix A We also fix random seeds so that all architectures pre-train on precisely identical data sequences.
"The original Mamba2 has no MLP layers: each Mamba layer has 6d parameters (for hidden size d). compared with 12d in Llama. Thus, we configure Mamba2 with 24 or 16 layers to match Llama's size. Mamba2(mlp) alternates Mamba and gated MLP blocks, thus keeping 12 or 8 total layers. See details in Appendix C
8

---

3.1 Initial Comparison Results
attention GLA performs weakest overall, Mamba2 excels in knowledge tasks (CAPO, MANO), and Llama(RoPE) performs best reasoning tasks (DEPO, BREVO, l.No).. These results validate the effectiveness of our synthetic playground: however, we avoid deeper in- terpretation this point. As shown later, Llama and GLA lack critical architectural component, making this initial comparison incomplete, unfair, and less informative.
From Figure 4, linear-attention
For now, we highlight several key remarks.
3x4 4 mini scaling laws. Randomness may affect outcomes. For example, in Task MANO, despite two seeds and four learning rates per configuration, smaller models sometimes outperform larger ones. Thus, robust statistical comparisons are crucial. We address this by testing synthetic tasks systematically at three data scales and four architecture sizes (even more for Task CAPO). These "3 mini scaling laws enable clearer visual comparisons, reducing variability.
Benefits of synthetic tasks. Synthetic tasks clarify architectural differences starkly (e.g., 90% vs 5%), clearly exposing strengths and weaknesses. By contrast, real-world experiments often produce modest differences (e.g., 2%) buried in noise. Thus, synthetic pretraining environments allow clean evaluations of architectures' scalability and true capabilities.
Interpreting task failures. If specific architecture (of given size) fails at certain difficulty level (c.g.. large N or k), it does not imply the model cannot learn the skill given infinite training. Our comparison uses fixed, limited training budget: all architectures train for the same number of steps with identical data and shuffling, reporting best accuracy across multiple learning rates. Thus, differences in the speed of skill acquisition, not absolute capability
results should be
Predicting future pipelines. Synthetic tasks simulate idealized, high-quality pretraining conditions targeting core skills like multi-hop reasoning (DEPO). Unlike datasets such as Fine Web-edu or SlimPajama, which contain sparse reasoning examples obscured by simpler content, synthetic tasks highlight core capabilities. Currently, 100B-token pretraining fails even simplest 2-hop reasoning (Result 12). As training pipelines evolve -via improved data curation or RL-based post-training synthetic tasks like DEPO may better predict models' potential and guide architectural choices.
The remainder of this paper is deferred to the full version.
Acknowledgements
ZA sincerely thanks Vahab Mirrokni for the invitation to the Yale workshop in October 2023, where this research was sparked through enlightening discussions with Vahab Mirrokni and Peilin Zhong- Canon layers build on the idea of uniform attention previously explored in [6]. ZA thanks Alberto Alfarano for introducing the papers [30 44 63 79), and the PyTorch scaled dot product attention function. At Meta, we extend our heartfelt gratitude to Lin Xiao and Kristin Lauter for their insight- ful discussions and unwavering supports, which made this research possible. Special thanks go to Wangzhi Dai, Sam Doud, Dinesh Kannappan, Niki Kim, Junjie Qian, Ammar Rizvi, Travis Seev- ers. and Stephen Hartken at Meta, as well as Abraham Leal from W&B; without their invaluable technical assistance, the experiments presented in this paper would not have been feasible. We are deeply grateful to Songlin Yang and Ali Behrouz for providing detailed instructions on replicating their academic-scale pretraining experiments, and Fangcheng Sun for many helpful conversations on architecture design in general.
Contribution statement. ZAA proposed all ideas, conducted all investigations, implemented all code, performed all experiments, authored the entire manuscript, and managed necessary compliance reviews and social promotions; the term Canon Layers was jointly conceived and designed with Xiaoli Xu.
Faster learning is practically important for example, model ideally learns reasoning skills quicker than pure memorization. Similar observations arise in knowledge capacity tasks [8], where architectural differences vanish with ample training but become pronounced when training budgets are limited.

---

References
[1] Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report. arXiv preprint arXiv:2412.08905. 2024.
[2] Zeyuan Allen-Zhu. Physics of Language Models: Part 4.2, Canon Layers at Scale where Syn-
thetic Pretraining Resonates in Reality, 2025. URL https://phyyscss.. awlplyysicc.lllenzauia.aaaa allen- com/
Code released at btttps://githut com/
fccebookreesarrdd/Pyyyssss....
[3] Zeyuan Allen Zhu and Yuanzhi Li. Can SGD Learn Recurrent Neural Networks with Provable Generalization? In NeurlPS, 2019. Full version available
01028
[4] Zeyuan Allen-Zhu and Yuanzhi Li. Backward Feature Correction: How Deep Learning Per- forms Deep (Hierarchical) Learning. In Conference on Learning Theory, COLT '23, 2023. Full version available at http: /ttt:://axxv... www.iv..rgg//baaee. httpg//bs/000..... 04413,
[5] Zeyuan Allen-Zhu and Yuanzhi Li. Physics of Language Models: Part Knowledge Storage and Extraction. In Proceedings of the 41st International Conference on Machine Learning, ICML 2024, 2024. Full version available at http://aaxiii http:://arxx..or//aaa/////
14316
[6] Zeyuan Allen-Zhu and Yuanzhi Li. Physics of Language Models: Part 1, Learning Hierar- chical Language Structures. Transactions on Machine Learning Research, 2025. Full version available at http:://rxiii
hts//bb/23000 13673
[7] Zeyuan Allen-Zhu and Yuanzhi Li. Physics of Language Models: Part 3.2, Knowledge Ma- nipulation, In Proceedings of the 13th International Conference on Learning Representations, ICLR 2025, 2025. Full version available at http:/ /att:///axxiv..e http/axxivvv.ro... 2309 14402
[8] Zeyuan Allen- Zhu and Yuanzhi Li. Physics of Language Models: Part 3.3, Knowledge Capac- ity Scaling Laws. In Proceedings of the 13th International Conference Learning Represen- tations, ICLR 2025, 2025. Full version available at /htt:://arxxii org/ abs/ 2404. 05405
[9] Simran Arora, Aman Timalsina, Aaryan Singhal, Benjamin Spector, Sabri Eyuboglu, Xinyi Zhao, Ashish Rao, Atri Rudra, and Christopher Ré. Just read twice: closing the recall gap for recurrent language models. arXiv preprint arXiv:2407 05483, 2024.
[10] Ali Behrouz, Peilin Zhong, and Vahab Mirrokni. Titans: Learning to memorize at test time.
arXiv preprint 2501::2501.00663, 2024.
[11] Yoshua Bengio. Jérôme Louradour, Ronan Collobert, and Jason Weston, Curriculum learning.
In Proceedings of the 26th annual international conference on machine learning, pages 41 41-48, 2009.
[12] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. PIQA: Reasoning about phys- ical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432-7439, 2020.
[13] Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Gold- ing, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Wein- bach. GPT-NeoX-20B: An open-source source autoregressive language model. In Proceedings of the ACL Workshop on Challenges & Perspectives in Creating Large Language Models, 2022.
URL BtLLhttp::// /tapp:://rxxivv .rg/ abwp/224/44.0 06745
[14] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah. Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language mod- els are few-shot learners. Advances in neural information processing systems, 33:1877-1901,
2020.
[15] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009. 14794, 2020.
[16] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmano, et al. Palm: Scaling language modeling with pathways. Journal of Muchine Learning Research, 24(240): 1-113, 2023.
10

---

[17] Christopher Clark, Kenton Lee, Ming-Wor Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2924 2936, 2019. doi: 1411653/11/N19-1300
[18] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the A12 reasoning challenge, arXiv preprint arXiv:1803.05457 2018.
[19] Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060, 2024. URL https:
//arxiv. org/ abs /ta.b///// .
[20] Soham De, Samuel L Smith, Anushan Fernando, Aleksandar Botev, George Cristian- Albert Gu. Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, et al. Grif- fin: Mixing gated linear recurrences with local attention for efficient language models. arXiv preprint
[21] Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over para- graphs. In Proceedings of the 2019 Conference of the North American Chapter of the Associa- tionfor Computational Linguistics: Human Language Technologies, Volume / (Long and Short Papers), pages 2368 2378, Minneapolis, Minnesota, 2019. Association for Computational /ttttssss/ccaaateeeeeeee wwwpthhhooooooooooooooooooo wwwg//19/110000
Linguisties. doi: 10.18853/v1/N19-111
[22] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion pa- rameter models with simple and efficient sparsity. The Journal of Machine Learning Research, 23(1):5232-5270, 2022.
[23] Daniel Y Fu, Tri Dao, Khaled Kamal Saab, Armin W Thomas, Atri Rudra, and Christopher Ré. Hungry hungry hippos: Towards language modeling with state space models. arXiv preprint arXiv:2212. 14052 2022. URL https:///
atgg/bb////// 14052
[24] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman. Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding. Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang. Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL httpt:://///eoo / records/1260860
[25] Olga Golovneva, Tianlu Wang, Jason Weston, and Sainbayar Sukhbaatar. Multi-token atten- tion. arXiv preprint arXiv:2504.00927, 2025.
[26] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. http//bb/232..2 00752
arXiv preprint urriv:23322.0071 2023. URL http:htps::/// /tttpp://arxivv
[27] Anmol Gulati, James Qin, Chung- Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convolution-augmentee transformer for speech recognition. arXiv preprint arXiv:2005.08100 2020.
[28] Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:22044 06654, 2024.
[29] Changho Hwang, Wei Cui, Yifan Xiong, Ziyue Yang, Ze Liu, Han Hu, Zilong Wang, Rafael Salas, Jithin Jose, Prabhat Ram, Joe Chau, Peng Cheng, Fan Yang, Mao Yang, and Yongqiang Xiong. Tutel: Adaptive mixture-of-experts at scale. CoRR, abs/2206.03382, June 2022. URL
https://arxx: //rs:v//axxvv..// org/pdf//////oooooooooogpp /206 03382. pdf
[30] Samy Jelassi, David Brandfonbrener, Sham M Kakade, and Eran Malach. Repeat after me: Transformers are better than state space models at copying. arXiv preprint arXiv:2402.01032, 2024.
[31] Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishi,, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1-38, 2023. doi: 10.1145/3571730. URL https://doi.
org/ /00 1145/3571730
[32] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh
11

---

Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
[33] Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computationall Linguistics (Volume I: Long Pupers),
pages 1601-1611 2017
[34] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention. In International conference on machine learning, pages 5156-5165. PMLR, 2020.
[35] Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HILT, pages 4171-4186, 2019.
[36] Yury Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. Babilong: Testing the limits of Ilms with long context reasoning-in-
haystack. Advances in Neural Information Processing Systems, 17:106519-1065533 2024.
[37] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Wei Chang, Andrew M. Dai, Jakob Uszko- reit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering search. Transactions of the Association for Computational Linguistics. 7:452-466 2019. doi: 10.1162/tacl 00276.
[38] Nayoung Lee, Ziyang Cai, Avi Schwarzschild, Kangwook Lee, and Dimitris Papailiopou- los. Self-improving transformers overcome easy-to-hard and length generalization challenges. arXiv preprini arXiv:2502.016122 2025. (RLLhhtps://aatt http:/www...r//ab///2000000..
[39] OpenAl. Gpt-4 technical report, 2023.
[40] Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAM-
BADA dataset: Word prediction requiring broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pages 1525-1534, 2016. doi: 10. 10.11833/v1/PL 1144.
[41] Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. htttts:s///aattt org/abs/ /bs/ 2406. 17557
[42] Bo Peng. Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, et al. Rwkv: Reinventing mns for the transformer era. arXiv preprint arXiv:2305. /3048, 2023.
[43] Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177 2022. URL htttp:://arxiv wwteervvvvvor/veee org/abs////eeeeooooogggo/ee httpa/bs222..0000 221222222277
[44] Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2 12409, 2021.
[45] Zhen Qin, Songlin Yang, and Yiran Zhong. Hierarchically gated recurrent neural network for sequence modeling. Advances in Neural Information Processing Systems, 36:33202-33221, 2023.
[46] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[47] Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784-789, Melbourne, Australia, 2018. Association for Computational Linguistics. doi: 10. 18.18853/w 11t3:///PP1112211 URL https: //aclanthology. .acooog..ogg/// wwwp/o/-////...
[48] Liliang Ren, Yang Liu, Yadong Lu. Yelong Shen, Chen Liang, and Weizhu Chen. Samba: Simple hybrid state space models for efficient unlimited context language modeling. arXiv
12

---

preprint arXiv:2406.07522, 2024.
[49] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An adversarial winograd schema challenge at scale. arXiv preprint 1xXiv:1907.1064 2019.
[50] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Con- ference Natural Language Processing (EMNLP-IJCNLP), pages 4463-4473, 2019. doi: 10.18653/v1/D19-114 1454.
[51] Eshika Saxena, Alberto Alfarano, Emily Wenger, and Kristin Lauter. Teaching transformers modular arithmetic at scale, arXiv preprint arXiv:2410.03569, 2024. URL https://////..
org/abs/2400. 0455/
[52] John Schulman, Filip Wolski, Prafulla Dhariwal, Alee Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 073477 2017.
[53]
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.00 03300, 2024.
[54]
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202. 2020.
[55] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gates gated mixture-of-experrss layer. In International Conference on Learning Representations, 2016.
[56] Jimmy TH Smith, Andrew Warrington, and Scott W Linderman. Simplified state space layers for sequence modeling. arXiv preprint ArXiv:2088. 220.03230, 2022.
[57] DR So, W Manke, H Liu, Z Dai, N Shazeer, and QV Le. Primer: Searching for efficient transformers for language modeling. arxiv 2021. arXiv preprint arXiv:2109.0866 2021.
[58] Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. deduplicated version of RedPajama. slimpajaman ama- a-
SlimPajama: A 627B token cleaned and
http:::/wwww.cerebaaaa bll/llgg//////
June 2023. URLhhttss:/huugiiggfaas http://aacc.co/ttatttee//e//s//e
[59] Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2021.
[60] Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.086211 2023.
[61] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim- othée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1 3977, 2023.
[62] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.0922
2023.
[63] Asher Trockman, Hrayr Harutyunyan, J Zico Kolter, Sanjiv Kumar, and Srinadh Bho- janapalli. Mimetic initialization helps state space models learn to recall. arXiv preprint arXiv:2410.11135. 2024.
[64] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[65] Sinong Wang. Belinda Z Li. Madian Khabsa, Han Fang. and Hao Ma. Linformer: Self- attention with linear complexity. arXiv preprint 40Xiv:2006.04768, 2020.
[66] Jason Weston Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart Van Merriënboer, Armand Joulin, and Tomas Mikolov. Towards ai-complete question answering: A set of pre- requisite toy tasks. arXiv preprint arXiv: 2205555099955 2015.
[67] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introuucigg convolutions to vision transformers. In Proceedings of the IEEE/CVF inter- national conference computer vision, pages 22-31, 2021.
13

---

[68] Songlin Yang and Yu Zhang. Fla: A triton-based library for hardware-efficient implementa- tions of linear attention mechanism, January 2024. URL httpp://github.. com/fla org/ 1lash-linear-ri
[69] Songlin Yang. Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear at- tention transformers with hardware-efficient training. arXiv preprini arXiv:2312.06633, 2023.
[70] Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule. arXiv preprint
(tte::24120064400 2024.
[71] Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length. arXiv preprint arXiv:2406. 06484, 2024.
[72] Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process. In Proceedings of the 13th Inter- national Conference on Learning Representations, ICLR 2025, 2025, Full version available at https://arxia: org/ abs/2407 20311
[73] Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen Physics of Language Models: Part 2.2. How to Learn From Mistakes Grade-School Math Problems. In Proceedings of the 13th International Conference on Learning Representations, ICLR 2025, 2025. Full version available at /tatt:://rxxi,. www./as.g//////////e//...t..aee/ abbs/408..o 16293
[74] Ping Yu, Jing Xu, Jason Weston, and Ilia Kulikov. Distilling system 2 into system 1. arXiv preprint arXiv:2407.0660
2024.
[75] Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, YX Wei, Lean Wang. Zhiping Xiao, et al. Native sparse attention: Hardware- and natively trainable sparse attention. arXiv preprint arXiv:2502. 1089, 2025.
[76] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4791-480X 2019. doi: 10.18663/v1/119-111
[77] Yu Zhang. Songlin Yang. Rui-Jie Zhu, Yue Zhang, Leyang Cui, Yiqiao Wang, Bolun Wang, Freda Shi, Bailin Wang, Wei Bi, et al. Gated slot attention for efficient linear-time sequence modeling. Advances in Neural Information Processing Systems, 37:1 16870-11689, 2024.
[78] Zhengyan Zhang, Yixin Song, Guanghui Yu, Xu Han, Yankai Lin, Chaojun Xiao, Chenyang Song. Zhiyuan Liu, Zeyu Mi, and Maosong Sun. Relu² wins: Discovering efficient activation functions for sparse 1lms. arXiv preprint arXiv:2402.03810 2024.
[79] Yongchao Zhou, Uri Alon, Xinyun Chen, Xuezhi Wang, Rishabh Agarwal, and Denny Zhou. Transformers can achieve length generalization but robustly. arXiv preprint drXiv:2402.09377,, 2024.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction state the main Cntributions-Canne layers and a synthetic pretraining framework-and these are supported throughout the paper with 12 results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
14

---

2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The paper notes that academic-scale pretraining introduces noise and that the current five-task synthetic suite, while sufficient for distinguishing base models, is not exhaustive.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
The paper does not present formal theorems or proofs, and is focused on empirical architectural evaluation.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
15

---

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Detailed training protocols, hyperparameters, data generation, and evaluation procedures are provided in the appendix of the full version of this paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results way produce the model (c.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The code and data generators are not yet released, but we plan to open-source them soon; at this point, we have tried to provide all the technical details needed for repro- ducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
16

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
wwwpp..ccppub.... www.sccggiii/sseee Cote://ooSSSubmmi issiissinnn wwweiissssoopeeiie onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Full experimental configurations are described in Appendix A B and B, in- cluding model sizes, optimizers, learning rates, batch sizes, and evaluation steps.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [Yes]
The paper quantifies noise due to random seeds (e.g., Figure 1) and only interprets performance differences above that margin.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
17

---

8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The paper specifies hardware (e.g., A100/H100 GPUs), training duration, batch sizes, and Canon layer runtime overhead (Appendix 4 and A).
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't t make it into the paper).
9. Code of ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http:pps....ppbbi/tttiiiccccc
Answer: [Yes]
Justification: The work uses only symthetic publicly available datasets and does not involve human subjects, privacy risks, or misuse scenarios.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The Conclusion discusses potential benefits of more reproducible and resource-efficien architecture evaluations, and foresecable negative impacts are iden- tified.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
18

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not release high-risk models or data; it works with synthetic benchmarks and academic-scale training only.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere usage guidelines restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yss]
Justification: All external models and datasets (e.g., Mamba2, SlimPajama) are cited with source, version, and licenses are respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, papprswi wwwodeetthheee...c com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documenta- tion provided alongside the assets?
19

---

Answer: [No]
Justification: New synthetic datasets and evaluation suites are introduced, but not yet leased; documentation provided in Appendix A and release is planned post-submission.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve any human subjects or crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should paid at least the minimum wage the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No human subjects were involved, and IRB approval was necessary.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if upplicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMSs if it is an important, original, or non-standard component of the core methods in this research?
Question:
Answer: [NA]
20

---

Justification: LLMs were not used as an important, original, or non-standard component of the core research methods described in this paper. Any LLM usage was for general writing assistance.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy Inttps://neuripss.o (ttres//eeurpppccc//oteeeeeeeee wtt.///oooeeeene///2//////// for what should or should not be described.
21


Reviews:
[{"review_id": "gkidYNJAxw", "reviewer": "Reviewer_bijY", "summary": "Authors propose some synthetic pretraining tasks with the motivation that they will be helpful for doing architectural research, which at the moment is difficult to do for pretraining acc to them. \n\n5 tasks - DEPO (reasoning depth), BREVO (reasoning breadth), CAPO (knowledge capacity), MANO (knowledge manipulation), and LANO (hierarchical language structure.\n\nTo show effectiveness of these tasks, they propose Canon layers. Then they use these layers to show improvement on real world benchmarks.", "questions": "I think that the most important thing for such a framework is to establish correlation between the intermediate/psuedo tasks and the goodness of a pre-trained model (how many tasks does it take?) how will you establish it? \nI like the premise but I think there is more work needed.", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strenghts:\n- The paper is well written and easy to follow\n- The authors have identified and are working on a very useful problem. Nice!\n\nWeaknesses:\n- I like the overall premise. But what is the guarantee that anything good on these tasks will be good on real world tasks? What is the guarantee that they are optimal tasks?\n- How do you know that all we need is to measure performance on these 5 tasks and that is all to predict goodness of pre-training.\n- Authors begin by saying that pre-training performance is more difficult to evaluate and it is correct. model has not been post-trained yet.\n- Canon layers themselves are able to revive weaker models. Finding components using your framework that can improve upon state of art architectures will be the true test of this scheme.", "was_revised": true, "final_justification": "The paper has a chance of becoming a good paper, but it needs more work.\nThe authors addressed 1 of my 3 concerns. Authors seemed defensive to feedback and were mostly just requoting their numbers again\n\nIf the other reviewers want to accept the paper, I won't stop it. But I think paper can be improved a lot.\n\nAlso, unhappy with Reviewer UT7i, who gives a 6 with a one-line strength and one line weakness review", "rebuttal": "**We thank Reviewer bijY for highlighting the clarity and usefulness of our work, and for appreciating our systematic approach to a difficult open problem. We appreciate your thoughtful questions about real-world relevance, task sufficiency, and evaluation, and address each below.**\n\n**Overall Message:**  \nWe believe your main concerns arise from healthy skepticism about whether our synthetic tasks are sufficiently predictive and representative of real-world pretraining gains. We share this concern, and have carefully designed both our tasks and experimental phases with this in mind. Please see responses below.\n\n---\n\n1. **Correlation with real-world tasks**  \n   **BLUF:** *Our synthetic tasks already predict some real-world improvements, especially for models like Canon+NoPE and Canon+GLA, and large-scale results (including 8B models) reinforce this correlation.*  \n   - In the paper, we show that models improved by Canon layers on synthetic tasks—such as Canon+NoPE and Canon+GLA—become highly competitive, sometimes matching or even exceeding the performance of more complex baselines (e.g., Canon+GLA approaches Mamba2).\n   - Since submission, we have validated these findings at **larger scale**: across 1B, 3B, and 8B-parameter Llama Transformers trained with 1–2T tokens, `Llama+Canon` consistently improve MMLU accuracy by 2%+ on top of `Llama`.\n   - The combined 8B result (with Canon) **even outperforms** `Llama3-8B` on standard `lm_eval_harness` benchmarks, with similar gains across other tasks.\n   - While these new results are not included in this submission (and cannot be cited due to anonymity), they are now fully open-sourced and visible to the community. They can't be possible without the guidance from the synthetic playground designed in this paper.\n\n2. **Why these 5 tasks? Are they “optimal”?**  \n   **BLUF:** *We carefully chose our tasks according to \"Criteria for Task Selection\" (Appendix A.1) to cover core skills relevant to real-world models, but the suite is extensible.*  \n   - As described in Appendix A.1, we put thought into selecting tasks that reflect essential skills for modern models, while avoiding redundancy or shallowness.\n   - Our approach is not to claim optimality or exhaustiveness, but rather to start with a minimal, interpretable set. If a future model’s real-world gains are not predicted by these tasks, it signals the need to augment the suite (see Appendix I, future work).\n   - We focus on tasks that best isolate atomic skills most relevant for downstream use.\n\n3. **How many tasks are needed to predict model quality?**  \n   **BLUF:** *The current 5 tasks reliably differentiate all our studied models; expansion remains future work as needed.*  \n   - Across both synthetic and academic-scale experiments, these tasks have distinguished all studied architectures and surfaced key design insights.\n   - If future advances reveal gaps, we view this as motivation for further task development—an ongoing research agenda.\n\n4. **Are synthetic results sufficient before post-training?**  \n   **BLUF:** *Synthetic tasks may allow us to efficiently predict which skills architectures can acquire, before costly real-world post-training.*  \n   - First of all, most model skills are believed to originate during pretraining, making pretraining the critical stage for understanding and improving architecture design.\n   - Second, to fairly compare architectures under real-world post-training (e.g., RL), all models must be pretrained at scale using exactly the same data—a process that is very costly and prone to noise at academic scale.\n   - Third, our synthetic playground is designed to efficiently predict which architectures will close skill gaps (e.g., multi-hop reasoning) that real post-training would reveal, while remaining affordable, controlled, and reproducible.\n\n\n---\n\n**In summary, we hope the reviewer sees our synthetic benchmark as a principled, extensible “starting point” for bridging controlled architecture research and real-world model advances. We appreciate your support and thoughtful questions!**"}, {"review_id": "BBIqQSHRX1", "reviewer": "Reviewer_UT7i", "summary": "Academic-scale pretraining (around 1.3 billion parameters and 100 billion tokens) yields results dominated by noise, making it difficult to discern true architectural differences. To overcome this, the authors design a controlled suite of five synthetic pretraining tasks, DEPO for reasoning depth, BREVO for breadth, CAPO for knowledge capacity, MANO for knowledge manipulation, and LANO for hierarchical structure, that cleanly isolate core capabilities. Based on insights from these tasks, they introduce Canon layers, lightweight horizontal residual links implemented as causal convolutions. Canon layers integrate into any sequence model and dramatically improve horizontal information flow, boosting reasoning depth by 2–4×, breadth by 30 percent, knowledge capacity by 10–15 percent, and enabling weaker architectures to match or exceed stronger baselines.", "questions": "- The plots are overwhelming at first, and CAPO plots are particularly hard to read; Figure 8 could probably be simplified to highlight only the key insights.\n\n- It would be interesting to include a small-scale ablation study on different Canon kernel sizes.\n\n- For DEPO, the task may be too straightforward as a depth benchmark; would be good if we can also have a more demanding multi-step reasoning task. One example may be adding a variant of BREVO with a k-step limitation (output only vertices reachable in exactly k steps from the query).\n\n- Nit: on page 34 it should state “kernel size 4” rather than 3.", "limitations": "yes", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strength:** The paper presents a rigorous, cost-effective framework using synthetic benchmarks and extensive experiments that convincingly demonstrate how Canon layers improve core model capabilities and enable clearer architectural comparisons.\n\n**Weakness:** Real-world evaluations are limited to relatively small scale due to budget constraints, though the authors acknowledge and address this; the Canon concept itself is not particularly novel, though its systematic validation across diverse tasks is valuable and compelling.", "was_revised": true, "final_justification": "My questions have been addressed and I think the paper is interesting and impactful. I maintain my score.", "rebuttal": "**We sincerely thank Reviewer UT7i for their strong endorsement and thoughtful feedback. We are grateful for your recognition of our experimental framework and the impact of Canon layers. Please find our responses below, along with updates from our ongoing work.**\n\n---\n\n1. **Figures and minor typo**  \n   **BLUF:** *Thank you for the suggestions; we will correct the typo and work to improve figure clarity in future versions.*\n\n2. **Ablation study on Canon kernel sizes and CUDA implementation**  \n   **BLUF:** *Our use of kernel size 4 is due to practical CUDA support; we are seeking broader collaborations for future improvements.*  \n   - We adopted `causal_conv1d` with kernel size 4 because this is what the CUDA implementation efficiently supports, allowing us to rapidly test Canon layers at scale.\n   - Our goal is to first demonstrate compelling results, which we believe will attract systems and hardware experts to help us develop even more efficient and flexible (possibly CUDA-friendly) alternatives for Canon layers in future.\n   - Notably, we also found that simple random averaging performs surprisingly well, suggesting there are many lightweight possibilities beyond the current implementation.\n\n3. **Multi-step reasoning in BREVO and DEPO**  \n   **BLUF:** *We have already explored BREVO with varying depths; Canon layers consistently increase reasoning depth.*  \n   - As shown in Figure 14 (page 23), we varied the depth parameter in BREVO and observed that Canon layers substantially increase the model’s effective reasoning depth.\n   - We chose to focus the main discussion on DEPO because it provides a more direct and interpretable probe of reasoning depth, but we agree that both perspectives are valuable.\n\n4. **Real-world validation and scale**  \n   **BLUF:** *Our latest large-scale results confirm that synthetic playground guidance translates to real-world gains.*  \n   - Since submission, we have trained 1B, 3B, and 8B Transformer models on 1–2T tokens each, mirroring real-world regimes.\n   - Adding Canon layers to Llama consistently increases MMLU by over 2% and gives similar improvements on other `lm_eval_harness` tasks. Note: our trained `LlamaCanon-8B` even outperforms the commercial `Llama3-8B`, thus this is not a toy setting.\n   - While these results are not included in the submission (and note cited for anonymity), they are open-sourced and strongly support your view that our synthetic framework provides practical guidance at scale.\n\n---\n\n**Thank you again for your detailed and encouraging review!**"}, {"review_id": "QXwY91APFN", "reviewer": "Reviewer_h3Cm", "summary": "This paper presents an intriguing exploration of synthetic pretraining tasks aimed at teasing apart architectural differences in large language models (LLMs). Using this framework, the authors identify Canon layers, lightweight components that enhance horizontal information flow across adjacent tokens, integrating seamlessly into diverse architectures, including Transformers, linear attention models, and state-space models. They validate these enhancements via both synthetic tasks and real-world academic-scale pretraining, demonstrating increased reasoning depth, reasoning breadth, and knowledge manipulation. \n\n\nI find this paper interesting with solid experiments that provide better readability than most current LLM-based benchmark papers. However, it still has several drawbacks, such as its difficulty to comprehend due to unclear writing. The authors present numerous findings, but I am curious about which one is the main finding the paper intends to highlight. Additionally, the Canon layer approach appears to me as a variant of sliding windows, and I wonder what additional computational overhead it might entail and why the window size is limited to 3. It seems this method might be particularly effective in certain scenarios, such as those demonstrated in lines 165-167 with the AB..A context. The effectiveness in other contexts remains uncertain, as the synthetic tasks designed by the authors naturally fit these scenarios, showcasing the Canon layer's promising results.", "questions": "What is the central contribution among the findings presented? Is there a particularly important conclusion?\n\nHow much computational overhead is introduced by the Canon layer? What rationale is given for the chosen window size?\n\nHow do the synthetic tasks relate to real-world complex tasks?", "limitations": "Insufficient Validation on Complex Tasks: Due to reliance on relatively simple validation tasks, the paper's applicability and impact in complex situations remain to be examined.\n\nOverly Technical Expression: The paper's technical language may limit rapid understanding and application of the methods by non-specialist readers.\n\nAs a Variant of Sliding Window: Canon layers, as a variant of sliding windows, are effective in specific scenarios, but their performance in other contexts is uncertain, especially without targeted task design.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\nInnovative Experimental Framework: The paper provides a new methodology for exploring language model architectures through synthetic tasks, presenting a novel approach for architectural understanding.\n\nRobust Experimental Design: The experiments cover multiple tasks and different architectures comprehensively, with clear comparisons and results.\n\nEnhanced Interpretability: By introducing Canon layers, the paper enhances the transparency of understanding how models perform reasoning and knowledge manipulation.\n\nWeaknesses:\n\nComplex and Lengthy Presentation: Although the experimental content is rich, the paper's narrative is somewhat challenging to decipher, lacking clear emphasis on primary findings.\n\nSimilar Design to Sliding Windows: The Canon layers resemble sliding window techniques, necessitating further exploration regarding computational overhead and the effectiveness of the fixed window size of 3.\n\nLimited Validation on Challenging Datasets: The validation utilizing GSM8k is relatively simple for current LLMs, thereby potentially restricting the paper's broader applicability and impact.", "was_revised": true, "final_justification": "i remain my current score.", "rebuttal": "**We thank Reviewer h3Cm for their detailed review, thoughtful questions, and recognition of our experimental innovation and architectural analysis. We appreciate your suggestions for clarity and deeper comparison. Please find our responses below.**\n\n---\n\n1. **Central contribution and main findings**  \n   **BLUF:** *Our two main contributions are (1) introducing Canon layers for robust horizontal information flow, and (2) establishing a synthetic pretraining playground as a systematic, interpretable benchmark for architecture design.*  \n   - Canon layers provide a lightweight, architecture-agnostic way to boost reasoning depth and knowledge skills across diverse models.\n   - The synthetic playground enables affordable, controlled, and reproducible evaluation of core architectural skills, setting a new standard for principled LLM design and comparison.\n\n\n2. **Clarity and emphasis**  \n   *We acknowledge the writing could be clearer and will further highlight key messages in future versions.*  \n\n3. **Canon layers: Novelty vs. sliding window, kernel size, and computational overhead**  \n   **BLUF:** *Canon layers are distinct from sliding window attention (SWA); kernel size 4 was chosen for CUDA compatibility; computational overhead is modest yet with consistent and considerable gains.*  \n   - Sliding-window attention (SWA) requires dense matrix multiplications (d × d), making it heavy to deploy in as many places as Canon layers. Furthermore, our experiments show SWA does not work as well unless the number of heads is very large, so it may not be worth it in practice.\n    - We chose kernel size 4 purely for CUDA implementation convenience (`causal_conv1d`), enabling rapid experimentation and reproducibility. This allowed us to quickly test the Canon layer idea and showcase its potential, with the goal of engaging SysML experts to help develop even more lightweight and CUDA-friendly variants in future work.\n   - Overhead is modest: for a 1B model with Canon-ABCD, total runtime increases by ~12% (footnote 7); for larger models (e.g., 8B), the cost drops to about 3%—yet with consistent and considerable gains in performance.\n\n4. **Synthetic task relevance and real-world validation**  \n   **BLUF:** *Our synthetic tasks test skills that are both challenging and predictive of large-scale real-world performance.*  \n   - Although our synthetic tasks may appear easier than real-life tasks, they can also be more challenging in certain respects. For example, extreme 8- or 16-hop retrieval scenarios almost never occur in Common Crawl, and would likely only emerge during post-training (RL). By stress-testing model architectures on these rare but demanding skills, our synthetic playground reveals limitations and strengths that real-life pretraining may not expose until much later.\n   - Since submission, we have further validated this approach: *Canon layers added to 1B, 3B, and 8B Transformers pretrained on 1–2T tokens* consistently yield >2% MMLU improvement, and similar gains on other `lm_eval_harness` tasks. Our `LlamaCanon-8B` model even outperforms the commercial `Llama3-8B`, suggesting that this is no longer a \"toy setting.\"\n   - While these results are not in the submission (and not cited for anonymity), they are fully open-sourced. We hope this further reinforces that insights from our synthetic benchmarks do transfer to real-world, large-scale settings.\n\n---\n\n**Thank you again for your constructive and encouraging review!**"}, {"review_id": "ygooxvkwIG", "reviewer": "Reviewer_9L9W", "summary": "The paper presents a set of synthetic tasks to test architectural changes in LLMs in a more principled manner and further introduces Canon layers: 1d convolution with kernel size 4 + optional residual connections. Canon layers can be interleaved both in the attention and MLP part of a transformer. The authors present 12 empirical results highlighting differences in performance between Transformers and other architectures such as Mamba and GLA and the benefit of Canon layers. Canon layers are also shown to improve the performance in standard benchmarks under pre-training.", "questions": "Q1 Can you include, even if in the appendix, a related work section on synthetic tasks proposed to evaluate LLM architectures? For example [1,2] introduce formal language tasks which have been included in recent works concerning novel architectures. A deeper discussion on this would help contextualize the set of  tasks proposed in the paper. \n\nQ2 Figure 1 presents a compelling argument against relying on pretraining + benchmarks to assess architectural improvements: the difference by varying the data or init seed is larger than the one given by changing architecture. How does the difference in performance change when varying seed? Does fixing the seed to compute such a difference mitigate the huge variability of the results with the data and initialization seed?  Moreover, do you see these large differences even in perplexity on e.g. lambada or wikitext?\n\n\nQ3 Concerning canon layers, is there some intuition behind the choice of 4 for the kernel size?\n\n\n[1] Deletang, Gregoire, et al. \"Neural Networks and the Chomsky Hierarchy.\" The Eleventh International Conference on Learning Representations.\n\n[2] Akyürek, Ekin, et al. \"In-Context Language Learning: Architectures and Algorithms.\" International Conference on Machine Learning. PMLR, 2024.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\nS1 Comprehensive experimental evaluation. The paper includes an impressive number of  experiments and for each of the synthetic tasks several different setups are evaluated: varying both the complexity of the task and the number of parameters for the models.\n\nS3 The proposed canon Layers are simple but impactful and justified both conceptually and experimentally,  although the novelty is limited given that Canon is similar to the 1d convolution present e.g. in Mamba.\n\nS2 The manuscript is clearly written, well organized and easy to follow.\n\n\n**Weaknesses**\n\nW1 Little related works concerning synthetic experiments previously proposed to test LLM architectures (See Q1 in Questions). \n\nW2. I found many figures (e.g Figure 4 and 5) too packed and therefore difficult to parse.", "was_revised": true, "final_justification": "The issues I outlined in the review (lack of related work on synthetic data and figures that are too packed) were minor and the authors promised to include a related work section and to improve the figures. The authors also addressed my question on the variability of the benchmarks when fixing the data seed and only varying the initialization seed with a qualitative answer saying that they still observe a significant difference, and promised to include this version in the final paper. \n\nI maintain the score of 5 given in the original review.", "rebuttal": "**We thank Reviewer 9L9W for their careful reading, positive feedback on our experimental scope, and for highlighting the conceptual and practical value of Canon layers. Please see our responses below.**\n\n---\n\n\n\n1. **Figure crowding and visual clarity (W2)**  \n   *We acknowledge that some figures are dense and will clarify/simplify visuals in future versions.*  \n\n2. **Variance from random seed/data (Q2)**  \n   **BLUF:** *Significant variance persists even when only model initialization seeds are varied, and this is consistent with our observations in Figure 8; perplexity-based evaluation may be misleading, especially for overfitting.*  \n   - For this rebuttal, we trained models with 3 different init seeds (fixing data seed) and found that the variance remains substantial, further confirming that performance instability is not just a function of data seeds.\n   - This may not be surprising given Figure 8: there, we fixed the data seeds and still observed that minor changes in model architecture can cause large fluctuations in benchmark outcomes.\n   - Regarding perplexity: Figure 8 shows large variance for LAMBADA perplexity but much less for WikiText. However, low variance on benchmarks like WikiText can reflect over-fitting and may fail to distinguish meaningful architectural differences. This supports our argument that perplexity alone may not a reliable proxy for real downstream capability.\n   - Overall, we think these findings highlight the importance of our synthetic playground for controlled, interpretable architectural evaluation.\n\n3. **Canon layer kernel size (Q3)**  \n   **BLUF:** *Kernel size 4 was chosen purely for CUDA efficiency and reproducibility; future work will explore alternatives.*  \n   - We adopted kernel size 4 because it is the maximum efficiently supported by the `causal_conv1d` CUDA implementation, enabling us to rapidly scale experiments and share reproducible results.\n   - Our goal is to demonstrate effectiveness and attract collaboration from SysML experts for designing even lighter and more flexible Canon variants --- after all, even random averaging works quite well, so there ought to be better implementation choices than `conv1d`.\n\n\n4. **Related work on synthetic tasks (Q1)**  \n   **BLUF:** *Our synthetic tasks are deliberately chosen to probe model skills most relevant for modern LLMs, and differ in both coverage and difficulty from the formal language tasks in [1]; [2] is complementary but distinct in focus.*  \n\n   - **[1] (Deletang et al., \"Neural Networks and the Chomsky Hierarchy\")**  \n      - Many of their tasks, such as Duplicate String and Bucket Sort, are now too easy for Transformers.\n      - Their Reverse String appears challenging, but only in the binary case; with broader vocabularies (as in our DEPO/BREVO), it becomes easy for Transformers. The same holds for their Parity Check and Stack Manipulation tasks—difficulty drops with more realistic input diversity.\n      - Their modular arithmetic task is similar in spirit to our MANO, but we intentionally use a much larger knowledge base (23×23 table) to better stress-test \"knowledge manipulation.\"\n      - They study multi-digit addition/multiplication, but as we explain in Figure 2 and Appendix A.1, we intentionally avoid skills like long arithmetic that are better delegated to external tools (e.g., a calculator or Python interpreter). Even Llama3-70B cannot reliably compute 452352+547647=999999, and this is not a good use of LLM's parameter capacity in practice.\n\n   - **[2] (Akyürek et al., \"In-Context Language Learning: Architectures and Algorithms\")**  \n      - This work advances the understanding of in-context learning by focusing on regular languages and sequence induction. Our approach is complementary: we aim to systematically decompose LLM capabilities into atomic skills directly aligned with reasoning and knowledge challenges found in real applications, rather than primarily statistical pattern induction.\n\n   - We appreciate the suggestion and agree that a more detailed related work discussion could be valuable in future versions.\n\n\n5. **Real-world relevance and generalization**  \n   **BLUF:** *Follow-up experiments confirm the synthetic playground's practical value at scale.*  \n   - Since submission, we have further validated this approach: *Canon layers added to 1B, 3B, and 8B Transformers pretrained on 1–2T tokens* consistently yield >2% MMLU improvement, and similar gains on other `lm_eval_harness` tasks. By the way, our `LlamaCanon-8B` model outperforms the commercial `Llama3-8B` on such benchmarks, suggesting that this is no longer a \"toy setting.\"\n   - These results, though not included in this submission (and not cited for anonymity), are now fully open-sourced and reinforce the predictive power of our synthetic benchmarking approach for real-world advances.\n\n---\n\n**Thank you again for your thorough and encouraging review!**"}]

Author rebuttal:

Meta-review: This paper introduces a suite of synthetic pretraining tasks (DEPO, BREVO, CAPO, MANO, LANO) designed to facilitate architectural research in language models, alongside Canon layers to improve reasoning and knowledge manipulation. The paper is well written, easy to follow, and addresses a timely and useful problem. The synthetic tasks and Canon layers provide a rigorous and cost-effective framework for evaluating architectural choices, with extensive experiments across multiple tasks and model sizes.

Strengths include the innovative experimental framework, comprehensive evaluation, and enhanced interpretability through Canon layers. While the novelty of Canon layers is somewhat limited given their similarity to existing operations (e.g., 1D convolutions in Mamba), the conceptual justification and experimental validation are strong.

Overall, the paper presents a broad and impactful research direction, with wide relevance for both the research community and practitioners. Most reviewers lean toward acceptance.
Decision: Accept (poster)