SurveyLLMArch/LLMArchitectureSurvey

Towards Next-Generation Architectures of Large Language Models

What the canonical decoder fixes, and what recent architectures change.

A reading guide to the survey: one question, one map, and the design space behind open-weight language models.

🎯 The question · 🖼️ Teaser · 📈 HLE · 🗺️ Survey map · 📖 Chapters · 🧩 Design space · 🏗️ Patterns · 📚 Literature · 📎 Citation

---

_{Opening view of the survey. The canonical decoder fixes a pre-norm residual block, a dense feed-forward network, the same depth for every token, and a token-level key–value cache that grows with generation. The four panels are the directions developed in Section 15, Beyond Components: A Unified View of Architectural Design.}

⚡ Processing Flexible and conditional: who runs, and how much work a token receives.

🧠 State Selective and flexible: what history is kept, and how that state is updated.

🔭 Lookahead Delayed commitment: predict further ahead, then decide what to keep.

✨ Dynamics Refined transforms: the nonlinearity and the path across depth.

---

🎯 The question

Large language models have improved through scale, data, and training recipes. Their architectures have changed with them. Most deployed systems are still variations of the decoder-only Transformer in Section 4, Basic Modules in Transformer: multi-head self-attention, a position-wise feed-forward network, positional encoding, residual connections, and normalization, with tokenization as the interface to raw text.

Section 1, Introduction states the constraint that organizes the rest of the survey. Self-attention is quadratic in sequence length, and the key–value cache grows linearly with context. Dense activation, a fixed layer schedule, and one-token-at-a-time decoding add further training and serving cost. The survey asks how an architecture trades representational power, optimization stability, and computational cost when it relaxes one of those choices and keeps the others.

It differs from a component survey in three ways, all set out in Section 1:

1. It covers the full decoder, from tokenization and position through attention, feed-forward layers, normalization, residuals, the output objective, and depth allocation.
2. Section 14, Popular Architectures records how open-weight families actually combine those pieces.
3. Section 15 rereads the same mechanisms as assumptions of the canonical decoder that each design preserves, refines, or relaxes.

📈 Where open-weight models stand

Section 1 closes with two snapshots. MMLU against release date covers roughly 2021–2025 and is reproduced in the literature map. By 2026 that benchmark was nearing saturation, so the survey turns to Humanity’s Last Exam for the recent open-weight generation.

_{Humanity’s Last Exam versus open-weight model size (Section 1). The plot is concentrated on models released in 2026; GLM-4.7, from late 2025, is kept as a reference. The horizontal axis is total parameters on a log scale. Marker area grows with active parameters for mixture-of-experts models and with total parameters for dense models. Dashed and dotted curves are the observed Pareto frontiers with and without tools. Both start from Qwen3.6-27B (Dense); that model’s tool-use protocol is unspecified. Scores that mix tool-enabled and non-tool protocols should be compared with caution.}

Four earlier trends, read off the MMLU snapshot in Section 1, still frame this picture.

Trend	What Section 1 reports
Scale	At a given moment, larger models tend to score higher than smaller contemporaries.
Recipe, not only size	Newer and smaller models often pass older and much larger ones, consistent with compute-optimal, data-rich training.
Sparse capacity	Mixture-of-experts designs reach the top of the range at high sparsity, with a lower active-parameter cost.
Open versus proprietary	Proprietary systems have defined the upper envelope. On HLE the gap remains, and tool use moves the strongest open-weight scores by well over ten points.

Nearly all of the models on the HLE figure are sparse mixture-of-experts models, and scores still rise with total parameter count along the observed frontiers.

---

🗺️ Survey map

Section 2, Overview is the roadmap. Sections 1 and 2 frame the tree; they are not a third technical family. The figure below is the taxonomy in Section 2, numbered as in the survey. Color follows the role of each block in the stack.

The color legend is the one used by the taxonomy in Section 2.

Color	Role in the stack	Sections
🔵 Blue	History and the Transformer foundation	3–4
💠 Light blue	Keep quadratic attention, shrink the KV state	5
🌊 Teal	Replace pairwise mixing with a linear-time operator	6
🟣 Purple	Conditional parameter activation	7
🍑 Peach	Nonlinearity, normalization, and residual paths inside a block	8–9
🟢 Green	The units and coordinates of the sequence	10–11
🪸 Coral	How much work is spent per generated token	12–13
🟡 Cream	How released models combine the above, and what assumption that changes	14–16

Sections 5 and 6 are two answers to the same cost: cheapen mixing over time. Sections 12 and 13 are two answers to another: decide how much computation a generated token receives.

📖 Chapters

Reading order is the order of the survey. Each note below is the claim of that chapter, not a second taxonomy.

🧭 1–2 · Frame

Section 1, Introduction places architecture next to scaling laws. Loss tracks model size, data, and compute fairly smoothly, but the architecture decides complexity, optimization dynamics, and what kind of state the model is allowed to keep. The section also locates modern attention in a longer line: LSTM pretrained models, then the 2017 Transformer, whose attention mechanism the survey traces to the 1991 unnormalized linear Transformer.

Section 2, Overview is the reading map drawn above. After the component chapters, the survey turns from layers to generation: how many tokens one forward pass is asked to predict, and how many layers each token actually traverses.

🧱 3–4 · Foundations

Section 3, History of Language Modeling Architectures starts from language as a stochastic process. Markov’s counts and Shannon’s n-gram source are the probabilistic problem. The architectural answers that follow are statistical n-grams, recurrent networks, fast-weight programmers, LSTM’s protected additive state, and then self-attention as a way to select over an explicit history instead of compressing it into a fixed state.

Section 4, Basic Modules in Transformer fixes the notation every later chapter uses.

	Module	What it fixes
4.1	Multi-head Attention	Pairwise mixing. In a decoder, causality limits each query to the past.
4.2	Feed-Forward Network	Position-wise channel mixing, and a large share of the model’s parameters.
4.3	Positional Encoding	Order. Full attention without a positional signal is permutation-equivariant; the causal mask is another, weaker source of order.
4.4	Residual Connections and Normalization	The path by which representations and gradients cross depth.
4.5	Tokenization and Input Embeddings	The discrete interface to raw text.
4.6	Output Projection and Autoregressive Generation	One next-token distribution, then one committed token.
4.7	Encoder / decoder layouts	Decoder-only is the LLM default. Encoder-only and encoder–decoder remain the other two configurations.
4.8	Computational Complexity	Training cost is a forward pass over length T. Decoding cost is per new token with a KV cache. For attention, the projection term dominates when T is smaller than the hidden width.

💠 5 · Efficient Attention and KV Cache Management

This chapter keeps quadratic attention and reduces the key–value state that is stored or read. The four axes are orthogonal in intent and not freely stackable in practice.

	Axis	What changes	Representative ideas
5.1	Token	Which past positions are kept, merged, or read	Eviction and compression; sparse reads over a retained cache; attention that is sparse by design, including StreamingLLM, SepLLM, MoBA, NSA
5.2	Head	How many distinct KV heads exist	Multi-query attention, grouped-query attention, and asymmetric or key-driven groupings
5.3	Dimension	The width of the cached vector	Multi-head latent attention, tensor-product factors, grouped latents, conversion into latent attention, and post-training width compression such as Palu and ThinK
5.4	Layer	Which layers own a KV state	Cross-layer merge or reuse, including MiniCache, cross-layer attention, YOCO, MLKV, and layer-condensed KV

Section 5.5, Future Directions singles out two open problems. Token reduction is the axis that can depend on the input, because sequence length is not an architectural constant. Combining axes is attractive and constrained: latent attention already shares one state across heads, so it does not sit cleanly on top of grouped-query attention or head-wise budgets.

🌊 6 · Linear Language Models

These models do not sparsify the attention matrix. They avoid explicit pairwise interactions, so cost scales linearly with length. Section 6 treats four families, then one skeleton.

	Family	State, in one line
6.1	Recurrent Neural Networks	A fixed-size hidden state. LSTM’s constant-error carousel is the early protected additive path. Modern instances include LRU, Griffin’s RG-LRU, RWKV-6, and xLSTM.
6.2	Fast Weight Programmers	The state is a fast weight, updated by an outer product. ULTRA (1991) is the early case. DeltaNet, Gated DeltaNet, and RWKV-7 are recent ones. The update can be read as one step of online regression at test time.
6.3	State Space Models	A linear dynamical system. Mamba makes the transition input-dependent. Mamba-2 and Longhorn sit in the same line.
6.4	Linearized Attention	A kernel replaces the softmax, which turns attention into a recurrent update. Gated random-feature attention, RetNet, gated linear attention, MetaLA, and Rodimus are instances.
6.5	Unification	The families share a recurrent update. They differ in the gate, in whether the state is expanded, and in whether the update matches a test-time regression step.

Section 6.6, Future Directions puts the limit on the transition. An input-independent transition, or an input-dependent but diagonal one, cannot route information between state channels. Under the survey’s bounded-depth, log-precision assumptions, that places many of these models in TC⁰ and outside NC¹-hard state tracking such as permutation composition. Richer non-diagonal transitions, depth, and hybrids restore some of that power. A practical pattern already at large scale is to interleave linear layers with softmax attention, often near a 3:1 ratio. Training and prefill use chunkwise parallel kernels; decoding uses a fused recurrent update. Sequence recurrence in this chapter is not the depth recurrence in Section 13.

🟣 7 · Mixture of Experts

An expert layer is a router plus a bank of networks, usually feed-forward blocks. Only the selected experts run, so cost tracks active experts rather than the size of the bank. Section 7.1, Evolution of MoE in LLMs is one subsection with three overlapping developments, not three separate chapters.

Development inside 7.1	What changed
Sparse gating	Top-k token choice, expert-choice routing, auxiliary balance losses, and the systems that made this trainable: GShard, Switch, BASE, ST-MoE, DeepSpeed-MoE.
Fine-grained shared experts	Many small experts, plus a shared expert that every token sees. DeepSeekMoE is the template that later open-weight models inherited.
Systems-oriented variants	Communication and expert-weight traffic become design variables: shortcut-connected experts, zero-computation experts, latent experts, and post-training compression of expert weights.

Section 7.2, Future Directions leaves three problems open: better routing, where to place experts across depth, and heterogeneous experts. Current training systems assume experts have the same shape.

🍑 8–9 · Inside the block

Section 8, Activation Functions follows the feed-forward nonlinearity through four stages in Section 8.1: rectification (ReLU and its variants); smooth gates (GELU, SiLU, and Swish); gated linear units, with SwiGLU the default after strong empirical results; and scale-driven departures, including squared ReLU in some hybrid Mamba–Transformer families and clamped SwiGLU where unbounded branches threaten numerics. Section 8.2 expects the next step to be layer- or task-specific, hardware-aware, and not confined to the feed-forward network. Gating has already been tried on attention outputs.

Section 9, Normalization and Residual Connections separates the normalizer from the wiring.

	Topic	Designs
9.1	Normalization methods	BatchNorm, LayerNorm, RMSNorm, QK-Norm, DeepNorm, and lighter variants such as ScaleNorm
9.2	How the normalizer meets the residual	Post-LN, Pre-LN, Sandwich-LN, norm-after, Mix-LN, HybridNorm, Hyper-Connections, and manifold-constrained hyper-connections
9.3	Open problems	Adaptive normalization, a tighter account of the stability–expressivity seesaw, and designs that drop LayerNorm entirely

Pre-RMSNorm is the wiring most current dense decoders share. That fact becomes one of the reference choices in Section 14.

🟢 10–11 · The sequence interface

Section 10, Positional Encoding supplies order to an operation that would otherwise be permutation-equivariant.

	Family	Role
10.1	Absolute	Sinusoidal or learned vectors tied to an index. Length extrapolation is the weak point.
10.2	Relative	Displacement inside the attention score. Rotary position embedding is the form most open-weight decoders use. ALiBi, T5 bias, and content-adaptive schemes are the other branch.
10.3	Context extension	Keep a trained positional scheme and stretch the usable range. Position interpolation, LongRoPE, YaRN, and CLEX update the model. Self-Extend and related chunked schemes only rewrite positions at inference.
10.4	Open problems	Content-dependent positions, no explicit positional encoding under a causal mask, hybrids such as interleaving rotary local layers with global layers that have none, and positions for data that are not a single sequence.

Section 11, Tokenization treats the vocabulary as architecture.

	Topic	Claim
11.1	Granularity	Characters, words, and subwords trade vocabulary size against sequence length. Tokenization-free models read bytes or characters. Byte Latent Transformer groups bytes into learned patches.
11.2	Vocabulary size	The compute-optimal vocabulary grows with the compute budget, more slowly than the rest of the parameters. Decoupling the input vocabulary from the output vocabulary, and growing the vocabulary during pre-training, are the two concrete levers.
11.3	Open problems	Vocabulary as a scaling axis, asymmetric input and output vocabularies, and hierarchical latent tokens between raw bytes and a fixed subword inventory.

🪸 12–13 · Compute per generated token

Section 12, Multi-Token Prediction changes the objective before it changes the serving stack. Next-token training supervises one future token. Multi-token training supervises a horizon of K future tokens, which includes the ordinary next token. The extra predictors can be dropped at inference, leaving a standard model, or kept as a drafter.

The classification is by factorization. Joint pre-training versus a retrofit is a separate choice of when the predictor is attached.

	Factorization	What is independent
12.1	Parallel prediction	K heads read the same backbone state, so the predictions are cheap and conditionally independent. Blockwise decoding, ProphetNet, multi-token prediction heads, and Medusa are in this group.
12.2	Sequential prediction	Each extra module sees the preceding future token, so the causal chain is kept. DeepSeek-V3’s single extra module is the recipe many later open-weight models adopted. EAGLE drafts at the feature level; EAGLE-3 drops feature regression and fuses several hidden states. FastMTP shares one module across the horizon.
12.3	Inference integration	Acceptance rules, token trees, adaptive draft length, and drafter–target alignment. Speculative decoding and speculative sampling define an acceptance rule that preserves the target distribution. SpecDec++ adapts the candidate length.
12.4	Open problems	Horizon as a scaling axis, hybrids that stay causal without a new module per offset, and keeping the drafter aligned after supervised or reinforcement post-training.

Section 13, Adaptive and Recurrent Depth varies how many layers a token executes. If a binary mask says whether token i runs layer ℓ, effective depth is the sum of that mask. The standard decoder sets every entry to one.

	Mechanism	Allocation
13.1	Depth reduction and early exit	A token stops at a prefix, or the network drops layers for every token. LayerDrop, Depth-Adaptive Transformer, CALM, and LayerSkip. LayerSkip’s early layers can draft for the full model, which ties this section to Section 12.
13.2	Conditional depth	A router sends a fixed-capacity subset of tokens through a block. Mixture-of-Depths does this around the block. CoLT5 keeps a light path for every token and a heavy path for a few.
13.3	Looped Transformers	The same block is applied R times. Universal Transformer, Huginn, Ouro, Mixture-of-Recursions, and fixed-point reasoners differ in topology, in who chooses R, and in how much weight tying is relaxed.
13.4	Training variable depth	Intermediate supervision, credit assignment through recurrence, dynamical stability, and whether extra recurrence extrapolates past the training budget. A reported recurrence-equivalence exponent near one half says shared recurrence only partly replaces unique depth.
13.5	Open problems	Stable training at greater recurrent depth, budget-conditioned inference, joint allocation with experts and multi-token prediction, and evaluation under matched compute.

Depth recurrence adds compute by revisiting a shared block. It does not summarize the sequence. That job belongs to Section 6.

🟡 14–16 · From components to a design space

Section 14, Popular Architectures stops listing modules and reads released models. Section 14.1 tells the history as four shifts: foundations and open replication; convergence on a reusable dense baseline; diversification of capacity and of state; then joint configuration of capacity, state, and prediction. Section 14.2 names the four patterns in Patterns. The survey’s dense-model table and mixture-of-experts table are the model-by-model record behind those patterns.

Section 15 is the teaser at the top of this page, written out as assumptions and alternatives. The dimensions are in Design space. Section 15.3, Interdependencies and Joint Design is the reason a module list is not enough: operator choice sets the form of history; sparse activation and compressed history spend different budgets inside one serving stack; tokenization sets the unit of both memory and the prediction horizon; residual paths have to stay stable when depth or the sequence operator changes; a longer draft creates state that may be discarded.

Section 16, Concluding Remarks draws three conclusions.

1. The canonical dense decoder, with a fixed schedule and the usual Pre-RMSNorm, SwiGLU, and rotary positions, is still the reference. Most of the survey relaxes one assumption and keeps the rest.
2. The valuable comparisons are joint configurations. Sparse capacity with compressed attention state, hybrid sequence and state processing, and prediction-augmented backbones show up because their pieces interact.
3. The usable design space is bounded by load balance, kernels, credit assignment, and caching. At least one hybrid family returned to full attention when the infrastructure was immature, and later moved to block-sparse attention. Those decisions get revisited.

The open measurement problems Section 16 names are scaling laws for prediction horizon, recurrent depth, and state size under matched training and inference compute; stable optimization of conditional and recurrent computation; and evaluations that report expressivity, efficiency, and stability separately.

---

🧩 Design space

Section 15.1 states the canonical computational structure. Every token uses a fixed tokenization and the same dense blocks. History is an append-only, token-level key–value cache. Each state is trained to predict the next token, and that token is committed before a longer continuation is compared.

Section 15.2 decomposes the four directions in the teaser into ten design dimensions.

Direction	Dimension	Canonical assumption	Alternative	Where it is developed
Flexible and conditional processing	Sequence interface	One fixed tokenization and positional scheme	Finer, multi-scale, or adaptive units; other positional structure	Sections 10 and 11
	Operator assignment	The same sequence mixer in every layer	Different operators across layers or heads	Sections 5, 6, and 14
	Parameter activation	Every token runs the same dense blocks	Input-dependent routing among experts	Section 7
	Computation allocation	A fixed path and a predetermined amount of work	Early exit, conditional depth, recurrent depth, or a drafter–verifier split	Sections 12 and 13
Refined transformation dynamics	Local nonlinearity	A fixed feed-forward nonlinearity, often SwiGLU	Other gates, such as squared ReLU or clamped SwiGLU	Section 8
	Depth propagation	Fixed residual and normalization pathways	Other placement, residual scaling, or gating	Section 9
Selective and flexible state	History representation	Individually addressable token states	Selective, compressed, recurrent, or hybrid history	Sections 5 and 6
	State update	Append-only growth	Eviction, merging, compression, branching, or rollback	Sections 5, 6, and 12
Lookahead and delayed commitment	Prediction horizon	Next-token supervision only	Multi-step, blockwise, or feature-level prediction	Section 12
	Output commitment	One committed token per forward pass	Drafting, parallel verification, delayed commitment	Section 12

A single mechanism can move more than one row. Mixture-of-experts changes parameter activation. Speculative decoding changes output commitment and, while candidates are alive, state update. The teaser draws the rows as four panels because that is how the survey asks the question, not because a model may enter only one panel.

🏗️ Patterns in released models

Section 14.2, Recurring Architectural Design Patterns defines a pattern as a combination. The four are not a partition. One backbone can instantiate more than one.

📐 14.2.1 · Canonical dense decoders Decoder-only stack, dense feed-forward blocks, fixed depth. Pre-RMSNorm, SwiGLU, and rotary positions became the usual way to fill that skeleton. LLaMA is the reference instance. Later dense families change the number of key–value heads or the attention span, as Mistral does with grouped-query attention and a sliding window, without leaving the dense path.

🎛️ 14.2.2 · Sparse capacity, compressed attention state Expert routing and cache compression spend different budgets. DeepSeek-V2 and DeepSeek-V3 pair fine-grained experts and a shared expert with multi-head latent attention. Other releases pair the same expert layout with grouped-query attention, or with alternating local and global attention. Co-occurrence is not evidence that either piece improves the modeling quality of the other.

🔀 14.2.3 · Hybrid sequence and state processing Softmax attention keeps an addressable history. A recurrent or linear operator compresses history as it is built. Griffin, Jamba, Nemotron, Qwen3-Next, and Kimi Linear interleave the two across layers. Hymba runs them inside a layer. LoLCATs converts a pretrained attention model after the fact. How much softmax attention remains decides how much of the cache is actually gone.

🔭 14.2.4 · Prediction-augmented backbones A multi-token objective or module sits on top of a dense, sparse, or hybrid trunk. DeepSeek-V3, MiMo-7B, LongCat-Flash, and MiMo-V2-Flash are instances. The module changes which future tokens supervise the representation. It does not, by itself, decide how a draft is verified or committed. That decision is Section 12.3.

---

🚀 Entry points

A longer, section-numbered literature map is in README_detailed.md. The rows below are the shortest path through the survey.

If you are reading	Start with
Section 3–4	Shannon’s mathematical theory of communication; LSTM; Attention Is All You Need; Kaplan et al. and Hoffmann et al. on scaling
Section 5	Multi-query attention; grouped-query attention; H₂O; StreamingLLM; SepLLM; DeepSeek-V2 / multi-head latent attention; MiniCache
Section 6	Transformers are RNNs; Linear Transformers Are Secretly Fast Weight Programmers; S4; Mamba; Mamba-2; Gated DeltaNet; test-time regression
Section 7	Adaptive mixtures of local experts; sparsely gated MoE; Switch Transformer; Expert Choice; DeepSeekMoE
Section 8–9	GELU; GLU variants / SwiGLU; LayerNorm; RMSNorm; Pre-LN analysis; Hyper-Connections
Section 10–11	RoPE; ALiBi; position interpolation; YaRN; SentencePiece; ByT5; scaling laws with vocabulary
Section 12–13	Speculative decoding; Medusa; EAGLE; DeepSeek-V3; Mixture-of-Depths; Universal Transformers; Huginn
Section 14	LLaMA; Mistral 7B; Mixtral; Jamba; Qwen3-Next

---

📎 Citation

If you find our paper helpful, please cite it. Thank you.

@article{llmarchsurvey2026,
    doi = {10.20944/preprints202610.1532.v1},
    url = {https://doi.org/10.20944/preprints202610.1532.v1},
    year = 2026,
    month = {October},
    publisher = {Preprints},
    author = {Yao Teng and Guoxuan Chen and Yimeng Chen and Han Shi and Piotr Piękos and Jie Ran and Yutao Zeng and Chuanyang Zheng and Jingyao Li and Li Nanbo and Haoli Bai and Chao Huang and Xihui Liu and Di He and Weiyang Liu and Jürgen Schmidhuber},
    title = {Towards Next-Generation Architectures of Large Language Models: A Survey},
    journal = {Preprints}
}

Paper on Preprints.org · doi:10.20944/preprints202610.1532.v1

👥 Authors

Yao Teng^1,* · Guoxuan Chen^1,* · Yimeng Chen^2,† · Han Shi^3,† · Piotr Piękos² · Jie Ran³ · Yutao Zeng⁴ · Chuanyang Zheng³ · Jingyao Li⁵ · Li Nanbo²
Haoli Bai³ · Chao Huang¹ · Xihui Liu¹ · Di He⁶ · Weiyang Liu^5,7 · Jürgen Schmidhuber^2,8

¹The University of Hong Kong  ·  ²Center of Excellence for Generative AI, KAUST  ·  ³Huawei Technologies  ·  ⁴ByteDance Seed
⁵The Chinese University of Hong Kong  ·  ⁶Peking University  ·  ⁷Max Planck Institute for Intelligent Systems  ·  ⁸The Swiss AI Lab, IDSIA-USI/SUPSI

_{* Equal contribution.    † Corresponding authors.}

---

_{Companion to Towards Next-Generation Architectures of Large Language Models: A Survey. Section-by-section reading lists are in the literature map.}