Hyper-SET: Designing Transformers via Hyperspherical Energy Minimization
Authors: Yunzhe Hu
Difan Zou
Dong Xu
Venue: ICLR 2026 Poster (2026)
Keywords: Transformer
Hopfield Energy
Principled Model Design
Primary area: interpretability and explainable AI
TL;DR: We present Hyper-SET, a framework to design energy-principled Transformer derived from minimizing quantified token dynamics on the hypersphere, achieving strong performance with interpretability and scalability.

Abstract: Transformer-based models have achieved remarkable success, but their core components, Transformer layers, are largely heuristics-driven and engineered from the bottom up, calling for a prototypical model with high interpretability and practical competence. To this end, we conceptualize a principled, top-down approach grounded in energy-based interpretation. Specifically, we formalize token dynamics as a joint maximum likelihood estimation on the hypersphere, featuring two properties: semantic alignment in the high-dimensional space and distributional uniformity in the low-dimensional space. By quantifying them with extended Hopfield energy functions, we instantiate this idea as a constrained energy minimization problem, which enables designs of symmetric attention and feedforward modules with RMS normalization. We further present *Hyper-Spherical Energy Transformer* (Hyper-SET), a recurrent-depth alternative to vanilla Transformers naturally emerging from iterative energy optimization on the hypersphere. With shared parameters across layers, Hyper-SET can scale to arbitrary depth with fewer parameters. Theoretically grounded and compact, it achieves competitive or superior performance across diverse tasks, including Sudoku solving, image classification, and masked image modeling. We also design novel variations under the proposed general principle, such as linear attention and gated feedforward layer, and showcase its scalability with depth-wise LoRA. Our results highlight Hyper-SET as a step toward interpretable and principled Transformer design. Code is availabel at https://github.com/huyunzhe/hyper-set.

Published as a conference paper at ICLR 2026
HYPER-SET: DESIGNING TRANSFORMERS VIA HY- PERSPHERICAL ENERGY MINIMIZATION
Yunzhe Hu', Difan Zou Dong Xu'
Sccooo of Computing and Data Science, The University of Hong Kong
Shenzhen Loop Area Institute {yzhu, dzou, dongxu 0 hku. .k
ABSTRACT
Transformer-based models have achieved remarkable success, but their core com- ponents, Transformer layers, largely heuristics-driver and engineered from the bottom up, calling for a prototypical model with high interpretability and practi- cal competence. To this end, we conceptualize a principled, top-down approach grounded in energy-based interpretation. Specifically, we formalize token dynam- ics as a joint maximum likelihood estimation on the hypersphere, featuring two properties: semantic alignment in the nigh-dimensional space and distributional uniformity in the low-dimensional space. By quantifying them with extended Hopfield energy functions, we instantiate this idea as a constrained energy mini- mization problem, which enables designs of symmetric attention and feedforward modules with RMS normalization. We further present /lyper-Spherical Energy Transformer SEYPER-SET) recurrent-depth alternative to vanilla Transformers naturally emerging from iterative energy optimization on the hypersphere. With shared parameters across layers, HYPER- can scale to arbitrary depth with fewer parameters. Theoretically grounded and compact, it achieves competitive or superior performance across diverse tasks, including Sudoku solving, image classification, and masked image modeling. We also design novel variations un- der the proposed general principle, such as linear attention and gated feedforward layer, and showcase the scalability with depth-wise LoRA. Our results highlight HYPER-SET as a step toward interpretable and principled Transformer design. Code is available at https://github.. httpgggiiiub.ohuuuuunaiiiie http///uuunzhhhypprrseeee
INTRODUCTION
Transformer- based models (Vaswani et al., 2017) have become foundational across diverse domains, including computer vision (Dosovitskiy et al., 2021; Bao et al., 2022; He et al., 2022: Peebles & Xie, 2023), natural language (Devlin et al., 2019; Lan et al., 2020; Brown et al., 2020), robotics (Brohan al., 2022; Chen et al., 2021), and scientific discovery (Jumper et al., 2021; Kamienny al., 2022). In recent years, there has been evidence that scaling up model size, dataset size, or computational budget during pre-training can yield unprecedented performance gains (Kaplan al., 2020), driving the proliferation of Transformer-based foundation models (OpenAl et al., 2024; Dubey et al., 2024; Anil et al., 2023; Oquab al., 2024).
Despite these achievements, the architecture of Transformers--eppecilly the role of individual lay- ers ers-remains largely heuristic. For instance, empirical studies have observed high redundancy in the deeper layers (Gromov et al., 2025; Men et al., 2025), uniformity of representations in the mid- dle layers (Sun et al., 2025), and robustness to permuting certain intermediate layers (Lad et al., 2025) in LLMs. These findings suggest convergent functionality that one layer represents, yet our understanding of its role in processing information and representation learning remains limited. While interpretability efforts to unveil the function underlying network layers, especially Trans- former blocks, exist spanning mechanistic interpretability (Elhage et al., 2021; Nanda et al., 2023; Wang et al., 2023; Huben et al., 2024), causal mediation analysis (Vig al., 2020; Meng et al., 2022), and visualization (Bricken et al., 2023; Olsson et al., 2022) most of them focus on post-hoc interpretation and phenomenological approaches. This motivates pivotal question:

---

Published as a conference paper at ICLR 2026
Can we find design a function prior that induces model interpretable by construction?
One approach to achieving intrinsic interpretability to emhed an explicit optimization process into neural networks, known as modcl-based deep learning (Shlezinger et al., 2023). Prior works have designed networks that solve domain-speciff problems such as constraint satisfaction (Wang al., 2019), optimal control (Amos & Kolter, 2017; Amos et al., 2018), physical simulation (Greydanus et al., 2019; Karniadakis et al., 2021). However, these models often rely on fixed task priors and lack generality.
Another more general avenue is energy-based learning (EBL) (Dawid & LeCun, 2024), which frames prediction as minimizing a scalar energy function Eg y) over outputs y conditioned on inputs Within this framework, Energy Transformer (Hoover et al., 2024) interprets Transformer layers as iterative optimization over the canonical continuous Hopfield energy (Ramsauer et al., 2021; Krotov & Hopfield, 2021) yet focuses on mechanistic analogies to associative memory with- out grounding its formulation in specific representational challenges. In contrast, our goal to find a design principle from top down that not only reinterpret existing components but is also generalizable to novel architecture design constructively.
In this work, therefore take fundamentally different approach by introducing a principle grounded in maximum likelihood estimation (MLE) for tokens on the hypersphere. Under mild assumptions, we interpret it under two complementary objectives for representation dynamics: mantic alignment (mode seeking) in high-dimensional space and distributional uniformity (mass covering) in low-dimensional subspace. To translate these objectives into optimizable quantities over tokens, we define two complementary Hopfield-style energy functions that quantify these ob- jectives and can be minimized through iterative optimization. This leads the Hyper-Spherical Energy Transformer (HYPER-SET)) symmetric attention, feedforward layers, RMSNorm, and skip connection emerge naturally from the optimization dynamics. With only one set of shared parameters across iterations, HYPER-SET is compact, interpretable by design, and empirically competitive across diverse tasks, including rea- soning, classification, and masked image modeling. Beyond single instantiation, this principle can induce novel architectural designs by generalizing the energy functions, enabling variants such as linear attention and gated feedforward layer. To enhance scalability, we introduce depth- depth-wise low-rank adaptation (LoRA), allowing flexible iteration-specific modulation with minimal parame- ter overhead. Our key contributions are summarized follows:
recurrent-depth model in which core components such as
Theoretical Formulation: We conceptualize general principle for information processing in layer dynamics based on maximum likelihood estimation on the hypersphere with two properties: uniformity and alignment, quantified via complementary Hopfield-styl style energy functions.
2. Energy-Driven Architecture: We derive a compact bransformer-based model through pure energy minimization, where core components cauponents-iicluuinng symmetric attention, feedforward, RMSNorm (Zhang & Sennrich, 2019), and skip connection emeere emerge naturally.
Competitive Performance: We show competitive performance to vanilla Transformer across reasoning, classification, and masked modeling while demonstrating generality to design novel components (e.g., linear attention, gated feedforward) and scalability with flexible computation.
2
RELATED WORK
2.1 ENERGY-BASED LEARNING
Energy-based learning (EBL) (LeCun et al., 2006; Dawid LeCun, 2024) provides unifying framework for modeling prediction minimizing an energy function. Early forms include Hopfield networks (Hopfield, 1982) and Boltzmann machines (Ackley et al., 1985). Modern developments in EBL span both generative modeling via energy functions (Du & Mordatch, 2019) or their gradients (as in score-based models (Sohl-Dickstein et al., 2015; Song & Ermon, 2019))- and representation learning. Another line of work views network layers the result of iterative energy minimiza- tion. Some approaches define energy implicitly through neural networks (Bai et al., 2019; Du et al., 2022; 2024), while recent work Energy Transformer (Hoover et al., 2024) draws analogies between
2

---

Published as a conference paper at ICLR 2026
Renation
Beration 3
leration 5
ltemioin
Deration
leratio 11
Subspace
Low-
High-dim. Space
Figure 1: Evolution of tokens in the forward pass. Top: Tokens projected onto subspaces are pro- gressively separated on the low-dimensional hypersphere. Bottom: Tokens gradually align with an- chor vectors in the high-dimensional hypersphere. Visualization is carried out in dice-dimensional space for illustrative purposes.
attention layers and explicit energy descent but mainly focuses on reinterpretation rather than prin- cipled derivation. Our work differs in that design the Transformer block by quantifying general principle that can induce variants through alternative energy.
Other works also explore energy formulations the hypersphere (Liu et al., 2018; Loshchilov et al., 2025), but mostly in the weight space. By contrast, we define our energies directly on the rep- resentation space. Additionally, recent theoretical studies memory capacity in modern Hopfield networks (Hu et al., 2024a; Wu et al., 2024a; Hu et al., 2024b) emphasize spreading patterns on the sphere but focus primarily on memory retrieval and cross-attention.
2.2 MODEL DESIGN FROM FIRST PRINCIPLES
While neural network architectures often shaped by engineering practices, recent work has plored designing or interpreting them through principled lenses like signal processing, information theory, and neurobiology. For example, deep unrolling of the sparse coding algorithms has led to the development of fully connected networks (Gregor & LeCun, 2010), convolution networks (Pa- pyan et al., 2017: 2018), and even graph neural networks through iterative algorithms (Yang al., 2021). Similarly, the sparse rate reduction principle has been used to derive the Transformer archi- tecture (Yu et al., 2023). Other approaches draw inspiration from approximation theory (Liu al., 2025) and brain computation (Kozachkov et al., 2023), further bridging the gap between theoretical insights and practical network design.
2.3 DEPTH RECURRENCE IN TRANSFORMERS
Recurrence has been scientifically considered core computational mechanism the biological visual system enabling flexible computational depth, integration of priors, and efficient allocation of compute under resource constraints (van Bergen & Kriegeskorte, 2020). Depth-wise recurrence in Transformer architectures, under the name of universal (Devlin et al., 2019). recursive/recureent (Bae et al., 2025; Geiping et al., 2025) or looped Transformers (Giannou et al., 2023), where cross- layer weights are shared, has emerged iterative reasoning. Its repeated reuse of the same layer has been demonstrated to be capable of em- ulating iterative algorithms (Schwarzschild et al., 2021; Saunshi et al., 2025). Crucially, has been demonstrated with particular strength mostly on systematic generalization (Csordás et al., 2021) and structured reasoning tasks (Schwarzschild al., 2021; Bansal al., 2022), while its applications on -eneral-domain tasks that need hierarchical processing (e.g., image perception) are under-explored.
critical avenue for reducing parameters and enabling
3 CONCEPTUALZATTION AND INSTANTIATION
To answer the introductory question, conjecture that effective representations should exhibit two complementary propertics: semantic alignment in a high- aigh-dimensional space and distributional uniformity in a low-dimensional subspace. This dual perspective reflects the balance of mode seek-
3

---

Published as a conference paper at ICLR 2026
xt
BMSS
Narm
JENES
Norm
RMS
2
Narm
MLP
Linear
erbodding
X
Figure 2: Overview of hyperspherical energy Transformer layer. It recovers sequential stacking of symmetric self-attention, feedforward, skip connection, and RMSNorm from sheer minimization of extended Hopfield energy. Adaptive step sizes learned given the current i. and initial input Xo-
ing and mass covering terms we characterize the interplay between information preservation and entropy collapse prevention in representation learning. Figure 1 shows an illustrative example.
We formalize this conceptualization under maximum likelihood estimation. Specifically, we instan- tiate the forward dynamics as an optimization over token-level vector æ: balancing two terms:
min
DEL (p(z) (l(=))lp((*)) x))
log po( (z).
uniformity
alignment
where R represents low-ding -iwddimesioonl projections of the high dighnssieenii representation
The first term encourages the projections = to approximate a prior uniform distribution p(z) on hypersphere, thus maximizing entropy and mitigating representational collapse. The second term promotes alignment between & and mean directions, which can be modeled using von distributions. A detailed justification and interpretation of this objective is provided in Appendix A.
This objective resonates with but differs from the contrastive learning objective that unifies align- ment and uniformity in shared latent space (Wang & Isola, 2020). Our work instead takes energy view to quantify these two key ingredients into optimizable functions of æ that can induce Transformer architectures.
HYPERSPHERICAL ENERGY TRANSFORMER FROM ITERATIVE ENERGY MINIMIZATION
In this section, we translate the proposed instantiation into two modified Hopfield energy functions defined on hyperspheres (see Appendix B for preliminaries and definition of Hopfield energy EMCH in Eq. 16). Through iterative energy minimization, the architectural components of Transformer layers naturally arise under this framework. The overview is presented in Figure 2.
4.1 HYPERSPHERICAL ENERGY
Let X = -xI..... NN denote set of N contextual token vectors, each w; E Rdd These tokens are projected into distinct subspaces via basis matrices W [Wi.... WH E dy, where each W, E _dxp spans a p-dimensional subspace. Additionally, we define second set of bases D= [d dd....,,yyy xdx to encode semantic directions the original space. Unless otherwise specified, we assume these basis vectors are incoherent and span the full space, i.e., Hp M = d.
4.1.1 OVERCOMING TOKEN SYNCHRONIZATION VIA REPULSIVE DYNAMICS
Motivated by a recent argument that the contextual tokens lie on a low-dimensional manifold their high-dimen dimensiooi ambient space (Yu al., 2023), study the projection of tokens with bases W; for a subspace spanned by W.. the latent representation of a token æ; can be written as
W. a7;-
4

---

Published as a conference paper at ICLR 2026
Canonical Hopfield energy EMCH tends to align vectors with stored patterns. This interaction of- ten occurs between dynamic tokens and static patterns. However, in Transformers self-attention, this interplay happens among all dynamic tokens simultaneously. Enforcing strict alignment among them risks collapsing representations into degenerate clusters, reducing expressiveness. This phe- nomenon has been observed empirically rank collapse (Dong et al., 2021), I), and theoretically characterized in (Geshkovski et al., 2023). It also relates to the synchronization effect in coupled systems (Acebrón al., 2005; Miyato al., 2025).
oversmoothing (Chen et al., 2022; Wu et al., 2024b) or
Therefore, to overcome this issue, we extend the Hopfield energy EMCH to model the repulsive force among tokens and quantify their distributional uniformity in each subspace, which serves as surrogate of the uniformity measure in Eq. 1. For subspace h. this energy is given by
N Σ
EXITN B
log
exp (3(2() (x/)),
where 3 is usually the inverse of temperature. Here we use the subseript ATTN as this energy will be shown to be related to the design of the attention layer, resembling that in Yu et al. (2023). Aggregating over all subspaces, the total energy that models interacting tokens would be
H
E. EATTN = Σ FATIN
subject to [ 2:lle = vp-
(4)
1=1
The constraint ensures that the dynamics take place on a low-dimensional hypersphere of radius vD. Minimizing EATTN thus encourages token spread evenly on multiple hyperspheres, mitigating collapse and promoting distributional uniformity.'
4.1.2 SEMANTIC ALIGNMENT VIA ATTRACTION TO BIGH-DIMENSIONAL BASES
While the subspace projections separate to occupy volume thus regularizing distribution, seek to enrich the high-dimensiona representations per se. From an information- -thoorrttc perspec- tive (Tishby al., 2000; Tishby & Zaslavsky, 2015), effective representations require compress- ing uninformative redundancy while preserving salient information. Hence, in the original high- dimensional space, we encourage token alignment with set of directions that contain knowledge from data to reduce entropy for minimal coding bits.
Motivated by empirical findings that feedforward layers in Transformers store much of their knowl- edge (Geva et al., 2021; Dar et al., 2023), we interpret the basis vectors D as the semantic directions. One surrogate function to implement this attractive energy for alignment in Eq. 1 defined as
N
M
EFF
- 2 i=1 m=1
1
(ReLU (d), subject to || x; VM.
(5)
Here we use the subscript FF as this energy relates to the design of the feedforward layer. This energy favors alignment between tokens and those basis directions forming acute angles (as filtered by ReLU), while maintaining the hyperspherical constraint in the original space. Geometrically, each token is drawn toward a union of attractive half- half-spaces defined by D. This could imply that each token may bind patterns combinatorially beyond the number of basis vectors defined by D.
4.1.3 DUAL ENERGY THE HYPERSPHERE
By combining these two hyperspherical energy functions, we introduce a unified objective function that characterizes the functionality the Transformer layer represents:
min EX
E(X; W, D) = FATTN + EFF,
(6)
#I....NEE
subject to I 2 = vp. - i-1,..., N.
Iteratively minimizing this energy under spherical constraints induces the core architecture of Trans- former layers: self-attention module arises from repulsive energy over subspaces and feedforward module arises from attractive energy in the ambient space. To solve optimization Eq. 6, we adopt an alternating minimization method by splitting it into sub-problems, following Yu et al. (2023).
Its asymptotic convergence to uniformity the sphere has been proven in Liu (2018).
5

---

Published as a conference paper at ICLR 2026
4.2 SYMMETRIC STRUCTURE INDUCED FROM ENERGY MINIMIZATION
4.2.1 ATTENTION MODULE FROM UNIFORM ENERGY
To show how have attention module derived from minimizing hyperspherical energy EXTTN in Eq. 4, we first establish the differential equation that models the evolution of tokens' interactions:
X
V x EATIN
H
WaW,
W
softmax (B(W x) (W x)) + softmax (B(W), x) (W x))
column-wise
row-wwis
where B VP as in vanilla Transformers. Derivations could be found in Appendix C.I.
The constraint on the low-dimensional hypersphere of radius vp corresponds to RMSNorm()) which bears resemblance to Query- Key Normalization (Henry et al., 2020), but here the normaliza-
tion is applied after projection by the same query-key-valuu matrix. The projections in subspace onto the hypersphere thus read as
ZRMs = RMSNNrm 2d) = RMSNorm W; X).
(8)
By discretizing the differential equation Eq. 7 with step size ar and maintaining the constraint Eq. 8, we obtain an self-attention module; let [QK]RMs. = ((( (Rss,)). then the update will be:
X X, Ott
Wn RMS, softmax ([QK)RMs.)) W,, Zd RMS, softmax ([QK) JRMS.)
column-wise
row-wiie
This update yields a doubly symmetric multi-head attention operator, where both the query-key product and attention weights are symmetric under row and column operations. This structure connects with formulations of Wasserstein gradient flows using doubly stochastic attention (Sander al., 2022), grounding our energy-based interpretation.
4.2.2 FEEDFORWARD MODULE FROM ALIGNMENT ENERGY
For the sub- uub-problem of minimizing the alignment energy EpF in Eq. 5, we have similar construction of the corresponding differential equation, with details deferred to Appendix C.2:
X == V EFF ReLU (DTx).
(10)
By further imposing the high-dimensional hyperspherical constraint via RMSNorm with discretiza- tion step size ?/s we can recover the feedforward layer that exhibits symmetry in the weight space:
X X, D. ReLU (RMSNorm (D x.))
(11)
4.3 LEARNING ADAPTIVE STEP SIZE
To make the step sizes more flexible, we choose to learn their embedding with a neural network conditioned on the current iteration t and the initial token (0) (usually the output of the tokenizer):
- (t, x00). 7/ 7t. (0)).
(12)
For each iteration, step size embeddings in Eq. 12 are applied channel-wise to each token, similar to techniques in Touvron et al. (2021); Peebles & Xie (2023) and detailed Appendix D.I. We also adopt the inn-iintiiaiiztionn of network parameters n and & from Bachlechner et al. (2021) to facilitate convergence when using larger iterations.
In summary, by combining all the components and techniques, we present the Hyper-Spherical En- ergy Transformer with only one layer of learnable parameters. This one-layer model is amenable to rigorous analysis and, demonstrated later, has competitive performance with vanilla Transformer.

---

Published as a conference paper at ICLR 2026
Board Accuracy (%)
60
Board Accuracy (%)
58
56.20
56.
73
90
57
03
57.
56
40
wy
54.77
Ours (5.20 M)
54-
Transformer (7.10 10 M)
Distriibution
Ours
20
Trausforiuer
52-
50.63
51.03
13 31.13 51.17 51.20
Energy Transformer
50
+1.330
CRATE
50
100
150
200
24
28
32
36
44
48
Training Epochs
Forward Iterations
(a) Sudoku training dynamics.
(b) Sudoku test-time extrapolation.
Figure 3: Left: Training dynamics of different approaches. Our model achieves superior training curve, while Energy Transformer and CRATE both fail to make non-trivial predictions (i.e., flat curves). Right: Test-time extrapolation w.r.t. forward iterations. Our model achieves better perfor- mance over five runs with fewer parameters, even when the iterations are beyond the training regime.
EXPERIMENT
In this section, we evaluate HYPER-SET against vanilla Transformer and other baselines on discrim- inative and generative tasks. For fairness, we remove biases and dropout, use the Pre-Norm style with RMSNorm, and set the MLP ratio to in Transformer. We use one-layer trainable parameters but vary the forward iterations for all models, including Transformers, unless otherwise specified."
5.1 SOLVING SUDOKU
Setups. We use the challenging dataset from Palm et al. (2018), featuring boards with only 17 to 34 known digits. We build on the code from Yang al. (2023) and follow the setting of training on 9k samples and evaluating on 1k. See Appendix D.2 for details.
Extrapolation to Out-of-Distribution Iterations. Under identical experimental conditions, model exhibits faster and superior training dynamics over Transformer, while Energy Transformer (Hoover et al., 2024) and white-box Transformer CRATE (Yu al., 2023) both fail on this task, as shown in Figure 3a. It also outperforms Transformer for in-distribution evaluation 55%7 vs. 49.30%), i.e. using the same forward iterations for training and inference.
Recent efforts also explore test-time compute scaling to enhance reasoming (Schwarzschild et al., 2021: Bansal al., 2022: Du et al., 2022; Banino et al., 2021), aiming to extrapolate the algorithms during training. Building on this idea, we increase test-time iterations up to 2x of training ones. As shown in Figure 3b, our model scales more effectively than Transformer, with larger accuracy gains. We attribute this extrapolation to learned adaptive step sizes that preserve energy minimization. In practice, we also find that trainable positional encoding vital for the extrapolation.
Sudoku could be challenging for some architectures to achieve non-zero results, the 9x9 grid re- quires the exact digits to meet the constraints, which may be hard to learn without proper inductive bias. Even some foundation models have zero accuracy (Wang et al., 2025). A reasonable explana- tion for HYPER-SET success, where CRATE and Energy Transformer fail, is that we have better modeling and more realistic assumptions about the design objective, which is based on fundamen- tal as maximum likelihood, making the architecture better aligned with the optimization procedure. This allows HYPER-SET enjoy both the principled design and the expressivity of Transformer.
5.2 IMAGE CLASSIFICATION
Setups & Results. We also compare HYPER-SET on CIFAR-10/100, ImageNet-110; and the full ImageNet-IK. against ViTs, CRATE (Yu et al., 2023) its variant CRATE-T that aims for more
FFor instance, 12 iterations
applying the layer repeatedly 12 times.
hhtpp://githubb.. httpp://gthhh.cmmmarressm www.///reasmmer//rcrrraoo transformer We use subset of ImageNet-1K from http:///ticccccoooiieneneooohohe
7

---

Published as a conference paper at ICLR 2026
Ours (Small Scale-up)
Ours (Base)
Transfommrr (Small)
66.00
20.0
Aocuracy (%)
65.00
Params (M)
64.38
15.0
61.00
63:00
10.0
62.00
5.0
61.00
0.0
(1 Li(Iee)))
L1 (Iee11)) L12((er))
Layer (# Iteration)
Layer (# Iteration)
Figure 4: Top-1 accuracy (Left) and model parameters (Right) on CIFAR-100 with different layer- iteration trade-offs. Our model (Base) consistently surpasses Transformer, even its upper bound performance, with parameter efficiency. Error bars represent standard deviation over five runs.
faithful implementations (Hu et al., 2024c), and Energy Transformer (Hoover et al., 2024). Detailed setups are Appendix D.3.
Table I: Top-1 accuracy (%) for image classification with single-layer recurrent-depth models. Pa- rameters measured on ImageNet- I. All models trained from scratch the listed datasets.
Width
# Params (M)
Dataset
Model
CIFAR- CIFAR-100
IN-100
IN-IK
Transformer
384
2.38
89.90
61.89
69.44
66.94
CRATE-T (Hu al., 2024c)
896
3.04
87.54
60.23
68.16
57.89
CRATE (Yu al., 2023)
768
3.00
84.81
58.22
68.52
57.00
Energy Transformer (Hoover et al., 2024)
512
2.39
76.39
50.60
36.68
34.24
HYPER-SET (Ours)
512
2.39
90.11
63.41
70.16
62.76
HYPER-SET (Ours)
640
3.40
89.96
64.60
69.31
66.21
Table shows that, under properly parameter-aligned settings, our model surpasses others on CIFAR- 10/100 and ImageNet-100 but lags behind Transformer large-scale ImageNet-IK. No- tably, our architecture achieves higher width-parameter ratio compared to Transformer, meaning more parameter-efficieet under the same width. When scaling up the width d, our model can nar- row the performance gap to Transformer. This implies that our principled model suits better under resource-constrained setting its inherent structural biases could limit its scaling on large datasets.
Layer-Iteration Trade-off. So far, the classification is conducted using one-layer model. A natural question how well the model performs when stacking multiple layers with different pa- rameters. To see this, first train Transformer with 12 layers-equivalle one layer with 12 iterations as an upper bound. We then vary the number of distinct layers and their iterations while keeping total depth constant, effectively introducing flexibility the basis vectors.
in effective depth to
In Figure 4, our scaled-up small 11 model has parameter efficiency across varied layer-iteration ratios, with this strength intensifying as independent layers trained. However, its architectural ef- ficiency limits scalability beyond two layers. Scaling to the Base configurations enables our model consistently outperform Transformer, exceeding the upper bound while retaining less parameters.
Non-Recuree Scenarios. To demonstrate the practicality in the non-recurrent settings, which are more common in large-scale applications, we further provide comparisons in a 12-layer non- recurrent setting Table 2, where we report results on CIFAR- 10/100 with fine-tuning or training from scratch and on large-scale ImageNet-IK. The ImageNet-1KK pretraining setup follows Table 1, with fine-tuning batch size 256, epochs 50, learning rate 1e-4, and weight decay 1e-5. Our model performs comparably standard Transformers while being more efffeiieeeeeffennt,, and this pa- rameter reduction constantly grows when stacking more distinct layers, as shown in Figure 4, thanks to our parsimonious design. This suggests that HYPER-SET potentially preserves the benefit of
We compare the parameter efficiency and computational cost in Appendix D.6.
8

---

Published as a conference paper at ICLR 2026
(betrog-)
Fer (00empp))
(Derrpp))
X110
Attention Module
X08P
FoodPorward Modole
w100
Astenttoo
FoodRorward
6.5
0.
455
0.0
3
5
5
6
15
Forward Ieraatinns
Forward Iterations Sonss
Forward Iterations
Figure 5: The attention and feedforward energy decrease Sudoku (Top) and CIFAR-10 (Down) even without sign constraints on step sizes. This suggests the layer aligns well with the optimization objective. Normalization is first applied meet constraints in Eq. 6 before computing the energy- In the right subfigure, the decrease in the overall energy corresponds to the increase in performance.
scaling in Transformers when extended practically, while being more useful in resource- constree- settings where model size matters, further underscoring its promise in practical scenarios.
Table 2: Top-1 accuracy (%) for image classification with 12-layer non-recurrent-depth models. Parameters are measured on ImageNet- IK. means first pretraining on ImageNet- K and then fine- tuning on CIFAR-10/100.
Width
# Params (M)
Dataset
Model
CIFAR-10
CIFAR-100
CIFAR-10*
CIFAR-
IN-IK
Transformer
384
21.86
87.44
62.84
96.95
83.10
67.90
CRATE (Yu et al.. 2023)
512
10.28
88.76
63.94
94.18
77.39
60.69
HYPER-SET (Ours)
512
8.17
88.82
64.98
95.76
80.89
66.26
HYPER-SET (Ours)
768
17.56
88.53
64.16
96.47
82.60
67.20
5.3 MASKED IMAGE MODELING
Setups & Results. Masked image modeling has recently regained its traction for autoregressive generation (Li et al., 2023; 2024), framed recovering images from 100% masking. Due to high computational demand, we attempt to demonstrate the power of our one-layer model specifically for image reconstruction on ImageNet-100. We build on prior work (Chang et al., 2022) and use the open-source repository. Concrete settings in Appendix D.4, with additional results and visualization in Appendix E.
Table 3 unveils that, under the same number of iterations, our model significantly reduces parameters but lags behind Transformer on all metrics. If we further increase its iterations and the width of feedforward module M to 8d, can fill in the performance gap but at the cost of more computation.
Table 3: Comparisons of masked image modeling performance on ImageNet-100 (5k). While our model is more compact, scaling the feedforward module width (M) allows it to match Transformer performance. All models use single layer.
Model
Iters
FF Ratio M
Params (M)
PSNR (1)
SSIM (1)
MS-SSIM (1)
LPIPS (L)
FID ())
Transformer HYPER-SET (Ours)
12
4d
8.85
15.953
0.417
0.599
0.327
43.428
ld
3.94
15.713
0.411
0.376
0.358
59.511
HYPER-SET (Ours)
24
8d
8.07
15.955
0.417
0.596
0.332
45.174
5.4 ENERGY EVOLUTION, EFFECTIVE RANK AND AVERAGE ANGLE
Figure shows energy trajectories of the attention (FATTN) and feedforward module (FFp). Even without a positive threshold for step sizes a and Yr, the energy Sudoku still decreases within

---

Published as a conference paper at ICLR 2026
Head
Head
Head
Head
bofoethey Falll Baank
Averr Angle (*)
EFFottvve Pull Ronk
Angle
Bixenn Rask
82
Pvl Rack
Avenng
Avernee Angle
10
Forward Iterations
Forward Iteratioon
Forward Iterations
Forward Iterations
(a) Sudoku.
(b) CIFAR-10.
Figure 6: The effective rank and average angle of tokens projected to subspace gradually in- crease, suggesting a larger volume spanned by these tokens. Results are from Sudoku test dataset (Palm et al., 2018) (Left) and CIFAR-I0 validation set (Right).
training iterations and extrapolates smoothly beyond them, indicating strong generalization of learned step sizes. On CIFAR-10, our designed energy exhibits monotonic decline well.
To verify our subspace uniformity objective, we track two metrics-effective rank and average an- gle-defined in Appendix D.5. As shown in Figure 6, the subspace effective rank steadily increases while the full rank remains unchanged. Meanwhile, average angles between tokens approach or- thogonality, aligning with our goal to prevent entropy collapse. Full results and comparisons with parameter-sharing Transformer are in Appendix F.
Table 4: Alternative design instantiations key
Table 5: ImageNet-100 accuracy (%) of HYPER-SET under different matrix rank (r) in LoRA. Depth-wise LoRA introduces rep- resentation flexibility at each iteration.
components measured by top-1 accuracy (%).
Components
Alternative Designs
CIFAR-10
CIFAR-100
Bi So--maa Attention (Default)
90.11
63.41
EXTTN
Sigmoid Attention Linear Attention
$4.38
56.97
Params (M)
Accuracy
ReL. FF (Defaltt)
90.11
6341
Eye
Softmax FF
$8.20
62.44
HYPER-SET (Ours)
1.93
70.16
Goted
+ depth-wise LoRA (r
70.36
Learned Step Size (Default)
90.11
63.11
depthh-ise LoRA (r
2.13
70.40
Step Size
0.5
25.81
$7.92
depth-wise LoRA (r
70.56
7.1
$1.45
$8.29
depth wise LoRA (r 32)
2.72
72.20
5.5
ALTERNATIVE DESIGNS AND SCALABILITY
key strength of our formulation Eq. lies in its generality- it supports a broad spectrum of model variants through alternative energy functions. For example, replacing the attention energy with kernel-based function yields novel attention mechanisms, including linear attention. Similarly, gat- ing in feedforward layers naturally arises by generalizing the feedforward energy. Table 4 shows the performance of these variants and results with different fixed step sizes, with details in Appendix G.1
To improve scalability with compact parameterizations, spired by Bae et al. (2025), where learnable low-rank adapters are added each forward iteration to modulate shared weights. This depth-wise adaptation, shown in Table 5, enhances performance without significantly increasing parameter count. Setups and additional scaling results on image and text modalities included in Appendix G.2.
introduce lightweight extension in-
CONCLUSION
We present SIPER-SET. Transformer architecture designed via iterative optimization of hyper- spherical energy functions, bridging energy-based learning and practical model design. By formu- lating dual energy the hypersphere under a general principle derived from maximum likelihood, HYPER-SET pursues distributional uniformity in the low-dimensional subspaces while promoting directional alignment with bases in the original high-dimensional space, constructing core Trans- former components with intrinsic interpretability. Empirically, HYPER-SET" matches or surpasses vanilla Transformers across diverse tasks with fewer parameters. Beyond a single architecture, our framework enables flexible design choices and scalable variants. This work contributes towards prin- cipled, more describable, and economical Transformer designs that are both theoretically motivated and practically effective. We discuss limitations and future directions in Appendix H.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work supported by the Research Grants Council (RGC) of the Hong Kong SAR under the General Research Fund (17203023), the Collaborative Research Fund (C5052-23G), and the NSFC/RGC Collaborative Research Scheme (CRS HKU703/24). The research work described in this paper was conducted in the JC STEM Lab of Multimedia and Machine Learning funded by The Hong Kong Jockey Club Charities Trust. Difan Zou acknowledges the support from NSFC 62306252, Hong Kong ECS award 27309624, Guangdong NSF 2024A1515012444, and the fund from SLAI.
REFERENCES
Juan A Acebrón, Luis I. Bonilla, Conrad Pérez Vicente, Félix Ritort, and Renato Spigler. The kuramoto model: simple paradigm for synchronization phenomena. Reviews of modern physics, 77(1):137-185, 2005.
David H Ackley, Geoffrey E Hinton, and Terrence J Sejnowski. A learning algorithm for boltzmann machines. Cognitive science, 9(1):147-169, 1985.
Brandon Amos and Zico Kolter. Optnet: Differentiable optimization as a layer in neural networks. In International conference on machine learning, pp. 136-145. PMLR, 2017. 2
Brandon Amos, Ivan Jimenez, Jacob Sacks, Byron Boots, and Zico Kolter. Differentiable mpe for end-to-end planning and control. Advances in neural information processing systems, 31, 2018.
Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- Bann-Baptiste Alayrae, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312. 1, 2023. 1
Thomas Bachlechner. Bodhisattwa Prasad Majumder, Henry Mao, Gary Cottrell, and Julian McAuley. Rezero is all you need: Fast convergence at large depth. In Uncertainty in Artificial Intelligence, pp. 1362-1361. PMLR, 2021. 6
Sangmin Bae, Adam Fisch, Hrayr Harutyunyan, Ziwei Ji, Seungyeon Kim, and Tal Schuster. Re- laxed recursive transformers: Effective parameter sharing with layer-wise loRA. In The Thirteenth International Conference on Learning Representations, 2025. URL https: oppeneevee...e. net pet/forum: wid.mmidddwwwwww WwWkkW.t. 3, 10,
Shaojie Bai, Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances in neural information processing systems, 32, 2019.
Andrea Banino, Jan Balaguer, and Charles Blundell. Pondernet: Learning to ponder. In 8th ICML Workshop on Automated Machine Learning (AutoML), 2021. URL httpt:t///// /ttp://///e/eeeeeeee/mme net/ normm 2ammmmddd-lllam 1d=1EuxRTe OWN. 7
Arpit Bansal, Avi Schwarzschild, Eitan Borgnia, Zeyad Emam, Furong Huang, Micah Goldblum, and Tom Goldstein. End-to-end algorithm synthesis with recurrent networks: Extrapolation with- out overthinking. Advances in Neural Information Processing Systems, 35:20232-20242, 2022. 3.7
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEit: BERT pre-training of image trans- formers. In International Conference on Learning Representations, 2022. URL https: //openzeview. net wwwenreview..nnt//oomaaaaaa hett//oorm/d=oo-/222222
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Con- erly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Haffield-Dodds, Alex Tamkin, Karina Nguyen, Brayden MeLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. https//transformer- htmml

---

Published as a conference paper at ICLR 2026
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06811, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020. 1
Huiwen Chang. Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11315-11325 2022. 24, 25
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34: 15084-10977 2021. 1
Tianlong Chen, Zhenyu Zhang, Yu Cheng, Ahmed Awadallah, and Zhangyang Wang. The principle of diversity: Training stronger vision transformers calls for reducing all levels of redundancy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 12020-12030 2022. 5
Róbert Csordás, Kazuki Irie, and Juergen Schmidhuber. The devil is in the detail: Simple tricks improve systematic generalization of transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp.
2021. 3
Róbert Csordás, Kazuki Irie, Jürgen Schmidhuber, Christopher Potts, and Christopher D Manning. MoEUT: Mixture-of-experts universal transformers. In The Thirly-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https:////// openreview net. forum?
1dd=7Vvek7BBj. 33
Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. Analyzing transformers embedding space. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pp. 16124-16 16770, Toronto, Canada, July 2023. Association for Computational Linguis- tics. doi: 10.18653/v 001/633//1/2223.. a2l223.cll-oon.882 acl-long. 8
URL https: / //a/aaaann weeennnnhooogyy org/2023.
Tim R. Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub M. Tomezak. Hyperspheri- cal variational auto-encoders. 34th Conference on Uncertainty in Artificial Intelligence (UAI-18), 2018. 20
Anna Dawid and Yann LeCun. Introduction to latent variable energy-based models: path toward autonomous machine intelligence. Journal of Statistical Mechanics: Theory and Experiment, 2024(10):00011 2024.7 2
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Papers), pp. 4171-4186 Minneapolis, Minnesota, June 2019. Association for Com- putational Linguistics. doi: 10. 1653/v1//19911444 1423. URL https: : //c.//acllee/ antho ogy.ogy...o..ooo.......... org// N1 9- 1233 1,3
Yihe Dong, Jean- Man-BBatiste Cordonnier, and Andreas Loukas. Attention is not all you need: pure attention loses rank doubly exponentially with depth. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceed- ings of Machine Learning Research, pp. 2793-2803. PMLR, 18 24 Jul 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recogni- tion at scale. In International Conference on Learning Representations, 2021. URL https: / /ppeneevieww net. wtt.//oormm////mm cbF
12

---

Published as a conference paper at ICLR 2026
Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. Advances in Neural Information Processing Systems, 32, 2019.
Yilun Du, Shuang Li, Joshua Tenenbaum, and Igor Mordatch. Learning iterative reasoning through energy minimization. In International Conference on Machine Learning, pp. 5570-5582. PMLR, 2022. 2, 7
Yilun Du, Jiayuan Mao, and Joshua B. Tenenbaum. Learning iterative reasoning through energy diffusion. In International Conference on Machine Learning (ICML., 2024. 2
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.2 / 1783, 2024.
Nelson Elhage. Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1(1): 2021. 1
Jonas Geiping, Sean Michael McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time com- pute with latent reasoning: A recurrent depth approach. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https: http///wnneeee.eeeeeee net/ forum? Id=$3GhJooMil 3, 33
Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet. The emergence of clus- ters in self-attention dynamics. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL /ttp::////ppeeevieewe net / forum? id=aMj aEkkXJx. 5
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer ford-forward layers are key-value memories. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 5484-5495, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10111861112222 wttp:/2...e.lpp.... httpps:/accaatthologg org/ 2021 wwnlp-ma in....ii...... 4
URL
Angeliki Giannou, Shashank Rajput, Jy-yong Sohn, Kangwook Lee, Jason Lee, and Dimitris Papailiopoulos. Looped transformers as programmable computers. In International Conference on Machine Learning, pp. 11442 PMLR, 2023. 3
Karol Gregor and Yann LeCun. Learning fast approximations of sparse coding. In Proceedings of the 27th international conference on international conference on machine learning. pp. 399-406, 2010. 3
Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. Advances in neural information processing systems, 32, 2019. 2
Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Dan Roberts. The unreasonable ineffectiveness of the deeper layers. In The Thirteenth International Confer- ence on Learning Representations, 2025. URL https:/ /otpr//veeereeeee.//mmo net/ forum? id= ngmEcEer8a.
Xiaojun Guo, Yifei Wang, Tianqi Du, and Yisen Wang. Contranorm: A contrastive learning perspec- tive on oversmoothing and beyond. In The Eleventh International Conference on Learning Rep- resentations, 2023. URL https: / ooperevieee net/ ftr.morrmmm//mm wtwpummmmiddsss.euuu/eueue
26
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Alex Henry, Prudhvi Raj Dachapally, Shubham Shantaram Pawar, and Yuxuan Chen. Query-key normalization for transformers. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Linguisties: EMNLP 2020, pp. 4246-4253 Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v URL https:://///...... /tttss://att http///aaattttttc oogg//orrroooo 2020.00222 f findi indd p.3...
http:///.0..iinnnoooeooe findings-emmlp.379
13

---

Published as a conference paper at ICLR 2026
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017. 25
Benjamin Hoover, Yuchen Liang, Bao Pham, Rameswar Panda, Hendrik Strobelt, Duen Horng Chau, Mohammed Zaki, and Dmitry Krotov. Energy transformer. Advances in Neural Infor- mation Processing Systems, 36, 2024. 7, 8, 26, 32
John Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(8):2554-2558 1982. 2
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022. URL https: wpt./eveeeeeeeeeeeeeeeeee hwtp:/iwwwwweeeeeeeeeeeeeeeeeeee wwt.eee//teoooe forum?
iddnZeVKeeFYE9. 31
Jerry Yao-Chich Hu, Thomas Lin, Zhao Song, and Han Liu. On computational limits of modern hopfield models: A fine-grained complexity analysis. In International Conference on Machine Learning. pp. 19327-19343. PMLR, 2024a. 3
Jerry Yao-Chieh Hu, Dennis Wu, and Han Liu. Provably optimal memory capacity for modern hopfield models: Transformer-compaibbe dense associative memories as spherical codes. In The eighth Annual Conference on Neural Information Processing Systems, 2024b. URL https: wptprevieeeeeeeeeeeeeeeeeeeeeeee net/ forum? 4d=
Yunzhe Hu, Difan Zou, and Dong Xu. An in-depth investigation of sparse rate reduction in transformer-like models. In The Tiirth- Annual Conference on Neural Information Pro- cessing Systems, 2024c. URL https: httpa////o/ttttt//////eeeeeeeee 26, 32
CAC7 4VuMMX. 8,
Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, 2024, URL https: /ttp//pppeeeeeeeeeeeeeeeeee net/forum?
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek, Anna Potapenko, al. Highly accurate protein structure prediction with alphafold. nature, 596(7873):583-888 2021.
Kamienny, Stéphane d' Ascoli, Guillaume Lample, and François Charton, End-to- end symbolic regression with transformers. Advances in Neural Information Processing Systems, 35:10269-10281. 2022.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URL https: / arxiv. org/ abs 2001. 08361.
George Em Karniadakis, Ioannis G Kevrekidis, Lu Lu, Paris Perdikaris. Sifan Wang, and Liu Yang. Physict-informed machine learning. Nature Reviews Physics, 3(6):422-440, 2021. 2
Md Kowsher, Nusrat Jahan Prottasha, Chun-Nam Yu, Ozlem Garibay, and Niloofar Yousefi. Does self-attention need separate weights in transformers? In Weizhu Chen, Yi Yang. Mo- hammad Kachuee, and Xue-Yong Fu (eds.), Proceedings of the 2025 Conference of the Na- tions of the Americas Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies (Volume 3: Industry Track), pp. 535-543, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89176-19400. URL https: / //aclantholoyy org/2025. naacl industry 44/.21 27
Leo Kozachkov, Ksenia V Kastanenka, and Dmitry Krotov. Building transformers from neurons and astrocytes. Proceedings of the National Academy of Sciences, 120(34))e22111551200, 2023. 3
Dmitry Krotov. A new frontier for hopfield networks. Nature Reviews Physics, 5(7):366-367) 2023.
22
14

---

Published as a conference paper at ICLR 2026
Dmitry Krotov and John J Hopfield. Dense associative memory for pattern recognition. Advances in neural information processing systems, 29, 2016. 21
Dmitry Krotov and John J. Hopfield. Large associative memory problem in neurobiology and machine learning. In International Conference on Learning Representations, 2021. URL
https:/ /oppreeeeeeeeeeeeeee net http///oommmmdyyyoo00
hX. 2, 22
Vedang Lad, Jin Hwa Lee, Wes Gurnee, and Max Tegmark. Remarkable robustness of LLMs: Stages of inference? In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https:ttt..... : wtt./ppeervviiieeee net/ forum? dd= 7NpJ.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations, 2020. URL Stth:t:::1 /ttttpps///peneeveeee net fformm? id-HleA7AEtvS.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, Fujie Huang, et al. A tutorial on energy- based learning. Predicting structured data, 1()), 2006. 2
Tianhong Li, Huiwen Chang, Shlok Mishra, Han Zhang, Dina Katabi, and Dilip Krishnan. Mage: Masked generative encoder to unify representation learning and image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11522,152, 2023.
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image gener- ation without vector quantization. In The ehihty Annual Conference on Neural Information Processing Systems, 2024. URL https: http://pppeeeiiieeeeee net forum pinorummiii-aaee VNB gmkb..
Weiyang Liu, Rongmei Lin, Zhen Liu, Lixin Liu, Zhiding Yu, Bo Dai, and Le Song. Learning towards minimum hyperspherical energy. Advances in neural information processing systems, 31, 2018. 3,5
Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruehle, James Halverson, Marin Soljacic, Thomas Hou, and Max Tegmark. KAN: Kolmogorov- networks. The Thirteenth International Conference on Learning Representations, 2025. URL htttps://opeereeee
net/
forum? net/forum? Ld=02o7q35vz1.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL https: https/////ppeeeeeeeeeeee net wtt.r/erwweeeee//odidiiiee Bkg6RicqY7. 24
Iyya Loshchilov, Cheng- theng-Ping Hsieh, Simeng Sun, and Boris Ginsburg. nGPT: Normalized trans- former with representation learning on the hypersphere. In The Thirteenth International Confer- ence on Learning Representations, 2025. URL https: http////ppneeie...eeee/ net www.:neee/o/otateem forum?id- http:::/openzevieew
se4vjm7h4E. 3
Xin Men, Mingyu Xu, Qingyu Zhang, Qianhao Yuan, Bingning Wang. Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. ShorGPT: Layers in large language models are more redundant than you expect. In Wanxiang Che. Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Computational Linguistics: Linggaatiionllsngniiiiiisss::: ACL 2025, pp. 20192- 20204, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8- 89176-256-5. doi: 10.18653/112025. findings-acl.. 1035. URL https: attpss//aclaattholoe org/2025. wwwdiinnnnniiee acl 1035/.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359-17372, 2022.
Takeru Miyato, Sindy Löwe, Andreas Geiger, and Max Welling. Artificial kuramoto oscillatory neurons. In The Thirteenth International Conference on Learning Representations, 2025. URL
https://cpenrevice net htw.pprrrreeeeeeeeoooooeeeeeeeee
15

---

Published as a conference paper at ICLR 2026
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability, In The Eleventh International Conference on Learn- ing Representations, 2023. URL
nnt/ wwwt/eerrr/iiii Eorumoiirree....... 9 SbDPmdW.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. Transformer Circuits Thread, 2022. -nn.uriiiinn--ddnnndctiooo
OpenAl, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Moham- mad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Sermadet-Shhpirge Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brock- man, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry. Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gib- son, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray. Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo. Chris Hal- lacy, Jesse Han, Jeff Harris, Yuchen He, Mike Hcaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Ka- mali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim. Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning. Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer. Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mchta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reicchir Nakano, Ra- jeev Nayak, Arvind Neelakantan, Richard Ngo. Hyeonwoo Noh, Long Ouyang. Cullen O' Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Prochl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross. Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry. Heather Schmidt, David Schnurr, John Schulman, Daniel Sel- sam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher. Felipe Petroski Such, Natalie Summers. Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian. Elizabeth Tseng, Pre- ston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vi- jayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang. Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Work- man, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024. URL https: //tpii///iivv/v// www.//os/////.oooooooooooooo 23303 08774.
16

---

Published as a conference paper at ICLR 2026
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Mare Szafraniec. Vasil Khali- dov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaacldin El-Nouby, Mido Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po- Yao Huang, Shang- Shang-Wen Li, Ishan Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features with- out supervision. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL 6zFt. Featured Certification.
Misra,
pettps:/cpperee
net/ forum?id-aat
Rasmus Palm, Ulrich Paquet, and Ole Winther. Recurrent relational networks. Advances in neural information processing systems, 31, 2018. 7, 10, 29
Vardan Papyan, Yaniv Romano, and Michael Elad. Convolutional neural networks analyzed via convolutional sparse coding. Journal of Machine Learning Research, 18(83):1-522 2017.
Vardan Papyan, Yaniv Romano, Jeremias Sulam, and Michael Elad. Theoretical foundations of deep learning via sparse representations: A multilayer sparse model and its connection to convolutional neural networks. IEEE Signal Processing Magazine, 35(4):72-89, 2018.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195 4195-4205, 2023. 1,
Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Lukas Gru- ber, Markus Holzleitner, Thomas Adler, David Kreil, Michael Kopp, Günter Klambauer, Jo- hannes Brandstetter, and Sepp Hochreiter. Hopfield networks is all you need. In International Conference on Learning Representations, 2021. URL https: / /pwpreeeeeee...e/////pp net//orum? 1d=tL8 icd. 2,22
Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In 2007
15th European signal processing conference, pp. 606-610. IEEE, 2007. 26
Michael E. Sander, Pierre Ablin, Mathieu Blondel, and Gabriel Peyré. Sinkformers: Transformers with doubly stochastic attention. In Gustau Camps-Vall Francisco R. Ruiz, and Isabel Valera (eds.), Proceedings of The 25th International Conference Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pp. 3515-3530, PMLR, 28-30 Mar 2022. URL https: pttppeeeieees....coooopppo
mlr wwwppppsssssse..... paressvv111. sander22ddddaaee htmll. 6
Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li, Sanjiv Kumar, and Sashank J. Reddi. Reasoning with latent thoughts: On the power of looped transformers. In The Thirteenth International Con- ference on Learning Representations, 2025. URL https: / opw.revieweew.eee//eoee net/ forum? id=din00lf2Fd.
Avi Schwarzschild, Eitan Borgnia, Arjun Gupta, Furong Huang, Uzi Vishkin, Micah Goldblum, and Tom Goldstein. Can you learn an algorithm? generalizing from easy to hard problems with recurrent networks. Advances in Neural Information Processing Systems, 34:6695-67066 2021. 3,7
Nir Shlezinger, Jay Whang, Yonina C Eldar, and Alexandros G Dimakis. Model-based deep learn- ing. Proceedings of the IEEE, 111(5):465-49) 2023.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan. and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei (eds.), Pro- ceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 2256-2265, Lille, France, 07-09 Jul 2015. PMLR. URL www.m.r.ppr... press v37 wwwssohhccstkeinnnss5s html.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Qi Sun, Mare Pickett. Aakash Kumar Nain, and Llion Jones. Transformer layers painters. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 2025.
17

---

Published as a conference paper at ICLR 2026
Shawn Tan, Yikang Shen, Zhenfang Chen, Aaron Courville, and Chuang Gan. Sparse universal transformer. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing. pp. 169-179, Singapore, December 2023. Association for Computational Linguistics. doi: 10.1 23.1855/v11/ h/t0//.emmlpmmaa......e 12. URL https: a aalaaacaaanttoooooyyoooooo .rgg Seargg/222 2023. emn emnlp-man p- i 12/.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 ieee information theory workshop (inw), pp. 1-5. IEEE, 2015. 5
Naftali Tishby, Fernando Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000. 5
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. Proceedings of the IEEF/CVE international conference on computer vision, pp. 32 32-42, 2021. 6
Ruben S van Bergen and Nikolaus Kriegeskorte. Going in circles is the way forward: the role of recurrence in visual inference. Current Opinion in Neurobiology, 16313, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio. H. Wallach, Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https:://proceese ngs neurips co pec/paper_fi weteeerffflssppppeeee /esssppppe/2011// 9334422233 fbd053c1 Malaa84aaa-appeee Pa..r.ddf.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. Advances in neural information processing systems, 33:12388-12401 2020. I
Guan Wang, Jin Li, Yuhao Sun, Xing Chen, Changling Liu, Yue Wu, Meng Lu, Sen Song, and Yasin Abbasi Yadkori. Hierarchical reasoning model. arXiv preprint arXiv:2506.211 27334 2025. 7
Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretabilityy in the wild: a circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations, 2023. URL https: / /ppenrevieww www.:enetttouuiiee Eor Mpsniidd-MMVSNLLLL..
Po- Po-Wei Wang. Priya Donti, Bryan Wilder, and Zico Kolter. Satnet: Bridging deep learning and log- ical reasoning using a differentiable satisfiability solver. In International Conference on Machine
Learning. pp. 6545-6554 PMLR, 2019. 2, 24
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International conference on machine learning, pp. 9929-99399, PMLR, 2020. 4
Zhou Wang, Eero Simoncelli, and Alan C Bovik. Multiscale structural similarity for image quality assessment. In The SSeith- Asilomar Conference on Signals, Systems Computers, 2003, volume 2, pp. 1 1398-1402. Ieee, 2003. 25
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE transactions image processing, 13(4):600- 612, 2004. 25
Dennis Wu, Jerry Yao-Chieh Hu, Teng- Yun Hsiao, and Han Liu. Uniform memory retrieval with larger capacity for modern hopfield models. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceed- ings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp- 53471-53514, PMIR, 21-27 Jul 2024a. URL https: ippoeeedings... mlr wwwpss/eeve.. www41. html.
Xinyi Wu, Amir Ajoriou, Zihui Wu, and Ali Jadbabaie. Demystifying oversmoothing in attention- based graph neural networks. Advances in Neural Information Processing Systems, 36, 2024b.
18

---

Published as a conference paper at ICLR 2026
Jiacheng Xu and Greg Durrett. Spherical latent spaces for stable variational autoencoders. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ichi Tsujii (eds.), Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 4503-4513, Brussels, Belgium, Octoher-Novemher 2018. Association for Computational Linguistics. doi: 10.18653/ 11/008-1480. URL httpt:tttp................. ogg org /D1 B --1080/1.00.0 20
Yongyi Yang, Tang Liu, Yangkun Wang. Jinjing Zhou, Quan Gan, Zhewei Wei, Zheng Zhang, Zengfeng Huang, and David Wipf. Graph neural networks inspired by classical iterative algo- rithms. In International Conference on Machine Learning, pp. 11773-11783, PMIR, 2021.
Zhun Yang, Adam Ishay, and Joohyung Lee. Learning to solve constraint satisfaction problems with recurrent transformer. In The Eleventh International Conference Learning Representationss 2023. URL https::/opeereveee wtwp://wwwe//ooutttemtee
Yaodong Yu, Sam Buchanan, Druv Pai, Tianzhe Chu, Ziyang Wu, Shengbang Tong, Benjamin Ha- effele, and Yi Ma. White-box transformers via sparse rate reduction. Advances in Neural Infor- mation Processing Systems, 36:9422-9444 -457,, 2023. 3, 4, 5, 7, 8, 9, 26, 31, 32, 33
Alan L Yuille and Anand Rangarajan. The concave-conver procedure. Neural computation, 15(4):
915-936, 2003.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Infor- mation Processing Systems, 32, 2019. 2
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiienees of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018. 25
19

---

Published as a conference paper at ICLR 2026
A THEORETICAL JUSTIFICATION OF MOTIVATION
We provide here a theoretical foundation for the objective in Eq. 1, showing how it arises naturally from a maximum likelihood estimation (MLE) framework under mild assumptions.
Let æ E Idd be random vector (considered as a token in the context of Transformer) in a high- J== dimensional representation space with a probability distribution p(æ). Let { be a set of random vectors in low- -iwddmeniiooall latent spaces IR (p < d) with distinct support, following prior joint probability distribution p(='..... ........... ="). We formulate information processing in the forward pass of neural networks as maximum likelihood estimation:
"0..,a,,"....,,,".... 0,, [)]
(13)
where 0 and á are parameters of the high- and low-dimensional encodings, respectively.
To make this optimization more tractable, we make the following basic and practical assumptions:
Assumption 1. The random vectors 21 ,...,,""" = are independent and follow the identical distribu- tion p(z) in distinct latent spaces, i.e., p(z) p(.,....,,? " = 11 pp(z)) and p(z²) =... p(z")= p(z).
Assumption 2. The prior distribution p(z) is a uniform distribution with support a hypersphere Sp-1
Assumption 3. The random vectors (z'...., ") ~ po.(2).....2" ......... x) from the posterior distri- bution are conditionally independent, i.e., po (=..)........ |x) - 11 (= x)).
Assumption 2 of hyperspherical uniform distribution can be perceived function as regulariza- tion on the latent representations to preserve maximum entropy and avoid representational collapse, which has been adopted to enhance auto-encoding (Xu & Durrett, 2018; Davidson al., 2018). Under the above basic and practical assumptions, the MLE objective can be reformulated as:
max ((x .e...... "...."))pppa "op(s) [logp(z, z²
()
(()) (i....... ()pp()) [logPo ,..., (z....,,, x)) + (E ))))()) [log po (xx]]
1
E Eghx(()) (z |x)] + log pa(x))
H
log (x xx)
H E Bgh)p(z) [logp(=*)] + log (x
E (x))
p(2²)
h=1
h=1
H
log p=(2"1 [x)]
H
E (x))()))
E Expp(x) [log((z) + (og
plz)
h=1
h=1
H
DKL (p(2))))lpp x)) - H x H(p(z)) + log pox),
(14)
where (-Il- denotes Kullback-Leibler (KL) divergence and H((-) means differential entropy. As the second term on entropy in Eq. 14 does not depend on variable æ, this objective ultimately reduces to Eq. which we restate here for completeness:
min Σ DILL (po()) x))
log(po (x))
uniformity
aligument
The first term encourages the posterior (1 x)) defined on the vector zh E R2 in latent space, which can be implemented by a transformation parameterized by Φ, to approximate uniform dis- tribution on a hypersphere. To see why the second term implies alignment, suppose the distribution pe(x) is parameterized as mixture of M von Mises-Fisher (vMF) distributions' with equal mixing
hhtps://ce.wwkipppeeee http::///ww..iiipiidiaaoo///nnon Mises-Fisher distribution
20

---

Published as a conference paper at ICLR 2026
coefficients:
M Σ m0
M Σ T
- -log(po(2))) log
f(x; Mm, Fim))
log
Ca(km) exp(Nmm
M
M
(15) where Mm E gd-1 denotes mean direction on (d - dimmeniional unit sphere and fim 2 0 is the concentration parameter while Ca(Ke)) the normalization constant; therefore, finding the vector a that minimizes this negative log-prohability Eq. 15 equals finding maximal inner product jr æ. thus aiming for directional alignment. practice, the mean direction μmm is learned by backpropagation and consequently contains certain statistical properties from the data.
summary, the objective Eq. suggests that for a representation vector a Rd the forward dynamics can be characterized by two complementary properties:
Mode Seeking: Achieving semantic alignment with directional vectors encapsulating specific information derived from data in the high-dimensional space.
Mass Covering: Maximally preserving the entropy embedded via regularizing distributional uni- formity in the low-dimensional space.
These principles underpin our design of token dynamics, and we propose to use energy functions to quantify these two properties as instantiations that can induce various Transformer-based models.
B PRELIMINARIES
B. HOPFIELD NETWORKS
Given network with N neurons ........ ... ENl that take binary values, the temporal evolution dynamics of these neurons are determined by scalar-value energy function:
1
Wazza = Wa, I, E (+1,-1)
where wi represents the strength of connectivity between node 2; and Zj. and the connectivity is assumed to be symmetric, i.e., Waj = Wjs- can further rewrite W == E E.E. as a set of patterns to be stored. The update rule of each node to retrieve the most relevant pattern follows the Hebbian learning rule used in neuroscience:
P E.E.
T++1 sign( Wx,) = sign Σ
This update rule tends to minimize the energy function with retrieved patterns as its attractor. It is an embodiment of the idea of "Neurons that fire together wire together.": If two neurons connect (Wij > 0), then they should have the same state (+1 for active and - for dead). The number of patterns the network can store and retrieve is ONN).
B.2 MODERN CONTINUOUS HOPFIELD NETWORKS
To overcome the limitation of linear storage capacity, modern Hopfield networks, also known as Dense Associative Memory (Krotov & Hopfield, 2016), introduce nonlinearity in the energy and the update of neurons' states and make them suitable for continuous variables:
P E' == 2 Σ f (E. x). Lt+1 = tanh Σ
E,f' (ET m1)
where tanh(-) ) is to ensure the neurons' states constrained to the interval [-1, 1] so that the energy bounded from below. Depending on the form of f. the network could have power or exponential storage capacity. If set [xx x2 this reduces the traditional networks with linear capacity.
21

---

Published as a conference paper at ICLR 2026
If we further make modifications to the non-linearity in the energy function with logsumexp()) which is inspired by contrastive normalization, we can define the Modern Continuous Hopfield (MCH) energy function with a quadratic regularization term æ:
P EMCH = log Σ
1 + 2 a.
exp (5, σ))
(16)
By leveraging the concave-conver procedure (Yuille Rangarajan, 2003), the update could be writ- ten
1 Esofumax xx),
where E = E RNxP. This formulation has proven to converge to stationary points of the energy function EMCH. and linked to the key-value memory similar to the attention mechanism (Ramsauer et al., 2021). Notice that this update rule is essentially the cross-attention given query vector and can only describe the independent evolution of that vector. fails faithfully cover the parallel interactions between contextual tokens in the self-attention adopted in the GPT or BERT style Transformers.
The construction of the modern continuous Hopfield energy and update rule can also be carried out from a biologically plausible view by extending the network with hidden neurons and establishing a group of coupled differential equations. We refer the readers to Krotov & Hopfield (2021); Krotov (2023) for more details.
C DERIVATION
C.1 DERIVATION OF THE GRADIENT OF FATTN
4
VR, EXTIN
H
N
Wn WTW,T Φ3 exp (3(W,) TT (W, xz))
N
W o, exp ( x;) (W xx))
E exp (3(W, m2 )x T(W, x.,))
E exp (B(W, x:) (W₁ a;)))
h=1
exp (B(W,, xx)) T(W, ax1))
H
WA Wi
[ xN]
Σ j=1
exp (3(W,) xx) Te)TT(( (W x,,) +
exp (B(W, mkk (W, kx))
exp (3(W)) TT)TT (W xx)))
شزكلك exp (S(W, x.) (W, x;)1)
W. WT x;
exp (3(W)) σNN) T (W x2)) Σ exp (B(W,T T
(W 2,1)
8
N WnWW X softmax (B(W,) X) (WT xb)) +
Wi W x, softmax (3(W,) x) (W, x:))
column
column
W.W. X softmax (B(W) h X)T ((,, mx)) W. W X softmax (3(W X) (W, x)) [A:.]]
columm
column
H
Wa W.W. X softmax (B(W, X) (W xx.) + W, W, X softmax (B(W,, X) (W₂ x)) x)){Ea) (1mb)
column
row
H
Wr W.W. X softmax (B(W, X) (W, æw)) + Wt WT X softmax (B(W,) x)),xX)TT_ T(W, x)) P..M
column
22

---

Published as a conference paper at ICLR 2026
X 1 xNl x EATTN
(WWT) X softmax (B(W) x)((W' [ + WWT X softmax (B(WTX) (
column-wise
row-wise
C.2 DERIVATION OF THE GRADIENT OF EpF
V an
EFF
ReI.U(d,, Md 0) dm
Σ
ReLU ( ( dm)
ReLU x&)
[d
dm}
ReLU(d))ax))
D ReLUU(D xkk)
X [ [N]] - Vx D ReLU(DD X)
D DETAILED EXPERIMENTAL SETUPS AND MODEL. CONFIGURATIONS
D.I NETWORK TO LEARN ADAPTIVE STEP SIZES
We propose to learn adaptive step sizes Cg Y E Rd given the current iteration t E [1, L]. where L the iteration number of the layer with unique parameters, and the initial token zx) Rd using the network shown in Figure 7 and configurations in Table 6.
Linear
GELU
Linear
GELU
Linear
a; |d ... ata
t
Time Embedding
Yt IVt Ytal
x(0)
Figure 7: Illustration of time embedding conditioned the input to learn adaptive step size.
Table 6: Model configurations of network lear adaptive step sizes.
Layer
Configuration
Time embedding
512
Linear
512 x d
GELU
Linear
x d
GELU
Linear
x 2d
23

---

Published as a conference paper at ICLR 2026
D.2 SOLVING SUDOKU
Solving a Sudoku puzzle requires filling 9 x 9 board, with some digits (1-9) known and unknown entries marked as The unknown entries must be filled with digits perfectly such that the board satisfies certain rule, which can be seen
logical reasoning task (Wang et al., 2019). We tackle this puzzle by predicting the digits to fill in, conditioned on the given digits. It can be viewed simplified masked modeling on synthetic data. Cross-entropy loss is computed exclusively on unknown entries. We train all models with 200 epochs, 16 batch size, AdamW (Loshchilov & Hutter, 2019) with 0.1 weight decay, and learning rate from 1 x 10 with cosine decay. The hidden dimension set to 768. Tables 7 and 8 show the training recipe and model configurations for solving Sudoku. We train the model with 24 iterations and evaluate beyond these iterations.
Table 7: Training recipe for solving Sudoku.
Table 8: Model configurations for solving Su- doku.
Hyperparameter
Value
Configuration
Value
Epochs
200
Vocabulary size Layer
10
Batch size
# GPU
x NVIDIA 3090
1
# Training samples
9,000
Iterations L
24
# Evaluating samples
1,000
Hidden dimension
768
Optimizer
AdamW
Feedforward ratio M
4d
$1.32
0.9, 0.95
Number of heads
12
Positional encoding
Learnable
Weight decay
0.1
Learning rate (Ir)
10-4
Time embedding condition
Xo
Time embedding frequency
512
LR decay
Cosine
Gradient clipping
1.0
Total parameters
5.20 M
D.3
IMAGE CLASSIFICATION
Tables and 10 present the training recipe and model configurations for image classification on CIFAR-10/100, while Tables 11 and 12 show those on ImageNet-100// with a learnable class token [CLS). In practice, we use absolute sinusoidal positional encoding and adopt conditioning X, for performance reasons. Table 13 lists the configurations of different sizes, and it applies to other tasks as well.
All models are trained
Table 9: Training recipe for image classification on CIFAR-10/100.
Table 10: Model configurations (Small Scaled- up) for image classification on CIFAR-10/100.
Hypersphere
Value
Configuration
Value
Epochs
200
Patch size
Batch size
128
Layer
# GPU
x NVIDIA 3090
Iterations L
12
Training samples
50,000
Hidden dimension d
512
# Evaluating samples
10,000
Feedforward ratio M d
Optimizer
Adam
Number of heads H Positional encoding
81,32
0.9, 0.999
Sinusoidal
Weight decay
X 10
Time embedding condition
X,
Max learning rate (Ir)
10
Time embedding frequency
512
Min learning rate (Ir)
1 x 10
LR decay
Cosine
Warmup epochs
Input size
D.4 MASKED IMAGE MODELING
We follow Chang et al. (2022) using VQ- VAE to tokenize the images to 16 x 16 latent code with the codebook size of 1024 after resizing the input to 256 x 256. The masking ratio is randomly
24

---

Published as a conference paper at ICLR 2026
Table 11: Training recipe for image classification on ImageNet-100// 100/1K.
Table 12: Model configurations (Small Scaled- up) for image classification on ImageNet- 100/1K.
Hyperparameter
Value
Configurations
Value
Epochs
200
Batch size
256
Patch size
# GPU
1 X NVIDIA A100
Layer Iterations
1
# Training samples
226,69/1,2211,10
# Evaluating samples
5,000/50,000
Hidden dimension d Feedforward ratio M Number of heads H
512
Optimizer
Adam
Br 82
0.9, 0.999
8
Weight decay
5 x 10
Positional encoding
Sinusoidal
Max learning rate (Ir)
1 10-3
Time embedding condition
X;
Min learning rate (Ir)
x 10-3
Time embedding frequency
512
LR decay
Cosine
Warmup epochs
5
Input size
224
Table 13: Model configurations of different sizes.
Configurations
Small
Small Scale-up Base
Hidden dimension d
384 512
768
Number of heads 6
12
chosen between [0, 0.4], and the masked region is replaced by a learnable token. Training loss is computed only for the masked tokens. We also follow the iterative decoding process in Chang al. (2022) with temperature time embedding and set the embedding frequency equal the hidden dimension save parameters, and we find out that this implementation works better. Tables 14 and 15 show the detailed training recipe and configurations.
1 and decoding step T 24. We also remove the MLP following the
We evaluate the quality of the reconstructed images of masking out 40% of the images with Base configurations. We report Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM) (Wang et al., 2004), Multi-Scale SSIM (MS-SSIM) (Wang et al., 2003), Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al., 2018) and Fréchet Inception Distance (FID) (Heusel et al., 2017) the validation set (5k).
Table 14: Training recipe for masked image modeling.
Table 15: Model configurations for masked im- age modeling.
Hyperparameter
Value
Configuration
Value
Epochs
300
Vocabulary size Layer
1,025
Batch size
256 (64 per GPU x 4) NVIDIA A100
# GPU
Iterations
12
# Training samples
126,689
Hidden dimension d Feedforward ratio M
768
# Evaluating samples
5,000
d
Optimizer
Adam W
Number of heads H
12
81,32
0.9, 0.96
Positional encoding
Sinusoidal
Weight decay
0.1
Time embedding condition
X.
Learning rate (Ir)
10-4
Time embedding frequency
768
LR decay
None
Total parameters
3.94 M
Gradient clipping
3.0
Input size
256
25

---

Published as a conference paper at ICLR 2026
D.5 DEFINITION OF EFFECTIVE RANK AND AVERAGE ANGLE
We provide the formal definition of effective rank Eq. 17 and average angle Eq. 18 below. The effective rank is continuous proxy of the full rank and, similar to the average angle, reflects the extent to which a set of vectors distributes uniformly.
Definition (Effective Rank). For matrix X RdxN, let Σ -- Frl be its singular values where r is its full rank and denote ps = 0sl E;=10; the discrete probability. The effective rank (Roy & Vetterli, 2007; Guo et al., 2023) is defined as the exponential of the entropy
exp( Σ Ps log
(17)
Definition 2 (Average Angle), Given a set of vectors X 11... [NN] angle of these vectors is
RdN,, the average
2
x 2j
(18)
N(N
D.6 COMPARISONS OF PARAMETER EFFICIENCY AND COMPUTATIONAL COST
Table 16: Comparisons of parameters and computational cost different architectures.
Models
# Params (M)
Memory (MB)
GFLOPs
Runtime (ms)
Transformer
2.38
528
12.27
4.99
CRATE-T (Hu et al., 2024c)
0.91
528
5.90
CRATE (Yu ul., 2023)
1.06
528
5.90
4.86
1.8610.19
Energy Transformer (Hoover et al., 2024)
1.50
670
6.19
21.95=0.35
HYPER-SET (Ours)
1.55
528
6.81
8.03
Table 17: Impact of step size and normalization strategies on computational cost and runtime. The configuration of our HYPER-SET is highlighted in gray which is designed to have RMSNorm placed after the subspace projection. Importantly, this configuration is not independent disentan- gled but rather derived based on the proposed principled objective.
Step Size Strategy
Pre-Norm
LayerNorm
# Params (M)
GFLOPs
Runtime (ms)
1.55
6.81
8.03
1.55
6.81
8.05
Learnable
1.55
6.84
7.31
1.55
6.82
7.21
0.91
4.98
5.70
0.91
4.98
5.53
Fixed
0.91
5.01
4.86
0.91
5.00
4.74
To show the differences in parameters and computational cost incurred only by distinct architectures, we provide comparisons under the same width d 384. Empirically, we compare FLOPs and runtime in one forward pass using cal lopss, measured on single NVIDIA A100 GPU, of our model and other baselines with 3 x 224 x 224 random input. Table 16 reports the results with mean and standard deviations averaged over 1,000 runs for robust measurements. Our model has fewer FLOPs as it has inherent structures like weight sharing due to mathematical design.
To better understand the runtime performance of our model, we break down the contribution of key components in Table 17. The results reveal that the primary bottleneck in execution time stems largely from the facts that:
We use an additional network to learn step sizes instead of keeping them fixed (~ 2.33 ms);
The current default implementation of torch. nn. RMSNorm (), which we use to meet the hy- perspherical constraints, is slower than torch. LayerNorm (~ 0.72 ms).
26

---

Published as a conference paper at ICLR 2026
Currently, we trade some efficiency allow for strong performance while keeping the principled de- sign as transparent as possible with the additional modulation network for learned step sizes, which is adopted only for performance reasons rather than being theoretically necessary component in our framework. In fact, our ablations that remove this extra network and fix the step sizes give sim- ilar runtime (5.70 ms in Table 17), but the performance decreases greatly (81.45 on CIFAR-10 and 58.29 on CIFAR-IOL 1100 in Table 4), So, we believe there is still room for improvement in optimizing the step sizes schedule and further enhancing this minimalist implementation future work.
E ADDITIONAL RESULTS OF MASKED IMAGE MODELING
Table 18 summarizes the results of masked image modeling with different masking ratios. When scaled to larger iterations and a wider feedforward module, our model achieves comparable results to Transformer but still slightly lags behind. This suggests the scalability of our model to large configurations may be a bottleneck for its development and deployment. More visual comparisons are provided Figure 8.
Table 18: Comparisons of masked image modeling performance of varied masking ratios.
Masking Ratio
Model
Iterations/ / FV Ratio Af
Parans (M)
PSNR (1)
SSIM (1)
MS-SSIM (T)
LPIPS (1)
RID (L)
Transfommer
12 12/d
8885
17,693
0,466
1,709
0.236
22.428
10%
HYPER-SET (Ours)
3,94
17.553
0.462
0.701
24.665
HYPER-SET (Ours)
24/88 Sd
8.07
17.673
0.465
0.708
0.236
22.517
Transformec
12/ 4d
8.85
17,185
0.451
0.678
0.261
27,320
20%
HYPER-SET (Ours)
12
3,94
16,988
0.444
01662
0.275
33.637
HYDER-EET (Ours)
/dd
17.170
0.450
0.676
Transformn
12/4d
8.85
16.616
0.435
0.642
0,291
35,095
30%
HYPER-SET (Ouns)
12/d
3.91
16.365
0.427
0.621
0.314
45.642
HYPER-SET (Ouns)
24/8d
8.07
16.590
0,434
0.638
0.254
35,128
F RANK AND AVERAGE ANGLE OF EACH HEAD
F.1 SUDOKU DATASET
Figures 9a and 9b capture the evolution of the effective rank and average angle of all heads. Most of them follow the separation dynamics on the hypersphere where tokens tend to be netroorthogonal, corroborating our design goal of attention energy.
F.2 CIFAR- DATASET
The full results CIFAR-10 also possess similar trends to those the Sudoku dataset, as shown in Figures 10a and 10b.
F.3 COMPARISONS WITH TRANSFORMER WITH SHARED QUERY, KEY, AND VALUE MATRIX
A notable connection between HYPER-SV SET and vanilla Transformer lies in the shared query (Q) key (K), and value (V) projection matrix, which has been studied recently (Kowsher et al., 2025). To verify whether HYPER-SET captures essential Transformer behaviors, we adapt vanilla Transformer to have shared QKV projections and measure its effective rank and average angle among projected tokens. Furthermore, we also include comparisons with HYPER-SEE SET that adopts fixed step sizes set as 0.1 to evaluate the necessity of learned ones.
As shown in Figures la and 1b, both HYPER-SET and its fixed-step variant exhibit increasing token separation across subspaces, confirming the emergence of distributional uniformity and the benefit of learned step sizes. This dynamics is also mirrored in shared QKV Transformer, which cross-validates our insights on distributional uniformity in subspaces, suggesting the promise of this parameter-ssaring design. In contrast, modifying the shared Transformer to reverse the update direction of ciinntoon-siilee highlighting a breakdown in uniformity. This contrast emphasizes that HYPER-ST SET not a heuris- tic tweak of vanilla Transformer but principled architecture.
the design in Eq. 9 leads to decline in both rank and angle,
27

---

Published as a conference paper at ICLR 2026
Orgina
Maded
Tazaeezer 6.5
(labaraa) Boy
Origina
Tundeeeer BAG
(l//bwag) Koy
Traateenrr bds
Figure 8: Visual comparisons of masked image modeling on ImageNet 256x 256. Our model, when increased to Transformer scale with additional compute, can achieve similar reconstruction quality with 40% masking ratio.
28

---

Published as a conference paper at ICLR 2026
Boeeee (Fdd Banh
Fernal baccies
Paraad beraicee Bead
Berned Bentha Heat
Ferote hrreic Bkad
Parcari Ioraticss
Reved Bertieas
Heal
Bad
Eltettnn Pall Utaale
Pall Daal Daak
hill Fuk
7d bia
Renol 1izeao
Porvani Domdicts
Rexed Heniioa
Renal Tameaso
Forvarl Domticts
Rened Bertiaa
(a) Effective and full rank.
Teat
Bead
Bosd
Tied
Sana Augle
Averag Aegle
10
Foad
Heal
Hoal
Lilgine drang
Average angle.
Figure 9: Geometric analysis on Sudoku. The effective rank (a) and average angle (b) of tokens projected each subspace on the test (Palm al., 2018).
Real
Boal
Bel
Hoal
Stetie Back
20 the TUAk
Bunkk
Forood
(a) Effective and full rank.
Feraad Timmation
Reveri DerelKes
Frevard Timsiices
Fermadd Boratoon
Rewonl Demeaes
(b) Average angle.
Figure 10: Geometric analysis on CIFAR-10. The effective rank (a) and average angle (b) of tokens projected to each subspace on the test set (Palm et al., 2018).
29

---

Published as a conference paper at ICLR 2026
(Poed Sop Stoj
The mocdooo Shand goo)
Turdfower (Saad DeV Negali Anoedao))
Hod:
Had
Beals
Hard
Read
Burk
Terned Zerskee
Nevunl Beakeo
Firsed Zereikes
Flonard Deratice
Doroad Baraiion
(a) Effective rank.
(Podd Sty Sua)
Trandersss ISandd QUV
Trankers iSkaed QJT Suparis Arneike)
Hal
Red
Heal
(b) Average Angle.
Figure 11: Impact of weight sharing and update rules. This layer-wise evolution of effective rank (Top) and average angle (Bottom) the CIFAR-10 validation set verifies that our architecture and the variant using fixed step sizes share a similar functional role with Transformer with shared Query/Key/Value (QKV) matrices.
G ADDITIONAL RESULTS ON ALTERNATIVE DESIGNS AND SCALABILITY
G.1 ALTERNATIVE DESIGNS ENERGY FUNCTIONS
Our proposed energy functions EATTN and ÉFF provide an avenue to quantify the objective on uni- formity and alignment in an amenable way for opúmization. To manifest the significance of our conceptualization in designing variety of Transformer- based models, we extend the energy func- tions to more general forms and provide alternative instantiations of them that can induce novel structures. Specifically, we generalize energy functions in Eq. and Eq. 5 to the following forms:
EXTIN = Σ E
ΣΣ'
N
N
K(W, 2. WT. )
(19)
h=1
h=1
1m1
M
EFF
g
h(d x.,)
(20)
m=11
where K R R + R is a kernel function and fig. R + IR are non-decreasing scalar functions. For clarity, omit the hyperspherical constraint but its correspondence to RMSNorm still holds.
By choosing different variations on these functions, we arrive alternative energy functions and their consequent attention and feedforward operators, of which we summarize the specifications in the following tables. Remarkably, Table 19, we can derive an attention with linear complexity O(N) by specifying the kernel function with the inner product of an element-wise transformation ::R- + I., bridging our energy view with recent advances linear attention design. In practice, we choose it as the sigmoid function (z) 1/(1 exp(-xp -x)), but other designs can also be possible. In Table 20, we specify the outer function g in the feedforward energy as a quadratic function, there is novel summation and a Hadamard product operation emerging with the transformation 4, similar to the gating mechanism. We also specify & with the sigmoid function.
In summary, these design choices demonstrate that HYPER-SET is more than just single model, but a blueprint for constructing principled Transformer variants. Each component-selff self-attention, feedforward, and normalization- can be systematically interpreted and designed within energy minimization framework, providing pathway for principled, modular innovation in sequence model architectures.
30

---

Published as a conference paper at ICLR 2026
Table 19: Alternative designs on attention energy EATTN and the induced operators.
Operater
((0)
Kion mngl y)
Flars
Ux Rarrs
Bi-Sofimax (Deianl))
R- Vngle)
Fq
Sigmood Attestion
(ste
2 2 (pWW))) Wille)
Wow (OEM)) wX.x)
Linear
(MMa)r 6(x))*
WLelw X) (36( Xiewy o.W, X))
Astigst
Table 20: Alternative designs on feedforward energy EFF and the induced operators.
Operator
g(x)
h(x)
Eip
EiF
ReLU FF (Default)
1 ReI.U(r))
Eq.
Eq. 10
N
M
Softmax FF
log(x)
exp(x)
Σ
log
Σ exp(d
D softmax (D XX)
column-wise
Gated FF
2
(di,
D x D' X)
columa sum
G.2 PRELIMINARY RESULTS ON SCALABILITY
Setups for Depth-wise Flexible Computation. To equip our model with flexible computation while maintaining its core recurrence-bast parameter sharing, we add an independent low-rank adaptation (Hu et al., 2022) matrix W AB to every iteration of the Transformer layer sharing the same base parameters. Inspired by but unlike depth-wise adaptation of pre-trained models in Bae et al. (2025), we train the base parameters and the adaptation matrix together. Both matrix A E Rddxr, B E IRrdd with rank r are initialized with Gaussain of 0.02 standard deviation. The scaling factor before the adaptation matrix is set
Scaling Results on Images. To further demonstrate the capability of our model when scaling up its size, we perform a preliminary evaluation on image classification. Following the experimental setups in the main paper to configure the models with one layer and repeat with 12 iterations, we scale up the width of Transformer to 1152 and ours to 1536, resulting in similar parameter size. The result presented in Table 21. These explorations showcase the potential HYPER-SET in improving model capacity without significantly increasing the total parameter count.
Table 21: Top-1 accuracy for image classification when scaling up the model size. Our model surpasses other baselines while being parameter-efficient compared to vanilla Transformer.
Dataset
Model
Width
# Params (M)
Accuracy (%)
Transformer
1152
16.0
89.42
CRATE (Yu et al., 2023)
1152
4.1
88.77
CIFAR-10
Energy Transformer (Hoover et al., 2024)
1152
8.0
76.21
HYPER-SET (Ours)
1536
14.3
90.62
Transformer
1152
16.1
62.83
CRATE (Yu al., 2023)
1152
4.2
63.39
CIFAR-100
Energy Transformer (Hoover al., 2024)
1152
8.1
55,47
HYPER-SET (Ours)
1536
14.4
66.30
Scaling Results on Texts. We also provide the results for text classifications to show the potential from Hugging Face as the tokenizer with a max-
of HYPER-SEE We use
imum sequence length of 128. The number of recurrence is set (L1R6) and width is set to 384. The training recipe is the same as the image classification in Table 9 except we train for 10 epochs. The results on yelp review full dataset, a 5-way classification task, shown in Table 22.
31

---

Published as a conference paper at ICLR 2026
Table 22: Top-1 accuracy for text classification on yelp review. dataset.
Model
Width d
Params (M)
Accuracy (%)
Transformer
384
13.49
60.09
CRATE-T (Hu et al., 2024c)
384
12.02
59,43
CRATE (Yu al., 2023)
384
12.17
59.18
Energy Transformer (Hoover et al., 2024)
384
12.61
54.93
HYPER-SET (Ours)
384
12.66
60.75
H DISCUSSION, LIMITATIONS AND FUTURE DIRECTIONS
H.1
CONNECTIONS AND DIFFERENEES
ENERGY TRANSFORMER
Both our HYPER-SET and ET (Hoover et al., 2024) are grounded in the idea of interpreting Trans- former components as gradient flows that minimize an energy function. However, the two ap- proaches diverge significantly in motivations, theoretical formulation, and architectural design.
Motivation
HYPER-SET: centers dual objective of semantic alignment (mode seeking) and uniformity (mass covering) under hyperspherical constraints grounded in maximal likelihood estimation. The proposed Hopfield-style energy aims quantify it into optimizable objective and serves as a specific instantiation of this more general principle, which fundamentally differs from ET.
ET: maintains the mechanistic interpretation of associative memory and does not directly con- nect the energy formulation to any particular representational challenge. Moreover, ET adopts Hopfield energy more as starting point than as motivation-driven design.
Methodology
HYPER-SET: provides more rigorous formulation of energy minimization. Our energy func- tions are defined directly on tokens Eq. 6 under hyperspherical constraint. This formula- tion enables us to derive RMSNorm as natural outcome of energy minimization in low- dimensional subspaces Eq. 8.
ET: defines energy over pre-normalized tokens (see (q.(1) in Hoover et al. (2024)) rather than tokens per se, bypassing the constrained optimization step. As result, the role of normal- ization in ET is more heuristic than principled, following standard pre-norm practices rather than emerging from the underlying energy.
Implementations
HYPER-SET: a) applies alternating minimization that results in attention and feedforward modules sequentially, reflecting the original Transformer structure; b) adopts adaptive, learn- able step sizes conditioned the input and iteration index Eq. 12, allowing the model mod- ulate its energy descent dynamically.
ET: a) performs energy updates via auto-differrnniation that results in a parallelized fashion; b) uses fixed step sizes.
Empirical Verification
HYPER-SET: a) confirms that the designed energy decreases the forward pass in Figures and 6; b) supports generalizations beyond softmax attention (c.g.. different energy functions leading to alternative attention schemes), as illustrated Tables 19 and 20.
ET:: a) does not offer such explicit verification, leaving unclear whether its dynamics faith- fully track the energy descent objectives; b) Additionally, HYPER-SET achieves competitive performance with vanilla Transformers on tasks such as image classification and inference (e.g..
Sudoku), whereas ET does not demonstrate comparable results in these domains.
H.2 PRACTICAL IMPLICATIONS AND BROADER IMPACT
Why Study This Model: This work proposes principled approach to Transformer design by modeling representation learning as an energy minimization on the hyperspace. Unlike prior ef- forts such Energy Transformer (Hoover et al., 2024) and CRATE (Yu al., 2023), which
32

---

Published as a conference paper at ICLR 2026
either diverge from their theoretical formulations lack generality derive architectures, our model directly formulates energy functions on tokens with improved rigorousness, while sup- porting spectrum of alternative designs.
Why CRATE Is a Fair Baseline: CRATE (Yu et al., 2023) also pursues transparency and princi- pled design, which shares a similar spirit with our goal, making it an appropriate baseline. While engineering heavy ViTs may excel in benchmarks, they often involve significant redundancy. We aim to advance more compact, describable, and empirically competitive model design for next- generation architectures.
Interpretabillty: We view the forward pass of HYPER-SET as dynamical system. It features greater interpretability than vanilla Transformer in the sense that this dynamics more read- ily describable and characterized by meaningful quantity-the energy function-and grounded in well-established principles as maximum likelihood estimation. Beyond being merely conceptual, this dynamics is quantitatively verifiable, providing an interpretable and testable framework for understanding representation evolution.
General Principle Beyond Canonical Hopfield Energy: The Hopfield energy we em- ploy in the main paper serves one instantiation under the proposed general principle. Our formulation allows for broader energy-based designs-sue as kernel-based alternatives to gesubeep--enablin principled generalizations beyond standard attention mechanisms.
H.3 LIMITATIONS
While HYPER-SET offers a principled and empirically competitive formulation for Transformer design, also comes with several limitations that highlight directions for future work:
First, the choice of subspace in our conceptualization less explored. The choice of uniform prior the hypersphere in assumption could be too strong in practice. Overly enforcing uniformity may be restrictive some tasks. Moreover, the number of subspaces and its dimension p are chosen heuristically. Their relationship, if any, with the real data distribution remains unclear.
Second, the modulation network to learn step sizes introduces complexity. Although we use modulation network learn step sizes adaptively, tuning the configurations of this network still requires considerable effort. In addition, introduces more computational complexity despite that the overall architecture more parameter-efficient.
Third, our experiments on scalability are still preliminary. We confirm competitive and supe- rior performance on less than 20 million parameters and prove depth-wise LoRA scaling effec- tiveness. However, extensions to truly large-scale settings- e.g. billion-level- yet to be demonstrated.
H.4 FUTURE DIRECTIONS
We provide several promising future directions:
Autoregressive Modeling: HYPER-SET currently lacks a causal structure, limiting its use in autoregressive sequence modeling. Adapting to GPT--style models with causal masking is an important future step.
The iterative updates in HYPER-SET resemble neural ODEs, suggesting potential connections to flow matching techniques that may unify Transformer- based models with generative modeling.
Flow Matching and ODE Connections:
3. Scalability and Adaptive Computation: Our initial results with depth-wise LoRA promis- ing but preliminary. Future work could explore dynamic iteration depth, inspired by latent space reasoning (Geiping et al., 2025), sparsity (Tan al., 2023), and mixture-of-experts (Csordás et al., 2024).
33


Reviews:
[{"review_id": "6WDT3wFwI7", "reviewer": "Reviewer_PVNu", "summary": "This paper proposes a new transformer component, based on the principle of optimizing a hyperspherical energy.", "questions": "N/A", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper seeks to formulate model architecture design through the principle of energy minimization, which is an interesting and ambitious attempt.\n- The experiments span several domains, showing the effectiveness of the proposed method.\n- This paper is well presented and easy to follow.", "weaknesses": "- It is unclear if the proposed \"energy\" is actually minimized. I don't see any empirical verification or any theoretical proof regarding this.\n- Also, there is no justification for whether the energy decrease actually correlates with performance.\n- The resulting architecture appears extremely similar to a standard Transformer with RMSNorm or other normalization layers. It is unclear what the \"energy\" formulation contributes beyond rephrasing existing operations in geometric language.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3fBdXQ29ET", "reviewer": "Reviewer_jreb", "summary": "The authors introduces HYPER-SET, a principled that reframes the Transformer as an iterative energy minimizer on a hypersphere. This hyperspherical energy is designed to balance two key goals: semantic alignment with important features and distributional uniformity to prevent representational collapse. By deriving the math for this optimization, core components like symmetric attention, feedforward layers, and RMSNorm emerge naturally rather than hand-designed. The final recurrent-depth model is parameter-efficient and demonstrates competitive performance on diverse tasks, including Sudoku solving, image classification, and masked image modeling.", "questions": "Q1. The model is slower than a standard Transformer (Table 15) and underperforms on general-domain tasks (Weakness 1). Given these latency and performance gaps, how do the authors justify the practical value of its parameter efficiency?\n\nQ2. Does the training memory cost scale linearly with the number of iterations to store activations for backpropagation?\n\nQ3. The paper's most significant performance gap appears on the Sudoku task, where both CRATE and Energy Transformer completely fail on this task. What is the authors' hypothesis for why these other \"white-box\" models fail catastrophically on this reasoning task, while HYPER-SET succeeds?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The primary strength of this work is its \"white-box\" design. The authors start from a unified principle (hyperspherical energy minimization) and derive the architectural components (Bi-Softmax attention, FFN, RMSNorm) as the mathematical solution.\n\n2. Figure 5 demonstrates that the designed energy function decreases during the forward pass. Figure 6 shows that the effective rank and average angle of tokens increase, empirically confirming that the \"distributional uniformity\" objective is being met. This link between theory and empirical dynamics is therefore good.\n\n3. HYPER-SET outperforms other models in the \"white-box\" design class (CRATE and Energy Transformer).\n\n4. Table 3 provides an ablation that validates the design choices. It shows that the components derived from the theory outperform more common alternatives.", "weaknesses": "W1. The model’s good results on iterative reasoning tasks (like Sudoku) do not generalize to general-domain tasks. On standard benchmarks like ImageNet-1K and masked image modeling, the vanilla Transformer baseline remains superior.\n\nW2. The recurrent-depth design is a major practical drawback. As confirmed by the authors' runtime analysis in Table 15, the 1-layer, 12-iteration HYPER-SET model is significantly slower than a standard Transformer, despite having fewer parameters. This high latency severely limits its practicality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yvOWT3mdS7", "reviewer": "Reviewer_Dhku", "summary": "This paper presents a principled, \"top-down\" derivation of components of the Transformer architecture from an energy-based learning perspective. The authors posit that effective token representations should simultaneously achieve two goals: semantic alignment (clustering of similar concepts) and distributional uniformity (spreading representations out to avoid collapse). They formulate these two objectives as a single, combined hyperspherical energy function. The paper then demonstrates that an iterative optimization process to minimize this energy function naturally yields the core components of a Transformer block: symmetric self-attention emerges from the uniformity objective, a symmetric feedforward network from the alignment objective, and RMSNorm from the hyperspherical constraint. The resulting model, HYPER-SET, is a recurrent, parameter-shared architecture. The authors evaluate HYPER-SET on tasks like Sudoku solving, image classification, and masked image modeling, comparing its performance and parameter efficiency to standard Transformers.", "questions": "For Sudoku extrapolation (Figure 3b), the authors attribute the gains to the proposed energy minimization dynamics. However, these improvements could maybe come from implicit regularization introduced by the symmetric parameter sharing and hyperspherical constraints, instead of the iterative energy optimization itself? Did the authors conduct ablations to disentangle the effects of the energy-based formulation from those of architectural regularization?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is very well-written. The argument is presented with great clarity, smoothly taking readers from a conceptualization to mathematical derivations and finally to empirical validation.\nThe core premise of the work is very interesting. It is not the traditional heuristic-driven design of modern architectures; the \"first principles\" approach is compelling. The conceptualization of token dynamics as a balance between \"semantic alignment\" and \"distributional uniformity\" is intuitive and provides a compelling theoretical foundation for the architecture.\nThe mathematical derivations that connect these two principles to the standard Transformer components are elegant. Showing how attention, FFNs, and RMSNorm emerge as natural solutions to a single constrained energy minimization problem is the central theoretical strength of this work.\nThe empirical evaluation is extensive and respectable. The authors are transparent about the model's performance, showcasing its strengths in parameter efficiency and extrapolation on tasks like Sudoku, while also honestly reporting its limitations on larger-scale datasets. This thoroughness provides a complete picture of the model's utility and grounds the compelling theoretical framework in practical results.", "weaknesses": "The primary drawback, which the authors acknowledge through their experiments, is the model's difficulty to scale to larger datasets (like ImageNet-1K) and more complex generative tasks. On these, it lags behind standard Transformer baselines. While the authors are not expected to conduct a full-scale SOTA-level experiment, and their attempt to scale by stacking two distinct layers is a valuable inclusion, the paper would be significantly strengthened by a more in-depth discussion on how this scaling limitation could be overcome. The paper falls short in hypothesizing why this limitation exists. Is the hard parameter-sharing of the recurrent model creating an optimization bottleneck that a deep stack of independent layers avoids? The work would be more complete if it drew parallels to existing literature on deep vs. recurrent models or the challenges of weight-sharing in large-scale settings. A discussion on potential paths forward would provide a more comprehensive outlook. Without this, the proposed model, while theoretically compelling, feels disconnected from the path toward practical, large-scale application.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8jYcDT8E2C", "reviewer": "Reviewer_fUR7", "summary": "The authors propose Hyper-SET architecture, a recurrent-depth transformer-like sequence-to-sequence architecture. It is constructed as a discretization of a constrained gradient flow on a modified Hopfield energy function restricted to the product of spheres. This network is a single layer iterated an arbitrary number of times, with one layers' worth of parameters being shared across every application of the layer. The layer itself is a novel construction obtained via gradient descent-like steps on different parts of the aforementioned Hopfield-like energy. The authors demonstrate via experimental results that under this highly parameter-constrained setup, the model performs competitively against other architectures formed in the same way (i.e., single layer iterated arbitrarily many times) with different architectures for the layer, e.g., a standard transformer module, across a diverse set of problems.", "questions": "Q1: Can you elaborate a little on the surprising performance of the baselines in the experiments? Namely,\n- Q1.1: For Figure 3a, do you have intuition why otherwise reasonably-successful architectures completely fail this task and get a 0?\n- Q1.2: For Table 2, do you have intuition why the energy transformer performance drops by a large amount on IN1K?\n\nQ2: Can you train some non-weight-shared networks with more parameters, e.g., using a largely validated pipeline such as the original ViT pipeline [1], to simultaneously establish fair comparisons w.r.t. deep network baselines, and show in some more detail how the proposed network can be made practically performant (in either the single-layer recurrent-depth or deep settings)? \n\n[1]: Dosovitskiy, Alexey. \"An image is worth 16x16 words: Transformers for image recognition at scale.\" arXiv preprint arXiv:2010.11929 (2020).", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The mathematical derivation seems complete, correct, and interesting: while cooking up some objective whose optimization unrolling/discretization reproduces a transformer-like architecture has been done before [1, 2, 3], only a few previous works such as [3] attempt to give a holistic interpretation to the objective or optimization procedure.\n- The new block is novel and shows a genuine attempt at developing a prescriptive theory to improve the current practice.\n- The paper is written well; the connection to previous work and situation in the literature is clear; and the current approach is well-motivated and rigorously exposited.\n\n[1]: Ramsauer, Hubert, et al. \"Hopfield networks is all you need.\" arXiv preprint arXiv:2008.02217 (2020).\n\n[2]: Yang, Yongyi, and David P. Wipf. \"Transformers from an optimization perspective.\" Advances in Neural Information Processing Systems 35 (2022): 36958-36971.\n\n[3]: Yu, Yaodong, et al. \"White-box transformers via sparse rate reduction.\" Advances in Neural Information Processing Systems 36 (2023): 9422-9457.", "weaknesses": "Some meaningful weaknesses:\n- By the authors' own admission, the proposed network \"suits better [sic] under a resource-constrained setting as its inherent structural biases could limit its scaling on large datasets.\" This seems like a downside compared to the regular Transformer whose benefits over other architectures _only_ manifest at scale. With this in mind, maybe comparisons to some simpler RNNs/convnets may be meaningful?\n- Comparisons against other single-layer recurrent-depth baselines require some more explanation and elaboration (beyond Appendix D). See \"Questions\" Q1 below.\n- The single-layer recurrent-depth setting is interesting but impractical. A well-trained ViT with 12 (non-tied) layers can easily get > 90 on CIFAR 10 for instance (cf Table 1 where the high score is 90, whereas even the original ViT paper [1] boasts 99% in Table 6). Almost all practical use cases do not use this weight sharing. With this in mind, the empirical results currently do not demonstrate the practical efficiency of such networks. See \"Questions\" Q2 below.\n- Relatedly, the baseline deep networks in Figure 4 seem under-tuned, e.g., the baseline 12-layer ViT is said to get 66% on CIFAR100 in Table 4 but [1] Table 6 shows that a 12 layer ViT can get > 90% on CIFAR100. See \"Questions\" Q2 below.\n\nSome nits:\n- It does not really make sense to write two separate ODEs to show that you optimize each part of the features (e.g. (7) and (10)), which would suggest a parallel-structure network. Instead it is more mathematically sound to have a combined ODE with both RHS terms summed together, and obtain the network architecture via operator splitting (or some similar technique).\n- Table 2 empirical performances in the first two columns are close and seem within-noise, requiring 5 significant figures on the benchmark to distinguish the models; within this context, bolding is done slightly incorrectly, e.g., only bolding Hyper-SET numbers in the case of a tie (0.417 in the second column). On the other hand, the text referencing Table 2 is slightly unfair to Hyper-SET since it can emphasize parity in the first two columns instead of saying it lags in all metrics.\n\n[1]: Dosovitskiy, Alexey. \"An image is worth 16x16 words: Transformers for image recognition at scale.\" arXiv preprint arXiv:2010.11929 (2020).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)