NRGPT: An Energy-based Alternative for GPT
Authors: Nima Dehmamy
Benjamin Hoover
Bishwajit Saha
Leo Kozachkov
Jean-Jacques Slotine
Dmitry Krotov
Venue: ICLR 2026 Poster (2026)
Keywords: energy-based model
GPT
LLM
small language models
Primary area: generative models
TL;DR: we introduce an energy-based GPT model, where inference is gradient descent on the energy; we evaluate it on three datasets Listops, Shakespeare and OWT

Abstract: Generative Pre-trained Transformer (GPT) architectures are the most popular design for language modeling. Energy-based modeling is a different paradigm that views inference as a dynamical process operating on an energy landscape. We propose a minimal modification of the GPT setting to unify it with the EBM framework. The inference step of our model, which we call eNeRgy-GPT (NRGPT), is conceptualized as an exploration of the tokens on the energy landscape. We prove, and verify empirically, that under certain circumstances this exploration becomes gradient descent, although they don’t necessarily lead to the best performing models. We demonstrate that our model performs well for simple language (Shakespeare dataset), algebraic ListOPS tasks, and richer settings such as OpenWebText language modeling. We also observe that our models may be more resistant to overfitting, doing so only during very long training.

Published as a conference paper at ICLR 2026
NRGPT: AN ENERGY-BASED ALTERNATIVE FOR GPT
Nima Dehmamy
Benjamin Hoover"
Bishwajit Saha" IBM Research
IBM Research
IBM Research
Leo Kozachkov
Jean-Jacques Slotine MIT
Dmitry Krotov* IBM Research
Brown University
ABSTRACT
Generative Pre-trained Transformer (GPT) architectures are the most popular de- sign for language modeling. Energy-based modeling is a different paradigm that views inference as a dynamical process operating on an energy landscape. We propose minimal modification of the GPT setting to unify it with the EBM frame- work. The inference step of our model, which we call eNeRgy-GPT (NRGPT), is conceptualized as an exploration of the tokens on the energy landscape. We prove, and verify empirically, that under certain cireumstances this exploration be- comes gradient descent, although they don't 't necessarily lead to the best performing models. We demonstrate that our model performs well for simple language (Shake- speare dataset), algebraic ListOPS tasks, and richer settings such OpenWebTex. WebText language modeling. We also observe that our models may be more resistant to overfitting, doing only during very long training.
Transformers represent a dominant paradigm in autoregressive language modeling (Vaswani et al., 2017). In typical setting, a sequence of tokens describing a text is passed through several transformer layers and mapped onto new sequence, which is a copy of the original one shifted by one token and appended by the token that follows the initial sequence. At training time, this network is trained through self- supervised training, and at inference time the network is used for next token prediction. This is the standard Generative Pre-trained Transformer (GPT) setting, which is the first step in Large Language Model (LLM) design (Radford et al., 2018).
Energy-based modeling (LeCun et al., 2006) is another prominent paradigm in modern Al landscape that historically goes back to Hopfield Networks (Hopfield, 1982). this framework the operation of the neural network is defined by scalar energy function. Proper samples generated by the model (those that resemble training data) correspond to low energy states, while unrealistic samples (with large deviations from the training data distribution) correspond to high energy states. Thus EBMs particularly appealing for their theoretical properties, because they view the "forward pass" through a deep network an explicit optimization problem that maximizes the likelihood of the input data. Practically, this approach unlocks better tooling for systematic exploration of the solution space and enables natural solutions for both variable computation and model alignment via regularizers (see our extended discussion on EBMs for LLMs in Appendix C). Although at face value these two approaches look very different, in recent years growing number of studies hint at deep connections. Von Oswald et al. (2023) showed evidence that in-context learning (ICL) may be gradient descent by constructed an explicit weights such that the forward pass was GD MSE loss. Ahn et al. (2024) further showed that transformers learn preconditioned GD for ICL. However, both of these works make significant simplifications, such as considering only linear transformers, omitting the softmax.
Other works have have attempted to reconcile transformers and EBM from several angles. For instance, the Energy Transformer (Hoover et al., 2023) is architecture, which is simultancously a transformer and an energy-based model. In the image domain, the typical setting would be to reconstruct set of masked tokens (patches) given the set of open tokens. The network solves this task by performing a gradient descent of the energy on the space of tokens at inference time. This architecture is inspired by associative memory models (Krotov and Hopfield, 2016) and for this reason solves the following problem: given a partially incomplete pattern - complete it in a meaningful way.
"qqual contribution.

---

Published as a conference paper at ICLR 2026
This aspect of the core design makes it difficult to apply Energy Transformers to GPT settings, in which the sequence needs to be transformed to a shifted sequence by means of going through the network. Intuitively, in Energy Transformers the masked tokens need to evolve rapidly to match the missing parts of the pattern (e.g.. image or graph), while the open tokens need to stay almost constant to harely adjust for the smooth transitions between the masked and the open tokens within the pattern. This is in drastic contrast with the GPT setting, in which there are no masked tokens at all. Rather, every token needs evolve into the following token in the sequence.
A different line of work is inspired by "System 2" thinking and attempts to design an energy-based network for processing language (Gladstone et al., 2025). In this study, transformers are used as an architectural motif that casts text into scalar energy function. While models of this nature have benefits for language processing, they belong exclusively to the class of energy-based models, and are unrelated to the GPT settings, commonly used in most LLMs.
Individual modules within the transformer block, such as attention, have also been studied from the perspective of inference time optimization (Geshkovski et al., 2023; 2024). In this line of work, peculiar clustering properties of tokens have been observed. Energy-based optimization has also been studied in (Yang et al., 2022) from the perspective of enjorization-miniiizzln algorithms.
Despite this growing list of studies dedicated to synergies between autoregressive transformers and energy-based models, at present it remains unknown how cast the commonly used GPT setting into a well-defined energy-based framework. Our work tackles this gap. We refer to our model as eNeRgy Generative Pre-trained Transformer NRGPT. The input sequence of tokens is mapped onto shifted sequence of tokens, which includes the next word, see Figure 1. The mapping is performed by neural network, which recurrently applies the NRGPT block to the sequence of tokens. Each application of the block uses gradients of the network energy functions to update the state of the tokens. Each token has its own energy landscape, which dependent on the states of other tokens. Specifically, our contributions are:
We design an energy function and an update rule that describes the GPT setting with several possible variants including learnable inference rate and normalization operations: LayerNorm and RMSNorm.
We obtain excellent results on nested ListOPS tasks, including arithmetic operations. min/max selection, etc.
We show the feasibility of using NRGPT for language modeling on Shakespeare and Open WebTee datasets.
We do systematic comparison of performance scaling of recurrent transformers and NRGPT.
We study empirically the properties of dynamical trajectories of tokens on the energy landscapes of our models.
ENERGY-BASED MODELING
In generative modeling our goal is to generate samples with a distribution close to observed datapoints. If we manage learn an approximate likelihood function for the dataset, we can generate data by sampling. This is also the premise Energy-Based Models (EBM). An example of EBM would be Dense Associative Memory (Krotov and Hopfield, 2016), where datapoints are stored in minima of an energy function. But more generally, the energy can represent negative-log- lo--loolll-ioood. E(x) log Pxx). In this case, the global minima of the energy represent maximum likelihood solutions. The deeper the energy, the higher the likelihood of that datapoint. One strategy to train an EBM is to first learn the energy function by fitting the distribution of the data. The sampling process would then be separate from learning the energy.
However, in high dimensional data, learning the distributions is notoriously difficult due to the curse of dimensionality. Diffusion models solve this problem by starting from high noise and cooling down. Diffusion models do not learn an explicit energy function, only its gradients, the score function. Yet, having explicit energy function could enable to explore the solution space in ways not easily afforded by the implicit score function of diffusion models. So can we build a model which learns the energy directly?
2

---

Published as a conference paper at ICLR 2026
Logits
d.I.
be
or
not
to
NRGPT Block
..1.
(++1)
Each token explores energy landscape
Block sums foed fan-rfonwrrddee energies
NRGPT
E
E
Vggat
V
Token prediction recurrent dynamics on energy landscape
recurrantl follows energy gradient
Artn energy
FF energy
Energy
Input Tokens
x!=1) 2 v/ g
Leorm
One block. t= 1:2 unrolled "layers"
To
x(9
Time
J..I
Figure 1: NRGPT casts the standard GPT setting into an energy-based framework. The network defined as the sum of two energies: an attention energy and feedforward energy. Each token is transformed into the next token by exploring the energy landscape. Recurrent application of the NRGPT block produces dynamical system where each token can be thought of as a particle moving on the network's energy landscape.
Similar to diffusion models, we use an end-to-end process, where learning the energy and generating datapoints are all done one pass. The key idea is to have a differentiable sampling process which allows us to learn the parameters of the energy during sampling. Since real datapoints should have low energies, we choose a gradient-based sampling process. Note that do not need to descend all the way to minimum (i.e. maximum likelihood solution), since we want diverse samples. Instead, we do fixed number energy GD steps and demand that the final point matches real datapoints.
Generated data: (T)
(++1) =x(0 17 (()(())
(1)
for a fixed number of steps T, where (") is some random initial point. Here r7(1) is matrix that may depend on x. This matrix has many different names, e.g., kinetic rates in physics, preconditioner in optimization, etc. We will call this matrix the inference rate, since it determines the size of the steps that the inference dynamics takes on the energy landscape. But how do we judge whether the output matches real datapoint? One way would be have judge, like the discriminator in GAN. Another setting where judging the output is more natural is autoregressive language modeling where the datapoints the next tokens and can be matched the training text. In this case R represents real data sequence of length N embedded in D dimensions. In causal language modeling the energy should take X (x1 ... XN-1) input and predict x.v. in
((+1)
1) 7
Following the observations the Energy Transformer (ET), we will show that one can choose parametrization for E such that the process of T -tee GD closely resembles the forward through a T layer GPT transformer with a seiggt-sharing pattern.
2 NRGPT MODULE
H log
N Σ B=1
1
Σ exp (BgEJ" ga 8.))
EA
gB W2
W WigB
(3)
3
BCA
In this section will start from the structure of the transformer model and derive the energy function whose gradients yield layer which is very close in structure to a transformer layer. Let E D be an input sequence of length N embedded in D dimensions. We will denote its components by X with A = =1..N and i = ..... D, or X suppressing the embedding index, but keeping the token index. Let (() be the output sequence of layer i of the model with (() = x. A conventional transformer layer has Attention layer (AT) followed by two-layer feedforward (FF) and LayerNorm (LN) in series
x( (++1)
x(())
FF LN x()) AT((N(((11))
(4)
3

---

Published as a conference paper at ICLR 2026
But subsequent works such GPT-J (Wang and Komatsuzaki, 2021), PaLM (Chowdhery al., 2023), and Energy Transformer (Hoover et al., 2023) showed that the following parallel design has good performance too
Parallel Transformer: x x + AT(g + "),
g [t) LN(x!0)
(5)
We choose this parallel transformer design as it is suitable for goal of replacing the transformer layer with the gradient of an energy.
If passing through layer becomes one step of energy decent (ED), then all layers need share weights. Therefore, our model will consist of a single module replacing the transformer block. Instead of different layers, we will be recurrently feeding the output of the layer back into itself, so that x(!) will become step of the ED instead of the layer number.
Update Rule An important point to note is that the update rule of NRGPT is slightly different from conventional gradient descent and is of the form
x (1+1) x(() x(0)= -np OF
(6)
0g(1)
where (7) E DxD is an inference rate matrix, which can be learnable. Nevertheless, this can be valid descent on E as can show that E(++)) - E(4 < 0 under certain conditions, which depend on the normalization operation g. We will derive these conditions for LayerNorm as well
as RMSNorm, as well as when g i.e., no normalization in Section 2.2. Next, we introduce the energy of NRGPT module.
2.1 ENERGY OF NRGPT
Matching our update rule (6) the parallel transformer (5), define two terms in the energy, EAT and FF
E E= EAT + FFF
no, = - =
AT)
no, = FF((g)
(7)
We begin by introducing the attention layer and deriving the energy function the self-attention mechanism. Then, we derive the energy function for the FF. Finally, we combine the two energy functions obtain the total energy function for the transformer layer.
Attention. Consider multi head attention module with H heads, and hidden dimension Y D/H, index h enumerates heads and runs h = 1... ....H. H. Its query, key, value and projection weights are
Wºº WK K W WP HRXxDD
(8)
Using the standard K WK g, 1o g, V WV g. The MHA output for token A is!
AT = Σ W "V.''V V,SM (K. QAh)
(9)
denoting J [W*]] W the softmax is defined as (we omit the self-interaction term C A)
cxp (Bg JgA) (M(KQQ)BA = Eccs exp Jgg)
== VY
(10)
Following Hoover et al. (2023), define the attention energy
E (g)
1 Σ On log
Σ BeA
exp (Bgh))
(11)
where a E Rd is learnable weight.
Usualll the projection weights V are defined D) D and the head outputs are concatenated, into an N (Y N matrix before multiplying by P.. This is equivalent our definition.
4

---

Published as a conference paper at ICLR 2026
Taking the gradient of EAT w.r.t. 94 and using (7), the resulting attention layer becomes
(EAT(g) (g)
AT(g)A -17 ŌgA
Σ ann gSM (g" JngA).
(12)
Both standard attention and this update have the form (g) = MMgM(gg Jg), but in the standard attention M [W W and NRGPT attention is M anJ". That is,
[W T W = Energy: o
Original:
(13)
In principle W and W can be merged into matrix. (He and Hofmann, 2024) also experimented with removing W and W and found that in the setting without skip connections, these two weights could be largely omitted.
Feed-Forward network. The FF network generally has two layers FF(gA) = W (W
with weights W W E RMXD, with M being the size of the hidden layer. A possible choice for this network is a Dense Associative Memory (Krotov and Hopfield, 2016). In this case
ET = Σ A=1
1 (w) gA), s.t. =o =a
FF
(g( = -11 Ōgg
W (W gA)
(14)
where 1 is an M-dimensional vector of ones. Both the standard MLP in transformers and the FF update in Equation (14) have the form Mo(W'g). In the standard MLP, M W whereas in NRGPT we get M W'n That is,
Original: W Energy: W'n"".
(15)
As example, in order for EFF reproduce the FF of transformers with (z) ReLU(()) max(z, 0), the function F should be
F(z) - 702)
(16)
Of course, the module can replaced by other, more general, MLP networks. Essentially, any scalar function, which is additive in token index, can serve valid form of FF network. In the experiments (Section 3) we will detail our choices of FF
Distinction from prior work NRGPT's energy is distinct from prior works like that of Energy Based Transformers (EBT) (Gladstone et al., 2025) and Energy Transformer (ET) (Hoover et al., 2023). Specifically, EBT computes energies of next tokens output of standard transformer forward pass, while NRGPT describes a parameterized energy whose gradient returns a transformer block. Repeatedly minimizing the energy by following this gradient resembles complete transformer forward pass. On the other hand, ET's architecture appears more similar to NRGPT's design, but it's design was unsuited for causal token generation. We expand on these differences further in Appendix B.
2.2 NORMALIZATION OF TOKENS
These normalizations have the form
(17)
21 +
with μ E[z] for LayerNorm, and μ 0.8 8 0 for RMSNorm. Here 6, R and @ is elementwise multiplication. Many recent models such as Qwen Bai et al. (2023) and Llama Grattafiori et al. (2024) use RMSNorm Zhang and Sennrich (2019).
5

---

Published as a conference paper at ICLR 2026
Proposition 2.1 (Energy Descent). The update rule (6) results in asymptotically decreasing energy.
EA E F O, if the inference rate is n diag(n) with Ryo- This asymptotic behavior begins after transient phase where previous tokens are converging.
Sketch of proof. See Appendix E for full proof. The Jacobian of 9 can be written as òga/Ox, A = AA, where T diag(7); PΛ > some norm of æA4 and PA an approximate D D for B < A. get Σ
projection matrix and p.s.d. Using this and (6), once ãB -A Σx Tr 0g EAT 0g. ET). When 17 T get E 0.
There also exist z-dependent solutions of the form n M (x) where (x) is an arbitrary positive semi-definin (psd) matrix and c 0, but they mean n is itself neural network. To remain close to the structure of conventional transformers, we work with the z-independent 77 cf solution. In principle, 17 can also have part contributing to the anti-symmetric part of PP''" but we could not find an z-independent solution for it. While r7()) = I puts severe restrictions on the preconditioning matrix, each layer is still allowed to have a different e! constant. Yet, as we show Appendix LayerNorm together with Lipschitz activation functions in FF (e.g. ReLU) will lead to bounded energy, which cannot explode. In this case, a generic 17 may still yield convergence. In fact, only a small set of 77 which yield perpetually oscillating solutions may not lead to convergence, although the exact conditions need to be derived.
Preconditioner without layer normalization. Several works have shown careful initialization and scaling replace normalization entirely (Huang al., 2020; Wang al., 2024). Doing so would remove much the restrictions on the preconditioner r). In an Tayerr transformer, Instead of layer normalization, T-Fixup (Huang al., 2020) and DeepNet (Wang et al., 2024) initialize some weights and z by scale proportional to 1/4 and change most layer outputs to f(g) + af) with x T -12/2, In this case, 1/ becomes much less restricted.
Proposition 2.2 (É without normalization). When g æ. to get É 0, the symmetric part, 7 = (1) + 1/2 needs to be psd.
Proof. In this case ě = -nd E and
E
Σ
Tr |de Emd ET| = Σ Tr En
(18)
which is negative when 17+ is psd.
The antisymmetric part 7- (n / Hence, without layer normalization,
22tx is not constrained by this and can be arbitrary this point. have (9) learnable parameters of the form
U D VERDXD
(19)
where D' can be arbitrary, and each layer (depth) can have different learnable U V(o.
2.3 ASYMPTOTIC STABILITY OF NRGPT
A peculiar aspect of NRGPT is the phenomenon of asymptotic stability. In order to illustrate it, consider a simplifying case when the inference rate matrix r is identity. In this case dynamical equations for tokens be written
0FA
F:A = 0giA
(20)
Additionally, 7 can always be absorbed into J and the weights of the FF model. This way, the Jacobian becomes 0 PA. which is p.s.d.. The key observation is that due to causal attention mask the energy EA of token A only depends on the states of tokens < A. Thus, for the first token
0E
(21)
0gi

---

Published as a conference paper at ICLR 2026
Since the energy of that token decreases with time
aE dgi dæ
E
- P 0
(22)
Ögi dzy dt
since P1 is psd. Additionally, since energy only depends on g layernormalized tokens it is bounded from below. Thus, the dynamics of has to converge to fixed point. This means that after a transitory period of time Tu the derivative dgi vanishes. dt
Now, consider the network of two tokens
8E2 0gi
0E2 0ga dæ² = = Pada $
ES
(23)
ōg. dt
0g2 dx² dt
the second equality is written assuming that are looking at this quantity at L > Ti. Thus the first term is zero. Same argument applies, the energy decreases with time and is bounded from below. Thus, eventually De freezes.
One can apply this argument recursively to each token and conclude that after a transitory period of time, all tokens stabilize and all 9A will eventually become constants. From the perspective of energy profiles, this leads to the following behavior: during transitory regime energies of individual tokens will evolve in time (they can both increase and decrease). After that transitory period is over the energies must stabilize and reach their constant values that become unchanged in the future. One can run the inference dynamics as long as desired after that, but no changes in energies will occur. This behavior is apparent from the numerical profiles of energies, see Figure 2. It is distinct aspect of our models the phenomenon that we call asymptotic stability.
3 EXPERIMENTS
We tested model three datasets: ListOps, Shakespeare and Open Web Text (OWT). The details of the experimental settings and hyperparameters can be found in Appendix F. To evaluate the quality of text in the Shakespeare and OWT experiments, we use number of metries, including perplexity and diversity scores, explained in Appendix F.2.
The choice of the energy function is equivalent to the choice of architectures in neural networks. As our goal is to be as close to the original transformer architecture as possible, we experimented with a few settings for the FF network and found the following variants to be the best performing:
Model Variants.
1. NRGPT_H_FFT EFF = (W same as in equation 14, but with GEELU.
2. NRGPT_H_FF2W. EFF = =- -- W W gn), which yields
FF(ga) = =-n (W (W2o(WW/ 9A) o' W' ga)" w Wol (W229a)
(24)
Here, the first term the conventional FF of transformers, but the second is somewhat odd network.
In case with two weights, we choose the hidden dimension between W1 and W to be 4 x D. All of these performed well, with the residual version showing best performance on ListOps, learning at even smaller sizes than our baseline, while also easily training at larger sizes with embedding size over 256. However, since some these models deviate significantly from the FF of recurrent GPT (the gradient results in an FF module with four layers), decided to focus more on NRGPT_H __ FF1 and NRGET. H. FF2W, which are much closer to the GPT FF. As Baselines, we used GPT Rec parallaaaaa 1el, which is GPT- model with single transformer layer, feeding back recurrently into itself for a fixed number of times (mimicking number of layers). On Shakespeare and OWT, we also show results of conventional GPT2-style deep transformer model.
3.1 ENERGY DYNAMICS
7

---

Published as a conference paper at ICLR 2026
Total Accuracy (learning > 0.8)
1.0
GFT Jocpern
ListOpp Minimum Number Parameters for Selected Models
1.5x 30
GPTLM Rec, Rec_paralie NRGPT FF2W
1,4X 10
1.3x
WRGPT_H
Mic Trait Loak
1,2x100
100
101
GPTLM paraful
NAGPT H. FF2W
10a
6000
NRGPPT H. FF1
10
10a
10
10
10'
10*
Number Parameters
Number of Parameters
Figure 3: Learning ListOps: NRGPT variants match performance with recurrent GPT model on ListOps accuracy parameter-transition points (top) and training/validation losses (bottom). The accuracy of models is tested on nested, mixed arithmetic tasks of maximum, median and sum modulo 20. For all plots, the x axis shows the total parameter count the model. The yellow star indicates the transition to learning, which we define as where the logistic fit hit 80% accuracy. The bascline model GP T Rec parallel shows the earliest learning transition at size 2.3x x 104, but our NRGPT variants are also similar, with NRCPT FF1 at 2.4 x 10 and NRGPT H FF2W at 2.98 x 10*.
NRGPT without constraints on the inference rate 7j is not
Total Energy
forced to strictly decrease energy during inference and it
30-
may learn other exploration strategies for inference. Nev- ertheless, we would like examine whether models which are explicitly forced to perform GD and reduce energy dur- ing inference can learn the tasks well. To better understand update rule performs inference, we ran experiments on ListOps with large number of recur- rent steps (30 steps). For these experiments, we set 1) = 1, which according to Proposition 2.1 forces the update rule to decreases energy. Figure 2 shows the evolution of total E. EAT and EFF along these trajectories. We observe that indeed in all trajectories the final energy is lower than initial. Each individual token trajectory is not required to be monotonically decreasing, the dynamies of tokens is coupled. however, in accordance with our result in Sec- tion 2.3, after a transient stage, once all previous tokens start converging, the energy of the next token monotoni- cally decreases.
Attention Energy
how our gradient-basse
Feed-Forward Energy
10 15 20
25
30
Timestep ("aayer)
Figure 2: In NRGPT, tokens converge stable states of low energy where the causal attention mask allows each token energy to fluctuate during infer- ence. Shown are 64 tokens passed to an NRGPT model trained to predict ListOps equations.
3.2 LISTOPS
We perform experiment on nested math operations on lists of integers, which are a version of ListOps (Nangia and Bowman, 2018). Our ListOps setting consists of three functions: maximum, median and sum modulo 20. Our inputs range from 0 to 19, Each
data sample begins with nested equations like SUM (2, MAX (4, 13,, 1) MED (5, 3, 3,1 ) As performance metrics, we looked accuracy on the mixed task, as well as the training and validation
loss. Figure shows the results for two of our model variants, NRGPT H. FF1 and _F22 compared to GPT GPT__ec_________________________ paral lel.
8

---

Published as a conference paper at ICLR 2026
Best Validation Loss
Best Train Loss
14
1.5
1.25
H
15
GPT paralll!
GPT
34
0.00
Number
Figure 4: Shakespeare scaling: NRGPT achieves performance parity with recurrent GPT Shakespeare across parameter sizes, as measured by best validation loss per number of parameters. For many embedding sizes, NRGPT also follows the parameter as both GPT and recurrent GPT baselines. However, NRGPT does not overfit Shakespeare large parameter sizes. Connecting lines show the best performance fixed parameter sizes. Transparent dots show different choices of hyperparameters a larger spread indicates more sensitivity hyperparameters. See Appendix F.4 for details.
optimal training loss trajectory-per -pe--
Table OWT performance at Neymeed 768.
Training loss min
Model
mean + std
max
# runs
# Param.
GPT
2.905 +: 0.006
2.900
2.914
5
124M
GPT Rec paralell
3,447 + 0.046
3.395
3.494
5
85M
NRGPT H FF2W
3.456 0.076
3.391
3.540
3
90M
Validation loss
Model
mean +: std
min
max
# runs
# Param.
GPT
2.921 + 0.005
2.915
2.929
5
124M
GPT Rec Rec_Reee__aeeeeeeeeeeeeeeeeeeR
cec_parallel
3.454 0.037
3.411
3.491
85M
FF22
3,467 = 0.073
3.404
3.548
3
90M
3.3 SHAKESPEARE
We compare the performance of our NRGPT __FF2 and NRGPT_H_ER GPT_Rec_paralle and deep GPT for embeddings sizes less than 1024 (Figure 4 In larger sizes, we ran many sweeps to find suitable hyperparameters such as the range of learning rates, resulting in the wide spread. Interestingly, our best models large sizes achieve validation losses similar to the GPT baselines, but do so with much less overfitting to the training set.
with
3.4 OPEN WEB TEXT
Table 2 shows the best model configuration for baseline GPT, RGPT- and our model NRGPT with the respective generation quality metries. We see that the generation quality of NRGPT is very competitive with GPT and RGPT-parallel while it contains around 34M less parameters than GPT. Figure 5 shows example of generated text by GPT, RGPT-parallel and NRGPT for which the generation quality metrics are provide in Table 2. In these experiments, we consider transformer blocks configured the same width please see Appendix F.5 for more experiments across equal parameter counts, including experiments that show NRGPT's advantage on downstream tasks such as MMLU (Hendrycks al., 2020).
LIMITATIONS
NRGPT an appealing theoretical construct for the inference process of GPT. In our experiments, we observe that NRGPT achieve similar performance to GPT and its recurrent variants ListOps,

---

Published as a conference paper at ICLR 2026
Table 2: Best Model Configurations and Quality Metries for OWT. Note abbreviations: of parameters + u_ param, grammar quality score + gqs, average pariwise cosine similarity - apcs, distinct-1 + d-I and distinct-2 d-2.
Configuration
Metries
Model
min_
layer
head
n_embedd
n_params
perplexity
g4y
apes
GPT
7e-4
76:5
12
768
124M
0.978
0.306
0.619
0.965
GPT_Rec
Rec_Parallel
Rec Parallel
62-4
4e-4
12
12
768
85M
99
0.976
0.336
0.615
0.975
NRGPT_H FFFW
le-4
7e-5
12
12
768
90M
104
0.966
0.306
0.674
0.984
Shakespeare, and OWT. However, NRGPT is computationally the gradient of an energy, which enforces weight sharing and limits how flexibly we can parameterize the architecture. We observe that this constraint also causes larger amount of hyperparameter sensitivity than GPT variants. We also notice that the configurations of NRGPT that achieve competitive performance at comparable parameters actually have a slightly larger FLOP count than their standard transformer counterparts. Please see our extended discussion Appendix D. contrast to standard transformers, increasing the number of attention heads in NRGPT actually increases the parameters. We additionally observe that NRGPT has a more difficult time overfitting the training set, which is beneficial in small data regimes but is undesirable in the massive datasets used to train modern LLMs.
DISCUSSION AND CONCLUSIONS
We have presented NRGPT, minimal modification of the GPT architecture that unifies autoregressive language modeling with energy -ased modeling. Our analysis shows that under specific conditions on the inference rate matrix 77, this process provably decreases energy after a transient period of time, providing principled foundation for the dynamics, phenomenon that call asymptotic stability. Moreover, relaxing this constraint allows the model to learn its own energy exploration strategy for inference. Thus, work complements previous studies suggesting that transformers perform GD during inference. Unlike past work, in our model inference is explicitly gradient-based dynamics, while still maintaining architecture very similar GPT. Our experiments show that this framework performs comparably to a fully recurrent GPT model across parameter sizes while generally requiring fewer parameters, NRGPT represents a meaningful step toward understanding the architecture of transformers using energies.
REPRODUCIBILITY STATEMENT
The code for our model, experiments, and analysis is available Shoov/nrgee in self-contained environment. The code began as fork of the excellent nanoGPT repository and such all experiments and models are implemented using PyTorch (Paszke al., 2019). Each Shakespeare and ListOps experiment was conducted single H100 GPU. OWT experiments were conducted over nodes of 8xH100 GPUs.
com/
ACKNOWLEDGMENTS
The results presented here were obtained while Dmitry Krotov was employed by IBM Research. At the time of the camera-ready submission Dmitry Krotov no longer employed by IBM Research.
REFERENCES
K. Ahn, X. Cheng, H. Daneshmand, and S. Sra. Transformers learn to implement preconditioned gradient descent for in-context learning. Advances Neural Information Processing Systems, 36, 2024.
Bai, S. Bai, Y. Chu. Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, et al. Qwen technical report, arXiv preprint arXiv:2309, 16609, 2023.
10

---

Published as a conference paper at ICLR 2026
T. Barrus. pyspellchecker: Pure python spell checking, 2020. URL URLhttss///gthub.com barrust/pyssse paaruss/paapellcheeee:
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners, Advances in neural information processing systems, 33:1837:11-1 187--191,, 2020.
A. Chowdhery, Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, Barham, H. W. Chung. C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Rexearch, (4(240):1-113,, 2023.
Q. Dong, L. L. D. Dai, C. Zheng. J. Ma, R. Li, H. Xia, J. Xu, Z. Wu, B. Chang, et al. A survey on in-context learning. In Proceedings of the 2024 conference on empirical methods in natural language processing, pages 1107-1128, 2024.
R. Flesch. A readability yardstick. Journal of applied psychology, 32(3):221, 1948.
B. Geshkovski, C. Letrouit, Y. Polyanskiy, and Rigollet. A mathematical perspective on transform- 2023.
ers. arXiv preprint arXiv:23/2.10794, 2023.
B. Geshkovski, C. Letrouit, Y. Polyanskiy, and P. Rigollet. The emergence of clusters in self-attention dynamics. Advances in Neural Information Processing Systems, 36, 2024.
A. Gladstone, G. Nanduru, M. M. Islam, P. Han, H. Ha, Chadha, Y. Du. H. Ji, J. Li, and T. Iqbal. Energy-based transformers are scalable learners and thinkers. arXiv preprint arXiv:2507.02092, 2025.
A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The llama herd of models. arXiv preprint arXiv:2407.2178. 2024.
B. He and T. Hofmann. Simplifying transformer blocks. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https://p///.cc.ttt RtDok9es3s.
https:://pperewieeeeee/ /owpr//pwereeeeeeeeeeeee net/ forum?id-
D. Hendrycks, C. Burns, Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300) 2020.
B. Hoover, Y. Liang, Pham, Panda, H. Strobelt, D. H. Chau, M. Zaki, and D. Krotov. Energy transformer. Advances in neural information processing systems, (6:27532-275551 2023.
J. Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(8):2554-2558, 1982.
X. S. Huang, F. Perez, Ba, and M. Volkovs. Improving transformer optimization through better initialization. In International Conference on Machine Learning, pages 1475-4483 PMLR, 2020.
D. Krotov and J. Hopfield. Dense associative memory for pattern recognition. Advances in neural information processing systems, 29, 2016.
Y. LeCun, S. Chopra, R. Hadsell, M. Ranzato, F. Huang, et al. A tutorial on energy-based learning. Predicting structured data, 1(0), 2006.
N. Nangia and S. Bowman. Listops: A diagnostic dataset for latent tree learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 92-99, 2018.
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, seghrrerformence deep learning library. Advances in neural information processing systems, 32, 2019.
A. Radford, K. Narasimhan, T. Salimans, Sutskever, et al. Improving language understanding by generative pre-training, 2018.
11

---

Published as a conference paper at ICLR 2026
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
N. Reimers and I. Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint 1rXiv:1988.10053 2019.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
J. Von Oswald, E. Niklasson, E. Randazzo. J. Sacramento, A. Mordvintsev, A. Zhmoginov, and M. Vladymyrov. Transformers learn in-context by gradient descent. International Conference Machine Learning, pages 35151-35174. PMLR, 2023.
B. Wang and A. Komatsuzaki. Gpt-j-6b: A 6 billion parameter autoregressive language model, 2021.
H. Wang, Ma, L. Dong, S. Huang, D. Zhang, and F. Wei. Deepnet: Scaling transformers to 1,000 layers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(10):6761-677 6774, 2024.
Y. Yang, D. P. Wipf, et al. Transformers from optimization perspective. Advances in Neural Information Processing Systems, 35:36958-36971, 2022.
B. Zhang and R. Sennrich. Root mean square layer normalization. Advances in neural information processing systems, 32, 2019.
12

---

Published as a conference paper at ICLR 2026
A LLM USAGE STATEMENT
LLMs were not used for ideation, experiments, or model design. LLM generated code helped with experimental analysis (e.g.. plot layouts) and grammar checking of the submission.
B NRGPT COMPARED TO EBT AND ET
B. DISTINCTION FROM EBT
Both the Energy-Basee -Based Transformers (EBT) of (Gladstone et al., 2025) and NRGPT are methods that can used minimize an explicit energy during autoregressive generation. However, the methods differ in how that energy is modeled. Specifically, the EBT paper uses standard, feed-forwer transformer architectures with modified attention masks and a standard loss function on predicted tokens to turn the transformer predictions into an energy, whereas we use novel, causal energy formulation of the transformer block inspired by the Energy Transformer (Hoover et al., 2023).
NRGPT is most comparable to the autoregressive EBT. The EBT paper mentions that "the au- toregressive EBT presents greater implementation challenges, primarily due to the potential for information leakage in naïve implementations" and they discuss the design challenges in Sec C.3 of their Appendix, which requires reframing how tokens are processed by the standard transformer architecture. To prevent information leakage, they duplicate sequence of tokens into observed tokens and predicted tokens while carefully tuning both the attention masks and the contraction over observed values.
We believe that NRGPT's solution for a causal EBM is more succinct and elegant than EBT's, preventing duplicate tokens and the need for careful tuning of attention masks. Our solution is expressed in Equation (3), where E' describes the energy to predict token A. By restricting the summation of the logsumexp to include summation only over previous tokens < and then by minimizing the energy w.r.t. token g.s. we are guaranteed to propagate information causally without any of the engineering tricks EBT.
B.2 DISTINCTION FROM ET
The architecture of Energy Transformer (ET) (Hoover et al., 2023) is more similar to that of NRGPT, but it is fundamentally different. This is because ET is designed for MASKed-token prediction tasks whereas NRGPT is a special energy function designed for autoregressive token prediction, a paradigm that is not compatible with the original ET's bidirectional attention design (where attention signal propagates both forward and backwards in time such that past tokens can attend to future tokens). A simple causal mask on ET attention breaks ET's guarantee of monotonic energy minimization.
The key innovation of NRGPT is to model the sequence collection of token-wist energies EA ΣA EA. We discover that tokens
for token index A, rather than a global sequence energy still converge even when all tokens minimize their individual energy simultaneously (see Figure 2), and we argue that this generalizes the ideas of ET to allow tokens to explore meaningful energy landscape during causal token generation.
We also thcoretically and empirically study the projection matrix that is present in all standard transformer attention but that is noticeably absent in ET's attention formulation. We interpret this matrix as an "inference rate" matrix 17 in the gradient descent step, where under certain conditions guaranteed to maintain token convergence. Including this projection matrix and playing with various choices for FF of Equation (3) makes the NRGPT block as expressive as a recurrent, standard GPT block that arguably go toe-to-toe with similarly configured GPT models on causal language modeling tasks up to the size of OWT (see Table 5 and Table 4). These conclusions and experiments were not evident in the original ET paper.
In summary, NRGPT is distinct from ET because:
1. NRGPT performs causal language modeling by minimizing a per-token energy. ET was restricted to strict energy minimization an entire sequence, paradigm that is not compatible with the parallel, autoregressive language modeling of GPT-style transformers.
13

---

Published as a conference paper at ICLR 2026
2. NRGPT uses learnable inference rate matrices 77 during token prediction. Meanwhile, ET was restricted to a fixed, scalar gradient descent step which did not allow additional exploration of the energy landscape.
3. NRGPT explores alternative mergy-replacenne module. ET used a single-layer Hopfield Network with energy G(Éga), which results in the weights of the layers be and <²_ In NRGPT, explore general form EFF (e.g., Equation (24)) for the feed-forward module and find improved results on the causal language modeling task.
for the feed-forward (FF) MLP
C ADVANTAGES OF EBMS
EBMs offer paradigm for generative modeling that is inherently about optimization, where the model samples new token from transition probability deseribed by the prior context. Conventional transformers are trained to implicitly learn this transition probability, but is hidden in the architectural design. The key appeal of an EBM (like NRGPT) is that it models the transition probability explicitly. This offers several advantages which we hope to explore in future work:
1. Systematic exploration of the solution space. An explicit likelihood function enables us to systematically explore the space of solutions in LLMs using well-established methods in optimization and statistical physics, such as alternative gradient descent methods, minimum- energy paths, saddle-poin analysis, and metastability. These tools are not available when the energy is implicitly encoded in deep architecture, and we believe that finding alternative or creative solutions may correspond to exploring different local minima of the energy during inference. It also means that as a formal optimization process, which despite the prevalence of research around in-context learning (ICL) (Dong et al., 2024), is not widespread belief.
now view the forward process of causal LMs like GPT
2. Variable computation using early stopping criteria. Both the energy and the norm of the energy's s gradient can be used as signals to stop model computation "early" for easier problems, or to continue thinking longer for harder ones. This is a primary motivation of other explicit EBM language models like EBT (Gladstone et al., 2025), and we discuss this advantage further below.
3. Model alignment using energy regularizers. Note that Equation (7) of our paper shows NRGPT's architecture as the sum of two energies: token-mixing atlention energy FAT and a token-wise feed-forward energy FFFF. Per the precedent of ET (Hoover et al., 2023), these are chosen to be faithful to the original transformer's ' design. However, any scalar objective can theoretically be added to NRGPT's block, and we imagine that adding regularizer terms to bias the energy landscape toward favorable solutions (or away from undesirable ones) is unique benefit EBMs that is more robust to prompt injection attacks than current LLMs.
Early stopping Adaptive stopping criteria for "early stopping" an incredible advantage for EBMs over traditional methods in the context of language modeling. If you can guarantee that all of NRGPT's tokens B < A are fixed, then the energy of token A is guaranteed to decrease, and early stopping based on energy values and gradient norms (as you suggested) would just work we are iteratively improving a token't 's embedding until the convergence guarantees are met and there is no purpose to "thinking longer". This iterative improvement needs no fine-tuning it is baked into the model design. However, the constraint of a fixed preceiding context has a strong "disadvantage doing this would discard the transformer's unique advantage of full parallelism across tokens. For the causal energy formulation with parallel token evolution studied in this work, we are actually not guaranteed to monotonically decrease the energy and energy deltas and gradient norms would be unreliable (see Figure 2 for roken-wise energy trajectories).
D ON FLOP COMPLEXITY
NRGPT is parameter efficient compared to standard GPT and recurrent GPT, but how does it compare on the FLOP cost of the model? When we consider the FLOPs/block (per iteration), we discover that NRGPT is anywhere from 1 1-2x the FLOP cost of equivalent RecGPT at constant parameters,
14

---

Published as a conference paper at ICLR 2026
where the factor of 2 difference appears depending which FF variant (NRGPT H FF1 or NRGPT H FF2W). We expand on the contributions of our architectural choices below:
NRGPT Attention: For a single head, the attention block of NRGPT costs approximately the same FLOPS as recurrent and normal GPT. fact, NRGPT uses slightly fewer FLOPs since the key weights Jg are used for both the keys and the values of the attention update (in contrast to the standard attention where distinct values W gB are computed for each key B). For multiple heads however, NRGPT costs more because our current solution uses x D weights Jn. per head, while GPT attention uses increasingly narrow KQV matrices you increase the number of heads (i.e., the dimension of W is of shape D x (D/ //HHH H). which decreases as becomes larger). This is convention that can be easily adapted to NRGPT but did not explore in this work.
NRGPT FF: This FLOP count for this FF module can vary, but in this paper we tested configurations that are always ~2 the FLOP count of a standard transformer's FF of the same width. We discuss the two FF modules studied in this work below:
FFI, where E | W g)ll" In ListOps, we found that FFI competes with Rec- GPT only when the hidden dimension of W was 8D instead of the standard 4D used in GPT. This makes FF1 a constant- constent-parameter operation, but because the wider weight matrix is used twice it actually doubles the FLOPS cost of NRGPT.
FF2W, where F = g" g W> g). In this config, the update rule V leads to two terms, each being two layer neural net. We keep the 4D expansion used by standard transformers and thus the same parameter count, but this leads to a ~2 FLOPS in this module compared to standard transformers.
Thus, though NRGPT is more parameter efficient in general, seems that parameter savings need to be exchanged for FLOPs. Note that we use pytorch's 's functional autograd interface to compute gradients through this network which slows down the wall-clock time evaluation compared to manual implementation of the update rule.
E LEARNING RATE AND PRECONDITIONER OF THE FORWARD PASS
RMSNorm:
9Ai %/8
AA
VD
(25)
D Σ, 233
V
E E[CAA]
JA
LayerNorm: 9Ai
+
(26)
v Varl
NA) +
where I are the components with 1....N and E 1...D.
Proof of Proposition 2.1: Energy Descent. Using chain rule
θE Ögs tc 0gB dxc
ÊA
Σ
B.C
θE 0gB
θE
Ögs dac "
Ōgc
(27)
B.C
Defining diag(r), the Jacobian of g becomes
Ög Ow
SAB (8jp yAJUAp),
T
(28)
rA
where
RMSNorm:
TA = || /VD,
DA = IA/TA
(29)
LayerNorm:
FA /arr [ +c,
Y4 (x MA)/TA-
(30)
15

---

Published as a conference paper at ICLR 2026
Since typically Therefore
10-6 6 is small constant, ||y = 1 for LayerNorm, and exactly 1 for RMSNor..
1
TPA. AA
PA PX
P PA + O(e)
(31)
A
where the approximate projection matrix PA is positive semi-definite, with DA/ |/alll being its sole null eigenvector. Plugging into equation 27
ÉA Σ T
DE dgB
PPn
.EB
(32)
UB
dgB
B
Note that when B > A, 0Ex/0gB = 0. Hence, equation 34 can be separated into A B and B <
TEE Ögn Ögs dzp B
BB TI [EATTPAT 0ga
OE
DEA Ōga
ÉA = Σ Tr BEA
(33)
In order for token A to converge as well, we only need to find conditions under which the second term in equation 33 converges. This is because the same conditions would then lead the second term in all En for B < A to converge. Then, by induction all En will eventually converge. Thus we want just the second term
1 Tt SA
P
OEA T
(34)
TA
Öga
Öga
satisfy AEAA O. solutions to this are
which can be achieved if the symmetric part of I'PAn" p.s.d.. Two simple
7 cr,
7 = M()
(35)
where (x) is an arbitrary p.s.d. matrix and >> 0. If we want 17 to be simple weights instead of an x dependent neural network, the solution is 77 = cl'.
Note that equation 34 does not restrict the anti-symmetric part of P Using n cT + B, the antisymmetric part satisfies Td PaT = -PPA B. Since PA is rank D for each A, the anti- symmetric part doesn't seem to have an z-independent solution. The fact that PA is different for each token severely restricts the form of 17 to get convergence. However, The boundedness of the energies EA due to boundedness 9.1 can tame the dynamics and lead more choices for n yielding convergence. For example, damped harmonic oscillator has dynamics in which the state oscillates damped fashion, but the energy is constantly decreasing.
E. EVOLUTION OF LOSS
We always absorb I into the weights of AT and FF, will assume I I from here on. To find ÉA need to chain rule with derivatives w.r.t. all tokens B < A.
ÉA Σ Onp EXOBgR④B
Σ 0gn EA EAPEpgp EB
Σ 2g ET PBLB 0g F PAnO,A FLA B<A
(36)
because of the coupling between different tokens, it is not clear whether ÉA is negative or not. Yet. due to the causality of the B < A interaction, token B is not affected by any future token. It follows that if token B dynamics converges, meaning dB - for all B < A, we have
(EA Öga
θEA PAn
if tB 0, vB< < A: EA
- -
(37)
Ōga
16

---

Published as a conference paper at ICLR 2026
where the last part assumes is invertible. Hence, token A will converge if PAn is p.s.d. Since PA a projection away from AA and Pan should be p.s.d. for all A, it follows that on the data manifold only be constant times identity. If the the data is on low-rank subspace, on the orthogonal subspace. More formally, let PI represent projection onto S| Span (x (VA]
be arbitrary
meaning 1 aI ||> Let Pi be the projection the subspace orthogonal all data AA meaning x/ PITA O, and so P| = The Mi
PinPi cPI for some c > 0, while Ml PimP1 is an arbitrary p.s.d. matrix with null space including Sii-
By induction, we can show that the 77 above leads convergence all tokens. For the first token
A = 1, there are no past tokens to convergence requires
(OE) Og
T
0E 0gi
ES =
PMn
<0
(38)
For the second token, when the first token converges, 21 0, resulting in the same condition for Ez. Assume 77 I and introduce the shorthand ΕA,B 0Ea/0gn, and the Mahalonobis norm || 2" Mz. For any A
EAAPAAA -1| llp - ||
ÉA = BA=EAAPASE EAAPAAA+ + Σ
EABBBBB
BCA
-1IiAA +
Σ
EABBPRãB
(39)
B<A
EABBPRBB
(40)
BCA
Since EA are bounded from below (shown next), AA < should eventually lead to convergence. For E1, there are no past tokens and Ej always, resulting in convergence.
F.2 BOUNDEDNESS OF THE ENERGY.
First, observe that EA EAT + EFF is bounded from below, given some assumptions about FF. For AT, since 9B = / | + B = B, we have
11 1 = +1+ UA
+ -1 I Ig 1 1 +1+ 2|
9 JBB J/ + 8- (JūB + J
(41) where ||||| is the spectral norm of J equal to the largest singular value of J. Therefore, using monotonicity of log and exp
(1 + || J.B
(1 B|)
EAT = log Σ
exp
explgA Jgl S log(A max/Amaxxeepppppp ig gel})
B<A
BCA
< log A max
(g (gA gB) S BeA
log (1 2|
(42)
and similarly for the lower bound resulting
(1 ||| < EAA" log A s (1 211111)\]
(43)
For FF, we need to assume a form for the energy. First, considering FFFF (g^) = -looolloWa (Wgx))l2 with activation being Lipschitz, in ReL.U GeLU. The Lipschitz condition means
[|o(z) - a (v) < L||x yll for some Lipschitz constant T. > setting y = 0 and æ n we get
(44)
|[a ga) a0)) < L||Wgal gall L | S L|[W]|2(1 [|3]])
(45)
17

---

Published as a conference paper at ICLR 2026
using triangle inequality ||a + bl| 1 ||||| + |[b]] with (W a00) and b = o00) we get
Ilor ga)ll _ |lo(W 9A) - ((0) + ||o < W| ga1l + [|o(0)]
- W [1 + [| + [0)]
(46)
and similarly, using the other side of triangle inequality ||a|| 2 ||a + bl| - ||a|| with a = o ga) and b (Wga)) + (0) we have
(lo(Wga)) 2 [o(())] - (Wo - -(0) 2 ||o(0)] - + |||I
(47)
therefore
(Olo + (1 + 181I)) S E - -
11
(48)
For more general FF, as long as they are MLP with Lipschitz activations, the normalized nature of gA should guarantee boundedness of EF and, consequently, E.
F EXPERIMENTS
F.1 ARCHITECTURE DETAILS
NRGPT serves as a drop-in replacement for standard GPT block, where tokens and positions are embedded using learnable embedding matrices following the precedent of GPT-2. Text is tokenized using the default OpenAl BPE tokenizer from GPT-2 (with just over 50k vocab tokens). For prediction, tokens are "unembedded" using a linear projection layer whose weights are shared by the input's embedding matrix.
We describe the predictions of NRGPT as sampling process obtained by minimizing the energy of each token, which serves as recurrent replacement of the complete forward pass through the transformer. This is easiest to explain for when descent on the energy EA of token index A (these energy trajectories are shown in Figure 2). Gradient descent (GD) provides fast, differentiable way to move initial points to locations which have lower energy (higher likelihood). Each new token starts from an initial position x = 0), and the forward pass does GID evolving interpreted as the log- transition probability. In all GPT-style models and therefore also in NRGPT, we choose the initial point x = to be x the embedding of the previous token, but tokens could be initialized to anything in theory.
1 where the forward pass is explicit gradient
point x.A (1 Ly) which has lower energy. In this case, FA can be
In the case where v * 1, the forward pass follows more complex dynamics than pure GD, but in general the model learns dynamics that convert XA-1 to XA. We refer to this process as "sampling" throughout the paper; however, we note that this not a strict sampling process for any value of 17.
F.2 EVALUATION METRICS
To assess the generation quality of our language models, we utilize several complementary metrics. We use Perplexity as a measure of model uncertainty, computed using a pretrained GPT-2 model to evaluate how well the generated text aligns with expected language patterns. Lower perplexity indicates more fluent and predictable text, with scores typically ranging from 10 (excellent) to 1000+ (poor quality). For lexical diversity, of unique unigrams and bigrams to total n-grams (or total words) in the generated text, respectively. These metries range from 0 to where higher values indicate greater vocabulary diversity and less repetitive generation. A Distinct-1 score near 0 suggests highly repetitive text, while scores above 0.8 indicate rich vocabulary usage. We utilize Average Pairwise Cosine Similarity using Sentence- BERT embeddings (Reimers and Gurevych, 2019) to measure semantic diversity within generated samples. This metric calculates the mean cosine similarity between all pairs of generated sentences, ranging from -1 to 1. Optimal values fall between 0.3 and 0.6, balancing semantic diversity with topical coherence. Values below 0.3 indicate excessive divergence with potentially incoherent topic-jumping between sentences, while values above 0.7 suggest repetitive redundant content with insufficient variation. The target range of 0.3-0.6 represents healthy diversity where generated sentences explore different aspects of topic while maintaining semantic relevance and coherent narrative flow.
utilize Distinct-1 and Distinct-2, which measure the ratio
SSpecifically, GPT-2's implementation by the nanoGPT repository
18

---

Published as a conference paper at ICLR 2026
Finally, We compute the Grammar Quality Score (GQS), composite metric that combines rule- based grammar error detection, spelling accuracy via spellchecker (Barrus, 2020), and readability assessment using Flesch-Kincaid Kincaid grade level (Flesch, 1948). GQS ranges from 0 (poor grammar) to (perfect grammar), weighting grammatical correctness (50%), spelling accuracy (30%), and readability (20%). The metric identifies errors across categories including subject-verb agreement, tense consistency, and punctuation, with severity-weighted scoring. For complete context and to understand what the ideal ranges for all of these metrics, see Table 3.
Table 3: Ideal ranges for generation quality metrics
Metric
Good Range
Interpretation
Perplexity Distinct-1 1
15-50
Lower is better (fluency)
0.6-0.9
Higher better (vocabulary diversity)
Distinct-2
0.8-0.95
Higher is better (bigram diversity) Higher is better (grammatical quality)
GQS Avg. Cosine Similarity
0.8-1.0
0.3-0.6
Moderate values best (semantic diversity)
F.3 LISTOPS
We perform experiment on nested math operations on lists of integers, which are version of ListOps (Nangia and Bowman, 2018). Our ListOps setting consists of three functions: maximum, median and sum modulo 20. Our inputs range from to 19.
F.4 SHAKESPEARE
As training data used the full Shakespeare training set, tokenized such that each character constitutes a single token. Models were evaluated across the same held out validation subset. Across all models, we used a context window of 256 tokens, dropout of 10%, the AdamW(3;=0k) 32=0.99), and 40k maximum update iterations using a minibatch size of 64. We varied model sizes by sweeping over embedding dimensions (32, 64, 128, 256, 380, 512, 768) and the number of attention heads (1, 2, 8). For the recurrent models, we additionally varied the number of layers across (3, 6, 8), though this does not affect the parameter count.
We found the recurrent models to be quite sensitive to choices of learning rate and learning rate schedules. Hence, we explored several different maximum learning rates (1e 3, 7.5e 4, 4. 4), schedules (cosine, exponential), and minimum learning rates (10x, 20 and 100x smaller than the max learning rate). LR warm-up was 100 updates for all experiments.
In Figure 4 we emphasize the best losses were able to achieve for each model size. In addition, we capture the model's sensitivity to hyperparams by showing the top 50% performing models across all hyperparameters.
F.4.1 BEST MODEL CONFIGURATIONS AND METRICS
Table 4 shows the best model configuration for baseline GPTs and our model NRGPT with the respective generation quality metrics. We see that NRGPT outperforms the baseline GPT, RGPT and RGP'T-parallel in terms of generation quality while it has only half of the nparams of GPT.
F.5
OPEN WEXTEXT
We perform experiments on natural language modeling using the OpenWehTee corpus, which is open-source recreation of GPT-2's WebText dataset (Radford et al., 2019). The datasct contains approximately 17GB of text with 9B tokens that came from million documents. We tokenize using byte-pair encoding (BPE) with a vocabulary size of 50,257 tokens. Our training sequences fixed-length contexts 1024 tokens. Table lists the hyperparameters and respective values/ranges used in experiments.
Table shows the best model configuration for baseline GPT and RGPT-parallel and our model NRGPT with the respective generation quality metrics. We see that the generation quality of NRGPT
19

---

Published as a conference paper at ICLR 2026
Table 4: Best model configurations and quality metries for Shakespeare. Note abbreviations: RGPT- parallel - RGPT-P, no of parameters - n_ param, grammar quality score + gqs. average pariwise cosine similarity + apes, distinct-1 d-1 and distinct-2 d-2.
Configuration
Metrics
Ir
min
layer
n_head
n_eibeed
p_params
perplexity
84s
ape
d-1
d-2
GPT
5e-3
Se-4
8
4
64
0.4M
294
0.913
0.198
0.751
0.978
RGPT
le:3
le-4
8
256
0.8M
476
0.896
0.164
0.794
0.995
RGPT-P
2o-3
20-4
128
0.2M
0.888
0.190
0.838
NRGPT H. FFI
3e-4
3e-5
6
2
512
2M
318
0.901
0.218
0.797
0.995
URGPT_H_PPE FF2W
1o-3
1o-4
128
0.2M
0,975
0.176
0.765
0.995
Table 5: Best Model Configurations and Quality Metrics (average of four generations) for OWT with similar no parameters. Note abbreviations: no parameters - n_param, best train loss + t_loss, best validation loss loss, grammar quality score gqs, average pariwise cosine similarity apes, distinct-1 + d-1 and distinct-2 + d-2.
Configuration
Metrics
Ir
min_Ir
laya
n. head
n. n_embeed
R_ puparss
[
v_loss
perplesity
1208
apps
d-2
GPT
Te-4
7e-5
12
12
768
124M
2.90
2.93
78
0.947
0.274
0.638
0.963
GPT Parallll
0.950
0.962
NRGPT FF2W
3e-5
3--6
1536
128M
3.29
3.30
105
0.946
0.306
0.650
0.960
very competitive with GPT and RGPT- parallel while it contains around 34M less parameters than GPT. Figure 5 shows example of generated text by GPT, RGPT-paralled and NRGPT.
We report the best OWT training configurations for the metrics reported in the main paper in Table 2. We additionally test models at comparable 125M parameter scale in Table 5.
F.5.1 MMLU
Perplexity and simple generation metrics are insufficient to understand the performance of language models on downstream tasks. Thus, we additionally characterize the equal-paramees models of Table 5 the MMLU dataset (Hendrycks et al., 2020) using five-shot generation in Table 6. Remarkably, we find that NRGPT outperforms both GPT and RecGPT at constant parameter count on the average MMLU score.
As a caveat, we note that transformers at the 125M scale generally perform poorly on these downstream tasks. For example, the original MMLU paper (Hendryeks et al., 2020) shows that GPT-3-small (2.7 billion params) and GPT-3-medium (6.7 billion params) (Brown et al., 2020) achieve 25% success rate the exact same random guessing. A large GPT-2 model (1.5 billion params) (Radford et al., 2018) achieves slightly better 32.4% accuracy, but this is still only 7% higher than random prediction. Our NRGPT model 125M parameters (an order of magnitude smaller than the MMLU results of GPT-2) achieves a 29.3% accuracy, which is the best performance per parameter count on this task that we are aware
We believe that our results in Table 6 should be taken with a grain of salt, since we are testing small models and comparing between low accuracies. However, we also believe that these hint at NRGPT's advantage at greater scales.
Table 6: MMLU performance comparison using five-shot generation. The models tested in this work are insufficient size and scale achieve good performance on all of MML.U, so we report results by subject. Best are bold.
Model
Astronomy
Science
CS
Economics
Medicine
Math
Global Facts
Average
NRGPT (128M)
32.1
30.0
30.5
29.2
19.0
29.3
RGPT (126M)
27.1
28.0
24.1
17.0
GPT (124M)
36.8
29.5
26.7
31.2
27.3
18.0
28.0
20

---

Published as a conference paper at ICLR 2026
Table 7: OWT Hyperparameters and range of values.
Hyperparameter
Used Values
batch_sire size
block_sizz size
1024
n layer
[3 9, 24]
n head
[1. 4, 6, 12, 16]
a_embed
[768. 1020, 1536]
learning rate,
[1e-3- 1e-5]
min Ir
[l<100 Ir]
betal
0.9
beta2
0.99
weight_decay
[1e-1, 1c-2]
gradient accumulation steps
eval interval
1,000
eval iters
200
iters
100000
warmup_iters
[100, 2000]
dropout
0.0
Table 8: Best Model Configurations and Quality Metrics for OWT. Note abbreviations: RGPT-parallel - RGPT-P, no of parameters - n_param, grammar quality score + gqs, average pariwise cosine similarity + apes, distinct-1 d-1 and distinct-2 + d-2.
Configuration
Metrics
Model
min Ir
n layer
n_head
n embed
params
perplexity
2qs
apcs
d-2
GPT
7c-4
76-5
12
12
768
124M
75
0.978
0.306
0.619
0.965
RGPT-P
6e-4
4e-4
12
12
768
99
0.976
0.336
0.615
0.975
NRGPT
le4
76-5
12
12
768
104
0.966
0.306
0.674
0.984
F.6
UNCONSTRAINED PROJECTION MATRIX
We wonder how much it is necesary constrain the inference rate matrix 77 such that NRGPT continues to exhibit convergence, as shown in Figure 6. To test this, we repeat the experimental setting of Figure 2, training 100-layer NRGPT model ListOps that achieves 100% total accuracy. We then take 64 tokens from validation set and minimize their energy. Remarkably, despite using 7) that is completely unconstrained (and therefore with no guarantee of energy minimization), observe that the energy of tokens eventually stop fluctuating and converge to stable state.
Though this observation is empirical and not guaranteed to be the case should we have trained model for much longer time, it is still an interesting observation that suggests that NRGPT's convergent properties may be encouraged by the objective function, freeing us from needing to constrain the inference rate matrix v).
21

---

Published as a conference paper at ICLR 2026
Figure 5: Best Generation Examples from GPT (left column), RGPT-paralle (middle column) and NRGPT (right column).
Total Energy
10-
-10
-50
Attention Energy
30-
20-
-10-
-20-
-ordard Energy
-00
10 20
30 35
45
50
55
65 70
75 85 95100
Timestep ("Layer")
Figure NRGPT's inference rate matrix 77 does need be constrained to induce convergent dynamics. Shown is a 100-layer NRGPT model that achieves 100% accuracy on ListOps. Shown are 64 tokens from validation set passed NRGPT, where token dynamics are shown to stabilize and converge without any constraint on 77.
22


Reviews:
[{"review_id": "US7Jar9DAl", "reviewer": "Reviewer_xLun", "summary": "This paper presents eNeRgy-GPT (NRGPT) a framework for autoregressive language modeling that converts a Generative Pre-trained Transformer (GPT) into an energy-based model. NRGPT inference is conceptualized as an exploration on an energy landscape, where each layer becomes a gradient descent step on an energy function comprising attention and feedforward components. The authors establish theoretical conditions under which this process guarantees energy descent.\nOn three datasets, nested math operations on lists of integers (ListOps), character-level modeling (Shakespeare), and standard language modeling (OpenWebText) NRGPT achieves comparable performance with GPT baselines across different parameter sizes. Notably, the model exhibits greater resistance to overfitting compared to standard transformers.", "questions": "- Why not comparing NRGPT with recurrent GPT and GPT at approximately similar number of parameters?\n- When comparing models in Open Web Text (Section 3.4), are you using the same tokenizer across all models?\n- How do GPT/Rec-GPT differ from NRGPT in terms of computational cost. How much does the need of the inference rate matrix affect FLOPs?\n\n\nComments:\n\nl114: This is also the premise Energy-Based Models (EBM)\n\nl121: the final point match real datapoints\n\nl322: TO evaluate the\n\nl323 ends abruptly", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors unify the well-known GPT architecture with Energy-based models, providing explicit energy functions for both attention and feedforward components. This theoretical contribution is novel and advances recent work exploring these connections.\n    \n- The paper is well-written and easy to follow.\n    \n- While the experiments on ListOps, Shakespeare, and Open Web Text cover a narrow range of tasks, they are nonetheless informative for drawing meaningful comparisons between models.", "weaknesses": "- While the theoretical connection between energy-based models and transformers is interesting, the paper does not clearly state why this direction is practically valuable or how it could lead to meaningful advances beyond conceptual unification.\n- While NRGPT achieves performance comparable to a recurrent GPT, this comparison has limited practical relevance. The meaningful benchmark remains the standard GPT architecture.\n- NRGPT requires weight sharing across layers (recurrent steps), which posits a constraint in terms of architectural flexibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nD60DdNUvM", "reviewer": "Reviewer_cL58", "summary": "The paper reformulates the GPT as an EBM. Instead of stacking multiple transformer layers, it uses a single recurrent module that performs iterative energy descent steps. Each forward pass updates the token representations by following the gradient of a total energy function that combines attention and feedforward energies, turning inference into motion along an energy landscape.\n\nThe authors derive energy formulations whose gradients reproduce the transformer’s attention and feedforward operations. They introduce a learnable inference rate matrix η(t), prove that energy monotonically decreases under LayerNorm or RMSNorm, and show asymptotic stability of the update dynamics.\n\nExperiments demonstrate that NRGPT achieves performance comparable to standard and recurrent GPTs on ListOps, Shakespeare, and OpenWebText benchmarks, while requiring fewer parameters and showing greater resistance to overfitting.\n\nOverall, the work provides a minimal and theoretically grounded link between GPT and EBM frameworks, showing that transformer inference can be interpreted as an explicit gradient-based energy minimization process without sacrificing model quality.", "questions": "1. Is the proposed energy formulation essentially unique, or could multiple energy functions induce the same transformer dynamics? Any intuition or simple example would help clarify interpretability.\n\n2. Is one-step energy descent meant to be exactly equivalent to a transformer layer, or a principled approximation? If approximate, what governs the approximation quality (e.g., β, normalization, η structure)?\n\n3. Beyond fixed iteration counts, have the authors explored adaptive criteria (energy change, gradient norm, or confidence signals) to decide when to stop the recurrent updates?\n\n4. Do energy trajectories correlate with uncertainty, generation quality, or failure cases? Could energy serve as a runtime diagnostic or control signal (e.g., confidence gating, early stopping)?\n\n5. Since the formulation emphasizes iterative refinement, can NRGPT naturally support test-time “extra thinking steps” (longer inference) without tuning? If so, are there tasks where this yields measurable gains?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "This paper stands out for its originality in redefining the GPT architecture through an energy-based modeling perspective. Instead of proposing a new model class, it reinterprets existing transformer mechanics as iterative energy minimization, offering a fresh theoretical lens rather than a purely architectural innovation.\n\nIts technical quality is solid: the authors construct explicit energy functions for both attention and feedforward components, establish convergence and stability guarantees, and validate the model on multiple benchmarks. The experiments are carefully designed and demonstrate that the energy-based formulation can match GPT performance while using fewer parameters.\n\nThe clarity of presentation is strong, with concise mathematical derivations and intuitive explanations that make the energy interpretation easy to follow.\n\nIn terms of significance, the work bridges a conceptual gap between transformers and energy-based models, opening a path toward more theoretically grounded, interpretable, and potentially more efficient generative models.", "weaknesses": "First, the experiments are limited to small and medium-scale models (≤90M parameters) and simple datasets like ListOps and Shakespeare, making it unclear whether NRGPT remains stable and efficient at larger LLM scales. Scaling experiments on multi-billion parameter models and broader benchmarks such as MMLU or GSM8K would make the conclusions more general.\n\nSecond, the theoretical guarantees rely on a learnable inference rate matrix η(t) and specific normalization (LayerNorm/RMSNorm), but the paper lacks ablations on their robustness and training feasibility. Systematic studies of different η parameterizations and normalization variants would clarify when the energy monotonicity and stability conditions hold in practice.\n\nThird, the paper does not analyze runtime efficiency or expressivity trade-offs. Because the model enforces weight sharing, it may have higher iteration costs or reduced fitting capacity, which should be quantified under equal FLOPs or wall-clock settings.\n\nFinally, the paper should discuss its relationship to *Energy-Based Transformers are Scalable Learners and Thinkers* (arXiv:2507.02092). While that work focuses on scalable, multimodal reasoning with “System 2” inference, NRGPT formalizes energy descent within standard GPT. A clear comparison of their goals, formulations, and implications would strengthen the paper’s positioning and highlight its distinct contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1EcLgxMXQw", "reviewer": "Reviewer_JUki", "summary": "This paper proposes an energy-based framework for next-token prediction by making key edits to the Transformer architecture to achieve a recurrent block. This model is learned via a differentiable sampling process where parameters are learned during sampling. The energy function mimics the Transformer via a parallel block with an attention-based component and a feed-forward based component, as in parallel transformer designs. Tokens are transformed into the next via exploring the energy landscape. The block is repeatedly applied to mimic layer depth in a recurrent fashion. Results on simple tasks like domain-specific language modeling and list ops show performance comparable to a layer-recurrent Transformer.", "questions": "1. How should the original: and energy: expressions be interpreted in equations 13 and 15?\n2. While it is clearly not the point of this work to outperform current language models given the exploratory and new nature of this work, I am still curious to know what are the run-time, memory, FLOP/compute use differences between this model and normal GPT? \n\nNotes:\n1. Cutoff on line 323, sentence not finished, “For”. \n2. Cutoff on line 353, “To reach”, sentence not finished\n3. Citations for Qwen, Llama, and RMSNorm are missing on line 245.", "limitations": "", "rating": 2, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. This work extends prior work on energy based modeling and the Transformer archtecture to the new and popular setting of next token prediction, advancing from prior work largely focusing on masked infilling based modeling. \n2. The experiments testing NR-GPT on ListOPs and Shakespeare data show competitive results with a recurrent version of a transformer model, indicating that this different learning paradigm has promise and can perform competitively with a modficiation on a dominant paradigm.", "weaknesses": "1. It is unclear how much novelty is presented in this work versus the immediate prior work of Energy Transformer. For example, the parallel block structure for EBM was proposed in the prior work, but not attributed directly in this work. Additionally, while proper attribution was given for the attention energy, this architecture and energy formulation are also directly from the prior work, constituting a major part of the current paper's claimed contributions. \n2. The motivation for this avenue of exploring EBMs and Transformers is lacking in the current work - while I believe that studying new avenues for language modeling are important, it is not clear from this work what precise benefit EBMs may give over current language modeling, even if they are \"not there yet.\" \n3. Key details on how sampling actually occurs in this model are not present. How does the arcitecture embed tokens into the network? And more importantly, how are samples actually decoded into tokens?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i6tEYXDczA", "reviewer": "Reviewer_d1Ma", "summary": "The paper introduces an interesting and novel approach to generative modelling that integrates energy-based methods with standard GPT like methods.", "questions": "The proposed method has similarities to diffusion models. Can it be used to enhance diffusion models (e,g, to generative images instead of applying it to LLMs, using a GAN discriminator to provide a score)?\n\nCan proposition 2.1 be strengthened to provide a weaker constraint on the required inference rate matrix? (i.e. are there other inference rate matrices where the energy decreases?) There is some discussion of this, but I am mainly wondering if there are proofs where the inference rate is something that is more useful for the problem at hand.\n\nNot necessarily a weakness, but the issue of asymptotic stability raises the question of short-term stability. Does this affect the operation and training of the network? How long does one have to wait to reach quasi-stability where things have settled down enough for reliable inference?\n\nWhat is it about the EBM approach that results in needing fewer parameters than equivalent GPT models (table 2)?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Work is novel and interesting.\nModifies standard GPT methods with energy-based methods.\nThe work complements previous studies suggesting that transformers perform GD during inference. Unlike past work, in the proposed model inference is explicitly a gradient-based dynamics, while still maintaining an architecture very similar to GPT.\n\nExtensive experiments show results comparable to standard GPT implementations but requiring fewer parameters.", "weaknesses": "Has a drawback similar to diffusion models in that inference is an iterative process and thus takes time and incurs a heavy computational burden. As the paper mentions, a T-step gradient descent can be mapped to a T-layer GPT transformer architecture, but the computational burden remains if the required number of steps T is large.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)