DS-LLM: Leveraging Dynamical Systems to Enhance Both Training and Inference of Large Language Models
Authors: Ruibing Song
Chuan Liu
Chunshu Wu
Ang Li
Dongfang Liu
Ying Nian Wu
Tong Geng
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Model;  nature-powered computing; dynamic physical system
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose an algorithm framework to bridge existing LLMs and powerful Dynamical-system-based machines to enhance both inference and training of LLMs.

Abstract: The training of large language models (LLMs) faces significant computational cost challenges, limiting their scalability toward artificial general intelligence (AGI) and broader adoption. With model sizes doubling approximately every 3.4 months and training costs escalating from 64 million USD for GPT-4 in 2020 to 191 million USD for Gemini Ultra in 2023, the economic burden has become unsustainable. While techniques such as quantization offer incremental improvements, they fail to address the fundamental computational bottleneck. In this work, we introduce DS-LLM, a novel framework that leverages dynamical system (DS)-based machines, which exploit Natural Annealing to rapidly converge to minimal energy states, yielding substantial efficiency gains. Unlike traditional methods, DS-LLM maps LLM components to optimization problems solvable via Hamiltonian configurations and utilizes continuous electric current flow in DS-machines for hardware-native gradient descent during training. We mathematically demonstrate the equivalence between conventional LLMs and DS-LLMs and present a method for transforming a trained LLM into a DS-LLM. Experimental evaluations across multiple model sizes demonstrate orders-of-magnitude improvements in speed and energy efficiency for both training and inference while maintaining consistent accuracy. Additionally, we provide an in-depth analysis of the challenges and potential solutions associated with this emerging computing paradigm, aiming to lay a solid foundation for future research.

Published as a conference paper at ICLR 2025
DS-LLM: LEVERAGING DYNAMICAL SYSTEMS TO ENHANCE BOTH TRAINING AND INFERENCE OF
LARGE LANGUAGE MODELS
Ruibing Song', Chuan Liu', Chunshu Wu', Ang Lie, Dongfang Liu", Yingnian Wu', Tony (Tong) Gengb
Department of Electrical and Computer Engineering. University of Rochester
Physical & Computational Sciences Directorate, Pacific Northwestern National Laboratory?
Department of Computer Engineering, Rochester Institute of Technology
Department of Statisties and Data Science, University of California, Los Angeles*
ABSTRACT
The training of large language models (LLMs) faces significant computational cost challenges, limiting their scalability toward artificial general intelligence (AGI) and broader adoption. With model sizes doubling approximately every 3.4 months and training costs escalating from $64 million for GPT-4 in 2020 $191 million for Gemini Ultra in 2023, the economic burden has become unsus- tainable. While techniques such as quantization offer incremental improvements. they fail to address the fundamental computational bottleneck. In this work, we introduce DS-LLM,, a novel framework that leverages dynamical system (DS)- based machines, which exploit Natural Annealing to rapidly converge to minimal energy states, yielding substantial efficiency gains. Unlike traditional methods, DS-LLM maps LLM components to optimization problems solvable via Hamilto- nian configurations and utilizes continuous electric current flow in DS-machines for hardware- mardware-nativ gradient descent during training. We mathematically demon- strate the equivalence between conventional LLMs and DS-LLMS and present a method for transforming trained L.M into DS-LLM. Experimental evalua- tions across multiple model sizes demonstrate orders-of-magnitude improvements in speed and energy efficiency for both training and inference while maintaining consistent accuracy. Additionally, we provide an in-depth analysis of the chal- lenges and potential solutions associated with this emerging computing paradigm, aiming to lay solid foundation for future research.
INTRODUCTION
Large language models (LLMs) are propelling rapid advancements in AI, with model sizes doubling approximately every 3.4 months. This exponential growth necessitates unprecedented computa- tional resources for both training and inference, with training costs soaring from S64 million for GPT-4 in 2020 to $191 million for Gemini Ultra in 2023. As result, a significant portion of cloud and HPC resources is now dedicated to LLMs. raising socictal, environmental, and energy concerns. Meanwhile, Moore's Law is losing momentum, exacerbating the situation. While optimizing tra- ditional computing methods remains essential, the demand for new, energy-efficient architectures is growing. The challenge is to sustainably scale Al models, particularly LLMs, without incurring prohibitive costs.
Several alternative computing paradigms have emerged, including quantum computing (Kerenidis al. 2024), optical computing (Anderson al.) 2004), and computing-in-menon (Tu al. 2023). While promising, these technologies face significant technical barriers and require further develop- ment. In the near term, is it feasible to rely on mature IMOO--ased technology to accelerate LLM training from 10 million hours to 10,000 hours while reducing energy consumption from 20 tera- joules to 200 megajoules?
Recent advances in dynamical system-basee (DS) machines provide compelling solution. DS- machines incorporate an electrodynamics-urn based model that naturally converges to the minimum energy state by following the physical dynamics of electrons. By correctly setting the initial state,

---

Published as a conference paper at ICLR 2025
DS-LLM: Mapping LLMs
DS
Enhance LLM Training&Inffrence via Natural Annealing
Training cost /++20T
Training
Original LLMs
GPUs
Add, Sub, Mult operations...
Back Propagation
Expensive Inférence
GP1/2, 6PM2MM OM17.38...
CUDM cores
Sec 3.2-3.3
Training Sec 3.4
DS machine
Electronic-
Natural Annealing
speedup
Map to Hamiltonian
Current-Loss
Energy Landscape
Energy Save
Figure 1: Overview of DS-LLM.
boundary conditions, and final state, various problems be embedded into this "Natural Anneal- ing" process. As no external energy is theoretically required during such a process, DS-machines are extremely energy efficient and have already demonstrated remarkable potential in some real-world applications. For example, solving complex optimization problems like MAX-CUT (Hamerly et al., 2019), working on graph learning problems including traffic predictions, air quality, taxi demand, and pandemic progression (Wu et al., 2024]. Given their CMOS CMOS-compatibe compatible analog implementa- tion, DS-machines operate at room temperature, achieving remarkable efficiency gains. Previous research has demonstrated over 1, x000 speedup and 100, x00 energy reduction for graph learn- ing tasks while improving accuracy compared to state-of-the-ee Graph Neural Networks (GNNs) running commercial GPUs.
Can DS- machines be leveraged to accelerate LLMs? Unfortunately, no prior work has successfully mapped existing deep learning models like LLMs onto DS-machines. Previous approaches (Wu et al. 2024) have ignored the architectural structure of neural networks, instead directly fitting data the energy landscape of DS-machines. However, for complex tasks such as natural language processing (NLP), directly modeling the energy landscape of a DS system would be prohibitively challenging. Instead, could we re-purpose well-researched LLM architectures guide the energy landscape formation, effectively enabling DS-machines to execute LLM computations through Nat- ural Annealing?
To address this challenge, we introduce Dynamical System-based Large Language Model (DS- LLM),, the first algorithmic framework that bridges LLMS and DS-machines. Our approach con- structs the energy landscape of DS-machines based on reference LLM, ensuring they can accu- rately reproduce LLM outputs while leveraging Natural Annealing for inference. Additionally, we propose
(ECL) training method, which exploits the continuously evolving electric currents in DS machines as hardware-nativ gradients to optimize model parame- ters dynamically. We mathematically prove the equivalence of DS-LLM and traditional LLMs for both inference (Sections 3.2-3.3) and training (Section 3.4), and we empirically validate our ap- proach using models ranging from GPT-2-1 1PTM-2-124M LLaMA-2-7B. Our experimental results demon- strate orders-of- -aggnnitude speedup and energy reduction while preserving accuracy.
This work represents early-stage exploration into leveraging dynamical systems for efficient LLM computation. Our analysis is supported by theoretical proofs and empirical simulations conducted using standard commercial analog hardware simulation flows. While further research is needed to assess real-world performance using fabricated DS hardware, our results highlight significant potential. We hope this work inspires the community to further investigate DS DS-based architectures as promising avenue for sustainable LLM scaling.
The main contributions of this work are summarized as follows:
We propose DS-LLM, the first framework that unlocks the extraordinary inherent computational power of dynamical systems to revolutionize LLM computational efficiency.
We introduce online continuous training method that enables DS-Machine perform instant on-device lifelong training, extending its remarkable computational efficiency from Al test to training.
Experimental results across models from GPT-2-124M to LLaMA-2-7B on five downstream datasets show that, average, DS-Machine achieves a x.3 102 speedup for training and 2.4 102 for inference, along with a reduction in energy consumption of 2.3 x 10 during training and x.4x 10 during inference.
2

---

Published as a conference paper at ICLR 2025
2 BACKGROUND AND PRELIMINARY KNOWLEDGE
2.1 SRAAMICA-SYSSTE--BBEE MACHINE
(1) Hamiltonian and Natural Annealing: DS m5-ciness operate based on an energy function called the Hamiltonian, a key concept in physics that describes the total energy of system. The Hamil- tonian of the backbone DS-machine (Wu et al. (2024) is augmented from the Ising model which is statistical physics model widely used in the modeling of interacting spins. The Hamiltonian defined as:
N
N Jiyoia + Σ Noo 0, E
-
where system spins, J are coupling parameters representing the correlations among spins, and h are spins' self-reaction intensity to external influence. While retaining the strengths of the Ising model, this new Hamiltonian lifts its binary constraint to support real values, achieving high performance on graph learning problems.
The computing power of DS-machines stems from the process of Natural Annealing, inherent characteristic of dynamical systems. In systems such as interacting spin models, the Hamiltonian naturally decreases due to spin interactions. From a physics perspective, this occurs because spins tend to settle into lower energy states, guiding the system toward optimal solutions. To harness Natural Annealing, the parameters and h are programmed based on the target problem, shaping the Hamiltonian's ss energy landscape to align the desired outcomes with its minimum states. As the programmed DS-machine initiates Natural Annealing from random initial conditions, the system rapidly converges to an energy minimum, with the spin dynamics stabilizing at the target results.
(2) DS-machine Hardware: Since programming interacting spins can be prohibitively expensive, the backbone DS-machine (Wu et al.) (024) is built on electronic dynamical systems, which are implemented using current CMOS technology.
D6 mschise Hundware
As Fig. 2 shows, corresponding to equation [ each spin o, is implemented as a nano-scale capacitor within a node unit (Na), with its voltage representing spin value. Each ca- pacitor is coupled with a programmable resistor, forming a feedback loop within the node unit that serves the self- reaction parameters h. Capacitors from different node units (N,&N,) are structurally connected by a programmable re- sistor in coupling unit CUyy to perform the coupling parame- ters J. Natural Annealing this system is driven by voltage imbalances across capacitors, which guide the natural movement of electrons toward equilibrium, propelling the system to evolve toward energy minima. The convergence of the Natural Annealing, in other words, the spontaneous energy decrease of the Hamiltonian with time ddt S 0) can be theoretically guaranteed by Lyapunov stability analysis (Blaquiere) 2012). As Fig. Shoows, for each spin o, its electrodynamics behavior is designed as:
Coloss Pre www.greennnggg...
Figure 2: Backbone DS-machine.
do: Juja, h4o,
1 8H
dt
2 do
Then, following the chain rule we have:
dtl
07 do: do: dt
- Σ 0H1 )2 < 0o:
Σ
(3)
dt
As the equations show, the system's 's electrodynamics inherently drive the Hamiltonian toward a local minimum. To escape local minima and achieve better solutions, several techniques can be employed such as spin- piin-flippng and noise injection (Afoakwa et al. (021).
2.2 LARGE LANGUAGE MODELS
In this paper, LLMs refer to pre-trained Transformer models such as GPT, Llama, Gemini, and Claude, which are among the most popular models today. Since the Transformer model was intro- duced in (Vaswani 2017), it has achieved remarkable success across various domains, including nat- ural language processing (NLP), computer vision (Wang et al.) 2023), and audio processing (Ghosal et al. 2023). While modern LLMs are evolving towards multi-modal capabilities that integrate
3

---

Published as a conference paper at ICLR 2025
diverse types of data, this study focuses the classic Transformer architecture early-stage exploration.
A classic Transformer consists of both an encoder and decoder, whereas mainstream commercial models like GPT are decoder-only. Despite this distinction, the key components of both encoder and decoder similar: multi-head self-attention, feed-forward Multilayer Perceptrons (MLPs), linear projection layers, and layer normalization. This work primarily investigates decoder-based models, such GPT-2 and OPT, while maintaining adaptability for other Transformer variants.
Although matrix multiplication remains fundamental computational task in LLMs as in most neural networks, the primary computational bottleneck in L.LMs is the attention layer. The multi-heal self- attention mechanism, which significantly enhances LLM performance, imposes an exceptionally high computational burden. Considering each token x; as a row vector, the break down of multi- head self-attention of each head is:
= w, h; =2,WK, = wv
(4)
f(g;, = exp( Tik3
(5)
vd
A, = Σ [(1 kiy)03 3=1 ΣØ s(1
N
(6)
where Wo. WK, and Wy are learned weight matrices, and qi k; 82 are the row vectors representing
the query, key, and value for token 24, respectively. Function f(g;, ky)) measures the similarity between query and key. Consequently, the computation in the attention layer involves weighted sum, matrix multiplications, and exponential function. Based on the above decomposition, the next chapter will demonstrate how the Transformer model can be implemented on DS machines.
DS-LLM: MAPPING LLM ONTO DS-MACHINE
3.1 OVERVIEW
In this chapter, we provide detailed, step-by-step guide leveraging DS-machines enhance both the inference and training of LLMs. As illustrated in Fig. [ traditional transformer models are executed on digital GPUs using CUDA and tensor cores, whereas DS-LLM maps the origi- nal model onto the energy landscape of DS-machines, utilizing Natural Annealing for the forward pass. Furthermore, while conventional transformers rely on offline training via backpropagation, training method to enable rapid back-
DS-LLM introduces an online Catrtri-CCurret-LLossssss. propagation. By harnessing the immense computing power of DS-machines, DS-LLM accelerates both inference and training, offering fundamentally more efficient approach to LLM optimization.
3.2 MAPPING METHOD
The mapping of existing models onto DS-machines can follow two distinct approaches: one at the Hamiltonian level and the other at the electrodynamics level. To illustrate these approaches, we begin with single linear layer. Consider the input matrix X E Raxd.., the output matrix Y E Raxd.ut, and the weight matrix W E RRde along with an optional bias vector b e Rdo.. Since the bias term b can be absorbed into the matrix multiplication operation, we omit it for simplicity. Consequently, the transformation performed by this layer can be expressed as a single matrix multiplication Y XWT.
On the Hamiltonian level, to leverage the Natural Annealing process on DS-machines, we must shape the energy landscape that its minimum energy state corresponds to the desired output. To achieve this, we define a target function F that minimizes the squared Frobenius norm of the difference (or Euclidean distance) between the output state of the DS-machine, Yos, and the desired output, Xps WT, This forms the following minimization problem:
= \Ds Xps XpsW 11:
-((va - Σ wiy
where w. I, and y represent elements of the matrices W. Xps, and Yos, respectively, and i.j. and l correspond to the dimensions n,din, and dous- The target function F reaches its minimum, Fmin = O, when Yos XpsW.F Since the absolute value of Fmin is not crucial, and E, WifTst remains constant during inference, simplify the target function as:
4

---

Published as a conference paper at ICLR 2025
Linwar Layer [Matrix Multipiccatio)
*) Query-KKy
b) Exparential approzimation
Weiihtted-SSm
CUs
y1
Yi
to
to by
ClUS
to
to hi
Wj
kg
(q.kk)
cus
Figure Implementation of key operations.
N
= Σ (2.1)
vu -
(ua (2.5)
wzzz))
(8)
After transforming the linear layer's computation into a minimization problem, we program the DS- machine by aligning its Hamiltonian with this simplified target function, F. As shown in equation F is a special case of the Hamiltonian described in equation | In this case, the spins o are divided into two groups: Fijz representing the input, and yit, representing the output. The self-reaction parameters h2 are set to for Yır, while the coupling parameters J are assigned 20ni between cor- responding spins ztt and yu. with all other elements set to O. The value of h4 for zij does not affect the convergence in equation because the input Taj is fixed, resulting in dæ.z/ /tt = 0.
This configuration maps the target function F to the Hamiltonian of the DS machinen enabling the computation of this layer via Natural Auncaling process. During this process, the Hamiltonian will continuously decrease until it reaches a minimum (which is also the minimum of F), where Yos X naturally emerges.
On the electrodynamics level, we can arrive at a similar conclusion. As seen in equation 2 the electrodynamics behavior is governed by the coupling parameters J and self-reaction parameters h. According to Lyapunov stability analysis (Blaquiere) 2012), all spins should stabilize at a specific value when the system reaches stable point, i.c., a local minimum. Hence, the electrodynamics of all spins must satisfy a boundary condition where do / dt = O. Dividing the spins into two groups, as before before-input x and output y along with the boundary condition, we arrive at:
Jaj h4
(9) In this scenario, the spin electrodynamics exhibit solvable stable point, enabling us to directly program and map the desired matrix multiplication results onto the spin dynamics. Notably, this leads to the same mapping setup as derived from the Hamiltonian analysis.
y/
is a fortunate coincidence that the backbone DS machine is inherently well-suited for matrix multiplication, a fundamental computational operation in LLMs. In the next subsection, we extend this mapping approach to more general functions, including other key operations L.MMs, where slight augmentations to the backbone DS-machine are required for full support.
3.3 TRANSFORMER IMPLEMENTATION
Recall the decomposition of the self-attention layer. Aside from the linear projection layer in equa- tion which has already been implemented, three key operations remain the attention mecha- nism: (a) Query-Key matrix multiplication, (b) the exponential function, and (c) the weighted-sum operation. Operations (a) and (b) are defined in equation while (c) given in equation (
Fig. 3 illustrates the implementation of these three key operations, with detailed explana- tions provided below. For reference, the previously introduced implementation of the linear responieeeoe for matrix multiplication between weights and features- is also included in the figure.
a) Query-Key Matrix Multiplication: Unlike the typical multiplication in linear layer, this op- eration occurs between two feature matrices generated online, rather than between features and weights. In the linear layer analysis, we assumed the weight matrix W is obtained offline and loaded onto programmable resistors in the coupling units. Fortunately, these resistors in the back- bone DS-machine are implemented using transistors, and programming them is accomplished by adjusting the voltage on their control ports a common technique in CMOS design. Mcanwhile, the spins in DS-machines represented by capacitors, where the voltage corresponds to the spin
5

---

Published as a conference paper at ICLR 2025
value. Thus, map the feature matrix K onto the programmable resistors by connecting the output voltage of the spins to the control ports of the resistors with necessary scaling circuits.
b) Exponential approximation: The exponential function is computationally expensive and quires a high-order Taylor expansion for approximation. Based on a pre-experiment, we explore the trade-off between model accuracy and hardware cost and select 3rd-order Taylor expansion as approximation. Details of this trade-off can be found Appendix.
erp(1, Tayloes + x + 1/2x² 2 + 1/6x.
(10)
As Fig. B shows, both the second and third order terms are implemented in the same way as the Query- -ey Matrix Multiplication.
Weighted-Sumn Similar to the matrix multiplication, we can build the target function:
F Σ (y: E, E. - Σ, flg,, kiy) - Σ, flg, ky) )²
(11)
Ej flg
Ej f(q,, kj)
Since the probability of) [ai k;)) keeping zero is negligible in evolving dynamical system, we multiply it on equation 11 and reduce the constant terms:
N f= - Σ -= ky) + Σ (9
(12)
In this setup, we program f(a; ky)) to Jig like in the Query-Key Matrix Multiplication. Notice that 2 Ej f(g, k2) can be regarded as matrix multiplication operation between f(g;kj) and an all-ones vector, which can be mapped to DS -S-machine as a simplified linear layer. The results are connected to the programmable resistors in node units, where h; 1 Ej f(g,,k)) for yıs and set h; 0 for Vi. With this setup, the Weighted-Sum operation in equation 6 can also be mapped to the Hamiltonian described in equation
Now have most of the essential components of a transformer model. The other operations which have relatively lower computing demanding like LayerNorm and activation are handled by auxiliary functional units. Details of other operations how to map the entire transformer model onto a DS-machine. Assuming the original model consists of multiple layers, each decomposed into operations, its function can be expressed as:
be found in the appendix. Next, we demonstrate
y ((P f(P-1) ....o f(2) (2)
(13)
Since we have confirmed that the mapping method works for each computational component, we can now construct the general target function follows:
(P) 1=)=-1))=- (x++) (()
(14)
Here, we combine the mapping of all individual target functions by using the output of each lower layer as the input to the higher layer, where x is the initial input and -P-1 is the fi- nal output. It's important to note that when combining them, the influence between spins is unidirectional from lower to higher layers-ie.., oF(++) /Ox(p) 0. Therefore, the Natural Annealing process converges each operation, ensuring that ((P) reaches its minimum when (P++)) ((P) ((P-1) o... o f
In this setup, the entire transformer- based model can be mapped onto DS-machines. With global natural annealing, the model achieves the desired output when the system reaches its global energy minimum.
3.4 ONLINE TRAINING WITH
After accelerating the forward pass of LLMs using Nat- ural Annealing, we further enhance their training. The core idea is that well-trained DS-mace machine should reach its energy minimum when its output spins align with the our
l1
J
I 1-
E1p
ground truth from the training data. Building electrodynamics analysis of the mapping method, we in- troduce Carrri--Crrent-LLoc Loss-based training method,
In
-
cipecterr
sestel
Figure 4: The feedback path of Electric- Current-Loss based training.

---

Published as a conference paper at ICLR 2025
which leverages the physical currents in the DS-machine to enable rapid online training.
As shown in Fig. 4 when Natural Annealing converges, for each output spin yı, the total incoming electric current i - Σÿ Jejır the node unit must balance the internal current f hey flowing through the resistor within the same node, ensuring that the capacitor voltage (representing the spin values) remains stable (dys/dt = 0). Referring to equation | if we map the ground truth output yi onto the output spins and fix their values, can define the internal current reference 1 h.Ght
The difference between the incoming current 1i and the reference current I forms a new loss function, expressed as LEC = Σ, (Ti - ? This difference is equal to the current through the sampling resistor r, H. = Pn II. We then update Jis (h, = is constant) using gradient descent:
θL θL 0iL 27
AJ
(15)
JJ..
aIL DJú
As depicted in Fig. the multiplication of the loss current TL by the spin values zy ji can be implemented at the circuit level. The resulting current is then fed back to the control port of the programmable resistors in the coupling units (CUs) and nodes. Consequently, the parameters are updated as Jjj - Jıj AJ At by integrating the result current on the control capacitor of the programmable resistors over a time interval At.
We propose this training method, named Electric-Current- Current-Loss (ECL) online training, which aims minimize the loss function LEc. Currently, this method only supports single-layer DS- machines, where a ground truth ý is available. In this early-stage research, we leverage conventional back- propagation training to provide a baseline on how to combine ECL with backpropagation to enable efficient LLM training DS DS-machines.
In the training of most LLMs, the loss in the output layer is typically computed using softmax with cross-entropy, yielding 0L/Oz; y4 Di. where represents the input logits. Referring to equation and equation [4, after the DS-machine completes the Natural Annealing process, the output spin value yi should converge to the desired computational result f (ax()). If we map the ground truth vi to the output spin of the final layer, we obtain:
OL
T
T
V Us
=
(16)
Dz,
hi
Thus, using the ECL training method, the gradi- ents of the logits can be represented as an elec- tric current. As shown Fig. [5] the gradients in other layers computed based the ac- tivations and the electric current loss from the subsequent layer. Since the gradient of ma- trix multiplication operation is itself a matrix multiplication, this computation can also be ef- ficiently mapped onto DS-machines. For cer- tain non-polynomial operations. additional aux- iliary circuits handle the gradient calculation.
NP)
P-1)
H1 1
Actiration
Autheotinn
AP)
Saftmas& CressEntropy
CT's
CU's
Figure 5: ECL-based backpropagation.
Moreover, because the forward mapping process is directional, the connections and computational units (CUs) from later layers to earlier layers remain unused during inference. We exploit these idle CUs for gradient computation without incurring additional cost. The gradients are then integrated on the controlling capacitors, updating the parameters accordingly. In this way, we seamlessly combine ECL with backpropagation, enabling an online training method that eliminates output readout while leveraging the fast forward computation of DS -Scacciis...
EVALUATION
4.1 MODEL AND TRAINING SETUP
Models, Tasks and Datasets: We apply DS-LLM several open-source models: GPT-2 (base and medium) (Radford et al., 2019), OPT-1.3B and OPT-2.7B (Zhang et al., 2022). and Llama2- 7B Touvron et al. 2023). For all models, we fine-tune and evaluate on five datasets from the GLUE
7

---

Published as a conference paper at ICLR 2025
benchmark (Wang et al.; 2019): SST-2 (Socher al. 2013) for Single- Tasks, MRPC
(Dolan & Brockett 2005) and QQP (DataCanary et al. 2017) for Paraphrase Tasks, QNLI Ra-
jpurkar et al. 2016 and RTE (Dgan a 2006) for In Inference Tasks. Additionally, pre-tran
GPI--2mmedim from scratch on OWT (Gokaslan & Cohen 2019).
The fine-tuning on the GLUE benchmark was conducted on 4 Nvidia A100 40GB GPUs The global batch size was set to 32 for GPT-2 (124M), 16 for GPT-2-medium (355M) and OPT-1.3B, and 8 for OPT-2.7B and Llama2-7B. We fine-tuned all models for epochs on QQP and 3 epochs on the other datasets. The optimizer used was Adam W with an initial learning rate of 2e-5, and all other parameters were kept as default, as provided by the Hugging Face Transformers library. For pre-training GPT-2-medium, we utilized 80 Nvidia A100 40GB GPUs with global batch size of 480 and trained the model for 120,000 iterations. The optimizer was Adam with 6e-4 initial learning rate, and other parameters were also kept at default settings. The inference experiments are using the same global batch size as in the fine-tuning experiments. For DS- -S-Mach evaluation, the Natural Annealing process is evaluated with standard Finite Element Analysis (FEA) software emulator. Additionally, the power consumption of DS-Machine is generated using the Cadence Mixed-Signa Design Environment with 45 nm CMOS technology.
Experiments Setup:
4.2 ACCURACY COMPARISON BETWEEN DS-LLM AND ORIGINAL LLMS
In order to verify the accuracy loss of the proposed mapping method (DS-LLM) and training method (ECL), fine-tune and evaluate the models on selected datasets compare the accuracy across three model types: a) the original LLMs, trained offiine and inferred on GPUs; b) DS-LLM models, trained offline on GPUs but inferred DS-machines; and c) DS-LLM-ECL models, trained online and inferred on DS DS-machines. As shown in Table | the accuracy of DS-LLM models is comparable to that of the original LLMs, with some even outperforming them, demonstrating that the approx- imation loss during mapping is minimal and the mapping method is effective. The DS-LLM-ECI. models also maintains good accuracy, validating the feasibility of the ECL online training method.
Table 1: Accuracy comparison (in Accuracy (%)): the higher the better.
Task
Paraphrase Tasks
Inference Tasks
Single-Sentence Tasks
Dataset
MRPC
QQP
RTE
QNLI
SST2
GPT2
75.00
88.43
63.18
88.03
91.63
GPT2-DS
76.72
89.04
63.54
88.28
91.29
GPT2-DS-ECL
76.91
89.24
63.11
87.94
90.82
GPT2-M
79.66
90.57
68.59
91.05
93.58
GPT2-M-DS
79.17
90.55
70.40
90.33
93.35
GPT2-M-DS-ECI
79.31
90.09
70.41
89.73
93.06
OPTL3B
84.07
90.94
77.26
91.09
92.43
OPTL-3-DS
87.01
88.48
78.70
91.41
92,20
OPTL.38-D--EC
86.57
88.59
78.05
91.45
91.81
OPT2.7B
86.52
91.03
82.33
93.15
94.08
OP
86.54
90,98
82,48
93,27
94.04
OPT2.7B-DS-ECI
86.74
90.67
82.44
93.41
94.25
Llama2-7B
90.01
91.10
88.45
95.75
96.58
Lam-a2-BB-DS
89.47
90.95
88.70
95.69
96.32
Llama2-7B-DS-ECL
89.56
91.08
88.57
95.73
95.79
Finetune of GPT2 on CgP
Finetune of GPT2-M on COP
Finetune of OPT1.38 on DDP
Pretrein of GPT2-M on OWT GIT2 G-72---05
GP72
G972
OPT1 38 OPT1.115-C oS
GP72-05
2.05
GP12.9005
fecrotorn
Figure& 6: Visualization of training trajectories.
The fine-tuning trajectories the original LLMs DS-LLM, and DS-LLM-ECL models are visu- alized in Fig. Additionally, we present the training curves of the original LLMs and DS-LLM during GPT-2 Medium pretraining; unfortunately, we are unable to pretrain DS-LLM-ECL. models due to computational limitations. Notably. both DS-LLM and DS-LLM-ECL exhibit convergence curves that closely match those of the original models, demonstrating that the mapping and online training methods effectively replicate the performance of traditional LLMSs on DS- DS-machines.
8

---

Published as a conference paper at ICLR 2025
Table 2: Performance comparison between GPU and DS-machines.
Training
Inference
Throughput
Energy Efficiency
Time to First Token
Token Generation Rate
Energyy Efficiency
Metric
(loken/s)
(wken/KKWh)
(8)
(loken/s)
(luken/KWh)
GPT2
1.37+*0
6.19E#07
6.46E-05
1.55E#04
1.39E+08
GPT2-DS
4,70E006
8,93E+12
1.20E-06
8.33E+05
1.11E+12
gp12-m
3.24E403
1,466407
1.80F.04
5.55E+03
5.18E+07
gpt2-m-DS
3.27E406
4,36E412
2.48E-06
4,03E+05
5,38E+11 II
OPT1,3B
1.20E+03
5,41E+06
2.76E-04
3.62E+03
3,41E+07
OPTL3EDDS
3.08F+06
9.56F+11
2.58F:06
3.92F:405
1,22F+11
OPT2.7B
3.97E+02
1,78E+06
8.12E-04
1.23E+03
1.13E+07
OPT277B-DS
2.33E:+06
4,37E+11
3,41E-06
2,93E:+05
5,50E+10
Llama2-7B
1.34E+02
6,05E+05
2.51E-03
3.98E+02
3.82E+06
Llama2-BB-DS
2.244:+06
1.56+111
3.57E-06
2. 2.808+00
1,95++00
Thrsppput (Tekre)))
1847
Scaling Training
IK-13
Tolon Goerztton Note
18:6
Scaling
Performance
18-136
Batrag Bilicenn
Earrgo Eltree
Eodinatedd
(Tikmn))
Estimated
1E+4
IE44
12:05
18+09
1E-10
12:11
122
13
1X 409
18-10 Madel Size
18
12-12
Model Stze
Trabing sughest Delomak
Mati skip Trilklag Throogpppn (hione
Takoe Georration Rate (alleei))
Matii diip Tabee Grare atie Ratr (hebne)))
Figure 7: Scaling curves of single-chip and multi-chip solutions.
4.3 PERFORMANCE COMPARISON
Table. 2 compares the average training and inference performance on all dataset and estimated en- ergy consumption between the implementation on GPUs and our solution. The power consumption of the 4 A100 GPUs estimated at 800 watts, and the inference evaluation on GPUs is based on Kown et al. 2023) for fair comparison.
As shown in Fig 7 we visualized the scaling trends of DS-LLM to show how the performance change with model size, illustrating linear reductions in token rate and energy efficiency model size increases. The mod- els larger than 7B are estimated by linearly extrapolation from 7B. There two scaling solutions shown in the figure, including single-chip solution which is used in Ta- ald multi-chip solution which suggest handle larger models. The motivation of a multi-chip solution comes from the considerations on hardware implementation. As we demonstrate Table. 3 the chip area and power consumption increase quickly with model size under single-chip solution and the area needed for a 66B model even larger than commercial GPUs. Building extremely big single chip will increase the practical risks due to physical limitations such process yield and ther- mal management. Though the industry has built some wafer-scale extremely large chips (Hu et al.] 2024), multi-chip scaling out solution is a more mature and widely used solution in AI accelerator industry. However, while multi-chip solution involves fixed chip sizes, additional communication overhead will reduce the system efficiency shown in the figure. The performance of actual multi- chip systems will also be influenced by the implementation of memory hierarchy, communication bandwidth, and many system level trade-offs.
Table 3: Hardware cost with single-chip scaling.
Model size
Area (mm-)
Power (W)
GPT2-DS
19,8
2.7
gpt2-m-DS
19.8
2.7
OPTL-8-DS
OPT2.7B-DS
131.1
19.2
Llama2-7B-DS
332.4
51.6
OPT30B-DS (est.)
907.5
138.2
OPT66B-DS (est.)
1448.6
226.7
For a more comprehensive evaluation on inference, we also compared our DS-LLM with some low- precision implementations for edge devices, including low-precision CPen et al. 2023) and current SOTA accelerator for LLM,, Cambricon-LLLM Yu et al., 2024). As shown in Table. DS- I.IM outperforms the references both token rate and energy efficiency. Results of Cambricon- LLM are sourced from the original paper.
Table 4: Comparison Llama2-7B with low owwprrccssion CPU and edge devices.
Solutions
Token Generation Rate (tokens/s) Energy Efficiency (lokens/K Wh)
Low Precision CPUs et al. 2023
45.4
1.63E+6
Cambricon-LL.N Yu et 2024
3.55
3.60E+6
DS-LLM (this work)
3,03E+5
4,045+10
We emphasize that our assessment follows standard analog simulation flows, while acknowledging that it may not fully capture the complexities of real physical chip measurements. Nonetheless,

---

Published as a conference paper at ICLR 2025
believe the demonstrated orders-of-magee -aagnnitudd improvements in speed and energy efficiency high- light the potential of DS-LLM as a promising solution for overcoming bottlenecks in LLM compu- tation.
DISCUSSION
While DS-machines demonstrate significant theoretical potential as an emerging solution, this sec- tion explores the practical challenges they may face, aiming to lay solid foundation for future research and development in this promising field. We address key aspects such as scalability, model flexibility, physical practicality, system integration feasibility, robustness, and precision constraints, supported by preliminary verification. Due to space limitations, some discussions are included in the Appendix.
Scalability: While prior work like NP-GL was designed for small graphs with fewer than 1,000 nodes, the capacity of DS-machines First, DS-machines have demonstrated linear complexity with respect to the number of nodes (Song al. 2024), making them inherently scalable with increased chip area. For context, NP-GL oc- cupies only about 5 mmn? whereas modern GPUs like the H100 have a die size of 814 mm3 and wafer-scale chips-such as those with up 46,255 mm2-are are emerging (Hu et al.) 2024). Based on linear complexity, single-chip solution could theoretically support millions of nodes within single -m-machene. Second, for even larger models or faster training. multi-chip approaches offer a viable path forward. Existing research has explored multi-chip solutions for DS-machines (Sharma jtt al. 2022), where individual chips perform annealing with periodic synchronization. We are also investigating promising techniques such deploying models across multiple DS-machine chips to achieve pipeline parallelism. Overall, the scalability of machhees theoretically well-founded, offering both single-chip and multi-chip solutions meet the demands of increasingly large and complex models.
be significantly expanded to handle much larger scales.
This work focuses on classic operations in Transformer- based LLMs, recogniz- ing that modern LLMs may incorporate different operations, such as diverse activation functions or embedding methods. Despite the variety of LLM architectures, these operations can gener- ally be categorized as either polynomial or non-polynomial. Polynomial operations, which can be decomposed into basic addition and multiplication, are directly mappable DS DS-machines. Non- polynomial operations, such as the exponential function, require either transformation into polyno- mial approximations (c.g., via Taylor expansion) or the addition of auxiliary circuits, which may slightly increase latency depending on their complexity. Fortunately, most high-computational demand operations, particularly those in attention layers and feed-forward MLPs, are polynomial or even linear. Thus, DS-machines offer significant flexibility and adaptability for a wide range of models.
Model Flexibility:
6 RELATED WORK
The carly-stage research on DS-machines is rooted in the Ising model, which supports only binary spin values and primarily addresses binary optimization problems (Afoakwa et al.) 2021). Our backbone DS-machine was proposed in NPGL (Wu et al.) (024)) and applied to graph learning problems. However, NPGL employs an individual learning method that ignores the architecture of Neural Networks, limiting its ability to leverage existing technologies effectively.
There are several variants of dynamical systems, such as optical (Inagaki et al. al.] 2016) and oscillator- based (Lo et al., 2023) Ising machines. Although these approaches show promise, they have yet to be successfully integrated into machine learning applications. To the best of our knowledge, this work represents the first attempt to combine existing machine learning models, particularly Large Language Models (LLMs), with DS- machinec.
7 CONCLUSION
In this work, we introduce DS-LLM, the first algorithmic framework that bridges LLMs to existing DS-machines, harnessing the power of Natural Annealing to efficiently execute LLMs on DS hard- ware. The mathematical equivalence between DS-LLM and original LLMs is proven and validated through experiments on models from GPT-2 to Llama2-7B. Results demonstrate consistent accuracy speedup and energy reduction both training and inference.
while achieving onders-of-maanitudde In conclusion, DS- LS-L presents a promising new solution for the community with significant op- portunities for further exploration in future studies.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work is supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, in support of the MEERCAT Microclectronics Science Research Center, under Contract DE-AC05-76RL018330 This work is also supported by NSF under Award No. 2326494.
REFERENCES
Richard Afoakwa, Yiqiao Zhang, Uday Kumar Reddy Vengalam, Zeljko Ignjatovic, and Michael Huang. Brim: Bistable resistively-coulled ising machine. In 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pp. 749-760. IERE, 2021.
Maxwell Anderson, Yuan Ma, Tianyu Wang. Logan Wright, and Peter McMahon. Optical transformers. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL
htttps:/cppereviiiee wttp://wwwweetooemmmmmm
net. Forumr (irrrmmmdddxxxwedFFF@.
Austin Blaquiere. Nonlinear system analysis. Elsevier, 2012.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment chal- lenge. Joaquin Quiñonero-Candela, Ido Dagan, Bernardo Magnini, and Florence d' Alché Buc (eds.), Machine Learning Challenges. Evaluating Predictive Uncertainty. Visual Object Classification, and Recognising Tectual Entailment, pp. 177-190, Berlin, Heidelberg, 2006. Springer Berlin Heidelberg. ISBN
DataCanary, hilfialkaff, Lili Jiang, Meg Risdal, Nikhil Dandekar, and tomtung. Quora question pairs, 2017. URL https:: /kagg e com/ www/compet it wtt.iiiiiiosuraaeeeeoooee quest on pa irs
William B. Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005. URL
https://aclantion wwtpogg.ooo..ooooooooooooooooo.o www.oo////O---11 I05-5002
Deepanway Ghosal, Navonil Majumder, Ambuj Mehrish, and Soujanya Poria. Text-to-audio gen- eration using instruction-tuned Ilm and latent diffusion model, 2023. URL htttt://///:xiiv.
org/abs 2tt.aa///34.333333
Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. Sttp://8kyLLonO00.ggthhh @OeenWebTextCorppus 2019.
Ryan Hamerly, Takahiro Inagaki, Peter L. McMahon, Davide Venturelli, Alireza Marandi, Tat- suhiro Onodera, Edwin Ng, Carsten Langrock, Kensuke Inaba, Toshimori Honjo, Koji Enbutsu, Takeshi Umeki, Ryoichi Kasahara, Shoko Utsunomiya, Satoshi Kako, Ken ichi Kawarabayashi, Robert L. Byer, Martin M. Fejer, Hideo Mabuchi, Dirk Englund, Eleanor Rieffel, Hiroki Take- sue, and Yoshihisa Yamamoto. Experimental investigation of performance differences between coherent ising machines and a quantum annealer. Science Advances, 5()):caau0823, 2019, doi: 10. 10.1112//siady..aaul 200202
Dan Hendrycks and Kevin Gimpel. Bridging nonlinearities and stochastic regularizers with gaussian error linear units, 2017.
Yang Hu, Xinhan Lin, Huizheng Wang, Zhen Hc, Xingmao Yu, Jiahao Zhang, Qize Yang. Zheng Xu, Sihan Guan, Jiahao Fang, Haoran Shang, Xinru Tang, Xu Dai, Shaojun Wei, and Shouyi Yin. Wafer-scale computing: Advancements, challenges, and future perspectives [feature]. IEEE Cireuits and Systems Magazine, 24(1):52 81, 2024. doi: 10.
Takahiro Inagaki, Yoshitaka Haribara, Koji Igarashi, Tomohiro Sonobe, Shuhei Tamate, Toshimori Honjo, Alireza Marandi, Peter L. McMahon, Takeshi Umeki, Koji Enbutsu, Osamu Tadanaga, Hi- rokazu Takenouchi, Kazuyuki Aihara, Ken ichi Kawarabayashi, Kyo Inoue, Shoko Utsunomiya, and Hiroki Takesue. A coherent ising machine for 2000-node optimization problems. Science, 354(6312):603 606, 2016. doi: 10. 2et./ciincc..aa424... URL https: w/www... science. arg/doi/abst /0. 1126/ sci eenieennee... aah4243

---

Published as a conference paper at ICLR 2025
Iordanis Kerenidis, Natansh Mathur, Jonas Landman, Martin Strahm, Yun Yvonna Li, et al. Quan- tum vision transformers. Quantum, 8:1265, 2024.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention, 2023, httthttss///raae
org/ ab///b////// 2309. 06180 6180
Hao Lo. William Moy, Hanzhao Yu, Sachin Sapatnekar, and Chris H Kim. An ising solver chip based on coupled ring oscillators with a 48-node all-to-all connected array architecture. Nature Electronics, 6(10):771-778) 2023.
Guy Ohayon, Theo Adrai, Michael Elad, and Tomer Michaeli. Reasons for the superiority of stochastic estimators over deterministic ones: robustness, consistency and perceptual quality. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. models are unsupervised multitask learners. OpenAl blog, 2019.
Language
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text, 2016. URL https: wttpaaaww.....a./////a/aaaaaaaaa
05250
Anshujit Sharma, Richard Afoakwa, Zeljko Ignjatovic, and Michael Huang. Increasing ising ma- chine capacity with multi-chip architectures. In Proceedings of the 49th Annual International Symposium Computer Architecture, ISCA '22, pp. 508 521,, New York, NY, USA, 2022. As- sociation for Computing Machinery. ISBN 9781450386104. 11 30055447444555 3527414
doi: 10.11457340049633527444
URL htttt:::://oi.orppgtpttttpoo
Haihao Shen, Hanwen Chang, Bo Dong, Yu Luo, and Hengyu Meng. Efficient 11m inference on cpus, 2023. URL http:::t/ttrxtttt org/ / /aall////1....... 00502
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng. and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In David Yarowsky, Timothy Baldwin, Anna Korhonen, Karen Livescu, and Steven Bethard (eds.). Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631-1642, Seattle, Washington, USA, October 2013. Association for Computa- tional Linguistcs... URL www.ltttss../ :/ac http://caaatthoooo/ooo wowetooooooy..ooooooooo www.ooogooooooo.oooooo..oooo.... 311-1770
Ruibing Song, Chunshu Wu, Chuan Liu, Ang Li, Michael Huang, and Tony Tong Geng. DS-GL: Advancing graph learning via harnessing nature's power within scalable dynamical systems. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), pp. 201.77.02222000114
45-57, 2024. doi:
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshor,, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. TRRLhhtps::/// //ariii////iiiv////aa//a org/ abwaabaa30000 20288097778888888
Fengbin Tu, Zihan Wu, Yigi Wang, Weiwei Wu, Leibo Liu, Yang Hu, Shaojun Wei, and Shouyi Yin. Multcim: Digital computing-in-ie -n---------e in-memory-hasee multimodal transformer accelerator with attention-token-bit hybrid sparsity. IEEE Journal of Solid-State Circuits, 2023.
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
12

---

Published as a conference paper at ICLR 2025
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference Learning Representations, 2019. htttttts//p/eeeeeses
net/forum? :darmmmmi.r....meee
Wenhai Wang. Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, and Jifeng Dai. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. In A. Oh, T. Nau- mann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 61501-61513, Curran Associates, Inc., httprr/ffeeeeeapee/eee/22
2023. URL. httpps://procedd ngs. neurips httppp.ccccccceeee Paaeraaeeeeeaa-aaaaa--areeeeeeee
File/cif7bledd Fww.eeeee3ee7755eeeee5e
Christopher Wolters, Xiaoxuan Yang. Ulf Schlichtmann, and Toyotaro Suzumura. Memory is all you need: An overview of compute-in-memoor architectures for accelerating large language model inference, 2024. Uttttss:://aaxii wtw.oo///.bbbbba.ae.......eeea.s
08413
Chunshu Wu, Ruibing Song, Chuan Liu, Yunan Yang. Ang Li, Michael Huang, and Tong Geng. Extending power of nature from binary real-valued graph learning in real world. In The Twelfth International Conference on Learning Representations. 2024. URL
Zhongkai Yu, Shengwen Liang, Tianyun Ma, Yunke Cai, Ziyuan Nan, Di Huang, Xinkai Song, Yifan Hao, Jie Zhang, Tian Zhi, Yongwei Zhao, Zidong Du, Xing Hu, Qi Guo, and Tianshi Chen. Cambricon-IIm: A chiplet-based hybrid architecture for on-device inference of 70b Ilm, 2024.
URLpssss://aaxx org/ abs///0s//////// 15654
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.
A APPENDIX
A.1 DISCUSSION
As an early-stage research initiative, DS-LLM aims to introduce a promising new computing paradigm to meet the growing computational demands of LLMs. While still in its infancy, this section discusses and analyzes its potential, challenges, and future directions.
First, we emphasize that there are no fundamental challenges to integrating DS-machines into existing computing systems. Although architecturally distinct from digital processors, "S-machines built using AMM0Sccoppaible technology, ensuring they can be seamlessly integrated as co-processors (similar to TPUs or NPUs) via interfaces like PCLe. No major hardware adaptations are theoretically required. From system perspective, while this work is still in its early stages, we believe the integration of DS-DDSS-iii machiness into existing computing infrastructures holds great promise for future exploration. There is significant potential in developing software toolchains, such as compilers, optimizing memory management, and pipelining tasks between DS- machines and other processors. With these advancements and supporting software tools, we are confident that DS-man DS-machines are inherently feasible for integration into digital systems as a new type of co-processor, akin to GPUs, TPUs, or NPUs. Future work could explore hybrid use cases that combine CPUs, GPUs, and DS- DS-machines, leveraging their unique strengths to achieve optimal performance. Overall, while DS- DS-machines are not yet fully mature, their fundamental fcasibility opens the door exciting opportunities for integration into existing computing infrastructures.
a) System Integration Feasibility:
b) Robustness on Circuit Non-idealities: For a comprehensive discussion, we address two types of Non-idealities: dynamic noise (e.g., thermal noise) and static offset (e.g., fabricated non-linearity and mismatch).
Dynamic noise: It has been proved that stochastic process like the natural annealing in DS-machines is usually more robust to noise than deterministic process Ohayon et al. 2023). We setup an exper- iment to evaluate the impact of varying noise levels DS-LLLM, where two models fine-tuned
13

---

Published as a conference paper at ICLR 2025
Table 5: Influence of dynamic noise accuracy of DS-LLM.
Model
Noise Level
MRPC
QQP
RTE
QNLI
SST2
gpt2-DS ESL
76.91
89.24
63.11
87.94
90.82
gpt2-DS BEL-DD-ECC
0.03
76.84
89.20
63.02
87.83
90.77
gpt2-DS ECL
0.10
76.55
88,93
62.71
87.56
90.33
Epi2-DS-ECC ECL
0.15
75.89
88.12
62.05
86.98
89.84
OPT1.3B- 1-----D--CCCL
0
86.57
88.59
78.05
91.45
91.81
OPT13BB-DS-BC
0.05
86.43
88.52
77.99
91.40
91.75
OPTT.3B-DS-BBC
0.10
86.15
88.31
77.63
91.22
91.43
OPTI.3B-DS-ER
0.15
85.67
87.75
77.24
90.89
91.01
Accuracy v.s. Time
100
Accuracy (%)
90
Variation
80
Variation 0.15
HIL HIL-Variation = 0.15
Variation 0.3
HIL-Variation
60
50
50
100
150
200
250
300
Time (ms)
Figure 8: Accuracy recovery for pre-trained GPT2 model during finetuning on DS-machines.
on the downstream datasets based on our work with dynamic noise injected into the simulation. The dynamic noise was modeled standard Gaussian noise with standard deviation ranging from 0.05 to 0.15. The results demonstrate that DS-machines exhibit high robustness to dynamic noise.
Static offset: Static offset is a common challenge in analog circuits, arising from factors like hard- ware non-linearity and mismatches during fabrication. Our proposed online training method ad- dresses this effectively by training and performing inference on the same hardware device. This ensures that the model inherently adapts to the hardware's biased patterns during training, result- ing in accurate inference despite non-idealities. Moreover, when deploying pre-trained model on different devices, promising solution is to fine-tune the model on the target hardware for few batches. This allows the model to quickly adapt to the specific biased pattern the new hardware. We evaluate this method by finetuning a pre-trate ree-traine GPT2 model DS machines with different offset level. The offset of DS-machines is modeling as standard gaussian noise with standard vari- ation from 0 0.3. As shown in Fig- [ the dash line represents the inference accuracy without finetuning, while the solid line represents the inference accuracy with finetuning. The results illus- trates the recovery of model accuracy during the fine-tuning process, underscoring the practicality and effectiveness of this approach.
The precision of computation in DS-machines is inherently continuous due to their analog nature. However, the precision of Maalog-to-Digiinn Converters (ADCs) and Digital-to-Anas -Analo Converters (DACs) used the system do affect the overall accuracy. Fortunately, ADCs and DACs are well-established technologies with numerous mature solutions that allow for various design trade-offs- instance, achieving 16-bit precision with lower power consumption or 32-bit precision with higher power consumption. Such trade-off's align with and theoreti- cally compatible with existing quantization techniques, providing flexibility to balance precision and power efficiency based on the application requirements.
Precision Constraints:
d) Advantages over other emerging computing diagram: DS OS-machines show high potential to satisfy the increasing LLM computing demanding, while there are also other emerging computing diagrams like quantum computing, optical computing and Computing-In-Mema
works. We would like to briefly compare DS-machines with other approach and highlight our key advantages.
14

---

Published as a conference paper at ICLR 2025
Quantum computing: Quantum computing is promising avenue but is still constrained by scala- bility issues and the need for complex еπтоr correction. As quantum systems scale, errors and noise increase, demanding advanced crror-corrcttion techniques that not yet fully mature. Existing largest quantum computer from IBM has about I 1100 qubits, which too small to support L.M computing tasks. Meanwhile, due to the technology requirements, nowadays building and running quantum computers are still very expensive. In the contrast, DS-machines are built on CMOS- compatible technologies, which are highly mature and the cost of its fabrication is at the same level of digital processors.
The optical computing solutions also rely on special technology and hard to be integrated with digital systems. Building complex and accurate optical circuits can be big chal- lenge and very expensive, making the stability and feasibility of optical computer larger challenge than DS-machines.
Optical Computing:
Conputing-In-Memee (CIM): CIM technology is relatively feasible and CMOS compatible. Ex- isting CIM works only supports inference, achieving up to 100s times acceleration and 2000 times energy reduction (Wolters et al. 2024), which much lower than our solution especially on energy reduction. The insight behind this is that CIM still follows a traditional instruction-bassee paradigm, completing computations step by step. In contrast, DS- maccnnhs are driven by physical processes, specifically natural annealing, which doesn't 't require step-by-ster control. This allows DS-machines to naturally perform complex tasks automatically without extra energy for controlling, achieving a much higher energy efficiency.
A.2 TRADE-OFF ON TAYLOR EXPANSION
Table 6: Trade-off on Taylor Expansion order.
Taylor Expansion order Validation loss
Loss Drop
Hardware cost (units)
Inf (baseline exponential)
3.25
0
3.35
0.10
3
3.28
0.03
5
3.27
0.02
15
3.27
0.02
Before training the models, we conducted a pre-experiment by fine-tuning pre-trained GPT-2 model on small dataset (Shakespeare, 300k) with different orders of Taylor expansion. The results show that the 3rd-order expansion achieves sufficiently low validation loss, while the improvement from higher orders is marginal. In terms of hardware requirements, the resource cost scales ap- proximately linearly with the order of each term. For instance, a polynomial like "x3 + x² + x" requires around 6 resource units, while adding Based on this trade-off, we selected the 3rd-order expansion as the most balanced design choice. For those prioritizing accuracy over hardware efficiency, higher-order expansions can be adopted and are compatible with our framework.
like "25" would demand an additional units.
A.3 IMPLEMENTATION OF OTHER OPERATIONS
Due to the page limit, we only introduce the most important computing demanding operations in Section 3. Here we provide the analysis and implementation of other operations.
a) Activation Functions:
The activation function used in GPT-2 at its initial publication was the ReLU function. However, many large language models (LLMs) have since transitioned to the Gaus- sian Error Linear Unit (GELU) (Hendrycks & Gimpel 2017) for enhanced performance.
The ReLU function is straightforward to implement in hardware by simply turning off the output for spins with negative values. In contrast, the GELU function is more complex and is defined as follows:
V2/m(x + 0.0447152
GELU(x) 0.5x 1 tanh
(17)
15

---

Published as a conference paper at ICLR 2025
The GELU function can be decomposed into matrix multiplication operations and the hyperbolic tangent function. To approximate the tanh function, we employ the same Taylor series expansion method used for the exponential function:
tan (2)
(18)
3
Consequently, the implementation of the GELU function can be transformed into series of matrix multiplication operations, similar to the approach introduced for the exponential function Fig. in Section 3.3.
b) Layer Normalization: Layer normalization crucial function in Large Language Models (LLMs). Unlike traditional Convolutional Neural Networks (CNNs), where normalization is per- formed in the batch direction and can fixed during inference, layer normalization involves com- plex computations that cannot be avoided:
H
(19)
H
i=1
1 H
0²
224
11)
(20)
(21)
V +e
y = 7Φ + 3
(22)
In this formulation, we observe that the calculations of μ and be directly mapped series of matrix multiplication operations, as previously discussed. Additionally, reduction circuits, such as differential amplifiers, must be incorporated Furthermore, computing å requires an additional circuit to manage the division operation. Thus, the entire layer normalization operation can be effectively mapped to DS DS-machines.
handle the difference between I; and μ.
16


Reviews:
[{"review_id": "3xKTSyewq0", "reviewer": "Reviewer_xU44", "summary": "The paper introduces DS-LLM, a novel framework that employs Dynamical System (DS)-based machines to optimize the training and inference processes for Large Language Models (LLMs). By incorporating an electrodynamics-based model and a Natural Annealing process, DS-LLM dramatically reduces the computational costs associated with these models. The approach is mathematically and empirically validated, demonstrating significant speedups and energy reductions without sacrificing accuracy compared to conventional LLMs.", "questions": "1. Can you provide specific examples of Dynamical System-based machines being used in real-world applications? \n2. What are the main challenges in integrating DS machines with the existing computational infrastructures used for LLMs?\n3. Could you provide more specifics about the Finite Element Analysis (FEA) software emulator used for the DS machine simulations? \n4. How closely do the results from the FEA simulations align with what you would expect from real hardware implementations? Are there any known discrepancies or performance variations between the simulated outcomes and actual hardware tests?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper pioneers the use of DS machines in the context of LLMs, leveraging their inherent Natural Annealing processes to optimize both training and inference, which is both novel and potentially transformative for the field.\n2. The reported results show a 1,090× speedup and a 102,559× reduction in energy during training, along with a 127× speedup and a 37,545× reduction during inference. These improvements are substantial and address critical cost and efficiency barriers in LLM deployment.\n3. The evaluation is thorough, with tests across multiple models and datasets demonstrating that DS-LLM achieves comparable accuracy to traditional LLMs. This thorough testing helps validate the practicality of the proposed approach.", "weaknesses": "1. While the paper demonstrates impressive theoretical and simulated results, the practical implementation of such systems might be challenging due to specialized hardware requirements. To better understand the feasibility, could you discuss specific hardware requirements, potential adoption challenges, and necessary adaptations to existing ML infrastructures for implementing DS-LLM?\n2. The paper discusses the theoretical and simulated benefits of DS machines but lacks concrete examples of these machines being implemented in real-world scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U7mR5z2Z86", "reviewer": "Reviewer_kykQ", "summary": "This work proposes a method for mapping LLM inference and training to Ising machines (which the authors refer to as \"dynamical system machines\"). They build upon existing work that introduces a room temperature Ising machine built with CMOS technology and work that maps graph learning problems to these substrates. The resulting DS-LLM is orders of magnitude more energy efficient than conventional hardware.", "questions": "1. The prior work on NP-GL utilized relatively small graphs, resulting in circuits with 100s of nodes. This is not the case for a transformer model where the number of nodes is much larger. Is it possible to manufacture Ising machines at this scale? If not, would we need to rely on a multi-chip solution?\n2. The description of back-propagation implemented on the Ising machine is not clear. What is computed by the auxiliary circuit and what is computed by the Ising machine? Is there a separate \"reverse\" network for computing gradients? If so, how are the reverse and forward networks kept aligned (typically referred to as the \"weight transport problem\" in brain-inspired learning settings)?\n3. Does the time and energy evaluation refer to the total training / inference time over the full dataset or a single sample?\n4. Does your proposed hardware solution perform training with batches? If not, wouldn't the total training time be prohibitive?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The problem tackled in this paper—the energy consumption of LLMs—is both timely and well-motivated. The approach taken by the authors is unique and the results appear quite significant. The methodology is clearly explained despite covering a broad range of background fields.", "weaknesses": "- Evaluation of inference improvements focus on a comparison against GPUs instead of low-precision, quantized implementations for edge devices. For example, comparing against low-precision CPU inference implementations (https://arxiv.org/pdf/2311.00502) or novel edge devices (https://arxiv.org/pdf/2409.15654, https://arxiv.org/pdf/2305.18691)\n- There is no discussion of the possible limitations which is extremely relevant for ICLR readers as they may not be familiar with the hardware used in this paper. In particular, focus on scalability issues (like described in Question 1) or model flexibility issues (are there operations that would be difficult to map to the proposed substrate, and are all existing transformer variants mappable?).\n- Small suggestion: the introduction would be clearer if the authors removed hyperbolic/unnecessary language surrounding the capabilities of natural annealing\n    - e.g. Line 067: \"Recently, Dynamical-System-based (DS) machines have emerged as a promising solution of such, which leverages nature itself as a computer!\"\n    - Paragraph on Lines 077-090 can be shortened as it mostly restates the previous paragraph with longer phrasing. Instead, simply state that no existing framework maps LLMs to DS machines, the prior work is specific to graph learning, and the difference in computation and scale between LLMs and graph learning poses a challenge for finding such a mapping.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IjgI0IPRYj", "reviewer": "Reviewer_v14i", "summary": "This paper proposes a novel approach for constructing large language models (LLMs) using analog dynamic system (DS) machines, offering an alternative to conventional CMOS-based hardware. The authors present two methods for mapping existing Transformer-based models onto DS machines, thoroughly analyzing key components within Transformers to facilitate a complete transformation compatible with DS architecture. The proposed DS-based models achieve significantly faster training and inference times, along with notably reduced energy consumption compared to traditional CMOS-based systems.", "questions": "* As LLMs scale up, are additional considerations or modifications necessary for DS machines to handle larger model architectures effectively?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper introduces a creative application of analog DS machines for LLM construction, advancing research on energy-efficient AI model deployment.\n* By analyzing and adapting key Transformer components for DS architecture, the authors provide a clear methodology for how LLMs can be mapped effectively, which contributes to the feasibility of DS-based machine learning.\n* The substantial reductions in training and inference time and energy consumption demonstrate the practical advantages of DS machines over CMOS for LLMs.", "weaknesses": "* The paper could benefit from a broader benchmarking analysis that includes different LLM sizes. \n* While DS machines are promising in terms of efficiency, their limitations (e.g., stability and precision constraints) are not fully discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vtVTccwIVf", "reviewer": "Reviewer_HAWr", "summary": "This paper proposes a novel approach to training and inference for large language models on dynamic systems, a new type of computing mechanism. Inference is facilitated through Hamiltonian configurations of the dynamic system machines, whereas training necessitates additional analog hardware for gradient computations. The authors assert that their approach yields substantial benefits in terms of speedup and energy reduction compared to traditional GPU-based methods.", "questions": "**Regarding novelty and differentiation from prior work**\n\n1. Can you provide a detailed comparison of your work with [1], highlighting the specific contributions and differences in your approach?\nHow does your use of dynamical systems for machine learning improve upon or depart from existing methods, particularly those that also employ attention mechanisms?\n2. What are the key advantages of your approach over prior work (and other emerging compute alternatives such as compute-in-memory), and how do you demonstrate these advantages in your experiments?\n\n**Regarding circuit non-idealities**\n\n1. How do you plan to address the potential impact of thermal noise and non-linearity in analog circuits on model accuracy?\n\n2. Can you provide a justification for the exponential approximation using Taylor expansion in Section 3.3, and explain why a 3rd-order expansion was chosen?How does the choice of Taylor expansion order affect model accuracy and hardware requirements, and can you provide experimental or theoretical evidence to support your choice?\n\n**Regarding model scalability**\n\n1. Can you provide a theoretical analysis of how your approach scales with increasing model sizes, such as those demonstrated in recent works (e.g., 405B model in llama3)?How do you expect the speedups and energy consumption to change as model sizes increase, and what are the implications for hardware requirements? Can you provide additional experiments or simulations to demonstrate the scalability of your approach?\n\n**Regarding comparisons and metrics**\n\n1. Can you clarify the basis for the claimed 1000x speedup, and provide more details on the experimental setup and metrics used?\n\n2. Did you consider using the latest techniques for fully utilizing GPU resources, such as vLLM[3], and if not, why not? Why was batched inference not used, particularly for smaller models (e.g., 124M), and how would this affect the energy calculations and comparisons with A100 GPUs?\n\n3. In terms of LLM inference/training benchmarking, what is the comparision results on meaningful metrics, such as Time to First Token (TTFT), Token Generation Rate, and Energy (Token/KWh).\n\n4. Can the authors also compare with (to a rough level) on dynamical systems with other emerging computing candidates (as introduced in the second paragraph of the introduction), such as quantum computing, optical computing, computing in memory etc? \n\n**Minor recommendations**: The abstract of the paper consists of 3 paragraphs and is quite long, can the authors abbreviate the abstract into a single paragraph to breifly explain the papers novelty and contributions?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "(1) New computing paradigm. The authors explore the use of DS machines, a novel computing paradigm for both training and inference of large language models, a timely contribution given the substantial resources currently consumed by LLM pretraining.\n\n(2) Significant gains. The authors claim 1000x speedups and 100,000x energy reduction for training while maintaining consistent model accuracy.\n\n(3) Mathematically rigorous. Although the reviewer may not have understood every detail, the problem formulation of LLM inference into Hamiltonian and Natural Annealing in dynamical systems appears mathematically solid and promising.", "weaknesses": "(1) **Lack of clear differentiation from prior work**: One of the major contributions of using dynamical systems for machine learning appears to be extremely similar to prior work [1]. The reviewer struggles to understand the significant differences and contributions of this work compared to [1], particularly given that some graph neural networks in [1] also employ attention mechanisms. A clearer explanation of the novelty and significance of this work is necessary.\n\n(2) **Inadequate consideration of circuit non-idealities**: The training process relies on specialized analog circuits to achieve gradient calculations and backpropagation. However, analog circuits are prone to non-idealities such as thermal noise and non-linearity [2]. The authors seem to overlook these non-idealities and their potential impact on model accuracy. Furthermore, the exponential approximation using Taylor expansion in Section 3.3 lacks justification, and the choice of a 3rd-order expansion is not fully explained. The authors should provide more insight into how this choice affects model accuracy and hardware requirements.\n\n(3) **Limited analysis of model scalability**: The authors conduct analysis on model sizes of 124M, 355M, and 1.3B, but recent works have demonstrated the importance of scaling up model sizes, such as the 405B model in llama3. Can the authors provide more theoretical analysis or insight into how the speedups and energy consumption scale with the size of the models?\n\n(4) **Unfair/Unclear comparisons**: The authors compare their energy calculations to A100 GPUs (800W), but the basis for the claimed 1000x speedup is unclear. Specifically, did the authors consider the latest techniques for fully utilizing GPU resources, such as vLLM[3]? Moreover, was batched inference used, considering that only 1.3B models were tested? It is unclear how/why this was not done, as four 40GB A100s would not be required for 124M models, for example. Especially in terms of LLM inference benchmarking, what is the comparision results on meaningful metrics, such as Time to First Token (TTFT),Token Generation Rate , and Energy in Token/KWh [4,5].?\n\n\nReferences\n\n[1] EXTENDING POWER OF NATURE FROM BINARY TO REAL-VALUED GRAPH LEARNING IN REAL WORLD\n\n[2] Design of Analog CMOS Integrated Circuits\n\n[3] Efficient Memory Management for Large Language Model Serving with PagedAttention\n\n[4] https://www.anyscale.com/blog/comparing-llm-performance-introducing-the-open-source-leaderboard-for-llm\n\n[5] https://huggingface.co/spaces/optimum/llm-perf-leaderboard", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)