An Expressive and Self-Adaptive Dynamical System for Efficient Function Learning
Authors: Chuan Liu
Chunshu Wu
Ruibing Song
Ang Li
Ying Nian Wu
Tong Geng
Venue: ICML 2025 poster (2025)
Keywords: Dynamical systems
Function learning
Novel computing paradigms
Primary area: general_machine_learning->hardware_and_software

Abstract: Function learning forms the foundation of numerous scientific and engineering tasks. While modern machine learning (ML) methods model complex functions effectively, their escalating complexity and computational demands pose challenges to efficient deployment. In contrast, natural dynamical systems exhibit remarkable computational efficiency in representing and solving complex functions. However, existing dynamical system approaches are limited by low expressivity and inefficient training. To this end, we propose EADS, an Expressive and self-Adaptive Dynamical System capable of accurately learning a wide spectrum of functions with extraordinary efficiency. Specifically, (1) drawing inspiration from biological dynamical systems, we integrate hierarchical architectures and heterogeneous dynamics into EADS, significantly enhancing its capacity to represent complex functions. (2) We propose an on-device training method that leverages intrinsic electrical signals to update parameters, making EADS self-adaptive with exceptional efficiency. Experimental results across diverse domains demonstrate that EADS achieves higher accuracy than existing works, while offering orders-of-magnitude speedups over traditional neural network solutions on GPUs for both inference and training, showcasing its broader impact in overcoming computational bottlenecks across various fields.

An Expressive and Self-Adaptive Dynamical System for Efficient Function Learning
Chuan Liu' Chunshu Wu
Ruibing Song !
Ang Li Ying Nian Wu+
Tony (Tong) Geng 15
Abstract
1. Introduction
Function learning is essential for modeling, analysis. and prediction across a wide range of scientific and engineering tasks. Modern ML methods, particularly neural networks (NNs), have demonstrated exceptional capability in approxi- mating complex functions by learning from data. Despite their remarkable achievements, the computational demands of ML models have soared due to the escalating model com- plexity. Even on the most powerful GPU, training these models remains prohibitively expensive, and the waning of Moore's Law exacerbates this challenge. As result, the quest for alternative, efficient computational paradigms has become increasingly urgent.
Function learning forms the foundation of numer- ous scientific and engineering tasks. While mod- ern machine learning (ML.) methods model com- plex functions effectively, their escalating com- plexity and computational demands pose chal- lenges to efficient deployment. In contrast, nat- ural dynamical systems exhibit remarkable com- putational efficiency in representing and solving complex functions. However, existing dynami- cal system approaches are limited by low expres- sivity and inefficient training. To this end, we propose EADS, Expressive and self-Adaptive Dynamical System capable of accurately learning a wide spectrum of functions with extraordinary efficiency. Specifically, (1) drawing inspiration from biological dynamical systems, we integrate hierarchical architectures and heterogeneous dy- namics into EADS, significantly enhancing its capacity to represent complex functions. (2) We propose on-device training method that lever- ages intrinsic electrical signals to update param- eters, making EADS self-adaptive with excep- tional efficiency. Experimental results across di- verse domains demonstrate that EADS achieves higher accuracy than existing works, while offer- ing orders-of sfceersoffmagnnnne -angnnituee speedups over traditional neural network solutions on GPUs for both infer- ence and training, showcasing its broader impact in overcoming computational bottlenecks across various fields.
Nature offers an elegant remedy to the growing computa- tional burden of modern MI. methods. Natural dynamical systems exemplify how complex functions can be efficiently modeled and solved through their intrinsic processes. For example, partial differential equations (PDEs) that govern physical phenomena or chemical reactions are inherently solved by the dynamical systems they describe. This natural process can be interpreted as the system evolving over an energy landscape, where lower energy states correspond to higher statistical likelihood. Driven by thermodynamic principles, such systems instinctively evolve toward the equi- librium state, generating optimal solutions. This process, which we refer to as natural annealing, exhibits excep- tional efficiency. Motivated by this potential, this work investigates whether dynamical systems can be leveraged to develop a novel ML paradigm that effectively learns various functions with significantly improved efficiency.
Although early theoretical work highlighted the promise of dynamical systems for ML (Weinan, 2017; Li & Weinan, 2021; Weinan et al., 2022), practical progress has been hin- dered by the lack of suitable hardware embodiments. For- tunately, recent breakthroughs in programmable electronic dynamical systems (Afoakwa et al., 2021; Böhm et al., 2022) have marked turning point. Leveraging these electronic dynamical systems, previous studies have demonstrated that the power of nature can be leveraged to tackle some simple learning problems with remarkable efficiency (Wu et al., 2024; Song et al., 2024b; Böhm et al., 2022). However, the applicability and broader impact of electronic dynamical
University of Rochester, Rochester, NY, USA PPacific North- west National Laboratory, Richland, WA, USA Universityy of Washington, Seattle, WA, USA University of California Los An- geles, Los Angeles, CA, USA 'Rice University, Houston, TX, USA. Correspondence to: Tony (Tong) Geng <tg62@ @ rric..uuuu
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

An Expressive and Self-Adaptive Dynamical System
systems remain significantly limited due two key chal- lenges: 1. Low Expressivity: Existing dynamical systems are governed by energy functions that entail simple inter- actions among nodes, and hence limiting their capacity to represent complex functions. 2. Inefficient Training: Ex- isting approaches realize inference on dynamical systems through on-device natural annealing; however, the training process to construct the desired energy landscape is still per formed on digital processors, resulting in high training costs. This decoupling of training and inference deviates from the intelligence observed in natural systems, preventing this emerging ML paradigm from addressing the most pressing computational challenge in ML development: extremely high training costs. Therefore, substantial advancements are needed fully realize the potential of dynamical systems.
10a.
Normalized Speed
10a
Neural Networks
Dynamical Systems
104
Enhanced exprasiiiity $31
On-device training $3.2
10a
OnOPU
Previous Work
1Z
This Work
3
Figure The overview of EADS.
ternal electrical signals with exceptional efficiency.
Experimental results demonstrate that EADS accu- rately learns various functions, achieving orders-of- magnitude speedups (10 x) over baselines on GPUs.
Notably, numerous scientific studies (Wills al., 2005; Fris- ton, 2010; Inagaki et al., 2019) suggest that the brain also functions as dynamical system, performing inference and training in a collocated manner. Inspired by the brain, a highly efficient and powerful dynamical system, we propose enhancing the electronic dynamical system two ways: (I) improving the system's 's expressivity through hierarchi- cal architectures and heterogeneous dynamics; (2) enabling on-device training to fully leverage its extraordinary com- putational power. Specifically, to enhance expressivity, we improve the dynamical system with a hierarchical struc- ture and heterogencous dynamics, facilitating progressive information refinement through distinct processing stages. Furthermore, we propose a learning method that allows the dynamical system to leverage its intrinsic electrical signals to self-construct its energy landscape, align with target data distributions, and achieve on-device training with excep- tional efficiency.
2. Preliminaries and Related Work
This section provides preliminaries of ML. through the lens of dynamical systems, including both theoretical develop- ments and recent advances with hardware embodiments. Subsequently,
review related work that uses dynami- cal systems to address various tasks, from combinatorial optimization to advanced ML applications.
2.1. Preliminaries
ML via Dynamical Systems. The potential of dynami- cal systems in ML was highlighted by (Weinan, 2017: Li & Weinan, 2021; Weinan et al., 2022), demonstrating their abil- ity to model complex, high- high-dimensional nonlinear functions through continuous transformations. These works highlight that deep NNs succeed by composing simple functions to approximate complex ones, while dynamical systems ex- tend this compositional approach to an infinitesimal limit. Compared to deep NNs, dynamical systems offer several ad- vantages: (1) greater flexibility in imposing constraints and incorporating domain-specific structures, facilitating more transparent theoretical analysis than purely discrete-layer architectures; and (2) easier integration of ML techniques with physical models, enabling seamless interaction with real-world physical processes. Despite these promising theoretical advancements, practical adoption of dynamical systems in ML. has been limited by the lack of suitable hardware embodiments.
To this end, this work introduces EADS, a nature- powered ML paradigm that leverages the computational power of dynamical systems for accurate and efficient function learn- ing. By expanding the applicability of dynamical systems to encompass functions from diverse domains, e.g. real-world problems, PDEs in scientific computing, and MI. kernels, EADS holds the potential to overcome persistent computa- tional bottlenecks and drive advancements across various fields. The overview of EADS shown in Figure 1, and the contributions of this paper are summarized as:
We propose FADS, an expressive and self-adaptive dynamical system capable of accurately and efficiently learning functions across diverse domains.
Fortunately, recent advancements in programmable elec- tronic dynamical systems have revived interest in this field, Originally conceived physical embodiments of the binary Ising model for solving binary combinatorial optimization problems, these systems have since expanded to tackle bi- nary learning tasks (Pan et al., 2023; Liu et al., 2023). (Wu et al., 2024) later extended the binary Ising model to support real-valued nodes, enabling real- velluv graph learning tasks
We introduce hierarchical structures and heterogeneous dynamics to enhance the dynamical system's 's capacity to represent complex functions.
We propose an on-device training method that enables the dynamical system to train its parameters using in-
2

---

An Expressive and Self-Adaptive Dynamical System
(Liu et al., 2025b). This extension results in the following energy function (also referred to Hamiltonian):
Node Unit
Node Unit Control
cu
CU
cU
Couping Uni Contoo
I
Y
N
N
R
Hrp
Jeyaayy
+
Ti E IR. (I)
CU
Here, Jiy represents the interaction strength between nodes 3y and v.y. while h; denotes the self-reaction strength of ri external influences. Assuming Boltzmann distribution Pre =e -- /Z, where the partition function Z serves normalization constant that ensures that the probabilities sum up to one, the energy landscape is mapped to probabil- ity distribution, with the lowest energy state corresponding to the highest probability state. The system's 's dynamics are designed as:
cu
CU
CU
Coupling Unit
Figure 2. The backbone electronic dynamical system.
trained the model using computationally expensive statisti- cal methods executed on digital processors, mainly GPUs. Specifically, the training process begins by estimating the node values using methods such as conditional likelihood maximization (Wu et al., 2024). The discrepancies between the estimations and the ground truths are evaluated using metrics such as Mean Absolute Error (MAE). These met- rics serve as loss functions to update the model parameters, thereby reconstructing the energy landscape to align with the data distribution. During inference, the well-trained pa- rameters mapped onto the circuit, then natural annealing drives the system toward the lowest energy state, enabling it to find the solution with the highest probability for the target problem.
dx;
or Dz.
N Σ j7i
(Jig + -yi))) xzj 2h,
(2)
dt
which guarantees the spontaneous energy decrease of the system:
dHry
8Hro dx; Ox: dt
N 8thr Or,
0.
dt
(3) When applied to graph learning tasks, a subset of nodes is fixed to input values, while the remaining nodes, serving as output nodes, are randomly initialized and evolve according the designed dynamiis. Given a well-trained Hamiltonian that accurately captures the correlation between inputs and outputs, the spontaneous energy decrease makes the system instantly anneal to desired solutions.
2.2. Related Work
Dynamical systems have gained significant attention as an efficient computing paradigm, initially applied to optimiza- tion problems (Sharma et al., 2023a; Sun et al., 2025) and subsequently extended machine learning tasks (Wu et al., 2024; Liu al., 2025a; Song et al., 2024a). The Ising ma- chine, one of the earliest hardware implementations lever- aging dynamical systems, embodies the Ising model orig- inally developed for ferromagnetism in statistical physics. Ising machines have demonstrated breakthrough efficiency in solving numerous binary optimization problems (Böhm et al., 2019; Mohseni et al., 2022; Lo et al., 2023). For instance, researchers have employed Ising machines to ad- dress satisfiability (SAT) problems (Sharma et al., 2023a;b; Sun et al., 2025), as well MAX-CUT and graph color- ing problems (Wang & Roychowdhury. 2019; Böhm et al., 2019; Liu et al., 2025c).
Physical Embodiment of Dynamical Systems. This dy- namical system is physically realized using programmable electronic components, such as resistors and capacitors, as illustrated in Figure 2. The key idea behind this embodiment is to precisely and efficiently realize the node dynamics us- ing electronic components. In this design, each node 24 is implemented as a nanoscale capacitor within a node unit (N₂), with its voltage representing the node value. Each ca- pacitor is coupled with a resistor resistance RG = 1/(2),). forming a resistive current within the node unit, realizing the term 2/, in the node dynamics. Additionally, capaci- tors from different node units (N; and Ný) structurally connected via programmable resistor in the coupling unit (CU,)), with resistance R.; 1/Jíj- This configuration effectively incorporates the term E (Jey + Jyt) zy in the node dynamics, implementing a resistively coupled capaci- tor network.
Recognizing their potential, researchers have explored dy- namical systems in ML applications such as unsupervised NN training (Böhm et al., 2022), graph learning (Pan et al., 2023), and collaborative filtering (Liu et al., 2023). While these studies provide valuable insights into leveraging dy- namical systems for ML tasks, their scope and applicability are limited by the binary nodes of Ising machines, hindering progress in more complex, real-valued scenarios. Recent work (Wu et al., 2024; Wu et al.) introduced real-valued
Offline Training of Dynamical Systems. Training dy- namical system involves optimizing the parameters and in the Hamiltonian Hys to construct an energy landscape that reflects the target data distribution. Previous works have
3

---

An Expressive and Self-Adaptive Dynamical System
dynamical systems accelerate inference in graph learning problems; however, their contributions are constrained by two key limitations. First, while the proposed Hamiltonian supports real-valued nodes, it only accounts for simple inter- actions, limiting its ability to capture the intricate patterns present in many complex problems. Second, their approach utilizes the power of dynamical systems exclusively during the inference phase, leaving the computationally intensive training process unaddressed. Although some simple on- device training methods have been developed (Liu et al., 2025a; Wu et al., 2025), they have only demonstrated ef- fectiveness on simple models. These limitations, which significantly constrain the broader impact of dynamical sys- tems, will be addressed this work.
dhg di
Σ j=1
P.jaa - rihhi
(4)
where 2; E = 1,2..... .N denotes the j-th input node, h4 E R (i 1,2, H) denotes the i-th hidden node. The projection weight P.g IR denotes the connection from 2j to h, analogous to dendritic integration in biological neurons. The term ri denotes the self-reaction strength of the hidden node h;,
Once the hidden nodes in Eq. 4 have stabilized, they further evolve through internal coupling, reflecting the dense local connectivity observed in cortical circuits:
dh; dl
H [ Jashue
3. Methodology
(5)
The pursuit of powerful and highly efficient computing sys- tems has been profoundly influenced by the extraordinary capabilities of biological systems, particularly the human brain. By contrasting the brain's remarkable capabilities with existing physically embodied dynamical systems, two fundamental limitations emerge: (1) insufficient expressivity and (2) inefficient training. This section introduces solu- tions to address these limitations. In particular, Section 3.1 introduces hierarchical, heterogeneous dynamical system to boost expressivity, and Section 3.2 presents an on-device training method that makes the system self-adaptive with exceptional efficiency.
where Jak R (i, h: 1,2..... H) represents inter-node interaction weights, and is a hardware friendly nonlinear function, such as ReLU. This nonlinear function can be efficiently implemented using diodes to regulate current flow, thereby enabling nonlinear processing capabilities.
Finally, the processed hidden states are mapped to the output space through:
H Qmahs
dym dl
-m Mm
(6)
where Dm E I (m 1,,,..., M) represents output nodes,
3.1. Expressivity Enhancement
Qmi E R is the output projection weight matrix, and T'm. denotes output self-reaction strength. This hierarchical pipeline yields significantly enriched expressivity than tradi- tional flat and homogeneous dynamical systems, as demon- strated in Section 4.
Existing dynamical systems with physical embodiments, while promising, exhibit several limitations when compared to the brain. (1) Flat Structure. Unlike the brain's 's hierar- chical organization, which processes information through multiple layers of increasing abstraction, existing dynamical systems maintain a flat structure. (2) Homogeneous Dynam- ics. In contrast to the brain's rich repertoire of nonlinear processing mechanisms, where different regions exhibit di- verse dynamics, current dynamical systems rely on uniform dynamics, characterized by linear interactions among nodes. These constraints limit their ability model intricate, non- linear functions.
Physical Embodiment of Enhanced Dynamical Systems. The physical implementation of the enhanced dynamical sys- tem builds upon the foundational design shown in Figure 2. Following the same design strategy, node values are mapped to capacitor voltages, enabling the natural realization of continuous-time dynamics. Each capacitor is coupled with a resistor of resistance r, or Ti. forming the resistive current within the node unit. The parameters J, Q are config- ured as conductances of programmable resistors, thereby implementing the system dynamics as electrical currents. Specifically, the terms Σ/= Pij.T,, a (Ei/+ Juthk) and
Brain-Inspired Enhancements. To address these limita- tions, we propose two key enhancements inspired by the architecture and functionality of the brain: (1) a hierarchical structure and (2) heterogeneous dynamics. Our enhanced system implements a multi-stage processing pipeline in- spired by the brain's 's information processing mechanisms.
Σ'" Quihh are mapped as the flow-in currents into the re- spective node units. The terms r.h; and m'ml mapped as internal currents within each node unit. The design of input nodes zy and output nodes Um remains consistent with the original architecture, while the hidden nodes h, have been extended to incorporate the newly introduced dynam- ies, as illustrated in the "Improved design of h," section
The enhanced system initiates with a projection that trans- forms inputs into an abstract hidden space, modeled by the following dynamics:
4

---

An Expressive and Self-Adaptive Dynamical System
In
[R]
[R|
Tin
Improved design of hi
Trainable
Trainable Jik
Trainable Qmi
Feedback from Ym
Figure The key components of the enhanced dynamical system with on-device training support.
of Figure 3, where the added components are highlighted in yellow. As shown, two switches control the dynamics of each hidden node: when the black switch is closed, the circuit realizes the input-to-hide hidden dynamics described by Eq. 4; when the yellow switch is closed, implements the inter-hidden dynamics of Eq. 5. The nonlinear function is implemented by using diodes to restrict the current, enabling effective hardware implementation of the nonlinearity.
computed as: aT,jQmm: = 6m 11g- Since output nodes Vm are clamped to their ground truth values during training, I' essentially acts as scaling factor for the ground truth signal. To achieve an efficient hardware implementation, we make I'm = 1, thereby streamlining both the training and inference processes.
However, for the we face unique challenge: unlike output nodes, we lack ground truth values for hidden states h;- To address this, we employ the Adjoint Sensitivity Method (Pontryagin, 2018), a powerful technique from optimal control theory that en- ables gradient computation through an auxiliary dynamical system. This approach is particularly suitable as it: (1) elim- inates the need for ground truth hidden states, (2) maintains mathematical rigor while being hardware-realizzable For- mally. we introduce an adjoint node a, aL/Oh, for each hidden node hú, with initial value being Σm. Q Qmi and dynamics governed by:
coupling weights Jak.
3.2. Instant On-Device Training
Despite these enhancements, the system's advantages main constrained without efficient training support, train- ing is the most computationally intensive process. Therefore, to extend the extraordinary computational power of dynami- cal systems to the training process, we propose an efficient on-device training method, EC-Train. This novel approach utilizes the intrinsic electrical signals of the dynamical sys- tem feedback for on-device parameter adjustment, en- abling self self-adaptation the target data distribution. EC- Train significantly reduces training costs, achieving orders- of-magnitude improvements in efficiency over conventional offline training on digital processors.
da: dt
H Σ
aiJu..
(9)
1=1
Here, I is boolean indicator defined
On-Device Instant Training Approach. According to fundamental physical principles (Kirchhoff's current law), each output node yo stabilizes when its flow-in current 1-1 Qmih; balances its internal resistor current
Σ Jich > 0)
11
>0) - EJhhha > >0).
41 = o'
(10)
p DH1 I Tmmm- When is clamped to its ground truth value, the difference between Iin and R provides a direct measure of error. Consequently, the on-device training process of EC- Train aims to minimize the difference between /t / for all output nodes when their values (voltages) set to ground truth values. In this way, the loss function of EC-Train can be formulated as:
The gradient with respect to coupling weights Ju is then computed through a dynamical process:
aL = Jr arlihadl,
-
(11)
OJ.
which involves the evolution of on and h; from T backward to 0 to accumulate the updates.
M Σ m=1
For the parameters Po and r; involved the input-to- hidden projection stage, the stabilization the hidden state h; follows Kirchhoff's current law. Specifically, hy reaches steady state when its incoming current 1.th =E P.jaj is balanced by the current through its internal resistor, I rih;. Given that each input xj is clamped to its ground-truth value, the equilibrium condition implies a stable hidden state value of h, = > P.jæ., assuming r: = without loss of generality, as it functions as a scaling factor. The error signal propagated from the inter-hidden
2
M
m
The error signal &m emerges intrinsically from the dynami- cal system, serving as natural feedback signals for parameter optimization:
8m
(1
1R).
Specifically, the gradients with respect Qms are then
5

---

An Expressive and Self-Adaptive Dynamical System
stage is denoted as 6; = a; t=0 Consequently, the gradient with respect to the projection weight P.j is denoted as:
the performance of EADS in graph learning tasks that it is originally designed for, showing the performance of EADS in learning complex functions in real-word problems. Then, we show its potential on other tasks, including PDE solving in scientific computing and approximating important kernels in Large Language Models (LLMs).
aL/OP.y = Fzg.
(12)
Detailed derivations are provided in the Appendix.
Physical Embodiment of EC-TTaiin The proposed training approach introduces simple yet effective hardware modifi- cations that enable self- training through electrical current feedback. As shown in Figure 3 (highlighted in red), we in- troduce additional feedback signal paths for each parameter. These feedback paths allow the electronic dynamical system to propagate signals to the coupling units, facilitating param- eter adjustments through the rapid charging or discharging of programmable resistors. Specifically, for parameters Jik. access to their unmodified values is crucial for computing the adjoint nodes. To achieve this, we incorporate addi- tional capacitors (highlighted as red "=" in Figure 3): one dedicated to receiving feedback signals for parameter up- dates and another to preserve the original values required for adjoint and hidden node calculations. This dual-capacitor configuration ensures accurate gradient computation while enabling efficient, high-speed on-device learning, reinforc- ing the system's 's capability for real-time adaptation. The hardware implementation of the newly-introduced adjoint nodes a, is encoded as a node unit, as depicted in the "a, " section of Figure 3. With EC-Train, the system performs continuous updates within each natural annealing cycle, rapidly reshaping the energy landscape to achieve instant training with exceptional efficiency compared to traditional training on digital processors. The EC-Train training pro- cess is detailed as follows:
Experimental Platforms. We conduct our experiments us- ing an NVIDIA A100 40GB SXM GPU for non-dynamical system based baselines, measuring total training time, in- ference latency per sample, and accuracy. For dynamical system based approaches, we build upon the original hard- ware embodiment BRIM (Afoakwa et al., 2021), using a custom
Finite Element Analysis (FEA) simulator to assess the training time, inference latency, and accuracy. Since the dynamical system based baseline NP- GL (Wu et al., 2024) only achieves inference on dynamical systems, its training time is still measured on an A100 GPU using its own offline training method.
4.1. Graph Learning
Datasets and Baselines. For complex function learning in real-world problems. we evaluate the performance of EADS in
prediction tasks including six real-world datasets from four applications. (1) Traffic flow prediction with two datasets PEMS04 and PEM08 (Chen et al., 2001), ). (2) Air quality prediction including PM2.5 and PM10 (Kong et al., 2021). (3) Taxi demand prediction (NYC Taxi): pre- dicting the hourly number of taxi trips (New York City Taxi and Limousine Commission, 2024). (4) Pandemic progres- sion prediction (Texas COVID): predicting the daily number of new cases (Centers for Disease Control and Prevention, 2024). We compare EADS with SOTA spatial-temporas prediction baselines, including Graph WaveNet ( (Wu et al., 2019), MTGNN (Wu et al., 2020), DDGCRN (Weng et al., 2023), MegaCRN (Jiang et al., 2023), and the dynamical system based method NP-GL (Wu et al., 2024). The number of hidden nodes in EADS is set to 128, and baselines are implemented following the experimental setups detailed in their respective original papers.
1. Initialization: The capacitor voltages representing in- puts and outputs are set their ground truth values, while the trainable parameters are randomly initialized.
2. Natural Annealing:
The system undergoes natural annealing and generates the electrical current 1 -I IR which serves as the feedback signal to adjust the system parameters.
3. Parameter Adjustment: The trainable parameters are updated based on the feedback signal.
We report the test MAE of baselines and EADS in Table 1, where lower values indicate better performance. The results show that EADS outperforms all baselines across all datasets, achieving an average MAE reduction of 23.91% %. Notably, EADS reduces MAE by up to 16.29% compared to the best baseline on the Texas Covid dataset. Furthermore, when compared to the dynamical system based baseline NP-GL, EADS achieves an average MAE reduction of 8.08% across all datasets, highlighting the improved system expressivity.
Experimental Results.
4. Iterative Training: The update of trainable parameters results in a new electrical current fir to the node units ym. updating the feedback signal f R and in- stantancously initiating a new training iteration. This iterative process continues across the training set until convergence is reached.
4. Evaluation
Additionally, Figure 4 presents the training time and in- ference latency of EADS compared to the baselines, where EADS exhibits remarkable computational efficiency. Specif-
As a pioneering effort demonstrating the significant potential of physically embodied dynamical systems, we first evaluate
6

---

An Expressive and Self-Adaptive Dynamical System
Table Spatial-temporal prediction performance in MAE. EADS consistently outperforms all haselines.
Dataset
PEMS04
PEMS08
PM2.5
PM10
NYC Taxi
Texas Covid
Graph WaveNet
20.84
15.77
1,82
1.95
10.22
82.96
MTGNN
19.96
15.15
1.83
1.99
7.08
84.17
DDGCRN
18.97
14.64
1.71
1.88
3.06
23.94
MegaCRN
17.65
13.70
1.65
1.74
6.08
83.73
NP-GL
17.07
13.51
1.62
1.73
3.03
22.04
EADS
16.92
13.43
1,53
1.62
2.46
18.45
Normalizzed Train Time
10a
10
Inference Latency
10-2
10:3
10a
10
10
10
100
10
10-1
10
PM2.5
PEMS04
Covid
GraphWaveNet
MTGNN
DDGCRN
MegaCRN
NP-GL
EADS
Figure Training time and inference latency for spatial-temporal prediction. EADS shows higher efficiency than GPU-hased baselines.
ically, EADS delivers an average training speedup of 356x compared to NP-GL. Across all baselines, EADS achieves an average training speedup approximately x00 Fur- thermore, in terms of inference latency, EADS delivers an average speedup of around 1000 compared the base- lines executed GPUs. These substantial improvements in both training and inference efficiency highlight the signif- icant potential of EADS for real-time applications.
better accuracy than FNO, underscoring its effectiveness even against advanced operator learning methods.
Beyond accuracy, we compare the training times and infer- ence latencies of all models across the datasets, as illustrated in Figure 5. EADS exhibits exceptional training and infer- ence efficiency, substantially outperforming methods im- plemented on GPUs. On average, EADS achieves training and inference speedups of around 1000x, highlighting its extraordinary computational performance. The results un- derscore the potential EADS to accelerate simulations in scientific computing and to support applications demanding
4.2. PDE Solving
Datasets and Baselines. Following (Li et al., 2020a), we evaluate our method on the Burgers' equation and the Darcy Flow equation. The datasets are generated following the same procedure as (Li et al., 2020a), ensuring consistency benchmarking. The data is collected on grids of varying spatial resolutions: 16 x 16, 32 32, and 64 64. The model trained to learn the mapping from the initial condition (or coefficient field) to the corresponding solution at specific time, under the same spatial resolution. We compare our method against several established benchmarks, including: NN (Li et al., 2020a), FCN (Zhu Zabaras, 2018), GNO (Li et al., 2020b), FNO (Li et al., 2020a), and the dynamical system based method NP-GL (Wu et al., 2024). All base- lines are implemented following the setups detailed in their respective original papers.
Table 2. Test MAE for PDE solving. EADS achieves superior accuracy compared to baselines.
Burgers
Methods
S1=256
S2=1024
S3=4096
NN
1.26
10 3
1.32 x 10
1.69 x 10 3
FCN
1.37
10
1.35
10
1.76
10
GNO
6.95 10
6.97
10-5
7.42 10
FNO
1.41 10-5
1.42
10-5
1.53
10
NP-GL
1.59 10-4
1.57
10-4
1.83
10
EADS
1.32 10-5
1.46
10-5
1.51 10
Darcy Flow
Methods
S1=16x16
S2=32x32
S3=64x64
The performance of EADS and the baseline methods on the selected PDEs summarized in Table Across all evaluated resolutions and PDEs, EADS consistently achieves lower test MAE than NN, FCN, GNO, and NP-GL. Notably, EADS also exhibits marginally
Experimental Results.
8.48 10
8.30
10
3.86 10
FCN
5.83
10
6.14
10-6
3.77
10
GNO
5.71 10
5.62
10
1.26 10
FNO
4.72 x 10-6
3.36 x 10-6
1.03 x 10-5
NP-GI.
6.51 10
6.72 10-6
3.65 10
EADS
4.31 x 10
3.17 x 10
1.12 x 10

---

An Expressive and Self-Adaptive Dynamical System
Normalized Train Time
104
10
10a
Inference Latency
10
10-3
10a
10
10-5
10
10
10
Burgers-S1 Burgers-Se 52 Burgers-S3
Darcy-S1
Darcy-S2
Darcy-S3
Burgers-S1 Burgers 52 Burgers-S3
Darcy-S1
Darcy-S2
Darcy-53
NN
FCN
GNO
FNO
NP-GL
EADS
Figure 5. Training time and inference latency for PDE solving, EADS is markedly more efficient than GPU-based baselines.
rapid PDE solving, such as interactive design optimization and time-critical decision support systems.
ter performance. For comparison, we also evaluated NP-GL (Wu et al., 2024), the dynamical system baseline, to provide reference benchmark.
4.3. LLMs
Experimental Results. As shown in Table 3, EADS demon- strates remarkable consistency and robustness across all kernel positions, with only minimal performance degrada- tion ranging from 1.54 to 1.95 PPL points compared the original GPT-2. This stability indicates that EADS success- fully captures and reproduces the complex transformations encoded within each kernel. Notably, EADS substantially outperforms NP-GL across all kerne positions, achieving an average perplexity reduction of approximately 1.49 points. Furthermore, as illustrated in Figure 6, EADS achieves a remarkable training speedup of roughly 800 compared to NP-GL on GPUs. For inference latency, EADS delivers speedup of approximately 102 GPUs, underscoring its potential as promising approach to enhancing LLM efficiency.
Experimental Setups. To evaluate the capability of EADS in learning complex functions embedded within advanced ML models, conducted experiments in the context of LLMs. Specifically, we adopt the GPT-2 small model ( (Wolf, 2019), which consists of 12 transformer decoders, each con- taining a causal self-attention kernel. For each attention kernel, we construct training dataset by extracting input- output pairs from GPT-2's forward pass on the LAMBADA dataset (Paperno et al., 2016), thereby capturing the transfor- mation performed by that kernel. We then train separate EADS for each of the 12 attention kernels to assess EADS's s ability to replicate the underlying complex transformations. During evaluation, we replace one attention kernel in GPT- with its corresponding trained EADS while keeping other GPT-2 components unchanged. The performance of the modified GPT-2 evaluated using test perplexity (PPL) on the LAMBADA dataset, where lower values indicate bet-
the kernel rumning
5. Conclusion
Modern ML methods have demonstrated exceptional capa- bility in approximating various functions, yet their increas- ing complexity and substantial computational costs pose significant challenges to efficient development. In contrast, nature effortlessly models complex functions through dy- namical systems. Inspired by this, we introduce EADS, a nature-inspired ML paradigm that leverages an expressive and self-adaptive dynamical system to learn various func- tions with unprecedented efficiency. The proposed EADS incorporates hierarchical architectures and heterogeneous dynamics to enhance the expressivity of existing dynamical systems. In addition, we introduce an on-device training method that enables the dynamical system to optimize its pa- rameters using intrinsic electrical signals, thereby achieving exceptional training efficiency. Experiments across func- tions from diverse domains show that EADS achieves higher accuracy than existing methods while delivering orders-of- magnitude speedups over traditional GPU-based approaches for both inference and training. These results underscore the potential of EADS to overcome the computational bot- tlenecks across various critical fields.
Table 3. Test perplexity (PPL) on LAMBADA.
Kereess
2
3
5
GPT2
35.13
35.13
35.13
35.13
35.13
35.13
NP-GL
37.79
37.90
38.17
38.33
38.41
38.48
EADS
36.67
36.77
36.86
36.83
36.95
36.94
Kemels
7
8
9
10
11
12
GPT2
35.13
35.13
35.13
35.13
35.13
35.13
NP-GL.
38.41
38.38
38.72
38.31
38.79
38.77
EADS
36.75
36.93
36.96
36.87
36.94
37.08
Normalized Train Time
10a
Inference Latency (s)
10
102
10-1
10
10
100
10-7
A100 GPU
NP-GL
EAD5
Figure 6. Training time and inference latency on LAMBADA.
8

---

An Expressive and Self-Adaptive Dynamical System
Acknowledgements
Proceedings of the AAAI conference on artificial intelli- gence, volume 37, pp.
2023.
This work is supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Comput- ing Research, in support of the MEERCAT Microelcctron- ics Science Research Center, under Contract DE-AC05- 76RL01830. This work is also supported by DARPA un- der Contract W912CG25CA007, by NSF under Award No. 2326494, and by NERSC through the perlmutter supercom- puter for computational resources.
Kong, L., Tang, X., Zhu, J., Wang, Z., Li, J., Wu, H., Wu, Q. Chen, H., Zhu, L.,, Wang, W., et al. A -ear--oo high-resolution air quality reanalysis dataset in china based the assimilation of surface observations from eneme. Earth System Science Data, 13(2):529-570, 2021.
Li, Q. and Weinan, E. Machine learning and dynamical systems. Collections, 54(09), 2021.
Impact Statement
Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhat- tacharya, K., Stuart, A., and Anandkumar, A. Fourier neural operator for parametric partial differential equa- tions.
This paper presents work whose goal is to advance the field of dynamical systems for machine learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
arXiv preprint arXiv:2010.08895, 2020a.
Li, Z., Kovachki, N., Azizzadeneshelii K., Liu, B., Bhat- tacharya, K., Stuart, A., and Anandkumar, A. Neural operator: Graph kernel network for partial differential equations. arXiv preprint MiXi::0003.034835 2020b.
References
Afoakwa, R., Zhang, Y., Vengalam, U. K. R., Ignjatovic, Z., and Huang, M. Brim: Bistable resistively-couple ising machine. In 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pp. 749-760. IEEE, 2021.
Liu, C.,, Song, R., Wu, C., Haghi, P., and Geng, T. Instatrain: Adaptive training via ultra-fast natural annealing within dynamical systems. The Thirteenth International Con- ference on Learning Representations, 2025a.
Böhm, F., Verschaffelt, G., and Van der Sande, G. A poor man's 's coherent ising machine based on opto-electronic feedback systems for solving optimization problems. Na- fure Communications, 10(1):3538, Aug 2019. ISSN 2041- 1723.
Liu, C., Wu, C.,, Song, R., Chen, Y., Li, A., Huang, M., and Geng, T. Nature-gl: A revolutionary learning paradigm unleashing nature's ' power real-world spatial-temporal graph learning. In Proceedings of the 30th Asia and South Pacific Design Automation Conference, pp. 865- 871, 2025b.
Böhm, F., Alonso-Urquijo, D., Verschaffelt, G., and Van der
Sande, G. Noise-injected analog ising machines enable ul- trafast statistical sampling and machine learning. Nature Communications, 13(1):5847, 2022.
Liu, Y., Sun, L., Huang, M., and Wu, H. Integrated hardware annealing based on langevin dynamics for ising machines. In 2025 Design, Automation & Test in Europe Conference (DATE), pp. 1 1-6. IEEE, 2025c.
Centers for Disease Control and Prevention. Covid data tracker, 2024. URL https: http///ovidddoooo cdc..oo....oooooooo covid-data aecaaa-trackeee trackerakeere Accessed: September 20, 2024.
Liu, Z., Yang. Y., Pan, Z., Sharma, A., Hasan, A., Ding, C., Li, A., Huang, M., and Geng, T. Ising-ef: A path- breaking collaborative filtering method through efficient ising machine learning. In 2023 60th ACM/EEF Design Automation Conference (DAC), 2023.
Chen. C., Petty, K., Skabardonis, A., Varaiya, P., and Jia, Z. Freeway performance measurement system: mining loop detector data. Transportation research record, 1748(1): 96-102, 2001.
Lo, H., Moy, W., Yu, H., Sapatnekar, S.,, and Kim, C. H. An ising solver chip based on coupled ring oscillators with 48-node all-to-all connected array architecture. Nature Electronics, 6(10):771-77 2023.
Friston, K. The free-energy principle: a unified brain theory? Nature reviews neuroscience, 1272):127-138, 2010.
Inagaki, H. K., Fontolan, I.., Romani, S., and Svoboda, K. Discrete attractor dynamics underlies persistent activity in the frontal cortex. Nature, 566(7743):212 2019.
Mohseni, N., McMahon, P. L., and Byrnes, T. Ising ma- chines as hardware solvers of combinatorial optimization problems. Nature Reviews Physics, 4(6):363-779, 2022.
Jiang, R., Wang, Z., Yong, J., Jeph, P., Chen, Q.. Kobayashi, Y., Song, X., Fukushima, S., and Suzumura, T. Spatio- temporal meta-graph learning for traffic forecasting. In
New
York City Taxi and Limousine Commis- sion. The trip record data, 2024. URL https: wttp//www../////t//ttttooe/

---

An Expressive and Self-Adaptive Dynamical System
cle-trip-record page. September 20, 2024.
Accessed:
Weinan, E., Han, J., and Li, Q. Dynamical systems and optimal control approach to deep learning. Mathematical Aspects of Deep Learning, pp. 422, 2022.
Pan, Z., Sharma, A., Hu, Y.-C., Liu, Z., Li, A., Liu, H., Huang, M., and Geng. T. Ising-trafin trangttaffic: Using ising machine learning to predict traffic congestion under un- certainty. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 9354-9363, 2023.
Weng, W., Fan, J., Wu, H., Hu, Y., Tian, H., Zhu, F., and Wu, J. A decomposition dynamic graph convolutional recur- rent network for traffic forecasting. Puttern Recognition, 142:109670, 2023.
Wills, T.J., Lever, C., Cacucci, F., Burgess, N., and O' O'Keefe, J. Attractor dynamics in the hippocampal representation of the local environment. Science, 308(5723):733887, 2005.
Paperno, D., Kruszewski, G., Lazaridou, A., Pham, Q. N., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint arXiv:1606.0603/ 2016.
Wolf, T. Huggingface's transformers: State-of-the- art natural language processing. arXiv preprint arXiv: 1910.03771, 2019.
Pontryagin, L. Mathematical theory of optimal processes. Routledge, 2018.
Wu, C., Song, R., Liu, C., Wang, Y., Chen, Y., Li, A., Liu, D., Wu, N., Huang, M., and Geng, T. Np-nds: nature- pooure--owweedd nonlinear dynamical system for power grid forecasting, In The Third Learning on Graphs Con- ference.
Sharma, A., Burns, M., Hahn, A., and Huang, M. Aug- menting electronic ising machine to effectively solve boolean satisfiability. Scientific Reports, 13(1):22858, 2023a.
Sharma, A., Burns, M., and Huang, M. C. Combining cubic dynamical solvers with make/break heuristics to solve sat. In 26th International Conference on Theory and Applications of Satisfiability Testing (SAT 2023). Schloss- Dagstuhl-Leibniz Zentrum für Informatik, 2023b.
Wu, C., Song, R., Liu, C., Yang, Y., Li, A., Huang, M., and Geng, Extending power nature from binary to real-valued graph learning in real world. In The Twelfth International Conference Learning Representations, 2024.
Song, R., Liu, C., Wu, C., Li, A., Liu, D., Wu, Y. N., and Geng, T. Ds-llm: Leveraging dynamical systems to enhance both training and inference of large language models. In The Thirteenth International Conference Learning Representations, 2024a.
Wu, C., Song, R., Liu, C., Haghi, P., Li, A., Huang, M., and Geng, T. Ds-tpu: Dynamical system for on-device lifelong graph learning with nonlinear node interaction. In Proceedings of the 52nd Annual International Symposium on Computer Architecture, pp. 1867-1879, 2025.
Song, R., Wu, C., Liu, C., Li, A., Huang, M., and Geng, T. T. Ds-gl: Advancing graph learning via harness- ing nature's power within scalable dynamical systems. In 2024 ACM/IEEE 51st Annual International Sympo- sium on Computer Architecture (ISCA), pp. 45-57, IEEE, 2024b.
Wu, Z., Pan, S., Long, G., Jiang, J., and Zhang, C. Graph wavenet for deep spatial-temporal graph modeling. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pp. 1907-1913, 2019.
Wu, Z., Pan, S., Long, G., Jiang. J., Chang, X., and Zhang, C. Connecting the dots: Multivariate time series forecasting with graph neural networks. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 753-763, 2020.
Sun, L.., Burns, M. X., and Huang, M. C. Enhancing escillator-based ising machine models with amplitude dynamics and polynomial interactions. arXiv preprint 2rXh::2504.00329, 2025.
Zhu, Y. and Zabaras, N. Bayesian deep convolutional encoder- decoder networks for surrogate modeling and uncertainty quantification. Journal of Computational Physics, 366:415-447, 2018.
Wang, T. and Roychowdhury, J. Oim: Oscillator-based ising machines for solving combinatorial optimisation problems. In McQuillan, I. and Seki, S. (eds.), Uncon- ventional Computation and Natural Computation, pp- 232-256, Cham, 2019. Springer International Publishing.
Weinan, E. A proposal on machine learning via dynamical systems. Communications in Mathematics and Statistics, 15):1111, 2017.
10

---

An Expressive and Self-Adaptive Dynamical System
A. Appendix
Below, we present the detailed mathematical derivations of the proposed EC-Train method. The proposed BC-Train approach aims minimize the difference between 1 I after clamping the output nodes to their ground truth values. this way, the EC-Train loss function can be formulated as:
M
1
Σ M
(13)
The error signal 6m emerges intrinsically from the dynamical system, serving as natural feedback signals for parameter optimization:
6m
M (Tin
(14)
Specifically, the gradients with respect to Qmi are then computed as:
ØT. 6m h4.
(15)
aQ
Since output nodes Vm are clamped to their ground truth values during training, 7m essentially acts as scaling factor for the ground truth signal. To achieve an efficient hardware implementation, we make rm 1, thereby streamlining both the training and inference processes.
For the inter-hidden-node coupling weight Jik that encoded in the following dynamical process:
dh;
a
Σ
H
Junhe
(16)
di
where Jis represents inter-node interaction weight and (I)) a nonlinear function defined as:
x; x 0, 0,
x
(17)
x < 0.
Since their final states (I) h2(T), hy (T)) determines the output node values, we have:
dL
Qmaom
(18)
(T)
To compute parameter gradients, following the adjoint sensitivity method, we introduce the adjoint node a,, satisfying:
da:
H
0f Oh.
aj
(19)
dt
1=1
where
H
fi = 0
Σ Jish;
(20)
By taking the partial derivative, obtain:
0f d' 0h;
Σ "Jnto
Jii:
(21)
where (z) is the derivative of (xx), given by:
1, x > 0,
d(xx)
(22)
0, x <

---

An Expressive and Self-Adaptive Dynamical System
Consequently, the adjoint node evolves as:
da; dt
H J1i
H Σ 1=1
Σ 1=1
000
Σ J1shi
a
(23)
with the initial value:
ØL (T(T)
E
(24)
(T)
Here, 1 a boolean indicator defined as:
H Jishs > 0 =1
11
Jiahi > 00.
==0 o'
Jiai
(25)
Using the adjoint nodes, the gradient of L with respect to Jik is computed as:
ÖL
[
a 0f
S"e a, Σ Jiatie
H1
hadt [" h,I,
(26)
aJ.
JJık
For the input-to-hidden dynamical process:
dh, dt
Pijay r2hi;
(27)
where h; the dynamical node, æ, is the fixed input, Ps the parameter. According Kirchhoff's current law, h, reaches steady state when its incoming current I," = Σÿ= Piyæry is balanced by the current through its internal resistor, I," = rgh Given that each input æy is clamped to its ground-truth value, the equilibrium condition implies a stable hidden state value of
N
h; Pajag
(28)
assuming r; without loss of generality, it functions scaling factor. These equilibrium states then the initial states for the inter-hidden dynamical process in Eq. 16, thus the error signal propagated back from the inter-hidden stage to this input-to-hidden stage is ó; ailc=o- Consequently, the gradient with respect to the projection weight Pij is given by:
0L aP.
8 Iz.
(29)
12


Reviews:
[{"review_id": "bo4AKRxPmA", "reviewer": "Reviewer_yCbr", "summary": "The authors propose a method to learn equations efficiently from data. Current methods such as neural networks have high complexity and high operational cost which hinders widespread applications. Recently electronic dynamic systems have shown great promise in solving simple learning problems with great efficiency. However, current electronic dynamic systems lack sufficient expressivity to learn complex equations, and also lack effective training support. To mitigate the current limitations of electronic dynamic systems for learning, the authors propose Expressive and self Adaptive Dynamic System (EADS) that integrates hierarchical architecture and heterogeneous dynamics to increase expressivity and also propose an on-device learning method for efficiency. Experiments show that EADS has lower training times and higher/comparable accuracy compared to baselines.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your positive feedback and constructive suggestions. Below, we address each of your points in detail to further improve the manuscript.\n\n**1.Additional explanations for comparable accuracy experiments**\n\nThank you for this insightful suggestion. In our evaluations, we compare EADS with NP-GL and deliberately selected state-of-the-art digital baselines that have consistently demonstrated exceptional accuracy through rigorous validation in prior research. Although the accuracy differences may appear similar in magnitude, the actual improvements are significant. Specifically, compared to NP-GL, EADS achieves an average MAE reduction of 8.92% on spatial-temporal prediction, a 71.4% MAE reduction in PDE solving, and a 6.32% reduction in PPL on LLM tasks. Furthermore, compared to the best digital baseline, EADS yields a 15.10% MAE reduction on spatial-temporal prediction, and a 4.62% MAE reduction on PDE solving, while substantially reducing computational demands. We will expand our analysis with additional explanations to better highlight the advantages of EADS.\n\n**2.Terminology clarification**\n\nThank you for highlighting this potential confusion and for your valuable suggestion. We agree that the term “function learning” offers a more precise description of our work. We will change “equation learning” to “function learning” throughout the manuscript and differentiate our focus from symbolic regression in the Introduction.\n\n**3.Add visual diagram for EADS workflow**\n\nWe appreciate your valuable suggestion. We will include a new figure in the Methodology section to illustrate the overall EADS workflow. As we cannot directly upload figures in the rebuttal, we provide a detailed description of the proposed diagram below. The diagram will be divided into two main components:\n- **Expressivity Enhancement:** We will visualize the hierarchical structure and heterogeneous dynamics with clear illustrations of information flow. This visual will demonstrate how EADS progressively refines the input through multiple processing stages with heterogeneous dynamics.\n- **Instant On-Device Training:** This part will detail the parameter adjustment process. It will depict how intrinsic electrical signals are used as feedback to drive learning, illustrating the feedback loop where the output nodes’ electrical currents guide rapid, on-device parameter updates.\n\n**4.Enhanced figure and table captions to highlight key messages**\n\nThank you for recommending more informative captions. In our revision, we will update the captions to explicitly highlight the key performance takeaways:\n- **Table 1:** Spatial-temporal prediction performance in MAE. EADS consistently outperforms all baselines across all datasets (best results in bold).\n- **Figure 4:** Training time and inference latency for spatial-temporal prediction. EADS demonstrates significantly higher efficiency than GPU-based baselines.\n- **Table 2:** Test MAE for PDE solving. EADS achieves superior accuracy compared to all baselines, with marginal improvements over FNO.\n- **Figure 5:** Training time and inference latency for PDE solving. EADS is markedly more efficient than GPU-based baselines.\n- **Table 3:** Test perplexity (PPL) on LAMBADA. EADS significantly outperforms NP-GL.\n- **Figure 6:** Training time and inference latency on LAMBADA. EADS exhibits superior efficiency compared to GPU-based methods.", "claims_and_evidence": "The authors claim their method provides low training times and more expressive power. Experiments show that the training times are lower. The accuracy is clearly higher for one experimental setup, while for others they are comparable. The comparable accuracy experiments can use some additional possible explanations.", "methods_and_evaluation_criteria": "The proposed method makes sense, as the authors clearly identify 2 gaps in current literature, and their proposed method incorporates possible remedies for the identified gaps. The training times are measured for efficiency comparison, while accuracy is measured for expressivity comparison.", "theoretical_claims": "None", "experimental_designs_or_analyses": "I looked at the experimental designs, at quick glance the experimental designs look good, however I haven’t checked the details thoroughly.", "relation_to_broader_scientific_literature": "Energy efficient training is a major concern at present for complex machine learning models, the authors work can significantly push the limit of training efficiency using hardware optimizations through dynamic systems. As scientific literature in many domains now heavily relies on machine learning methods and model training, this can have a broad impact on overall literature.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "The authors present a very good introduction, clearly identifying gaps in current literature and justifying their proposed solutions. I have few suggestions that the authors can consider for improving the current manuscript:\n1. Equation learning can be confused with symbolic equation learning/symbolic regression, I think a better name could be function learning. To avoid confusion, the authors can add a line or two in the introduction.\n2. It will be great if the authors can provide a simplified visual diagram of EADS workflow, that will be very helpful to interested readers.\n3. The captions can be more detailed, highlighting the key message i.e., figure shows EADS is more efficient", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZVfpO0NBt8", "reviewer": "Reviewer_17Sp", "summary": "The paper proposes an Expressive and self-Adaptive Dynamical System (EADS) that can learn a wide range of equations with efficiency. The authors propose an efficient on-device learning method that leverages intrinsic electrical signals to update parameters, making EADS self-adaptive at a reduced cost. The authors explore the accuracy of EADS and compare it to existing works, showing that EADS can provide speedups and energy efficiency over other methods.\n\n## update after rebuttal\n\nIn view of the authors rebuttal, I updated my score 2 -> 3.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thorough review and constructive feedback. We address each concern in detail.\n\n**1.Relation to Equation Discovery**  \nWe clarify that our work fundamentally differs from equation discovery: rather than discovering equations, our work focuses on efficiently solving equations with high speed and low computational cost while maintaining high accuracy. EADS works by optimizing its parameters through the proposed on-device training method to capture the joint distributions between inputs and outputs. Once trained, it can efficiently generate solutions for new inputs. In PDE solving, equation discovery methods would attempt to identify the underlying equations, while EADS aims to generate solutions to PDEs under varying inputs (different coefficients or initial conditions) as described in [1]. Importantly, our work extends to solve analytically intractable equation. We will add a subsection in the Related Work to state the relation. \n\n**2.Clarification of Learn in Dynamical Systems**  \nWe apologize for any confusion. As noted in Q1, learning is indeed misleading since our focus is on solving equations efficiently rather than learning them. We agree that EADS do not learn their inherent dynamics. However, it's worth noting that natural dynamical systems do adjust the parameters within their dynamics. A typical example is the human hippocampus, which functions as a dynamical system and adjusts synaptic strengths (i.e., parameters) between neurons [2]. We will revise the manuscript to replace “learn equations” with “tune the parameters within dynamics.”\n\n**3.Formality of language, Sentence Clarifications and Key Term Definitions**  \nWe will revise the manuscript to use more precise language. \n- Line 327: Revised to “in graph learning tasks that it is originally designed for.”\n- Inference Latency: Defined as the time delay between submitting an input to a trained model and receiving its output.\n- Hard-to-Define Equation Learning: Revised to “For complex real-world problems where underlying equations are unknown or not readily expressible in closed form.”\n- Abstract: Revised to “While modern machine learning methods approximate complex functions, their escalating complexity and computational demands pose challenges to efficient deployment.”\n\n**4.Hamiltonian Training and Example**  \nTraining Hamiltonian involves optimizing its parameters to capture correlations between nodes. Consider $\\mathcal{H} = -\\sum_{i\\neq j}^{N} J_{ij} x_i x_j + \\sum_{i=1}^{N} h_i x_i^2$, with trainable parameters $J,h$. Using a conditional likelihood method [3], the estimated state at equilibrium is: $\\hat{x_i} = \\frac{1}{2h_i}\\sum_{j\\neq i}^{N} (J_{ij}+J_{ji}) x_j.$ Then, the MSE loss is minimized via standard backpropagation to optimize parameters. Detailed explanation and example will be included in Section 2.1.\n\n**6.EADS Training in PDE Solving**  \nIn PDE solving, EADS approximates solutions under varying inputs (coefficients or initial conditions). Training is optimizing system parameters ($P,J,Q$) to capture the correlations between inputs and outputs. In one of the Darcy Flow example [1], we solve the 2D Darcy Flow equation on the unit square with Dirichlet conditions. Each training sample comprises a 16×16 grid of coefficient a(x) and its corresponding solution u(x). EADS learns mapping from a to u, then rapidly generate solutions without iterative solvers. A detailed description will be added in the Appendix.\n\n**7.Interpretability and Comparison with Equation Discovery and FEM**  \nAs explained in our response to Q1, our method does not aim to derive explicit equations from data.\nSince the goal and required training data are significantly different between our method and methods in equation discovery methods, it is difficult to incorporate genetic programming based methods in our evaluation for a fair comparison. To provide a reference, a method in the genetic programming-based equation discovery domain that also evaluated on the Burgers' equation achieves an MSE of $4.33×10^{-5}$ [4], while our method achieves an average MSE of $9.37×10^{-6}$ under our evaluation.\nRegarding the comparison with FEM, our evaluation employs outputs from the FEM as ground truth, so we only compare their efficiency. On average, the inference latency of EADS is at the level of 10e-7 seconds, while the latency of typical FEM generally exceeds 10e-3 seconds. Therefore, the efficiency of EADS is significantly better than FEM, especially on complex PDEs with higher resolutions. \n \n[1] Li, Z., et al. Fourier Neural Operator for Parametric Partial Differential Equations. ICLR.  \n[2] Wills, T.J., et al. Attractor dynamics in the hippocampal representation of the local environment. Science, 2005.  \n[3] Wu, C., et al. Extending power of nature from binary to real-valued graph learning in real world. ICLR.  \n[4] Chen, Y., et al. Symbolic genetic algorithm for discovering open-form partial differential equations (SGA-PDE). Physical Review Research.", "claims_and_evidence": "The paper show evidence of good performance of the proposed method.", "methods_and_evaluation_criteria": "The method is evaluated on a set of benchmark datasets. Some baselines are missed in the experimental evaluation. Notably, a genetic programming method should be included in the comparison and the Finite Element Method (FEM) for the PDE solving task.", "theoretical_claims": "Theoretical claims are not discussed in the paper.", "experimental_designs_or_analyses": "I did not run experiments to verify the results.", "relation_to_broader_scientific_literature": "The paper does not emphasize enough the relation of the proposed method with the literature on equation discovery (aka symbolic regression). The authors should discuss how their method relates to existing methods in the field of equation discovery.", "essential_references_not_discussed": "The paper does not discuss any reference in the field of equation discovery, also known as symbolic regression. The authors should discuss how their method relates to existing methods in the field of equation discovery.", "other_strengths_and_weaknesses": "- Strengths\n    - The paper addresses the important question of efficiency in inference and training in machine learning.\n- Weakness\n    - In my opinion, the language in the paper is not precise and too colorful for a scientific paper. For example, \"Nature presents an elegant solution to this computational crisis.\" or \"Can we harness dynamical systems to create a nature-powered ML paradigm that learns and solves equations with revolutionary efficiency?\". I would suggest the authors to use a more formal language.", "other_comments_or_suggestions": "1. The paper repeatedly uses the idea of a dynamical system being able to \"learn\" equations. (e.g., \"... dynamical systems effortlessly learn complex equations ...\", \"Natural systems effortlessly learn and solve complex equations through inherent dynamical processes.\", etc). In mathematical physics, the term \"dynamical system\" refers to a system that evolves over time according to a set of rules, typically described by a system of differential equations. (From Wikipedia: \"In mathematics, a dynamical system is a system in which a function describes the time dependence of a point in an ambient space, such as in a parametric curve.\") Therefore, the equation that describes the dynamics of a dynamical system is not \"learned\" by the system, but rather is a fundamental property of the system. It is very important that the authors clarify this use of the term \"learn\" in the context of dynamical systems.\n\n2. Rewrite the line 327: \"...we first evaluate the performance of EADS in graph learning tasks that it is originally deployed, showing the performance of EADS ...\". Do you mean \"in graph learning tasks that it is originally designed for\"?\n\n3. A definition of \"inference latency\" will be pertinent for the reader to understand the results presented in the paper. The authors should provide a definition of this term in the paper.\n\n4. Line 306 \"For hard-to-define equation learning in real-world problems, ...\" What do the authors mean by \"hard-to-define equation learning\"? This sentence is vague and should be clarified.", "questions_for_authors": "1. (Abstract) \"While modern machine learning (ML) methods are powerful equation learners, their escalating complexity and high operational costs hinder sustainable development.\" What do the authors mean by \"equation learners\"? This is not a standard term in the ML literature. Is a LLM a \"equation learner\"? What do the authors mean by \"sustainable development\" in this context? This sentence is vague and should be clarified.\n\n2. (Section 2.1). \" Given a well-trained Hamiltonian that accurately captures the correlation between inputs and outputs, ...\" How is the Hamiltonian trained? Can the authors provide an example on how to do this for a toy problem? I think that will help the reader to understand the concept better.\n\n3. (Section 4.2) The authors present examples of PDE Solving. How does the Hamiltonian of the system is trained in this case? Can the authors provide an example on how to do this for a simple Parabolic PDE?\n\n4. Experimental Results. A critical aspect of learning equations is the ability to inspect the learned equations and assess their interpretability. Can the authors provide some examples of the learned equations and discuss their interpretability?\n\n5. Table 1. How does the method compare to other methods in the field of equation discovery (aka symbolic regression)? In particular, a well-stablihed method in the field is genetic programming. How does the proposed method compare to genetic programming in terms of accuracy and efficiency?\n\n6. (4.2 PDE Solving) The authors should compare with a Finite Element Method (FEM) solver for the PDEs. How does the proposed method compare to a FEM solver in terms of accuracy and efficiency?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wbvPRxBejX", "reviewer": "Reviewer_z4kd", "summary": "This paper introduces EADS, an framework for learning equations efficiently\n. EADS is inspired by the efficiency of natural systems in learning and solving complex equations. The authors argue that EADS overcomes the limitations of traditional ML methods, such as high computational cost and complex models. They demonstrate EADS's superior performance in various tasks, including graph learning, PDE solving, and LLMs.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your insightful comments. We will address your questions below.\n\n**1.The number of parameters**  \nWe will report the number of parameters in the Appendix. Due to the character limit, we present parameters for a dataset below. For the PEMS04 dataset: Graph WaveNet: 250,689; MTGNN: 555,169; DDGCRN: 567,109; MegaCRN: 316,225; NP-GL: 188,498; EADS: 94,976. \n\n**2.Include NP-GL in Comparison** \n\nNP-GL supports inference on a dynamical system, its training is conducted offline on GPUs. According to [1], NP-GL operates with similar power for inference to EADS. Both methods achieve similar inference latency, resulting in comparable inference energy efficiency. However, for training, compared to NP-GL, EADS achieves an average ~$10^{2}$ training speedup, ~ $10^5$ greater energy efficiency.\n\n**3.Include Theoretical Computational Complexity**  \nThe computing paradigm of EADS is inherently different from that of digital methods, rendering their complexities not directly comparable. Specifically, GPU-based methods execute explicit instructions sequentially, while EADS operates via natural annealing—where electrons (dis)charge capacitors to seek equilibrium. To provide a clear understanding, we define one operation in our DS as a single substantial (dis)charge event of a capacitor, taking ~0.2 ns. For a digital processor, a single instruction execution requires ~0.25 ns. Given that the operation execution times are comparable, we evaluate computational complexity by comparing their total number of operations. Our results show that EADS requires ~$10^3$ operations per inference, while GPU-based solutions require ~$10^5$ operations per inference. We will incorporate detailed discussion.\n\n**4.Experiment Details on LLMs**  \nWe design our LLM experiments to assess EADS’s ability to learn the complex transformations embedded within each decoder of GPT-2. The entire GPT-2 model is not implemented on EADS; rather, only one decoder is implemented by EADS. Although our system has the potential to implement all decoders, such an extension is beyond the current scope of our work. We will provide details in the manuscript.\n\n**5.Include the Settings of EADS**  \nThanks for your valuable suggestion. We will add a comprehensive section in the Appendix specifying EADS configurations. \n\n**6.Scalability and Flexibility of EADS**  \n- Scalability: Dynamical systems have demonstrated strong scalability through single-chip and multi-chip solutions, as evidenced in [2-3]. Specifically, [2] propose a single-chip solution, while [3] explored multi-chip solutions. Overall, the scalability of dynamical system machines is well-founded.  \n- Flexibility of EADS: Since EADS enables on-device training, its parameters can be dynamically adjusted to suit different problems without hardware modifications, making it highly versatile.\n\n**7.Why Is the Method Called Dynamical System**  \nWe refer to EADS as a “dynamical system” because EADS is not merely a model—it is a software-hardware co-design implemented using programmable electronic components.\n\n**8.Under What Circumstances Is On-Device Training Needed?**  \nOn-device training is essential for several reasons:\n1. Enabling a New Computing Paradigm: Instant on-device training extends the exceptional computational power of dynamical systems beyond inference to include training, thereby establishing a new AI paradigm with remarkable efficiency.\n2. Applications Requiring Real-Time Adaptation or Facing Training Costs: \n- By implementing training directly on-device, EADS can update parameters automatically in response to new data in real time without the delays associated with off-device training, ensuring accurate predictions as underlying patterns evolve.  \n- In scientific computing, the potential of data-driven equation solving is often hindered by prohibitive computational costs. As ML models scale to achieve higher accuracy, training resources may exceed those required by traditional numerical solvers, limiting their practical benefits. Our on-device training directly addresses this challenge.\n3. Advancing AI Paradigm Development: Recent research suggests that collocating inference and training on the same hardware can significantly reduce training costs while more closely mirroring biological intelligence [4]. There is also evidence indicating that the human hippocampus functions as a dynamical system with collocated training and inference [5]. Our proposed on-device training method offers a meaningful exploration in this domain.\n\n[1] Extending power of nature from binary to real-valued graph learning in real world\n\n[2] DS-GL: Advancing Graph Learning via Harnessing Nature’s Power within Scalable Dynamical Systems\n\n[3] Increasing ising machine capacity with multi-chip architectures\n\n[4] The forward-forward algorithm: Some preliminary investigations\n\n[5] Attractor dynamics in the hippocampal representation of the local environment", "claims_and_evidence": "The main claim is that EADS achieves higher accuracy than traditional ML methods. This is justified by experimental results on graph learning, PDE solving, and LLMs show that EADS outperforms baselines in terms of accuracy. The proposed method also achieves much higher efficiency compared with those methods running on dedicated gpus, for both inference and training.", "methods_and_evaluation_criteria": "The performance of EADS is evaluated based on accuracy, training time, inference latency, and energy consumption. \nI have concerns regarding the evaluation baselines.\n\n1.Regarding accuracy (e.g, table 2), it is difficult to judge the effectiveness of the proposed method without mentioning the number of parameters in each method.\n\n2.Regarding energy consumption (e.g section 4.4), why a dedicated all purpose gpu is introduced for comparison with the proposed hardware customized to the specific model design? A fair comparison should include NP-GL, which also significantly reduce the energy cost on CMOS-compatible computers.\n\n3.The above concern can also apply to training time, as the major contribution is from not using gpu instead of the proposed method.\n\n4. Including theoretical computational complexity could be a more suitable evaluation criterion for all algorithms, as it is independent of hardware constraints.", "theoretical_claims": "n/a", "experimental_designs_or_analyses": "In figure 6, EADS achieves much faster training time than NG-PL due to training implementation on device. In section 4.3, it says \"we replace a single decoder block with its corresponding EADS model while keeping all other components of GPT-2 unchanged\", is GPT-2 also implemented on the device? can you specify the details?\n\nThere also lacks details regarding the architecture of the proposed method for each experiment setting (e.g., number of parameters, hidden units, input/output dimensions.)", "relation_to_broader_scientific_literature": "n/a", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "n/a", "other_comments_or_suggestions": "n/a", "questions_for_authors": "How does EADS compare to other methods in terms of scalability and flexibility? Do we need to customize the hardware if the model is modified?\n\nWhy the method is called \"dynamical system\" rather than \"dynamical system model\" or more specific models like Ising model? EADS is essentially a model to capture the system behavior, not the system itself.\n\nI'm curious under what circumstance on-devide training is needed?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The proposed method improves on the high complexity of existing methods for equation learning. The proposes an interesting novelty in combining algorithmic and hardware aspects to equation learning. The empirical evaluation demonstrated the improved efficiency of the method.
Decision: Accept (poster)