ODE-based Smoothing Neural Network for Reinforcement Learning Tasks
Authors: Yinuo Wang
Wenxuan Wang
Xujie Song
Tong Liu
Yuming Yin
Liangfa Chen
Likun Wang
Jingliang Duan
Shengbo Eben Li
Venue: ICLR 2025 Spotlight (2025)
Keywords: Reinforcement Learning
Smooth Control
Low-pass Filter
Neural ODE
Primary area: reinforcement learning
TL;DR: This paper proposes a neural unit structure with smooth properties, and based on it, proposes a smoothing policy neural network. This work is the first time that Neural ODE method is used for action smoothing in deep reinforcement learning.

Abstract: The smoothness of control actions is a significant challenge faced by deep reinforcement learning (RL) techniques in solving optimal control problems. Existing RL-trained policies tend to produce non-smooth actions due to high-frequency input noise and unconstrained Lipschitz constants in neural networks. This article presents a Smooth ODE (SmODE) network capable of simultaneously addressing both causes of unsmooth control actions, thereby enhancing policy performance and robustness under noise condition. We first design a smooth ODE neuron with first-order low-pass filtering expression, which can dynamically filter out high frequency noises of hidden state by a learnable state-based system time constant. Additionally, we construct a state-based mapping function, $g$, and theoretically demonstrate its capacity to control the ODE neuron's Lipschitz constant. Then, based on the above neuronal structure design, we further advanced the SmODE network serving as RL policy approximators. This network is compatible with most existing RL algorithms, offering improved adaptability compared to prior approaches. Various experiments show that our SmODE network demonstrates superior anti-interference capabilities and smoother action outputs than the multi-layer perception and smooth network architectures like LipsNet.

Published as a conference paper at ICLR 2025
ODE-BASED SMOOTHING NEURAL NETWORK FOR RE- INFORCEMENT LEARNING TASKS
Yinuo Wang', Wenxuan Wang', Xujie Song', Tong Liu', Yuming Yin",
Liangfa Chen', Likun Wang', Jingliang Duan 1.3- Shengbo Eben Li
School of Vehicle and Mobility & College of AI, Tsinghua University
2 School of Mechanical Engineering, Zhejiang University of Technology
3 School of Mechanical Engineering, University of Science and Technology Beijing
syn23mmails. ts tsinghua edu. en, duanj
com, lishboltsinghhn edu, cn
ABSTRACT
The smoothness of control actions is a significant challenge faced by deep rein- forcement learning (RL) techniques in solving optimal control problems. Existing RL-trained policies tend to produce non-smooth actions due to high-frequency input noise and unconstrained Lipschitz constants in neural networks. This article presents a Smooth ODE (SmODE) network capable of simultaneously addressing both causes unsmooth control actions, thereby enhancing policy performance and robustness under noise condition. We first design smooth ODE neuron with first-order low-pass filtering expression, which can dynamically filter out high frequency noises of hidden state by learnable state-based system time constant. Additionally, we construct a state-based mapping function, and theoretically demonstrate its capacity control the ODE neuron's Lipschitz constant. Then, based on the above neuronal structure design, we further advanced the SmODE network serving as RL policy approximators. This network is compatible with most existing RL algorithms, offering improved adaptability compared to prior approaches. Various experiment, show that our SmODE network demonstrates superior anti-interference capabilities and smoother action outputs than the multi- layer perceptron and smooth network architectures like LipsNet.
INTRODUCTION
Recently, deep reinforcement learning (RL) has emerged an effective method for solving optimal control problems in the physical world Guan al. (2022); Peng al. (2021); Kaufmann al. (2023); Li (2023); Wang al. (2024). RL algorithms commonly employ neural networks (NNs) to learn optimal control policies due to their universal approximation capabilities Sonoda & Murata (2017); Schäfer & Zimmermann (2006). However, in practical optimal control scenarios, the outputs of NNs are often sensitive to noise disturbances, as noted by Molchanov et al. (2019). Inadequately addressing this sensitivity result in severe consequences. For instance, oscillations in control actions may cause drone crashes Shi et al. (2019), increased wear in robotic arm components Yu et al. (2021), and heightened safety risks in autonomous driving Wasala al. (2020); Chen et al. (2021).
To optimize NN performance in optimal control scenarios, research has primarily concentrated on improving the smoothness of NN-based control systems. Current approaches be classified into four principal categories: filtering methods, action penalty methods, adversarial perturbation methods, and network enhancement methods.
Filtering methods like Kalman and extended Kalman filtering Chen et al. (2023) effectively suppress noise and reduce output oscillation by estimating the current state from multi-step historical data. These methods work well with Gaussian noise but struggle with non-Gaussian noise. Particle filtering Wang et al. (2021), in contrast, samples directly from the probability density function address nonlinear and non-Gaussian noise, making it more suitable for such environments. However, it is computationally intensive due to the need for many samples and can suffer from particle degeneracy, affecting its accuracy Daum & Huang (2011).
Corresponding authors.

---

Published as a conference paper at ICLR 2025
Action penalty methods penalize significant shifts in actions to enhance stability and smoothness during policy learning. Mysore et al. (2021) incorporated two regularization components within the policy loss function: one mitigates variance between consecutive actions over time, and another promotes action consistency across similar states. Similarly, Kobayashi (2022) introduced the 1.2C2 algorithm with dual losses: one for action congruence and another for coherence in the value function across similar states, adjusting action penalties based on value function congruity. While these methods improve stability and smoothness, fine-tuning hyperparameters without diminishing system performance is challenging.
Adversarial perturbation techniques aim to reduce oscillatory output actions by integrating optimized perturbation data during training. The main goal is to enhance the agent's 's resistance to noisy data Zhao et al. (2022), improving control effectiveness unpredictable or noisy environments. Shen et al. (2020) employed projected gradient ascent to identify the most effective perturbation noise, maximizing action divergence under genuine and adversarial conditions. This approach effectively mitigates the oscillation issues caused by noise. However, the algorithm increases complexity by generating adversarial states, and it faces compatibility challenges with mainstream RL algorithms and limited generalizability.
The aforementioned methods each have their drawbacks: the filtering method necessitates multi-step historical data, action penalties may compromise control optimality. and adversarial perturbations complicate RI. methods. Network enhancements add noise resistance directly to the NN through structural improvements, avoiding major modifications to the RL algorithm. Miyato et al. (2018) employed spectral normalization to reduce the NN's Lipschitz constant, enhancing smoothness. Similarly, Song et al. (2023) introduced LipsNet, which adaptively modulates the local Lipschitz constant, effectively dampening action oscillation Gouk et al. (2021). Nonetheless, with high observation noise, controlling the NN's Lipschitz constant alone inadequately suppresses action fluctuations. Additionally, the neural ordinary differential equation (ODE) network Chen et al. (2018); Hasani et al. (2021): Asikis et al. (2022); Hasani al. (2022); Ruiz-Balet Zuazua (2023), defined by ODEs, emerges promising approach due to its flexibility in autonomous ODE design. To the best of our knowledge, we are the first to attempt using neural ODE to simultancously address the action non-smoothness problem in deep RL caused by high-frequency input noise and large Lipschitz constant.
To address the aforementioned challenges, this study introduces the Smooth ODE (SmODE). Initially, the research presents a smooth ODE neuron designed to estimate action rate changes near the current state. Our theoretical proof demonstrates that this computation effectively controls the maximum state transition between adjacent temporal neurons. We then developed a SmODE neural network incorporating these smooth ODE neurons, which reduces action fluctuations by integrating additional regularization terms into the original policy objective. The primary goal of this network is to enhance control output smoothness and function as versatile, plug-and-play policy approximator for broad range of RL algorithms.
The key contributions of this paper are the following:
We design a smooth ODE to function as a neuron of a NN for smooth control. This ODE neuron employs a mapping function to estimate the speed of change of the action in the neighborhood of the current state. Utilizing the estimated rate of change, is possible to efficiently moderate the extent of neuronal hidden state alterations contiguous time points, consequently reducing the difference in output from neighboring temporal neurons.
The SmODE network is developed by utilizing the smooth ODE as neurons. Our network comprises three modules: the input module, the smooth ODE module, and the output module. The input module is a multi-layer perceptron (MLP) network and the output module is lincar transformation layer, with spectral normalization applied. The smooth ODE module consists of three layers, and the number of smooth ODE neurons in each layer can be selected according to the task complexity. This design endows the SMODE network with disturbance rejection and smoothness capabilities.
We propose an SmODE-based RL algorithm designed to smooth action fluctuations. This algorithm incorporates the classical Actor-Critic architecture and integrates SMODE network as its policy network. Our method reduces action fluctuations by combining two regularization terms with the original policy objective, aimed at augmenting state filtering
2

---

Published as a conference paper at ICLR 2025
and controlling action fluctuation suppression in the SmODE network. In three-degree- of ff-freedom vehicle trajectory tracking task, our approach achieves an 81.7% reduction in action fluctuation rate, while preserving performance, compared to using traditional MLPs as the policy network, under a Gaussian noise variance setting of 0.2.
Supplementary experimental outcomes confirm that the SmODE architecture surpasses MLPs and LipsNet smoothing output while incurring negligible performance trade-offs. To accelerate adoption and further research, have encapsulated SmODE as Py module, with the code available in the attached files.
Section provides simple introduction online RI,, a metric for measuring the ratio of action fluc- tuation in control outputs, and introduction to neural ODE. In Section 3, new network architecture called SmODE is proposed, which includes smooth ODE neurons to smooth control outputs. The experimental results obtained from applying the proposed method are reported in Section 4. Section provides the conclusions of this paper.
PRELIMINARIES
ONLINE REINFORCEMENT LEARNING
Standard RL settings involve discrete-time agent-environment interactions, typically modeled as continuous- state and continuous-action Markov Decision Processes (MDP) Sutton & Barto (2018). Feedback is provided through a bounded reward function r($e, ae), and state transitions are determined by the probability p( ((+--1/2 dz). State-action pairs represented (s, a) for current and (s',a') for subsequent. The agent's actions at state are guided by stochastic policy π T assigning probabilities possible actions based on the current state.
In online RL, an agent learns and makes real-time decisions through interactions with its environment. transition, (st, Ge, T', St41), captures this interaction and stored in an experience replay buffer, R. During training, sampling from R. produces data batches, promoting stable model training. The primary goal of online RL is develop a policy that maximizes the expected cumulative return:
J= (3 ( y tr(s ai)
where 7 E (0, 1) represents the discount factor. The Q-value for a state-action pair (s, a) given by
Qss a) E, Σ r'r(s,,
s, 0,00
RL primarily uses an actor-critic architecture (2023), consisting of a policy function, æ. and corresponding Q-value function, Q". The policy iteration framework, used to derive the optimal policy x", alternates between policy evaluation and policy improvement. During policy evaluation, Qa is updated based on the self-consistency principle of the Bellman equation:
Q (s, = r(s, " (s"(s'
In the policy improvement phase, an enhanced policy Nnew is sought by optimizing current Q-value Quoia:
Tnew arg max E Bawd,
aw .wn Fu (s, a)]-
Practically, neural networks typically parameterize the policy and value functions, indicated as ma and Q3- These functions are honed using gradient descent techniques minimize the actor and critic loss functions, (0) and (o),), respectively, which are formulated based on equation 4 and equation 3.

---

Published as a conference paper at ICLR 2025
2.2 ACTION FLUCTUATION RATIO
To measure the action fluctuation of the control policy, Song et al. (2023) defined the action fluctuation ratio for continuous action settings:
T Σ a - a-11
<(TT = ETrope
(5)
T
where Ps is the state-action trajectory distribution induced by the policy π, T is the episode length, dg and 04-1 represent the action value the current and previous time steps, respectively. It can be observed that the control smoothness is negatively correlated with the action fluctuation ratio S(T).
2.3 NEURAL ODE
Neural ODE treats the computation of NN as a process of solving ODE, enabling the model to efficiently handle continuous-time sequence problems and describe its dynamics through differential equation methods. Chen et al. (2018) proposed that the hidden state of a neural ODE can be defined by the solution of
dx(()
f
(6)
dt
where x(t) represents the hidden states, [tt represents the input, represents time, f is a NN with parameter 6.
In control theory, a first-order low-pass filter Yuce & Minaci (2012) can be expressed in terms of an ODE
dx((() xx(1) T(I)
dt
where T a time constant of the system. larger value corresponds to a higher degree of filtering.
Instead of directly defining the derivatives of the hidden state using neural network f. more stable continuous- carimuouu-timeeee recurrent neural network can be employed by the following equation Funahashi & Nakamura (1993):
dx((t) dt
x(() + (x((),(()))
(8)
Hasani et al. (2021) proposed the liquid time-constant (LTC), further explored the impact of the ODE structure on representation performance and proposed replacing f ((t)), t, 0) in equation 8 with (a=)t), (t), (A xx))), where A represents learnable parameter.
Due to the reliance on advanced numerical ODE solvers, the training and inference speed of neural ODE is slow. This issue worsens the complexity of the data, tasks, and state space increases. To address this, Hasani et al. (2022) derived closed-form continuous-depth (CFC) model that preserves the modeling capabilities of ODE-based models without requiring a solver for data modeling.
SMOOTH ODE NETWORK
In this section, we first introduce the design of the ODE neuron. Then, we will describe the structure of the SmODE network in this paper. Following this, propose RL training approach devised to improve the smoothness of the policy while maintaining good control performance.
3.1 SMOOTH ORDINARY DIFFERENTIAL EQUATION
In order to address both the issue of figh-frreuency noise and the action non non-smoothness caused by unbounded Lipschitz constant, have designed the ODE follows.
To address the issue of figghfreeuency noise, we design the ODE with a low-pass structure, similar to equation 7. While the large time constant of the system ensures excellent action smoothness. it also introduces additional delay. These delays can significantly harm control performance when the system needs fast response. To address this issue, introduce a learnable function f(x(t), T(t), i, 0) that
4

---

Published as a conference paper at ICLR 2025
maps the input signal (t) and the neuronal hidden state x=t) to the inverse of the time constant 3 The equation is shown as
dx((1) -f ((),,, ((t)), r(t):
dt
where f is NN with parameter 0. Since the time constant must be positive number, the function f be greater than
The magnitude the Lipschitz constant can be controlled by constraining the size of d constraint must be state-dependent; otherwise, may impair performance in regions where certain systems require a faster response.
di and this
In this paper, replace I(L) on the far right side of equation with learnable function g(x((), T( 0), resulting in the following equation:
dx((t)
x(t) + (=()), (()))
(10)
Based equation 10, can draw the following theorem:
Theorem 1. Let 24 denote the hidden state of a neuron i within the smooth ODE, identified by equation 10, and let neuron i receive some incoming connections. Then, the hidden state of any neuron i, on finite interval Int E [O, is bounded as follows:
m, g(x((t, 0) S (t( S
)))
(11)
Proof. See Appendix A.I.
Theorem 1 suggests that g (I)) [(),, 1, 0), which we designed, guarantees that the hidden state of neuron remains bounded by equation 11 for finite time. Additionally, (z:(1), T(t), 1, 0) is state-dependent, allowing for the adaptive adjustment of the hidden state boundaries of neurons based on the current state. This conclusion is further supported by the analytical results presented in Appendix D.
Using bionic modeling method similar to that in Lechner al. (2020), we can obtain the specific formulation of our smooth ODE neuron, which is presented as follows:
dæ dt
Σ
Wij Cm.
oi (2z) 24
Way
(Iz) tanh(h (2,, 01) + Tleaki
(12)
Cm.
where wyy E (0.001, 1.0) denotes the synaptic weight from neuron to neuron j. and Cm, E (0.4, 0.6) signifies the membrane capacitance. The term Tieak, refers to the resting potential of a neuron. The sigmoid function a; (zz) 1+e 1+e=")yfy--ur is introduced, where "hy and Piy are trainable parameters
with initial values ranging from and 0.3 to 0.8, respectively. Furthermore, [(((1), T(1), expressed as was Ci o, (I,), and I(t) is equal to zj. g (z(t), t,t), 0) is equal to tanh(h(z,,0)) 0)).
representing a NN.
Based on equation 10, equation 11 and equation 12, we can also obtain the following theorem:
Theorem 2. Lel di denote the hidden state of a neuron i within the smooth ODE, identified by equation 10. Then, the absolute value of the derivative of the hidden state concerning time for any neuron i has an upper bound controlled by M Mxx()), T()), 0), as follows
dx:(t) < ((),4), C,
(13)
di
where (ax() 8) min [(), = M positive constant.
C a bounded
Proof. See Appendix A.2.
Theorem shows that the hidden state of smooth ODE neuron has an upper bound on the absolute value of the temporal derivative controlled by M (z(t), (t), i, 0); Therefore, we can suppress the value of dz((() by suppressing the value of M (x(1), T(O), (,);
5

---

Published as a conference paper at ICLR 2025
The nonlinear characteristies of semantics present challenges in deriving analytical solution for equation 12. As a result, we opt for a numerical ODE solver. To strike balance between computational efficiency, solution accuracy, and stability, we select the fixed time-step semi-implicit Euler discretization method Ethier & Bourgault (2008) to solve this equation. We can unroll a given dynamical system of the form dx(()
dr [=x((), sft + At)) by
(t + At) x(t) + At (x((), x (t+
(14)
Applying the fixed time-step semi-implicit Fuler discretization method to equation 12, we can obtain
(t) Cm, + Cm,
Ii (t + At)
Cml + Cm. + Ejenn wiyo; (z)(t)) Σ jehy Wiyo; (xz( tanh(h (ry (1))))))
(15)
+ +m.. + +E jet, Wayo, (t))
where In represents the set of neurons that have connections neuron i. During the training phase of solving ODE, we initialize the hidden states uniformly to zero. During the sampling phase of solving ODE, the hidden state is initially set to zero in the first sampling step, followed by using the hidden state value from the preceding step for subsequent initialization. In this study, the numerical ODE solver has iteration step size of 6 and discrete interval time of 1.
3.2 THE SMODE NETWORK ARCHITECTURE
To improve the smoothness of control outputs, we further introduce the SmODE network, employing the smooth ODE its neuron. It is applicable frameworks. The architecture of the SmODE is shown in Fig. I, which is structured with an input module, smooth ODE module, and output module. The input module MLP network, and the output module is a linear transformation layer, with spectral normalization applied. The smooth ODE module consists of three layers, and the number of smooth ODE neurons in each layer can be selected according to the task complexity.
policy network across wide range of RL
3.3 SMODE-BASED RL.
To facilitate smoother control, the SmODE network is utilized for parameterizing the actor in RL, represented by тe. where 0 denotes the respective network parameters. MLP is still utilized for parameterizing critic in RL, represented by Qs.
The magnitude of the [[x(t),((t)) t,)),, value indicates the extent of filtering; smaller value cor- responds to a higher degree of filtering, thereby more effectively suppressing high-frequency noise interference. Therefore, we add the coefficient f (at(),, T(t), 1, 0) associated with the filtering as a regularization term. The function (an)()(x(t).. T(t), 0)) regulates the range of values for the hidden state of the neuron; a smaller absolute value of h(x((t, T(t), r) results in more pronounced inhibition of the magnitude change of hidden state across neighboring time steps. Therefore, the coefficient A2 (z(t), (t), t, 0) is associated with the hidden state boundary value as a regularization term. Both regular terms are added to the original RL training loss. The modified actor loss is
N 12²()
min Lx E
+
Σ + AORER
A2(-)
(16)
where X and A are the regularization factors, R is the replay buffer, the first regularization term is named the time constant term, the second regularization term named state boundary term, and N the number of smooth ODE neurons of the SMODE network. The pseudocode of SmODE-based RI. is illustrated in Algorithm I.
EXPERIMENTS
4.1 EXPERIMENTAL ENVIRONMENT
In this study, ten types of experimental environments are adopted to validate the efficacy of the SmODE network: vehicle trajectory tracking task, linear quadratic regulator problem, and eight

---

Published as a conference paper at ICLR 2025
Boundary eniarges
Xy1 (t)
Neuron(i)
Neural state
(0)+0)))))))
x00
Dynamics change slowly
112(1)
POX
Boundary decreases
Sp(())
x,( 1)
(()))).((t)c...))
x,(0) a
Dynamics change quickly
(a) Neuron model
x(te) x(tx-1)
Numerical ODE solver
Control
States
100
x(ty-1)
actions
s())
Input
Output Module
a(t)
Module
Smooth Module
(b) The SmODE network architecture
Figure 1: Designing SmODE network with smooth ODE neuron. (a) The neural state, xz((), of smooth ODE neuron integrates inputs from neurons j2, j3 and its previous state. The system dynamics, 9 (x(t), (t)), f, 0), allow for adaptive state boundary adjustments during the solving process by the numerical ODE solver, akin to feedback control mechanism. (b) The SmODE network consists of an input module, smooth ODE module, and output module. The input module MLP network, while the output module features a linear transformation layer with spectral normalization applied. The smooth ODE module contains three layers, with the number of neurons in each layer tailored to the task's s complexity. I' denotes the number of iterations performed by the numerical ODE solver.
robotic control tasks in Mujoco Todorov et al. (2012). All experiments were conducted on eight AMD Ryzen Threadripper 3960X 24-core processors with 128G of RAM each. The time required for Mujoco tasks with an average training step length of 1 million is 14h.
Vehicle trajectory tracking is a significant problem in autonomous driving. We simulated the motion of the vehicle using the vehicle dynamics model proposed by Ge et al. (2021). Furthermore, we chose an LQR problem with two states and one action as an ablation experiment task. Detailed introductions to the two experimental environments are provided in the Appendix C.
Mujoco is benchmark RI. environment that integrates several robot control tasks. The specific simulation tasks, depicted in Fig. 4, include Humanoid, Pusher, Hopper, Reacher, Walker2d, Ant, InvereedDoubleeenduluum and CarRacing.
We will use the following two types of RL algorithms. Infinite-time approximate dynamic program- ming LINFADP)Li (2023) typical model-based RL algorithm. Distributional soft actor-crite critic (DsAC)Duan et al. (2021) is typical model-free RL algorithm. All experiments were conducted in general optimal control problem solver Wang et al. (2023), and the results are averaged over five random seeds.
7

---

Published as a conference paper at ICLR 2025
04
Laken
130
1.0
125
3..0
Figure 2: Results in vehicle trajectory tracking environment. In this experiment, MPC operates without adding noise, and its control outcomes will serve as a benchmark for the optimal policy. On the first line the result of the sine curve, and on the second line is the result of the double lane-change curve.
4.2 TEST RESULTS ON VEHICLE TRACKING PROBLEM
Table 1:
Performance analysis of tracking double lane-change using the INFADP algorithm. Results are expressed as mean + standard deviation of five independent environmental seeds.
Gqussian standard deviation
Policy network
0.10
SmODE E(8)
TAR
TAR
TAR
E(x)
TAR
0.110=0.013
00830=00026
41876211.012
4.926cH0.025
MLP
0.312:0.022
-1.111a0.038
0.64550.038
01.725=0.049
0.84240.047
2.26620.000
0.964220059
-3.08720.047
LTC
0.39210:020
-1.281200331
0.58910.021
-1884820.046
0.77810.034
-2.33420.036
LipsNet MPC
0.06920.013
-0.893+0.018
0.12:00.019
-0.934=0.000
0.169#0.019
-0.987=0.021
0.221=0.027
-1.112+0.033
0.238=0.008
.08114400013
0.439+0.010
-12248=0017
0.55440.015
-15509+0.028
0,706=0.022
-2225240.026
We illustrate our approach by tracking the double lane-change curve, employing five distinct methods: INFADP with MLP, INFADP with SMODE, INFADP with LipsNet, INFADP with LTC and MPC Holkar & Waghmare (2010). Table displays the performance metrics for these methods, noting that Gaussian noise the noise type used. In this context, TAR denotes the total average return, and s(x) represents the action fluctuation ratio.
In four distinct noise environments with varying levels, our algorithm consistently outperformed others. As shown in the table, SmODE, acting as policy network, significantly reduces the action fluctuation ratio and enhances the TAR compared MLP. Notably, with Gaussian noise variance of 0.2, our network lowers the action fluctuation ratio by about 81.7%. demonstrating superior smoothness. The results for the first and third algorithms indicate that our neural network architecture, combined with modified actor loss, greatly mitigates the action fluctuation rate. Moreover, in noisy environments, our approach exceeds the recent LipsNet enhancement performance and proves more effective than the classical MPC controller.
This resilience is largely due to the low-pass filtering effect and SmODE's ability to suppress its Lipschitz constant. Given the common presence of noise in real-world settings, SmODE's robustness in noisy environments is of significant practical value.
Furthermore, with Gaussian noise variance of 0.05, our analysis of experimental results using MLP and SmODE as policy networks for tracking sine and double lane-change curves shows notable differences. We used the MPC algorithm a baseline for noise-free comparison. As illustrated in Fig. 2, SmODE not only exhibits lower action fluctuation ratio than MLP but also smaller variations in lateral velocity, enhancing vehicle comfort and safety.
8

---

Published as a conference paper at ICLR 2025
4.3 TEST RESULTS ON MUJOCO BENCHMARK
In this experiment, we focused on eight robotic control tasks within the Mujoco environment. We employed DSAC Duan et al. (2021) as the fundamental RL algorithm, configuring the policy networks as MLP, LipsNet, LTC, and SMODE. The assessment was performed under two levels of Gaussian noise to mimic various real-world conditions. Since the state values of different Mujoco tasks vary greatly, we set two levels of Gaussian noise for the eight tasks, as shown in Table 2.
Table 2: Variance of different levels of Gaussian noise for different Mujoco tasks.
Noise level
Iivete//DuutccPenhtlumaa
Reacher-v2
Humanoid-va
Pusher-v2
Hopper-v3
Walker2d-v3
Ant-v3
CarRacing-vl
lovel 2
0.005
0.050
0.020
0,050
0,050
0.050
0.050
0.150
0,015
0,100
0,050
0.100
0.100
0.070
0,250
For the whole task, noise is added to states. The results, which are the averages of five seeds over I million training steps, are shown in Table 3.
Under different levels Gaussian noise, SmODE, functioning as a policy network, achieved the lowest average action fluctuations compared to LTC, LipsNet, and MLP. Additionally, SmODE exhibited the best performance in most Mujoco tasks. Given that the pursuit of action smoothness and high performance was not achieved all experimental settings. Moreover, we also experimented with TD3 Fujimoto al. (201 8) in the Walker2d-v3 and Ant-v3 environments and obtained similar results, shown in Appendix E.
be somewhat contradictory, it is understandable that the best performance
Table Average control performance of SMODE. LTC, LipsNet, and MLP for different Gaussian noise levels, where level is on the left column and level 2 is on the right column. The average action fluctuation rate is indicated in parentheses. Results are expressed as mean + standard deviation of five independent environmental seeds.
Network structure
InverteDDeahleeeenbllnss
Reacher-v2
SMODE
9357=2 (0.15)
9340=2 (0.44)
-5.67#1 (0.22)
-9.16=1 (0.32)
LTC
935512 (0.25)
933613 (0.64)
-6.0912 (0.31)
-10.29113 (0.42)
LipsNet MLP
0357+2 (0.20)
9338=2 (0.50)
-5,94+1 (0.26)
-9.8422 (0.39)
935742 (0.27)
933524 (0.68)
-5.7343 (0.30)
-10.4943 (0.44)
Network structure
Humanoid-v3
Pusher-v2
Hopper-v3
SmODE
10819=81 (0.45)
10746=101 (0.50)
-40=1 (0.90)
-51#1 (1.39)
3265#232
(0.70)
2532=302 (1.01)
LTC
10626+128 (0.60)
10578=245 (0.66)
.4:2 (1.51)
-86=8 (2.30)
2724+287 (0.88)
1398+345 (1.25)
LipsNet
10872=89 (0.57)
10715=104 (0.62)
.4+22 (1.23)
-55=3 (2.03)
2905+301 (0. 84)
1787429] (1.21)
MLP
10892+342 (0.62)
10567=512 (0.69)
-4923 (2.01)
-71 (2.60)
1282+322 (0.93)
1108+231 (1.30)
Network structure
Walker2d-33
Ant-v3
CarRacing-v1
SmODE
6039:112 (0.94)
5037:114 (1.30)
3564:184 (1.68)
1677241 (1.93)
916:27 (0.83)
873:14 (0.96)
LTC
58611482 (1.10)
2352a604 (1.71)
2872-341 (2.04)
10841298 (2.16)
(0.87)
694s116 (1.03)
LipsNet
6032+238 (1.05)
49812423 (1.45)
3721=212 (1.93)
1532:109 (2.10)
896231 (0.85)
821:56 (1.00)
56631508 (1.21)
1597a815 (1.80)
1086:1246 (2.16)
197=120 (2.30)
870:38 (0.88)
751a86 (1.05)
4.4 ABLATION STUDY
To demonstrate how the time constant and state boundary regularization terms contribute to the final smoothing action, we conducted ablation experiments. We still use the model-based RL method, INFADP Li (2023), to train in this environment. All ablation experiments are performed on the linear quadratic regulation problem for two-dimensional states and -ie-dimenioonl action.
The following are the specific designs of three ablation experiments: 1) SMODE w/o time constant term: ablation time constant regular term. This ablation experiment aimed to validate the impact of incorporating the time constant of the system as regular term in actor loss on the smoothing of action output. 2) SmODE w/o state boundary term: ablation regular term for the boundary of neuron states. This ablation experiment involved removing the regular term from the actor loss, a term that adaptively controls neuron state boundaries by predicting the rate of change in actions near the current state. 3) Baseline: ablation the both regular terms. This ablation experiment simultancously removes the two regular terms added to the actor loss and replaces the MLP with a neural ODE network only.

---

Published as a conference paper at ICLR 2025
SmiO0E SeiO0E Floadliie
0-
TOOh overage retur
04
-000
0.3
-15
811
Be220 conmart tarn Bex00 boundary lomn
Beacino
00
-29
Nobee
(a) Action fluctuation ratio
(h) Total average
Figure 3; Results in the LQR environment. The X-axis is the noise variance.
In Fig. 3, we present results from three ablation studies in the LQR environment using the INFADP algorithm. We introduced various levels of uniform noise into the observed state. The results show that compared the baseline, both regularization terms effectively reduce the action fluctuation ratio and increase the total average reward, with the state boundary regularization term having particularly notable impact. Notably, with noise variance of 0.6, SmODE decreases the action fluctuation rate by 12% and boosts the total average return by 79%. As noise levels increase, SmODE shows slower rise in action fluctuation ratio and a more gradual decrease in total average return compared to the baseline neural ODE network, highlighting its superior noise resistance and smoothing capabilities.
In addition, we conducted ablation experiments on whether the output module used spectral normal- ization (SN) techniques in the Walker2d and Humanoid tasks, with the experimental results shown in Table 4.
Table 4: Control performance of different network structures under different Gaussian noisy variance. The average action fluctuation rate is indicated parentheses.
Network structure
Walker2d-v3
Humanoid-v3
SmODE 6039+112 12 (0.73)
5037 (1.03)
10819481 (0.45) 10746+101 (0.50)
SmODE-wo-SN
6013+202 (0.80)
5165+164 (1.12)
10821 (0.48) 10739=122 (0.54)
The experimental results indicate that using SN techniques can further reduce action fluctuation, with minimal impact overall performance. It is worth noting that the reduction in action fluctuation due to the use of SN techniques relatively small compared to the overall decrease.
CONCLUSION
In this study, we introduce the SmODE network to tackle non-smooth action outputs deep reinforcement learning. The network features smooth ODE as a key component of its neurons, enabling adaptive state boundary adjustments and low-pass filtering. This design grants the neurons disturbance rejection and smoothness capabilities. As policy network, SmODE enhances control output smoothness and increases average rewards various RL algorithms over MLP and LipsNet. We hope contributions advance real-world RL applications.
LIMITATION
Solving the Neural ODE using numerical methods requires N iterations. In our case, we balanced solution accuracy and computational efficiency by adopting = 6, a value commonly used in previous related work. As result, the training time for SMODE increases by factor of 2 to compared to MLP, which is an issue that needs to be addressed. In future work, we plan to explore the latest techniques for training neural ODE networks to accelerate the backpropagation process.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGEMENTS
This study is supported by National Key R&D Program of China with 2022YFB2502901. It is also partially supported by Tsinghua University-Toyota Joint Research Center for AI Technology of Automated Vehicle, and supported by Tsinghua University Initiative Scientific Research Program.
REFERENCES
Thomas Asikis, Lucas Böttcher, and Nino Antulov-Fantulle Neural ordinary differential equation control of dynamics on graphs. Physical Review Research, 4 4(1):013221, 2022.
Badong Chen, Lujuan Dang, Nanning Zheng, and Jose C Principe. Kalman filtering. In Kalman Filtering Under Information Theoretic Criteria, pp. 11-51. Springer, 2023.
Chen Chen, Hongyao Tang, Jianye Hao, Wulong Liu, and Zhaopeng Meng. Addressing action oscillations through learning policy inertia. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 7020-7027, 2021.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in Neural Information Processing Systems, 31, 2018.
Fred Daum and Jim Huang. Particle degeneracy: root cause and solution. In Signal Processing, Sensor Fusion, and Target Recognition XX, volume 8050, pp. 367-377. SPIE, 2011.
Jingliang Duan, Yang Guan, Shengho Eben Li, Yangang Ren, Qi Sun, and Bo Cheng. Distributional soft actor-critic: Off-policy reinforcement learning for addressing value estimation errors. IEEE Transactions Neural Networks and Learning Systems, 33 33(11):658446596 2021.
Mare Ethier and Yves Bourgault. Semi-implicit time-discretization schemes for the bidomain model.
SIAM Journal on Numerical Analysis, 46(5):2443-2468. 2008.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference on Machine Learning, pp. 1587-1596. PMLR, 2018.
Ken-ichi Funahashi and Yuichi Nakamura. Approximation of dynamical systems by continuous time recurrent neural networks. Neural Networks, 6(6):801-806, 1993.
Qiang Ge, Qi Sun, Shengbo Eben Li, Sifa Zheng, Wei Wu, and Xi Chen. Numerically stable dynamic bicycle model for discrete-ting time control. In 2021 Intelligent Vehicles Symposium Workshops (IV Workshops), pp. 128..34. IEEE, 2021.
Henry Gouk, Eibe Frank, Bernhard Pfahringer, and Michael J Cree. Regularisation of neural networks by enforcing lipschitz continuity. Machine Learning, 10:3933446,, 2021.
Yang Guan, Yangang Ren, Qi Sun, Shengbo Eben Li, Haitong Ma, Jingliang Duan, Yifan Dai, and Bo Cheng. Integrated decision and control: Toward interpretable and computationally efficient driving intelligence. IEEE Transactions Cybernetics, 53(2):859-88) 2022.
Ramin Hasani, Mathias Lechner, Alexander Amini, Daniela Rus, and Radu Grosu. Liquid time- constant networks. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 7657-7666, 2021.
Ramin Hasani, Mathias Lechner, Alexander Amini, Lucas Liebenwein, Aaron Ray, Max Tschaikowski, Gerald Teschl, and Daniela Rus. Closed-form continuous-time neural networks. Nature Machine Intelligence, ((11):92211003 2022.
KS Holkar and Laxman M Waghmare. An overview of model predictive control. International Journal of Control and Automation, 3(4):47-63, 2010.
Elia Kaufmann, Leonard Bauersfeld, Antonio Loquercio, Matthias Müller, Vladlen Koltun, and Davide Scaramuzza. Champion-lewe drone racing using deep reinforcement learning. Nature, 620 (7976)99229987) 2023.
11

---

Published as a conference paper at ICLR 2025
Taisuke Kobayashi. L2c2: Locally lipschitz continuous constraint towards stable and smooth reinforcement learning. In 2022 International Conference on Intelligent Robots and Systems (IROS). pp. 4032-4039, IEEE, 2022.
Mathias Lechner, Ramin Hasani, Alexander Amini, Thomas A Henzinger, Daniela Rus, and Radu Grosu. Neural circuit policies enabling auditable autonomy. Nature Machine Intelligence, 2(10): 642-652, 2020.
Shengbo Eben Li. Reinforcement Learning for Sequential Decision and Optimal Control. Springer Verlag, Singapore, 2023.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv: /00..008000555555 2018.
Artem Molchanov, Tao Chen, Wolfgang Hönig, James A. Preiss, Nora Ayanian, and Gaurav Sukhatme. Sim-to-(multi)-renl Transfer of low-level robust control policies to multiple quadrotors. In 2019 International Conference on Intelligent Robots and Systems (IROS), pp. 59-66. IEEE, 2019. doi: 10.1 L00.////R00000701199999777777770
Siddharth Mysore, Bassel Mabsout, Renato Mancuso, and Kate Saenko. Regularizing action policies for smooth control with reinforcement learning. In 2021 International Conference on Robotics and Automation (ICRA), pp. 1810-1816. IEEE, 2021.
Baiyu Peng, Qi Sun, Shengbo Eben Li, Dongsuk Kum, Yuming Yin, Junqing Wei, and Tianyu Gu. End-to-end autonomous driving through dueling double deep q-network. Automotive Innovation, 4:328-337, 2021.
Domènec Ruiz-Balet and Enrique Zuazua. Neural ode control for classification, approximation, and transport. SIAM Review, 65(3):735-773, 2023, doi: 10.1137/1IM141 1433.
Anton Maximilian Schäfer and Hans Georg Zimmermann. Recurrent neural networks are universal approximators. In Artificial Neural -CCNN 2006: 16th International Conference, Athens, Greece, September 10- 14, 2006. Proceedings, Part I 16, pp. 632-640. Springer, 2006.
Qianli Shen, Yan Li, Haoming Jiang, Zhaoran Wang, and Tuo Zhao. Deep reinforcement learning with robust and smooth policy. In 2020 International Conference on Machine Learning (ICML., pp. 8707-8718. PMLR, 2020.
Guanya Shi, Xichen Shi, Michael O'Connell, Rose Yu, Kamyar Azizzadenesheli, Animashree Anandkumar, Yisong Yue, and Soon-Jo Chung. Neural lander: Stable drone landing control using learned dynamics. In 2019 International Conference Robotics and Automation (ICRA), pp. 9790- IEEE, 2019.
Xujie Song, Jingliang Duan, Wenxuan Wang, Shengbo Eben Li, Chen Chen, Bo Cheng, Bo Zhang, Junqing Wei, and Xiaoming Simon Wang. Lipsnet: A smooth and robust neural network with adaptive lipschitz constant for high accuracy optimal control. In 2023 International Conference on Machine Learning (ICML). PMLR, 2023.
Sho Sonoda and Noboru Murata. Neural network with unbounded activation functions is universal approximator. Applied and Computational Harmonic Analysis, 43(2):233-268, 2017.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In Intelligent Robots and Systems, 2012.
Wenxuan Wang, Yuhang Zhang, Jiaxin Gao, Yuxuan Jiang, Yujie Yang, Zhilong Zheng, Wenjun Zou, Jie Li, Congsheng Zhang, Wenhan Cao, et al. Gops: A general optimal control problem solver for autonomous driving and industrial control applications. Communications in Transportation Research, 3:100096, 2023.
Yinuo Wang, Shujuan Liu, Jingyuan Zhou, and Tengxuan Sun. Particle filtering based biome intelligence algorithm. 2021 International Conference on Security, Pattern Analysis, and Cybernetics (SPAC), pp. 156-161. IEEE, 2021.
12

---

Published as a conference paper at ICLR 2025
Yinuo Wang, Likun Wang, Yuxuan Jiang. Wenjun Zou, Tong Liu, Xujie Song, Wenxuan Wang, Liming Xiao, Jiang Wu, Jingliang Duan, et al. Diffusion actor-critic with entropy regulator. arXiv preprint ar 4rXiv:2405.15177, 2024.
Asanka Wasala, Donal Bymee Philip Miesbauer, Joseph O'Hanlon, Paul Heraty, and Peter Barry. Trajectory based lateral control: A reinforcement learning case study, Engineering Applications of Artificial Intelligence, 94:103799, 2020.
Haonan Yu, Wei Xu, and Haichao Zhang. Taac: Temporally abstract actor-critic for continuous control. Advances in Neural Information Processing Systems, 34:29021-29033, 2021.
Erkan Yuce and Shahram Minaei. Derivation of low-power first-order low-pass, high-pass and all-pass filters. Analog Integrated Circuits and Signal Processing, 70:151-156, 2012.
Zhigen Zhao, Simiao Zuo, Tuo Zhao, and Ye Zhao. Adversarially regularized policy learning guided by trajectory optimization. Learning for Dynamics and Control Conference, pp. 844-857. PMLR, 2022.
13

---

Published as a conference paper at ICLR 2025
A THEORETICAL RESULTS
A.1 PROOF OF THEOREM 1
Theorem 1 Let 2; denote the hidden state of neuron i within the smooth ODE, identified by equation 10, and let neuron receive some incoming connections. Then, the hidden state of any neuron i, on finite interval Int E [0, T), is bounded as follows:
min(0, ((x(()()))))))))))))))) ((), m) S (re()) & max(0,.g((()) T(0),,0);
(17)
Proof. Let us insert M = max(0, (.0gg()))) (.):uux) the neural state of neuron i, xz(() into equation 10:
da
dt = - f(x; (t), + f(x, (t) g(x, (t),t,0) 0).
(18)
The right-hand side of equation 18 is negative, considering the constraints on M, the positivity of weights, and the fact that / (xx, is positive. Consequently, the left-hand side must also be negative. Employing an approximation the derivative term yields the following relationship:
dx; dt S
dæ =
z,(t At) At
xz(t) 00
(19)
dt
By substituting (t) with M, we get:
x(t + At)
M
0 - x(t + At) S M,
(20)
At
which means (1) S max (0, maxx.
We can also obtain similar results min(0, g(-)rin)) < (1)
A.2 PROOF OF THBOREM 2
Theorem 2 Let H; denote the hidden state of a neuron i within the smooth ODE, identified by equation 10. Then, the absolute value of the derivative of the hidden state concerning time for any neuron has an upper bound controlled by M (xx(1) T(),, 1, 0)., as follows
dr.
< (zz((),
(21)
dt
where max((g(<<()) ((1,,4,) () (() = ((),4) C' a bounded positive constant.
Proof.
dx(()
dt
(x)() - (4000000000000000000000000000000
dt
St00://00.000K000000000000000000 According to equation 17
s T()40)
where (()),,, Cm,
wii
sigmoid((), wij E (0.001, 1.0), Cm, E (0.4, C' is bounded
positive constant.
The output module is simple layer of linear mappings wr + b, with spectral normalization
applied, so there da(t) dz(()) holds.
14

---

Published as a conference paper at ICLR 2025
M (()()) C'
= 1 dd((t)
dt
where C" is a bounded positive constant.
B PSEUDOCODE
Algorithm Training method SmODE-bart RL.
Input: 8, o, A1. λ2, 3g. 3 for each iteration do
Collect a batch of samples (s, a, r. 8') with policy me
Store the samples in replay buffer
for each update step do
Sample data from R
Update actor using 0 t 0 - Vo
Update critic using o t 16-0-pp o. (0) end for end for
C EXPERIMENTAL ENVIRONMENT INTRODUCTION
C.I VEHICLE TRAJECTORY TRACKING ENVIRONMENT
Table 5 provides detailed descriptions of the states and actions in the vehicle trajectory tracking task.
Table 5: List of states and actions
Varible
Description
Uint
longitudinal position
lateral position
m
heading angle
rad
State
u
longitudinal velocity
m/s
lateral velocity
m/s
w yaw rate at center of gravity (C.G.)
rad/s
longitudinal acceleration
m/s2
Action
front wheel angle
rad
In the vehicle trajectory tracking experiment, selected sine and double lane-change curves for tracking. The reward is designed
0.04 (x 2ret) 0.04 (y Bret) 2
-0.02 (yp - pret) 0.02 (u Usel 2
(22)
-0.01 0.015 0.01a²
where Fref, lief, SPref, Mref represent reference states.
The vehicular parameters listed in Table 6, where C.G. means the center of gravity.
15

---

Published as a conference paper at ICLR 2025
Table 6: Vchicular parameters
Parameter
Description
Value
m
mass of the vehicle distance between C.G. and front axle
1412 kg
4
1.06 m
distance between C.G. and rear axle
1.85 m
front axle equivalent sideslip stiffness rear axle equivalent sideslip stiffness yaw inertia of vehicle body control frequency
128916 N/rad
kir
-85944 N/rad
I
1536.7 kg -m²
f
10 Hz
C.2 LINEAR QUADRATIC REGULATION PROBLEM
The state-space equation is
X AX + BU,
(23)
where
0 1
B 00
1
A =
(24)
0 0
The reward is designed as
G - x, -X, RU,,
(25)
where
Q (2 R =
(26)
diag means a diagonal matrix.
C.3 MUJOCO ENVIRONMENTS
Mujoco serves as a benchmark RL environment comprising various robot control tasks. The specific simulation tasks, shown Fig. 4, include Humanoid, Pusher, Hopper, Reacher, Walker2d, Ant, Inverted Double Pendulum, and Car Racing.
D LANDSCAPE OF AVERAGE [h(z,, 0)
In the LQR problem used for the ablation experiments, plot the average values of [h(2,, 0) in SmODE, as shown in Fig. 5. Notably, [h(x,, 0) exhibits larger values around the states (-1,-1] and (1, 1). as these states indicate a departure from the steady state (0, 0). Consequently, the Lipschitz constant may be larger, necessitating an increase in the range of values for the hidden state of the neuron.
E TD3 WITH SMODE
In order to demonstrate the smoothing ability of SmODE in other RI algorithms, we experimented with TD3 example in Walker2d-v3 and Ant-v3 environments, as shown in Table 7.
Table Control performance of different network structures under Gaussian noisy variance of 0.05 (left column) and Gaussian noisy variance of 0.1 (right column) conditions. The average action fluctuation rate indicated parentheses.
Network structure
Walker2d-v3
Ant-v3
SmODE
39624361 (0.87) 3504+773 (1.11)
4158+524 (1.01)
3857+754 (1.67)
LipsNet
3578+392 (0.95)
3226#623 (1.32)
4002+531 (1.24)
3398+482 (1.88)
MLP 3226+360 (1.08)
2063+520 (1.60)
38524227 (1.72)
862+242 (2.19)
16

---

Published as a conference paper at ICLR 2025
a
(c)
(d)
0064
(c)
(f)
g
hh)
Figure 4: Simulation tasks. (a) Humanoid-v3:($ X a) E R376 R7 (b) Pusher-v2: (s x a)
2 x R? (c) Hopper-v3: (s x a) E R x R (d) Reacher-v2: (s x a) E R x R² (c) Walker2d-
v3: (s x
IR'" x R6 (f) Ant-v3:
(s a) Bill RR. (g) InveredDDabbeePendulunyy
x a) E Rll X (h) CarRacing-vl:
(s a) 199696633 x (image-input).
Average value of [h(x,, 8)]
0.40
0.35
0.30
-0.55
0.20
0.15
-1.00
-0.75
0.50
0.25
0.00
0.25
0.50
0.75
1.00
Figure Landscape of average [Mx,,, 0)}.
TRAINING DETAILS
In Mujoco tasks, hyperparameters unrelated to SmODE were consistent with those in the DSAC paper. The parameters that needed adjustment were only A₁ and Ay, as well as the number of neurons in the three-layer network of the smooth ODE module. The variables ê and X were adjusted using a controlled variable method to find the relatively optimal results. The configuration the neuron numbers in the smooth ODE follows the rule that the number of neurons in the second and third
17

---

Published as a conference paper at ICLR 2025
layers equals the dimensionality of the environment actions, and the number of neurons in the first layer is greater than that of the latter two layers.
F.1 TRAINING DETAILS ON VEHICLE TRAJECTORY TRACKING ENVIRONMENT
We employ the infinite-time approximate dynamic programming (INFADP) Li (2023), model- based RL algorithm, for training in the vehicle trajectory tracking environment. We use the same hyperparameters for sine and double-line scenarios. The hyperparameters of INFADP are listed in Table 8.
Table 8: Algorithm hyperparameter
Parameter
Setting
Replay buffer capacity
1000000
Buffer warm-up size
1000
Batch size
64
Discount y
0.99
Target network soft-update rate
0.2
Initial random interaction steps
Interaction steps per iteration
Network update times per iteration Prediction step
10
Action bound
[-0.4, 0.4]
Exploration noise std. deviation
0.2
Hidden layers in input module
[64, 64]
Numbers of adaptive ODE neurons in each layer
[4, 2, 2]
Hidden layers in critic network
[64, 64]
Activations critic network Optimizer
ReLU
Adam
Actor learning rate
1 10
Critic learning rate
1 10-03
Weight Å
2 10-2
Weight X2
2 2.10-3
F.2 TRAINING DETAILS ON LQR
The classical optimal control problem is characterized linear quadratic regulation (LQR) problem with two-dimensional states and one-dimensional action. We use the INFADP algorithm to train this environment. The hyperparameters of INFADP are listed in Table 9.
F.3 TRAINING DETAILS ON MUJOCO TASKS
Mujoco Todorov et al. (2012) is a simulation engine designed primarily for research in RL and roboties. It provides versatile and physics-based platform for developing and testing various RL algorithms. Core features Mujoco include a highly efficient physics engine, realistic modeling of dynamic systems, and support for complex articulated robots. Currently, one of the most recognized benchmark environments for RL and continuous control.
We distributional soft actor-critic (DSAC)Duan et al. (2021), model-free RL algorithm train these eight robot control tasks. The hyperparameters of DSAC are listed in Table 10. The weights \1, Ag and numbers of smooth ODE neurons of each layer are listed in Table 11.
G NETWORK MODULES AND TEST WEIGHT SELECTION METHODS
The output module is primarily used for complex control tasks. If the selected control task is relatively simple and has a small action space, such as the www-dde-oo--ffeeeeeeeee vehicle trajectory tracking control task in this paper, the output module is unnecessary. Additionally, the final test weights should
18

---

Published as a conference paper at ICLR 2025
Table Algorithm hyperparameter
Parameter
Setting
Replay buffer capacity
1000000
Buffer warm-up size
1000
Batch size
Discount Y
0.99
Target network soft-update rate
0.2
Initial random interaction steps
Interaction steps per iteration
Network update times per iteration Prediction step
Action bound
[-5, 5]
Exploration noise std. deviation
0.2
Hidden layers input module
[64, 64]
Numbers of adaptive ODE neurons in each layer
[2, 1, 1]
Hidden layers critic network
664, 64]
Activations in critic network
ReLU
Optimizer
Adam
Actor learning
3 10-5
Critic learning rate Weight Å
8 10
2 10
Weight A2
2 10-3
Table 10: Algorithm hyperparameter
Parameter
Setting
Replay buffer capacity
1000000
Buffer warm-up size
10000
Batch size
256
Discount 7
0.99
Initial alpha
0.27
Target network soft-update rate 7
0.005
Initial random interaction steps Interaction steps per iteration
0
Network update times per iteration Prediction step
Action bound
[-1, 1]
Convolution kernel sizes (CarRacing)
[4, 3, 3.
Convolution channels (CarRaeing) Convolution strides (CarRacing) Convolution activation (CarRacing)
[8. 16, 32, 64, 128, 256]
2, 2, 2, 1,
ReLU [256, 256, 256]
Hidden layers in input module Hidden layers critic network Activations in critic network
[256, 256, 256]
GeLU
Policy act distribution Policy min log std Policy max log std Policy delay update Optimizer
TanhGauss
-20
0.5
Adam
Actor learning rate
1 10 -4
Critic learning rate
1 10
Alpha learning rate
3 10
Target entropy
dim (A)
19

---

Published as a conference paper at ICLR 2025
Table 11: Weight A1, 22 and numbers of smooth ODE neurons of each layer on Mujoco
Env
weight À
weight X2
Numbers of smooth ODE neurons
Humanoid-v3
1-100-2 2
10-2
[20 17 17]
Pusher-v2
1.10-3
1 10-2
[10 7]
H0pper-v3
1.10-3
1.10-2
[6
Reacher-v2
1.10-2
1.10-2
[422]
Walker2d-v3
1.10-2
1.10-2
[1066]
Ant-v3
1.10-5
1.10-3
[10 8]
InverredDoubbeelendtuu--3
1.10-2
1.10-2
[2 1]
CarRacing-vl
1.10-3
1-10-a
[4 2
be selected based on comprehensive evaluation of strategy performance and action smoothing inhibition ability, rather than simply adopting the training weights from the last iteration.
20


Reviews:
[{"review_id": "jvhpS0131Y", "reviewer": "Reviewer_6QXf", "summary": "To adress the issue of smoothness in a policy in RL the authors propose an ODE-based approach to perform a low-pass filtering of the methods.", "questions": ".", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper deals in an important area of research: finding stable control policies (where smoothness is one aspect) is a relevant area of research\n- the derivations are sound and the concepts are explained in a clear way\n- overall the paper is written quite well\n- the authors perform experiments on different domains\n- relevant work is mentioned as far as I can tell", "weaknesses": "- One key problem is that the authors motivate their method e.g. by: \" Filtering methods like Kalman and extended Kalman filtering Chen et al. (2023) effectively suppress noise and reduce output oscillation by estimating the current state from multi-step historical data. These methods work well with Gaussian noise but struggle with non-Gaussian noise.\" \n\nHowever, in the experiment the authors only test on settings with Gaussian noise:\n  - MuJoco:  Table 2,3 and 4 specify a Gaussian noise level\n  - vehicle trajectory tracking environment: it is unclear what the noise shape is here (perhaps its partial-observable and thus exhibits non-Gaussian noise). In that case the authors should perform an analysis of the shape of stochasticity in this benchmark. \n\nEither way: I would advise either testing again a Filtering method, such as an extended Kalman filter, re-designing the experiments under non-Gaussian settings and/or clearly present how the vehicle trajectory tracking is a RL problem with non-standard noise.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5lrDatsiGg", "reviewer": "Reviewer_kjCQ", "summary": "This paper introduces a novel neural ODE-based architecture designed for reinforcement learning tasks, specifically addressing the problem of action fluctuation. The authors use ODEs as low-pass filters on network hidden states, with provided theoretical justifications. Their method controls the network's Lipschitz constant in a state-dependent manner, allowing for large actions when needed and smooth actions otherwise. The work demonstrates improved performance over state-of-the-art methods through comprehensive experimentation and ablation studies.", "questions": "How does this work relate to ODE-based Recurrent Model-free RL for POMDPs (https://arxiv.org/abs/2309.14078)?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "1. The novel approach of integrating smoothing into neural ODEs for resolving action fluctuation is an interesting idea.\n2. Having state-dependence is shown to be important for performance, which is important knowledge for the community.\n3. The comprehensive ablation studies show that the individual choices made are important for the performance of the method.\n4. Testing with multiple noise levels on mujoco tasks is a good way of benchmarking such methods, and the results show clear improvements over existing techniques.\n5. The authors provide proofs showing how their method can bound the Lipschitz constant.\n6. The authors describe how their ODE integration is performed carefully making choices so that the method is not too expensive for practical use.", "weaknesses": "## Major issues\n\n1. Limitations discussion hidden in appendix instead of main paper under future work\n3. Neural ODE section explanation is unclear, particularly \n4. Notation used is often confusing or partially defined:\n    - Equation 12 has the variable j which is never quantified, I believe there is a missing summation\n    - some undefined terms such as g(.)^max, this is not clear what the max is over\n    - In equation 14 l is defined to take one parameter but then takes 2 in the equation\n5. \"Bionic modeling\" terminology is confusing and includes notions of membrane capacitance with arbitrary values (0.4-0.6) which lack justification, and it is unclear if the model aims for biological plausibility or mathematical abstraction\n6. The authors mention that the limitation of Kalman filters is that they only function well with Gaussian noise but their tests only include Gaussian noise.\n\n## Minor Issues\n1. Line 096: \"Multi-layer perception\" should be \"Multi-layer perceptron\"\n2. Line 332: \"Regular\" should be \"Regularization\"\n4. Line 376: DSAC is mentioned as state-of-the-art, but newer methods outperform DSAC, such as TQC and CrossQ, however this does not detract from the contribution of the paper\n3. Line 746: \"According to Eq. equation 17\" should be \"According to Eq. 17\"\n## Suggestions\n1. Move limitations section to main paper\n2. Add clarity to mathematical notations, such as defining what g(.)^max is\n3. Fix equation 12 as it's missing a summation\n4. Use h(t) instead of x(t) for hidden states as previous literature does would be clearer\n5. Use non-gaussian noise in some tests", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EsBl7zGiVu", "reviewer": "Reviewer_7dLr", "summary": "This paper presents a Smooth Ordinary Differential Equation-based Neural Network (SmODE) architecture, designed to address the issue of unsmooth control actions in deep reinforcement learning (RL). The SmODE network mitigates high-frequency input noise and restricts the neural network’s inherent high Lipschitz constants by employing a dual mechanism that combines low-pass filtering with Lipschitz constant control. This approach enhances the smoothness and robustness of the policy.", "questions": "1. **Provide a Detailed Description of Equation 12**:\n    - Offer a comprehensive explanation of each parameter in Equation 12 (\\(w_{ij}\\), \\(C_{mi}\\), \\(\\gamma_{ij}\\), \\(\\mu_{ij}\\)), including their the influences of value ranges, and specific roles in controlling action smoothness. Provide additional theoretical support or empirical results to demonstrate how the introduction of these parameters influences the model's performance.\n2. **Discuss Computational Efficiency and Scalability**:\n    - Provide a more detailed analysis of the computational overhead, the training time and inference speed of the SmODE network. Explore the influences of the ODE solving process or the number of iterations, to improve the model’s computational efficiency and real-time performance.\n3. **Expand Experimental Comparisons and Discussions**:\n    - [**Smooth Filtering Neural Network for Reinforcement Learning**](https://ieeexplore.ieee.org/abstract/document/10643291/): This work is highly relevant to the current paper and should be prominently compared to highlight the advantages and characteristics of the proposed method.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. **Innovative Approach**\n    - **First Combination of Neural ODE to Address Dual Issues**:\n        - This work is the first to attempt using Neural Ordinary Differential Equations (Neural ODE) to simultaneously address high-frequency input noise and action unsmoothness caused by the network’s Lipschitz constants. The SmODE network provides a comprehensive solution by integrating low-pass filtering with Lipschitz constant control, effectively improving policy smoothness and robustness rather than optimizing for a single issue alone. The design cleverly mimics the structure of a classical first-order low-pass filter but replaces the fixed time constant with a state-dependent learnable function. This not only retains the noise suppression capability of the low-pass filter but also enhances the model's adaptability and dynamic response through the learning mechanism. Additionally, it leverages the continuity properties of Neural ODEs.\n    - **Dual Smoothing Mechanism**:\n        - The SmODE network introduces learnable state-dependent time constants \\(\\tau(x)\\) and state mapping functions \\(g(x(t), I(t), t, \\theta)\\), controlling the rate of state change via the time constants and regulating the Lipschitz constants through the state mapping functions. This simultaneously suppresses high-frequency noise and controls the network’s sensitivity, achieving smooth and robust action outputs.\n    - **Biologically-Inspired Neuron Design**:\n        - SmODE neurons emulate the characteristics of biological neurons by combining synaptic weights (\\(w_{ij}\\)), membrane capacitance (\\(C_{mi}\\)), and resting potential (\\(x_{leaki}\\)). Through low-pass filtering, they enable adaptive adjustment of state boundaries. This design not only enhances the biological plausibility of the model but also improves its performance and stability in practical control tasks.\n2. **Technical Depth**\n    - **Theoretical Foundation**:\n        - The paper provides formal theorems and proofs (Theorem 1 and Theorem 2), theoretically demonstrating the bounds on hidden states and the upper limits of derivatives, showcasing a deep understanding of Neural ODEs and control theory.\n    - **Extensive Experimental Tasks**:\n        - The authors conducted comprehensive experimental validations across various reinforcement learning tasks, including vehicle trajectory tracking, linear-quadratic regulation problems, and eight robot control tasks in Mujoco. The experimental results demonstrate that the SmODE network significantly outperforms traditional Multi-Layer Perceptrons (MLP) and methods like LipsNet in terms of action smoothness and noise robustness, showcasing superior control performance.\n    - **Validation of Key Component Contributions**:\n        - Ablation studies demonstrate the significant contributions of the time constant term and state boundary adjustment term in enhancing action smoothness. By individually removing these key components, the experimental results show a substantial degradation in the smoothness effect of SmODE, further validating the rationality and effectiveness of its design choices.", "weaknesses": "1. **Insufficient Description of the Core Neuron Model (Equation 12)**:\n    - While similar works are referenced in the paper, the transition from the general smooth ODE neuron model (Equation 10) to the specific biological neuron model (Equation 12) deserves more detailed exposition, as it serves as the core design of the SmODE network. The roles, value ranges, and impacts on model performance of key parameters (\\(w_{ij}\\), \\(C_{mi}\\), \\(\\gamma_{ij}\\), \\(\\mu_{ij}\\)) are not thoroughly discussed. A more comprehensive explanation would enhance the paper's logical flow and motivation. Moreover, the introduction of multiple parameters in Equation 12 increases the model's complexity and raises concerns about the reliability of experimental results due to the extensive parameter tuning space. The introduction of multiple parameters in Equation 12 increases the model's complexity and the difficulty of parameter tuning. There is a lack of clear guidelines or empirical rules for selecting these parameters.\n2. **Incomplete Theoretical Assumptions and Proofs**:\n    - **Missing Boundedness Assumption for Function \\( g(\\cdot) \\)**:\n        - Theorem 1 asserts that the hidden state of the neuron is bounded by the maximum and minimum values of the function \\( g(x(t), I(t), t, \\theta) \\), but it does not sufficiently explain or prove whether \\( g(\\cdot) \\) itself is bounded. If \\( g(\\cdot) \\) is unbounded, the validity of the theorem is questionable.\n    - **Incompleteness and Rigorousness of Theorem Proofs**:\n        - **Theorem 1**: The proof lacks a detailed discussion on the boundedness of \\( g(\\cdot) \\) and does not adequately explain why \\( \\frac{dx_i}{dt} \\leq 0 \\) holds in all cases. The derivation steps for the lower bound \\( \\min(0, g(\\cdot)_{\\min_i}) \\leq x_i(t) \\) are not sufficiently clear.\n        - **Theorem 2**: The derivation process lacks detailed explanations regarding the boundary control of \\( f(x(t), I(t), t, \\theta) \\) and \\( g(x(t), I(t), t, \\theta) \\). The origin of the constant \\( C \\) and its relationship with \\( M(\\cdot)_i \\) are not clearly explained, resulting in the final inequality \\( \\left| \\frac{dx_i(t)}{dt} \\right| \\leq M(\\cdot)_i \\cdot C \\) lacking sufficient mathematical justification.\n3. **Insufficient Discussion on Computational Efficiency**:\n    - Although the paper mentions that the training time of SmODE increases due to the use of numerical ODE solvers, it lacks an in-depth discussion of this computational overhead and an analysis of its impact on scalability and real-time performance.\n4. **Lack of Direct Comparisons with Some Related Works**:\nRelated research focuses on addressing action smoothness in reinforcement learning, with methodologies highly relevant to SmODE. Without comparing SmODE to these methods, readers would find it difficult to understand SmODE's specific advantages.\n    - Related similar studies such as **\"Smooth Filtering Neural Network for Reinforcement Learning\"** are not included in the comparisons, potentially missing opportunities to demonstrate the relative advantages of SmODE against the latest methods.\n    - Recent relevant methods like **Neural CDEs (2020)** and **Stable Neural Flows (2021)** are not compared experimentally. These methods are relevant to handling continuous-time sequences and controlling stability, aligning closely with the goals of SmODE.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pc9VftjTcr", "reviewer": "Reviewer_DVZD", "summary": "The authors propose an ODE that can serve as a neuron in a neural network. They then propose an actor-critic algorithm that incorporates the ODE neuron into the policy network, thereby allowing for smoother action outputs compared to existing methods. The authors present relevant theoretical results as well as extensive empirical validation that showcases the usefulness of their method.", "questions": "1.\tTheorem 2 is based on the specific setup in Eq (12) according to line 245 on p.5. But the proof does seem to rely on Eq (12). Can you discuss the generalizability of your main results. \n2.\tHow do the terms added to the loss in Equation 16 affect the performance? What are the guidelines on how to set the two weights (\\lambda_1 and \\lambda_2)?\n3.\tWhat is the computational burden due to the ODE solver?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper provides a theoretically-sound method for smoothing the action output, which, based on the empirical results, leads to better performance overall. The authors provide useful and sound theoretical results, as well as convincing empirical evidence that shows how their method generally outperforms existing methods. The paper is polished, and presented in an easy-to-read manner. Overall, this paper provides a solid contribution.", "weaknesses": "The authors heavily rely on the Action Fluctuation Ratio (AFR) as the key metric for their implementation and analysis. Given that the AFR was only proposed recently (in Song et al., 2023), the paper would benefit from a more thorough discussion on why this metric is an acceptable one to use. In fact, one could argue that if there is no universally-agreed upon metric, perhaps the authors should consider multiple candidates for metrics in their analysis and justify (in the paper) why AFR is the best one to use.\n\nSimilarly, aside from MPC, the other baselines used in the experiments are not well-motivated in the text. In particular, it should be made clearer why the baselines used are the correct ones to use, and why they constitute a reasonably complete set of relevant baselines to consider.\n\nFinally, in lines 231-232, “We also think that…” should be framed in a more scientific manner. Appendix C is not convincing either (while it shows an evidence).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)