LipsNet++: Unifying Filter and Controller into a Policy Network
Authors: Xujie Song
Liangfa Chen
Tong Liu
Wenxuan Wang
Yinuo Wang
Shentao Qin
Yinsong Ma
Jingliang Duan
Shengbo Eben Li
Venue: ICML 2025 spotlightposter (2025)
Keywords: Deep Reinforcement Learning
Policy Network Design
Action Fluctuation
Control Smoothness and Robustness
Primary area: reinforcement_learning->deep_rl
TL;DR: We unified the filtering and control capabilities into a single policy network in RL, achieving SOTA noise robustness and action smoothness in real-world control tasks.

Abstract: Deep reinforcement learning (RL) is effective for decision-making and control tasks like autonomous driving and embodied AI. However, RL policies often suffer from the action fluctuation problem in real-world applications, resulting in severe actuator wear, safety risk, and performance degradation. This paper identifies the two fundamental causes of action fluctuation: observation noise and policy non-smoothness. We propose LipsNet++, a novel policy network with Fourier filter layer and Lipschitz controller layer to separately address both causes. The filter layer incorporates a trainable filter matrix that automatically extracts important frequencies while suppressing noise frequencies in the observations. The controller layer introduces a Jacobian regularization technique to achieve a low Lipschitz constant, ensuring smooth fitting of a policy function. These two layers function analogously to the filter and controller in classical control theory, suggesting that filtering and control capabilities can be seamlessly integrated into a single policy network. Both simulated and real-world experiments demonstrate that LipsNet++ achieves the state-of-the-art noise robustness and action smoothness. The code and videos are publicly available at https://xjsong99.github.io/LipsNet_v2.

LipsNet++: Unifying Filter and Controller into a Policy Network
Xujie Song' Liangfa Chen 2 Tong Liu
Wenxuan Wang' Yinuo Wang' Jingliang Duan 12" Shengho Eben Li"
Shentao Qin Yinsong Ma
Abstract
environments (Bertsekas, 2019; Guan al., 2021; Li, 2023; Wang et al., 2024b). Neural networks, capable of modeling complex nonlinear functions (Horiik et al.. 1989; Kidger & Lyons, 2020), are commonly used as the container for the control policy fitted by RL. However, RL-trained policy net- works often encounter the action fluctuation problem, where consecutive actions exhibit significant variations despite minor differences in the adjacent observations. While this problem is often overlooked during simulation and training stages, it will result in serious issues in real-world applica- tions like performance reduction, actuators' wear, and safety risk (Song et al., 2023; Mysore et al., 2021; Chen et al., 2021; Wang et al., 2024a; 2025). This problem preva- lent in various scenarios, including drone control (Mysore et al., 2021; Shi al., 2019), robot locomotion (Zhang et al., 2024), robot manipulation (Yu al., 2021). and autonomous driving (Cai et al., 2020; Chen et al., 2021; Wasala et al., 2020; Lee et al., 2024), etc. These reported action fluctua- tion problem indicates that it constitutes a considerable gap in sim-to-real RL applications.
Deep reinforcement learning (RL) is effective for decision decision-making and control tasks like au- tonomous driving and embodied AL. However, RL policies often suffer from the action fluctu- ation problem in real-world applications, result- ing in severe actuator wear, safety risk, and per- formance degradation. This paper identifies the two fundamental causes of action fluctuation: ob- servation noise and policy non-smoothness. We propose LipsNet++, novel policy network with Fourier filter layer and Lipschitz controller layer to separately address both causes. The filter layer incorporates a trainable filter matrix that automat- ically extracts important frequencies while sup- pressing noise frequencies in the observations. The controller layer introduces Jacobian regu- larization technique achieve low Lipschitz constant, ensuring smooth fitting of policy func- tion. These two layers function analogously to the filter and controller in classical control theory, sug- gesting that filtering and control capabilities can be seamlessly integrated into a single policy net- work. Both simulated and real-world experiments demonstrate that LipsNet++ achieves the state-of- the-art noise robustness and action smoothness. The code and videos are publicly available at https:/xjsongg99 http:////soon9.....u.ooooo///// github.io/LipsNet..y v2.
In order make RL more applicable in real-world scenarios. researchers working hard to solve the problem. CAPS (Mysore et al., 2021) and 1.2C2 (Kobayashi, 2022) mitigate fluctuations by introducing penalty terms in the actor loss, enforcing action similarity across successive time steps or similar states. SR (Shen et al., 2020; Zhao et al., 2022) employs adversarial noise and minimizes action differences between actual and perturbed states. PIC (Chen et al., 2021) and TAAC (Yu et al., 2021) design two-stage policies by us- ing one network to output the current action, and the other to output action inertia scalar or make choice between the cur- rent and the last action. MLP-SN (Takase et al., 2020) and LipsNet (Song et al., 2023) smooth control actions by con- straining the Lipschitz constant of policy network. However, both CAPS and L2C2 require sensitive hyperparameter tun- ing and involve compulsory sampling of neighboring states. SR PIC, and TAAC need special policy evaluation or pol- icy improvement mechanisms. MLP-SN suffers from severe performance loss and difficulty in Lipschitz constant tuning, LipsNet, the previous state-of-the-ar (SOTA) method, has network structure limitation and is unsuitable for high-real- time tasks due to slow inference speed. Most importantly, none of them explicitly address observation noise-a factor
1. Introduction
Deep reinforcement learning (RL) has become powerful approach for addressing optimal control tasks in physical
'School of Vehicle and Mobility & College of AI, Tsinghua University. Beijing, China "School Mechanical Engineering, University of Science and Technology Beijing, Bei- jing, China 'Laboratory for Computational Sensing and Robotics, Johns Hopkins University. Baltimore, USA. Correspondence to: Shengbo Eben Zlishbo@tsinghua.coo wdw.ghuaa.du.n...... Jingliang Duan <duanjl ustb.cdu.cn..
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

LipsNet++: Unifying Filter and Controller into Policy Network
that our study identifies key cause of action fluctuation. In summary, achieving smooth control actions in a manner that is effective, simple, and broadly applicable across RL
return (TAR) by 3.4% and reduces the action fluctuation ratio (AFR) by 35.5% compared to LipsNet (Song al., 2023), which is the previous SOTA network. Additionally, an experiment on physical mini-vehicles is implemented for real-world testing, where the vehicle is going to track given trajectories and avoid moving obstacle under various noise levels. Results show that LipsNet++ increases the TAR by 5.8% and reduces the AFR by 90.0% compared to MLP.
algorithms remains an open challenge.
L
Env
Actor
Critic
Technical contributions. LipsNet++ is novel network, addressing the action fluctuation problem in the real-world applications of RL. Our contributions are four-fold: (1) We identify the two fundamental reasons that cause action fluctuation, and propose a policy network LipsNet++ to explicitly tackle the two reasons decoupled manner; (2) We propose a Fourier filter layer in LipsNet++, capable of automatically extracting valuable frequencies while sup- pressing noise frequencies in observations; (3) We propose a Lipschitz controller layer with Jacobian regularization tech- nique to enhance the smoothness of policy fitting, which breaks LipsNet's structure limitation and inference speed limitation; (4) LipsNet++ suggests that filtering and con- trol capabilities can be seamlessly integrated into a single policy network, providing valuable insights for future pol- icy network design. The code is released to facilitate the implementation and future research.
MLP
Fluctuated action
Replace
Control outouts
+ipsNet++ (ours)
mooth action
Figure I: LipsNet++ outputs smooth action.
In this paper, we proposed novel policy network structure, LipsNet++ achieving action smoothing in RI. effectively, simply and flexibly. We identified the fundamental rea- sons for causing action fluctuation are the non-smoothness of policy network and the existence of observation noise. LipsNet++ adopts two corresponding network modules to explicitly tackle them at the same time. Firstly, we pro- pose a Fourier filter layer to filter observation noise. In this layer, fast Fourier transform (FFT) is used to obtain the frequency features of sequential observations, and a trainable filter matrix automatically extracts important fre- quencies and suppress noise frequencies in observations. Secondly, Lipschitz controller layer is introduced, whose Lipschitz constant is constrained by a proposed Jacobian regularization technique. The controller layer can be ar- bitrary derivable network structures, breaking the network structure limitation of LipsNet who only supports multilayer perceptron (MLP) with piecewise linear activation functions. This layer no longer requires the computation of the Jaco- bian matrix during inference, thereby breaking the inference speed limitation of LipsNet. The Fourier filter layer and Lip- schitz controller layer function analogously to the filter and controller in classical control theory, respectively, suggest- ing that filtering and control capabilities can be seamlessly integrated into single policy network.
2. Preliminaries
2.1. Actor-Critic Reinforcement Learning
Actor-critic method, consisting of an actor network and a critic network as shown in Figure 1, forms the backbone of many RL algorithms. The actor network fits policy π S + A that mapping from state space to action space. Therefore, the actor network is also called as policy net- work. The goal of RL. is to train a policy æ maximizing the expected return:
Jx = E Eyeep
Σ x
where Pe is the distribution of state-action trajectory induced by policy T is the termination time of an episode, 0 < 7 S 1 is the discount factor, and r; represents the reward. The critic network fits value function V(s) or Q(s, a). mapping from the state-action pairs to the expected returns, to evaluate the actions taken by the actor.
In policy evaluation phase, the critic is updated by minimiz- ing the temporal difference (TD) error. For example, the Q-value network in DDPG (Lillicrap et al., 2015) parame- terized by is updated by
Experiment results. Simulated and physical experiments verify that LipsNet++ has achieved the SOTA performance. For the simulated tasks, we conduct experiments on the double integrator environment and DeepMind control suite benchmark (DMControl). For example, in DMControl's 'ss walker environment, LipsNet++ increases the total average
min Eaaan [(Qp(s,)) - - -7Q Prax 110 (s')
2

---

LipsNet++: Unifying Filter and Controller into Policy Network
where D is the replay buffer, s' is the next state, a' is the next action obtained by the target actor network, and Qpeare is the return estimated by the target critic network.
Based on the above analysis, the two fundamental causes of action fluctuation can be identified: (1) the non-smoothness of policy network, and (2) the existence of observation noise, corresponding to the red and blue terms, respectively.
In policy improvement phase, the actor is updated by maxi- mizing the expected return predicted by the critic. Taking DDPG as an example again, the actor network is updated by minimizing the actor loss function:
Non-smoothness of policy network. A non-smooth policy network means that RL fits a non-smooth policy function mapping from the states to control actions. The mapping function has significant output differences even if the inputs are closely adjacent. Consequently, when the state changes with time, non-smooth action sequence is generated. Ap- pendix B visualizes the effect of a non-smooth policy.
L B (-Qp(s,
2.2. Action Fluctuation Ratio
Action fluctuation ratio (AFR) is an index quantitatively measure the fluctuation level of control actions (Chen et al., 2021; Song et al., 2023). It is defined as
Existence of observation noise. The noise results in the discontinuous changes in observations, making the actions produced by the policy network at the adjacent time stamps erratically differ. Even if the policy function fitted by the policy network is smooth enough, actions can still be fluctu- ated because of the erratic observation noise.
T
{(T) = E Eropp
1 T
Σ t=1
||.
a-1l -
Therefore, achieving sufficiently smooth control actions re- quires addressing both fundamental reasons simultancously. However, previous works have neither clearly identified these two causes nor considered them together. While some studies acknowledge the impact of observation noise, they attempt to enhance robustness by reducing the Lipschitz constant of the policy network (Takase et al., 2020; Song et al., 2023), i.e. improving the smoothness of policy net- work, rather than directly filtering the observation noise. Such a non-decoupled approach results in actions being insufficiently smooth, and performance degradation when higher action smoothness is required. In this paper, intro- duce the Fourier filter layer and Lipschitz controller layer to explicitly and decoupledly address these two fundamental causes, providing a more effective and principled solution.
where Pg is the distribution of state-action trajectory induced by policy т, T the termination time of episodes, a and a--1 are two adjacent actions, and] || is the norm of action difference vector !
Beside the total average return (TAR), AFR is also an impor- tant indicator to evaluate policies' performance in the real world. The smaller AFR is, the smoother action sequence policy æ has.
3. Method
3.1. Reasons Identification of Action Fluctuation
To ensure that RL agents produce smooth actions, it is nec- essary to first identify the root cause of action fluctuation. In decison-making and control tasks, the actions are calculated by the policy network т according to the current observa- tion ot, i.e. a₂ = (ot). And the current observation og is composed by the current state 8 and observation noise ot. i.e. 0 St + de- The rate of action change over time is
3.2. Fourier Filter Layer
Fourier Transform is a widely used frequency analysis tool, which can also be employed in neural networks for feature extraction (Lee-Thorp et al., 2022; Rao et al., 2021; Tan et al., 2024). To mitigate the action fluctuation caused by observation noise, we propose the Fourier filtering layer based on fast Fourier Transform. The workflow of Fourier filter layer is shown in Figure
das d=(a.) des then we can derive that dt do. de
da
(o)) do
ds; dt
da: dt
+
(1)
Given N historical observations o,, 0:-1. R where D denotes the dimension of features, the Fourier filter layer concatenates them as a matrix x E NDD and calculates the frequency feature matrix X E CNDD using 2D discrete Fourier transformation:
O
To mitigate action fluctuation, || || must be controlled within reasonable range. From Equation (1), we know Il is affected by three parts: red term of policy derivative, reflecting the level of policy smoothness; blue term of noise change rate, reflecting the level of ob- servation noise; and inherent derivative term ds. de of
N-1 D-1
X uuo
Σ
Σ
Fn,d E f2 (2F
the target dynamics system which cannot be modified.
n=0
Throughout the paper, || denotes the 2-norm of vector or
where In,d denotes the d-th feature of the n-th observation signal, X X,,r denotes the element located at the u-th row
a matrix.
3

---

LipsNet++: Unifying Filter and Controller into Policy Network
FFT
N+-++1
X
conjugate symmetrize
IFFT
N+1+1
X
HT
Figure 2: Workflow of Fourier filter layer. Firstly, FFT converts historical observations to frequency feature matrix X. Then, half of X is multiplied by a trainable filter matrix H, and complete matrix X is generated by conjugate symmetrizing. Finally, IFFT converts X to filtered time-domain signals.
and u-th column of the frequency feature matrix X, and represents the imaginary unit. When the length of historical observations is less than N, the missing parts are padded with In FFT, Zero-padding does not alter the primary frequency components of the signal, and it merely increases the spectral resolution (Jung et al., 2019). The magnitude of Xu,v denotes the signal intensity at the frequeney combina- tion (u, v), where u and are frequency indices rather than the actual frequency values. Since the observations only consist of real values, the resulting matrix exhibits conju- gate symmetry, i.c. Xu,v = -N means that half of X could represent the complete information contained in the signal.
Because X is conjugate symmetric matrix, the matrix I E RNx becomes a real matrix. By slicing rows from the matrix ě, the filtered features o₁, ōt-1, ... Oo-N+1 E R are obtained. The signal o,, representing the filtered feature corresponding to the current timestamp, selected as the input for the subsequent Lipschitz controller layer.
3.3. Lipschitz Controller Layer
Definition 3.1 (Local Lipschitz Constant), Suppose f RD + R is continuous neural network. The K K(x) is defined as the local Lipschitz constant of on the neighbor- hood of x::
Then, half of X, denoted Xhair CN is sub- jected to a Hadamard product with trainable filter ma- trix H E CNx111+1, After that, a complete matrix X E CNDD is restored by conjugate symmetrizing the product matrix:
(f(x)) f(x2))
K(x)
max (Bx.p)
(6)
|
where Bix p) denotes the open ball area with radius p >> 0 centered the point x the Euclidean space, i.e. B(x,, p) {x' || x|| < p}.
X = symmelrive @ H).
(3)
Lipschitz constant characterizes the landscape smoothness of function. By viewing the policy network as map- ping function from states to actions, Lipschitz constant can reflect the smoothness of the policy function. A lower Lip- schitz. constant means a smoother policy function, leading to smoother actions (Ames et al., 2016; Kobayashi, 2022: Song et al., 2023: Takase al., 2020). MLP-SN (Takase et al., 2020) constrains the Lipschitz constant by applying spectral normalization (SN) (Miyato et al., 2018) on each layer of policy network. However, it leads to tuning dif- ficulty and severe performance loss, because the desired network-wise Lipschitz continuity realized by layer-wise Lipschitz constraints (Bhaskara et al., 2022; Wu et al., 2021). Instead, LipsNet (Song et al., 2023) proposes network- wise method, Multi-dimensional Gradient Normalization (MGN). However, LipsNet is not applicable in high-real- time tasks due to the Jacobian matrix calculation during forward inference, and its network structure is limited to MLP with piecewise linear activation functions.
By choosing H complex matrix instead of real matrix, the Fourier filtering layer can not only alter frequency ampli- tudes but also perform feature extraction. The magnitudes of the elements in H determine which frequency is suppressed or strengthened. To enable the noise filtering capability of policy network, we encourage the magnitudes of elements in H to be as small as possible. In this way, policy network can automatically extract valuable frequencies and filter less relevant frequencies where noise may exist. The tailored actor loss becomes
L' An H
(4)
where || is the Frobenius norm of H. and X is hy- perparameter coefficient.
Finally, the resulted frequency feature matrix X is recovered to the Lime-domain signals by 2D inverse discrete Fourier transformation:
N-1D-1 ΣΣ Xm,v
2n.d
To overcome the above challenges, we propose the Jacobian regularization method to conveniently reduce the Lipschitz
ND

---

LipsNet++: Unifying Filter and Controller into Policy Network
at
L" = C'
IlVIl
V."
Controller
Lipschitz Controller Layer
acobian Regularization
Policy NN
IFFT
0t-1
Oz-N41
F Filter
Fourier F ilter Layer
frequency feature
trainable filter matrix
Observations
FFT
Fourier F Filter Layer
Figure 3: Overall structure of LipsNet++. Historical observations processed by Fourier filter layer, where trainable filter matrix is used for frequency selection. The filtered feature o; is inputted into Lipschitz controller layer whose Lipschitz constant is constrained by Jacobian regularization. The parameters in LipsNet++ are updated by tailored actor loss L"
constant of our controller layer. The Jacobian norm is a com- monly used index of function smoothness and robustness (Hoffman et al., 2019; Lee al., 2023).
is superior to the Lipschitz constraint methods used MLP- SN and LipsNet because: (1) It a network- wise rather than layer-wise constraint method, avoiding severe performance loss; (2) It does not require predefined initial/target Lips- chitz constant, avoiding the tuning difficulty; (3) It does not need to calculate Jacobian matrix during forward inference, making it applicable in high-real-time tasks; (4) It is suitable for arbitrary derivable network structures.
Theorem 3.2 (Lipschitz's Jacobian Approximation). Let R" + Rm be continuously differentiable neural network. The Jacobian norm =. is approximation of f's local Lipschitz constant within neighborhood B(x, p), centered at x with radius The approximation error is
The overall structure of LipsNet++ is shown in Figure The pseudocode of LipsNet++ is illustrated in Appendix C.
[V= max 6EB(0.p)
+o(8)
where o()) is a higher-order infinitesimal term with respect to 8. Moreover, - O, the Jacobian norm converges to the exact local Lipschitz constant, i.e.
3.4. User-friendly Packaging
We have packaged LipsNet++ as an user- user-friendly PyTorch (Paszke et al., 2019) module and released the code. When network's backward propagation is called, backward hook function will awake to automatically replace the gradient VL by VL". In this way, LipsNet++ can be easily inte- grated into almost all actor-critic RL algorithms like DDPG (Lillicrap et al., 2015), TD3 (Fujimoto et al., 2018), PPO (Schulman et al., 2017), TRPO (Schulman et al., 2015), SAC (Haarnoja al., 2018) and DSAC (Duan et al., 2025). As shown below, practitioners
lim v= K(x). p00
Proof. See Appendix A in the supplementary material.
know that Lipschitz constant reflects policy's 's landscape smoothness (Takase al., 2020; Song et al., 2023) and The- orem 3.2 proves that Jacobian norm is approximation of conveniently
the local Lipschitz constant, therefore enhance the policy smoothness by reducing Jacobian norm. The actor loss is tailored from L' in Equation (4) into
it simple MI.P
net LipsNet ++ ()
out net (input)
L" = X V
where f means the Lipschitz controller layer and A a constant coefficient. The proposed Jacobian regularization
loss. backward ()
5

---

LipsNet++: Unifying Filter and Controller into Policy Network
4. Experiments
increase in TAR and 75.0% reduction in AFR. Therefore, LipsNet++ achieves new SOTA performance with sig- nificant advantage in action smoothness.
In this section, we comprehensively evaluate LipsNet++ with model-based, model-free, deterministic, and stochastic RL algorithms in both simulated and real-world tasks.
Furthermore, an ablation study for the two techniques in LipsNet++ is implemented in Appendix E, the sensitivity analysis for hyperparameters λ and A is provided in Ap- pendix F, and the sensitivity analysis for hyperparameter N is provided in Appendix G. Additionally, policy net- works' computational efficiency are evaluated in Appendix H, including the time usages of forward and backward prop- agations. Based on all the results, a performance radar chart is depicted in Figure 7(c), which implies the overall perfor- mance of LipsNet++ is much better than previous works.
4.1. Double Integrator
Double integrator is a classic linear quadratic control task, which is commonly used to test the performance of con- trollers. In the environment, particle is moving along axis without resistance (Song et al., 2023). The observations include position z and velocity v of the particle. The control action is particle acceleration a that parallel to the axis. A schematic diagram of the environment is shown in Figure 4.
4.2. DeepMind Control Suite
V
The DeepMind Control Suite (DMControl) (Tassa et al., 2018) consist of several well-designed continuous control tasks. Currently, it stands as one of the most recognized benchmarks in the fields of RL and continuous control (Mu et al., 2022). In this paper, focus on four of its envi- ronments: Cartpole, Reacher, Cheetah, and Walker. The visualization of these environments are shown in Figure 5, and more information are described in Appendix I.
Figure 4: Double integrator task. A particle is moving along an axis without resistance.
22 The reward function is r = -2x u2 a², which in- centives the particle to remain stable at the origin, i.c. == 0, v = 0, = 0. The Infinite-time Approximate Dy- namic Programming (INFADP) (Li, 2023), a model-based RL algorithm, is used for train without noise. When testing policy networks, the particle has nonzero initial position and velocity, and the noise for each observation dimension is dis- tributed in U' (.20 0.2). More details and hyperparameters are shown in Appendix D.
(a) Cartpole
(b) Reacher
(c)
Cheetah
(d)
Walker
The results are presented in Figure 6 and 7. In Figure 6(a), 30 episodes are simulated starting from the same initial state. The solid line and shadow area respectively denote the mean and standard deviation of actions. The shadow areas imply the action fluctuation amplitude of LipsNet++ is much smaller than that of MIP,, and is on par with LipsNet. Figure 6(b) depicts action trajectories for a single episode, which reveals that LipsNet++ has better action continuity than LipsNet under the same level of action fluctuation amplitude. This conclusion is confirmed again by Figure 6(c), where action trajectories are decomposed by FFT and the action frequency induced by LipsNet++ is shown to be more distributed in the low-frequency range.
Figure 5: DeepMind control suite benchmark. Four envi-
ronments in DeepMind control suite are selected for testing.
We employ the Twin Delayed Deep Deterministic Policy Gradient (TD3) (Fujimoto et al., 2018), a model-free RL al- gorithm, for training. The hyperparameters for TD3 remain consistent across all environments, except for the coeffi- cients Ag. An, and the length of historical observations N. All hyperparameters are listed in Appendíx J. To evaluate comprehensively, networks are tested on both noise-free and noisy environments. Figure 9 visualizes the results in noisy environments. The learned filter matrix H is visualized in Figure 24 to show the noise filtering ability of LipsNet++ All results are summarized in Table 11 and 12, from which we can find that LipsNet++ has the highest TAR and the lowest AFR in all cases. For example, LipsNet++ increases the TAR by 3.4% and reduces the AFR by 35.5% in Walker environment compared to LipsNet, which is the previous SOTA network. Appendix K shows a comparison in Cart- pole environment between LipsNet++ and reward penalty method. All these results imply that LipsNet++ has perfect action smoothness and noise robustness.
To further evaluate LipsNet++, we set different observation noise amplitudes and compare with previous works. As Fig- ure 7(a) shows, when noise increases, LipsNet++ maintains the highest TAR and its TAR declines the slowest rate. As Figure 7(b) shows, when noise increases, LipsNet+++ main- tains the lowest AFR and its AFR grows at the slowest rate. We then compare the performance in high-noise environ- ment, i.c. noise amplitude is 0.3. Compared to Lips the previous SOTA network, LipsNet++ achieves an 8.2%
6

---

LipsNet++: Unifying Filter and Controller into Policy Network
3.50
0.06
0.5
LipsNee LipnNidde
0.06
0.00
0.0
Amao fudd
-0.50
0.02
-1.0
LioaNucL
0.01
LppNNet++
LipoNoc++
1.5
0.00
5.0
12.5
150
0.0
7.5
10.0
Time (6)
Time (6)
Frequercy (H12)
(a) Action fluctuation amplitude
(b) Action trajectory
(c) Action frequency spectrum
Figure 6: Action visualization in double integrator environment. (a) The action fluctuation amplitude of LipsNet++ is smaller than that of MLP, and is on par with LipsNet. (b) LipsNet++ has better action continuity than MLP and LipsNet. (c) LipsNet++'s action frequency is more distributed in the low-frequency range.
0.5
Tosal oversge roturn
fuclustooratio
6A
LppeNet-G
MUP-SN
control performann nc
(lw-nools))
backward
speed
RLP
CAPS
1352
MIP-N
Lpsket
Upskett
MLP
0.2
sm smoothness flow-noc nccle
speed
nwward
MLF-SN
LipsNet-G
8.1
Lpshett
Liphiote
performanc (high nolse)
smoothness (highinelse)
0.00
0.36
0.10
0.16
0.20
0.25
0.30
0.00
0.05
0.10
0.15
0.20
0.25
0.50
Obsarvation noise ampiltude
Observation naise amplltude
(a) TAR
(b) AFR
(c) Performance radar chart
Figure 7: Performance comparison in double integrator environment. (a) The TAR of LipsNet++ declines at the slowest rate when noise increases. (b) The AFR of LipsNet++ grows at the slowest rate when noise increases. (c) LipsNet++ has the best overall performance in the evaluation metries of control performance, action smoothness, and forward speed, etc.
Reference robol Obibucin
Reference robot
-55
25
Obetacle
Reference
trajectory
20
20
spood
Obstacle
MB
20
Time
Time
Time
Thm
-00
RL robot
20
20
Time
Tmm
(a) Scenario illustration
(b) Vehicle trajectory (MLP)
(c) Vehicle trajectory (LipsNct++)
Longiudinal atc.
Yaw accelaration
MLP
Net
LppNNet
10
10
Time
Time
(d) Action 1: LLongitudinal accelcration
(c) Action 2: Yaw acceleration
Figure 8: Result of scenario 3. The noise amplitude is 10. (a) The RI. robot aims to turn left. (b,c) The vehicle states and trajectories produced by MLP and LipsNet++. (d,e) The control actions produced by MLP and LipsNet++. LipsNet++ produces much smoother control actions than MLP.

---

4 go straight LipsNet++: Unifying Filter and Controller into Policy Network
1000
actions result in smoother vehicle states, i.e. speed and yaw rate, which are shown in Figure 8(b)(c). These results consistently hold true across all scenarios, as illustrated in Appendx M. Furthermore, in scenario 4 with 10 times noise, the MI.P-driven robot crashed while LipsNet++ successfully completed the task, as shown Figure 48 and 49.
TAR
500
0
Cartpole
Reacher
Cheetah
Walker
The learned filter matrix H is visualized Figure 12 to show the noise filtering ability of LipsNet++. Figure 12(a) and 30(b) show the frequency distributions of observation in noise-free and noisy environments, respectively. Figure 12(b) implies that the learned filter matrix mainly focus on the frequencies containing observation information, and rarely focus on the frequencies containing noises.
MLP
AFR (L)
LipsNet-G G
LipaNet-L
+
Cartpolc
Reacher
Cheetah
Walker
Figure Performance comparison DMControl. It shows the TAR and AFR in noisy environments. LipsNet++ has the highest TAR and the lowest AFR in all cases.
The average TAR and AFR for the first three scenarios are depicted in Figure 13, As Figure 13(a) shows, when noise increases, LipsNet++ maintains the highest TAR and its TAR declines much slower than MLP's. As Figure 13(b) shows, when noise increases, LipsNet++ maintains the low- est AFR and its AFR grows much slower than MLP's. In the high environment (noise amplitude is 20), LipsNet++ achieves 5.9% increase in TAR and 90.0% reduction in AFR. More results listed in Appendix M, implying LipsNet++ has much better action smoothness and noise robustness.
4.3. Mini-Vehicke Driving
Vehicle trajectory tracking is an important task in tonomous driving (Guan et al., 2022; Mu et al., 2020). To validate LipsNet++ in the real world, we conduct an ex- periment on physical vehicles. As Figure 26 shows, the vehicle moves by two differential wheels, aiming to track reference trajectory and velocity while avoiding obstacle. The observations and actions are listed in Table 15. We up four diverse scenarios, described in Table and visualized in Figure 28. Detailed introduction of the vehicle, control mode, and scenarios are described in Appendix L. The Distributional Soft Actor-critic (DSAC) (Duan et al., 2025), a model-free RI. algorithm, is used for training. The tests in all scenarios are accomplished by the same networks. For real-world highway vehicles, RL observations rely on perception results where sensor noise is amplified by per- ception algorithms. To precisely simulate such scenario, assigned various noise amplitudes.
Historical absv. rank
0.4
0.2
15.
19
Observation dimension
(a) Obsv. frequency (noise-free)
Historical obsv. rank
0.5
Table Scenario descriptions.
5.
10-
0.3
Scenario No.
RL robot
Obstacle robot
1
go straight
stationary moving moving aggressive
15
0.1
2
go straight
3
turn left
1
3
5
Observation dimension
(b) Filter matrix
In scenario 3 with 10 times noise, the results shown in Figure 8, its video snapshots are recorded in Figure 11. The RL robot successfully tracks the reference trajectory and avoids obstacle by slightly shifting to yield. As shown in Figure 8(d)(e) it evident that LipsNet++ produces much smoother control actions than MLP. The smoother
Figure 12: Observation frequency and filter matrix. The color in (a) and (b) means the frequency intensity of noise- free observations and the element magnitude in filter matrix. The matched color distribution implies LipsNet++ can ex- tract important frequencies and filter noise frequencies.
8

---

LipsNet++: Unifying Filter and Controller into Policy Network
t= Os
Reference trajectory
t= 3s
t= 10s
t= 17s
RL robot
RL robot
Obstacle
Obstacle
Obstacle
Obstacle
RL robot
RL robot
Figure 10: Snapshots of scenario The RL robot first shifts left to navigate around the stationary obstacle robot, then
shifts right to resume tracking the reference trajectory. All test videos are available at https://jjsong99.gihbnn wttp://9.iiiubb.ooipll/loo v2.
Os
6s
t= 11s
t 18s
Obstacle
Obstacle
robot
RL robot
RL robot
Obstacle
Reference trajectory
RL robot
Figure 11: Snapshots of scenario 3. Video snapshots of Figure 8(c). The RL robot first shifts left to yield to the moving
obstacle robot, then resumes tracking the left-turn reference trajectory. All videos: http://Assogg.ggiuubbiiLLissss github.io/LipsNee....
5. Conclusion
Total average (1)
260
In this paper, we identify the two fundamental causes of action fluctuation and propose a novel policy network Lip- sNet++, where Fourier filter layer and a Lipschitz con- troller layer introduced explicitly tackle the two causes. Fourier filter layer has observation noise filtering capability based on learnable filter matrix. Lipschitz con- troller layer uses Jacobian regularization for smoothly fitting policy functions, breaking LipsNet's structure lim- itation and inference speed limitation. LipsNet++ can be easily integrated into most actor-critic RL algorithms. Sim- ulated and real-world experiments show that LipsNet++ has excellent action smoothness and noise robustness, achieving new SOTA performance. The unifying of filtering and control capabilities providing valuable insights for future policy network design. We hope the research could make contribution to the real-world applications of RL.
255
250
245
240
LipsNet**
10
15
20
Nolse amplitude
(a) Average
2.5
MLP
LipsNot++
Action fiuctuation
1.5
1.0
Acknowledgements
0.5
This study is supported by National Key R&D Program of China with 2022YFB2502901, and Tsinghua University- Didi Joint Research Center for Future Mobility.
0.0
10
15
20
Noise amplitude
(b) Average AFR
Impact Statement
Figure 13: Performance trend with increasing noise. They show the average TAR and AFR of scenarios 1-3. (a) The TAR of LipsNet++ declines much slower than MLP's. (b) The AFR LipsNet++ grows much slower than MLP's.
LipsNet++ has positive impacts on the AI community by effectively addressing the action fluctuation problem in RL. LipsNet++ breaks through the bottleneck of action fluctu- ation and poor robustness faced by RI,, which accelerates

---

LipsNet++: Unifying Filter and Controller into Policy Network
the process of RL's real-world applications. It mitigates the wear of actuators, safety risks, and performance reduction caused by fluctuated actions. The unifying of filtering and control capabilities providing valuable insights for future policy network design. LipsNet++ benefits many industrial fields, including drone control, decision-maln making and control of autonomous vehicles, and embodied AI.
conference on machine learning, pp. 1861-1870. PMLR, 2018.
Hoffman, J., Roberts, D. A., and Yaida, S. Robust learning with jacobian regularization. arXiv preprint 2019.
Hornik, K., Stinchcombe, M.. and White, H. Multilayer feedforward networks are universal approximators, Neu- ral Networks, 2(5):359 1989.
References
Ames, A. D., Xu, X., Grizzle, J. W., and Tabuada, Control barrier function based quadratic programs for safety crit- ical systems. IEEE Transactions on Automatic Control, 62(8):3861-3876 2016.
Jung, Y., Cho, J., Lee, S., and Jung, Y. Area-efficient pipelined fft processor for zero-padded signals. Elec- tronics, 8(12):1397, 2019.
Kidger, P. and Lyons, T. Universal approximation with deep narrow networks. In Conference Learning Theory, pp- 2306-2327. PMLR, 2020.
Bertsekas, D. Reinforcement learning and optimal control, volume 1. Athena Scientific, 2019.
Bhaskara, S.,
T., Jepson, A. D., and Levinshtein, A. Gran-gan: Piecewise gradient nor- malization for generative adversarial networks. In Pro- ceedings of the IEEE/CVF Winter Conference on Appli- cations of Computer Vision, pp. 3821-3830 2022.
Kobayashi, T. L.cc:: Locally lipschitz continuous constraint towards stable and smooth reinforcement learning. In 2022 IEEF/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 4032-4039, IEEE, 2022.
Lee, I., Cao, H.-G., Dao, C.-T., Chen, Y.-.,, and Wu, I.- C. Gradient-basst regularization for action smoothness in robotic control with reinforcement learning. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 603-610. IEEE, 2024.
Cai, P., Mei, X., Tai, L., Sun, Y., and Liu, M. High-speed autonomous drifting with deep reinforcement learning. IEEE Robotics and Automation Letters, 1(2):1247-1254, 2020.
Chen, C., Tang, H., Hao, J., Liu, W., and Meng, Z. Address- ing action oscillations through learning policy inertia. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 35, pp. 7020-7027, 2021.
Lee, S., Park, J., and Lee, J. Implicit jacobian regularization weighted with impurity probability output. In Inter- national Conference on Machine Learning, pp. 19141- 19184. PMLR, 2023.
Duan, J., Wang, W., Xiao, L., Gao, J., Li, S. E., Liu, C., Zhang, Y.-Q.. Cheng, B., and Li, K. Distributional soft actor-critic with three refinements. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(5))3935- 3946, 2025.
Lee- Thorp. J., Ainslie, J., Eckstein, I., and Ontanon, S. Fnet: Mixing tokens with fourier transforms. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. +296-4313 2022.
Fujimoto, S., Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. In Interna- tional Conference on Machine Learning, pp. 1587-1596 PMLR, 2018.
Li, S. E. Reinforcement learning for sequential decision and optimal control. Springer Verlag, Singapore, 2023.
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning. arXiv preprint SKKiv:/509.097// 2015.
Guan, Y., Li, S. E., Duan, J., Li, J., Ren, Y. Sun, Q., and Cheng, B. Direct and indirect reinforcement learning. International Journal of Intelligent Systems, 36(8):4439- 4467, 2021.
Miyato, T., Kataoka, T., Koyama, M., and Yoshida, Y. Spec- tral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018
Guan, Y., Ren, Y., Sun, Q. Li. S. E., Ma, H., Duan, J., Dai, Y., and Cheng, B. Integrated decision and control: toward interpretable and computationally efficient driving intelligence. IEEE Transactions on Cybernetics, 2022.
Mu, Y., Peng, B., Gu, Z., Li, Liu, C., Nie, B., Zheng, J., and Zhang, B. Mixed reinforcement learning for efficient policy optimization in stochastic environments. In 2020 20th International Conference on Control, Automation and Systems (ICCAS), pp. 1212-1219 IEFE, 2020.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic; Off- policy maximum entropy deep reinforce- ment learning with stochastic actor. International
10

---

LipsNet++: Unifying Filter and Controller into Policy Network
Mu, Y. M., Chen, S., Ding, M., Chen, J., Chen, R., and Luo, P. Ctrlformer: Learning transferable state representation for visual control via transformer. In International Con- ference on Machine Learning, pp. 16043-16061. PMLR, 2022.
Tan, H., Liu, S., Ma, K., Ying, C., Zhang, X., Su, H., and Zhu, J. Fourier controller networks for real-time decision- making in embodied learning. In International Confer- ence on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 47607-47626. PMLR, 21-27 Jul 2024.
Mysore, S., Mabsout, B., Mancuso, R., and Saenko, K. Regularizing action policies for smooth control with rein- forcement learning. 2021 IEEE International Confer- ence on Robotics and Automation (ICRA), pp. 1810-1816. IEEE, 2021.
Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. L.,, Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. Deepmind control suite. arXiv preprint 2018.
Todorov, E.. Erez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ Inter- national Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012. doi: 10.1109/RROS.011 6386109.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z.,, Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32, 2019.
Wang. W., Duan, J., Song, X., Xiao, I.., Chen, I.., Wang, Y., Cheng, B., and Li, S. E. Smooth filtering neural network for reinforcement learning. IEEE Transactions on Intelligent Vehicles, 2024a.
Rao, Y. Zhao, W. Zhu, Z., Lu, J., and Zhou, J. Global filter networks for image classification. Advances in neural information processing systems, 34:980-993, 2021.
Rumelhart, D. E., Hinton, G. E., and Williams, R. J. Lear- ing representations by back-propagating errors. nature, 323(6063):533-536 1986.
Wang, Y., Wang, L., Jiang, Y., Zou, W., Liu, T., Song, X., Wang, W., Xiao, L., Wu, J., Duan, J., et al. Diffusion actor-critic with entropy regulator. Advances in Neural In- formation Processing Systems, 37:54183-54204 2024b.
Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. Trust region policy optimization. In International Con- ference on Machine Learning, pp. 1889-1897. PMLR, 2015.
Wang, Y., Wang, W., Song, X., Liu, T., Yin, Y., Chen, L., Wang, L... Duan, J., and Li, S. E. Ode-based smoothing neural network for reinforcement learning tasks. In The Thirteenth International Conference on Learning Repre- sentations, 2025.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv: arXiv: 1707.06347, 2017.
Wasala, A., Bymee D., Miesbauer, P., O' O'Hanlon, J., Heraty, P., and Barry, P. Trajectory based lateral control: A rein- forcement learning case study. Engineering Applications of Artificial Intelligence, 94: 2020.
Li. Shen, Li, Y., Jiang, H., Wang, Z.. and Zhao, T. Deep reinforcement learning with robust and smooth policy. In International Conference on Machine Learning, pp. 8707-8718. PMLR, 2020.
Wu, Y.-L., Shuai, H.-H., Tam, Z.R.., and Chiu, H.-Y. Gradi- ent normalization for generative adversarial networks. In Proceedings of the JEEE/CVF International Conference on Computer Vision pp. 6373-6382, 2021.
Shi, G., Shi, X., O'Connell, M., Yu, R., Azizzadenesheli, K., Anandkumar, A., Yue, Y., and Chung, S.-J. Neural lander: Stable drone landing control using learned dy- namics. In 2019 International Conference on Robotics and Automation (ICRA), pp. 9784 9784-9790. IEEE, 2019.
Yu, H., Xu, W., and Zhang, H. Taac: Temporally abstract actor-critic for continuous control. Advances in Neural Information Processing Systems, 14:29021-29033 2021.
Song, X., Duan, J., Wang, W., Li, S. E., Chen, C.,, Cheng, B., Zhang, B., Wei, J., and Wang, X. Lipsnet: smooth and robust neural network with adaptive lipschitz constant for high accuracy optimal control. In International Con- ference on Machine Learning, pp. 32253-32272. PMLR, 2023.
Zhang, Y., Nie, B., and Gao, Y. Robust locomotion policy with adaptive lipschitz constraint for legged robots. IEFE Robotics and Automation Letters, 2024.
Zhao, Z., Zuo, S., Zhao, T., and Zhao, Y. Adversarially regu- larized policy learning guided by trajectory optimization. In Learning for Dynamics and Control Conference, pp. 844-857. PMLR, 2022.
Takase, R., Yoshikawa, N., Mariyama, T., and Tsuchiya, T. Stability-certffied reinforcement learning via spectral normalization. arXiv preprint arXiv:20/2. 13744, 2020.

---

LipsNet++: Unitying Filter and Controller into Policy Network
A. Theoretical Results
Lemma A.1 (Equivalent Form of Lipschitz Constant), Suppose f: R + Rm is a continuously differential neural network. Then its local Lipschitz constant (x) has an equivalent form besides equation (6):
K(x)
max x(EB(a.p)
[V
(8)
Proof. We assume the local Lipschitz constant of over (ix,, P) is Ka, which means K₂ max,, (xEB((.))) (a) Firstly, we prove that |[V & K2, Va' E p). Because the local Lipschitz constant is Kg. we know that 11((n)) (1=2)] S K2 |21 æ2/1, |l, 22 E ()r,
[f(z))-
| |
Let h(t) f(x' + u) where x' E B(I, p). t E R. and E R", then its first-order derivative function is ((())
ff(w' +t v. From the Laaton-Lebbnizformmuue we know
((a)-h0p = f d(())dt dt,
which means
J( aa | VS(a' + +l. dt.
By taking the 2-norm on both sides and considering the condition (9), we get
1%
Vf(a' u) dt [1 + +a f('')l
< <aallloll
Divide a both sides then let a 0+, get
[V lla S Kn ||a||| Ve.
From the definition of matrix norm, we know
Vf(x') all K2 Ya' E B(x.p). 0. |||||
max
(b) Secondly, we prove that (ax,)) [[)((' 2 Kg. Let h(t) f(z + t(z2 x1)) where E (0. 1) and Ii, 42 E B(x, p). then its first-order derivative function is (((()= = Vʃ(zi + 1(x2 - =1)) (x2 x1). From the Newton- Leibniz formula, we know
h(1) - h(0) = [ H'(t)dt dt,
which means
f(x2)-f(x)) = /' Vf(ai + + x1)) (x2 - xτ) dt
= (f vf(xi + - dt (xz -1))
Take the 2-norm on both sides, get
(l(())) - S(x))l = ( vs(m + 1(x2 [111+((122-)
21)) dt)
(x2 (x2-x))) m1)
<
Vf(zi + t(x2 - mz)) dt
||اا xill
[ 1(52 -1)iil dt
|02 x1l
max
| ||| ||ا| =1ll-
x(B.pp.))
12

---

LipsNet++: Unifying Filter and Controller into Policy Network
Therefore,
[[((m) [(2))]
max ='EB(z.p)
[Vf(2'I
22 E B(x, p):
|2] x2
which means
[1((r)) f(22)]
max z' (B(,p)
[Vf(2')I|
max
| I2l
K2.
Considering both (a) and (b), we know that [V'(l')| S Kz, Yal E p)x,p) and max [V 2 Kz. Therefore, we can conclude that maXp))B(z.p)) [V)((') Kg.
Theorem A.2 (Lipschitz's ' Jacobian Approximation). Let f: : + R be a continuously differentiable neural network. The Jacobian norm ,'Il is an upproximation of f's local Lipschitz constant within neighborhood B(x, p), centered with radius p. The approximation error is
(V. 8 + o(8)
max BEB(O.P)
where o()) a higher-order infinitesimal term with respect to 6. Moreover, as p + the Jacobian norm converges the exact local Lipschitz constant, i.e.
lim p
V ((x).
Proof. By Definition 3.1 and Lemma A.1, we know that
[1((1) [(x2)
K(x) in E(
2 x2ll
max a'EB(z,))
fVx''l
max 80
By conducting the first-order Taylor expansion for Vf(r + 5) we get that
K(x)
max 8EB(O.,)
[V+
+ v>f()))))' 8 + o())
||=x((>)|
max 6EB(0.p)
V
V
8 8 o(0)
Therefore, the Jacobian norm ||V.][| is approximation of the local Lipschitz constant K(x), with the approximation enror max SEB(0,p) (V> [x)]) T o()) Furthermore, when p - 0, the approximation enror also approaches 0. This means that [|ll fl + K(r) when + 0.
B. Fundamental Reasons of Action Fluctuation
Non-smoothness of policy network. A non-smooth policy network means that RL fits a non-smooth policy function mapping from the state to control action. The mapping function has significant output differences even if the inputs closely adjacent. Consequently, when the state changes with time, a non-smooth action sequence is generated. Figure 14 visualizes the effect of policy non-smoothness..
Existence of observation noise. The noise results in the discontinuous changes in observations, making the actions produced by the policy network at the adjacent time stamps erratically differ. Even the policy function fitted by the policy network smooth enough, actions can still be fluctuated because of the erratic observation noise.
13

---

LipsNet++: Unifying Filter and Controller into Policy Network
Policy function
Action trajectory
Action
Action
2
2
-3
State 2
State
20
40
60
80
100
Time
(a) Smooth policy function
(b) Smooth action trajectory
Policy function
Action trajectory
Action
Action
-3
State 2
State
10
20
40
90
00
100
Time (s)
(c) Non-smooth policy function
(d) Non-smooth action trajectory
Figure 14: Effect of policy non-smoothness.
C. Pseudocode of LipsNet++
Algorithm Forward and backward propagations of LipsNet++
1: Input: last N step observations oz, 0:-1,' -+-N+1, original actor loss L, original network parameter 6.
/* Forward propagation x/
2: x + [o 01-1 ... NN-N+1] T
3: +- FF)(2)
4: X symmetrizo X H)
5: f IFFT
6: o + the first row in ã
7: 4. t f(ō))
/* Backward propagation +/
8: L" + L d V.. + A, | [|)
9: &new V
10: Output: control action 0r, updated network parameter Onew-
14

---

LipsNet++: Unifying Filter and Controller into Policy Network
D. Double Integrator: Detailed Implementation and Results
The double integrator classic control task with linear dynamics and quadratic cost function, namely linear quadratic (LQ) control task. The environment used in this paper a particle-moving enviromment. We train in noise-free environment and test in noisy environment with various noise level to comprehensively evaluate policy networks. We use model-based RL algorithm, INFADP (Li, 2023), train different policy networks including MLP (Rumelhart al., 1986), MLP-SN (Takase et al., 2020), LipsNet-G (Song et al., 2023), LipsNet-L. (Song et al., 2023), and LipsNet++. The hyperparameters for INFADP are listed in Table 2.
We set different observation noise amplitudes and compare the performances of MLP, MLP-SN, LipsNet-G, LipsNet-L, and LipsNet++. Table 3 and Table summarize the TAR and AFR, respectively. Figure shows the variation trends TAR and AFR the noise increases. As shown in Figure 7(a), the TAR of LipsNet++ decreases much slower than that of the other networks. As shown in Figure 7(b), the AFR of LipsNet++ increases much slower than that of the other networks. These results indicate that LipsNet++ has superior action smoothness and noise robustness compared to previous works.
Table 2: Hyperparameters for INEADP.
Hyperparameter
Value
Replay buffer capacity
100000
Buffer warm-up size
1000
Batch size
64
Discount y
0.99
Target network soft-update rate
0.2
Initial random interaction steps
0
Interaction steps per iteration
8
Network update times per iteration Prediction step
1
Action bound
5,
Exploration noise std. deviation
Hidden layers in subnetwork f Activations in subnetwork f Hidden layers in critic network Activations in critic network Optimizer
[64, 64
ReLU
664, 64
ReLU
Adam
Actor learning rate
3 10-4
Critic learning rate length historical obsv. coefficient Ae
8 10-4
0.01
coefficient A
1
Table 3: Comparison of TAR on double integrator environment. The observation noise in each dimension is distributed
U (-o,o). The data this table is visualized in Figure 7(a).
Noise
Methods MLP-SN
MLP
CAPS
L2C2
LipsNet-G
LipsNet-L
LipsNet++
0.01
-51.0 2.11
-52,1 201
-55.0
-62.0 101
-53.2 201
-55.3 =
-56.5 6.1
0.05
-53.5 02
-53.0 02
-55.4
-62.3 1044
-54.3 203
-55.6
-56.8
0.1
-59.5 0.
-56.9 a06
-57.5 =05
-62.8 +0z
-54.2 207
-56.0
-57.1 0.6
0.2
-78.4 118
-67.9 +:0
-63.4 155
-65.9 1:17
-59,8 +111
-58.7
-57.9 4.66
0.3
-103.2
17
-85.9 430
-82.3 444
-71.8 2:3
-74.3 #211
-65.3 1.6
-59.9 19
15

---

LipsNet++: Unitying Filter and Controller into Policy Network
Table 4: Comparison of AFR on double integrator environment. The observation noise in each dimension is distributed
in U (-o,0). The data this table is visualized in Figure 7(b).
Noise
Methods MLP-SN
MLP
CAPS
L2C2
LipsNet-G
LipsNet-L
LipsNet++
0.01
0.02
0.01 0:001
0.01 0:001
0.01 0:00
0.01 0.00
0.01 +: 0.0
0.00 0.01
0.05
0.11 0001
0.07 4 0.01
0.05 1 0.1
0.03 =
0.04 = 0.01
0.03 = 0.0
0.01
0.1
0.19
0.15
0.10
0.06
0,07
0.06
0.02
0.2
0.34 0.02
0.26
0.20
0.13
0.17
0.12 not
0.03
0.3
0.48 0.02
0.37 0.02
0.33 0.02
0.20 0.1
0.28 0.01
0.20 + 0.1
0.05 0.01
E. Ablation Study for Two Techniques
In this appendix, we implement ablation study for the two techniques proposed in our paper, i.e. Jacobian regularization and Fourier filter layer. The two techniques respectively tackle the two fundamental reasons of action fluctuation, as described in Section 3.1. The Jacobian regularization enhances the smoothness of policy network by introducing the Jacobian norm in actor loss function. Similarly, the Fourier filter layer enhance the noise robustness of policy network by introducing the Frobenius norm of the filter matrix in actor loss function. The resulted actor loss is illustrated in Equation
= C+ ~ V + Ao. H
In order to validate the effectiveness each technique, the two coefficients Å and Å are set to zero in turn. The performance result on double integrator environment is shown in Figure 15. The performance result on DMControl's Cheetah and Walker environments is shown in Table 5. These results shows that setting either coefficient to will lead rapid decrease in TAR and rapid increase in AFR when the noise increases. It indicates that both the Jacobian regularization and Fourier filter layer are effective techniques and they are both indispensable.
-50
0.5
Av 0, An
À = 0.01, A, =
Action fluctuation ratio
0.4
0.01,
Total average return
-70
0.3
-80
0.2
-90
Àk = A, 1
0.1
-100
A = 0.01, A
A = 0,01,Ar =1
0.0
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.00
0.05
0.10
0.15
0.20
0.25
0.30
Observation noise amplitude
Observation noise amplitude
TAR
(b) AFR
Figure 15: Ablation study for Jacobian regularization and Fourier filter layer double integrator environment.
F. Sensitivity Analysis for D and An
In this appendix, we provide the sensitivity analysis for the hyperparameters A and XІ- We design experiments to demonstrate that the two hyperparameters have low sensitivity, making LipsNet++ convenient for tuning and easy to use.
Similar to the approach in Appendix E, we fix one hyperparameter and then vary the other to observe the changes in TAR and AFR on double integrator environment. As shown Figure 16, when X fixed at and \ varies between 0.001,
16

---

LipsNet++: Unitying Filter and Controller into Policy Network
Table 5: Ablation study on Cheetah and Walker. The result shows that setting either coefficient to zero will lead to an increase in AFR, which indicates the two techniques are all effective and indispensable.
Environment
A
An
Total average return
Action fluctuation ratio
10-3
10-3
822
0.94 100
Cheetah
0
10-3
822 15
1.08 0.0
10
0
821 + 17
1.21 + 002
10-2
10 10-3
961 12
0.78 +
Walker
0
10
958 15
0.98 0.02
10
0
940 14
1.89 0.1
0.01, and 0.1, the performance differences are significant. However, when A is fixed at 1 and ) varies between 0.01 and 0.02, the performances are essentially consistent. A similar phenomenon can also be found for hyperparameter An as shown Figure When As fixed at 0.01 and Ån varies between 0.1, and 10, the performance differences are significant. However, when A fixed at and 0 varies between and 2, the performances are essentially consistent.
0.30
A = 0.001, A.
-60
A. 0.01, A,
Total average retum
-70
Action fluctuation ratio
A 0.02, An =1
0.20
A = 0.1, A,
-80
0.15
100
0.10
0.001, An =1
-110
A 0.01, An
0.05
0.02,
-120
A 0,
0.00
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.00
0.05
0.10
0.15
0.20
0.25
0.30
Observation noise amplitude
Observation noise amplitude
(a) TAR
(b) AFR
Figure 16: Sensitivity analysis for Ag-
The above results imply that the hyperparameter Å and Å have low sensitivity. Only the magnitude of hyperparameters have significant impact on performance, while changing the hyperparameters within appropriate magnitude has minimal effect performance. Therefore, when tuning parameters, the user only need to set appropriate magnitude. It makes LipsNet++ convenient for tuning and easy to use.
G. Sensitivity Analysis for N
In this appendix, we provide the sensitivity analysis for hyperparameter N, which represents the length of historical observations. We design experiments to demonstrate that N exhibits low sensitivity when the length of historical observations is sufficiently long, making LipsNet++ convenient for tuning and easy to use.
The values of N are set to range from to 32 in the double integrator environment. The Figure 18(a) and (b) show the trend TAR and AFR when N changes. The result shows that the performance no longer improves once exceeds a threshold, suggesting a low sensitivity. Therefore, users only need to set a relatively large value N, making LipsNet++ very convenient for tuning.
17

---

LipsNet++: Unitying Filter and Controller into Policy Network
A 0.01, 0.1
-60
0.25
Ag 0.01, A, =1
-70
Action fluctuation ratio
A = 0.01, An =2
Total average return
-80
0.01, 10
-90
0.15
-100
-110
0.10
0.01, 0.1
-120
0,01, An 11
0.05
A 0.01, A,
-130
0.01, 10
140
0.00
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.00
0.05
0.10
0.15
0.20
0.25
0.30
Observation noise amplitude
Observation noise amplitude
(a) TAR
(b) AFR
Figure 17: Sensitivity analysis for An.
-56
0.18
A 0.01. 0.0
0.16
Total average relurn
-58
Action fluctuation ratio
0.12
-60
0.10
-62
0.08
-64
0.04
À = =, 0.
12
32
32
Length of history obs. N
Length of history obs.
(a) TAR
(b) AFR
Figure 18: Sensitivity analysis for N.
18

---

LipsNet++: Unitying Filter and Controller into Policy Network
H. Computational Efficiency Analysis
To evaluate the computational efficiency, we provide the detailed forward and backward processing time of policy networks. All policy networks are from the network trained in double integrator environment. This analysis implemented on AMD Ryzen Threadripper 3960X 24-Core Processor. For MLP-SN, the number of power iterations is set to 1, whose time usage included in the backward stage. Similarly, the computation times for Jacobian norm and Frobenius norm in LipsNet++ included in the backward stage. The length of historical observations used in LipsNet++ is 8.
The results are summarized in Table 6. Compared to the previous SOTA network LipsNet, LipsNet++ has significantly faster speed for forward propagation. This allows LipsNet++ to be applied in high-real-ting tasks. We acknowledge that backward propagation speed of LipsNer++ is relatively slow, but we have devised a solution to accelerate this in future work by using multiple forward propagation and zero-order gradient estimation to compute the Jacobian matrix.
Table Forward and backward propagation time comparison.
Settings
Policy network
Propagation
Batch size
MLP
MLP-SN
LipsNet-L
LipsNet++
1
0.10 ms
0.11 ms
0.75 ms
0.16 ms
forward
100
0.11 ms
0.12 ms
1.41 ms
0.25 ms
1
0.17 ms
0.76 ms
0.45 ms
1.98 ms
backward
100
0.28 ms
0.89 ms
0.73 ms
2.48 ms
Since network propagation times constitute only a part of the overall RL training process, we next compare the total training wall-clock times. Table 7 presents the wall-clock times for 1M iterations of TD3 on the DMControl environments. The results show that, on average, the training time of LipsNet++ is 1.6 times that of MLP. The difference in wall-clock time is not significant as the difference in backward time shown in Table 6, as RL algorithms involve additional time-consuming steps beyond backward, such as sampling and evaluation.
Table 7: Training wall-clock time comparison. The data show the wall-clock times used for IM iterations in TD3. On average, the training time of LipsNet++ is 1.6 times that of MLP.
Env Cheetah Walker
Network
Total
Cartpole
Reacher
MLP
120 min
118 min 128 min
125 min
491 min
LipsNet++
194 min
195 min
206 min
204 min
799 min
In conclusion, LipsNet++'s forward time is under 0.2 ms, making it suitable for real-time applications. While the training wall-clock time shows slight increase, it remains acceptable and has clear pathways for future optimization.
I. Deep.Mnnd Control Suite Benchmark
The DeepMind Control Suite (DMControl) (Tassa et al., 2018) encompasses a collection of meticulously crafted continuous control tasks. These environments feature consistent structures, rewards that are both interpretable and normalized, facilitating more straightforward comparison of performance across different algorithms. Developed in Python and leveraging the MuJoCo physics engine (Todorov et al., 2012). DMControl currently stands one of the most esteemed benchmarks for evaluating RL and continuous control tasks.
In DMControl, the term "domain" denotes specific physical model, whereas a "task" corresponds to an instantiation of that model with a defined Markov Decision Process (MDP) structure. For instance, within the cartpole domain, the distinction between the swingup and balance tasks lies the initial orientation of the pole: it is initialized pointing downward in the swingup task and upward in the balance task, respectively. In the following figures, we provide detailed descriptions of the domains used in this paper, with each domain's 's name followed by tuple of three integers that denote the dimensions of the state, action, and observation spaces, respectively, formatted as (dim (8), dim (A), dim (0)
19

---

LipsNet++: Unifying Filter and Controller into Policy Network
Figure 19: Cartpole(4, 1, 5): This domain features cart connected to a pole via an unactuated joint. It encompasses set of four distinet tasks. In the context of our experimental setup, focus on the swi ngup task. Here, the pole is initially positioned downward, and the objective is to apply appropriate forces to the cart swing the pole upward and maintain its upright position.
Figure 20: Reacher(4, 2, 7): This domain comprises two interconnected poles with sphere whose initial position is randomly determined. One end the linked poles is anchored at the origin of the coordinate space, while the other remains free to move. The domain offers two distinet tasks, and we focus on the easy task. The task requires the application of forces the pendulum ensure that its endpoint remains within the red area at all times.
This domain features planar bipedal and it able crawl forward by its legs. It involves single task, namely the run task. In the initial state of the environment, the agent's s pose is random, typically in non-standing position. In this task, the challenge is to control the planar biped to achieve an upright standing position and subsequently propel it forward into a running motion with targeted forward velocity.
Figure 21: Cheetah(18, 17):
Figure 22: Walker(18, 6, 24): This domain includes planar walker. This environment simulates simple locomotion task of humans, with the agent possessing two legs and advancing in an upright posture. It comprises three distinet tasks, and our experiment focus the walk task. In this task, the objective is to control the walker to maintain an upright torso posture, achieve the specified torso height, and maintain consistent forward velocity.
J. DMControl: Detailed Implementation and Results
We employ the Twin Delayed Deep Deterministic Policy Gradient (TD3) (Fujimoto et al., 2018), model-free RL algorithm, to train on DMControl. The hyperparameters for TD3 remain consistent across all environments, except for the coefficients Av: An, and the length of historical observations N. The hyperparameters for TD3 are listed in Table 8. The environment- related byperparameters are listed in Table 9.
To evaluate comprehensively. networks are tested on both noise-free and noisy environments. For noisy environments, the noise amplitudes are listed in Table 10. We compare LipsNet++ with MLP, LipsNet-G, LipsNet-L using 10 seeds. All results are summarized in Table 11 and 12, from which we can find that LipsNet++ has the highest TAR and the lowest AFR in all cases. These results imply LipsNett+ has good action smoothness and noise robustness.
For comparing LipsNet++ and MLP-SN, train them on DMControl Reacher environment. We use a 3-layer MLP-SN network and manually tuning its spectral norm of each layer by grid search. The global Lipschitz constant of MLP-SN is the product the spectral norms of all layers. The results are listed in Table 13, from which we can find that LipsNet++ outperforms MLP-SN under all hyperparameter settings. As shown in Figure 23, the trends of TAR and AFR imply that even with fine adjustments to the spectral norm, the overall performance of MLP-SN remains inferior to that of LipsNet++. We refrain from comparing LipsNet++ MLP-SN across all environments used in this paper, because this would necessitate the manual tuning of spectral norm hyperparameters for each layer, which have unwieldy number of potential hyperparameter combinations.
Additionally, the learned filter matrix H in LipsNet++ is visualized in Figure 24 to show the noise filtering ability, Figure 24(a) and 24(b) show the frequency distributions of observation noise-free free environment and noisy environment, respectively. Their shades of color represent the intensity of frequency. The color in Figure 24(c) denotes the magnitude of elements in matrix H, which determines which frequencies are suppressed or strengthened. The result implies that
20

---

LipsNet++: Unifying Filter and Controller into Policy Network
Table 8: Hyperparameters for TD3.
Hyperparameter
Value
Replay buffer capacity
1000000
Buffer warm-up size
1000
Batch size
100
Discount
0.99
Target network soft-update rate т
0.005
Target noise
0.2
Target noise limit
0.5
Exploration noise std. deviation
0.1
Policy delay times
2
Initial random interaction steps Interaction steps per iteration Network update times per iteration
25000
50
50
Hidden layers in subnetwork f Activations in subnetwork
[64, 64
ReLU
Hidden layers in critic network Activations in critic network
664, 64
ReLU
Optimizer
Adam
Actor learning rate
1 10-3
Critic learning rate
1 10-3
Table 9: Environment-relatedd hyperparameters in DMControl.
Env
A
A
Length of his. obsv. N
Cartpole
10-2
10-2
Reacher
10-2
10-3 3
5
Cheetah
10-3
10
5
Walker 10-2
10-3
10
Table 10: Observation noise in DMControl. The observation noise in each dimension distributed in U(-o,o).
Env
Noise amplitude
Cartpole
(0.1,0.1,0.1,0.2,003
Reacher
[0.001 repeats 7 times]
[0.01, 0.01, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.5, 0.05, 0.1, 0.5, 0.5, 0.5, 0.5, 0.5)
Cheetah
Walker
[0.25 repeats 24 times]
21

---

LipsNet++: Unifying Filter and Controller into Policy Network
Table 11: Total average return in DMControl.
Environment
ML.P
LipsNet-G
LipsNet-I.
LipsNet++
Cartpole
805 + as
691 + 10
831 +
841 +
noise-free
Reacher
981 = 10
979 11
983 + 10
988 : 10
env
Cheetah
816 +
702 + 10
822 444
829 + 15
Walker
926 +:
956 20
945
962 1
Cartpole
763 9
517 4:1
823 6
825 1
noisy env
Reacher
972 + 25
973 + 18
978 1
982 + 10
Cheetah
813 +
680 17
818 11
822 :
Walker
911 +: 26
942
929
961 :
Table 12: Action fluctuation ratio in DMControl.
Environment
MLP
LipsNet-G
LipsNet-L
LipsNet++
Cartpole
0.04 1: 0.00
0.08 i 0.000
0.01 i 0.00
0.01 :: 0.00
noise-free
Reacher
2.07 + 0.600
0.13 0:24
0.01 + 0.00
0.01 : 0.00
env
Cheetah
1.08 4: 0.2
0.92 0.0
0.94 + 0.0
0.90 0.01
Walker
1.89
1.25
0.93 + 0.01
0.74
Cartpole
0.58 + 0.03
0.75 + 0.09
0.17 + 0.01
0.13 0.00
noisy env
Reacher
2.41 0.288
0.04 +: 0.00
0.04 + 0.03
0.01 = 0.00
Cheetah
1.13 1: 0.02
1,00 0.01
1.08 :: 0.01
0.94 0.01
Walker
2.02 1 0.03
1.68 0.01
1.21 1 0.01
0.78 :: 0.01
Table 13: Performance of LipsNet++ and MLP-SN on DMControl Reacher.
Network
Total average return
Action fluctuation ratio
Spectral norm for each layer
Name
5.0
760 +
0.01 = 0.00
5.5
831 102
0.01
MLP-SN
5.8
954
0.08 0.05
6.0
967 4:
0.13 0.08
LipsNet++
988 ::
0.01 :: 0.00
22

---

LipsNet++: Unifying Filter and Controller into Policy Network
LipsNet*+
MLP-SN
MLP-SN (5.8)
(6.0)
Total average retum (t)
900
MLP-SN (5.5)
800
MLP-SN (5.0)
700
0.00
0.06
0.10
0.15
Action fluctuation ratio
Performance comparison of LipsNet++ and MLP-SN on DMControl Reacher. The numbers in parentheses are the spectral norm of each layer in MLP-SN. Points that closer to the top-left corner indicate better performance. It shows that even with fine adjustments to spectral norm, MLP-SN's overall performance remains inferior to that of LipsNet++.
Figure 23:
the learned filter matrix mainly focus on the frequencies that containing observation information, and rarely focus on the frequencies that containing noises. In other words, LipsNet++ can automatically extract the important frequencies and filter out the noise frequencies.
10
14
Historical 0
Historical 00 rank
Hictorical obev. rank
12
6
2
12
6
10
11
9 10
Observation dimension
Observation dimension
Observation dimension
(a) Obsv. frequency (noise-free)
(b) Obsv. frequency (noisy)
(c) Filter matrix
Figure 24: Filter matrix and observation frequency in walker environment. The color in (a) and (b) represents the intensity of frequency. The color in (c) represents the magnitude of elements in matrix H. The color distribution in (c) implies LipsNet++ can automatically extract the important frequencies and filter out the noise frequencies.
K. Comparison to Reward Penalty
Punishing the difference between consecutive actions in the reward is an effective way to smooth the actions in some environments. However, such an approach breaks the Markov property, which affects the performance, albeit to a minor extent certain environments. Moreover, we found that adding reward penalty sparse reward environment increases action fluctuation rather than smoothing it, which is consistent with the finding by Chen et al. (2021) and Song et al. (2023).
Cartpole in DMControl is sparse reward environment. The reward is when the pole is within 30 of the vertical and 0 otherwise. We implement TD3 in this environment, punishing the difference between consecutive actions in the reward. Specifically, the new reward is Torigin + a ||q++1 - azll, where Torigin is the original sparse reward, a is the penalty coefficient and a,+1 the output of actor network under $1+1- The experiment results are summarized in Table 14. The results imply that simply adding reward penalty in the sparse reward environment increases the action fluctuation ratio. Superiorly, LipsNet++ can smooth actions even in the sparse reward environment.
23

---

LipsNet++: Unitying Filter and Controller into Policy Network
Table 14: Comparison to reward penalty.
Method
Penalty coefficient
Total average return
Action fluctuation ratio
TD3 (MLP, reward penalty)
0.01
825 0.5
0.27
TD3 (MLP, reward penalty)
0.1
819
0.21
TD3 (MLP, reward penalty)
13 0.5
0.02 : 0.00
TD3 (MLP)
805
0.04
TD3 (LipsNet-G)
691 = 1.0
0.08 1: 0.00
TD3 (LipsNct-L)
831 1 0,9
0.01 0.00
TD3 (LipsNet+++)
841 0222
0.01 0.00
L. Mini- Vehicee Driving: Introduction of Vehicle and Task
The vehicle rohot is driven by differential wheels, which is shown in Figure 26. The task for the robot is track given reference trajectory and reference velocity while avoiding obstacle. The setting of observations and actions in this environment is described in Table 15.
Figure 25: Physical vehicle robots.
For the perception, the vehicle equipped with LiDAR, obtaining its position by matching with a pre-scanned point cloud map generated by SLAM. In this way, vehicle can detect its horizontal coordinate æ. vertical coordinate y, and heading angle 6. The vehicle is also equipped with speed sensor that measures the linear velocity and angular velocity To increase the complexity of the task, another vehicle is used as obstacle vehicle. Both vehicles exchange real-time state information with each other via WiFi communication.
For the decision-making and control, a policy network trained by RL is deployed on the vehicle. After inputting the perceived observation into the network, control actions are computed, namely linear acceleration and angular acceleration W. Then, control actions are sent to motor to execute the command. The overall control mode is shown in Figure
As illustrated in Section 4.3, there are four diverse scenarios in this environment. The scenario descriptions are listed in Table 1. To describe the scenario settings more clearly, Figure 28 shows the map and vehicle routes for each scenario. Figure 29 shows the corresponding snapshot for each scenario. In scenarios 1 1-3, the obstacle vehicle goes straight with constant speed. In scenario 4, the obstacle vehicle is manipulated by human.
24

---

LipsNet++: Unifying Filter and Controller into Policy Network
Table 15: Variables in mini-vehicle driving env.
Variable
Description
longitudinal speed
yaw rate
71
Heading direction
dy
trajectory offset
Y
6Φ
heading angle error
8y
Tangent line
8u
speed error
8Φ
Obsv.
Ar
obstacle's ' relative position
Ay
obstacle's 's relative position
Reference trajectory
AΦ
obstacle's s relative angle
Au
obstacle's 's relative speed
Aw
obstacle's ' relative yaw rate
Figure 26: Vehicle kinematics model. The vehicle moves by two differential wheels, tracking the reference trajectory.
longitudinal acceleration yaw acceleration
Action
Reference trajectory
vehicle speed
Policy network
Sensors
v, w
vehicle pose
LiDAR
x,y,
WiFi 0yy
obstacle state
LipsNet++
x' x'y', 1..610 w
Action
ú, a
Surrounding car (obstacle)
Ego car (RL robot)
Motor
Figure 27: Flowchart of vehicle control mode.
25

---

LipsNet++: Unifying Filter and Controller into Policy Network
Obstacle
robot
>>
Reference trajectory
RL robot (0)
D>
>>
Reference trajectory
Obstacle
(a) Scenario
(b) Scenario
Reference
trajectory
Obstacle
Obstacle
Reference trajectory
RL robot
RL robot
(c) Scenario
(d) Scenario
Figure 28: Scenario illustration of mini-vehicle driving environment.
26

---

LipsNet++: Unifying Filter and Controller into Policy Network
Reference trajectory
Reference trajectory
Obstacle
Obstacle
RL robot
RL robot
(a) Sccnario
(b) Scenario 2
Reference trajectory
Reference trajectory
Obstacle
Obstacle
RL robot
RL robot
(c) Scenario
Scenario
Figure 29: Scenario snapshots of mini-vehicle driving environment.
Historical obsv. rank
03
0.5
Historical absu. rank
0.5
0.4
0.4
0.4
0.3
10
0.3
10
0.3
Historical
0.2
0.2
0.2
a5.
15.
0.1
Observation dimension
Observation dimension
Observation dimension
(a) Obsv. frequency (noise 0)
(b) Obsv. frequency (noise 10)
(c) Filter matrix
Figure 30: Filter matrix and observation frequency mini-vehicle driving environment. The color in (a) and (b) represents the intensity of frequency. The color in (c) represents the magnitude of elements in matrix H. The color distribution in (c) implies LipsNet++ can automatically extract the important frequencies and filter out the noise frequencies.
27

---

LipsNet++: Unifying Filter and Controller into Policy Network
M. Mini- Vehicle Driving: Detailed Implementation and Results
In the training stage, observation noise set to zero. In the vehicle testing stage, multiple different magnitudes of observation noise are added to thoroughly test the performance of policy networks. The noise magnitude is adjusted using the coefficient Joof IR {0}, such that noise is distributed in co Dbses). And the baseline noise obse set to:
base
[0.01
0.03
0.01
0.03
0.03
0.01
The reward function is defined constant minus the penalties related to tracking error, vehicle instability, and collision violation:
r= - 0.4(6y)² 0.1(6p)² - 1.3 - 0.01 - 0.01 - 0.01 - 2 I[p < 0.94),
where o represents the distance between the centers of the two vehicles, calculated speed is set to 0.3m/s, meaning dv = 0.3.
/Ar² + Ay². The reference
The Distributional Soft Actor-critic (DSAC) (Duan et al., 2025), model-free RL algorithm, is used to train the vehicle robot.
The hyperparameters for DSAC are listed in Table 16. The tests all scenarios are accomplished by the same networks.
All results shown Figure 31~49, Table 17 lists the figure index for each scenario.
Table 16: Hyperparameters for DSAC.
Hyperparameter
Value
Replay buffer capacity
1000000
Buffer warm-up size
10000
Batch size
256
Discount y
0.99
Target network soft-update rate т
0.005
Policy delay times
Temperature parameter
0.2
Hidden layers in critic network
[256, 256]
Activations in critic network
ReLU
Hidden layers in subnetwork f
[256, 256]
Activations subnetwork f
ReLU
Optimizer
Adam
Critic learning rate
1 10-4
Actor learning rate
1 10-4
Coefficient 2
0.1
Coefficient A
0.04
Length of historical obsv. N
20
28

---

LipsNet++: Unitying Filter and Controller into Policy Network
Table 17: Figure indices for the results of mini-vehicle driving environment.
Noise amplitude
Scenario and network
Snapshots
10
MLP
Figure 31
Figure 33
Scenario
Figure 35
LipsNet++
Figure 32
Figure
MLP
Figure 36
Figure 38
Scenario
Figure 40
LipsNet++
Figure 37
Figure 39
ML.
Figure 41
Figure 43
Scenario
Figure 45
LipsNet++
Figure 42
Figure 44
MLP
Figure 46
Figure 48
Scenario
URL
LipsNet++
Figure 47
Figure 49
Table 18: Performance summary in mini-vehicle driving environment.
Task setting
Scenario
Scenario
Scenario
Policy network
Noise
TAR
AFR
TAR
AFR
TAR
AFR
amplitude
234.7
0.02
252.6
0.04
287.5
0.03
235.2
0.02
252.0
0.04
288.5
0.03
LipsNet++
232.8
0.08
254.1
0.08
289.6
0.08
10
233.6
0.14
249.6
0.16
290.3
0.14
224.5
0.27
252.7
0.28
281.3
0.23
238.4
0.04
254.6
0.17
293.5
0.15
237.8
0.58
250.4
0.58
293.0
0.55
MLP
232.7
1.68
250.0
1.62
289.6
1.58
10
225.0
2.03
247.2
2.24
283.3
2.17
20
209.8
2.53
238.9
2.65
267.9
2.65
29

---

LipsNet++: Unifying Filter and Controller into Policy Network
Refarence trajectory
Longitudinal
RL robot
2-
Obstancle
20
15
Time
Y
(b) Action I: Longitudinal acceleration
10
accelaration
Speed
Yow
-5
Time
Time
5
10
15
20
25
2
Time
(a) Vchicle trajectory
(c) Action Yaw acceleration
Figure 31: MLP performance in scenario The noise amplitude is 0.
25
Reference trajectory
Longitudinal
RL robot
0-
2-
Obstancle
20
-1-
1-
10
15
20
25
15
Time
(b) Action I: Longitudinal acceleration
Yaw accelaration
Spent 0.4
rate
20
Time
Time
10
15
20
Time
(a) Vehicle trajectory
(c) Action 2: Yaw acceleration
Figure 32: LipsNet++ performance scenario 1. The noise amplitude
Longitudinal acc.
25
Reference trajectory
20
1-
10
15
20
25
15
Time
(b) Action 1: Longitudinal acceleration
10
accelaration
Speed
Yow rate
0
20
20
Time
Time
10
2
Time
(a) Vehicle trajectory
(c) Action 2: Yaw acceleration
Figure 33: MLP performance in scenario 1. The noise amplitude is 10.
30

---

LipsNet++: Unifying Filter and Controller into Policy Network
Reference trajectory
Longltudinal
RL robot
2-
Obstancle
20
10
18
Time
(b) Action I: Longitudinal acceleration
10
Yaw accelaration
Spees
Wwwmmm
You rata
20
Time
Time
10
15
20
25
2
Time
(a) Vehicle trajectory
(c) Action 2: Yaw acceleration
Figure 34: LipsNet++ performance in scenario 1. The noise amplitude is 10.
t= Os
Reference trajectory
t= 3s
Obstacle
Obstacle
RL robot
RL robot
(a) t00s
(b) t=3s
t 10s
t= 17s
RL robot
RL robot
Obstacle
Obstacle
(c) 1=10s
(d) 1=17s
Figure 35: Snapshots of scenario 1.
31

---

LipsNet++: Unifying Filter and Controller into Policy Network
Refarence trajectory
Longitudinal
RL robot
2-
Obstancle
20
15
Time
(b) Action I: Longitudinal acceleration
sook 30r
10
Yaw accelaration
Speed
-5
Time
Time
5
10
15
20
25
2
Time
(a) Vchicle trajectory
(c) Action Yaw acceleration
Figure 36: MLP performance in scenario 2. The noise amplitude is 0.
25
Longitudinal
Reference trajectory
RL robot
0-
2-
Obstancle
20
-1-
1-
10
15
20
25
15
Time
0
(b) Action I: Longitudinal acceleration
200 1000000
Yaw accelaration
Spent
rate
0-
20
Time
Time
10
15
20
(a) Vehicle trajectory
(c) Action 2: Yaw acceleration
Figure 37: LipsNet++ performance scenario 2. The noise amplitude
Longitudinal acc.
25
Reference trajectory
20
1-
10
15
20
25
15
Time
(b) Action 1: Longitudinal acceleration
10
accelaration
Spoed
Yow rate
0
20
20
Time
Time
2
X 00-
Time
(a) Vehicle trajectory
(c) Action 2: Yaw acceleration
Figure 38: MLP performance in scenario 2. The noise amplitude is 10.
32

---

LipsNet++: Unifying Filter and Controller into Policy Network
Reference trajectory
Longltudinal
mmm
RL robot
2-
Obstancle
20
18
Time
(b) Action I: Longitudinal acceleration
10
Yaw accelaration
Spee
hmm
Yow rate
20
Time
Time
10
15
20
25
2
Time
(a) Vehicle trajectory
(c) Action 2: Yaw acceleration
Figure 39: LipsNet++ performance in scenario 2. The noise amplitude is 10.
Reference trajectory
t Os
t t= 5s
Obstacle
Obstacle
RL robot
RL robot
(a) t=
(b) t=5s
t = 11s
t t 20s
RL robot
Obstacle
RL robot
(c) t=lss
(d) 1=20s
Figure 40: Snapshots of scenario 2.
33

---

LipsNet++: Unifying Filter and Controller into Policy Network
Refarence trajectory
Longitudinal
RL robot
25
Obstancle
20
0
Time
>
XXXXUUXX
(b) Action I: Longitudinal accelcration
20
2-
10
Yaw accelaratio
Yew
5
Time
0-
15 Time
20
25
-5
-3
(a) Vchicle trajectory
(c) Action Yaw acceleration
Figure 41: MLP performance in scenario 3. The noise amplitude is 0.
Reference trajectory
Longitudinal
RL robot
0-
Obstancle
-1-
4-
20
15 Time
20
25
Spece
3-
1000OO
15
(b) Action I: Longitudinal acceleration
Time
10
accelaration
Sow rate
1
-5
Time
15
20
Time
(a) Vehicle trajectory
(c) Action 2: Yaw acceleration
Figure 42: LipsNet++ performance scenario 3. The noise amplitude
Reference trajectory
Longitudinal
25
20
100
15
20
25
Speed
0.4
Time
30000 XO
15
(b) Action I: Longitudinal acceleration
Time
accelaration
2-
10
2
YeN rale
0
/ MwMlsmmn
-1
20 Time
Time
(a) Vehicle trajectory
(c) Action 2: Yaw acceleration
Figure 43: MLP performance in scenario 3. The noise amplitude is 10.
34

---

LipsNet++: Unifying Filter and Controller into Policy Network
Reference trajectory
Longltudinal
ba
RL robot
25
5
Obstancle
20
15
20
Speed
04
Time
(b) Action I: Longitudinal acceleration
20
2
10
Yaw accelaration
Yew rale
mmmmm
5
Time
0
15 Time
20
25
-5
-3
-2
X
(a) Vehicle trajectory
(c) Action 2: Yaw acceleration
Figure 44: LipsNet++ performance in scenario 3. The noise amplitude is 10.
t t= Os
t= 6s
Obstacle
Obstacle
Reference trajectory
RL robot
RL robot
(a) t=0s
(b) t=6s
t 11s
t t= 18s
RL robot
RL robot
Obstacle
(c) t=ls
(d) 1=18s
Figure 45: Snapshots of scenario 3.
35

---

LipsNet++: Unifying Filter and Controller into Policy Network
Refarence trajectory
Longitudinal
M
RL robot
2-
Obstancle
15
40
30
Time
(b) Action I: Longitudinal acceleration
20
accelaration
Speed
rale
10
Time
Time
10
15
20
25 Time
35
40
45
2
(a) Vchicle trajectory
(c) Action Yaw acceleration
Figure 46: MLP performance in scenario The noise amplitude is O.
50
Reference trajectory
Longitudinal
RL robot
0-
2-
Obstancle
440
10
15
20
25 Time
30
35
40
45
30
(b) Action I: Longitudinal acceleration
acoelaration
Spent
mi
Time
Time
10
15
20
25
30
35
40
Time
(a) Vehicle trajectory
(c) Action 2: Yaw acceleration
Figure 47: LipsNet++ performance scenario 4. The noise amplitude
50
trajectory
Longitudinal
Crash!
40
1-
10
15
20
25 Time
30
35
45
50
30
(b) Action 1: Longitudinal acceleration
20
Speed
Yow rate
MM Aq
10
-1
20
20
Time
Time
2
Time
(a) Vehicle trajectory
(c) Action 2: Yaw acceleration
Figure 48: MLP performance in scenario 4. The noise amplitude is 10.
36

---

LipsNet++: Unifying Filter and Controller into Policy Network
50
Reference trajectory
Longitudinal
Mrd
RL robot
0-
2-
Obstancle
40
-1-
10
15
20
25
30
35
40
45
50
Time
(b) Action I: Longitudinal acceleration
Yaw accelaration
Spees
0.4
rafts
2
10 M /m mm
mmmm
10
0-
Time
Time
10
15
20
25
30
35
40
45
50
2
(a) Vehicle trajectory
(c) Action 2: Yaw acceleration
Figure LipsNet++ performance in scenario The noise amplitude is 10.
The results of TAR and AFR for scenario 1-3 are listed in Table 18. The result for scenario 4 is not listed because the obstacle vehicle is manipulated by human, which means each trial has great randomness. The data in Table 18 is visualized in Figure 50. As shown in Figure (e()) when noise increases, LipsNet++ maintains the highest TAR and its TAR declines much slower than MLP's. As shown in Figure (0()) when noise increases, LipsNet++ maintains the lowest AFR and its AFR grows much slower than MLP's. These results imply LipsNet++ has excellent action smoothness and noise robustness.
N. Limitation and Future Work
LipsNet++ achieves smoother and more robust control with slight increase in training time, as illustrated in Appendix H. The increase in training wall time caused by this limitation becomes acceptable due to the significant reduction in action fluctuation rate.
In the future, the backward time of LipsNet++ can optimized to overcome the above limitation. can be accelerated by using multiple forward propagation and zero-order gradient estimation to compute the Jacobian matrix.
Furthermore, attention mechanism for the filter matrix H can be introduced in future works. In this way, H can vary according different observation inputs. Additionally, employing LipsNet++ more real- world tasks, such highway vehicles and embodied AI, is also promising direction.
37

---

LipsNet++: Unifying Filter and Controller into Policy Network
2.5
MLP
235
Action fluctuation ratio (L)
LipsNet++
Total average return
2.0
230
1.5
225
1.0
220
215
0.5
MLP
LipsNet++
210
0.0
5
10
15
20
5
10
15
20
Noise amplitude
Noise amplitude
(a) TAR in scenario
(b) AFR in scenario
255.0
2.5
MLP
Total average return (t)
252.5
LipsNet++
Action fiuctuation ratio
2.0
250.0
247.5
1.5
245.0
1.0
242.5
MLP LipsNet++
0.5
240.0
0.0
0
5
10
15
20
5
10
15
20
Noise amplitude
Noise amplitude
(c) TAR in scenario 2
(d) AFR in scenario
2.5
2.5
MLP LipsNet++
Action fluctuation ratio (L)
Total average return (t)
290
2.0
285
1.5
280
1.0
275
0.5
MLP
270
LipsNet++
0.0
5
10
15
20
5
10
15
20
Noise amplitude
Noise amplitude
(c) TAR in scenario
AFR scenario
Figure 50: Performance trend with increasing noise in mini-vehicle driving environment.
38


Reviews:
[{"review_id": "miYbX42IHq", "reviewer": "Reviewer_Exvu", "summary": "The paper introduces LipsNet++, a novel policy network designed to mitigate action fluctuation in reinforcement learning (RL). The authors identify two primary causes of action fluctuation: observation noise and policy non-smoothness. To address these, LipsNet++ integrates: A Fourier filter layer, which processes historical observations in the frequency domain and applies a trainable filter matrix to suppress noise. And, a Lipschitz controller layer, which applies Jacobian regularization to constrain the Lipschitz constant, ensuring smooth control outputs.\n\n\n## update after rebuttal\nThe authors did address most of my concerns, which is why I increased my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback!\nWe are encouraged by the positive aspects of your assessment, i.e., ***well-supported*** claims, ***mathematically sound*** derivations, ***valuable*** insight, ***comprehensive*** related works, ***novel*** integration of filtering and control, ***clear*** empirical improvements, and ***good*** ablation studies.\n\nWe now address your comments in detail:\n\n---\n#### **> Examine Impact of Control Types**\nAll DMControl environments in our paper (Cartpole, Reacher, Cheetah, Walker) are torque control. To examine the impact of control types as you suggested, we additionally train on **position control (DMControl Fish)** and **velocity control (Gymnasium Bipedal Walker)** environments. The Fish task needs to control 5 actions (tail, tail_twist, fins_flap, finleft_pitch, finright_pitch) under position control type, and the Bipedal Walker needs to control 4 motor speed values under velocity control type. Their TAR and ARF are summarized as follows:\n\nTAR comparison: [[Link - Table R3]](https://github.com/ICML-anonymous-2025/LipsNet_v2/blob/main/rebuttal/Table_R3.png), AFR comparison: [[Link - Table R4]](https://github.com/ICML-anonymous-2025/LipsNet_v2/blob/main/rebuttal/Table_R4.png)\n\nThese results show that LipsNet++ achieves good action smoothness across all three control types. Additionally, some studies on robotics control also support the effectiveness of LipsNet-related structures in torque control [1] and position control [2].\n\nIn fact, we believe LipsNet++ does not assume the control type to be torque, position, or velocity; it ensures action smoothness in all cases and serves as a general solution. The choice of control type should depend on the task requirements.\n\n#### **> Other Comments Or Suggestions**\n- Formula in Line 98: Typo fixed.\n- Thank you for your suggestion. We **have renamed** Section 3.2 to “Understanding Key Factors Behind Action Fluctuation” and **refined** the contribution description accordingly. Additionally, we believe analyzing the root causes of action fluctuation is essential. Equation (1) mathematically identifies two fundamental causes, which play a crucial role in guiding the design of the subsequent network layers (Fourier Filter Layer & Lipschitz Controller Layer). Previous works did not distinguish the two causes and treated them as one; therefore, our analysis actually contributes to the broader community.\n\n#### **> Q1: Scalability to Image-based RL**\n**Yes**, LipsNet++ can scale to image observations. The **core question** here is whether LipsNet++ can handle high-dimensional observation tasks. To demonstrate this, we added two experiments: DMControl Cartpole (pixel observation) and Gymnasium Humanoid (348 observations), with the former has image-based observations and the latter has high-dimensional observations.\n\nGiven the high-dimensional input, we adopt the following LipsNet++ architecture:\n- DMControl Cartpol: *[Convolutional Layer -> Fourier Filter Layer -> Lipschitz Controller Layer]*\n- Gymnasium Humanoid: *[Linear Layer -> Fourier Filter Layer -> Lipschitz Controller Layer]*\n\nIn the Convolutional Layer and Linear Layer, they perform the feature extraction and dimension reduction.\n\nThe TAR and AFR are summarized in [[Link - Table R5]](https://github.com/ICML-anonymous-2025/LipsNet_v2/blob/main/rebuttal/Table_R5.png). These results demonstrate that **LipsNet++ can handle high-dimensional observation tasks and can scale to image-based tasks**.\n\n#### **> Q2: Gradient Flow for Matrix $H$**\n$H$ receives gradients from the policy improvement loss $\\mathcal{L}'$, which includes the critic value as described in Equation (4) and the last equation in Section 2.1; but $H$ does not directly receive gradients from the critic itself, as the critic does not have a Fourier Filter Layer.\n\nIntegrating a Fourier Filter Layer into the critic is an interesting direction. In this case, the actor and critic could potentially share one $H$ matrix, raising an interesting question of whether $H$ should receive gradients only from the actor or from both. Thank you for your insightful perspective. We will explore this in future work.\n\n---\n#### **Summary**\nWe hope our revisions adequately address your concerns. Please let us know if you have any further questions.\n\nThank you again for your time and valuable feedback!\n#### ***References***\n*[1] Y. Zhang, et al. Robust Locomotion Policy with Adaptive Lipschitz Constraint for Legged Robots. IEEE RAL, 2024.*\n\n*[2] G. Christmann, et al. Benchmarking Smoothness and Reducing High-Frequency Oscillations in Continuous Control Policies. IEEE IROS, 2024.*", "claims_and_evidence": "The paper presents several claims, the majority of which are well-supported by experimental results:\n\n- The Fourier filter layer effectively suppresses observation noise.\n- The Lipschitz controller layer enhances policy smoothness, supported by theorem 3.2.\n- The approach is applicable across various continuous control tasks.\n- LipsNet++ is easily integrable into different network architectures and RL frameworks.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria make sense. However, one aspect that is missing is the influence of control type. Depending on the environment, the control inputs could be torque, velocity, or position-based. It would be particularly interesting to investigate how LipsNet++ performs across these different control modalities and whether its effectiveness varies between them.", "theoretical_claims": "I did take a look at theorem 3.2. The derivations are mathematically sound and align with prior work on smooth policy learning.", "experimental_designs_or_analyses": "The experiments are well-structured, incorporating multiple environments and ablations:  \n- Double Integrator: Assesses basic control performance and robustness to noise.  \n- DMControl Suite: Evaluates performance on complex continuous control tasks.  \n- Mini-Vehicle Driving: Demonstrates real-world applicability in a robotics setting.  \n\nIt would have been interesting to examine the impact of control type, as position control inherently applies some degree of smoothing compared to torque control. A useful addition would be to construct two alternative control interfaces (for position and velocity; DMC already uses torque) within DMC and perform an ablation study over them. Given the exploratory nature of this analysis, it may not be necessary to compare all baselines for this ablation.", "relation_to_broader_scientific_literature": "The paper builds on prior work in:\n- Lipschitz-constrained policy learning (Takase et al., 2020; Song et al., 2023).\n- Fourier-based filtering in neural networks (Lee-Thorp et al., 2022; Tan et al., 2024).\n- Smooth control in RL (Mysore et al., 2021; Yu et al., 2021).\n\nThe connection to classical control theory is a valuable insight, suggesting that RL policy networks can explicitly incorporate filtering and control mechanisms.", "essential_references_not_discussed": "The related work section appears comprehensive, and I did not identify any missing references.", "other_strengths_and_weaknesses": "Strengths:\n- Novel integration of filtering and control into a policy network.\n- Clear empirical improvements in smoothness and robustness.\n- Good ablation studies.\n- Public PyTorch implementation for reproducibility.\n\nWeaknesses:\n- Limited discussion on scalability to image-based RL.\n- Some interesting ablations are missing (see above on control type influence)\n- Certain claimed contributions seem overstated relative to their novelty.", "other_comments_or_suggestions": "- second formula: squared term needs to be inside the expectation\n- “3.1. Reasons Identification of Action Fluctuation” not a good section name. I also found that section not really novel and satisfying. Calling it one of the four core contributions of the paper looks like an overstatement to me", "questions_for_authors": "- how well does this scale to image observations?\n- does the trainable H matrix receive gradients from the critic? Or did you stop these gradients?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OQmerIOCnq", "reviewer": "Reviewer_QXnu", "summary": "Action fluctuation is a major issue in reinforcement learning. The fluctuation in action can be caused due to measurement noise or steep changes in the policy due to large Jacobians. This paper addresses the measurement noise issue using Fourier filter and steep Jacobian issue using Jacobian regularization. The approach is well-motivated and I believe the paper makes a strong contribution.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback!\nWe appreciate your recognition of our work as a ***well-motivated*** approach that makes a ***strong contribution*** and as an ***excellent specimen of leveraging theoretical insights to solve a crucial issue in RL***. Your positive assessment greatly encourages us.\n\nWe now address your comments in detail:\n\n---\n#### **> Theoretical Improvement**\nWe appreciate your valuable suggestion! We **have refined** Theorem 3.2 by **explicitly incorporating the approximation error and its asymptotic behavior**. The revised theorem is as follows:\n\n**Theorem 3.2** (Lipschitz’s Jacobian Approximation): *Let $f:\\mathbb{R}^n \\to \\mathbb{R}^m$ be a continuously differentiable neural network. The Jacobian norm $\\left\\Vert\\nabla_xf\\right\\Vert$ serves as an approximation of the local Lipschitz constant of $f$ within the neighborhood $\\mathcal{B}(x, \\rho)$, centered at $x$ with radius $\\rho$. The approximation error is given by $\\mathop{\\max}_{\\ \\delta\\in \\mathcal{B}(0,\\rho)} \\left[ \\left(\\nabla_x\\left\\Vert \\nabla_x f(x) \\right\\Vert\\right)^\\top\\delta + o(\\delta)\\right].$ Moreover, as $\\rho \\to 0$, the Jacobian norm converges to the exact local Lipschitz constant, i.e. $\\left\\Vert\\nabla_xf\\right\\Vert \\to K(x)$.*\n\nThis refinement provides a more precise characterization of the approximation error, as you suggested.\n\n#### **> Question on Lemma A.1**\nWe appreciate your question regarding Lemma A.1. Theorem 5.19 in Rudin's book is an **existential result** that relates the function’s increment to its derivative at some point, **without involving** the Lipschitz constant or providing an explicit expression for Lipschitz constant; whereas Lemma A.1 **explicitly gives an equivalent expression** for the Lipschitz constant. In summary, Theorem 5.19 primarily focuses on the bounding variation of a function, while Lemma A.1 focuses on the local Lipschitz property and provides an approximation method for calculating the Lipschitz constant.\n\n---\n#### **Summary**\nWe hope our revisions adequately address your concerns. Please let us know if you have any further questions.\n\nThank you again for your time and valuable feedback!", "claims_and_evidence": "Yes, the exposition in the paper supports the claim in the introduction.", "methods_and_evaluation_criteria": "Yes from my perspective.", "theoretical_claims": "Yes, the mathematical development is correct, but it can be made more rigorous. In the statement of Theorem 3.2, the authors should also include the approximation error for the Lipschitz constant instead of just using $\\approx$ notation, to make the theorem more precise. The approximation error appears well articulated in the proof in the appendix, so suggest just using that formulation.", "experimental_designs_or_analyses": "The experimental design appears valid to me, but I am more of a theoretically inclined researcher, I'll leave the critique on experiments to more experimentally oriented researchers.", "relation_to_broader_scientific_literature": "The literature survey is sufficient to put the contribution in context.", "essential_references_not_discussed": "None that I know of.", "other_strengths_and_weaknesses": "This paper is an excellent specimen of leveraging theoretical insights, especially using Lipschitz constant estimates to solve a crucial performance issue in RL.", "other_comments_or_suggestions": "None.", "questions_for_authors": "Isnt Lemma A.1 just the mean value theorem for vector valued functions? Any standard math textbook reference would suffice, e.g., see Principles of Mathematical Analysis by Rudin, Theorem 5.19.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2bifIYO8qX", "reviewer": "Reviewer_K8nQ", "summary": "The paper proposes a new policy network, LipsNet++, to mitigate the action fluctuation problem in real-world robotic applications. The proposed network uses a Fourier filter layer to smooth the observation input which is then fed into a MLP network with its local Lipschitz constant regularized via Jacobian regularization. On a set of simulated and real robotic tasks, the proposed policy network exhibits a lower action fluctuation level and higher robustness to observation noises. In particular, on the double integrator task, the authors compare LipsNet++ against a set of prior approaches designed for mitigating the action fluctuations and show that the proposed network excels at all metrics, including control performance and action smoothness when subject to different amount of observation noises.\n\n## update after rebuttal\nAll my concerns have been addressed. With the new experiments, I am now convinced that LipsNet++ indeed outperforms previous approach on the action smoothness while not sacrificing performance. \n\nI have increased my score from 1 to 3.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback!\nWe appreciate your recognition of our ***well-written and convincing*** results and analyses, ***reasonable*** evaluation criteria, ***nice*** ideas, the proposed methods that ***make sense***, and the ***easy-following*** nature of our paper. The positive aspects of your assessment encourages us.\n\nWe now address your comments in detail:\n\n---\n#### **> Improved Theoretical Claim**\nWe appreciate your valuable suggestion! We **have refined** Theorem 3.2 by explicitly **incorporating the approximation error, defining the neighborhood area, and describing its asymptotic behavior** when $\\rho\\to 0$, as you suggested. The revised theorem is as follows:\n\n**Theorem 3.2** (Lipschitz’s Jacobian Approximation): *Let $f:\\mathbb{R}^n \\to \\mathbb{R}^m$ be a continuously differentiable neural network. The Jacobian norm $\\left\\Vert\\nabla_xf\\right\\Vert$ serves as an approximation of the local Lipschitz constant of $f$ within the neighborhood $\\mathcal{B}(x, \\rho)$, centered at $x$ with radius $\\rho$. The approximation error is given by $\\mathop{\\max}_{\\ \\delta\\in \\mathcal{B}(0,\\rho)} \\left[ \\left(\\nabla_x\\left\\Vert \\nabla_x f(x) \\right\\Vert\\right)^\\top\\delta + o(\\delta)\\right].$ Moreover, as $\\rho \\to 0$, the Jacobian norm converges to the exact local Lipschitz constant, i.e. $\\left\\Vert\\nabla_xf\\right\\Vert \\to K(x)$.*\n\n#### **> Additional Comparisons with Baselines on DMControl**\nBeyond the comprehensive baseline comparisons in the double integrator environment, Section 4.2 evaluates LipsNet++ against three baselines (MLP, LipsNet-G, LipsNet-L) across all DMControl environments and Appendix J compares it with one baseline (MLP-SN) on Reacher environment. Extending MLP-SN to all environments is impractical due to excessive hyperparameter tuning (Line 1039).\n\nTo further address your concerns, we **have added experiments with additional baselines on DMControl** environments. Now, LipsNet++ is evaluated against all baselines on DMControl, including MLP, CAPS, L2C2, MLP-SN, LipsNet-G, and LipsNet-L.\n\nTAR comparison: [[Link - Table R1]](https://github.com/ICML-anonymous-2025/LipsNet_v2/blob/main/rebuttal/Table_R1.png), AFR comparison: [[Link - Table R2]](https://github.com/ICML-anonymous-2025/LipsNet_v2/blob/main/rebuttal/Table_R2.png)\n\nAs shown in the above tables, LipsNet++ **maintains SOTA** performance on the **non-toy DMControl** tasks.\n\n#### **> Improved Discussion with Additional References**\n- Jacobian regularization: Prior works [1,2] use Jacobian regularization to enhance adversarial robustness for neural networks, with [2] proposing an efficient Jacobian norm approximation method (already cited in Line 214). While common [1-4], it is rarely explored in RL. Studies [5,6] have identified high Lipschitz constant of policy network as a cause of action fluctuation. Our Theorem 3.2 shows the Jacobian norm is an approximation of Lipschitz constant, introducing Jacobian regularization as a practical way to control Lipschitz constant in RL.\n\n- Observation filtering: Methods [7-10] enhance RL robustness in algorithm level, e.g., state-adversarial DRL with SA-MDP modeling for inaccurate observations [7] and robust training framework for safe RL with rigorous theoretical analysis and empirical results [8]. In contrast, our approach improves action smoothness and robustness in network level by designing policy network (Fourier Filter Layer), embedding prior knowledge directly into its structure.\n\nWe have added a detailed discussion in introduction section and don't elaborate here due to space constraints.\n\n---\n#### **Summary**\nWe hope our revisions adequately address your concerns. Please let us know if you have any further questions.\n\nThank you for your time and valuable feedback!\n#### ***References***\n*[1] D. Jakubovitz, et al. Improving dnn robustness to adversarial attacks using jacobian regularization. ECCV 2018.*\n\n*[2] J. Hoffman, et al. Robust learning with jacobian regularization. arXiv 2019.*\n\n*[3] W. Yuxiang, et al. Orthogonal jacobian regularization for unsupervised disentanglement in image generation. ICCV 2021.*\n\n*[4] K. Co, et al. Jacobian regularization for mitigating universal adversarial perturbations. ICANN 2021.*\n\n*[5] R. Takase, et al. Stability-certified Reinforcement Learning via Spectral Normalization, arXiv 2020.*\n\n*[6] X. Song, et al. LipsNet: A Smooth and Robust Neural Network with Adaptive Lipschitz Constant for High Accuracy Optimal Control, ICML 2023.*\n\n*[7] H. Zhang, et al. Robust deep reinforcement learning against adversarial perturbations on state observations. NeurIPS 2020.*\n\n*[8] Z. Liu, et al. On the robustness of safe reinforcement learning under observational perturbations. ICLR 2023.*\n\n*[9] M. Xu, et al. Trustworthy reinforcement learning against intrinsic vulnerabilities: Robustness, safety, and generalizability. arXiv 2022.*\n\n*[10] H. Zhang, et al. Robust reinforcement learning on state observations with learned optimal adversary. ICLR 2023.*", "claims_and_evidence": "\"Simulated and real-world experiments show that LipsNet++ has excellent action smoothness and noise robustness, achieving a new SOTA performance.\" -- I am not convinced by this claim because the only environment where the authors compare LipsNet++ with the prior baseline approaches (CAPS, L2C2, MLP-SN) is the toy double integrator environment. It is unfair to say that LipsNet++ achieves SOTA when it is only really shown to be the best on a toy environment.", "methods_and_evaluation_criteria": "The proposed method makes sense and the evaluation criteria are also reasonable.", "theoretical_claims": "The statement for Theorem 3.2 is very vague -- What does \"$||\\nabla_x f|| \\approx K(x)$\" mean? I would suggest to use a more precise statement (e.g., define what infinitesimal neighborhood means and state that as $\\rho \\rightarrow 0,  ||\\nabla_x f|| \\rightarrow K(x)$).", "experimental_designs_or_analyses": "The experimental results and analyses are well written and convincing. My only concern is that the authors did not compare the proposed method against important baselines on environments other than the toy double integrator environment. This raises the important question that whether the proposed method is as effective as claimed to be (SOTA).", "relation_to_broader_scientific_literature": "The paper is closely related to LipsNet with the same goal: mitigating action fluctuation in robotic applications. The key contributions of this paper are\n- Observation filtering module with a learnable filter matrix that can help smooth out noises in the observations.\n- Jacobian regularization to encourage the local Lipschitz constant to be small for the policy network", "essential_references_not_discussed": "- One of the key component of the proposed policy network is the jacobian regularization of the network. However, this exact technique has been used in many prior works (especially in adversarial robustness literature, e.g., [1, 2].) but the paper fails to attribute this technique properly \n- The other key component of the proposed policy network is the observation filtering with the goal of being robust to observation noises. This is also widely studied in the literature (e.g., [3, 4]). The paper also fails to discuss how the proposed technique relates to prior techniques.\n\nWhile the ideas in the paper are nice, it is hard to contextualize this paper in the literature without proper discussions with the broader literature.\n\n[1] Hoffman, Judy, Daniel A. Roberts, and Sho Yaida. \"Robust learning with jacobian regularization.\" arXiv preprint arXiv:1908.02729 (2019).\n[2] Jakubovitz, Daniel, and Raja Giryes. \"Improving dnn robustness to adversarial attacks using jacobian regularization.\" Proceedings of the European conference on computer vision (ECCV). 2018.\n[3] Zhang, Huan, et al. \"Robust deep reinforcement learning against adversarial perturbations on state observations.\" Advances in Neural Information Processing Systems 33 (2020): 21024-21037.\n[4] Liu, Zuxin, et al. \"On the robustness of safe reinforcement learning under observational perturbations.\" arXiv preprint arXiv:2205.14691 (2022).", "other_strengths_and_weaknesses": "Strengths:\n- The paper is easy to follow and the proposed method makes sense for addressing the action fluctuation problem.\n\nWeaknesses:\n- The empirical comparison of the prior methods is lacking. While the authors evaluate the proposed method on many environments, baseline results are only available on a toy double integrator environment. This alone is not sufficient to show the effectiveness of the proposed method.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z94POkmemD", "reviewer": "Reviewer_qhMJ", "summary": "Deep reinforcement learning suffers from the action fluctuation problem. This problem has been studied in many different forms in the past. Initial works constrained the Lipschitz constant of various aspects of the policy and various value functions to achieve desired smoothness. This paper proposes two methods for this problem. First, they propose smoothing observation noise by frequency domain filters with the filtering matrix learnt via gradient descent. Second, they propose to penalize the norm of the Lipschitz constant of the neural networks. Experimental results are shown on standard reinforcement learning benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback!\nWe are encouraged by the positive aspects of your assessment, i.e., ***method effectiveness, sound experimental design, and strong benchmarks and figures***. Your recognition of the frequency-domain filtering, Section 3.2, and Fig. 12 is especially appreciated.\n\nWe now address your comments in detail:\n\n---\n#### **> Improved Theoretical Claims**\nIn Line 626, **the equality $K=\\max_{x'}\\left\\Vert\\nabla f(x')\\right\\Vert$ follows from Equation (8) in Lemma A.1**, rather than a direct application of the mean value theorem. So, the inequality $\\leq$ does not apply here. To ensure a more rigorous presentation, we have explicitly included the error term into the theorem in the main text and discussed the limiting case $\\rho\\to0$. Please refer to our response to reviewer QXnu for the revised theorem.\n\n#### **> Improved Discussion with Additional References**\nThank you for pointing out some theoretical filters, such as [1-3]. These works exhibit strong theoretical foundations. In contrast, our approach is more experimental—**we don't make any assumptions about system linearity, convexity, or noise distribution.** Instead, we introduce a learnable filtering matrix that **adaptively learns from data which frequencies are important and which are noise**. We have added a detailed discussion in introduction section and don't elaborate here due to space constraints.\n\n#### **> Weakness Concerns**\nIn fact, LipsNet++ differs significantly from LipsNet in structure. **First**, LipsNet only addresses excessive Lipschitz constant, overlooking the dual causes of action fluctuation. In contrast, our approach introduces the Fourier Filter Layer and Lipschitz Controller Layer, which not only control the Lipschitz constant but also directly address observation noise. **Second**, while LipsNet has a rigorous Lipschitz constraint, it relies on a strong assumption—only for MLP with piece-wise linear activations (ReLU). Our Jacobian regularization, however, generalizes to any network architectures. **Third**, LipsNet’s Lipschitz constraint method is too slow for real-time inference, whereas LipsNet++ achieves a 4× faster inference speed, as shown in Appendix H. In summary, the two techniques in our work are both highly novel and impactful, with a structure entirely distinct from LipsNet.\n\n#### **> Comments & Suggestions**\nThe lag when operating at the edge of capability is an interesting topic. Notably, LipsNet++ prevents catastrophic failures when the only option is to use a big acceleration and stop to a halt. The DMControl-Cheetah environment exemplifies this scenario, requiring actions to switch rapidly between -1 and 1 for high-speed movement. The [[Link - Figure R1]](https://github.com/ICML-anonymous-2025/LipsNet_v2/blob/main/rebuttal/Figure_R1.png) visualizes the first two action dimensions (rapid switching occurs across all dimensions). As shown in the figures, LipsNet++ preserves the necessary rapid reaction behavior to maintain good control performance (see TAR in Fig. 9) while reducing action fluctuation as much as possible (see AFR in Fig. 9). LipsNet++ **can adapt action smoothness dynamically** by learning local Lipschitz constants that vary with different obs input.\n\n#### **> Fixed Typos & Unclear Words**\nLine 160: Typo fixed (*even* $\\to$ *even if*)\n\nLine 262: Typo fixed (*dose* $\\to$ *does*)\n\nLine 206: We have clarified this statement by explicitly noting that LipsNet’s Jacobian matrix calculation during inference introduces significant computational overhead. The inference time and their comparisons are already detailed in Appendix H.\n\nLine 557: Fixed by removing *'real'*\n\nAppendix typos: Fixed all typos of *'acceleration'* and *'obstacle'*\n\n#### **> Questions**\nQ1: We have not tested with adversarial noise, which is a promising direction for future work.\n\nQ2 & Q3: Training is conducted without noise. Our method doesn't require the presence or absence of noise during training. During testing, different levels of noise are added to evaluate performance and action fluctuation under varying noise intensities. Apart from this, the training and test setups remain identical. The ability to achieve filtering even with noise-free training is attributed to Equation (4): the term $\\lambda_h\\left\\Vert H\\right\\Vert_F$ drives the filter coefficients of frequencies unrelated to control performance to 0, as shown in Fig. 11, 22, and 28. This mechanism is one of our core ideas.\n\n---\nThank you again for your time and valuable feedback! Please let us know if you have any further questions.\n\n#### ***References***\n*[1] H. Elad, K. Singh, and C. Zhang. Learning linear dynamical systems via spectral filtering. NeurIPS 2017.*\n\n*[2] H. Elad, H. Lee, K. Singh, C. Zhang, and Y. Zhang. Spectral filtering for general linear dynamical systems. NeurIPS 2018.*\n\n*[3] A. Sanjeev, E. Hazan, H. Lee, K. Singh, C. Zhang, and Y. Zhang. Towards provable control for unknown linear dynamical systems. ICLR Workshop 2018.*", "claims_and_evidence": "The paper proposes two methods to smooth the actions of RL policies. They are a little ad-hoc but there is some experimental evidence to suggest that the methods are effective. For the first improvement on constraining the Lipschitz constant of the neural network, the authors claim certain training time improvements from the simplification. Given that the proposed regularization is network-wise and not layer-wise, it is less susceptible to training difficulty. Further, this simplification is claimed to affect performance to a lesser extent. The other method of using a learned frequency-domain filter is new and is claimed to give additional improvements. An additional claim is that they are less sensitive to hyper-parameters than the benchmarks. These improvements are illustrated on an extensive set of experimental benchmarks.", "methods_and_evaluation_criteria": "The methods are justifiable and the evaluation criteria of the action fluctuation ratio is standard for these benchmarks. However, given the control applications of this paper, the impact of filtering and normalizing the Lipschitz constant on the stability and safety of the controllers is worrying. This suggests incorporating evaluation criteria relevant to these aspects to gain deeper understanding of the problem.", "theoretical_claims": "The theoretical section in the appendix is not very new and only serves as an addendum to the reader. In theorem A.2, it should be the case that $K(x)\\leq max_{x^{\\prime}} \\Vert \\nabla f(x) \\Vert$. The approximation in the theorem statement is concerning given that there is theoretical research on the lipschitz constant of a neural net.", "experimental_designs_or_analyses": "The experimental design is shown on the double integrator, deepmind control suite and mini-vehicle driving. The mini-vehicle driving is illustrated on physical vehicles. The experimental design is sound.", "relation_to_broader_scientific_literature": "Action smoothness is a very important problem to be tackled for control and robotics applications. The frequency domain filtering method is interesting and deserves mention. The other method is similar to previously known literature and is not a standout approach.", "essential_references_not_discussed": "RL theory proposes some filtering methods such as below. Some theoretical filters could be discussed.\n\nHazan, Elad, Karan Singh, and Cyril Zhang. \"Learning linear dynamical systems via spectral filtering.\" Advances in Neural Information Processing Systems 30 (2017).", "other_strengths_and_weaknesses": "Strengths\n\nThe effectiveness of the method despite its simplicity is the strength of the paper. The experimental benchmarks and figures are strong. The results are not compelling on all benchmarks. Further, experiments with adversarial noise could be interesting to evaluate further.  The result in Figure 12 is strong.\n\nWeaknesses \n\nThe structure and presentation is very similar to Lipsnet. The Lipschitz constant optimization of Lipsnet has more rigor.\n\nOnly one of the two proposed methods fit the criteria of being high on both novelty and impact. These weaknesses are reflected in my score for this paper even though the experimental results have strength.", "other_comments_or_suggestions": "For control problems, observation filters could create a lag which can affect the system stability and safety. It would be interesting to look at these filters from a perspective of safety when the system is operating at the edge of its capability. Will smoothing the actions cause catastrophic failures when the only option is to use a big acceleration and stop to a halt? An adaptive smoothing scheme that remains cognizant of invariant constraints is much important for this line of work to expand in scope.\n\nSection 3.2 is very interesting to me. \n\nLine 160: The mapping function has significant output differences even if the inputs are closely adjacent\n\nLine 162: typo with “does” not\n\nLine 206: \"However, LipsNet is not applicable in high-real-time tasks due to the Jacobian matrix calculation during forward inference,\" -unclear statement\n\nPage 11, Line 557: It is unclear what is the “real Lipschitz constant”.\n\nTheorem A.2. There are some mistakes in line 626 which should be an upper bound.\n\nThe appendix has multiple typos for “acceleration” and “obstacle”.", "questions_for_authors": "In terms of the claims of robustness, do we know if the proposed method is better or worse on adversarial noise distributions?\n\nIs the noise distribution same in training and test?\n\nWhat aspects of the training was changed during the test?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel policy network, LipsNet++, that aims to address the action fluctuation problem in RL by integrating a Fourier filter layer (to address observation noise) and a Lipschitz controller layer (to ensure policy smoothness). In experiments, the paper demonstrates convincing improvements and/or on-par results for total average return, while yielding significant improvements in action fluctuation ratio, on DMControl benchmark tasks compared to the previous SOTA. Physical experiments on mini-vehicles are also implemented.

While several reviewers initially raised concerns about the rigor of the theoretical analysis, the generalizability of experimental settings, and the number of baselines compared against, and the potential effect of noise, the authors were largely able to address these concerns via updated analysis and additional experiments during the rebuttal period. Overall, the reviewers largely agreed that this is a valuable contribution.
Decision: Accept (spotlight poster)