Enhancing Robustness in Deep Reinforcement Learning: A Lyapunov Exponent Approach
Authors: Rory Young
Nicolas Pugeault
Venue: NeurIPS 2024 poster (2024)
Keywords: Reinforcement Learning
Robust Reinforcement Learning
Stable Reinforcement Learning
Lyapunov Exponents
Chaos Theory
Primary area: reinforcement_learning
TL;DR: We show reinforcement learning controllers produce chaotic states and reward trajectories in continuous control environments. To mitigate this we propose a regularisation method which minimises chaos and improves the robustness of Dreamer V3.

Abstract: Deep reinforcement learning agents achieve state-of-the-art performance in a wide range of simulated control tasks. However, successful applications to real-world problems remain limited. One reason for this dichotomy is because the learnt policies are not robust to observation noise or adversarial attacks. In this paper, we investigate the robustness of deep RL policies to a single small state perturbation in deterministic continuous control tasks. We demonstrate that RL policies can be deterministically chaotic, as small perturbations to the system state have a large impact on subsequent state and reward trajectories. This unstable non-linear behaviour has two consequences: first, inaccuracies in sensor readings, or adversarial attacks, can cause significant performance degradation; second, even policies that show robust performance in terms of rewards may have unpredictable behaviour in practice. These two facets of chaos in RL policies drastically restrict the application of deep RL to real-world problems. To address this issue, we propose an improvement on the successful Dreamer V3 architecture, implementing Maximal Lyapunov Exponent regularisation. This new approach reduces the chaotic state dynamics, rendering the learnt policies more resilient to sensor noise or adversarial attacks and thereby improving the suitability of deep reinforcement learning for real-world applications.

Enhancing Robustness in Deep Reinforcement Learning: A Lyapunov Exponent Approach
Rory Young"
Nicolas Pugeault
School of Computing Science
University of Glasgow
Abstract
Deep reinforcement learning agents achieve state-of-the-art performance in wide range of simulated control tasks. However, successful applications to real- world problems remain limited. One reason for this dichotomy is because the learnt policies are not robust to observation noise or adversarial attacks. In this paper, we investigate the robustness of deep RL policies perturbation in deterministic continuous control tasks. We demonstrate that RL policies can be deterministically chaotic, as small perturbations the system state have a large impact on subsequent state and reward trajectories. This unstable non-linear behaviour has two consequences: first, inaccuracies sensor readings, or adversarial attacks, can cause significant performance degradation; second, even policies that show robust performance terms of rewards may have unpredictable behaviour in practice. These two facets of chaos in RL policies drastically restrict the application of deep RL to real-world problems. To address this issue, we propose improvement the successful Dreamer V3 architecture, implementing Maximal Lyapunov Exponent regularisation. This new approach reduces the chaotic state dynamics, rendering the learnt policies more resilient to sensor noise or adversarial attacks and thereby improving the suitability of deep reinforcement learning for real-world applications.
single small state
Introduction
Deep neural networks (DNNs) have revolutionised reinforcement learning (RL)
25 enabling 23 24]. However, trained
agents to excel diverse set of simulated control tasks [12 16 22
deep RL policies not robust controllers DNNs vulnerable to adversarial attacks 77] 26)
Adding small amount of noise to each observation can cause these policies to make poor decisions, considerably degrading their overall performance [10 [3]. This lack of stability poses significant threat when applying deep RL to real-world environments, where inaccurate sensors can easily argue that even high-performing deep RL policies are
introduce noise (5). In this work, robust controllers as they can create a chaotic closed-loop control system [] [4]. These systems are characterised by high sensitivity to initial conditions, with small changes in initial system states producing vastly different long-term outcomes.
In this paper, we the spectrum of Lyapunov Exponents 15 of the policies learnt by state-of-the-ard deep RI. approaches subject to small state perturbations. We show that these controllers can produce chaotic state and reward dynamics when controlling continuous environments. Consequently, a single noisy observation has a dramatic long-term impact on these control systems, with the subsequent approximate state and reward trajectories diverging significantly (Figure[] []. This instability poses two problems for the safe deployment of deep RL in real-world environments.
empirically measure the stability
"Corresponding email: R. Young www.eearcc.gllaa...oo.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

1.00
0.99
Reward
Torso Angle
0.98
0.0252
0.0257
0.97
0.0262
100
200
300
400
500
Steps
Figure Reward attained when a trained deterministic Soft Actor-Critic agent controls the deterministic Walker Walk environment. Each system has the same imitial configuration other than the torso angle, which perturbed by significantly diverge after 50 steps due to the chaotic nature of the control interaction. Consequently, this affects overall performance as there is significant variation in the total reward attained.
10 degrees. This small perturbation causes the systems to
1. The chaotic state dynamics create fractal return surface 28 which is highly sensitive small changes in the system state. The high-frequency oscillations this function cause lack of robustness small state perturbations produce significantly different total rewards.
2. Even for high- performing policies with stable returns, it remains impossible to accurately predict the long-tern behaviour of these chaotic control systems they rely noisy partially observable sensors to attain an observation. This unpredictability means that safe and reliable behaviour cannot be guaranteed.
To address these issues, propose Maximal Lyapunov Exponent regularisation for Dreamer V3 [9] This novel technique estimates the local state divergence using the Recurrent State Space Model and incorporates this term into the policy loss. We demonstrate that this regularisation term significantly reduces the chaotic dynamics produced by this state-of- of-tt--rrt deep RL controller. This increased stability dramatically improves the robustness of the policy, thus improving the feasibility of deep RI. agents for real-world continuous control tasks.
Background
2.1 Robust reinforcement learning
Reinforcement learning provides data-driven method for solving sequential decision-making problems. This control interaction represented by deterministic Markov Decision Process (MDP) with state space S C R", action space A C R scalar reward function r S A - R. state transition function f SS x A -> and initial state distribution Po S S. The objective of an RI. agent is to learn policy S - A which maximises the sum of discounted returns (Equation I for a given discount factor 7 E [0, 1).
((0) =
Σ
r x r( or) St+1 = f(8,, (1),,,))::)))) az = Te()s)
Sum-pa
RL. policies are said to robust if they can maintain consistent behaviour and reliable performance in the face of noise adversarial attacks. This stability is crucial for the safe deployment of deep RL

---

Table Stability of a dynamical system for different values of the Maximal Lyapunov Exponent (A1)
and Sum of Lyapunov Exponents (Ax:).
A
X
Stability
Stable
+
Chaotic
+
Unstable
to real-world applications where observation noise is inevitable. Recently, a wide range of attack methods have been developed which can easily compromise the performance of deep RI. policies with relatively low levels of intervention. In their seminal work, Huang et al. [00] extended the Fast Gradient Sign Method 7 attack to deep RL policies, demonstrating that the addition a small amount of strategic noise to every observation is sufficient to degrade the performance of trained deep RL agents. Furthermore, Kos & Song [11] showed that this attack does not need to occur every step, less frequent attacks with small Gaussian noise still produce drop in performance. Additionally, they demonstrated that only perturbing the observation when the value function surpasses set threshold still produces poor performance while significantly decreasing the frequency of the attack. A similar result was established by Lin et al. [13] who showed that specifically attacking when the relative performance gain between best and worst action surpasses defined threshold can successfully degrade performance. These findings demonstrate that consistent and inconsistent small perturbations to the system can easily confuse deep RL policies, resulting in unintended and detrimental behaviour. Note that small observation noise is frequent in real-world systems, and therefore, this lack of robustness severely limits the application of deep RL real-world environments.
2.2 Measuring stability: Lyapunov Exponents
Lyapunov Exponents 15 20 provide method for quantifying the stability of complex, non-linear,
high-dimensional systems by measuring the deformation rate of a small hyperellipsoid under the effects of a transition function. In general, for dynamical system with N degrees of freedom, there are N Lyapunov Exponents, each representing the exponential growth rate of a unique principal axis of the hyperellipsoid. Given a set of N ordered exponents (A1 2 A 2 2 Ax), the volume of the hyperellipsoid grows proportionally
cAAr-\2+...Ax>) and the length grows proportionally From this definition, the Maximal Lyapunov Exponent (MLF) (Equation?) (2) and the Sum of Lyapunov Exponents (SLE) (Equation 3) are used to determine if system stable, chaotic or unstable, as outlined in T
edxit
1st - $c
\1 = lim
lim
In
(2)
6-000
8g
s0
sol
A
A
T=
Dynamical systems with negative MLE (A1 I 0) are stable as all principal axes of the hyperellipsoid exponentially decrease to zero (2I] In these systems, any trajectories produced from similar initial positions converge to the same trajectory given sufficient time. Conversely, systems with positive MLE (A) 0) and SLE (As; > 0) are unstable as resulting state trajectories diverge at an exponential rate. However, for a positive MLE (A > 0) and negative SL.E (Ax: < 0), similar trajectories will diverge at an exponential rate but remain confined to a subregion of the phase space known chaotic attractor [4] [4]. This bounded exponential divergence means trajectories in this region of the state space are unstable and only replicable given the exact same starting state: small perturbations to the starting state will produce significantly different long-term outcomes which appear random and uncorrelated. As a result, it is impossible to predict the long- long-term behaviour of a chaotic system given approxination of the initial state.
To measure the stability of a known dynamical system, the full spectrum of Lyapunov Exponents can be estimated using the approach outlined by Benettin et al. 2 3 This method represents the spectrum as set small perturbation vectors which are iteratively updated using the known transition function. To avoid all vectors collapsing in the direction of maximal growth, they are periodically
3

---

PM
CB
C5
WS
WW
WR
CR
1000
Total Reward
800
600
400
200
0
SAC
TD3
DR3
None
SAC
TD3
DR3
None
SAC TD3
DR3
None
SAC
TD3
DR3
None
SAC
TD3
DR3
None
SAC
TD3
DR3
None
SAC
TD3
DR3
None
Figure 2: Total episode reward for the Pointmass Easy (PM). Cartpole Balance (CB), Cartpole Swingup (CS), Walker Stand (WS), Walker Walk (WW), Walker Run (WR) and Cheetah Run (CR) environments when controlled by trained instances of SAC, TD3, Dreamer V3 (DR3) and an agent which takes no actions (None). Each poliey-environment combination is independently trained with three random seeds and the average interquartile episode reward with bootstrapped 95% confidence interval is reported over 80 evaluation episodes each with a fixed length of 1000 steps.
Shan-Schiidt orthonormalised so that each vector maintains a unique direction. Performing this orthonormalisation allows for the detection of both positive and negative Lyapunov exponents up the dimension of the phase space. The spectrum of Lyapunov Exponents is then determined as the average log rate of divergence of the perturbation vectors they are Gram-Schmidt orthonormalised. Estimation of the full spectrum of Lyapunov Exponents using this method allows for the estimation of and Ay which can quantify the stability of the dynamical system.
2.3
Chaos in reinforcement learning
Previous studies have investigated the chaotic state and reward dynamics produced by reinforcement learning policies. Rahn et [19] showed that the policy optimisation landscape can contain high- frequency discontinuities in the vicinity of a trained policy. A similar result was established by Wang et al. (8), who proved that control systems with Lipschitz continuous reward and transition functions only have a Lipschitz continuous objective function if Y < - In()). When \ > In(()) the objective function is fractal and a-Hölder continuous with holder exponent a = -ln(7)/Ar Consequently. single update to the policy in these chaotic control systems produce substantially different total rewards. Furthermore, Parmas et al. [77 demonstrated that chaos exists in model-based RL methods due to repeated nonlinear predictions and this instability causes gradients to explode during training.
While these works use chaos theory to highlight an important issue in the field of RL policy learning, they are focused primarily on the stability of the policy subject to policy parameter perturbations during training. Our work uses similar concepts but instead focuses on the stability of fully trained RI. policies subject to state perturbations and the adverse effect this has on the total reward attained in realistic environments. To our knowledge, we the first to use the spectrum of Lyapunov Exponents to estimate the level of chaos produced by trained DNN policies in continuous control tasks.
Chaotic state dynamics
In this section, we use Lyapunov Exponents identify the level of chaos produced by various state-of-the-an deep reinforcement learning policies in continuous control environments. We claim that the presence chaos in the MDP implies that the policies are not robust controllers, as trivial changes to the system state produce significantly different long-term state trajectories. This instability poses a significant problem for real-world control systems where consistent and predictable behaviour is necessary.

---

PM
CB
CS
WS
WW
WR
CR
MLE
0.2
0.0
SLE
-10
SAC
TD3
DR3
None
SAC
TD3
DR3
None
SAC
TD3
DR3
None
SAC
TD3
DR3
None
SAC
TD3
DR3
None
SAC
TD3
DR3
None
SAC
TD3
DR3
None
Figure 3: Estimated Maximal Lyapunov Exponent (MLE) and Sum of Lyapunov Exponents (SLE)
for the Pointmass (PM), Cartpole Balance (CB), Cartpole Swingup (CS), Walker Stand (WS),
Walker Walk (WW), Walker Run (WR) and Cheetah Run (CR) environments when controlled by trained instance of SAC, TD3, Dreamer V3 (DR3) and agent which takes no actions (None). Each policy-environmen
combination independently trained with three random seeds and the interquartile average MLE & SLE for each seed is calculated using 20 initial states. A bootstrapped 95% confidence interval included show the variation in MLE and SLE across random seeds.
To closely match real-world applications, estimate the stability of tasks from the DeepMind Control Suite [27 when controlled by deep RL policies. These simulated environments provide a range of deterministic control problems with continuous state spaces, as outlined in Appendix A.] For each control task we independently train three instances of Soft Actor-Criti (SAC) (8). Twin Delayed Deep Deterministic Policy Gradients (TD3) 6 and Dreamer V3 ( as these represent state-of-the of-tth--ttttt off off-polie pellicl and model-based methods for continuous control tasks. Furthermore, determine trained deep RL policies directly influence the stability of each control system, we also introduce passive controller that takes no actions. All models are based
the Stable Baselines [8 implementation with the parameters outlined in Appendix A.. (..2 and trained using an Intel Core i7-8700 CPU workstation with an Nvidia RTX 2080 Ti GPU and 32GB of RAM. The average interquartile reward and bootstrapped 95% confidence interval [ for each policy type are reported in Figure 2 These results show that all three algorithms learn policies which performed well and significantly better than the no-action baseline. However, this does not speak to the types of dynamics produced or how robust these policies are to local state perturbations.
To identify the stability of each policy-enviromment interaction, Lyapunov Exponents using the method proposed by Benettin et al. 3.. The spectrum is calculated using states sampled from the initial state distribution for each environment. perturbation vector initialised for each state dimension at distance of 10 from the sample state. This represents an arbitrarily small change the control system without introducing numerical precision errors. The spectrum is calculated over 1000 environment steps and perturbation vectors are orthonormalised every 10 steps prevent divergence saturation. This process repeated with 20 initial states and the average value for each exponent is used to calculate the MLE and SLE. Ablation studies for these constants are provided Appendix B
estimate the full spectrum of
Figure provides the bootstrapped 95% confidence interval for the MLE and SLE produced by each policy-environmen pair. These results indicate that all the environments are naturally invariant to small perturbations as the no action baseline has ê = 0. Conversely, when these environments are controlled by deep RL policies, ê1 can non-zero, indicating that DNN controllers directly influence the stability of these control systems.
5

---

Pole Angle Cart Position
Torso Angle Torso Height
0.00
-0.05
-0.2
-0.10
0.00
0.000
-0.025
200
400
20
60
100
Timestep
Timestep
(a) Cartpole Balance
(b) Walker Walk
Figure 4: Partial state trajectory produced by Dreamer V3 when controlling Cartpole Balance and Walker Walk subject to single initial state perturbation. Initially, each system separated by only 10-1 units but the subsequent state trajectories diverge significantly as the control interaction is chaotic.
We find that SAC and TD3 produce stable state dynamics in simple dow-dimesiional environments (Pointmass, Cartpole Balance and Cartpole Swingup) they have negative MLE. Therefore, if small perturbation is made to any of the states in these environments, the subsequent state trajectories converge. This result is consistent with the definition of the reward function for each of these tasks as they provide high rewards for maintaining the system at fixed location. However, when controlled by Dreamer V3, these simple systems exhibit low levels of chaos as they have positive MLE and negative SLE. As result, state trajectories in these environments are highly sensitive to initial conditions, with similar states producing significantly different long- long-term term outcomes as shown in Figure 4a Instead of converging to single state, these trajectories exhibit chaotic behaviour, continuously orbiting within a region which yields high rewards.
Furthermore, all deep RL methods produce chaotic dynamics in the complex high-dimensional environments (Walker Stand, Walker Walk, Walker Run and Cheetah Run), indicated by the positive MLE and negative SL... This means that these policies cannot account for arbitrarily small changes in the system's 's state since small changes produce exponentially diverging state trajectories, illustrated in Figure 4b) This poses a significant problem for real-world applications of RL, as observation perturbations are easily introduced via imperfect measurements sensor noise. Therefore, for these complex high-dimensional environments controlled by deep RL policies, it is impossible to guarantee stability as the system cannot correct itself after single inaccurate observation.
4 Chaotic rewards
In this section, we show that chaotic state trajectories can also impact a policy's performance and produce chaotic reward trajectories as determined by the Maximal Lyapunov Exponent. This instability creates fractal return surface in which small state perturbations produce significantly different total rewards. We argue that adversarial attack methods could leverage these high-frequency oscillations, repeatedly injecting perturbations which cause the agent to follow the state trajectories that attain the lowest total reward. This lack of robustness poses significant problem for real- world control systems, where the worst-case performance often more significant than the average performance.
By considering the reward over state trajectory trajectory in a one-dimensional reward space, the stability of the reward can also be measured using Lyapumov Exponents. Given that this space one-dimensional, only one Lyapunov Exponent exists; however, this single exponent can still be used to reliably identify the stability of reward trajectories. Negative X values indicate that small perturbations to the system's state still produce converging long-term rewards. Moreover, for a bounded reward function, the reward trajectories cannot diverge indefinitely; thus positive Ån indicates that the long-term reward chaotic as small changes to the system state produce exponentially diverging bounded reward trajectories.

---

PM
CB
CS
WS
WW
WR
CR
Reward MLE
0.25
0.00
-0.25
SAC
TD3
DR3
SAC
TD3
DR3
SAC
TD3
DR3
SAC
TD3
DR3
SAC
TD3
DR3
SAC
TD3
DR3
SAC
TD3
DR3
Figure 5: Reward MLE interquartile mean for the Pointmass (PM), Cartpole Balance (CB), Cartpole
Swingup (CS), Walker Stand (WS), Walker Walk (WW), Walker Run (WR) and Cheetah Rum (CR) when controlled by SAC, TD3 and Dreamer independently trained with three random seeds and the reward MLE for each seed is calculated using 20 initial states. bootstrapped 95% confidence interval is included to show the variation reward stability across random seeds.
(DR3). Each policy-envirommen combination
1.0
400
0.8
Total Reward
300
Reward
0.6
200
0.4
100
0.2
0.0
0.0000
0.0005
0.0010
0
100
200
300
400
500
Perturbation Size
Steps
Figure Left: Total reward attained by deterministic SAC policy when controlling the deterministic Walker Walk environment subject to initial perturbation with fixed direction and varying magnitude. Right: Rewards attained by three best and three worst state trajectories subject to this perturbation.
Figure 5 provides the average reward MLE and bootstrapped 95% confidence interval for each policy-environment pair defined in Section 3 These graphs indicate that the reward trajectories produced in simple low-dimensional environments (Pointmass, Caripole Balance, Cartpole Swingup) are stable as they have negative reward MLE. As a result, single small state perturbation does not harm the total reward attained the subsequent reward trajectories converge. Conversely, in high- high-dimensional control systems (Walker Stand, Walker Walk, Walker Run and Cheetah Run), the reward MIEE is positive, indicating that the reward is highly sensitive arbitrarily small changes in the system state. Consequently, reward trajectories in these systems are unstable, as similar initial states produce significantly different sequences of rewards. Plotting the reward trajectories for of these chaotic interactions (Figure shows that a single perturbation with varying magnitude can have huge detrimental impact on the total reward attained despite performing well on average. This poses a significant problem for real-world applications of RL as is impossible to guarantee worst-case performance in chaotic control system.
5 Maximal Lyapunov Exponent regularisation
In Sections 3 & we established that deep RL policies can produce chaotic state trajectories in continuous control tasks and that this can have large detrimental impact on performance. To address this, we propose a novel regularisation method which improves the stability of RL policies by constraining the Maximal Lyapunov Exponent during policy updates. This improved stability is crucial step towards the safe and reliable deployment of RL policies in real-world domains where local perturbations are common.
7

---

Algorithm MLE regularisation
Require:
Policy ( 2
Encoder (9s : S x H. - - P(Z))
Decoder (po H
+ P(S)
Dynamics Predictor (go H. P(Z))
Sequence Model (fe H x A - H)
Current State (s S) Current Hidden State (h E H) Time Horizon (T E N)
Ensure: _^ (0)
J (0) <
# Initialise the MLE regularisation loss
== [2] ~ (0 h))
# Update stochastic representation
for t 1,., do
A +- {ar To (H, Zi)
# Generate set of sample action
Н + [h, = =f(HH Zı, H=
# Update hidden representation
Z + { (z(Hi))
# Update stochastic representation
S {st (HH, Zi)
# Generate set of predicted states
_^1 + (²) + Var(S) + Var(H)
# Update the MLE regularisation loss
end for return (0)
We base our regularisation on Dreamer V3 9 a general-purpose model-based RL algorithm which attains state-of-the-an art performance across diverse set of control tasks. To achieve this, Dreamer V3 uses a Recurrent State Space Model (RSSM) consisting of an Encoder (gs : S x H + P(2)),
Decoder (p& H 2 + P(S)), Dynamies Predictor (gp H P(Z) and Sequence Model (fe xx 2 A H) to predict state trajectories (st), bootstrapped A-return trajec-
25 and state value trajectories (17()))) over a short time horizon T. The policy is then trained maximise the normalised advantage estimates using REINFORCE gradients [2) and an entropy regulariser (H(-l) (30) with weighting coefficient v). The full loss function used to train Dreamer V3's policy is outlined in Equation 4
tories (R
T Σ
R vo(st) log (a(a)))) + n/H (a)(ar))))
C (D))
-
sg
(4)
max S1
1=1
T
L2 (0) = Σ
Var(S.) Var(HHe)
(5)
L L
Palicy (0) = (0) + (^)
(6)
At its core, Dreamer V3 uses a stochastic RSSM to predict the state and reward trajectories over predefined time horizon given an initial starting state 8n and internal representation hn- Due to the stochastic nature of this model, repeating the same trajectory predictions T, E N times produces set of state trajectories (S, = (St,1, .t...... st,L)) St,,,,
and internal representation trajectories (H2 = (hr,,, hu.,.....h..... ))), each of which provides a plausible estimate of the future states. The variance between trajectories (Varz (-)) thus provides an estimation of the local state divergence as the state perturbation size approaches O. Therefore, to minimise X and improve the stability of Dreamer V3 subject to state perturbation, we propose incorporating the regularisation term outlined in Equation 5 into the policy loss (Equation(o) Including this regularisation term as an additional weighted term forces agents to consider the stability the system during the optimisation process. This incentivises the policy to produce stable state trajectories which attain high rewards instead of solely optimising the expected return. The complete algorithm for calculating the MLE regularisation term is provided in Algorithm using the notation outlined by Hafner et al. 99.
8

---

Table Average total reward and average MLE produced when Dreamer V3 (DR3) and Dreamer V3 with MLE regularisation (MLE DR3) when controlling various environments sampled from the Deep Control Suite 227 Each policy-enviromment combination independently trained with three random seeds using the hyperparameters outlined in Appendix A.2
Reward
MLE
Environment
DR3
MLE DR3
DR3
MLE DR3
Pointmass
869.5
880.5
0.0326
-0.0275
Cartpole Balance
978.6
970.5
0.0249
0.0231
Cartpole Swingup
781.4
866.4
0.0149
0.0235
Walker Stand
973.0
961.6
0.1688
0.0654
Walker Walk
948.6
950.7
0.1614
0.1405
Walker Run
646.3
698.4
0.1345
0.1106
Cheetah Run
737.7
675.2
0.0337
0.0283
WS
1000
800
Texal Raward
600
SAC TD3 DR3
MLE DR3
Figure 7: Total episode reward for the Cartpole Balance (CB), Cartpole Swingup (CS), Walker Stand (WS), Walker Walk (WW), Walker Run (WR) and Cheetah Run (CR) environments when controlled by trained instances of SAC, TD3, Dreamer V3 (DR3) and Dreamer V3 with MLE regularisation (MLE DR3) subject to (000, σr) Gaussian observation noise. Each policy-environment combination independently trained with three random seeds and the average episode reward with bootstrapped 95% confidence interval is reported over 80 evaluation episodes each with fixed length of 1000 steps.
Experiments
In this section, we investigate the impact that the proposed MLE regularisation has on Dreamer V3. We show that the inclusion this term reduces the chaotic state dynamics produced by the control policy and that this improved stability increases performance when noise is introduced. For these experiments, we train three instances of Dreamer V3 with MILE regularisation and reuse the SAC, TD3 and Dreamer V3 policies from Sections 3 & When estimating state divergence, the RSSM predicts
plausible future trajectories over which the state and internal representation variance measured. Increasing L will produce more accurate estimate of state divergence; however, this will require more computational resources. Therefore, to maintain a similar training time to that of Dreamer V3, we set L 3. All other hyperparameters are consistent with the Dreamer V3 baseline.
Table@provides the average reward and estimated MLE produced by the regularised and unregularised Dreamer V3 models for each control task. This indicates that MLE regularisation successfully minimises the chaotic state dynamics produced by Dreamer V3 while maintaining similar performance. However, MLE is still positive in the majority of environments, the control interaction still produced chaotic state trajectories. Despite this, the regularised policies are more stable the of divergence has significantly decreased.

---

Torso Angle Torso Height
0.0
Torso Angle Torso Height
-0.5
1.0
0
-2
-2
100
300
400
500
0
200
300
400
500
Timestep
Timestep
Dreamer V3
(b) Dreamer V3 MLE regularisation
Figure 8: State trajectories produced when Dreamer V3 and Dreamer V3 + MLE regularisation control the Walker Stand environment with (0,,, 0.5) Gaussian observation noise.
To identify how robust these regularised Dreamer V3 policies are continual state perturbations, we measure their performance when Gaussian noise is added to each observation. Each policy is trained without observation noise (a 0) in the fully deterministic variant of each environment and then tested with Gaussian noise (1 0 and E [0, I]. Performance is measured over 240 episodes with a fixed length of 1000 steps and maximum reward per step of Figure 7 shows the interquartile mean and bootstrapped 95% confidence interval for the total reward attained by interaction subject to various levels of Gaussian noise. This shows that MLE regularisation significantly improves the performance of Dreamer V3 in four of the noisy control systems, while the other two environments (Walker Stand and Cheetah Run) attain similar performance to the Dreamer V3 baseline. Furthermore, examining the state trajectories produced by Dreamer V3 and Dreamer V3 + MLE regularisation when controlling the Walker Stand task (Figure shows that the regularisation improves the stability of the control interaction the trajectories do not diverge significantly. These findings indicate that MLE regularisation can improve the robustness of Dreamer V3 subject to observation noise produces consistent and reliable behaviour in the face of uncertainties.
each policy-environnen
7 Conclusion
key issue preventing the application of deep reinforcement learning to real-world environments is the need for guaranteed stability and performance the face of noisy observations and adversarial attacks. In this work, we set out to identify the impact a single perturbation has on the long-term behaviour of deep RL policies in continuous control environments. Using the spectrum of Lyapunov Exponents, we established that the MDP can produce chaotic state and reward trajectories which are highly sensitive initial conditions. This instability poses two threats to the application deep RL to real-world problems, where it is infeasible to attain an accurate measurement of the system state. First, small state perturbation can have large impact on the performance of trained deep RL policies, even where the average performance is good. This can create hazards in real-world conditions where observation noise prevalent and can be exploited by adversarial attack methods. Second, even when deep RI policies perform well, they can produce unpredictable behaviours, which is undesirable in most real-world applications.
To mitigate these chaotic dynamics and improve robustness we propose Maximal Lyapunov Exponent regularisation for Dreamer V3. This novel approach uses the Recurrent State Space Model to estimate the local state divergence and incorporates this into the policy loss. In effect, the agent optimises its confidence in future trajectories jointly with expectations of rewards. While MLE regularisation helps improve the robustness of the agent's policies, this approach assumes an accurate estimation of the local state divergence. environments where the RSSM struggles to capture state dynamics, effectiveness of the proposed regularisation may be diminished. However, in our experiments, we demonstrate that this regularisation improves the stability of the learnt policies, thereby making them more robust to state perturbations.
10

---

References
[1] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C. Courville, and Mare Belle- mare. Deep reinforcement learning at the edge of the statistical precipice. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 29304 29320. Curran Associates, Inc., 2021.
[2] Giancarlo Benettin, Luigi Galgani, Antonio Giorgilli, and Jean-Marie Strelcyn. Lyapunov characteristic exponents for smooth dynamical systems and for hamiltonian systems; method for computing all of them. part I: Theory. Meccanica, 15:9-20, 1980.
[3] Giancarlo Benettin, Luigi Galgani, Antonio Giorgilli, and MannMarie Strelcyn. Lyapunov characteristic exponents for smooth dynamical systems and for hamiltonian systems; method for computing all of them. part 2: Numerical application. Meccanica, 15:21-30, March 1980.
[4] Robert Devaney. First Course in Chaotic Dynamical Systems: Theory and Experiment. CRC Press, second edition, 2020.
[5] Gabriel Dulac-Arnold, Nir Levine, Daniel Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. Challenges of real-world reinforcement learning: definitions, bench- marks and analysis. Muchine Learning, 110:2419-2468, 2021.
[6] Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Jennifer Dy and Andreas Krause, editors. Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1587-1596. PMLR, 2018.
[7] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adver- sarial examples. 3rd International Conference Learning Representations, 2015.
[8] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1861-1870. PMLR, 2018.
[9] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301 100444 2023.
[10] Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks on neural network policies. In Sth International Conference on Learning Representations, April 2017.
[11] Jernej Kos and Dawn Song. Delving into adversarial attacks on deep policies. In Sth Interna- tional Conference on Learning Representations, 2017.
[12] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learn- ing. In Yoshua Bengio and Yann LeCun, editors, 4th International Conference Learning Representations, San Juan, Puerto Rico, Conference Track Proceedings, 2016.
[13] Yen-Chen Lin, Zhang- Wei Hong. Yuan Soan-Hong Liao, Meng-Li Shih, Ming Yu Liu, and Min Sun. Tactics of adversarial attack on deep reinforcement learning agents. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, pages 3756-3762, 2017.
[14] Edward N. Lorenz. Deterministic nonperiodic flow. Journal of Atmospheric Sciences, 20:130- 141, 1963.
[15] Aleksandr Lyapunov. The General Problem of the Stability of Motion. Control Theory and Applications Series. Taylor & Francis, 1992.

---

[16] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller. Andreas K. Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 5 1188::993333 2015.
[17] Paavo Parmas, Carl Edward Rasmussen, Jan Peters, and Kenji Doya. PIPPS: Flexible model- based policy search robust to the eurse of chaos. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceed- ings of Machine Learning Research, pages 4065-4074 PMLR, 2018.
[18] Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernsstus, and Noah Dormann. caable-asselnes's Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22:1 12:1-8,77 2021.
[19] Nate Rahn, Pierluca D'Oro, Harley Wiltzer, Pierre-Luc Bacon, and Mare Bellemare. Policy optimization in a noisy neighborhood: On return landscapes in continuous control. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 06688306440. Curran Associates, Inc., 2023.
[20] David Ruelle. Sensitive dependence on initial condition and turbulent behavior of dynamical systems. Annals of the New York Academy of Sciences, 316:408 416, 1979.
[21] Hua Shao, Yuming Shi, and Hao Zhu. Lyapunov exponents, sensitivity, and stability for non- autonomous discrete systems. International Journal of Bifurcation and Chaos, 28:1850088, 2018.
[22] David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe. John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529:484-489, 2016.
[23] David Silver, Thomas Hubert, Julian Schrittwieser, loannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362:1 140-1144, 2018
[24] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of go without human knowledge. Nature, 550:354-359, 2017.
[25] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018
[26) Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In 2nd International Conference Learning Representations, 2014.
[27] Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap. Nicolas Heess, and Yuval Tassa. tim_control, Software and tasks for continuous control. Software Impacts, 6: 10002,, 2020.
[28] Tao Wang. Sylvia Herbert, and Sicun Gao. Fractal landscapes in policy optimization. In A. Oh, Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36. pages 4277-4294. Curran Associates, Inc., 2023.
[29] Ronald J. Williams. Simple statistical gradient- algorithms for connectionist reinforce- ment learning. Machine Learning, 8:229-256, 1992.
[30] Ronald Williams and Jing Peng. Function optimization using connectionist reinforcement learning algorithms. Connection Science, 3:241-266 1991.
12

---

A Appendix
A.1 State space composition
Table 3: State space definition for each environment used in this paper. Each dimension of the state space represents a unique aspect of the control system and contain any real value. Positions measured in metres (m), angles are measured in radians (rad), velocities are measured in metres per second (m/3 s) and angular velocities are measured in radians per second (rad/s).
Render
Name
Degrees of freedom
Axis Representations
Point position Point mass x y velocity
Pointmass
Cart position
Cart velocity
Cartpole Balance
Pole angle
Cartpole Swingup
Pole angular velocity
Torso x & z position
Torso x & velosity
Torso angle
Torso angular velocity
Walker Stand
Left & right hip angle
18
Walker Walk
Left & right hip angular velocity
Walker Run
Left & right knee angle
& right
angular velocity
right ankle angle
Left & right ankle angular velocity
Torso x z position
Torso x & z velocity
Torso angle
Torso angular velocity
Front back hip angle
Cheetah Run
18
Front & back hip angular velocity
Front & back knee angle
Front & back knee angular velocity
Front & back ankle angle
Front hack ankle angular velocity
13

---

A.2 Hyperparameters
Table 4: Hyperparameters used train SAC, TD3 and Dreamer V3
SAC
TD3
Parameter
Value
Parameter
Value
Environment steps
5 x
10
Environment steps
5 x 10
Buffer size
106
Buffer size
106
Parallel environments
8
Parallel environments
Update period
8
Update period
Updates per step
8
Updates per step
Discount factor
0.99
Discount factor
0.99
Learning rate
0.0003
Learning
0.0003
Batch size
1024
Batch size
1024
Polyak update coefficient
0.005
Polyak update coefficient
0.005
Networks activation
Tanh
Networks activation
Tanh
Networks depth
3
Networks depth
Networks width
256
Networks width
256
Dreamer V3
Parameter
Value
Environment steps
106
Buffer size
105
Parallel environments
8
Update period
8
Updates per step
Discount factor
0.99
Learning rate
10
Batch size
15
RSSM batch length
64
Imagination horizon
15
Network activation
LayerNorm + SiLU
Networks depth
2
Networks width
512
Recurrent state size
4096
Number of latents
32
Classes per latent
32
B Lyapunov Exponent ablation study
B.1 Default values
Table 5: Parameters used to calculate the spectrum of Lyapunov Exponents using the method outlined
by Benettin et al. 2 B
Parameter
Value
Total timesteps
1000
Number of iterations
100
Normalisation period
10
Number samples
Perturbation size
0.0001
14

---

B.2
Number of iterations ablation study
Cartpole Balance
Cartpole Swingup
Walker Stand
SAC
1.0
TD3
MLE
DR3
0.5
0.0
Walker Walk
Walker Run
Cheetah Run
1.0
0.5
0.0
0
50
100
50
100
0
50
100
Number of Iterations
Figure 9: Estimated Maximal Lyapunov Exponent of environments sampled from the DeepMind Control Suite when controlled by SAC, TD3 and Dreamer V3 (DR3). Each combination is independently trained with three random seeds and evaluated using the parameters in 5 except the number of iterations, which varies from to 100. The mean and 95% confidence interval indicate that MLE converges after 100 iterations.
B.3
Number of samples ablation study
Cartpole Balance
Cartpole Swingup
Walker Stand
0.4
SAC
TD3
MLE
DR3
0.2
0.0
Walker Walk
Walker Run
Cheetah Run
0.4
MLE
0.2
0.0
0
100
50
100
50
100
Number of Samples
Figure 10: Estimated Maximal Lyapumov Exponent of environments sampled from the DeepMind Control Suite when controlled by SAC, TD3 and Dreamer V3 (DR3). Each policy-environment combination is independently trained with three random seeds and evaluated using the parameters in Table 5) except the number of initial samples, which varies from 1 to 20. The mean and 95% confidence interval indicate that MLE converges with 20 initial samples.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction claim the Markov Decision Process can produce chaotic state trajectories and that this greatly impacts the stability of the reward. These They also claim that reduce
issues addressed in Sections 3 and 4 the chaotic state dynamics produced by Dreamer V3 thereby improving its robustness to adversarial attacks and this is studied in Sections (aacionnSaann@
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
This paper continually highlights important limitation of deep RL methods, the instability of deep RI. controllers. We also propose a method for minimising the chaotic state dynamics and the limitations of this method are discussed in Section 6 and
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specificattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
16

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: This paper discloses the hyperparameters used to train each agent in Ap- pendix A.2 the method for estimating MLE in Section 3 Appendix B and the method for minimising chaos in Secions
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility.
17

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: This paper provides open access to the code, with instructions faithfully reproduce the main experimental results outlined in Sections 3 3 and Appendix A.2 2
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/ mublic/guidese otte///eummmimissiooooooiiooeiie
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips. http/uullcggguucee https//ooeSSumiissseeee ssionPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All hyperparameters used for training and evaluation are outlined in Ap- pendix A.2 and Appendix B respectively.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All necessary figures report the mean and bootstrapped 95% confidence interval
using the method outlined by Agarwal al [I]. Full details this outlined in the text. Figures 1 6 and 8 do not include envor bars as they explicitly show the variability between state or reward trajectories. Reducing this normal distribution removes any nuances produced by the control interaction.
18

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the eπror bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Compute resources are outlined in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics stctps:/nneuriss.com http:///wawppp..cppoooc/ccccaaaa
Answer: [Yes]
Justification:
Our work does not involve human subjects participants.
Our work uses environments [27] and RL agents [8] which are publicly available.
do not publish any new datasets.
We ensure reproducibility by including hyperparameters and code.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
19

---

10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This paper discusses the important societal issue of safe and reliable control using RL methods.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks as datasets and models are not released.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
20

---

Justification: This paper uses several existing assets and this is outlined in Section 3 The control environments are sampled from the DeepMind Control Suite [27] and the RL models are implemented using Stable Baselines 3 [8] The original papers for each model are also cited in the text.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
patasets.paaerestt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
21

---

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "8psSroJq0i", "reviewer": "Reviewer_TDUf", "summary": "To address the issue of stability of deep reinforcement learning, the authors first gauge the chaotic behavior of various state-of-the-art deep reinforcement learning policies in continuous control environments, and quantify the stability of those policies with significant impact of their applicability to real world problems. Then, the authors proposed an improvement based on implementing a Maximal Lyapunov Exponent regularization in the RL architecture, and demonstrate the improvement with examples.", "questions": "Could more explanation of all the terms and parameters in Equation (4) be provided? Also, how is the proposed regularizing term in (5) is related to (4)?", "limitations": "The authors adequately addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Lyapunov exponent is an important concept developed in dynamical system. There are many related sophisticated research conducted. It is beneficial to both machine learning and dynamical system to have used it in the analysis of deep RL. \n\nSec 3 & 4 present a reasonable analysis on the stability study of deep RL.", "weaknesses": "The proposed method on maximal Lyapunov exponent regularization needs to be further explored and studied, both in terms of the theory involved and amount of experiments conducted. The idea is novel, but not fully explored and explained. The current set of numerical experiments are limited and not truly convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your time and effort in reviewing our work. Your questions are very insightful, and we would like to offer our thoughts on it.\n\n### Weaknesses\n- **The proposed method on maximal Lyapunov exponent regularization needs to be further explored and studied, both in terms of the theory involved and amount of experiments conducted.** \\\nWe appreciate your suggestion for further exploration and study into the impact of MLE regularisation. We would like to highlight that we have outlined the theory used for the regularisation method in Section 5 and have provided extensive experimental results using multiple seeds in Section 6. These results use the same environments as the original Dreamer V3 paper and demonstrate that the inclusion of MLE regularisation improves the stability of the control interaction and often the total reward attained (Table 2). We acknowledge that extending these experiments to more complex control systems is desirable. However, we believe that the results outlined in our work sufficiently demonstrate the benefits of MLE regularisation.\n\n### Questions\n\n- **Could more explanation of all the terms and parameters in Equation (4) be provided?** \\\nPlease, see the comment outlined in the Global Response, in which we address the definition of Equation 4.\n\n- **How is the proposed regularizing term in (5) related to (4)?** \\\nThe loss function used to train the MLE regularised Dreamer V3 policy is $\\mathcal{L}^\\text{Policy}(\\theta) + \\mathcal{L}^{\\lambda_1}(\\theta)$. This will be clarified in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hT3tIVzubJ", "reviewer": "Reviewer_pxFr", "summary": "Deep RL methods are usually lack of robustness in control tasks whose dynamics are chaotic, thereby having positive maximal Lyapunov exponents (MLEs). This paper proposes an approach that improves the stability of trained deep RL controller through MLE regularization.", "questions": "* Does the policy network architecture affect the performance of your approach?\n\n* Is it possible to find two trajectories with equal rewards, where one is stable and the other is chaotic? In other words, the reward itself may not be sufficient to reflect the stability of a trajectory.\n\n* How does the approach perform in the case that it is unable to estimate the MLE accurately in practice?", "limitations": "It has discussed its empirical limitations in Section 6.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The problem studied in this paper is well-motivated and important. \n\n* Characterizing the robustness using maximal Lyapunov exponents is promising.\n\n* The presentation is straightforward.", "weaknesses": "* This paper dedicates many pages to the chaotic phenomena in RL (Sections 3 and 4), which have been addressed in previous works as introduced in Sections 1 and 2. The core contribution, which I believe is the regularization technique, needs more elaboration and analysis.\n\n* There is no concrete algorithm provided that the audience can follow to easily implement the approach proposed in this paper. Imagine a person with little or no knowledge of dynamical systems. Can they manage to implement the algorithm after reading the paper?\n\n* (minor) Some information is missing. For example, what are $S$, $H$ and $v_\\phi(s_t)$ in equation (4)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and thoughtful review. We value your insightful questions, and you can find our response below.\n\n### Weaknesses\n- **This paper dedicates many pages to the chaotic phenomena in RL (Sections 3 and 4), which have been addressed in previous works as introduced in Sections 1 and 2.**\\\nWhile the idea of chaos in RL has been addressed in prior works, they focus on the ***sensitivity of the value function subject to policy parameter perturbations during training***. In these works, it is demonstrated that the return surface can have a fractal structure, and this can produce poor policy updates during training. \\\nIn contrast, the analysis in Sections 3 and 4 focuses on the ***sensitivity of a fixed control system subject to state perturbations after training***. The results in these sections demonstrate that trained state-of-the-art model-based and model-free methods produce a chaotic control interaction in which a small change in the system state can have a profound impact on the long-term state trajectories and the reward attained. As such, these deep-RL methods cannot provide the stability guarantees necessary for real-world control systems. \n\n- **The core contribution, which I believe is the regularization technique, needs more elaboration and analysis.**\\\nWe appreciate your suggestion for further exploration and study into the impact of MLE regularisation. We would like to highlight that a series of experiments have been conducted using multiple seeds which demonstrate that the inclusion of MLE regularisation significantly improves the stability of the control interaction and often increases the total reward attained (Table 2). Furthermore, this increased stability improves the performance of Dreamer V3 when noise is added to the observation space (Figure 7). Examining the state trajectories produced by Dreamer V3 and Dreamer V3 + MLE regularisation when controlling the *Walker Stand* task (Figure 8) shows the regularisation improves the consistency of the control interaction as the trajectories do not diverge significantly. These results provide strong evidence that MLE regularisation leads to significant improvements in the stability of the state trajectories produced by Dreamer V3. \n\n- **There is no concrete algorithm provided that the audience can follow to easily implement the approach proposed in this paper.**\\\nWe appreciate your comment regarding the clarity of the proposed regularisation method. We would like to highlight that in Section 5, we outline the additional loss term used and that this can be calculated by estimating the state and hidden-state trajectories. However, we acknowledge this is unclear, so we will include a formal algorithm in the camera-ready version.\n\n- **What are $S$, $H$ and $v_\\phi(s_t)$ in equation (4)?**\\\nPlease, see the comment outlined in the Global Response, in which we address the definition of Equation 4.\n\n### Questions\n- **Does the policy network architecture affect the performance of your approach?**\\\nFrom our preliminary experiments, we find that the network architecture does not impact stability however, this is an active area of research which we are currently investigating. For our current work, we maintained the same network architecture for the baseline Dreamer V3 and MLE regularised Dreamer V3 to allow for a fair comparison. \n\n- **Is it possible to find two trajectories with equal rewards, where one is stable and the other is chaotic?**\\\nYes, it is possible for two trajectories to attain equal rewards despite having differing levels of stability. Consider the Cartpole Balance task as an example. In this control system, the agent is provided +1 reward per step for maintaining the pole within 1° of the vertical upright position and within 0.25 meters of the centre of the track. As such, a trajectory that maintains a stable vertical position will attain the same reward as a chaotic trajectory that remains within the high-reward region.\n\n- **How does the approach perform in the case that it is unable to estimate the MLE accurately in practice?**\\\nAs the estimation of MLE used for the regularisation term uses a learned model of the system dynamics, it is possible for the estimation to be inaccurate. When this incorrect estimation of MLE is included in the policy loss, the updated policy can produce more unstable dynamics. However, avoiding this inaccurate estimation requires an improvement in dynamics prediction models, which is outside the scope of this paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6jj8WnlIWd", "reviewer": "Reviewer_2W37", "summary": "This work studies adversarial RL and policy stability from the perspective of the Lyapunov spectrum. A regularization term is introduced to encourage more robust policies.", "questions": "- Can you please provide legends in the figures? If you have the data, can you include a third panel in Fig. 6 corresponding to the regularized policy? (otherwise this figure is ambiguous without comparison)\n- Maybe I missed it, but what is $S$ in Eq. 4?\n- Can you  measure the divergence in trajectories such as in Fig. 4 & 8 and compare to the calculated LE?\n- By how much do you weight the regularization in Eq. 5?\n- Can you discuss the use of hidden states $h$? It isn't clear to me how this affects the loss beyond what the first term contributes\n\nThese questions may open new areas of research, and I don't intend for them to be answered within this paper, but I'm curious to hear your thoughts on the following:\n- How does this work relate to the idea of imposing Lipschitz constraints / regularization on DNNs? I see it as an orthogonal direction, but can they solve the same problem? If so, can they be compared?\n- Can we use pre-trained policies, then finetune them wrt the LE regularization to make them more robust?\n- In 2.3, can you discuss the case of $\\gamma \\to 1$?\n\nminor typographical:\n- Eq. 1, I believe you need a conditional on the state $s$ and an expectation over initial states.\n- L161: \"...low levels of chaos **as** they have...\"\n- Eq. 5 use parentheses over both t-dep terms\n- Please increase fontsizes in Fig 7", "limitations": "Can you please include limitations in Sec 7?", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- A novel idea being introduced from classical literature and implemented in the deep RL setting\n- The regularization introduced is effective at smoothing out trajectories generated by learned policies\n- This can have important real-world effects in safety-critical / sensitive applications\n- Opens work for future (theory/computational/applications) studies to build off", "weaknesses": "The weaknesses I've found are mostly minor, but I still hope you can address:\n\n- The correlation between Figure 2 and 3 is not immediately clear to me; can you please discuss this further?\n- On that note, the improvement in terms of reward is not clear (I know this is not necessarily \"the point\" but given the discussion of reward trajectories, it would be nice to see further iumprovemetn here\n- How error-prone is the calculation of LEs discussed in Sec 2.2?\n- It's not very clear to me the use of \"no actions\" in the figures. Can you explain how this \"benchmark\" should be interpreted? How about in environments where there is no such \"do nothing\" action?\n- Can you include non-scalar comparisons of algorithms (cf. https://github.com/google-research/rliable)\n- The appendix is a bit sparse. Can you provide any further experiment details?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for taking the time to review our work. We are delighted that you are interested in our paper! The question you raised is very insightful, and we would like to share some opinions about it.\n### Weaknesses\n- **The correlation between Figure 2 and 3 is not immediately clear to me**\\\nThese figures show three different metrics attained by various deep RL agents when controlling environments sampled from the Deep Mind control suite. Figure 2 provides the average reward attained, while Figure 3 provides the estimated MLE and SLE for each policy-environment pair. We include these figures to demonstrate that control systems which appear to be performing well can still produce chaotic and unstable state dynamics.\n- **The improvement in terms of reward is not clear**\\\nImproving the performance of Dreamer V3 is not the objective of this work, as we instead focus on improving the stability of the state trajectories. We expect that including the regularisation term introduces a tradeoff between stability and performance. Despite this, we find that regularised policies perform better than the state-of-the-art approach in four out of the seven test environments. Furthermore, when noise observation noise is introduced, we find our approach matches or outperforms all baselines in all the environments.\n- **How error-prone is the calculation of LEs discussed in Sec 2.2?**\\\nWhen calculating the LEs using the method outlined by Benettin et al., we need to define the initial perturbation size, time horizon, normalisation frequency, and the number of initial states. We ran ablation studies for all these values, which can be included in the Appendix of the camera-ready version. From these additional experiments, we find that the calculation of the LEs is consistent for all parameter values except initial perturbation size, which can cause differing LE values when too large. Therefore, the calculation of the LEs discussed in Section 2.2 is not error-prone, providing a suitably small initial perturbation size.\n- **It's not very clear to me the use of \"no actions\" in the figures.**\\\nThe \"no action\" benchmark is used to show that chaos is introduced by the DNN policy and not the environment. At each time step, this baseline applies the fixed action $\\textbf{0}^m$; thus, no additional torques are applied to the available joints. This baseline is used to evaluate the stability of each environment without any control intervention. Since each environment has MLE = 0 and SLE $\\leq$ 0 when controlled by the \"no action\" baseline, we find that these systems are naturally invariant to small perturbations. In contrast, when controlled by deep RL policies, these environments have nonzero MLE. Therefore, we can conclude that the level of chaos found in the control interaction is produced by the DNN policies.\n- **Can you include non-scalar comparisons of algorithms?**\\\nThis will be included in the camera-ready version.\n- **The appendix is a bit sparse. Can you provide any further experiment details?**\\\nFurther experiment details, including Lyapunov Exponent ablation studies, will be included in the appendix of the camera-ready version.\n### Questions\n- **Can you please provide legends in the figures? If you have the data, can you include a third panel in Fig. 6 corresponding to the regularized policy?**\\\nThis will be included in the camera-ready version.\n- **Maybe I missed it, but what is $S$ in Eq. 4?**\\\n$S$ is a normalising term used to approximately scale the returns to the range [0,1]. For further details of Equation 4, please refer to the comment in the Global Response.\n- **Can you measure the divergence in trajectories such as in Fig. 4 & 8 and compare to the calculated LE?**\\\nThese values will be included in the camera-ready version.\n- **By how much do you weight the regularization in Eq. 5?**\\\nThe regularisation term used in Equation 5 is weighted equal to the policy loss term. We are currently investigating the impact varying this weight has on the balance between performance and stability.\n- **Can you discuss the use of hidden states $h$?**\\\nWhen improving the stability of Dreamer V3, it is important to constrain the internal hidden states, as the policy network uses this and an embedding of the current observation to produce an action. Therefore, simply constraining the prediction observations can still create chaotic dynamics, as differing internal representations can produce different long-term outcomes.\n- **How does this work relate to the idea of imposing Lipschitz constraints/regularization on DNNs?**\\\nImproving the smoothness of DNNs by imposing Lipschitz constraints is similar to our problem, as they both consider the smoothness of a function with respect to its inputs. However, when determining the level of chaos, the function we consider is the repeated composition of the dynamical system transition function. Due to this repeated composition, a continuously differentiable function, and thereby Lipschitz continuous, can produce chaotic dynamics (e.g. The Lorenz Attractor). Therefore, imposing Lipschitz constraints/regularisation on the DNN policy does not guarantee stable long-term dynamics.\n- **Can we use pre-trained policies, then finetune them wrt the LE regularization to make them more robust?**\\\nThis is an active area of research which we are currently investigating.\n- **In 2.3, can you discuss the case of $\\gamma \\rightarrow 1$?**\\\nAs $\\gamma \\rightarrow 1$, the objective function becomes Lipschitz continuous for control systems with $\\lambda_1 < 0$ and non-differentiable for systems with $ \\lambda_1 > 0 $. For further details, please refer to the original paper by Wang et al.\n- **Typographical Errors**\\\nThese errors will be fixed for the camera-ready version.\n- **Limitations**\\\nThe limitations of our work have been discussed in Section 6; however, for clarity, we will include a limitations section in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to express our gratitude to all the reviewers for taking the time to review our paper and providing valuable feedback. Your comments have been insightful and have certainly contributed to the refinement of our work. We appreciate the effort you have put into the review process. After carefully considering your comments, we would like to address the general concerns and criticisms raised during the review.

- **Could more explanation of all the terms and parameters in Equation 4 be provided?** \
Equation 4 outlines the loss function used by Hafner et al 2024 to train Dreamer V3’s policy network. All the terms used are consistent with the original article, including Stop Gradient (sg), Return Estimates ($R^\lambda_t$), Critic Estimates ($v_\phi(s_t)$), Scaling Factor ($S$), Entropy Scale ($\eta$) and Entropy Regulairsor (H$[\cdot]$). This equation is included in Section 5 to provide background information for the reader as the MLE regularisation term (Equation 5) is added to this equation. This will be clarified in the camera-ready version.

Meta-review: This paper studied the robustness of deep RL policies to a single small state perturbation in deterministic continuous control tasks. It first showed the vulnerability of deep RL policies, as being deterministically chaotic against small perturbations, which further indicates two facets of chaotic behaviors of the policies in real-world applications. To address the issue, the paper proposed a novel regularisation technique based on Maximal Lyapunov Exponent. The new algorithm reduces the chaotic behaviors and made the learned policies more resilient, with extensive experimental results. It reaches a consensus that the observations in this paper are new and of great importance to deep RL, and the proposed workaround is novel. Recommend acceptance. I suggest the authors to address the remaining comments from the reviewers in preparing the final version of the paper.
Decision: Accept (poster)