Enhancing Robustness in Deep Reinforcement Learning: A Lyapunov Exponent Approach

Authors: rory  young
nicolas  pugeault
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 19:06:03

Abstract: Deep reinforcement learning agents achieve state-of-the-art performance in a wide range of simulated control tasks. However, successful applications to real-world problems remain limited. One reason for this dichotomy is because the learnt policies are not robust to observation noise or adversarial attacks. In this paper, we investigate the robustness of deep RL policies to a single small state perturbation in deterministic continuous control tasks. We demonstrate that RL policies can be deterministically chaotic, as small perturbations to the system state have a large impact on subsequent state and reward trajectories. This unstable non-linear behaviour has two consequences: first, inaccuracies in sensor readings, or adversarial attacks, can cause significant performance degradation; second, even policies that show robust performance in terms of rewards may have unpredictable behaviour in practice. These two facets of chaos in RL policies drastically restrict the application of deep RL to real-world problems. To address this issue, we propose an improvement on the successful Dreamer V3 architecture, implementing Maximal Lyapunov Exponent regularisation. This new approach reduces the chaotic state dynamics, rendering the learnt policies more resilient to sensor noise or adversarial attacks and thereby improving the suitability of deep reinforcement learning for real-world applications.

Main paper: 
 introduction: 
 Deep neural networks (DNNs) have revolutionised reinforcement learning (RL) [25] , enabling agents to excel in a diverse set of simulated control tasks [12, 16, 22, 23, 24] . However, trained deep RL policies are not robust controllers as DNNs are vulnerable to adversarial attacks [7, 26] . Adding a small amount of noise to each observation can cause these policies to make poor decisions, considerably degrading their overall performance [10, 11, 13] . This lack of stability poses a significant threat when applying deep RL to real-world environments, where inaccurate sensors can easily introduce noise [5] . In this work, we argue that even high-performing deep RL policies are not robust controllers as they can create a chaotic closed-loop control system [4, 14] . These systems are characterised by a high sensitivity to initial conditions, with small changes in initial system states producing vastly different long-term outcomes.In this paper, we use the spectrum of Lyapunov Exponents [15] to empirically measure the stability of the policies learnt by state-of-the-art deep RL approaches subject to small state perturbations. We show that these controllers can produce chaotic state and reward dynamics when controlling continuous environments. Consequently, a single noisy observation has a dramatic long-term impact on these control systems, with the subsequent approximate state and reward trajectories diverging significantly (Figure 1 ). This instability poses two problems for the safe deployment of deep RL in real-world environments.Figure 1 : Reward attained when a trained deterministic Soft Actor-Critic [8] agent controls the deterministic Walker Walk environment. Each system has the same initial configuration other than the torso angle, which is perturbed by ±5 × 10 -4 degrees. This small perturbation causes the systems to significantly diverge after 50 steps due to the chaotic nature of the control interaction. Consequently, this affects overall performance as there is significant variation in the total reward attained.1. The chaotic state dynamics create a fractal return surface [28] which is highly sensitive to small changes in the system state. The high-frequency oscillations in this function cause a lack of robustness as small state perturbations can produce significantly different total rewards.2. Even for high-performing policies with stable returns, it remains impossible to accurately predict the long-term behaviour of these chaotic control systems as they rely on noisy partially observable sensors to attain an observation. This unpredictability means that safe and reliable behaviour cannot be guaranteed.To address these issues, we propose Maximal Lyapunov Exponent regularisation for Dreamer V3 [9] . This novel technique estimates the local state divergence using the Recurrent State Space Model and incorporates this term into the policy loss. We demonstrate that this regularisation term significantly reduces the chaotic dynamics produced by this state-of-the-art deep RL controller. This increased stability dramatically improves the robustness of the policy, thus improving the feasibility of deep RL agents for real-world continuous control tasks.2 Background 
robust reinforcement learning: 
 Reinforcement learning provides a data-driven method for solving sequential decision-making problems. This control interaction is represented by a deterministic Markov Decision Process (MDP) with state space S ⊆ R n , action space A ⊆ R m , scalar reward function r : S × A → R, state transition function f : S × A → S and initial state distribution ρ 0 ⊆ S. The objective of an RL agent is to learn a policy π θ : S → A which maximises the sum of discounted returns (Equation 1) for a given discount factor γ ∈ [0, 1).EQUATIONRL policies are said to be robust if they can maintain consistent behaviour and reliable performance in the face of noise or adversarial attacks. This stability is crucial for the safe deployment of deep RL [10] extended the Fast Gradient Sign Method [7] attack to deep RL policies, demonstrating that the addition of a small amount of strategic noise to every observation is sufficient to degrade the performance of trained deep RL agents. Furthermore, Kos & Song [11] showed that this attack does not need to occur at every step, as less frequent attacks with small Gaussian noise still produce a drop in performance. Additionally, they demonstrated that only perturbing the observation when the value function surpasses a set threshold still produces poor performance while significantly decreasing the frequency of the attack. A similar result was established by Lin et al. [13] , who showed that specifically attacking when the relative performance gain between best and worst action surpasses a defined threshold can successfully degrade performance. These findings demonstrate that consistent and inconsistent small perturbations to the system can easily confuse deep RL policies, resulting in unintended and detrimental behaviour. Note that small observation noise is frequent in real-world systems, and therefore, this lack of robustness severely limits the application of deep RL to real-world environments.Lyapunov Exponents [15, 20] provide a method for quantifying the stability of complex, non-linear, high-dimensional systems by measuring the deformation rate of a small hyperellipsoid under the effects of a transition function. In general, for a dynamical system with N degrees of freedom, there are N Lyapunov Exponents, each representing the exponential growth rate of a unique principal axis of the hyperellipsoid. Given a set of N ordered exponents (λ 1 ≥ λ 2 ≥ ... ≥ λ N ), the volume of the hyperellipsoid grows proportionally to e (λ1+λ2+...λ N )t and the length grows proportionally to e λ1t . From this definition, the Maximal Lyapunov Exponent (MLE) (Equation 2) and the Sum of Lyapunov Exponents (SLE) (Equation 3) are used to determine if a system is stable, chaotic or unstable, as outlined in Table 1 .EQUATIONDynamical systems with a negative MLE (λ 1 ≤ 0) are stable as all principal axes of the hyperellipsoid exponentially decrease to zero [21] . In these systems, any trajectories produced from similar initial positions converge to the same trajectory given sufficient time. Conversely, systems with positive MLE (λ 1 > 0) and SLE (λ Σ > 0) are unstable as the resulting state trajectories diverge at an exponential rate. However, for a positive MLE (λ 1 > 0) and negative SLE (λ Σ < 0), similar trajectories will diverge at an exponential rate but remain confined to a subregion of the phase space known as a chaotic attractor [4, 14] . This bounded exponential divergence means trajectories in this region of the state space are unstable and only replicable given the exact same starting state: small perturbations to the starting state will produce significantly different long-term outcomes which appear random and uncorrelated. As a result, it is impossible to predict the long-term behaviour of a chaotic system given an approximation of the initial state.To measure the stability of a known dynamical system, the full spectrum of Lyapunov Exponents can be estimated using the approach outlined by Benettin et al. [2, 3] . This method represents the spectrum as a set of small perturbation vectors which are iteratively updated using the known transition function. To avoid all vectors collapsing in the direction of maximal growth, they are periodically environments when controlled by trained instances of SAC, TD3, Dreamer V3 (DR3) and an agent which takes no actions (None). Each policy-environment combination is independently trained with three random seeds and the average interquartile episode reward with a bootstrapped 95% confidence interval is reported over 80 evaluation episodes each with a fixed length of 1000 steps.Gram-Schmidt orthonormalised so that each vector maintains a unique direction. Performing this orthonormalisation allows for the detection of both positive and negative Lyapunov exponents up to the dimension of the phase space. The spectrum of Lyapunov Exponents is then determined as the average log rate of divergence of the perturbation vectors as they are Gram-Schmidt orthonormalised. Estimation of the full spectrum of Lyapunov Exponents using this method allows for the estimation of λ 1 and λ Σ which can quantify the stability of the dynamical system.Previous studies have investigated the chaotic state and reward dynamics produced by reinforcement learning policies. Rahn et al. [19] showed that the policy optimisation landscape can contain highfrequency discontinuities in the vicinity of a trained policy. A similar result was established by Wang et al. [28] , who proved that control systems with Lipschitz continuous reward and transition functions only have a Lipschitz continuous objective function if λ 1 < -ln(γ). When λ 1 > -ln(γ), the objective function is a fractal and is α-Hölder continuous with holder exponent α = -ln(γ)/λ 1 .Consequently, a single update to the policy in these chaotic control systems can produce substantially different total rewards. Furthermore, Parmas et al. [17] demonstrated that chaos exists in model-based RL methods due to repeated nonlinear predictions and this instability causes gradients to explode during training.While these works use chaos theory to highlight an important issue in the field of RL policy learning, they are focused primarily on the stability of the policy subject to policy parameter perturbations during training. Our work uses similar concepts but instead focuses on the stability of fully trained RL policies subject to state perturbations and the adverse effect this has on the total reward attained in realistic environments. To our knowledge, we are the first to use the spectrum of Lyapunov Exponents to estimate the level of chaos produced by trained DNN policies in continuous control tasks.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: This paper continually highlights an important limitation of deep RL methods, the instability of deep RL controllers. We also propose a method for minimising the chaotic state dynamics and the limitations of this method are discussed in Section 6 and 7.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. Justification:• Our work does not involve human subjects or participants.• Our work uses environments [27] and RL agents [18] which are publicly available.• We do not publish any new datasets.• We ensure reproducibility by including hyperparameters and code.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
chaotic state dynamics: 
 In this section, we use Lyapunov Exponents to identify the level of chaos produced by various state-of-the-art deep reinforcement learning policies in continuous control environments. We claim that the presence of chaos in the MDP implies that the policies are not robust controllers, as trivial changes to the system state produce significantly different long-term state trajectories. This instability poses a significant problem for real-world control systems where consistent and predictable behaviour is necessary. Each policy-environment combination is independently trained with three random seeds and the interquartile average MLE & SLE for each seed is calculated using 20 initial states. A bootstrapped 95% confidence interval is included to show the variation in MLE and SLE across random seeds.To closely match real-world applications, we estimate the stability of tasks from the DeepMind Control Suite [27] when controlled by deep RL policies. These simulated environments provide a range of deterministic control problems with continuous state spaces, as outlined in Appendix A.1.For each control task we independently train three instances of Soft Actor-Critic (SAC) [8] , Twin Delayed Deep Deterministic Policy Gradients (TD3) [6] and Dreamer V3 [9] , as these represent state-of-the-art off-policy and model-based methods for continuous control tasks. Furthermore, to determine if trained deep RL policies directly influence the stability of each control system, we also introduce a passive controller that takes no actions. All models are based on the Stable Baselines 3 [18] implementation with the parameters outlined in Appendix A.2 and trained using an Intel Core i7-8700 CPU workstation with an Nvidia RTX 2080 Ti GPU and 32GB of RAM. The average interquartile reward and bootstrapped 95% confidence interval [1] for each policy type are reported in Figure 2 . These results show that all three algorithms learn policies which performed well and significantly better than the no-action baseline. However, this does not speak to the types of dynamics produced or how robust these policies are to local state perturbations.To identify the stability of each policy-environment interaction, we estimate the full spectrum of Lyapunov Exponents using the method proposed by Benettin et al. [2, 3] . The spectrum is calculated using states sampled from the initial state distribution for each environment. A perturbation vector is initialised for each state dimension at a distance of 10 -4 from the sample state. This represents an arbitrarily small change to the control system without introducing numerical precision errors. The spectrum is calculated over 1000 environment steps and perturbation vectors are orthonormalised every 10 steps to prevent divergence saturation. This process is repeated with 20 initial states and the average value for each exponent is used to calculate the MLE and SLE. Ablation studies for these constants are provided in Appendix B.Figure 3 provides the bootstrapped 95% confidence interval for the MLE and SLE produced by each policy-environment pair. These results indicate that all the environments are naturally invariant to small perturbations as the no action baseline has λ 1 = 0. Conversely, when these environments are controlled by deep RL policies, λ 1 can be non-zero, indicating that DNN controllers directly influence the stability of these control systems. Walker Walk subject to a single initial state perturbation. Initially, each system is separated by only 10 -4 units but the subsequent state trajectories diverge significantly as the control interaction is chaotic.We find that SAC and TD3 produce stable state dynamics in simple low-dimensional environments (Pointmass, Cartpole Balance and Cartpole Swingup) as they have negative MLE. Therefore, if a small perturbation is made to any of the states in these environments, the subsequent state trajectories converge. This result is consistent with the definition of the reward function for each of these tasks as they provide high rewards for maintaining the system at a fixed location. However, when controlled by Dreamer V3, these simple systems exhibit low levels of chaos as they have positive MLE and negative SLE. As a result, state trajectories in these environments are highly sensitive to initial conditions, with similar states producing significantly different long-term outcomes as shown in Figure 4a . Instead of converging to a single state, these trajectories exhibit chaotic behaviour, continuously orbiting within a region which yields high rewards.Furthermore, all deep RL methods produce chaotic dynamics in the complex high-dimensional environments (Walker Stand, Walker Walk, Walker Run and Cheetah Run), as indicated by the positive MLE and negative SLE. This means that these policies cannot account for arbitrarily small changes in the system's state since small changes produce exponentially diverging state trajectories, as illustrated in Figure 4b . This poses a significant problem for real-world applications of RL, as observation perturbations are easily introduced via imperfect measurements or sensor noise. Therefore, for these complex high-dimensional environments controlled by deep RL policies, it is impossible to guarantee stability as the system cannot correct itself after a single inaccurate observation. 
chaotic rewards: 
 In this section, we show that chaotic state trajectories can also impact a policy's performance and produce chaotic reward trajectories as determined by the Maximal Lyapunov Exponent. This instability creates a fractal return surface in which small state perturbations produce significantly different total rewards. We argue that adversarial attack methods could leverage these high-frequency oscillations, repeatedly injecting perturbations which cause the agent to follow the state trajectories that attain the lowest total reward. This lack of robustness poses a significant problem for realworld control systems, where the worst-case performance is often more significant than the average performance.By considering the reward over a state trajectory as a trajectory in a one-dimensional reward space, the stability of the reward can also be measured using Lyapunov Exponents. Given that this space is one-dimensional, only one Lyapunov Exponent exists; however, this single exponent can still be used to reliably identify the stability of reward trajectories. Negative λ 1 values indicate that small perturbations to the system's state still produce converging long-term rewards. Moreover, for a bounded reward function, the reward trajectories cannot diverge indefinitely; thus a positive λ 1 indicates that the long-term reward is chaotic as small changes to the system state produce exponentially diverging bounded reward trajectories. Figure 5 provides the average reward MLE and bootstrapped 95% confidence interval for each policy-environment pair defined in Section 3. These graphs indicate that the reward trajectories produced in simple low-dimensional environments (Pointmass, Cartpole Balance, Cartpole Swingup) are stable as they have negative reward MLE. As a result, a single small state perturbation does not harm the total reward attained as the subsequent reward trajectories converge. Conversely, in high-dimensional control systems (Walker Stand, Walker Walk, Walker Run and Cheetah Run), the reward MLE is positive, indicating that the reward is highly sensitive to arbitrarily small changes in the system state. Consequently, reward trajectories in these systems are unstable, as similar initial states produce significantly different sequences of rewards. Plotting the reward trajectories for one of these chaotic interactions (Figure 6 ) shows that a single perturbation with varying magnitude can have a huge detrimental impact on the total reward attained despite performing well on average. This poses a significant problem for real-world applications of RL as it is impossible to guarantee worst-case performance in a chaotic control system. 
maximal lyapunov exponent regularisation: 
 In Sections 3 & 4, we established that deep RL policies can produce chaotic state trajectories in continuous control tasks and that this can have a large detrimental impact on performance. To address this, we propose a novel regularisation method which improves the stability of RL policies by constraining the Maximal Lyapunov Exponent during policy updates. This improved stability is a crucial step towards the safe and reliable deployment of RL policies in real-world domains where local perturbations are common.Policy (π θ : H × Z → P(A)) Encoder (q ϕ : S × H → P(Z)) Decoder (p ϕ : H × Z → P(S)) Dynamics Predictor (q ϕ : H→ P(Z)) Sequence Model (f ϕ : H × Z × A → H) Current State (s ∈ S) Current Hidden State (h ∈ H) Time Horizon (T ∈ N) Ensure: L λ1 (θ) L λ1 (θ) ← 0 # Initialise the MLE regularisation loss Z = {z l ∼ q ϕ (s, h)} L l=1 # Update stochastic representation for t = 1, 2, ..., T do A ← {a l ∼ π θ (H l , Z l )} L l=1 # Generate a set of sample action H ← {h l = f ϕ (H l , Z l , A l )} L l=1 # Update hidden representation Z ← {z l ∼ q ϕ (H l )} L l=1 # Update stochastic representation S ← {s l ∼ p ϕ (H l , Z l )} L l=1 # Generate a set of predicted states L λ1 (θ) ← L λ1 (θ) + Var(S) + Var(H)# Update the MLE regularisation loss end for return L λ1 (θ)We base our regularisation on Dreamer V3 [9] , a general-purpose model-based RL algorithm which attains state-of-the-art performance across a diverse set of control tasks. To achieve this, Dreamer V3 uses a Recurrent State Space Model (RSSM) consisting of an Encoder (q ϕ : S × H → P(Z)), Decoder (p ϕ : H × Z → P(S)), Dynamics Predictor (q ϕ : H → P(Z)) and Sequence Model (f ϕ : H × Z × A → H) to predict state trajectories (s t ), bootstrapped λ-return trajectories (R λ t ) [25] and state value trajectories (v ϕ (s t )) over a short time horizon T . The policy is then trained to maximise the normalised advantage estimates using REINFORCE gradients [29] and an entropy regulariser (H[•]) [30] with weighting coefficient η. The full loss function used to train Dreamer V3's policy is outlined in Equation 4.EQUATIONEQUATIONEQUATIONAt its core, Dreamer V3 uses a stochastic RSSM to predict the state and reward trajectories over a predefined time horizon given an initial starting state s 0 and an internal representation h 0 . Due to the stochastic nature of this model, repeating the same trajectory predictions L ∈ N times produces a set of state trajectories (S t = ⟨s t,1 , s t,2 , ..., s t,L ⟩) and internal representation trajectories (H t = ⟨h t,1 , h t,2 , ..., h t,L ⟩), each of which provides a plausible estimate of the future states. The variance between trajectories (Var L (•)) thus provides an estimation of the local state divergence as the state perturbation size approaches 0. Therefore, to minimise λ 1 and improve the stability of Dreamer V3 subject to state perturbation, we propose incorporating the regularisation term outlined in Equation 5 into the policy loss (Equation 6). Including this regularisation term as an additional weighted term forces agents to consider the stability of the system during the optimisation process. This incentivises the policy to produce stable state trajectories which attain high rewards instead of solely optimising the expected return. The complete algorithm for calculating the MLE regularisation term is provided in Algorithm 1 using the notation outlined by Hafner et al. [9] . ) subject to N (0, σ) Gaussian observation noise. Each policy-environment combination is independently trained with three random seeds and the average episode reward with a bootstrapped 95% confidence interval is reported over 80 evaluation episodes each with a fixed length of 1000 steps. 
experiments: 
 In this section, we investigate the impact that the proposed MLE regularisation has on Dreamer V3. We show that the inclusion of this term reduces the chaotic state dynamics produced by the control policy and that this improved stability increases performance when noise is introduced. For these experiments, we train three instances of Dreamer V3 with MLE regularisation and reuse the SAC, TD3 and Dreamer V3 policies from Sections 3 & 4. When estimating state divergence, the RSSM predicts L = 3 plausible future trajectories over which the state and internal representation variance is measured. Increasing L will produce a more accurate estimate of state divergence; however, this will require more computational resources. Therefore, to maintain a similar training time to that of Dreamer V3, we set L = 3. All other hyperparameters are consistent with the Dreamer V3 baseline.Table 2 provides the average reward and estimated MLE produced by the regularised and unregularised Dreamer V3 models for each control task. This indicates that MLE regularisation successfully minimises the chaotic state dynamics produced by Dreamer V3 while maintaining similar performance. However, as MLE is still positive in the majority of environments, the control interaction still produced chaotic state trajectories. Despite this, the regularised policies are more stable as the rate of divergence has significantly decreased. To identify how robust these regularised Dreamer V3 policies are to continual state perturbations, we measure their performance when Gaussian noise is added to each observation. Each policy is trained without observation noise (σ = 0) in the fully deterministic variant of each environment and then tested with Gaussian noise (µ = 0 and σ ∈ [0, 1]). Performance is measured over 240 episodes with a fixed length of 1000 steps and a maximum reward per step of 1. Figure 7 shows the interquartile mean and bootstrapped 95% confidence interval for the total reward attained by each policy-environment interaction subject to various levels of Gaussian noise. This shows that MLE regularisation significantly improves the performance of Dreamer V3 in four of the noisy control systems, while the other two environments (Walker Stand and Cheetah Run) attain similar performance to the Dreamer V3 baseline. Furthermore, examining the state trajectories produced by Dreamer V3 and Dreamer V3 + MLE regularisation when controlling the Walker Stand task (Figure 8 ) shows that the regularisation improves the stability of the control interaction as the trajectories do not diverge significantly. These findings indicate that MLE regularisation can improve the robustness of Dreamer V3 subject to observation noise as it produces more consistent and reliable behaviour in the face of uncertainties. 
conclusion: 
 A key issue preventing the application of deep reinforcement learning to real-world environments is the need for guaranteed stability and performance in the face of noisy observations and adversarial attacks. In this work, we set out to identify the impact a single perturbation has on the long-term behaviour of deep RL policies in continuous control environments. Using the spectrum of Lyapunov Exponents, we established that the MDP can produce chaotic state and reward trajectories which are highly sensitive to initial conditions. This instability poses two threats to the application of deep RL to real-world problems, where it is infeasible to attain an accurate measurement of the system state. First, small state perturbation can have a large impact on the performance of trained deep RL policies, even where the average performance is good. This can create hazards in real-world conditions where observation noise is prevalent and can be exploited by adversarial attack methods. Second, even when deep RL policies perform well, they can produce unpredictable behaviours, which is undesirable in most real-world applications.To mitigate these chaotic dynamics and improve robustness we propose Maximal Lyapunov Exponent regularisation for Dreamer V3. This novel approach uses the Recurrent State Space Model to estimate the local state divergence and incorporates this into the policy loss. In effect, the agent optimises its confidence in future trajectories jointly with its expectations of rewards. While MLE regularisation helps improve the robustness of the agent's policies, this approach assumes an accurate estimation of the local state divergence. In environments where the RSSM struggles to capture state dynamics, the effectiveness of the proposed regularisation may be diminished. However, in our experiments, we demonstrate that this regularisation improves the stability of the learnt policies, thereby making them more robust to state perturbations. . Each policy-environment combination is independently trained with three random seeds and evaluated using the parameters in Table 5 , except the number of initial samples, which varies from 1 to 20. The mean and 95% confidence interval indicate that MLE converges with 20 initial samples. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: This paper discusses the important societal issue of safe and reliable control using RL methods.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks as datasets and models are not released.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: 1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: The abstract and introduction claim the Markov Decision Process can produce chaotic state trajectories and that this greatly impacts the stability of the reward. These issues are addressed in Sections 3 and 4 respectively. They also claim that we can reduce the chaotic state dynamics produced by Dreamer V3 thereby improving its robustness to adversarial attacks and this is studied in Sections 5 and 6.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: This paper studied the robustness of deep RL policies to a single small state perturbation in deterministic continuous control tasks. It first showed the vulnerability of deep RL policies, as being deterministically chaotic against small perturbations, which further indicates two facets of chaotic behaviors of the policies in real-world applications. To address the issue, the paper proposed a novel regularisation technique based on Maximal Lyapunov Exponent. The new algorithm reduces the chaotic behaviors and made the learned policies more resilient, with extensive experimental results. It reaches a consensus that the observations in this paper are new and of great importance to deep RL, and the proposed workaround is novel. Recommend acceptance. I suggest the authors to address the remaining comments from the reviewers in preparing the final version of the paper.

Reviews: review: paper_summary: paper_summary: To address the issue of stability of deep reinforcement learning, the authors first gauge the chaotic behavior of various state-of-the-art deep reinforcement learning policies in continuous control environments, and quantify the stability of those policies with significant impact of their applicability to real world problems. Then, the authors proposed an improvement based on implementing a Maximal Lyapunov Exponent regularization in the RL architecture, and demonstrate the improvement with examples.
strength_weakness: strength_weakness: The proposed method on maximal Lyapunov exponent regularization needs to be further explored and studied, both in terms of the theory involved and amount of experiments conducted. The idea is novel, but not fully explored and explained. The current set of numerical experiments are limited and not truly convincing.
questions: questions: Could more explanation of all the terms and parameters in Equation (4) be provided? Also, how is the proposed regularizing term in (5) is related to (4)?
limitations: limitations: The authors adequately addressed the limitations.
paper_summary: paper_summary: Deep RL methods are usually lack of robustness in control tasks whose dynamics are chaotic, thereby having positive maximal Lyapunov exponents (MLEs). This paper proposes an approach that improves the stability of trained deep RL controller through MLE regularization.
strength_weakness: strength_weakness: * This paper dedicates many pages to the chaotic phenomena in RL (Sections 3 and 4), which have been addressed in previous works as introduced in Sections 1 and 2. The core contribution, which I believe is the regularization technique, needs more elaboration and analysis.

* There is no concrete algorithm provided that the audience can follow to easily implement the approach proposed in this paper. Imagine a person with little or no knowledge of dynamical systems. Can they manage to implement the algorithm after reading the paper?

* (minor) Some information is missing. For example, what are $S$, $H$ and $v_\phi(s_t)$ in equation (4)?
questions: questions: * Does the policy network architecture affect the performance of your approach?

* Is it possible to find two trajectories with equal rewards, where one is stable and the other is chaotic? In other words, the reward itself may not be sufficient to reflect the stability of a trajectory.

* How does the approach perform in the case that it is unable to estimate the MLE accurately in practice?
limitations: limitations: It has discussed its empirical limitations in Section 6.
paper_summary: paper_summary: This work studies adversarial RL and policy stability from the perspective of the Lyapunov spectrum. A regularization term is introduced to encourage more robust policies.
strength_weakness: strength_weakness: The weaknesses I've found are mostly minor, but I still hope you can address:

- The correlation between Figure 2 and 3 is not immediately clear to me; can you please discuss this further?
- On that note, the improvement in terms of reward is not clear (I know this is not necessarily "the point" but given the discussion of reward trajectories, it would be nice to see further iumprovemetn here
- How error-prone is the calculation of LEs discussed in Sec 2.2?
- It's not very clear to me the use of "no actions" in the figures. Can you explain how this "benchmark" should be interpreted? How about in environments where there is no such "do nothing" action?
- Can you include non-scalar comparisons of algorithms (cf. https://github.com/google-research/rliable)
- The appendix is a bit sparse. Can you provide any further experiment details?
questions: questions: - Can you please provide legends in the figures? If you have the data, can you include a third panel in Fig. 6 corresponding to the regularized policy? (otherwise this figure is ambiguous without comparison)
- Maybe I missed it, but what is $S$ in Eq. 4?
- Can you  measure the divergence in trajectories such as in Fig. 4 & 8 and compare to the calculated LE?
- By how much do you weight the regularization in Eq. 5?
- Can you discuss the use of hidden states $h$? It isn't clear to me how this affects the loss beyond what the first term contributes

These questions may open new areas of research, and I don't intend for them to be answered within this paper, but I'm curious to hear your thoughts on the following:
- How does this work relate to the idea of imposing Lipschitz constraints / regularization on DNNs? I see it as an orthogonal direction, but can they solve the same problem? If so, can they be compared?
- Can we use pre-trained policies, then finetune them wrt the LE regularization to make them more robust?
- In 2.3, can you discuss the case of $\gamma \to 1$?

minor typographical:
- Eq. 1, I believe you need a conditional on the state $s$ and an expectation over initial states.
- L161: "...low levels of chaos **as** they have..."
- Eq. 5 use parentheses over both t-dep terms
- Please increase fontsizes in Fig 7
limitations: limitations: Can you please include limitations in Sec 7?

Comments: title: Response to the rebuttal
Reply to rebuttal
comment: I appreciate the authors' responses, thank you for addressing the questions raised. I believe the revised version will make for a great paper. I will maintain the current score as it appears the reviewers have converged.
Thank you the the rebuttal. They address some of my concerns. I would also like to raise my score to (6).
Thank you very much for your positive feedback and for raising your score. We are delighted to hear that we have effectively addressed your concerns and appreciate your suggestion to add a formal algorithm. We agree that it will enhance the clarity of our work and will, therefore, incorporate this into our camera-ready version.
I would like to thank the authors for their rebuttal and it has resolved my concerns. I still recommend the authors to add a formal algorithm table for their approach so that the audience can easily implement it. I am raising my score to 6.