Policy Optimization in a Noisy Neighborhood: On Return Landscapes in Continuous Control
Authors: Nathan Rahn
Pierluca D'Oro
Harley Wiltzer
Pierre-Luc Bacon
Marc G Bellemare
Venue: NeurIPS 2023 poster (2023)
Keywords: deep reinforcement learning
continuous control
return landscape
stability
TL;DR: A study of the return landscape traversed by deep reinforcement learning algorithms for continuous control.

Abstract: Deep reinforcement learning agents for continuous control are known to exhibit significant instability in their performance over time. In this work, we provide a fresh perspective on these behaviors by studying the return landscape: the mapping between a policy and a return. We find that popular algorithms traverse noisy neighborhoods of this landscape, in which a single update to the policy parameters leads to a wide range of returns. By taking a distributional view of these returns, we map the landscape, characterizing failure-prone regions of policy space and revealing a hidden dimension of policy quality. We show that the landscape exhibits surprising structure by finding simple paths in parameter space which improve the stability of a policy. To conclude, we develop a distribution-aware procedure which finds such paths, navigating away from noisy neighborhoods in order to improve the robustness of a policy. Taken together, our results provide new insight into the optimization, evaluation, and design of agents.

Policy Optimization in a Noisy Neighborhood: On Return Landscapes in Continuous Control
Nate Rahn Mila, McGill University
Pierluca D'Oro"
Harley Wiltzer Mila, McGill University
Mila, Université de Montréal
Pierre-Luc Bacon Mila, Université de Montréal
Mare G. Bellemare
Mila, McGill University
Abstract
Deep reinforcement learning agents for continuous control are known exhibit significant instability in their performance over time. In this work, we provide a fresh perspective these behaviors by studying the return landscape: the mapping between a policy and return. We find that popular algorithms traverse noisy neighborhoods of this landscape, in which single update the policy parameters leads to a wide range of returns. By taking a distributional view of these returns, we map the landscape, characterizing failure-prone regions of policy space and revealing a hidden dimension of policy quality. We show that the landscape exhibits surprising structure by finding simple paths in parameter space which improve the stability of a policy. To conclude, we develop distribution-aver aware procedure which finds such paths, navigating away from noisy neighborhoods in order to improve the robustness of policy. Taken together, our results provide new insight into the optimization, evaluation, and design of agents.
Introduction
It is well-documented that agents trained with deep reinforcement learning can exhibit substantial variations in performance as measured by their episodic return. The problem is particularly acute in continuous control, where these variations make it difficult to compare the end product of different
algorithms or implementations of the same algorithm [I] (20] or even reliably measure an agent's progress from episode episode [9]. A recurring finding is that simply averaging the return produced by a set of policies may be insufficient for rigorous evaluation.
In this paper, demonstrate that high-frequency discontinuities in the mapping from policy pa- rameters 0 to the return R(0) are an important cause return variation. As consequence of these discontinuities, single gradient step perturbation to the policy parameters often causes important changes in the return, even settings where both the policy and the dynamics are deterministic. Because agent's parameters constantly change during training and should be robust to minute parametric perturbations, we argue that the distribution of returns in the neighborhood of 0 fact better representative of its performance, both from evaluation and optimization perspective.
Noisy neighborhoods in the return landscape. We call the return landscape the mapping from 0 to R(0), our main object of study. We show that the return often varies substantially within the vicinity of any given 0, forming what we call a noisy neighborhood of 0. Based on this observation, demonstrate the usefulness of studying the landscape through the distribution of returns obtained from small perturbations of 0. In the important case where these perturbations result from a single gradient step, call the resulting object the post-update return distribution.
'Equal contribution. Correspondence to (nathan. rahn, www.rraaannnnrraaa haaba.pperruca...ooooiiilaaaa
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Return Landscapes
Post-Update Returns
Behaviors
Updar Dhredtoat
Updute Unrretionn
Luil
300
1000
1500
2000
2000
Rettrm
11000
50
Updaae Diro
Update Dirertoon
500
1000)
1500
2006
2500
Figure I: A visualization for two policies visited by SAC in the hopper environment. We show the return landscape in their proximity, their post-update return distributions, and the visual appearance of their learned gaits. We plot the mean of each return distribution as an orange line. Despite featuring similar level of return, curved gait which is faster but more prone to failure, as visible in the thick left tail of the post-update return distribution.
observe that the policy in the noisy neighborhood performs an unstable
Diversity in gouully-performing policies. We show that different neighborhoods correspond to different post-update return distributions and agent behaviors. We discover that at equal average returns, different policies obtained by the same deep RL algorithm may in fact have substantially different distributional profiles, as measured by statistics of the post-update return distribution. Moreover, uncover that many of these distributions are long-tailed and find the source of these tails to be sudden failures from an otherwise successful policy.
Effect learning dynamics. We expose how the transition between noisy and smooth parts of the landscape happens. Surprisingly, while large valleys of low return are visible when linearly interpolating between similarly performing policies from different runs, show such valleys typically exist between policies from the same run. Based on this insight, we show that it is possible to find improved set of parameters 0 that achieves comparable return, but substantially lower post-update variation.
We believe the phenomenon we study is central to deep reinforcement learning in continuous control. Beyond its effect on learning dynamies (for example, through increased variance and implicit exploration (39]), it is also a potential driver of instability in sim2real settings, even in the face of seemingly small environmental changes. Additionally, it suggests that one should not simply deploy the policy obtained at the end of training run, and that further post-training tuning may be beneficial.
2 Background
In reinforcement learning, an agent interfaces with an environment. In this paper, we are interested in continuous control environments modelled as finite-horizon Markov Decision Process (MDP) M = (S.A, A, f.T. Po), where S RD is the state space, A Rm is the action space, x S R is a reward function, f S x + S is deterministic transition function, T is the horizon, and Po = U(s 8, + (3) is an initial state distribution with being an initial reference state, and 3 E Ra an pevironment-dependeee
parameter. We assume that each agent produces a stationary Markovian deterministic policy S - A within a parametrized family Th = {To 0 E θ Rd). In an episodic setting, the interaction the agent with the environment with given policy me from some state S produces trajectory in the environment and,

---

consequently, retur:
T rstt 0z)
Go(s) Σ t=1
(1)
s.t. St = f( a-11), (1)
We are interested in understanding how small changes to the policy parameter affect the associated return. To this end it is sufficient to study the return from the reference state $1 (in Appendix A.4 we show that similar effects occur across the state space). The return landscape our main object of study.
Definition 2.1 (Return Landscape). The return landscape is the mapping from policy parameters to return, starting from the initial reference state:
R(0) = Go(si).
(2)
Figure T (left) depicts small portions of the return landscape for particular environment and policy parametrization (we describe the visualization procedure below).
In this work, we will use the policies discovered by popular algorithms to characterize the topology of the return landscape. We focus policy- -bssed deep reinforcement learning algorithms for continuous control, such as Soft Actor-Critic (SAC) [19]. Twin-Delayed DDPG (TD3) [16]. and PPO [22 which use neural network function approximators to represent the policy. Such algorithms learn good behavior in the environment by maximizing the discounted return. In the process, they produce sequence of policies
0o, O,,..., ON. ; s.t. 0c+1 = u(O,, X,) for all 6
where u : θ x R + θ is the algorithmic policy update function, and X, is some random variable abstracting the stochasticity inherent to the update. For example, SAC and TD3 construct parametric updates by sampling small number of transitions (minibatches) from their replay buffer (2
3 A Distributional View on Return Landscapes
The return landscape arises from the interaction between an environment and a class of parameterized policies. We first consider how this landscape varies in the immediate vicinity (or neighborhood) of policies produced by deep reinforcement learning algorithms. Given a reference policy, a natural choice consider how the return is affected by single updates the policy parameters. To this end, we view the collection of possible returns obtained by evaluating the updated policy as a distribution over returns; as we will see, this distribution widely varies across the return landscape.
Definition 3.1 (Post- Update Return). Let IIg be a parametric space of deterministic policies and an update function. Given E le. its post-update return is defined as:
R(0)= R(u(0,X) X P.
where a an algorithm-dependent source of stochasticity.
The post-e return inherits randomness from the underlying training algorithm and it is thus a random variable. Clearly, a post-update return will have an associated policy and trajectory, which are in turn random variables. this work, we will leverage the distribution of post-update returns as a tool to investigate the properties of neighborhoods of the return landscape.
The different panels of Figure illustrate how the return landscape in the neighborhood of translates into different post-update return distributions. Here, the return landscape is visualized along two update directions computed by the training algorithm based on two different batches sampled from its replay buffer, such that 1.0 on each axis corresponds to a single parameter update in that direction (details Appendix A.2). The middle panel shows the corresponding post-update return distribution estimated using 10000 samples. We find that the distribution from the noisy neighborhood (top) exhibits significant left tail, while the distribution from the quieter neighborhood (lower) is concentrated around its mean. On the right, we illustrate the gait produced by the reference policies (the origin in the left panel). We find qualitatively that the policy in the noisy neighborhood exhibits a curved gait which is sometimes faster, but unstable, whereas the policy in the smooth neighborhood produces an upright gait which can be slower, yet is very stable. We include similar evidence for other environments in Appendix A.II
3

---

4000
4000
4000
Moont
2000
2000
-2000
2000
0
7001
1000
1500
20
10
0
055
0.156
1%
10%
Standard Deviation
Probsabiliity
*
x4
*
Deusty
0.002
0.0000
0.02
0.001
0.005
0.001
0,0005
0.000
0.000
0,0000
0.00
0.000
0.000
5000
5000
5000
0
5000
5000
5000
Roturn
Return
Rettum
Retiar
Figure 2: A scatter plot showing mean return and standard deviation, skewness or left-tail probability of the post-update return distribution of policies produced by three popular deep RL algorithms on the ant Brax task. Each point corresponds to a given policy's post-update return distribution, with six selected policies highlighted by star markers showing range of diverse distributions.
3.1 Post-Update Return Distributions as Characterization of the Return Landscape
The mean of the post-update distribution naturally captures the average behavior represented by an algorithm as traverses a given neighborhood. We further characterize this distribution by measuring its standard deviation (a measure of spread around the mean) and its skewness (a measure of asymmetry). In our context, a negative skewness describes a distribution with a heavy left tail, similar the one shown in Figure Such tail is especially interesting than-expected returns. However, we find that skewness is not directly interpretable as a numerical quantity. To capture these tails interpretably, we introduce metric we call left-tail probability. The left-tail probability of a random variable defined as
indicates lower-
LTP P/O < Y < (eemoode(Y)).
(5)
This quantity satisfies desirable properties within the context of our study. First, it uses the mode of the distribution as a reference value. This is by contrast with the mean of the distribution, which may not correspond the "majority" behavior (as illustrated in the top half of Figure [ also allows us to more easily compare the tailedness of distributions generated from policies of widely varying returns. Second, is an easily-interpretable quantity which measures the total probability mass falling in the left tail. For simplicity, here we assume that Y is positive, noting the idea can be naturally generalized random variables bounded below. In our analyses we write LTP = LTP 1/2 to measure drops from the mode of the post-update return distribution of at least 50%. In practice, we estimate the LTP by leveraging the Chernoff estimator [00, computing the mode as the midpoint of the interval of the most populated bin in 100-bin histogram.
Equipped with these metrics, we measure the return for a set of 600 policies produced, across trials and iterations, by three popular deep RL algorithms (TD3, SAC and PPO). We use 20 seeds per algorithm and 10 checkpoints per seed, for total of 200 policies per algorithm. These checkpoints are equally-spaced in time in training runs of million steps for TD3 and SAC and 60 million steps for PPO. Each of the 600 distributions is estimated by performing 1000 independent updates to the starting policy and then rolling the resulting deterministic policy out in the environment for 1000 timesteps to compute its return. Each update different due to different batch sampled from the replay buffer for TD3 and SAC, and to a different batch data from the environment collected by a randomly-perturbed policy for PPO. This amounts to millions of policy evaluations for which, for computational reasons, we primarily use the easily parallelizable environments from the Brax simulator [5]. We also include similar results on the post- update return distributions of policies trained DeepMind Control Suite [4 and games from the
and the other three statistics of the post-update

---

halfcheetah
walker2d
3000
Ng. palling
Ro. Dailng
Romaed
4000
Rowsed
2000
2000
1000
3000
'Tiue
Man, S. Suaccssffal
Tine
May Suocessful
Figure 3: A visualization of how failures occur in the halfcheetah and walker2d tasks. The left subplots compare the reward-pe geward-per-iee timestp obtained by a successful and failing trajectory generated by two policies in the same noisy neighborhood. The right subplots show the simultaneous evolution of returns for 10 such trajectory pairs (that can be thought of as a race to collect the most rewards), with the trajectory pair from the left indicated by matching star marker. The right subplots indicate that policies from the same neighborhood behave similarly (diagonal segments of the curve) until the failing policy makes a sudden misstep and collects low rewards (horizontal segments).
ALE 7 in Appendix A.5 and A.6 Additional experimental details, including the hyperparameters and implementations used for running these algorithms, can be found in Appendix A.
Figure 2 iilustrates how different policies produced by deep RL algorithms correspond to a wide range of post-update return distributions, measured by our chosen metries [ report the bootstrapped mean using 1000 resamples to account for sampling error in the post-update returns collected for given policy, and omit the corresponding bootstrapped confidence intervals for visual clarity, as they are very small. In particular, this scatter plot shows that different policy parameters achieve similar levels of returns (as measured by the distribution mean) but wide range of possible levels of variability, as measured by standard deviation, skewness and left-tail probability. This suggests, in similar way the example shown in Figure [ that algorithms discover behaviors which can be qualitatively very different from one another, and that leveraging the post-update return distribution can offer a new lens to investigate different dimensions of policy quality.
For each metric,
These results suggests that simply optimizing the mean return of policy might ignore its distributional aspect. In particular, a practitioner will likely prefer, for given level of return, a policy featuring a post-update return distribution with lower levels of standard deviation or left-tail probability. Intuitively, such a policy may correspond to safer behavior, both able to more robustly accommodate additional updates from its training algorithm and possibly to deal with other unexpected sources of perturbation during deployment.
3.2 Analyzing Failures
One characteristic feature of the post-update distributions studied above is the existence of a significant lower tail for many policies visited by the three deep RI. algorithms TD3, SAC and PPO. This visible in their skewness, but especially in their left-tail probability, which demonstrates that many policies produce returns which are unexpectedly poor after up to roughly 10% of updates. We now take closer look at the specific mechanism by which small changes in an agent's 's actions results in wide range of returns in continuous control.
Our experimental procedure is as follows. For each environment, we randomly select 10 policies from the logged checkpoints of 20 independent runs of TD3, conditioned on the fact that the policy has left-tail probability which is greater than zero. These are policies that we know are prone to poor returns following an update. For each policy, we compute the post-update return distribution by collecting trajectories in the environment after single update to the original policy. According to this procedure, we identify two trajectories drawn from the neighborhood around the policy: successful trajectory, characterized by return within 10% of the mean of the post-update distribution, and a failing trajectory, characterized by a return of less than 50% the mode of the post-update distribution, as in the left-tail probability.
NNote that the LTP is not properly defined for small number of policies achieving negative return, that appear points the right border of the scatter plot.
5

---

Our goal is to understand the differences between these successful and failing trajectories in order to explain how long-tail returns occur. To this end, Figure depicts two views of the trajectory data. For each environment, the left subplot considers single pair of successful/falinng trajectories corresponding to one of the chosen policies, and plots the reward per timestep earned in these two trajectories. These results suggest that the failing policies which make up the tail of the post-update distribution are capable of collecting similar rewards to the successful policies, yet are prone to missteps which result in episode termination (as in walker2d) or transition to a low-reward, quasi- absorbing state (as in halfcheetah). Figure shows an example of such a misstep in walker2d.
We present broader view of these observations through the right subplots, per-environment, in Figure [] Here, we plot each of the trajectory pairs as a parametric curve in time. For both the successful and failing trajectories, we compute the return up to time t, Rg: = Ei= 7(st, a:). Then, for each value of t, we plot Rgr for the successful and failing trajectories as a point on the curve, allowing us to visualize the simultaneous evolution of both trajectories. We assume that Rig++ = R.g. when the length of one trajec- tory exceeds the other, that is, no additional reward is collected after the trajectory terminates. The resulting visualization re- veals several notable findings. First, we show that nearly all trajectory pairs begin by following the line y = I, indicating that the respective policies collect rewards at almost exactly the same rate. Next, we observe that many curves rapidly diverge from this line to horizontal, indicating that the failing trajectory suddenly starts collecting little no reward, while the success- ful trajectory continues. In walker2d, these divergences reflect sudden terminations of the episode, represented by horizontal lines. In half fchffhhhh,, which does not terminate, we see that instead the failing agent gets stuck in low-reward absorbing states, but is sometimes able to recover and go back to collecting reward at the same rate as the successful trajectory. We include similar visualizations for the hopper and ant environments in Appendix A.8 which support the same conclusions.
The trajectory of successful (top) and failing (bot- tom) policy, both coming from the same post-update distribution in walker2d. They exhibit similar gait until right before the failure.
Figure 4:
Taken together, these results demonstrate that some policies exist on the edge of failure, where a slight update can trigger the policy to take actions which push it out of its stable gait and into catastrophe. Indeed, when we compare the gaits learned by policies of high left-tail probability to those which are more well- behaved under updates, we observe that the behaviors of the former are qualitatively more unstable (Figure with more examples in Appendix A.1
Navigating Return Landscapes
In the previous section, we took fine-graines look at the return landscape, using post-update return distributions to characterize the neighborhood of different policies learned by deep RL algorithms. We now consider this landscape on more global scale, specifically how the agent's 's return changes as one interpolates between different policies.
4.1 Connectivity in the Return Landscape
For our analysis, we use 200 policies generated by different runs of TD3. From these we select pairs of policies with different post-update return distributions, as measured by their standard deviation or left-tail probability, but similar mean. Consider two sets of policy parameters O and Oz. for which the post-update return distribution implied by 0, has higher LTP than the implied by 0g. We linearly interpolate these two to form family of parameters 0 = a0, + (1 0)2, E [0. 1]. For each such 0, we then record the return R( (0) obtained by a single simulation with the corresponding policy.
In Figure 5] we show the result of this interpolation for six pairs of policies in the hopper and walker 2d environments, in two distinct cases. In the first case, the two policies have been produced by the same run of TD3 (i.e., starting from the same initialization and history of batches); in the second case, the two policies have been generated by independent repetitions of the algorithm. The plot shows interesting information about the global structure of the return landscape: the interpolation

---

hopper
walker2d
Same Run
Different Run
Sume Run
Different Run
1200
3000
1500
1500
1000
800
Return
2000
1600
2000
1500
2000
1500
1000
1000
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Interplation
Interpalction
Interpolation
Inderpolation
Figure 5: Return of the policies obtained by linear interpolation of the parameters of policies of ap- proximately the same level of return in the hopper and walker2d environments. The neighborhoods traversed transition from being noisy to being smooth; policies from the same run are connected by paths with no valleys of low performance in the return landscape, even if separated by hundreds of thousands of updates (i.e., at least x 10 steps for all pairs of policies from the same run).
process traverses different parts of the landscape, highlighting transition between noisy part of the landscape to an inherently smoother one. Interestingly, the interpolations between policies from the same run and from different runs exhibit very different qualities. When interpolating between policies of different runs, the process traverses entire regions of the landscape of poor return, until the point in which it gets to the neighborhood of the second policy. By contrast, when interpolating between policies from the same run, the transition from a noisy to a smooth landscape happens without encountering any valley of low return even when these policies are separated by hundreds of thousands of gradient steps in training. This is particularly surprising given that O is a high-dimensional vector containing all of the weights of the neural network, and there is priori reason to believe that interpolated parameters should result in policies that are at all sensible.
To further quantify the phenomenon, we want to mea- sure the proportion of return collapses encountered when interpolating between policies. We use the following ex- perimental design. We sample for each environment a set of 500 pairs of policies from the same runs and set of 500 pairs of policies from different runs. Then, we linearly interpolate between policies in the pairs, producing 100 intermediate policies, and randomly perturb them using Gaussian noise with standard deviation tain an estimate of the mean of their (random) post-update return distribution. Then, for each pair of policies, we measure how frequently the return collapses in between the two extremes, by counting how many times it becomes less than 10% of the minimum return of the two original policies. We then average this Selow-Threshold Propor- tion across pairs, and across environments using rliable [I]. Figure(6Jshows shows that there is on average almost no drop in return when interpolating among policies from the same run. We additionally report similar results on four ALE games in Appendix A.6
Different Run
Same Run
0.15
0.30
0:45
Tohos-Thwsoold Progortion
Figure 6: Proportion of return collapses when interpolating between randomly- sampled policies produced by either the same or different runs in Brax. Far fewer return collapses are observed when in- terpolating between policies produced by the same run. Results are aggregated over all environments with 95% boot- strapped C.I and 500 pairs of policies.
10-4 ob-
We hypothesize this might be interpreted as each individual run of the algorithm specializing on a different family of behaviors, for which, due to the geometry of the return landscape, interpolation between policy parameters does not have any disrupting effect. This result can be interpreted as being related to linear mode connectivity 17 18). a phenomenon observed in supervised learning, for which different points in the loss landscape of neural networks can be connected by sear-constant-loss paths. In other words, it appears there is typically no barrier of low average return separating policies generated from the same run, even when those policies feature very different levels of stability. The existence of such phenomenon in the RI. setting is far from certain: the optimization objective is non-stationary and the evaluation metric (the return instead of the loss) depends on an environment and multiple forward passes from neural network.
7

---

Median (Log LTP Improvement)
IQM (Log LTP Impeovemeut)
Mean (Log LTP Improvement)
Without rejection-
With rejection
0.25
0.50
0.75
1.00
0.2
0.8
1.2
Figure 7: LTP reduction over 40 gradient steps without rejections (TD3) and with rejections (Algo- rithm 1). Data is aggregated over starting policies, environments, and five independent runs for each starting policy. We see that Algorithm is strictly superior to TD3 with respect to LTP reduction. Results are aggregated over environments with 95% bootstrapped confidence interval.
4.2 Stabilizing Policies by Navigating the Landscape
Overall, Figure demonstrates the existence of paths in the return landscape which are able to increase the level of stability of a given policy, but are not necessarily followed spontancous way by typical policy optimization algorithms. In the absence of desirable end policy to interpolate towards, we would like to understand if it possible to find similar stabilizing paths (as measured by the LTP), given starting policy inhabiting noisy neighborhood of the return landscape. We conjecture that this feasible by filtering the policy updates produced by an algorithm: In particular, we propose reject gradient updates that lead to policies with less favorable post-update return distributions.
In procedure, which is out- lined in Algorithm|1 [ we use the CVaR heuristic compare the stability of post-update re- it is effec- tively a measure of the left tail mean (38. Our procedure works as follows: starting with given policy, use TD3 interact with the environment, maintain a replay buffer, and compute up- dates to the policy and critic pa- rameters. However, before apply- ing proposed update, we first
Input: Initial policy parameter 0, CVaR level a, policy update function u, number of MC samples N, tolerance
while True do
RRg Ik
iid ((0) (Post-update return samples)
turn distributions
B random minibatch
0' uu0, B)
[R Jk-1 ud R(0')
CVaR., ((
(1- (( Ma)) then
0'
end end while
Algorithm 1: Post- Post-Update-CVVa Rejection
estimate the post-update return distributions of the post-update policies by sampling TD3 updates from random minibatches of the replay buffer and evaluating the returns of the corresponding policies. sufficiently high relative that of the post-update
the estimate of the post-update return CVaR return distribution of the current policy, the update is rejected, so that the networks and the replay buffer are reverted to the state that they were in before the update was computed. our experiments, we study the effect that such rejection mechanism has on the evolution of the L.TP by comparing the trajectories induced by this procedure without the ability to reject (i.e., regular TD3) and with the ability to reject.
In Figure 7 we show the improvement in LTP that this algorithm induces when applied to the same policy, aggregated across Brax tasks, using at least 10 policies per environment, after only forty gradient steps. We additionally present scatter plots demonstrating the effect of applying Algorithm I to individual policies in Appendix A.10 Our results demonstrate that this rejection procedure can be an effective tool for reducing the ITP of existing policy.
5 Related Work
Reliability of deep RL. The goal to avoid catastrophic drops in performance was at the core of the
development of foundational methods in deep RL based on conservative updates [41 42). Previous work also studied the development of safer algorithms for learning and exploration, both from the theoretical and the empirical standpoints
25 30
32
37 48 Our work focuses on understanding
the landscape visited by commonly employed policy optimization algorithms and shows that it is
See Appendix A.9 for further justification.
8

---

possible to relatively easily move from parts of the landscape that induce dangerous behaviors to safer policy parameter vectors. On higher-level, the sensitivity of deep RL algorithms to stochasticity and hyperparameters, and the extreme variability of results across seeds has been the object of previous studies 2 20]. which mostly focused on proposing more reliable evaluation metrics. Previous work 99 also explicitly advocated for measuring the stability of deep RL. algorithms over different axes and using a diverse set of metrics. Our paper proposes a complementary perspective, based on return landscapes and on distributional view on them. Our procedure which leverages the directions proposed by a policy optimization algorithm to improve the LTP of a policy is related to previous work based rejectiin//acckracking strategies 25 40
Return and loss landscapes. Return landscapes have been previously investigated a coarser level under the name of reward surfaces/landssapes.. In particular, they have been employed for studying the alignment of the gradient directions suggested by policy optimization algorithms to directions of improvement in the actual environment (23) and investigating performance degradation as long-term Our study of return landscapes with a distributional view in an otherwise fully deterministic setting sheds new light both the landscape itself and on how it can be leveraged to characterize individual policies. More generally, the investigation of the return that policies collect in an environment is related to the study of the loss landscape of neural networks in supervised learning, for which different techniques have been proposed [28]. Those techniques, together with RL-specific tools, have been employed to explore the loss landscapes of RL algorithms, by visualizing them [5). probing their interaction with entropy regularization [3] or larger neural networks [35. Our discovery of how policies from the same run connected by simple paths in parameter space is related to (linear) mode connectivity, which shows a similar behavior in the landscapes of neural networks trained in supervised learning tasks 14 17 18] Finally, our work is related to distributional RL. [6], but we specifically focus on the post-update return distribution as opposed to the distribution of returns under given policy.
optimization danger in such algorithms
Discussion and Future Work
In this paper, we have investigated return landscapes in continuous control tasks, as traversed by deep RL algorithms. We demonstrated the existence of noisy neighborhoods of the landscape, where a single update to the policy parameters produces a wide-breadth of post-update returns. By taking a distributional view on these neighborhoods, we revealed the existence of neighborhoods of similar mean return, yet different statistics, which correspond to qualitatively different agent behaviors. We studied the characteristics of failing policies and trajectories in such neighborhoods and attributed their subpar performance to sudden collapses in trajectory reward, rather than overall degradation in the policy. By focusing on linear paths through the global policy landscape, we showed that the landscape exhibits macro-scale variations which extend beyond specific local neighborhoods, and that policies from the same run can be surprisingly connected by linear paths with no valleys of low return. Finally, we demonstrated simple procedure which discovers paths towards smoother regions of the landscape, starting from a trained policy.
Our results suggest that some of the neeviously-obsevee reliability issues in deep reinforcement learning agents for continuous control may due to the fundamental structure of the return landscape for neural network policies. In particular, while the return of policy in a given neighborhood may be adequate, the distributional structure of the neighborhood characterizes additional dimensions of policy quality: How stable is this policy? What kind of behavior has the agent learned? Is it safe to perform further optimization of this policy? These nuances indicate the potential utility of a landscap-insppreed approach to the design of reliable deep RL algorithms.
In addition, our study of parameter interpolation on the return landscape reveals new curiosities surrounding the training behavior of deep reinforcement learning agents. It appears that many policies from the same run fall within single basin of the return landscape; we conjecture that this may correspond to the algorithm "specializing" on one particular behavior. Our demonstration of regions of lower and higher variability in returns along such paths further supports the possibility of robustifying existing policies, yet also raises the question of whether there are significantly different behaviors separated by barriers of low return, and whether our algorithms can find them. As they are beyond the scope of this paper, we reserve such questions for future work.

---

7 Acknowledgements
The authors thank Jesse Farebrother, Georg Ostrovski, David Meger, Rishabh Agarwal, Josh Greaves, Max Schwarzer and Pablo Castro for insightful discussions and useful suggestions on the early draft, the Mila community for creating stimulating research environment, and the Digital Research Alliance of Canada for computational resources. This work was partially supported by CIFAR, Fonds de recherche du Québec (FRQNT) and Gruppo Ermenegildo Zegna.
References
[1] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C. Courville, and Mare G. Bellemare. Deep reinforcement learning at the edge of the statistical precipice. In Neural Information Processing Systems, 2021.
[2] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C. Courville, and Marc G. Bellemare. Deep reinforcement learning at the edge of the statistical precipice. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages 29304-29320, 2021.
[3] Zafarali Ahmed, Nicolas Le Roux, Mohammad Norouzi, and Dale Schuurmans. Understanding the impact of entropy policy optimization. In International Conference on Machine Learning. 2018.
[4] Igor Babuschkin, Kate Baumli, Alison Bell, Surya Bhupatiraju, Jake Bruce, Peter Buchlovsky, David Budden, Trevor Cai, Aidan Clark, Ivo Danihelka, Claudio Fantacci, Jonathan Godwin, Chris Jones, Tom Hennigan, Matteo Hessel, Steven Kapturowski, Thomas Keck, Turii Kemaev, Michael King, Lena Martens. Vladimir Mikulik, Tamara Norman, John Quan, George Papa- makarios, Roman Ring. Francisco Ruiz, Alvaro Sanchez, Rosalia Schneider, Eren Sezener, Stephen Spencer, Srivatsan Srinivasan, Wojeiech Stokowiec, and Fabio Viola. The DeepMind JAX Ecosystem, 2020.
[5] Recep Yusuf Bekci and Mehmet Gümüiş. Visualizing the loss landscape of actor critic methods with applications in inventory optimization. arXiv preprint arX1::2009.0239,, 2020.
[6] Mare Bellemare, Will Dabney, and Mark Rowland. Distributional reinforcement learning, 2022.
[7] Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents (extended abstract). In International Joint Conference Artificial Intelligence, 2012.
[8] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPP programs, 2018
[9] Stephanie CY Chan, Samuel Fishman, Anoop Korattikara, John Canny, and Sergio Guadarrama. Measuring the Reliability of Reinforcement Learning Algorithms. In International Conference on Learning Representations, 2019.
[10] Herman Chernoff. Estimation of the mode. Annals of the Institute of Statistical Mathematics, 16(1):31-41, 1964.
[11] Cédric Colas, Olivier Sigaud, and Pierre- Oudeyer. A hitchhiker's guide statistical comparisons of reinforcement learning algorithms. In Reproducibility in Machine Learning, ICLR 2019 Workshop. New Orleans, Louisiana, United States, May 2019. OpenReview.net, 2019.
[12] Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In International conference on machine learning, pages 1318. PMLR, 2018.
10

---

[13] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. International Conference on Machine Learning. pages 3259-3269. PMLR, 2020.
[14] Daniel Freeman and Joan Bruna. Topology and geometry of half-rectified network optimiza- tion. In 5th International Conference on Learning Representations, ICLR 20/7, 2017.
[15] C. Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax differentiable physics engine for large scale rigid body simulation. ArXiv, abs/2106. 13281, 2021.
[16] Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. ArXiv, abs/1802.09477,, 2018.
[17] Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information processing systems, 31, 2018.
[18] lan J. Goodfellow and Oriol Vinyals. Qualitatively characterizing neural network optimization problems. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[19] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[20] Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[21] Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1-18, 2022.
[22] John D Hunter. Matplotlib: A 2d graphics environment. IEEE Annals of the History of Computing, ((03):90-95 2007.
[23] Andrew Ilyas, Logan Engstrom, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Closer Look at Deep Policy Gradients. In International Conference on Learning Representations, December 2019.
[24] Eric Jones, Travis Oliphant, and Pearu Peterson. SciPy: Open source scientific tools for Python. 2014.
[25] Pranav Khanna, Guy Tennenholtz, Nadav Merlis, Shie Mannor, and Chen Tessler, Never Worse, Mostly Better: Stable Policy Improvement in Deep Reinforcement Learning, August 2022.
[26] Thomas Kluyver, Benjamin Ragan- Kelley, Fernando Pérez, Brian E Granger, Matthias Bus- sonnier, Jonathan Frederic, Kyle Kelley, Jessica B Hamrick, Jason Grout, Sylvain Corlay, et al. Jupyter Notebooks-a publishing format for reproducible computational workflows. volume 2016. 2016.
[27] Uyya Kostrikov. JAXRL: Implementations of Reinforcement Learning algorithms JAX, 10 2021.
[28] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 31, 2018.
[29] Longxin Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine Learning, 8:293-321, 1992.

---

[30] Alberto Maria Metelli, Matteo Pirotta, Daniele Calandriello, and Marcello Restelli. Safe policy iteration: A monotonically improving approximate policy iteration approach. Journal of Machine Learning Research, 22(97):1-833 2021.
[31] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 51 518(7540):529-5333 2015.
[32] Evgenii Nikishin, Pavel Izmailov, Ben Athiwaratkun, Dmitrii Podoprikhin, Timur Garipov, Pavel Shvechikov, Dmitry Vetrov, and Andrew Gordon Wilson. Improving stability in deep reinforcement learning with weight averaging. In Uncertainty in Artificial Intelligence Workshop on Uncertainty in Deep Learning, 2018
[33] Travis E Oliphant. A guide to NumPy, volume 1. Trelgol Publishing USA, 2006.
[34] Travis E Oliphant. Python for scientific computing. Computing in Science Engineering, 93):10-20, 2007.
[35] Keita Ota, Devesh K. Jha, and Asako Kanezaki. Training larger networks for deep reinforcement learning. ArXiv, abs/2102.07920 2021.
[36] The pandas development team. devaa-de Pandas, February 2020.
[37] Matteo Papini, Matteo Pirotta, and Marcello Restelli. Smoothing policies and safe policy gradients. ArXiv, abs/1905.0323) 032311 2022.
[38] Tyrrell Rockafellar, Stanislav Uryasev, et al. Optimization of conditional value-at-risk. Journal of risk, 2:21 2000.
[39] Tom Schaul, André Barreto, John Quan, and Georg Ostrovski. The phenomenon of policy churn. ArXiv, abs/2206.00733 00030,, 2022.
[40] Jürgen Schmidhuber, Jieyu Zhao, and Marco Wiering. Shifting inductive bias with success- story algorithm, adaptive levin search, and incremental seff-improvement. Machine Learning, 18:105-, 1997.
[41] John Schulman, Sergey Levine, P. Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. ArXiv, abs/1502.05477, 2015.
[42] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv, abs/ 0e//17070063477 2017.
[43] Ryan Sullivan, Jordan K. Terry, Benjamin Black, and John P. Dickerson. Cliff Diving: Explor- ing Reward Surfaces in Reinforcement Learning Environments. Proceedings of the 39th International Conference on Machine Learning, pages 20744-20776. PMLR, June 2022.
[44] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv: 1000110000000 1801000690, 2018
[45] Stefan Van Der Walt, S Chris Colbert, and Gael Varoquaux. The numpy array: a structure for efficient numerical computation. Computing in science & engineering, 13(2):22-30, 2011.
[46] Guido Van Rossum and Fred L Drake Jr. Python tutorial, volume 620. Centrum voor Wiskunde en Informatica Amsterdam, 1995.
[47] Wes McKinney. Data Structures for Statistical Computing in Python. In Stéfan van der Walt and Jarrod Millman, editors, Proceedings of the 9th Python in Science Conference, pages 56 61, 2010.
[48] Qisong Yang. Thiago D Simāo, Simon H Tindemans, and Matthijs TJ Spaan. Wesac: Worst-case soft actor critic for sanety-constrained reinforcement learning. In AAAI, pages 10639- 2021.
12

---

A Appendix
A.1 Hyperparameters and implementation
For our analyses, we run TD3 and SAC for one million environment steps, with the default hy- perparameters reported in Table and collected checkpoints at 10 15 10 25 x 10 35 x 10 45 x 10 55 x 10², 65 x 10", 75 x 10", 85 x 10", 95 x 10 steps for 20 different seeds. We run 20 seeds of PPO for 60 million environment steps with the hyperparameters in Table [ to obtain comparable level of performance. being that PPO is notoriously less sample-efficient than the two other algorithms [19]. We use equally-spaced checkpoints, at approximately 6 x 10 12 x 10 18 x 10S, 24 X 10º, 30 x 10S, 36 x 10º, 42 x 10s 48 x 10º, 54 x 10S, 60 x 10 environment steps.
We acknowledge the Python community 34 46 for developing the core set of tools that enabled this work, including JAX [4] 8), Jupyter (26]. Matplotlib [22], numpy [33 45], pandas [36 47], and SciPy 24]. Our implementations of TD3 and SAC are based on jaxrl 27 and implementation
of PPO is based on CleanRL. (21).
Our code is available at http::////ttttloe////aaaaaaaaaaa
A.2 Details of return landscape visualization
To generate the return landscapes in Figure [ we start from a given policy of parameters O, (produced, in that case, by SAC), and update it two times independently R(0).0 0 = (6' 0.) + 3("" 0,) + ou a E [-3.3], directions into other directions to explore the neighborhood. The resulting directions equivalent to the ones that would be obtained by oversampling or undersampling some elements in a larger batch composed of elements from both the two sampled updates.
obtain O' and 0" Then, plot 3], E [-3,3] 3). combining the two gradient
A.3 Interpolation between policies
We now specify the details of the experimental protocol used for Figure 5 and report additional results. In Figure 9 we report results of interpolation for the other two Brax environments, ant and half ceeethhe,,, missing from Figure 5 for space reasons. We generate the checkpoints for these plots to be policies with similar level of return but different standard deviation of the post-update return distribution. A list of the checkpoints and seeds from TD3 used for producing these plots is shown provides explicit visual cues on the
in Table While the view provided in Figure Jnd and Figure 9 variability of the neighborhood, it does not leverage directly the post-update return distribution to characterize the neighborhood of the policies obtained by interpolation. To provide a more detailed view, we estimate the post-update return distribution of an unstructured update, by sampling, given a policy of parameters 0, 1000 perturbations from an isotropic Gaussian distribution (O, 0.0003) and evaluating the resulting policies. Figure g shows the mcan and the standard deviation (as an error band) of the post-update return distribution of the interpolated policies from the same runs shown in Table 2 This alternate view confirms the results: linearly interpolating between policies of similar mean for the post-update return distribution produces paths of of approximately constant (or smoothly mncrasing/ddcrraainng) mean post-update return, even when the standard deviation of their post-update return distributions is different.
A.4 Instabilities states other than the starting one
In our analyses, we only investigated the return landscape by computing the return on a fixed initial state. Even if the characteristics of the return from other states naturally propagate back to the initial state, it natural to wonder whether the failures analyzed in our paper also happen naturally in other states. To investigate this question, we compute the LTP of the post-update return distribution for 3 runs of TD3, when starting from one of the states collected by the algorithm during training. In Figure T0| for the ant and walker2d tasks, we show for each of these states, identified by their collection time (index in the replay buffer) the LTP averaged over all the checkpoints collected during training. The results indicate that, regardless of some variation across seeds, there appears not to be any special pattern in the presence of failures over states grouped by time, and that the tail of the post-update distributions is long even when measured from states other the initial one.
13

---

loques
willa2dd
halidronah
3.00
105
Figure 8: Mean and standard deviation, displayed as error band, of the (noise-basedd post-update return distributions for the four different Brax environments. The mean of the post-update return distribution remains approximately constant when linearly interpolating between checkpoints of similar level of return.
mat
halfchertah
Same Run
Differet Rum
Some Run
ooffernt Rom
2000
10000
2508
2000
2000
2000
1000
Figure 9: Results on linear interpolation among policies in the ant and halfcheetah.
walker22
15%
13%
10%
8%
MTP
59
2%
0%
0.50
0.75
1.00
0.75
1.00
Collection Time
x10"
Collection Time
x10f
seee 0
sxed
seed 2
Figure 10: Left-tail probability computed on different states, from replay buffer for seeds of TD3, averaged across all checkpoints.
14

---

Table 1: Hyperparameters for TD3, SAC and PPO.
TD3
Parameter
Setting
SAC
Discount factor
0.99
Parameter
Setting
Minibatch size
256
Discount factor
0.99
Optimizer (all)
Adam
Minibatch size
256
Optimizer (all): learning rate
0.0003
Optimizer (all)
Adam
Optimizer (all): 31
0.9
Optimizer (all): learning rate
0.0003
Optimizer (all): 32
0.999
Optimizer (all): 91
0.9
Optimizer (all):
0.00015
Optimizer (all): 32
0.999
Networks (all): activation
ReLU
Optimizer (all):
0.00015
Networks (all): n. hidden layers
2
Networks (all): activation
ReLU
Networks (all): hidden units
256
Networks (all): n. hidden layers
2
Replay Buffer Size
106
Networks (all): hidden units
256
Updates per step
1
Initial Temperature
1
Target update period
Replay Buffer Size Updates per step
10
T (EMA coefficient)
0.995
1
Exploration noise
0.1
Target update period
Actor Delay
2
T (EMA coefficient)
0.995
Policy Noise
0.2
Noise Clip
0.5
PPO
Parameter
Setting
Discount factor
0.99
Optimizer (all)
Adam
Optimizer (all): learning rate
0.0026
Optimizer (all): B1
0.9
Optimizer (all): B2
0.999
Optimizer (all):
0.00001
Networks (all): activation
ReI.U
Networks (all): n. hidden layers
2
Networks (all): hidden units
256
Parallel Envs
2048
Steps to update
16
GAE A
0.95
Num. Updates
4
Max Grad. Norm
A.5 Scatter plots of statisties of post-update return distribution
We show similar "maps" of the landscape Figure 2 for all of the Brax environments
study. Results are shown Figure [1 Figure 12 and Figure 13] We also include similar results for and humanoid-run 17 and [8] We believe that this type of scatter plot can provide a different perspective, compared the one only focused training curves, to study the interaction between a particular algorithm and an environment.
the quadruped-walk, quadruped-rum humanoid-stande humanoid-wall
tasks from DeepMind Control Suite [44] in Figures [4] 15
16
A.6 Discrete control
Despite the fact that the main focus of our paper is on continuous control, we also run small-scale experiments to understand if the tools we introduced generalize discrete-action environments. In particular, we leverage four games from the standard ALE benchmark: Qbert, MsPacman, Breakout, and Seaquest [7]. We run PPO on these environments for runs and 10 million steps each, and measure post-update return distribution statisties using the same protocol employed in the rest of
15

---

Table 2: List of checkpoints used for Figure Figure
and Figure
Same Run
Diff. Run
Seed
Ckpt
Ckpt 2
Seed
Ckpt
Seed
Ckpt
ant
ant
550000
150000
14
850000
750000
850000
650000
11
750000
550000
11
650000
250000
2
650000
4
350000
halfcheetal
half cheetah
3
250000
950000
10
950000
450000
6
250000
350000
7
750000
16
750000
13
250000
550000
250000
350000
hopper
hopper
2
450000
250000
7
350000
4
450000
15
550000
150000
250000
150000
17
950000
750000
550000
250000
walker2d
walker2d
750000
250000
14
350000
17
450000
950000
850000
550000
950000
550000
450000
17
350000
250000
2013
SAC
8000
0000
6000
Mesn
8000
4000
1000
2000
2000
2000
0.01% 0.1%
Standard Deviation
Starrutees
Let: Teil Probability
Figure 11: scatter plot showing mean return and standard deviation, skewness or left-tail probability of the post-update return distribution of policies produced by three popular deep RL algorithms the half cheetah Brax task.
3000
3000
3000
BAC
2500
2500
2500
2000
2000
2000
Moo
1500
1500
1000
1000
1000
0
500
1000
-30
0
50
0%
0.01%
0.1%
155
10%
Standard Devistion
Shermmnne
Figure 12: A scatter plot showing return and standard deviation, skewness left-tail probability of the post-update return distribution of policies produced by three popular deep RL algorithms on the hopper Brax task.
16

---

6000
4000
1000
3000
3000
3000
2000
2000
2000
1000
1000
1000
0.01%
0.1%
10%
Standadd Deviation
Lef: Tail Probability
Figure 13: scatter plot showing mean return and standard deviation, skewness left-tail probability of the post-update return distribution of policies produced by three popular deep RL algorithms on walker2d 2d Brax task.
1000
800
800
600
600
600
Mean
400
100
100
200
200
100
400
0%
0.156
19
10%
Standard Deviation
Slomrares
Lef: Teil Probability
Figure 14: scatter plot showing mean return and standard deviation, skewness left-tail probability of the post-update return distribution of policies produced by two popular deep RL algorithms on the quadruped-wee walk DeepMind Control task.
800
800
800
800
600
000
Menn
200
200
200
50
150
200
-30
-00
10
0%
0.11%
1%
10%
Probolility
Figure 15: scatter plot showing mean return and standard deviation, skewness left-tail probability of the upst-e return distribution of policies produced by two popular deep RL algorithms on the quadruped -rrruuupd-uun run DeepMind Control task.
our paper: we evaluate policies evenly-spaced across training, and perform 1000 independent updates to each policy. We find that post-update return distributions computed for these environments still exhibit remarkable degree of variation, as captured by their standard deviation (see Figure[] At the same time, the shape of the resulting distributions can be quite different compared to the ones observed robotic locomotion tasks, and, thus, it is not necessarily described in rich way by metrics such the LTP (see Figure 20).

---

800
800
500
600
600
Menu
400
400
400
Standard Deviation
Stawrees
Lef: Probability
Figure 16: scatter plot showing mean return and standard deviation, skewness left-tail probability of the post-update return distribution of policies produced by two popular deep RL algorithms on the humanoid d-stand DeepMind Control task.
RAC
600
Mest
50 100
150
20
06%
0.1%
156
10%
Standard Deviation
Sammuss
Lef: Tall Probobility
Figure 17: A scatter plot showing mean return and standard deviation, skewness or left-tail probability of the post-update return distribution of policies produced by two popular deep RI. algorithms on the humanoid-see walk DeepMind Control task.
175
175
175
VDs SAC
150
150
130
Moan
100
100
100
75
75
75
0.156
1%
Standard Deviation
Siomenees
Lef: Thll Probability
Figure 18: scatter plot showing mean return and standard deviation, skewness or left-tail probability of the upst-e return distribution of policies produced by two popular deep RL algorithms on the humanoid d- run DeepMind Control task.
We also run an interpolation experiment on the ALE investigate whether the connectivity phe- nomenon studied in Section 4.1 gennraalizes to discrete-contre environments. We produce quantitative results on the phenomenon with the following procedure. For each one of the four games, we sample a set of at least 20 pairs of policies from same runs and 20 pairs of policies from different runs of the algorithm. Then, we linearly interpolate between policies the pairs, producing intermediate policies, and randomly perturb them using Gaussian noise with standard deviation 0.0003 to obtain an estimate of the mean of their (random) post-update return distribution. Then, for each pair of policy, we measure how frequently the return collapses in between the two extremes, by counting
18

---

Different Run
Same Rum
0.16
0.24
0.32
0.40
Below-Threshee Threshold Proportion
Figure 19: Proportion return collapses when interpolating between randomly-sampled policies produced by either the same different runs in the ALE (QBert, Breakout, Ms PPacman, Seaquest). Far fewer return collapses are observed when interpolating between policies produced by the same run. Results are aggregated over environments with 95% bootstrapped C.I..
rums
Qhert-v5
MssPacman-v55
Seaquest-xi
0.006
0,0100
0.06
0.006
0.0076
Denetty
0.004
0.0050
0.002
0.0025
0.002
0.02
0.000
0.0000
0.000
0.00
1000
0
250
500
1000
0
200
Return
Remsrr
Retur
Return
Figure 20: Histograms the post-update return distribution for policies obtained by PPO on Atari environments. These distributions were selected for illustrative purposes to demonstrate that 1) returns can vary significantly after single policy update (see also Figure 2T) and 2) the distributions obtained can exhibit varied profiles.
how many times it becomes less than 10% of the minimum return of the two original policies. We then average this helow-Threshold Proportion across pairs, and across environments using rliable M Figure T9rree9sssww that the phenomenon, properly quantified, is still present when using very different class of environments (discrete-action, (ame-based))
A.7 Behavior eloning
While the results in the main paper demonstrate that policies learned by commonly-used RL algo- rithms are sensitive to updates, we were curious to understand whether behavior cloning produces policies which occupy less noisy neighborhoods of the return landscape. To do we conducted a set of additional experiments on Brax environments. The protocol was as follows: For each environment, consider 10 independent training throughout the run. For each of these policies, we train a new agent using behavior cloning on the data logged up until the collection time of the teacher policy, replacing the actions in the dataset with the actions the teacher policy, for million gradient steps. We log 10 policies throughout each training of behavior cloning. To compute the post-update return distribution for the policies obtained by behavior cloning, we used one additional gradient step on the MSE-based BC objective, and 1000 samples.
of TD3, and 5 policies distributed evenly
In Figure 22 (left), we compare statistics of the post-update return distributions for all pairs of BCC is obstained by behavior cloning n. We compute the policies (Tt) ... TT( D3) where policy n...
Pearson correlation coefficient statistics of the post-update return distributions of these policies: between the mean of each pair, and between the LTPs of each pair. We find that the means are highly correlated and that the learned BC policies are comparable in performance to their teacher policy. We additionally show that correlation in the LTP is much more variable across environments in general, cloning a policy of high or low LTP does not always lead to a cloned policy of the same LTP.
But does training policies by BC produce policies occupying less noisy neighborhoods of the return landscape, which would have correspondingly lower LTP overall? Figure 22 (right), we show the mean I.PP across policies and environments, along with its 95% bootstrapped confidence interval following the recommendations from rliable (I]. Our results demonstrate that BC does not produce fundamentally more stable policies, measured by the LTP.
19

---

14
1200
30
1000
3.0
4.3
233
Figure 21: Scatter plots of the mean and standard deviation of the post-update return distribution of policies learned by PPO across four Atari environments. In the majority of cases, the post- cost-u return distribution exhibits significant variability relative its mean, indicating that learned policies in Atari are also sensitive to single updates.
corTmean
COTTLTP
TD3
ant halfcheetah
0,994
0.763
BC
0,995
0,976
0.0%
5.0% LTP (Mean)
10.0%
walker2d 0.973
0.218
hopper 0.753
0.449
Figure 22: Comparing pairs of policies obtained by behavior cloning given TD3 policy. Left: BC learns policies which are well-correlated in mean return with the cloned policy, measured by Pearson's correlation coefficient. The correlation in LTP more variable across environments, indicating that cloning given policy does not always produce a new policy of the same LTP. Right: Average LTP of policies learned by TD3 and BC, aggregated across environments. Overall, BC learns policies of similar LTP to policies learned by TD3, Results are aggregated over environments with 95% bootstrapped confidence interval.
A.8
Complete results on comparing successful and failing trajectories
In Figure 23
we include similar results to Figure 3 for all the Brax environments we study.
halfchoctah
wallsordd
hopper
5000
6000
3000
1500
Fallin
3750
Rgn Palling
Failing
2000
Failing
2500
2000
1000
600
1250
1250 2500 3750 5000 Rgg. Suconsful
0
2000
4000
6000
0
1000
2000
3000
0
500
1000
1500
Rig. Stanenseffll
Rgy.
Sureessfill
Rgg. Sucoossful
Figure 23: visualization of how failures occur on four Brax tasks. The plots show the simultaneous evolution of returns for 10 trajectory pairs, of one successful and one failing drawn from the same post-update policy/trajectory distribution. We observe that these policies from the same neighborhood behave similarly (diagonal segments of the curve) until the failing policy makes sudden misstep and collects low rewards (horizontal segments). At divergence, we observe that the failing policy terminate, collect reward, even collect negative reward, yet it sometimes recovers and goes back to matching the performance of the successful policy.
A.9 Justification of rejection procedure
In Algorithm [ we proposed a procedure that rejects gradient updates based on estimates of the CVaR of the resulting policies' post-update returns. Since our goal with this procedure ultimately to decrease the LTP of an existing policy, one might rightfully question why CVaR was our heuristic of choice. Our reasoning for this choice was simply that the LTP can minimized by arbirarily poor policies: that is, policies that only achieve "low enough" returns will have very low LTP, because they cannot substantially fail relative to their existing performance. Such an issue can be circumvented by comparing the CVaR of the post-update returns: among all policies that have the same left-tail
20

---

ant
halfcheetah
Without Rejection
With Rejection
Without Rejoction
With Rejoction
6000
Oa
dgo
6000
4000
Mea
4000
2000
2000
1%
0.1%%
1%
10%
0%
0.1%
10%
0%
0.1%%
10%
LTP
hopper
walker2d
Without Rejection
With Rejection
Without Rejection
With Rejection
2000
3000
60
Xian
2000
1000
1000
15
10%
0.15%
1% 10%
0%
0.1%
1%
10%
0%%
0.1%%
1%
10%
LTP
10P
LTP
Figure 24: Experimental results for Algorithm on 4 Brax domains. Each empty circle denotes the starting policy, while the filled circle at the end of the arrow shows the policy obtained after application of Algorithm T X's show other policies obtained by TD3, for context.
probability, the CVaR is still able to order the policies by the quality of the returns in the left tail. To further justify the use of CVaR as a rejection heuristic, we visualized how different CVaR heuristics order policies with respect their post-update return distributions. Figure 25 illustrates how the V-C ranks policies. Note that a-CVaR in this instance is effectively measuring the mean among the post-update returns below the a-quantile of the post-update return distribution, so O-CVaR would measure the worst-case post-update return and 1.0-CVaR is equivalent to the mean of the post-update returns.
As expected, we see that ranking by the mean relatively agnostic to the LTP of the policies, but for < ranking by CVaR is effectively trading off larger mean post-update returns for lower LTP. Notably, the gradient of colors (rankings) in the 0.01-CVaR column is distinctly "top-left to bottom-right", indicating that only policies with both high returns and low LTP are ranked favorably. The 0.1 column is fairly similar, but the gradient is more vertical that is. it is more forgiving with respect to LTP than the 0.01-CVaR heuristic, but it still clearly prefers policies with lower LTP among those with similar levels of return. Finally. the gradient in the rightmost column is exactly vertical, meaning that the ranking agnostic LTP.
Consequently, we argue that CVaR is reasonable heuristic for identifying robust policies without neglecting their performance level. In experiments, found that the 0. 1.V-R heuristic for rejecting gradient updates struck the best balance between robustness and performance.
A.10 Additional post-update-CVil CVaR rejection results
In Figure 24 we include the results from our experiments with Algorithm I in four Brax domains for policies found by TD3.
A.11 Visualization of behaviors
We uploaded some visualization of the behaviors on website (https://siter) google. com/ landscapesshhmme) http://rrrruuu--aaaaassseee/eeae policies of similar mean for the post-update return different standard deviation exhibit different behaviors.
Biee/onnreetus
21

---

ant
Post-Update
by Upat-U 0.1-CVaRR
by Pass-Update Return
4000
4000
4000
Mean
2000
2000
2000
10
10
2
10
10
-
10
10
10
LTP
LTP
halfcheetah
Ronk Updtte Retarn
Rank by Post-Update Return 0.1-CVaR
by Poes-Update Return
6000
6000
6000
Menu
4000
4000
4000
2000
2000
2000
10-3
10-2
10-1
10-3
10 LTP
10
10-3
10
10
LTP
hopper
Update
by Post-Update Return 0.1-CValt
Rank Rpss-Update
2000
2000
2000
Mean
1000
1000
1000
10-3
10-2
10-1
10-3
10-2
10-1
10-3
10-2
10-1
LTP
LTP
walker2d
Post- Update Return by UpsttUpdate Return 0.1-CVaR
Puea-Update
3000
3000
3000
Meaa
2000
2000
2000
1000
1000
1000
10-3
10-2 LTP
10
10-3
10-2
10-1
10-3
10-2
10"
LTP
LTP
Figure 25; Rankings policies by the c-CVaR of their post-update returns, Points with darker colors are ranked higher, and stars denote the highest ranked policy.
A.12 Evidence of multi-scale structure
To investigate whether the shape of a neighborhood heavily depends on the size of the steps used for
exploring it, we "zoom-in" on the landscape plot from Figure[]b using the same granularity but on smaller window. We apply this process twice, with the results in Figure 27 giving evidence for scale-independence.
22

---

Rejection rates by environment
80%
60%
Rejection
40%
20%
0%
ant
walker2d
hopper
halfcheetah
Figure Frequency of rejections during execution of Algorithm 1 across environments. Bars depict the mean across starting policies and runs, and the error bars represent standard error.
72500
Rosiin
1000
1000
1000
11000 Updatee Ditrction
Update Direcion
-0g 20 Pule
Update Ditection 2
321
Aa al
Update Direstion
Update Direstion
Update Direction
Figure 27: Multiscale version of the plot in Figure Note the similarity between the original and the more granular versions, giving evidence the independence of the general shape of a neighborhood from the resolution at which it is visualized.
23


Reviews:
[{"review_id": "j2AeTGD9Bm", "reviewer": "Reviewer_ZEGo", "summary": "This work proposes a new perspective for investigating the stability of RL algorithms -- the return distribution of post-update policies. A simple rejection technique is developed correspondingly to increase the stability in that sense.", "questions": "see above", "limitations": "see above", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The paper is very well written and easy to follow. This new perspective of return distribution of post-update policies appears novel and makes much sense. I can imagine many followup directions of this work. The rejection technique is neat while effective. ", "weaknesses": "There is indeed a few weaknesses in the treatment of the problem.\n1. Why do the authors limit the investigation to deterministic transitions and continuous actions? In my understanding the concept the authors propose apply to stochastic environments and discrete actions as well.\n2. Maybe it's better to rename the work as \"Policy Optimization in a Noisy Neighborhood: On Return Landscapes in MuJoCo\" since the work in its current form does not have any other domains. In other words, it is interesting to see whether the observation is general across various domains, especially the stability of interpolation of policies inside a single run. \n\nThat being said, I believe the novelty of this work is significant and am happy to increase my score based on the authors' response to my concern.  ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback!\n\n> In my understanding the concept the authors propose apply to stochastic environments and discrete actions as well.\n\nWe thank the reviewer for the suggestion. We have run two experiments (Brax with stochastic policies and four games from the ALE, see General Response) to confirm that this is the case.\n\nFor the stochastic setting, we generalize our experimental protocol for the computation of the post-update return distribution to the case in which the policy is non-deterministic. For each policy checkpoint from a TD3 training run, we produce 100 policies with the TD3 update, and then evaluate the stochasticity policy obtained with the same random perturbation scale used during training 10 times and compute the resulting distribution statistics. We find that, for a given policy, this alternate post-update return distribution yields very similar results to the one based on its deterministic version (see Figure 2 in the pdf). In the paper, we focus on deterministic environments in order to understand the relationship between policy parameters and return without the confounding effect of environmental stochasticity.\n\n\n> Maybe it's better to rename the work as \"Policy Optimization in a Noisy Neighborhood: On Return Landscapes in MuJoCo\" since the work in its current form does not have any other domains.\n\nWe appreciate the point from the reviewer. Including the results in the Appendix and the new results on the discrete-action ALE, we have now a total of three different simulators (Brax, MuJoCo via DMC, and Atari). We believe this positions the method of analysis to be general, despite, for computational reasons, we have run most of our experiments on the efficient Brax simulator.\n\n> In other words, it is interesting to see whether the observation is general across various domains, especially the stability of interpolation of policies inside a single run.\n\nWe thank the reviewer for the suggestion. To test whether the phenomenon is present also on other domains, we run an interpolation experiment on the ALE under the same setting based on PPO described in the general response. As per suggestion of other reviewers, we produce quantitative results on the phenomenon with the following procedure. For each one of the four games, we sample a set of at least 20 pairs of policies from same runs and 20 pairs of policies from different runs of the algorithm. Then, we linearly interpolate between policies in the pairs, producing 50 intermediate policies, and randomly perturb them using Gaussian noise with standard deviation $0.0003$ to obtain an estimate of the mean of their (random) post-update return distribution. Then, for each pair of policy, we measure how frequently the return collapses in between the two extremes, by counting how many times it becomes less than 10\\% of the minimum return of the two original policies. We then average this _Below-Threshold Proportion_ across pairs, and across environments using rliable (Agarwal et al., 2021). Figure 1 (b) shows that the phenomenon, properly quantified, is still present when using a very different class of environments (discrete-action, game-based)."}, {"review_id": "ytcVryV0uq", "reviewer": "Reviewer_ErLG", "summary": "This paper looks at visualizing the return landscape in typical continuous control environments of deep RL. The authors propose a statistic over the returns profile, namely the left tail probability, and try to present this as a suitable measure of policy quality in a series of experiments and visualizations. Finally, they propose a high-level rejection sampling process for the policy updates using the proposed statistic and argue that it can lead to similar-mean but less-variable return profiles in the trained policies.", "questions": "I would like the authors to address my concerns in the weakness section regarding the premise of the paper and the made arguments.", "limitations": "Given the the recent reproducibility issues with deep RL, papers in this are must be held to a higher standard. In particular, for this paper: \n  1. Many depicted training curves are based upon single trainings.\n  2. 20 seeds were used for running the experiments. I encourage the authors to run the experiments for at least a 100 seeds.\n  3. Most figures lack any information about the uncertainty of the statistics. Finding ways to incorporate confidence intervals in the plots is important for this paper.\n  4. The number of environments considered in the paper are limited. Hopper, Walker, Half-Cheetah, and Ant were shown in the main paper, while there are many more environments in the standard gym benchmarks.\n  5. The quality of the environments in the paper could also be improved. In particular, while standard gym benchmarks are a good start, they do not depict realistic robotic artifacts. There are many realistically modeled robotic environments, such as the ones used in [4] and [5], that can be better representatives for validation of findings.\n  6. While the paper is specifically targeting continuous control, the authors never discuss the physical aspects of each environment and do not make physical variations to the underlying physical models.\n  7. The code is not shared open-source in the paper. While this is not a requirement for the conference, it is certainly of great important to the RL community. I encourage the authors to release an open-source link to the code in the main paper.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The idea of looking at return surfaces is not original (e.g., see [1], [2], and [3] among many others). However, trying to propose a remedy based upon such visualizations has been less common. In terms of quality, the paper tries to narrow down its focus to control and robotics applications, which is admirable. As for the clarity of the paper, too much theoretical complications did not hinder the point of the paper. Unfortunately, the results are less significant than expected, but the studied topic (i.e., better understanding of the return landscape for deep RL in control environments) is certainly of great importance and has been under-looked for too long.\n\n# References\n[1] Sullivan, R., Terry, J. K., Black, B., & Dickerson, J. P. Cliff Diving: Exploring Reward Surfaces in Reinforcement Learning Environments. ICML, 2022.\n\n[2] Bekci, Recep Yusuf, and Mehmet Gümüş. \"Visualizing the Loss Landscape of Actor Critic Methods with Applications in Inventory Optimization.\" arXiv preprint arXiv:2009.02391 (2020).\n\n[3] Ilyas, A., Engstrom, L., Santurkar, S., Tsipras, D., Janoos, F., Rudolph, L., & Madry, A. (2018). A closer look at deep policy gradients. ICLR 2020.\n\n[4] Hwangbo, Jemin, et al. \"Learning agile and dynamic motor skills for legged robots.\" Science Robotics 4.26 (2019): eaau5872.\n\n[5] Saleh, E., Ghaffari, S., Bretl, T., & West, M. (2022). Truly Deterministic Policy Optimization. Advances in Neural Information Processing Systems, 35, 8469-8482.", "weaknesses": "## Presentation and Writing\nFirst and for most, the paper lacks a clear set of contributions. I encourage the authors to dedicate the last paragraph of their introduction to list 3 or 4 main contributions, and commit to delivering them in the entire paper. This is a standard practice, and lacking this makes it extremely difficult to evaluate the merit of the work.\n\nSecond of all, the paper lacks a main message and a proper effort to convince the reader about the validity and the importance of it. As is, the manuscript looks more like a technical report draft rather than a conference paper. In particular, note that most NeurIPS papers focus on driving the main message in the initial sections, and avoid detailing the methods/data until the experiments section. However, this paper lacks such a structure, and tries to narrate a story by jumping from one experiment to another, while losing a cohesive context and spending a lot of space on detailing experimental details.\n\n## Rigor\nThe paper is not driven by any amount or kind of theory. Instead, the authors choose or abuse vague terms and intuitions to describe a series of hypotheses. For example, the notion of \"stability\" of a policy gets thrown around a lot in the paper (e.g., Line 246) without any mathematical standing or definition. This is despite the fact that the stability of a policy is a well-defined topic of research, and has many measures such as the Lyaponuv stability criterion. I doubt the authors were targeting this original notion of stability, and this is just one example of a technical term being abused.\n\nMany other vague terms are consistently used throughout the paper, including, but not limited to, (1) the \"quality\" of a policy, (2) \"noisy\" neighborhoods, (3) \"failure\" of a policy or a trajectory, (4) \"safer\" behavior, and so on.\n\n## Premise\n  1. Most deep Policy Gradient methods (e.g., PPO, TRPO, DDPG, TD3, SAC, etc.) optimize the mean of the discounted return in the MDP framework. Other statistics of the return distribution (e.g., the variance, the mode, left tail probability, etc.) are irrelevant to the optimization objective. When evaluating methods and policies, it is standard that higher mean payoffs are preferable even if the return distribution has higher variance.\n\n  If a particular application requires consistent returns from a policy, the user either has to (1) engineer the reward so that such variations lead to worse mean returns, or (2) use PG methods that optimize a different objective (e.g., the worst-case performance with min-max optimization methods). Both approaches are commonly used in control and robotics applications.\n\n  Of course, consistent policy performance is preferable but it shouldn't be confused with the main performance metric. The entire paper is written on the premise that variations in the return are inherently undesirable, and I don't necessarily agree with that. In my experience, Deep RL algorithms such as PPO, SAC, and TD3 gravitate towards such \"noisy\" regions at the end of the training because these regions can potentially yield higher returns than the stable areas, even if the return improvement is small.\n  \n  When evaluating such PG methods, only the mean return should be considered. The only way the paper's findings could be relevant is if high-variations in the return profile were shown to correlate with lower mean-returns at the end of training. I'm afraid the paper is not showing that.\n\n  2. Deep PG methods do not directly optimize deterministic policies. Each algorithm, even the deterministic ones such as DDPG and TD3, have an exploration parameter $\\sigma$. These parameters cause a filtering/convolutional effect on the return landscape, and smoothen the effective optimization objective. The paper mainly focuses on oscillations in the deterministic landscape, whereas the effectively optimized landscape can be much smoother and I'm not sure the same findings are applicable to the filtered landscape.\n\n## Arguments\n\nThe experiments and figures in the paper aren't always convincing or helpful. In particular, \n\n  1. Figure 5 doesn't prove a meaningful hypothesis. Such parameter interpolation behavior may not even be specific to RL problems, and \"non-noisy\" problems can also exhibit the same behavior. In other words, interpolating parameters within the same run in a deep classification task can lead to reasonably-performing parameters while inter-run interpolations wouldn't perform well. I'm not sure how this observation is a solid evidence for the existence of special patterns or connections, even if there were many gradient steps in between interpolated parameters.\n\n  2. Figure 1 consumes a lot of space, but carries little information. \n\n  3. Also, the paper spends a lot of time speculating. For instance, Lines 49-52, Lines 236-243, and Lines 202-207 are a few examples. Line 245 speculates about the existence of \"paths\" (I'm not even sure what it means), and such claims about high-dimensional optimization problems cannot be easily proven by the experiments conducted in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback!\n\n> The idea of looking at return surfaces is not original (e.g., see [1], [2], and [3] among many others).\n\nWhile the return landscape has been studied in other works, our distributional perspective on it has no precedent. It is also important to distinguish between works which study a loss landscape (e.g. [2]), and a landscape of returns from the environment, which we study.\n\nOf the works in the latter category, [1] emphasizes visualizations of the landscape and [3] studies its relationship with the optimization landscape. Neither of these works show how distributional properties of the landscape can characterize policies, nor do they demonstrate macro-scale structure of the landscape when interpolating between policies.\n\n> I encourage the authors to dedicate the last paragraph of their introduction to list 3 or 4 main contributions, and commit to delivering them in the entire paper.\n\nOur introduction already explicitly states our contributions, in the form of three paragraphs with bold title, each one corresponding to a specific contribution.\n\n>  the manuscript looks more like a technical report draft rather than a conference paper. \n\n>  most NeurIPS papers focus on driving the main message in the initial sections, and avoid detailing the methods/data until the experiments section.\n\nWe do not present any experimental detail in the introduction, and simply enumerate and explain our findings. Note that the overall presentation strategy has been praised by most of the other reviewers.\n\n\n> vague terms are consistently used throughout the paper, including, but not limited to, (1) the \"quality\" of a policy, (2) \"noisy\" neighborhoods, (3) \"failure\" of a policy or a trajectory, (4) \"safer\" behavior\n\nThese concepts are defined in the paper or in related work. Notions such as \"stability\" and \"failure\" are entailed by the post-update return distributions and the LTP. \"Noisy neighborhood\" is a name defined in line 31. \"Safer\" is used in the sense reported in the related work. Better \"quality\" means having lower level of LTP/std for a given level of mean return of a post-update return distribution.\n\n> variations in the return are inherently undesirable, and I don't necessarily agree with that. In my experience, Deep RL algorithms such as PPO, SAC, and TD3 gravitate towards such \"noisy\" regions\n\nAn important point behind our analysis based on post-update return distributions is showing that a policy optimization algorithm produces policies with the same level of return but very different levels of variability under further updates. Our experiments have the goal of quantifying and understanding this phenomenon, which is generally undesirable when deploying a policy, but also compelling from a purely scientific perspective.\n\n> The paper mainly focuses on oscillations in the deterministic landscape, whereas the effectively optimized landscape can be much smoother\n\nOur paper is about the return landscape resulting from evaluating a policy in the environment, regardless of how smooth the optimization objective used by the algorithm that produced this policy was.\n\n> interpolating parameters within the same run in a deep classification task can lead to reasonably-performing parameters\n\nThe reviewer is correct that a similar phenomenon exists in deep classification (see citations). However, the RL setting is different: the optimization objective is non-stationary and the evaluation metric (the return vs the loss) depends on an environment and multiple forward passes from the neural network. We believe this makes the existence in RL of a phenomenon akin to mode connectivity far from certain.\n\n> Line 245 speculates about the existence of \"paths\" (I'm not even sure what it means)\n\nA path in this context simply refers to a trajectory in parameter space. Thus, a linear path is one that traverses through parameter space along a line, which is given by the interpolation between two parameters.\n\n> Many depicted training curves are based upon single trainings.\n\nPlease note that our paper does not report any single training curve, since they are not the object of study of our work.\n\n> 20 seeds were used for running the experiments. I encourage the authors to run the experiments for at least a 100 seeds.\n\nWe disagree that additional seeds are required to support the results presented in the paper. Using 100 seeds instead of 20 would only cause the scatter plot in Figure 2 to be populated by more points, and not increase the significance on any of the new plots in the pdf, greatly increasing the computational cost of our experiments without any clear benefit.\n\n> Finding ways to incorporate confidence intervals in the plots is important for this paper.\n\nWe incorporated error bars in experiments in the pdf using bootstrapped CIs (as in Agarwal et al., 2021). First, we compute the uncertainty of the statistics of the post-update return distribution (see Figure 6), showing it is very small; second, we quantify the presence of return drops when interpolating across pairs of policies (see Figure 1 and response to Voa2); third, we show the improvement provided by Algorithm 1 across environments is statistically significant (Figure 8); fourth, we measured the number of rejections from Algorithm 1 (see Figure 7).\n\n> The number of environments considered in the paper are limited.\n\nSee Appendix and general response.\n\n> while standard gym benchmarks are a good start, they do not depict realistic robotic artifacts.\n\n> the authors never discuss the physical aspects of each environment\n\nOur paper is mainly directed at the deep RL community. Since simple robotic locomotion tasks or Atari games are standard in this community, we believe the extension of our analysis to more complex robotic artifacts is an interesting avenue for future research, but outside the scope of our paper.\n\n> I encourage the authors to release an open-source link to the code\n\nWe will release code once the paper is published."}, {"review_id": "R148OB2tID", "reviewer": "Reviewer_QuAp", "summary": "In this study, the authors explore the underlying causes of the instability observed in the performance of deep RL policies. Their investigation revolves around examining the relationship between a policy and its corresponding return, with a focus on analyzing how minor variations in policy parameters impact the overall return. The authors posit that regions where even slight perturbations in policy parameters result in significant changes in return are susceptible to failures. To address this issue, they introduce an algorithm designed to navigate away from these volatile neighborhoods, thereby enhancing the robustness of the policy. Lastly, the authors provide a set of experimental results to demonstrate their findings and validate the efficacy of their optimization algorithm through another set of experimental results.", "questions": "1. What will the results look like among policies trained with different algorithms on the same task?\n2. I have seen recent works studying how the mapping between the state-space and the policy look like and how this landscape evolves during training. How do those findings relate to the findings in this paper? (For reference, look at “Understanding the Evolution of Linear Regions in Deep Reinforcement Learning” from NeurIPS’22)\n3. I personally find figure 2 hard to understand. But for now, I don’t have any particular suggestions to improve the readability of these.\n4. It would be invaluable if the same experiments would be repeated for policies trained with behavior cloning. To first see how policies trained with behavior cloning on some task compare to policies trained with other RL algorithms on the same task. And second, see whether different runs of behavior cloning demonstrate non-noisy return landscapes.\n", "limitations": "Please refer to the previous sections.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Overall Strengths:\n1. The organization and writing of this paper are of good quality. The authors have succeeded in presenting the content in a clear and easily understandable manner, which greatly enhances the readability of the paper. A notable aspect is the structure of the experiments sections (Section 3 and Section 4), where the authors first introduce the research questions, followed by a detailed explanation of the experimental setup, and finally present the results along with their corresponding conclusions. This approach significantly aids in the comprehension of the experiments, in such heavily experiments-based paper.\n2. I personally find the high-level idea of studying why the performance of RL policies so unstable well-worth pursuing.\n", "weaknesses": "1. This paper would benefit from providing additional explanation and motivation regarding the significance of investigating the mapping between policy parameters and return. A more thorough exploration of why this particular landscape is crucial to the focus of the work is needed. Expanding on the importance of understanding this relationship would enhance the reader's understanding and emphasize the relevance of the research.\n2. The experimental setup is limited. I would personally be interested in how different algorithms compare with each other when trained on the same task as well as how a certain algorithm behaves when trained on different tasks.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback!\n\n> This paper would benefit from providing additional explanation and motivation regarding the significance of investigating the mapping between policy parameters and return. A more thorough exploration of why this particular landscape is crucial to the focus of the work is needed.\n\n\nBy investigating the return landscape with the set of tools we introduced in the paper, we can both advance our understanding of the policies traversed by policy optimization algorithms and improve their stability. Our work offers both a new perspective on previously-studied unstable learning dynamics of deep RL agents (e.g. Chan et al. 2020, Henderson et al. 2018), as well as a new way to look at the stability of a given policy in terms of its post-update return distribution, which we measure using statistics such as the left-tail probability and show to correspond to the qualitative robustness an agent's behavior. Through this landscape-oriented view, we can improve a policy along that dimension of policy quality, as we propose to do in Algorithm 1. We are happy to make revisions to the manuscript to emphasize the central role of the return landscape in the results obtained.\n\n> How do those findings relate to the findings in this paper? (For reference, look at “Understanding the Evolution of Linear Regions in Deep Reinforcement Learning” from NeurIPS’22)\n\nWe appreciate the reference. Although this work is similar in spirit to ours, in that it aims to characterize policies beyond common evaluation metrics, they study the complexity of the learned policy as a function of the input state, whereas we study the return in the environment as a function of the policy parameters. Naturally, these objects are related: we conjecture that a more complex policy may exhibit greater variability in returns under a single update to its parameters. We take this as an interesting direction for future work, and will cite this work in the final version of the paper.\n\n> I personally find figure 2 hard to understand.\n\n To clarify, Figure 2 considers several main objects. Each color in the legend corresponds to an algorithm, and each point corresponds to a policy produced by that algorithm. For each policy, we compute its post-update return distribution. On the three main scatter plots, we plot each policy according to different statistics of the distribution: skewness, standard deviation, and left-tail probability, plotted against the mean. Therefore, each policy appears once on each scatter plot. We select 6 policies of interest, indicated by stars, and plot histograms of their post-update distributions at bottom to make a correspondence between the statistics and visual appearance of the distribution. We will consider if it's possible to simplify the presentation of Figure 2 for the final paper.\n\n> What will the results look like among policies trained with different algorithms on the same task?\n\nFigure 2 depicts precisely how policies obtained by different algorithms trained on the Brax `ant` task compare to each other. Similar figures are reproduced for other environments across Brax and DeepMind Control in the appendix.\n\n> It would be invaluable if the same experiments would be repeated for policies trained with behavior cloning\n\nWe thank the reviewer for the compelling suggestion. We were especially curious to understand whether behavior cloning produces policies which occupy less noisy neighborhoods of the return landscape. To do so, we conducted a set of additional experiments on 4 Brax environments. The protocol was as follows: For each environment, we consider 10 independent training runs of TD3, and 5 policies distributed evenly throughout the run. For each of these policies, we train a new agent using behavior cloning on the data logged up until the collection time of the teacher policy, replacing the actions in the dataset with the actions of the teacher policy, for 1 million gradient steps. We log 10 policies throughout each training run of behavior cloning. To compute the post-update return distribution for the policies obtained by behavior cloning, we used one additional gradient step on the MSE-based BC objective, and 1000 samples.\n\nIn Figure 3 (left), we compare statistics of the post-update return distributions for all pairs of policies $(\\pi^{BC}_{i, j}, \\pi^{TD3}_i)$ where policy $\\pi^{BC}_{i, j}$ is obstained by behavior cloning $\\pi^{TD3}_i$. We compute the Pearson correlation coefficient of statistics of the post-update return distributions of these policies: between the mean of each pair, and between the LTPs of each pair. We find that the means are highly correlated and that the learned BC policies are comparable in performance to their teacher policy. We additionally show that correlation in the LTP is much more variable across environments -- in general, cloning a policy of high or low LTP does not always lead to a cloned policy of the same LTP.\n\nBut does training policies by BC produce policies occupying less noisy neighborhoods of the return landscape, which would have correspondingly lower LTP overall? In Figure 3 (right), we show the mean LTP across policies and environments, along with its 95% bootstrapped confidence interval following the recommendations from *rliable* (Agarwal et al, 2021). Our results demonstrate that BC does not produce fundamentally more stable policies, as measured by the LTP."}, {"review_id": "hZUBEop2f9", "reviewer": "Reviewer_Voa2", "summary": "The paper investigates the mapping from policy parameters $\\theta$ to returns $R(\\theta)$ and the sensitivity of $R(\\theta)$ to minor perturbations to $\\theta$. The authors show that popular RL algorithms tend to traverse \"noisy neighborhoods\" in policy parameter space where the return is highly varied even for small changes to $\\theta$. They find that policies with comparable average returns can have drastically different return distributions. Additionally, interpolating between policies from the same training run allows for transitioning from noisy to smooth return landscapes without degraded performance (the same is not true for policies from different runs). Finally, a practical algorithm is proposed which improves stability by rejecting gradient updates that lead to noisier return distributions.", "questions": "\"For our analysis, we use 200 policies generated by different runs of TD3. From these we select pairs of policies with different post-update return distributions, as measured by their standard deviation or left-tail probability, but similar mean\" (L214-216).\nI don't fully understand how pairs of policies were selected using these criteria. Were these handpicked? Is there a more systematic way of picking pairs of policies? Is it possible for us to consider more than (a handful+1) examples?\n\nWhat is the claimed contribution of Algorithm 1? Are the authors claiming that this update scheme should be used practically? Is this just an illustration due to other considerations (e.g. computational cost).\n\nI think you should probably mention distributional RL [1] as related work.\n\nIn Figure 3, labeling the successful/failing trajectory either in the plot or in the caption would be nice.\n\n===\nI have read the rebuttal and bumped up my score slightly in light of the additional experimental results.", "limitations": "No", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "Overall, I thought the paper was enjoyable to read and appears fairly novel (to the best of my knowledge). The figures are great and in general the story of the paper flows very well; Figure 2 is fantastic.\n\nThe landscape-inspired approach to deep RL is extremely interesting in my opinion. Finding novel ways to stabilize training by considering the local return landscape around the current set of policy parameters is highly relevant to scaling up and robustifying existing RL methods.\n\nI appreciate that the paper doesn't rely on overly technical jargon and, combined with the great illustrations, the paper is fairly easy to grasp without much math at all.", "weaknesses": "Section 4.1 makes pretty strong claims about interpolating between policies from the same run and from different runs, but it is only really substantiated by a few (six) qualitative examples. I suspect there is some way to quantify the claims in this section across a larger population of policy pairs, which I think would significantly strengthen this section.\n\nThe results in Figure 6 (and the description in Section 4.2) are a bit confusing to me. Is this just showing that the proposed algorithm succeeds in reducing LTP after 40 gradient steps? Why don't we compare the mean / variance of returns for complete training runs that do or don't use rejection? The 'x' markers show points reached by full training runs, so why can't we show that with/without the proposed approach? Or perhaps I'm misunderstanding, and Algorithm 1 is meant to by applied after a training run in order to robustify an already learned policy?\n\nI felt the experiments could be more comprehensive w.r.t environment diversity. Given that the main claims of the paper are substantiated only empirically, I think it is potentially a problem that the paper only considers a few DMC environments (ant, walker2d, halfcheetah, hopper) which all involve locomotion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback!\n\n> I suspect there is some way to quantify the claims in this section across a larger population of policy pairs, which I think would significantly strengthen this section.\n\nWe thank the reviewer for the suggestion. To quantify whether there is a statistically significant difference in the proportion of return collapses encountered when interpolating between policies, we use the following experimental design. We sample for each environment a set of 500 pairs of policies from the same runs and a set of 500 pairs of policies from different runs. Then, we linearly interpolate between policies in the pairs, producing 100 intermediate policies, and randomly perturb them using Gaussian noise with standard deviation $0.0003$ to obtain an estimate of the mean of their (random) post-update return distribution. Then, for each pair of policy, we measure how frequently the return collapses in between the two extremes, by counting how many times it becomes less than 10\\% of the minimum return of the two original policies. We then average this _Below-Threshold Proportion_ across pairs, and across environments using rliable (Agarwal et al., 2021). Figure 1 (a) shows a proper quantification of the phenomenon: on Brax, there is on average almost no drop in return when interpolating among policies from the same run. Additionally, similar results on Atari games in Figure 1 (b) (see response to ZEGo) show that the phenomenon also exists in other domains.\n\n\n> I felt the experiments could be more comprehensive w.r.t environment diversity.\n\nWe provide additional results in the pdf using a set of games from the ALE (see General Response). We also have additional results on DeepMind Control Suite in the Appendix.\n\n\n> Or perhaps I'm misunderstanding, and Algorithm 1 is meant to by applied after a training run in order to robustify an already learned policy?\n\nThis is correct -- the purpose of Algorithm 1 is to transport an existing good policy to a less noisy neighborhood of parameter space. Training an agent from scratch with the procedure in Algorithm 1 would slow down progress, especially at the beginning of training. The pipeline we study involves achieving a 'good' policy (with respect to its return) using existing RL techniques, and then 'fine-tuning' the resulting policy with Algorithm 1.\n\n> I think you should probably mention distributional RL [1] as related work.\n\nThanks for the suggestion, we will add that to the related work.\n\n> In Figure 3, labeling the successful/failing trajectory either in the plot or in the caption would be nice.\n\nThank you for the suggestion, we will add this to the final version of the paper."}, {"review_id": "KvprnsuX4z", "reviewer": "Reviewer_Lizq", "summary": "This paper explores the concept of return landscapes in deep reinforcement learning agents for continuous control. The authors demonstrate that the mapping between policy parameters and return can exhibit high-frequency discontinuities, leading to variations in agent performance. They introduce the concept of noisy neighborhoods in the return landscape and show that different policies can have substantially different distributional profiles, even when they achieve similar average returns. The authors also investigate the impact of noisy neighborhoods on learning dynamics and propose a distribution-aware procedure to improve policy robustness.", "questions": "1. I am particularly interested in whether distributional RL algorithms exhibit the issues mentioned in the paper. Given that distributional RL methods take into account reward distribution during policy optimization (albeit with significant differences from the post-update reward distribution studied in this paper), can these methods produce smoother reward landscapes?\n\n2. Can the authors analyze the limitations of Algorithm 1? In particular, are the main conclusions and Algorithm 1 still valid for policies updated using different methods (beyond PPO, SAC, and TD3)?\n\n3. On average, how many times will Algorithm 1 reject an update during a single policy parameter update in actual operation? Will the efficiency of the algorithm be severely affected if the number of rejected updates becomes too high?", "limitations": "Limitations are not explicitly discussed by the paper. Some comments and questions regarding limitations of the work are discussed in the section above.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper introduces a fresh perspective on the instability of deep reinforcement learning agents in continuous control by studying the return landscape.\n- The concept of noisy neighborhoods and the characterization of post-update return distributions provide valuable insights into the relationship between policy parameters and return.\n- The experiments and visualizations presented in the paper help illustrate the findings and support the claims made by the authors.", "weaknesses": "This paper is well-written, and the authors thoroughly explain the key concepts and conclusions by employing extensive experiments and visualizations. However, I believe there are areas for improvement:\n\n1. In Section 4.2, the authors state, \"while Figure 5 demonstrates that we can smoothly interpolate between levels of stability along a single gradient direction, Figure 1 illustrates that many gradient directions are not so forgiving. As such, we posit that the stability of policies over the course of training can be improved simply by rejecting gradient updates that lead to policies with less favorable post-update return distributions.\" I find the introduction of Algorithm 1 to be somewhat abrupt, as it does not seem closely related to the conclusions drawn earlier in the paper. For instance, an important observation made is that \"we can smoothly interpolate between levels of stability along a single gradient direction.\" In algorithm design, can we not only reject policy updates that create noisier neighborhoods but also leverage policy updates that result in more stable neighborhoods?\n\n2. The paper designs a simple yet effective algorithm based on crucial conclusions derived from in-depth analysis. However, due to the algorithm's conciseness and clarity, it becomes difficult for readers to link it to other existing methodologies and place it appropriately within the field. Therefore, a more thorough comparison or discussion with related work is necessary. For example, in the related work section, the authors have mentioned that their work is related to studies based on rejection/backtracking strategies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback!\n\n\n> \"while Figure 5 demonstrates that we can smoothly interpolate between levels of stability along a single gradient direction\"\n\nWe apologize for the confusion. This is a typo in the original version of our paper: the direction that is used to interpolate between policies is not a gradient direction, but it is instead the direction going from one vector to the other in the parameter space.\n\n> can we not only reject policy updates that create noisier neighborhoods but also leverage policy updates that result in more stable neighborhoods?\n\nWe appreciate the suggestion. We believe this is indeed an exciting avenue for future research, but we found this approach to be much less stable than the approach presented in Algorithm 1 in our experimentation. Moreover, we leverage Algorithm 1 to show that many, but not all, of the directions proposed by policy optimization algorithms during training are actually sensible directions of improvement in terms of LTP. We believe this observation could be useful for practitioners who wish to improve this class of algorithms.\n\n> Therefore, a more thorough comparison or discussion with related work is necessary. For example, in the related work section, the authors have mentioned that their work is related to studies based on rejection/backtracking strategies\n\nWe appreciate the suggestion. We will expand our discussion in the related work section.\n\n> I am particularly interested in whether distributional RL algorithms exhibit the issues mentioned in the paper. Given that distributional RL methods take into account reward distribution during policy optimization (albeit with significant differences from the post-update reward distribution studied in this paper), can these methods produce smoother reward landscapes?\n\nThis is an excellent question, and indeed, it was an approach that we experimented with extensively. We experimented with a distributional extension of TD3, where the critic was replaced with a distributional critic. Additionally, we experimented with risk-averse policy updates using the distributional critic, where the Conditional Value at Risk (CVaR) of the return distribution was reinforced rather than the mean return, with the hypothesis that the risk-averse policies should navigate towards smoother neighborhoods. Unfortunately, our findings showed no significant improvement with regard to the smoothness of the neighborhoods encountered by the distributional agents.\n\nWe also compared the post-update return distributions of the policies visited by TD3 and distributional TD3, and observed that there is no substantial difference in neighborhoods reached by the distributional algorithms relative to TD3. We found that the distributional critic does not provide a reliable estimate of the post-update return distribution. Indeed, the distributional Bellman equation characterizing the return distributions in distributional RL does not model such an object explicitly.\n\n> Can the authors analyze the limitations of Algorithm 1? In particular, are the main conclusions and Algorithm 1 still valid for policies updated using different methods (beyond PPO, SAC, and TD3)?\n\nTo understand the limitations of Algorithm 1 in an extreme case, we have run the same rejection procedure but using random directions proposed by simple Gaussian perturbations to the policy parameters instead of the directions produced by policy optimization algorithms. In this extreme case, we found that Algorithm 1 is no longer an effective procedure for improving the LTP, suggesting that the Algorithm might be sensitive to the presence of bad update directions.\n\nAnother limitation of the algorithm, which does not create particular problems for the specific evaluation setting leveraged in our paper but might create problems when aiming at sample efficiency, is the need to obtain rollouts from the environment to evaluate the post-update return distribution statistics and reject an update.\n\n> On average, how many times will Algorithm 1 reject an update during a single policy parameter update in actual operation? Will the efficiency of the algorithm be severely affected if the number of rejected updates becomes too high?\n\nThanks for the question. We compute the statistics for the frequency of rejections across five seeds in Figure 7 in the pdf.\n\nIndeed, the rate of improvement of the algorithm (with respect to the policy return) may be reduced when using Algorithm 1. However, Algorithm 1 is only meant as a procedure to improve an existing acceptable policy with respect to its LTP -- we are not looking for fast improvement at this stage. Having said that, as shown in Figure 6, the difference in mean return after 40 gradient steps of Algorithm 1 tends to be comparable to that of TD3."}]

Author rebuttal:
We thank the reviewers for their helpful comments and feedback. Overall, the reviewers found our perspective to be novel (ZEGo, Voa2, Lizq), our paper to be well-written (ZEGo, QuAp, Voa2, Lizq) and the quality of our figures and experiments to be high (Voa2, Lizq).


Multiple reviewers (ZEGo, ErLG, VoA2) were curious to know whether our method of analysis based on post-update return distributions and our results generalize to other domains. We provide an affirmative answer by showing that the tools we introduce are easily applicable to discrete-action environments (see Figures 4, 5, 1 (b) in the pdf), and that meaningful variations in the post-update return can be found across different settings.


As discrete-action environments, we employ four games from the standard ALE benchmark (Bellemare et al, 2013). We run PPO on these environments for 5 runs and for 10 million steps each, and measure post-update return distribution statistics using the same protocol employed in the rest of our paper: we evaluate 10 policies evenly-spaced across training, and perform 1000 independent updates to each policy. We find that post-update return distributions computed for these environments still exhibit a remarkable degree of variation, as captured by their standard deviation. At the same time, the shape of the resulting distributions can be quite different compared to the ones observed in robotic locomotion tasks, and, thus, it is not necessarily described in a rich way by metrics such as the LTP (see Figure 5).

Overall, these results reinforce the utility of return landscapes and post-update return distributions as a general tool to understand policies produced by deep reinforcement learning algorithms.

Reviewers also raised a range of individual issues, which we have addressed. This includes clarifying the motivation behind the paper, reporting more precise quantitative evidence for our findings, and clearing up some misunderstandings. We address these in replies to each reviewer below.

Meta-review: The paper empirically studies the return landscape of RL algorithms through the lens of "noisy post-update neighbourhood". This refers to the return of a policy after an update of an RL algorithm. The noise is because of the stochasticity of the update given the mini-batch. The paper considers a few deep RL algorithms (SAC, PPO, TD3) and shows that the distribution of this post-update return affects how reliable the learned policy is, among several other related observations .

Most reviewers are positive about this paper. They believe that the paper provides a fresh perspective, is insightful, and the experiments support the main claims of the paper in overall.

There were some negative concerns about this paper, for example the limitation of the environments to MuJuco ones (which was partially addressed during the rebuttals by adding some ALE results), and several others concerns raised by Reviewer ErLG. For example, the reviewer has objections in how terms such as stability or safety have been used in the paper. Stability has a specific meaning in control theory, which is different from this paper's usage.

I read the paper myself. I believe that the paper offers new tools and empirical methodology to analyze RL algorithms. I also see that one may question certain aspects of the paper, including whether the measured quantities are actually what matters or the terminologies are the most accurate ones.

All considered, I believe what this paper offers clearly surpasses its shortcomings. As a result, I **recommend the acceptance** of this work and encourage the authors to consider the comments by the reviewers in order to improve their work.


I have some additional questions, which I hope the authors address in their next iteration of their paper:

- The paper focuses on the return of the post-update policies, showing that there are regions of the policy parameter space with high variations and heavy lower tail. What about the variations of the post-update parameters? Of course, this is a bit tricky to measure, as the parameters are a vector and not a scalar as return is, but I think some statistics of the parameter change, perhaps simply the norm, can be reported.

- In Figure 5, what are the (average) distance in theta_0 and theta_1 for two different cases of Same and Different runs? Are they comparable? 
The dynamics of the training of a neural network is nonlinear, and it may converge to different basis of attractions for each run.

- The paper has a claim that "such a policy may correspond to a safer behavior, ... and possibly to deal with other unexpected sources of perturbation during deployment" (LL152-154). What kind of perturbations during the deployment are you thinking of? If the parameters of the policy are not corrupted, which is a safe assumption unless we lossy compress the neural network for deployment, the policy parameters do not change. Are the authors implying that having a smoother post-update neighbourhood somehow has robustness (in the sim2read context, for example) implications? If so, why?
Decision: Accept (poster)