Title: Policy Optimization in a Noisy Neighborhood: On Return Landscapes in Continuous Control
Venue: NeurIPS 2023
Accepted: yes
Keywords: deep reinforcement learning

continuous control

return landscape

stability
Abstract: Deep reinforcement learning agents for continuous control are known to exhibit significant instability in their performance over time. In this work, we provide a fresh perspective on these behaviors by studying the return landscape: the mapping between a policy and a return. We find that popular algorithms traverse noisy neighborhoods of this landscape, in which a single update to the policy parameters leads to a wide range of returns. By taking a distributional view of these returns, we map the landscape, characterizing failure-prone regions of policy space and revealing a hidden dimension of policy quality. We show that the landscape exhibits surprising structure by finding simple paths in parameter space which improve the stability of a policy. To conclude, we develop a distribution-aware procedure which finds such paths, navigating away from noisy neighborhoods in order to improve the robustness of a policy. Taken together, our results provide new insight into the optimization, evaluation, and design of agents.

Policy Optimization in a Noisy Neighborhood: On Return Landscapes in Continuous Control

Abstract

Deep reinforcement learning agents for continuous control are known to exhibit significant instability in their performance over time. In this work, we provide a fresh perspective on these behaviors by studying the return landscape: the mapping between a policy and a return. We find that popular algorithms traverse noisy neighborhoods of this landscape, in which a single update to the policy parameters leads to a wide range of returns. By taking a distributional view of these returns, we map the landscape, characterizing failure-prone regions of policy space and revealing a hidden dimension of policy quality. We show that the landscape exhibits surprising structure by finding simple paths in parameter space which improve the stability of a policy. To conclude, we develop a distribution-aware procedure which finds such paths, navigating away from noisy neighborhoods in order to improve the robustness of a policy. Taken together, our results provide new insight into the optimization, evaluation, and design of agents.

1 Introduction

It is well-documented that agents trained with deep reinforcement learning can exhibit substantial variations in performance - as measured by their episodic return. The problem is particularly acute in continuous control, where these variations make it difficult to compare the end product of different algorithms or implementations of the same algorithm [11, 20] or even reliably measure an agent's progress from episode to episode [9]. A recurring finding is that simply averaging the return produced by a set of policies may be insufficient for rigorous evaluation.

In this paper, we demonstrate that high-frequency discontinuities in the mapping from policy parameters θ to the return R ( θ ) are an important cause of return variation. As a consequence of these discontinuities, a single gradient step or perturbation to the policy parameters often causes important changes in the return, even in settings where both the policy and the dynamics are deterministic. Because an agent's parameters constantly change during training and should be robust to minute parametric perturbations, we argue that the distribution of returns in the neighborhood of θ is in fact a better representative of its performance, both from an evaluation and an optimization perspective.

Noisy neighborhoods in the return landscape. We call the return landscape the mapping from θ to R ( θ ), our main object of study. We show that the return often varies substantially within the vicinity of any given θ , forming what we call a noisy neighborhood of θ . Based on this observation, we demonstrate the usefulness of studying the landscape through the distribution of returns obtained from small perturbations of θ . In the important case where these perturbations result from a single gradient step, we call the resulting object the post-update return distribution .

Post-Update Returns

Behaviors

2500

2000

Return

1500

1000

500

0

3

2

1

Update Direction 1

3

01

0

Direction

2

3

500

1000

1500

2000

2500

Return

Update l

2500

LRNUIR

2000

Return

1500

1000

500

0

3

0

Update Direction 1

12

3

3

500

1000

1500

2000

2500

Return

Update I

Diversity in equally-performing policies. We show that different neighborhoods correspond to different post-update return distributions and agent behaviors. We discover that at equal average returns, different policies obtained by the same deep RL algorithm may in fact have substantially different distributional profiles, as measured by statistics of the post-update return distribution. Moreover, we uncover that many of these distributions are long-tailed and we find the source of these tails to be sudden failures from an otherwise successful policy.

Effect on learning dynamics. We expose how the transition between noisy and smooth parts of the landscape happens. Surprisingly, while large valleys of low return are visible when linearly interpolating between similarly performing policies from different runs, we show no such valleys typically exist between policies from the same run. Based on this insight, we show that it is possible to find an improved set of parameters θ that achieves comparable return, but substantially lower post-update variation.

We believe the phenomenon we study is central to deep reinforcement learning in continuous control. Beyond its effect on learning dynamics (for example, through increased variance and implicit exploration [39]), it is also a potential driver of instability in sim2real settings, even in the face of seemingly small environmental changes. Additionally, it suggests that one should not simply deploy the policy obtained at the end of a training run, and that further post-training tuning may be beneficial.

2 Background

In reinforcement learning, an agent interfaces with an environment. In this paper, we are interested in continuous control environments modelled as a finite-horizon Markov Decision Process (MDP) M = ⟨S , A , r, f, T, ρ 0 ⟩ , where S ≡ R n is the state space, A ≡ R m is the action space, r : S × A → R is a reward function, f : S × A → S is a deterministic transition function, T is the horizon, and ρ 0 = U ( s I -β, s I + β ) is an initial state distribution with s I being an initial reference state, and β ∈ R n an environment-dependent parameter. We assume that each agent produces a stationary Markovian deterministic policy π θ : S → A within a parametrized family Π Θ = { π θ : θ ∈ Θ ⊂ R d } . In an episodic setting, the interaction of the agent with the environment with a given policy π θ from some state s ∈ S produces a trajectory in the environment and,

consequently, a return :

We are interested in understanding how small changes to the policy parameter affect the associated return. To this end it is sufficient to study the return from the reference state s I (in Appendix A.4 we show that similar effects occur across the state space). The return landscape is our main object of study.

Definition 2.1 (Return Landscape) . The return landscape is the mapping from policy parameters to return, starting from the initial reference state:

Figure 1 (left) depicts small portions of the return landscape for a particular environment and policy parametrization (we describe the visualization procedure below).

In this work, we will use the policies discovered by popular algorithms to characterize the topology of the return landscape. We focus on policy-based deep reinforcement learning algorithms for continuous control, such as Soft Actor-Critic (SAC) [19], Twin-Delayed DDPG (TD3) [16], and PPO [42] which use neural network function approximators to represent the policy. Such algorithms learn good behavior in the environment by maximizing the discounted return. In the process, they produce a sequence of policies

where u : Θ × R → Θ is the algorithmic policy update function, and X t is some random variable abstracting the stochasticity inherent to the update. For example, SAC and TD3 construct parametric updates by sampling a small number of transitions (minibatches) from their replay buffer [29, 31].

3 A Distributional View on Return Landscapes

The return landscape arises from the interaction between an environment and a class of parameterized policies. We first consider how this landscape varies in the immediate vicinity (or neighborhood) of policies produced by deep reinforcement learning algorithms. Given a reference policy, a natural choice is to consider how the return is affected by single updates to the policy parameters. To this end, we view the collection of possible returns obtained by evaluating the updated policy as a distribution over returns; as we will see, this distribution widely varies across the return landscape.

Definition 3.1 (Post-Update Return) . Let Π Θ be a parametric space of deterministic policies and u an update function. Given π θ ∈ Π Θ , its post-update return is defined as:

where P is a an algorithm-dependent source of stochasticity.

The post-update return inherits randomness from the underlying training algorithm and it is thus a random variable. Clearly, a post-update return will have an associated policy and trajectory, which are in turn random variables. In this work, we will leverage the distribution of post-update returns as a tool to investigate the properties of neighborhoods of the return landscape.

The different panels of Figure 1 illustrate how the return landscape in the neighborhood of θ translates into different post-update return distributions. Here, the return landscape is visualized along two update directions computed by the training algorithm based on two different batches sampled from its replay buffer, such that 1.0 on each axis corresponds to a single parameter update in that direction (details in Appendix A.2). The middle panel shows the corresponding post-update return distribution estimated using 10000 samples. We find that the distribution from the noisy neighborhood (top) exhibits a significant left tail, while the distribution from the quieter neighborhood (lower) is concentrated around its mean. On the right, we illustrate the gait produced by the reference policies (the origin in the left panel). We find qualitatively that the policy in the noisy neighborhood exhibits a curved gait which is sometimes faster, but unstable, whereas the policy in the smooth neighborhood produces an upright gait which can be slower, yet is very stable. We include similar evidence for other environments in Appendix A.11.

0

500

1000

1500

Standard Deviation

-

2000

0

2000

4000

6000

Mean

f

e

c

d

a

b

-

20

-

10

0

Skewness

-

2000

0

2000

4000

6000

f

e

c

d

a

b

0%

0.1%

1%

10%

Left Tail Probability

-

2000

0

2000

4000

6000

PPO

TD3

SAC

f

e

c

d

a

b

0

5000

Return

0

.

000

0

.

002

Density

a

0

5000

Return

0

.

000

0

.

001

0

.

002

b

0

5000

Return

0

.

0000

0

.

0005

0

.

0010

c

0

5000

Return

0

.

00

0

.

02

d

0

5000

Return

0

.

000

0

.

001

e

0

5000

Return

0

.

000

0

.

005

f

3.1 Post-Update Return Distributions as a Characterization of the Return Landscape

The mean of the post-update distribution naturally captures the average behavior represented by an algorithm as it traverses a given neighborhood. We further characterize this distribution by measuring its standard deviation (a measure of spread around the mean) and its skewness (a measure of asymmetry). In our context, a negative skewness describes a distribution with a heavy left tail, similar to the one shown in Figure 1. Such a tail is especially interesting to us as it indicates lowerthan-expected returns. However, we find that skewness is not directly interpretable as a numerical quantity. To capture these tails interpretably, we introduce a metric we call left-tail probability . The left-tail probability of a random variable Y is defined as

This quantity satisfies some desirable properties within the context of our study. First, it uses the mode of the distribution as a reference value. This is by contrast with the mean of the distribution, which may not correspond to the 'majority' behavior (as illustrated in the top half of Figure 1). It also allows us to more easily compare the tailedness of distributions generated from policies of widely varying returns. Second, it is an easily-interpretable quantity which measures the total probability mass falling in the left tail. For simplicity, here we assume that Y is positive, noting the idea can be naturally generalized to random variables bounded below. In our analyses we write LTP ≡ LTP 1 / 2 to measure drops from the mode of the post-update return distribution of at least 50%. In practice, we estimate the LTP by leveraging the Chernoff estimator [10], computing the mode as the midpoint of the interval of the most populated bin in a 100-bin histogram.

Equipped with these metrics, we measure the mean and the other three statistics of the post-update return for a set of 600 policies produced, across trials and iterations, by three popular deep RL algorithms (TD3, SAC and PPO). We use 20 seeds per algorithm and 10 checkpoints per seed, for a total of 200 policies per algorithm. These checkpoints are equally-spaced in time in training runs of 1 million steps for TD3 and SAC and 60 million steps for PPO. Each of the 600 distributions is estimated by performing 1000 independent updates to the starting policy and then rolling the resulting deterministic policy out in the environment for 1000 time-steps to compute its return. Each update is different due to a different batch sampled from the replay buffer for TD3 and SAC, and to a different batch of data from the environment collected by a randomly-perturbed policy for PPO. This amounts to millions of policy evaluations for which, for computational reasons, we primarily use the easily parallelizable environments from the Brax simulator [15]. We also include similar results on the postupdate return distributions of policies trained on DeepMind Control Suite [44] and on games from the

0

500

1000

Time

0

5

10

Reward

0

2000

4000

6000

R

t

, Successful

0

2000

4000

6000

R

t

, Failing

halfcheetah

0

500

1000

Time

0

1

2

3

Reward

0

1000

2000

3000

R

t

, Successful

0

1000

2000

3000

R

t

, Failing

walker2d

ALE [7] in Appendix A.5 and A.6. Additional experimental details, including the hyperparameters and implementations used for running these algorithms, can be found in Appendix A.1.

Figure 2 illustrates how different policies produced by deep RL algorithms correspond to a wide range of post-update return distributions, as measured by our chosen metrics 2 . For each metric, we report the bootstrapped mean using 1000 resamples to account for sampling error in the post-update returns collected for a given policy, and omit the corresponding bootstrapped confidence intervals for visual clarity, as they are very small. In particular, this scatter plot shows that different policy parameters achieve similar levels of returns (as measured by the distribution mean) but a wide range of possible levels of variability, as measured by standard deviation, skewness and left-tail probability. This suggests, in a similar way to the example shown in Figure 1, that algorithms discover behaviors which can be qualitatively very different from one another, and that leveraging the post-update return distribution can offer a new lens to investigate different dimensions of policy quality.

These results suggests that simply optimizing the mean return of a policy might ignore its distributional aspect. In particular, a practitioner will likely prefer, for a given level of return, a policy featuring a post-update return distribution with lower levels of standard deviation or left-tail probability. Intuitively, such a policy may correspond to a safer behavior, both able to more robustly accommodate additional updates from its training algorithm and possibly to deal with other unexpected sources of perturbation during deployment.

3.2 Analyzing Failures

One characteristic feature of the post-update distributions studied above is the existence of a significant lower tail for many policies visited by the three deep RL algorithms TD3, SAC and PPO. This is visible in their skewness, but especially in their left-tail probability, which demonstrates that many policies produce returns which are unexpectedly poor after up to roughly 10% of updates. We now take a closer look at the specific mechanism by which small changes in an agent's actions results in a wide range of returns in continuous control.

Our experimental procedure is as follows. For each environment, we randomly select 10 policies from the logged checkpoints of 20 independent runs of TD3, conditioned on the fact that the policy has a left-tail probability which is greater than zero. These are policies that we know are prone to poor returns following an update. For each policy, we compute the post-update return distribution by collecting trajectories in the environment after a single update to the original policy. According to this procedure, we identify two trajectories drawn from the neighborhood around the policy: a successful trajectory, characterized by a return within 10% of the mean of the post-update distribution, and a failing trajectory, characterized by a return of less than 50% of the mode of the post-update distribution, as in the left-tail probability.

Our goal is to understand the differences between these successful and failing trajectories in order to explain how long-tail returns occur. To this end, Figure 3 depicts two views of the trajectory data. For each environment, the left subplot considers a single pair of successful/failing trajectories corresponding to one of the chosen policies, and plots the reward per timestep earned in these two trajectories. These results suggest that the failing policies which make up the tail of the post-update distribution are capable of collecting similar rewards to the successful policies, yet are prone to missteps which result in episode termination (as in walker2d ) or transition to a low-reward, quasiabsorbing state (as in halfcheetah ). Figure 4 shows an example of such a misstep in walker2d .

We present a broader view of these observations through the right subplots, per-environment, in Figure 3. Here, we plot each of the trajectory pairs as a parametric curve in time. For both the successful and failing trajectories, we compute the return up to time t , R ≤ t = ∑ t i =1 r ( s i , a i ) . Then, for each value of t , we plot R ≤ t for the successful and failing trajectories as a point on the curve, allowing us to visualize the simultaneous evolution of both trajectories.

We assume that R ≤ t +1 = R ≤ t when the length of one trajectory exceeds the other, that is, no additional reward is collected after the trajectory terminates. The resulting visualization reveals several notable findings. First, we show that nearly all trajectory pairs begin by following the line y = x , indicating that the respective policies collect rewards at almost exactly the same rate. Next, we observe that many curves rapidly diverge from this line to horizontal, indicating that the failing trajectory suddenly starts collecting little to no reward, while the successful trajectory continues. In walker2d , these divergences reflect sudden terminations of the episode, represented by horizontal lines. In halfcheetah , which does not terminate, we see that instead the failing agent gets stuck in low-reward absorbing states, but is sometimes able to recover and go back to collecting reward at the same rate as the successful trajectory. We include similar visualizations for the hopper and ant environments in Appendix A.8, which support the same conclusions.

Taken together, these results demonstrate that some policies exist on the edge of failure, where a slight update can trigger the policy to take actions which push it out of its stable gait and into catastrophe. Indeed, when we compare the gaits learned by policies of high left-tail probability to those which are more wellbehaved under updates, we observe that the behaviors of the former are qualitatively more unstable (Figure 1, with more examples in Appendix A.11).

4 Navigating Return Landscapes

In the previous section, we took a fine-grained look at the return landscape, using post-update return distributions to characterize the neighborhood of different policies learned by deep RL algorithms. We now consider this landscape on a more global scale, specifically how the agent's return changes as one interpolates between different policies.

4.1 Connectivity in the Return Landscape

For our analysis, we use 200 policies generated by different runs of TD3. From these we select pairs of policies with different post-update return distributions, as measured by their standard deviation or left-tail probability, but similar mean. Consider two sets of policy parameters θ 1 and θ 2 , for which the post-update return distribution implied by θ 1 has higher LTP than the implied by θ 2 . We linearly interpolate these two to form a family of parameters θ = α θ 1 +(1 -α ) θ 2 , α ∈ [0 , 1] . For each such θ , we then record the return R ( θ ) obtained by a single simulation with the corresponding policy.

In Figure 5, we show the result of this interpolation for six pairs of policies in the hopper and walker2d environments, in two distinct cases. In the first case, the two policies have been produced by the same run of TD3 (i.e., starting from the same initialization and history of batches); in the second case, the two policies have been generated by independent repetitions of the algorithm. The plot shows interesting information about the global structure of the return landscape: the interpolation

600

1200

Same Run

0

1000

800

1600

Return

0

800

0

.

0

0

.

5

1

.

0

Interpolation

α

1000

2000

0

.

0

0

.

5

1

.

0

Interpolation

α

0

1500

hopper

1500

3000

Same Run

0

1500

1600

2400

Return

0

2000

0

.

0

0

.

5

1

.

0

Interpolation

α

1000

2000

0

.

0

0

.

5

1

.

0

Interpolation

α

0

1500

walker2d

process traverses different parts of the landscape, highlighting a transition between a noisy part of the landscape to an inherently smoother one. Interestingly, the interpolations between policies from the same run and from different runs exhibit very different qualities. When interpolating between policies of different runs, the process traverses entire regions of the landscape of poor return, until the point in which it gets to the neighborhood of the second policy. By contrast, when interpolating between policies from the same run, the transition from a noisy to a smooth landscape happens without encountering any valley of low return - even when these policies are separated by hundreds of thousands of gradient steps in training. This is particularly surprising given that θ is a high-dimensional vector containing all of the weights of the neural network, and there is no a priori reason to believe that interpolated parameters should result in policies that are at all sensible.

0

.

15

0

.

30

0

.

45

Below-Threshold Proportion

Same Run

To further quantify the phenomenon, we want to measure the proportion of return collapses encountered when interpolating between policies. We use the following experimental design. We sample for each environment a set of 500 pairs of policies from the same runs and a set of 500 pairs of policies from different runs. Then, we linearly interpolate between policies in the pairs, producing 100 intermediate policies, and randomly perturb them using Gaussian noise with standard deviation 3 × 10 -4 to obtain an estimate of the mean of their (random) post-update return distribution. Then, for each pair of policies, we measure how frequently the return collapses in between the two extremes, by counting how many times it becomes less than 10% of the minimum return of the two original policies. We then average this Below-Threshold Proportion across pairs, and across environments using rliable [1].

Figure 6 shows that there is on average almost no drop in return when interpolating among policies from the same run. We additionally report similar results on four ALE games in Appendix A.6.

We hypothesize this might be interpreted as each individual run of the algorithm specializing on a different family of behaviors, for which, due to the geometry of the return landscape, interpolation between policy parameters does not have any disrupting effect. This result can be interpreted as being related to linear mode connectivity [17, 18], a phenomenon observed in supervised learning, for which different points in the loss landscape of neural networks can be connected by near-constant-loss paths. In other words, it appears there is typically no barrier of low average return separating policies generated from the same run, even when those policies feature very different levels of stability. The existence of such a phenomenon in the RL setting is far from certain: the optimization objective is non-stationary and the evaluation metric (the return instead of the loss) depends on an environment and multiple forward passes from a neural network.

0

.

25

0

.

50

0

.

75

1

.

00

With rejection

Without rejection

Median (Log LTP Improvement)

0

.

2

0

.

4

0

.

6

0

.

8

IQM (Log LTP Improvement)

0

.

4

0

.

8

1

.

2

Mean (Log LTP Improvement)

4.2 Stabilizing Policies by Navigating the Landscape

Overall, Figure 5 demonstrates the existence of paths in the return landscape which are able to increase the level of stability of a given policy, but are not necessarily followed in a spontaneous way by typical policy optimization algorithms. In the absence of a desirable end policy to interpolate towards, we would like to understand if it is possible to find similar stabilizing paths (as measured by the LTP), given a starting policy inhabiting a noisy neighborhood of the return landscape. We conjecture that this is feasible by filtering the policy updates produced by an algorithm: In particular, we propose to reject gradient updates that lead to policies with less favorable post-update return distributions.

In our procedure, which is outlined in Algorithm 1, we use the CVaR as a heuristic to compare the stability of post-update return distributions 3 , as it is effectively a measure of the left tail mean [38]. Our procedure works as follows: starting with a given policy, we use TD3 to interact with the environment, maintain a replay buffer, and compute updates to the policy and critic parameters. However, before applying a proposed update, we first

estimate the post-update return distributions of the post-update policies by sampling TD3 updates from random minibatches of the replay buffer and evaluating the returns of the corresponding policies. If the estimate of the post-update return CVaR is not sufficiently high relative to that of the post-update return distribution of the current policy, the update is rejected , so that the networks and the replay buffer are reverted to the state that they were in before the update was computed. In our experiments, we study the effect that such a rejection mechanism has on the evolution of the LTP by comparing the trajectories induced by this procedure without the ability to reject (i.e., regular TD3) and with the ability to reject.

In Figure 7, we show the improvement in LTP that this algorithm induces when applied to the same policy, aggregated across Brax tasks, using at least 10 policies per environment, after only forty gradient steps. We additionally present scatter plots demonstrating the effect of applying Algorithm 1 to individual policies in Appendix A.10. Our results demonstrate that this rejection procedure can be an effective tool for reducing the LTP of an existing policy.

5 Related Work

Reliability of deep RL. The goal to avoid catastrophic drops in performance was at the core of the development of foundational methods in deep RL based on conservative updates [41, 42]. Previous work also studied the development of safer algorithms for learning and exploration, both from the theoretical and the empirical standpoints [25, 30, 32, 37, 48]. Our work focuses on understanding the landscape visited by commonly employed policy optimization algorithms and shows that it is

possible to relatively easily move from parts of the landscape that induce dangerous behaviors to safer policy parameter vectors. On a higher-level, the sensitivity of deep RL algorithms to stochasticity and hyperparameters, and the extreme variability of results across seeds has been the object of previous studies [2, 11, 20], which mostly focused on proposing more reliable evaluation metrics. Previous work [9] also explicitly advocated for measuring the stability of deep RL algorithms over different axes and using a diverse set of metrics. Our paper proposes a complementary perspective, based on return landscapes and on a distributional view on them. Our procedure which leverages the directions proposed by a policy optimization algorithm to improve the LTP of a policy is related to previous work based on rejection/backtracking strategies [25, 40].

Return and loss landscapes. Return landscapes have been previously investigated at a coarser level under the name of reward surfaces/landscapes. In particular, they have been employed for studying the alignment of the gradient directions suggested by policy optimization algorithms to directions of improvement in the actual environment [23] and investigating performance degradation as a long-term optimization danger in such algorithms [43]. Our study of return landscapes with a distributional view in an otherwise fully deterministic setting sheds new light both on the landscape itself and on how it can be leveraged to characterize individual policies. More generally, the investigation of the return that policies collect in an environment is related to the study of the loss landscape of neural networks in supervised learning, for which different techniques have been proposed [28]. Those techniques, together with RL-specific tools, have been employed to explore the loss landscapes of RL algorithms, by visualizing them [5], probing their interaction with entropy regularization [3] or larger neural networks [35]. Our discovery of how policies from the same run are connected by simple paths in parameter space is related to (linear) mode connectivity, which shows a similar behavior in the landscapes of neural networks trained in supervised learning tasks [12-14, 17, 18]. Finally, our work is related to distributional RL [6], but we specifically focus on the post-update return distribution as opposed to the distribution of returns under a given policy.

6 Discussion and Future Work

In this paper, we have investigated return landscapes in continuous control tasks, as traversed by deep RL algorithms. We demonstrated the existence of noisy neighborhoods of the landscape, where a single update to the policy parameters produces a wide-breadth of post-update returns. By taking a distributional view on these neighborhoods, we revealed the existence of neighborhoods of similar mean return, yet different statistics, which correspond to qualitatively different agent behaviors. We studied the characteristics of failing policies and trajectories in such neighborhoods and attributed their subpar performance to sudden collapses in trajectory reward, rather than overall degradation in the policy. By focusing on linear paths through the global policy landscape, we showed that the landscape exhibits macro-scale variations which extend beyond specific local neighborhoods, and that policies from the same run can be surprisingly connected by linear paths with no valleys of low return. Finally, we demonstrated a simple procedure which discovers paths towards smoother regions of the landscape, starting from a trained policy.

Our results suggest that some of the previously-observed reliability issues in deep reinforcement learning agents for continuous control may be due to the fundamental structure of the return landscape for neural network policies. In particular, while the return of policy in a given neighborhood may be adequate, the distributional structure of the neighborhood characterizes additional dimensions of policy quality: How stable is this policy? What kind of behavior has the agent learned? Is it safe to perform further optimization of this policy? These nuances indicate the potential utility of a landscape-inspired approach to the design of reliable deep RL algorithms.

In addition, our study of parameter interpolation on the return landscape reveals new curiosities surrounding the training behavior of deep reinforcement learning agents. It appears that many policies from the same run fall within a single basin of the return landscape; we conjecture that this may correspond to the algorithm 'specializing' on one particular behavior. Our demonstration of regions of lower and higher variability in returns along such paths further supports the possibility of robustifying existing policies, yet also raises the question of whether there are significantly different behaviors separated by barriers of low return, and whether our algorithms can find them. As they are beyond the scope of this paper, we reserve such questions for future work.

References

Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C. Courville, and Marc G. Bellemare. Deep reinforcement learning at the edge of the statistical precipice. In Neural Information Processing Systems , 2021.

Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C. Courville, and Marc G. Bellemare. Deep reinforcement learning at the edge of the statistical precipice. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 29304-29320, 2021.

Zafarali Ahmed, Nicolas Le Roux, Mohammad Norouzi, and Dale Schuurmans. Understanding the impact of entropy on policy optimization. In International Conference on Machine Learning , 2018.

Igor Babuschkin, Kate Baumli, Alison Bell, Surya Bhupatiraju, Jake Bruce, Peter Buchlovsky, David Budden, Trevor Cai, Aidan Clark, Ivo Danihelka, Claudio Fantacci, Jonathan Godwin, Chris Jones, Tom Hennigan, Matteo Hessel, Steven Kapturowski, Thomas Keck, Iurii Kemaev, Michael King, Lena Martens, Vladimir Mikulik, Tamara Norman, John Quan, George Papamakarios, Roman Ring, Francisco Ruiz, Alvaro Sanchez, Rosalia Schneider, Eren Sezener, Stephen Spencer, Srivatsan Srinivasan, Wojciech Stokowiec, and Fabio Viola. The DeepMind JAX Ecosystem, 2020.

Recep Yusuf Bekci and Mehmet Gümü¸ s. Visualizing the loss landscape of actor critic methods with applications in inventory optimization. arXiv preprint arXiv:2009.02391 , 2020.

Marc G Bellemare, Will Dabney, and Mark Rowland. Distributional reinforcement learning, 2022.

Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents (extended abstract). In International Joint Conference on Artificial Intelligence , 2012.

James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018.

Stephanie CY Chan, Samuel Fishman, Anoop Korattikara, John Canny, and Sergio Guadarrama. Measuring the Reliability of Reinforcement Learning Algorithms. In International Conference on Learning Representations , 2019.

Herman Chernoff. Estimation of the mode. Annals of the Institute of Statistical Mathematics , 16(1):31-41, 1964.

Cédric Colas, Olivier Sigaud, and Pierre-Yves Oudeyer. A hitchhiker's guide to statistical comparisons of reinforcement learning algorithms. In Reproducibility in Machine Learning, ICLR 2019 Workshop, New Orleans, Louisiana, United States, May 6, 2019 . OpenReview.net, 2019.

Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In International conference on machine learning , pages 1309-1318. PMLR, 2018.

Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning , pages 3259-3269. PMLR, 2020.

C Daniel Freeman and Joan Bruna. Topology and geometry of half-rectified network optimization. In 5th International Conference on Learning Representations, ICLR 2017 , 2017.

C. Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax - a differentiable physics engine for large scale rigid body simulation. ArXiv , abs/2106.13281, 2021.

Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. ArXiv , abs/1802.09477, 2018.

Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information processing systems , 31, 2018.

Ian J. Goodfellow and Oriol Vinyals. Qualitatively characterizing neural network optimization problems. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861-1870. PMLR, 2018.

Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence , volume 32, 2018.

Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research , 23(274):1-18, 2022.

John D Hunter. Matplotlib: A 2d graphics environment. IEEE Annals of the History of Computing , 9(03):90-95, 2007.

Andrew Ilyas, Logan Engstrom, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. A Closer Look at Deep Policy Gradients. In International Conference on Learning Representations , December 2019.

Eric Jones, Travis Oliphant, and Pearu Peterson. SciPy: Open source scientific tools for Python . 2014.

Pranav Khanna, Guy Tennenholtz, Nadav Merlis, Shie Mannor, and Chen Tessler. Never Worse, Mostly Better: Stable Policy Improvement in Deep Reinforcement Learning, August 2022.

Thomas Kluyver, Benjamin Ragan-Kelley, Fernando Pérez, Brian E Granger, Matthias Bussonnier, Jonathan Frederic, Kyle Kelley, Jessica B Hamrick, Jason Grout, Sylvain Corlay, et al. Jupyter Notebooks-a publishing format for reproducible computational workflows. , volume 2016. 2016.

Ilya Kostrikov. JAXRL: Implementations of Reinforcement Learning algorithms in JAX, 10 2021.

Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in neural information processing systems , 31, 2018.

Longxin Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine Learning , 8:293-321, 1992.

Alberto Maria Metelli, Matteo Pirotta, Daniele Calandriello, and Marcello Restelli. Safe policy iteration: A monotonically improving approximate policy iteration approach. Journal of Machine Learning Research , 22(97):1-83, 2021.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature , 518(7540):529-533, 2015.

Evgenii Nikishin, Pavel Izmailov, Ben Athiwaratkun, Dmitrii Podoprikhin, Timur Garipov, Pavel Shvechikov, Dmitry Vetrov, and Andrew Gordon Wilson. Improving stability in deep reinforcement learning with weight averaging. In Uncertainty in Artificial Intelligence Workshop on Uncertainty in Deep Learning , 2018.

Travis E Oliphant. A guide to NumPy , volume 1. Trelgol Publishing USA, 2006.

Travis E Oliphant. Python for scientific computing. Computing in Science & Engineering , 9(3):10-20, 2007.

Keita Ota, Devesh K. Jha, and Asako Kanezaki. Training larger networks for deep reinforcement learning. ArXiv , abs/2102.07920, 2021.

The pandas development team. pandas-dev/pandas: Pandas, February 2020.

Matteo Papini, Matteo Pirotta, and Marcello Restelli. Smoothing policies and safe policy gradients. ArXiv , abs/1905.03231, 2022.

R Tyrrell Rockafellar, Stanislav Uryasev, et al. Optimization of conditional value-at-risk. Journal of risk , 2:21-42, 2000.

Tom Schaul, André Barreto, John Quan, and Georg Ostrovski. The phenomenon of policy churn. ArXiv , abs/2206.00730, 2022.

Jürgen Schmidhuber, Jieyu Zhao, and Marco Wiering. Shifting inductive bias with successstory algorithm, adaptive levin search, and incremental self-improvement. Machine Learning , 28:105-130, 1997.

John Schulman, Sergey Levine, P. Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. ArXiv , abs/1502.05477, 2015.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv , abs/1707.06347, 2017.

Ryan Sullivan, Jordan K. Terry, Benjamin Black, and John P. Dickerson. Cliff Diving: Exploring Reward Surfaces in Reinforcement Learning Environments. In Proceedings of the 39th International Conference on Machine Learning , pages 20744-20776. PMLR, June 2022.

Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690 , 2018.

Stefan Van Der Walt, S Chris Colbert, and Gael Varoquaux. The numpy array: a structure for efficient numerical computation. Computing in science & engineering , 13(2):22-30, 2011.

Guido Van Rossum and Fred L Drake Jr. Python tutorial , volume 620. Centrum voor Wiskunde en Informatica Amsterdam, 1995.

Wes McKinney. Data Structures for Statistical Computing in Python. In Stéfan van der Walt and Jarrod Millman, editors, Proceedings of the 9th Python in Science Conference , pages 56 61, 2010.

Qisong Yang, Thiago D Simão, Simon H Tindemans, and Matthijs TJ Spaan. Wcsac: Worst-case soft actor critic for safety-constrained reinforcement learning. In AAAI , pages 10639-10646, 2021.

A Appendix

A.1 Hyperparameters and implementation

For our analyses, we run TD3 and SAC for one million environment steps, with the default hyperparameters reported in Table 1, and collected checkpoints at 5 × 10 4 , 15 × 10 4 , 25 × 10 4 , 35 × 10 4 , 45 × 10 4 , 55 × 10 4 , 65 × 10 4 , 75 × 10 4 , 85 × 10 4 , 95 × 10 4 steps for 20 different seeds. We run 20 seeds of PPO for 60 million environment steps with the hyperparameters in Table 1, to obtain a comparable level of performance, being that PPO is notoriously less sample-efficient than the two other algorithms [19]. We use equally-spaced checkpoints, at approximately 6 × 10 5 , 12 × 10 5 , 18 × 10 5 , 24 × 10 5 , 30 × 10 5 , 36 × 10 5 , 42 × 10 5 , 48 × 10 5 , 54 × 10 5 , 60 × 10 5 environment steps.

We acknowledge the Python community [34, 46] for developing the core set of tools that enabled this work, including JAX [4, 8], Jupyter [26], Matplotlib [22], numpy [33, 45], pandas [36, 47], and SciPy [24]. Our implementations of TD3 and SAC are based on jaxrl [27] and our implementation of PPO is based on CleanRL [21].

Our code is available at https://github.com/nathanrahn/return-landscapes .

A.2 Details of return landscape visualization

To generate the return landscapes in Figure 1, we start from a given policy of parameters θ 0 (produced, in that case, by SAC), and update it two times independently to obtain θ ′ and θ ′′ . Then, we plot R ( θ ) , θ = α ( θ ′ -θ 0 ) + β ( θ ′′ -θ 0 ) + θ 0 , α ∈ [ -3 , 3] , β ∈ [ -3 , 3] , combining the two gradient directions into other directions to explore the neighborhood. The resulting directions are equivalent to the ones that would be obtained by oversampling or undersampling some elements in a larger batch composed of elements from both the two sampled updates.

A.3 Interpolation between policies

We now specify the details of the experimental protocol used for Figure 5 and report additional results. In Figure 9, we report results of interpolation for the other two Brax environments, ant and halfcheetah , missing from Figure 5 for space reasons. We generate the checkpoints for these plots to be policies with similar level of return but different standard deviation of the post-update return distribution. A list of the checkpoints and seeds from TD3 used for producing these plots is shown in Table 2. While the view provided in Figure 5 and Figure 9 provides explicit visual cues on the variability of the neighborhood, it does not leverage directly the post-update return distribution to characterize the neighborhood of the policies obtained by interpolation. To provide a more detailed view, we estimate the post-update return distribution of an unstructured update, by sampling, given a policy of parameters θ , 1000 perturbations from an isotropic Gaussian distribution N ( θ , 0 . 0003) and evaluating the resulting policies. Figure 8 shows the mean and the standard deviation (as an error band) of the post-update return distribution of the interpolated policies from the same runs shown in Table 2. This alternate view confirms the results: linearly interpolating between policies of similar mean for the post-update return distribution produces paths of of approximately constant (or smoothly increasing/decreasing) mean post-update return, even when the standard deviation of their post-update return distributions is different.

A.4 Instabilities in states other than the starting one

In our analyses, we only investigated the return landscape by computing the return on a fixed initial state. Even if the characteristics of the return from other states naturally propagate back to the initial state, it is natural to wonder whether the failures analyzed in our paper also happen naturally in other states. To investigate this question, we compute the LTP of the post-update return distribution for 3 runs of TD3, when starting from one of the states collected by the algorithm during training. In Figure 10, for the ant and walker2d tasks, we show for each of these states, identified by their collection time (index in the replay buffer) the LTP averaged over all the checkpoints collected during training. The results indicate that, regardless of some variation across seeds, there appears not to be any special pattern in the presence of failures over states grouped by time, and that the tail of the post-update distributions is long even when measured from states other the initial one.

0

1000

0

1000

Return

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Interpolation

α

0

1000

2000

hopper

0

2000

0

2000

Return

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Interpolation

α

0

2000

walker2d

0

1000

0

2500

5000

Return

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

2000

ant

0

5000

0

2500

5000

Return

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

1000

2000

halfcheetah

Interpolation

α

Interpolation

α

0

1500

Same Run

2500

5000

0

3000

Return

2500

5000

0

.

0

0

.

5

1

.

0

Interpolation

α

0

3000

0

.

0

0

.

5

1

.

0

Interpolation

α

0

2500

ant

3000

6000

Same Run

0

5000

0

4000

Return

0

4000

0

.

0

0

.

5

1

.

0

Interpolation

α

0

1500

0

.

0

0

.

5

1

.

0

Interpolation

α

0

4000

halfcheetah

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

Collection Time

i

×

10

6

0%

2%

5%

8%

10%

13%

15%

¯

P

0

.

5

(

s

(

i

)

;

G

)

ant

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

Collection Time

i

×

10

6

walker2d

seed 0

seed 1

seed 2

LTP

LTP

SAC

Parameter

Discount factor

Minibatch size

Optimizer (all)

Optimizer (all): learning rate

Optimizer (all):

β

1

Optimizer (all):

β

2

Optimizer (all):

ϵ

Networks (all): activation

Networks (all): n. hidden layers

Networks (all): hidden units

Initial Temperature

Updates per step

Target update period

τ

(EMA coefficient)

A.5 Scatter plots of statistics of post-update return distribution

We now show similar "maps" of the landscape to Figure 2 for all of the Brax environments we study. Results are shown in Figure 11, Figure 12, and Figure 13. We also include similar results for the quadruped-walk , quadruped-run , humanoid-stand , humanoid-walk , and humanoid-run tasks from DeepMind Control Suite [44] in Figures 14, 15, 16, 17, and 18. We believe that this type of scatter plot can provide a different perspective, compared to the one only focused on training curves, to study the interaction between a particular algorithm and an environment.

A.6 Discrete control

Despite the fact that the main focus of our paper is on continuous control, we also run small-scale experiments to understand if the tools we introduced generalize to discrete-action environments. In particular, we leverage four games from the standard ALE benchmark: Qbert , MsPacman , Breakout , and Seaquest [7]. We run PPO on these environments for 5 runs and for 10 million steps each, and measure post-update return distribution statistics using the same protocol employed in the rest of

Setting

0.99

256

Adam

0.0003

0.9

0.999

0.00015

ReLU

2

256

1

6

10

1

1

0.995

Same Run

Seed ant

1

6

11

halfcheetah

3

6

13

hopper

2

15

17

walker2d

1

4

4

Ckpt 1

550000

850000

650000

250000

250000

250000

450000

550000

950000

750000

950000

550000

150000

650000

250000

950000

350000

550000

250000

150000

750000

250000

850000

450000

0

500

1000

1500

2000

Standard Deviation

0

2000

4000

6000

Mean

-

10

-

5

0

Skewness

0

2000

4000

6000

0%

0.01%

0.1%

1%

10%

Left Tail Probability

0

2000

4000

6000

PPO

TD3

SAC

0

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

500

1000

Standard Deviation

0

500

1000

1500

2000

2500

3000

Mean

-

50

0

50

Skewness

0

500

1000

1500

2000

2500

3000

0%

0.01%

0.1%

1%

10%

Left Tail Probability

0

500

1000

1500

2000

2500

3000

PPO

TD3

SAC

0

0

0

0

0

0

0

0

0

0

0

.

.

.

.

.

.

0

.

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

0

0

.

5

1

.

0

0

500

1000

Standard Deviation

0

1000

2000

3000

4000

Mean

-

60

-

40

-

20

0

Skewness

0

1000

2000

3000

4000

0%

0.01%

0.1%

1%

10%

Left Tail Probability

0

1000

2000

3000

4000

PPO

TD3

SAC

0

0

0

.

0

.

0

0

.

0

0

.

0

.

0

0

.

0

0

100

200

300

400

Standard Deviation

0

200

400

600

800

1000

Mean

-

30

-

20

-

10

0

Skewness

0

200

400

600

800

1000

0%

0.1%

1%

10%

Left Tail Probability

0

200

400

600

800

1000

TD3

SAC

0

0

5

1

0

0

.

.

.

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

5

1

.

0

0

.

0

0

.

5

1

.

0

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

0

0

0

.

0

.

0

0

.

0

0

.

0

.

0

0

.

0

0

50

100

150

200

Standard Deviation

0

200

400

600

800

Mean

-

30

-

20

-

10

0

10

Skewness

0

200

400

600

800

0%

0.1%

1%

10%

Left Tail Probability

0

200

400

600

800

TD3

SAC

0

0

5

1

0

0

.

.

.

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

5

1

.

0

0

.

0

0

.

5

1

.

0

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

0

0

0

0

0

0

0

0

0

0

0

.

.

.

.

.

.

0

0

0

0

5

1

0

0

0

0

5

1

0

0

0

0

5

1

0

0

0

0

5

1

0

0

0

0

5

1

0

0

0

0

5

1

0

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

our paper: we evaluate 10 policies evenly-spaced across training, and perform 1000 independent updates to each policy. We find that post-update return distributions computed for these environments still exhibit a remarkable degree of variation, as captured by their standard deviation (see Figure 21). At the same time, the shape of the resulting distributions can be quite different compared to the ones observed in robotic locomotion tasks, and, thus, it is not necessarily described in a rich way by metrics such as the LTP (see Figure 20).

0

50

100

150

Standard Deviation

0

200

400

600

800

Mean

-

10

0

10

20

Skewness

0

200

400

600

800

0%

0.1%

1%

10%

Left Tail Probability

0

200

400

600

800

TD3

SAC

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

50

100

150

Standard Deviation

0

200

400

600

Mean

0

20

Skewness

0

200

400

600

0%

0.1%

1%

10%

Left Tail Probability

0

200

400

600

TD3

SAC

0

0

0

0

0

0

0

0

0

0

0

0

.

.

.

.

.

.

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

20

40

Standard Deviation

0

25

50

75

100

125

150

175

Mean

-

10

0

10

20

30

Skewness

0

25

50

75

100

125

150

175

0%

0.1%

1%

10%

Left Tail Probability

0

25

50

75

100

125

150

175

TD3

SAC

0

0

0

0

0

0

0

0

0

0

0

0

.

.

.

.

.

.

0

0

0

0

5

1

0

0

0

0

5

1

0

0

0

0

5

1

0

0

0

0

5

1

0

0

0

0

5

1

0

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

0

0

0

0

0

0

0

.

.

.

.

0

.

0

0

.

0

0

.

5

1

.

0

0

.

0

We also run an interpolation experiment on the ALE to investigate whether the connectivity phenomenon studied in Section 4.1 generalizes to discrete-control environments. We produce quantitative results on the phenomenon with the following procedure. For each one of the four games, we sample a set of at least 20 pairs of policies from same runs and 20 pairs of policies from different runs of the algorithm. Then, we linearly interpolate between policies in the pairs, producing 50 intermediate policies, and randomly perturb them using Gaussian noise with standard deviation 0.0003 to obtain an estimate of the mean of their (random) post-update return distribution. Then, for each pair of policy, we measure how frequently the return collapses in between the two extremes, by counting

0

.

16

0

.

24

0

.

32

0

.

40

Below-Threshold Proportion

Same Run

0

1000

Return

0

.

000

0

.

002

0

.

004

0

.

006

Density

0

250

500

Return

0

.

0000

0

.

0025

0

.

0050

0

.

0075

0

.

0100

Breakout-v5

0

1000

Return

0

.

000

0

.

002

0

.

004

0

.

006

MsPacman-v5

0

200

Return

0

.

00

0

.

02

0

.

04

0

.

06

Seaquest-v5

how many times it becomes less than 10% of the minimum return of the two original policies. We then average this Below-Threshold Proportion across pairs, and across environments using rliable [1]. Figure 19 shows that the phenomenon, properly quantified, is still present when using a very different class of environments (discrete-action, game-based).

A.7 Behavior cloning

While the results in the main paper demonstrate that policies learned by commonly-used RL algorithms are sensitive to updates, we were curious to understand whether behavior cloning produces policies which occupy less noisy neighborhoods of the return landscape. To do so, we conducted a set of additional experiments on 4 Brax environments. The protocol was as follows: For each environment, we consider 10 independent training runs of TD3, and 5 policies distributed evenly throughout the run. For each of these policies, we train a new agent using behavior cloning on the data logged up until the collection time of the teacher policy, replacing the actions in the dataset with the actions of the teacher policy, for 1 million gradient steps. We log 10 policies throughout each training run of behavior cloning. To compute the post-update return distribution for the policies obtained by behavior cloning, we used one additional gradient step on the MSE-based BC objective, and 1000 samples.

In Figure 22 (left), we compare statistics of the post-update return distributions for all pairs of policies ( π BC i,j , π TD 3 i ) where policy π BC i,j is obstained by behavior cloning π TD 3 i . We compute the Pearson correlation coefficient of statistics of the post-update return distributions of these policies: between the mean of each pair, and between the LTPs of each pair. We find that the means are highly correlated and that the learned BC policies are comparable in performance to their teacher policy. We additionally show that correlation in the LTP is much more variable across environments - in general, cloning a policy of high or low LTP does not always lead to a cloned policy of the same LTP.

But does training policies by BC produce policies occupying less noisy neighborhoods of the return landscape, which would have correspondingly lower LTP overall? In Figure 22 (right), we show the mean LTP across policies and environments, along with its 95% bootstrapped confidence interval following the recommendations from rliable [1]. Our results demonstrate that BC does not produce fundamentally more stable policies, as measured by the LTP.

0

200

400

Standard Deviation

250

500

750

1000

1250

1500

Mean

50

100

150

200

Standard Deviation

100

200

300

400

Mean

Breakout-v5

0

50

100

150

200

Standard Deviation

0

500

1000

1500

Mean

MsPacman-v5

0

50

100

150

Standard Deviation

0

100

200

300

400

Mean

Seaquest-v5

0.0%

5.0%

10.0%

LTP (Mean)

BC

TD3

A.8 Complete results on comparing successful and failing trajectories

In Figure 23, we include similar results to Figure 3 for all the Brax environments we study.

0

1250 2500 3750 5000

R

t

, Successful

0

1250

2500

3750

5000

R

t

, Failing

ant

0

2000

4000

6000

R

t

, Successful

0

2000

4000

6000

R

t

, Failing

halfcheetah

0

1000

2000

3000

R

t

, Successful

0

1000

2000

3000

R

t

, Failing

walker2d

0

500

1000

1500

R

t

, Successful

0

500

1000

1500

R

t

, Failing

hopper

A.9 Justification of rejection procedure

In Algorithm 1, we proposed a procedure that rejects gradient updates based on estimates of the CVaR of the resulting policies' post-update returns. Since our goal with this procedure is ultimately to decrease the LTP of an existing policy, one might rightfully question why CVaR was our heuristic of choice. Our reasoning for this choice was simply that the LTP can be minimized by arbirarily poor policies: that is, policies that only achieve 'low enough' returns will have very low LTP, because they cannot substantially fail relative to their existing performance. Such an issue can be circumvented by comparing the CVaR of the post-update returns: among all policies that have the same left-tail

0%

0.1%

1%

10%

LTP

2000

4000

6000

Mean

Without Rejection

0%

0.1%

1%

10%

LTP

With Rejection

ant

0%

0.1%

1%

10%

LTP

2000

4000

6000

Mean

Without Rejection

0%

0.1%

1%

10%

LTP

With Rejection

halfcheetah

0%

0.1%

1%

10%

LTP

1000

2000

Mean

Without Rejection

0%

0.1%

1%

10%

LTP

With Rejection

hopper

0%

0.1%

1%

10%

LTP

0

1000

2000

3000

Mean

Without Rejection

0%

0.1%

1%

10%

LTP

With Rejection

walker2d

probability, the CVaR is still able to order the policies by the quality of the returns in the left tail. To further justify the use of CVaR as a rejection heuristic, we visualized how different CVaR heuristics order policies with respect to their post-update return distributions. Figure 25 illustrates how the α -CVaR ranks policies. Note that α -CVaR in this instance is effectively measuring the mean among the post-update returns below the α -quantile of the post-update return distribution, so 0-CVaR would measure the worst-case post-update return and 1.0-CVaR is equivalent to the mean of the post-update returns.

As expected, we see that ranking by the mean is relatively agnostic to the LTP of the policies, but for α < 1 , ranking by CVaR is effectively trading off larger mean post-update returns for lower LTP. Notably, the gradient of colors (rankings) in the 0.01-CVaR column is distinctly 'top-left to bottom-right', indicating that only policies with both high returns and low LTP are ranked favorably. The 0.1-CVaR column is fairly similar, but the gradient is more vertical - that is, it is more forgiving with respect to LTP than the 0.01-CVaR heuristic, but it still clearly prefers policies with lower LTP among those with similar levels of return. Finally, the gradient in the rightmost column is exactly vertical, meaning that the ranking is agnostic to LTP.

Consequently, we argue that CVaR is a reasonable heuristic for identifying robust policies without neglecting their performance level. In our experiments, we found that the 0.1-CVaR heuristic for rejecting gradient updates struck the best balance between robustness and performance.

A.10 Additional post-update-CVaR rejection results

In Figure 24 we include the results from our experiments with Algorithm 1 in four Brax domains for policies found by TD3.

A.11 Visualization of behaviors

We uploaded some visualization of the behaviors on a website ( https://sites.google.com/ view/on-return-landscapes/home ): policies of similar mean for the post-update return but different standard deviation exhibit different behaviors.

ant

0

10

-

3

10

-

2

10

-

1

LTP

2000

4000

Mean

Rank by Post-Update Return 0.01-CVaR

0

10

-

3

10

-

2

10

-

1

LTP

2000

4000

Rank by Post-Update Return 0.1-CVaR

0

10

-

3

10

-

2

10

-

1

LTP

2000

4000

Rank by Post-Update Return Mean

0

10

-

3

10

-

2

10

-

1

LTP

2000

4000

6000

Mean

Rank by Post-Update Return 0.01-CVaR

0

10

-

3

10

-

2

10

-

1

LTP

2000

4000

6000

Rank by Post-Update Return 0.1-CVaR

0

10

-

3

10

-

2

10

-

1

LTP

2000

4000

6000

Rank by Post-Update Return Mean

halfcheetah

0

10

-

3

10

-

2

10

-

1

LTP

1000

2000

Mean

Rank by Post-Update Return 0.01-CVaR

0

10

-

3

10

-

2

10

-

1

LTP

1000

2000

Rank by Post-Update Return 0.1-CVaR

0

10

-

3

10

-

2

10

-

1

LTP

1000

2000

Rank by Post-Update Return Mean

hopper

0

10

-

3

10

-

2

10

-

1

LTP

0

1000

2000

3000

Mean

Rank by Post-Update Return 0.01-CVaR

0

10

-

3

10

-

2

10

-

1

LTP

0

1000

2000

3000

Rank by Post-Update Return 0.1-CVaR

0

10

-

3

10

-

2

10

-

1

LTP

0

1000

2000

3000

Rank by Post-Update Return Mean

walker2d

A.12 Evidence of multi-scale structure

To investigate whether the shape of a neighborhood heavily depends on the size of the steps used for exploring it, we "zoom-in" on the landscape plot from Figure 1 by using the same granularity but on a smaller window. We apply this process twice, with the results in Figure 27 giving evidence for scale-independence.

ant

walker2d

hopper

halfcheetah

0%

20%

40%

60%

80%

Rejection rate

Rejection rates by environment

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the concept of return landscapes in deep reinforcement learning agents for continuous control. The authors demonstrate that the mapping between policy parameters and return can exhibit high-frequency discontinuities, leading to variations in agent performance. They introduce the concept of noisy neighborhoods in the return landscape and show that different policies can have substantially different distributional profiles, even when they achieve similar average returns. The authors also investigate the impact of noisy neighborhoods on learning dynamics and propose a distribution-aware procedure to improve policy robustness.\n\nSTRENGTHS:\n- The paper introduces a fresh perspective on the instability of deep reinforcement learning agents in continuous control by studying the return landscape.\n- The concept of noisy neighborhoods and the characterization of post-update return distributions provide valuable insights into the relationship between policy parameters and return.\n- The experiments and visualizations presented in the paper help illustrate the findings and support the claims made by the authors.\n\nWEAKNESSES:\nThis paper is well-written, and the authors thoroughly explain the key concepts and conclusions by employing extensive experiments and visualizations. However, I believe there are areas for improvement:\n\n1. In Section 4.2, the authors state, \"while Figure 5 demonstrates that we can smoothly interpolate between levels of stability along a single gradient direction, Figure 1 illustrates that many gradient directions are not so forgiving. As such, we posit that the stability of policies over the course of training can be improved simply by rejecting gradient updates that lead to policies with less favorable post-update return distributions.\" I find the introduction of Algorithm 1 to be somewhat abrupt, as it does not seem closely related to the conclusions drawn earlier in the paper. For instance, an important observation made is that \"we can smoothly interpolate between levels of stability along a single gradient direction.\" In algorithm design, can we not only reject policy updates that create noisier neighborhoods but also leverage policy updates that result in more stable neighborhoods?\n\n2. The paper designs a simple yet effective algorithm based on crucial conclusions derived from in-depth analysis. However, due to the algorithm's conciseness and clarity, it becomes difficult for readers to link it to other existing methodologies and place it appropriately within the field. Therefore, a more thorough comparison or discussion with related work is necessary. For example, in the related work section, the authors have mentioned that their work is related to studies based on rejection/backtracking strategies.\n\nQUESTIONS:\n1. I am particularly interested in whether distributional RL algorithms exhibit the issues mentioned in the paper. Given that distributional RL methods take into account reward distribution during policy optimization (albeit with significant differences from the post-update reward distribution studied in this paper), can these methods produce smoother reward landscapes?\n\n2. Can the authors analyze the limitations of Algorithm 1? In particular, are the main conclusions and Algorithm 1 still valid for policies updated using different methods (beyond PPO, SAC, and TD3)?\n\n3. On average, how many times will Algorithm 1 reject an update during a single policy parameter update in actual operation? Will the efficiency of the algorithm be severely affected if the number of rejected updates becomes too high?\n\nLIMITATIONS:\nLimitations are not explicitly discussed by the paper. Some comments and questions regarding limitations of the work are discussed in the section above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the mapping from policy parameters $\\theta$ to returns $R(\\theta)$ and the sensitivity of $R(\\theta)$ to minor perturbations to $\\theta$. The authors show that popular RL algorithms tend to traverse \"noisy neighborhoods\" in policy parameter space where the return is highly varied even for small changes to $\\theta$. They find that policies with comparable average returns can have drastically different return distributions. Additionally, interpolating between policies from the same training run allows for transitioning from noisy to smooth return landscapes without degraded performance (the same is not true for policies from different runs). Finally, a practical algorithm is proposed which improves stability by rejecting gradient updates that lead to noisier return distributions.\n\nSTRENGTHS:\nOverall, I thought the paper was enjoyable to read and appears fairly novel (to the best of my knowledge). The figures are great and in general the story of the paper flows very well; Figure 2 is fantastic.\n\nThe landscape-inspired approach to deep RL is extremely interesting in my opinion. Finding novel ways to stabilize training by considering the local return landscape around the current set of policy parameters is highly relevant to scaling up and robustifying existing RL methods.\n\nI appreciate that the paper doesn't rely on overly technical jargon and, combined with the great illustrations, the paper is fairly easy to grasp without much math at all.\n\nWEAKNESSES:\nSection 4.1 makes pretty strong claims about interpolating between policies from the same run and from different runs, but it is only really substantiated by a few (six) qualitative examples. I suspect there is some way to quantify the claims in this section across a larger population of policy pairs, which I think would significantly strengthen this section.\n\nThe results in Figure 6 (and the description in Section 4.2) are a bit confusing to me. Is this just showing that the proposed algorithm succeeds in reducing LTP after 40 gradient steps? Why don't we compare the mean / variance of returns for complete training runs that do or don't use rejection? The 'x' markers show points reached by full training runs, so why can't we show that with/without the proposed approach? Or perhaps I'm misunderstanding, and Algorithm 1 is meant to by applied after a training run in order to robustify an already learned policy?\n\nI felt the experiments could be more comprehensive w.r.t environment diversity. Given that the main claims of the paper are substantiated only empirically, I think it is potentially a problem that the paper only considers a few DMC environments (ant, walker2d, halfcheetah, hopper) which all involve locomotion.\n\nQUESTIONS:\n\"For our analysis, we use 200 policies generated by different runs of TD3. From these we select pairs of policies with different post-update return distributions, as measured by their standard deviation or left-tail probability, but similar mean\" (L214-216).\nI don't fully understand how pairs of policies were selected using these criteria. Were these handpicked? Is there a more systematic way of picking pairs of policies? Is it possible for us to consider more than (a handful+1) examples?\n\nWhat is the claimed contribution of Algorithm 1? Are the authors claiming that this update scheme should be used practically? Is this just an illustration due to other considerations (e.g. computational cost).\n\nI think you should probably mention distributional RL [1] as related work.\n\nIn Figure 3, labeling the successful/failing trajectory either in the plot or in the caption would be nice.\n\n===\nI have read the rebuttal and bumped up my score slightly in light of the additional experimental results.\n\nLIMITATIONS:\nNo", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this study, the authors explore the underlying causes of the instability observed in the performance of deep RL policies. Their investigation revolves around examining the relationship between a policy and its corresponding return, with a focus on analyzing how minor variations in policy parameters impact the overall return. The authors posit that regions where even slight perturbations in policy parameters result in significant changes in return are susceptible to failures. To address this issue, they introduce an algorithm designed to navigate away from these volatile neighborhoods, thereby enhancing the robustness of the policy. Lastly, the authors provide a set of experimental results to demonstrate their findings and validate the efficacy of their optimization algorithm through another set of experimental results.\n\nSTRENGTHS:\nOverall Strengths:\n1. The organization and writing of this paper are of good quality. The authors have succeeded in presenting the content in a clear and easily understandable manner, which greatly enhances the readability of the paper. A notable aspect is the structure of the experiments sections (Section 3 and Section 4), where the authors first introduce the research questions, followed by a detailed explanation of the experimental setup, and finally present the results along with their corresponding conclusions. This approach significantly aids in the comprehension of the experiments, in such heavily experiments-based paper.\n2. I personally find the high-level idea of studying why the performance of RL policies so unstable well-worth pursuing.\n\nWEAKNESSES:\n1. This paper would benefit from providing additional explanation and motivation regarding the significance of investigating the mapping between policy parameters and return. A more thorough exploration of why this particular landscape is crucial to the focus of the work is needed. Expanding on the importance of understanding this relationship would enhance the reader's understanding and emphasize the relevance of the research.\n2. The experimental setup is limited. I would personally be interested in how different algorithms compare with each other when trained on the same task as well as how a certain algorithm behaves when trained on different tasks.\n\nQUESTIONS:\n1. What will the results look like among policies trained with different algorithms on the same task?\n2. I have seen recent works studying how the mapping between the state-space and the policy look like and how this landscape evolves during training. How do those findings relate to the findings in this paper? (For reference, look at “Understanding the Evolution of Linear Regions in Deep Reinforcement Learning” from NeurIPS’22)\n3. I personally find figure 2 hard to understand. But for now, I don’t have any particular suggestions to improve the readability of these.\n4. It would be invaluable if the same experiments would be repeated for policies trained with behavior cloning. To first see how policies trained with behavior cloning on some task compare to policies trained with other RL algorithms on the same task. And second, see whether different runs of behavior cloning demonstrate non-noisy return landscapes.\n\nLIMITATIONS:\nPlease refer to the previous sections.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper looks at visualizing the return landscape in typical continuous control environments of deep RL. The authors propose a statistic over the returns profile, namely the left tail probability, and try to present this as a suitable measure of policy quality in a series of experiments and visualizations. Finally, they propose a high-level rejection sampling process for the policy updates using the proposed statistic and argue that it can lead to similar-mean but less-variable return profiles in the trained policies.\n\nSTRENGTHS:\nThe idea of looking at return surfaces is not original (e.g., see [1], [2], and [3] among many others). However, trying to propose a remedy based upon such visualizations has been less common. In terms of quality, the paper tries to narrow down its focus to control and robotics applications, which is admirable. As for the clarity of the paper, too much theoretical complications did not hinder the point of the paper. Unfortunately, the results are less significant than expected, but the studied topic (i.e., better understanding of the return landscape for deep RL in control environments) is certainly of great importance and has been under-looked for too long.\n\n# References\n[1] Sullivan, R., Terry, J. K., Black, B., & Dickerson, J. P. Cliff Diving: Exploring Reward Surfaces in Reinforcement Learning Environments. ICML, 2022.\n\n[2] Bekci, Recep Yusuf, and Mehmet Gümüş. \"Visualizing the Loss Landscape of Actor Critic Methods with Applications in Inventory Optimization.\" arXiv preprint arXiv:2009.02391 (2020).\n\n[3] Ilyas, A., Engstrom, L., Santurkar, S., Tsipras, D., Janoos, F., Rudolph, L., & Madry, A. (2018). A closer look at deep policy gradients. ICLR 2020.\n\n[4] Hwangbo, Jemin, et al. \"Learning agile and dynamic motor skills for legged robots.\" Science Robotics 4.26 (2019): eaau5872.\n\n[5] Saleh, E., Ghaffari, S., Bretl, T., & West, M. (2022). Truly Deterministic Policy Optimization. Advances in Neural Information Processing Systems, 35, 8469-8482.\n\nWEAKNESSES:\n## Presentation and Writing\nFirst and for most, the paper lacks a clear set of contributions. I encourage the authors to dedicate the last paragraph of their introduction to list 3 or 4 main contributions, and commit to delivering them in the entire paper. This is a standard practice, and lacking this makes it extremely difficult to evaluate the merit of the work.\n\nSecond of all, the paper lacks a main message and a proper effort to convince the reader about the validity and the importance of it. As is, the manuscript looks more like a technical report draft rather than a conference paper. In particular, note that most NeurIPS papers focus on driving the main message in the initial sections, and avoid detailing the methods/data until the experiments section. However, this paper lacks such a structure, and tries to narrate a story by jumping from one experiment to another, while losing a cohesive context and spending a lot of space on detailing experimental details.\n\n## Rigor\nThe paper is not driven by any amount or kind of theory. Instead, the authors choose or abuse vague terms and intuitions to describe a series of hypotheses. For example, the notion of \"stability\" of a policy gets thrown around a lot in the paper (e.g., Line 246) without any mathematical standing or definition. This is despite the fact that the stability of a policy is a well-defined topic of research, and has many measures such as the Lyaponuv stability criterion. I doubt the authors were targeting this original notion of stability, and this is just one example of a technical term being abused.\n\nMany other vague terms are consistently used throughout the paper, including, but not limited to, (1) the \"quality\" of a policy, (2) \"noisy\" neighborhoods, (3) \"failure\" of a policy or a trajectory, (4) \"safer\" behavior, and so on.\n\n## Premise\n  1. Most deep Policy Gradient methods (e.g., PPO, TRPO, DDPG, TD3, SAC, etc.) optimize the mean of the discounted return in the MDP framework. Other statistics of the return distribution (e.g., the variance, the mode, left tail probability, etc.) are irrelevant to the optimization objective. When evaluating methods and policies, it is standard that higher mean payoffs are preferable even if the return distribution has higher variance.\n\n  If a particular application requires consistent returns from a policy, the user either has to (1) engineer the reward so that such variations lead to worse mean returns, or (2) use PG methods that optimize a different objective (e.g., the worst-case performance with min-max optimization methods). Both approaches are commonly used in control and robotics applications.\n\n  Of course, consistent policy performance is preferable but it shouldn't be confused with the main performance metric. The entire paper is written on the premise that variations in the return are inherently undesirable, and I don't necessarily agree with that. In my experience, Deep RL algorithms such as PPO, SAC, and TD3 gravitate towards such \"noisy\" regions at the end of the training because these regions can potentially yield higher returns than the stable areas, even if the return improvement is small.\n  \n  When evaluating such PG methods, only the mean return should be considered. The only way the paper's findings could be relevant is if high-variations in the return profile were shown to correlate with lower mean-returns at the end of training. I'm afraid the paper is not showing that.\n\n  2. Deep PG methods do not directly optimize deterministic policies. Each algorithm, even the deterministic ones such as DDPG and TD3, have an exploration parameter $\\sigma$. These parameters cause a filtering/convolutional effect on the return landscape, and smoothen the effective optimization objective. The paper mainly focuses on oscillations in the deterministic landscape, whereas the effectively optimized landscape can be much smoother and I'm not sure the same findings are applicable to the filtered landscape.\n\n## Arguments\n\nThe experiments and figures in the paper aren't always convincing or helpful. In particular, \n\n  1. Figure 5 doesn't prove a meaningful hypothesis. Such parameter interpolation behavior may not even be specific to RL problems, and \"non-noisy\" problems can also exhibit the same behavior. In other words, interpolating parameters within the same run in a deep classification task can lead to reasonably-performing parameters while inter-run interpolations wouldn't perform well. I'm not sure how this observation is a solid evidence for the existence of special patterns or connections, even if there were many gradient steps in between interpolated parameters.\n\n  2. Figure 1 consumes a lot of space, but carries little information. \n\n  3. Also, the paper spends a lot of time speculating. For instance, Lines 49-52, Lines 236-243, and Lines 202-207 are a few examples. Line 245 speculates about the existence of \"paths\" (I'm not even sure what it means), and such claims about high-dimensional optimization problems cannot be easily proven by the experiments conducted in this paper.\n\nQUESTIONS:\nI would like the authors to address my concerns in the weakness section regarding the premise of the paper and the made arguments.\n\nLIMITATIONS:\nGiven the the recent reproducibility issues with deep RL, papers in this are must be held to a higher standard. In particular, for this paper: \n  1. Many depicted training curves are based upon single trainings.\n  2. 20 seeds were used for running the experiments. I encourage the authors to run the experiments for at least a 100 seeds.\n  3. Most figures lack any information about the uncertainty of the statistics. Finding ways to incorporate confidence intervals in the plots is important for this paper.\n  4. The number of environments considered in the paper are limited. Hopper, Walker, Half-Cheetah, and Ant were shown in the main paper, while there are many more environments in the standard gym benchmarks.\n  5. The quality of the environments in the paper could also be improved. In particular, while standard gym benchmarks are a good start, they do not depict realistic robotic artifacts. There are many realistically modeled robotic environments, such as the ones used in [4] and [5], that can be better representatives for validation of findings.\n  6. While the paper is specifically targeting continuous control, the authors never discuss the physical aspects of each environment and do not make physical variations to the underlying physical models.\n  7. The code is not shared open-source in the paper. While this is not a requirement for the conference, it is certainly of great important to the RL community. I encourage the authors to release an open-source link to the code in the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

comments: SUMMARY:
This work proposes a new perspective for investigating the stability of RL algorithms -- the return distribution of post-update policies. A simple rejection technique is developed correspondingly to increase the stability in that sense.

STRENGTHS:
The paper is very well written and easy to follow. This new perspective of return distribution of post-update policies appears novel and makes much sense. I can imagine many followup directions of this work. The rejection technique is neat while effective.

WEAKNESSES:
There is indeed a few weaknesses in the treatment of the problem.
1. Why do the authors limit the investigation to deterministic transitions and continuous actions? In my understanding the concept the authors propose apply to stochastic environments and discrete actions as well.
2. Maybe it's better to rename the work as "Policy Optimization in a Noisy Neighborhood: On Return Landscapes in MuJoCo" since the work in its current form does not have any other domains. In other words, it is interesting to see whether the observation is general across various domains, especially the stability of interpolation of policies inside a single run. 

That being said, I believe the novelty of this work is significant and am happy to increase my score based on the authors' response to my concern.

QUESTIONS:
see above

LIMITATIONS:
see above
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-18