SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Nate Rahn*
Pierluca D'Oro*
Harley Wiltzer
Marc G. Bellemare
Deep reinforcement learning agents for continuous control are known to exhibit significant instability in their performance over time. In this work, we provide a fresh perspective on these behaviors by studying the return landscape: the mapping between a policy and a return. We find that popular algorithms traverse noisy neighborhoods of this landscape, in which a single update to the policy parameters leads to a wide range of returns. By taking a distributional view of these returns, we map the landscape, characterizing failure-prone regions of policy space and revealing a hidden dimension of policy quality. We show that the landscape exhibits surprising structure by finding simple paths in parameter space which improve the stability of a policy. To conclude, we develop a distribution-aware procedure which finds such paths, navigating away from noisy neighborhoods in order to improve the robustness of a policy. Taken together, our results provide new insight into the optimization, evaluation, and design of agents.
It is well-documented that agents trained with deep reinforcement learning can exhibit substantial variations in performance – as measured by their episodic return. The problem is particularly acute in continuous control, where these variations make it difficult to compare the end product of different algorithms or implementations of the same algorithm [11, 20] or even reliably measure an agent's progress from episode to episode [9]. A recurring finding is that simply averaging the return produced by a set of policies may be insufficient for rigorous evaluation.
In this paper, we demonstrate that high-frequency discontinuities in the mapping from policy parameters  $\theta$  to the return  $R(\theta)$  are an important cause of return variation. As a consequence of these discontinuities, a single gradient step or perturbation to the policy parameters often causes important changes in the return, even in settings where both the policy and the dynamics are deterministic. Because an agent's parameters constantly change during training and should be robust to minute parametric perturbations, we argue that the distribution of returns in the neighborhood of  $\theta$  is in fact a better representative of its performance, both from an evaluation and an optimization perspective.
Noisy neighborhoods in the return landscape. We call the return landscape the mapping from  $\theta$  to  $R(\theta)$ , our main object of study. We show that the return often varies substantially within the vicinity of any given  $\theta$ , forming what we call a noisy neighborhood of  $\theta$ . Based on this observation, we demonstrate the usefulness of studying the landscape through the distribution of returns obtained from small perturbations of  $\theta$ . In the important case where these perturbations result from a single gradient step, we call the resulting object the post-update return distribution.
*Equal contribution. Correspondence to {nathan. rahn, pierluca. doro}@mila. quebec.
Figure 1: A visualization for two policies visited by SAC in the hopper environment. We show the return landscape in their proximity, their post-update return distributions, and the visual appearance of their learned gaits. We plot the mean of each return distribution as an orange line. Despite featuring a similar level of return, we observe that the policy in the noisy neighborhood performs an unstable curved gait which is faster but more prone to failure, as visible in the thick left tail of the post-update return distribution.
Diversity in equally-performing policies. We show that different neighborhoods correspond to different post-update return distributions and agent behaviors. We discover that at equal average returns, different policies obtained by the same deep RL algorithm may in fact have substantially different distributional profiles, as measured by statistics of the post-update return distribution. Moreover, we uncover that many of these distributions are long-tailed and we find the source of these tails to be sudden failures from an otherwise successful policy.
Effect on learning dynamics. We expose how the transition between noisy and smooth parts of the landscape happens. Surprisingly, while large valleys of low return are visible when linearly interpolating between similarly performing policies from different runs, we show no such valleys typically exist between policies from the same run. Based on this insight, we show that it is possible to find an improved set of parameters  $\theta$  that achieves comparable return, but substantially lower post-update variation.
We believe the phenomenon we study is central to deep reinforcement learning in continuous control. Beyond its effect on learning dynamics (for example, through increased variance and implicit exploration [39]), it is also a potential driver of instability in sim2real settings, even in the face of seemingly small environmental changes. Additionally, it suggests that one should not simply deploy the policy obtained at the end of a training run, and that further post-training tuning may be beneficial.
In reinforcement learning, an agent interfaces with an environment. In this paper, we are interested in continuous control environments modelled as a finite-horizon Markov Decision Process (MDP)  $\mathcal{M} = \langle S, A, r, f, T, \rho_0 \rangle$ , where  $S \equiv \mathbb{R}^n$  is the state space,  $A \equiv \mathbb{R}^m$  is the action space,  $r: S \times A \to \mathbb{R}$  is a reward function,  $f: S \times A \to S$  is a deterministic transition function,  $T$  is the horizon, and  $\rho_0 = \mathcal{U}(s_I - \beta, s_I + \beta)$  is an initial state distribution with  $s_I$  being an initial reference state, and  $\beta \in \mathbb{R}^n$  an environment-dependent parameter. We assume that each agent produces a stationary Markovian deterministic policy  $\pi_\theta: S \to A$  within a parametrized family  $\Pi_\Theta = \{\pi_\theta : \theta \in \Theta \subset \mathbb{R}^d\}$ . In an episodic setting, the interaction of the agent with the environment with a given policy  $\pi_\theta$  from some state  $s \in S$  produces a trajectory in the environment and,
consequently, a return:
$$
G _ {\boldsymbol {\theta}} (s) = \sum_ {t = 1} ^ {T} r \left(s _ {t}, a _ {t}\right) \tag {1}
$$
$$
s. t. s _ {t} = f \left(s _ {t - 1}, a _ {t - 1}\right), a _ {t} = \pi_ {\boldsymbol {\theta}} \left(s _ {t}\right), s _ {1} = s.
$$
We are interested in understanding how small changes to the policy parameter affect the associated return. To this end it is sufficient to study the return from the reference state  $s_I$  (in Appendix A.4 we show that similar effects occur across the state space). The return landscape is our main object of study.
Definition 2.1 (Return Landscape). The return landscape is the mapping from policy parameters to return, starting from the initial reference state:
$$
R (\boldsymbol {\theta}) = G _ {\boldsymbol {\theta}} \left(s _ {I}\right). \tag {2}
$$
Figure 1 (left) depicts small portions of the return landscape for a particular environment and policy parametrization (we describe the visualization procedure below).
In this work, we will use the policies discovered by popular algorithms to characterize the topology of the return landscape. We focus on policy-based deep reinforcement learning algorithms for continuous control, such as Soft Actor-Critic (SAC) [19], Twin-Delayed DDPG (TD3) [16], and PPO [42] which use neural network function approximators to represent the policy. Such algorithms learn good behavior in the environment by maximizing the discounted return. In the process, they produce a sequence of policies
$$
\theta_ {0}, \theta_ {1}, \dots , \theta_ {N}, \text {s . t .} \theta_ {t + 1} = u \left(\theta_ {t}, X _ {t}\right) \text {f o r a l l} t, \tag {3}
$$
where  $\mathfrak{u}:\Theta \times \mathbb{R}\to \Theta$  is the algorithmic policy update function, and  $X_{t}$  is some random variable abstracting the stochasticity inherent to the update. For example, SAC and TD3 construct parametric updates by sampling a small number of transitions (minibatches) from their replay buffer [29, 31].
3 A Distributional View on Return Landscapes
The return landscape arises from the interaction between an environment and a class of parameterized policies. We first consider how this landscape varies in the immediate vicinity (or neighborhood) of policies produced by deep reinforcement learning algorithms. Given a reference policy, a natural choice is to consider how the return is affected by single updates to the policy parameters. To this end, we view the collection of possible returns obtained by evaluating the updated policy as a distribution over returns; as we will see, this distribution widely varies across the return landscape.
Definition 3.1 (Post-Update Return). Let  $\Pi_{\Theta}$  be a parametric space of deterministic policies and  $u$  an update function. Given  $\pi_{\theta} \in \Pi_{\Theta}$ , its post-update return is defined as:
$$
\mathcal {R} (\boldsymbol {\theta}) = R \left(\mathrm {u} (\boldsymbol {\theta}, X)\right), \quad X \sim P, \tag {4}
$$
where  $P$  is a an algorithm-dependent source of stochasticity.
The post-update return inherits randomness from the underlying training algorithm and it is thus a random variable. Clearly, a post-update return will have an associated policy and trajectory, which are in turn random variables. In this work, we will leverage the distribution of post-update returns as a tool to investigate the properties of neighborhoods of the return landscape.
The different panels of Figure 1 illustrate how the return landscape in the neighborhood of  $\theta$  translates into different post-update return distributions. Here, the return landscape is visualized along two update directions computed by the training algorithm based on two different batches sampled from its replay buffer, such that 1.0 on each axis corresponds to a single parameter update in that direction (details in Appendix A.2). The middle panel shows the corresponding post-update return distribution estimated using 10000 samples. We find that the distribution from the noisy neighborhood (top) exhibits a significant left tail, while the distribution from the quieter neighborhood (lower) is concentrated around its mean. On the right, we illustrate the gait produced by the reference policies (the origin in the left panel). We find qualitatively that the policy in the noisy neighborhood exhibits a curved gait which is sometimes faster, but unstable, whereas the policy in the smooth neighborhood produces an upright gait which can be slower, yet is very stable. We include similar evidence for other environments in Appendix A.11.
Figure 2: A scatter plot showing mean return and standard deviation, skewness or left-tail probability of the post-update return distribution of policies produced by three popular deep RL algorithms on the ant Brax task. Each point corresponds to a given policy's post-update return distribution, with six selected policies highlighted by star markers showing a range of diverse distributions.
3.1 Post-Update Return Distributions as a Characterization of the Return Landscape
The mean of the post-update distribution naturally captures the average behavior represented by an algorithm as it traverses a given neighborhood. We further characterize this distribution by measuring its standard deviation (a measure of spread around the mean) and its skewness (a measure of asymmetry). In our context, a negative skewness describes a distribution with a heavy left tail, similar to the one shown in Figure 1. Such a tail is especially interesting to us as it indicates lower-than-expected returns. However, we find that skewness is not directly interpretable as a numerical quantity. To capture these tails interpretably, we introduce a metric we call left-tail probability. The left-tail probability of a random variable  $Y$  is defined as
$$
\operatorname {L T P} _ {\alpha} (Y) = \mathbb {P} [ 0 \leq Y <   \alpha \cdot \operatorname {m o d e} (Y) ]. \tag {5}
$$
This quantity satisfies some desirable properties within the context of our study. First, it uses the mode of the distribution as a reference value. This is by contrast with the mean of the distribution, which may not correspond to the "majority" behavior (as illustrated in the top half of Figure 1). It also allows us to more easily compare the tailedness of distributions generated from policies of widely varying returns. Second, it is an easily-interpretable quantity which measures the total probability mass falling in the left tail. For simplicity, here we assume that  $Y$  is positive, noting the idea can be naturally generalized to random variables bounded below. In our analyses we write  $\mathrm{LTP} \equiv \mathrm{LTP}_{1/2}$  to measure drops from the mode of the post-update return distribution of at least  $50\%$ . In practice, we estimate the LTP by leveraging the Chernoff estimator [10], computing the mode as the midpoint of the interval of the most populated bin in a 100-bin histogram.
Equipped with these metrics, we measure the mean and the other three statistics of the post-update return for a set of 600 policies produced, across trials and iterations, by three popular deep RL algorithms (TD3, SAC and PPO). We use 20 seeds per algorithm and 10 checkpoints per seed, for a total of 200 policies per algorithm. These checkpoints are equally-spaced in time in training runs of 1 million steps for TD3 and SAC and 60 million steps for PPO. Each of the 600 distributions is estimated by performing 1000 independent updates to the starting policy and then rolling the resulting deterministic policy out in the environment for 1000 time-steps to compute its return. Each update is different due to a different batch sampled from the replay buffer for TD3 and SAC, and to a different batch of data from the environment collected by a randomly-perturbed policy for PPO. This amounts to millions of policy evaluations for which, for computational reasons, we primarily use the easily parallelizable environments from the Brax simulator [15]. We also include similar results on the post-update return distributions of policies trained on DeepMind Control Suite [44] and on games from the
Figure 3: A visualization of how failures occur in the halfcheetah and walker2d tasks. The left subplots compare the reward-per-timestep obtained by a successful and failing trajectory generated by two policies in the same noisy neighborhood. The right subplots show the simultaneous evolution of returns for 10 such trajectory pairs (that can be thought of as a race to collect the most rewards), with the trajectory pair from the left indicated by a matching star marker. The right subplots indicate that policies from the same neighborhood behave similarly (diagonal segments of the curve) until the failing policy makes a sudden misstep and collects low rewards (horizontal segments).
ALE [7] in Appendix A.5 and A.6. Additional experimental details, including the hyperparameters and implementations used for running these algorithms, can be found in Appendix A.1.
Figure 2 illustrates how different policies produced by deep RL algorithms correspond to a wide range of post-update return distributions, as measured by our chosen metrics  ${}^{2}$  . For each metric,we report the bootstrapped mean using 1000 resamples to account for sampling error in the post-update returns collected for a given policy, and omit the corresponding bootstrapped confidence intervals for visual clarity, as they are very small. In particular, this scatter plot shows that different policy parameters achieve similar levels of returns (as measured by the distribution mean) but a wide range of possible levels of variability, as measured by standard deviation, skewness and left-tail probability. This suggests, in a similar way to the example shown in Figure 1, that algorithms discover behaviors which can be qualitatively very different from one another, and that leveraging the post-update return distribution can offer a new lens to investigate different dimensions of policy quality.
These results suggest that simply optimizing the mean return of a policy might ignore its distributional aspect. In particular, a practitioner will likely prefer, for a given level of return, a policy featuring a post-update return distribution with lower levels of standard deviation or left-tail probability. Intuitively, such a policy may correspond to a safer behavior, both able to more robustly accommodate additional updates from its training algorithm and possibly to deal with other unexpected sources of perturbation during deployment.
3.2 Analyzing Failures
One characteristic feature of the post-update distributions studied above is the existence of a significant lower tail for many policies visited by the three deep RL algorithms TD3, SAC and PPO. This is visible in their skewness, but especially in their left-tail probability, which demonstrates that many policies produce returns which are unexpectedly poor after up to roughly  $10\%$  of updates. We now take a closer look at the specific mechanism by which small changes in an agent's actions results in a wide range of returns in continuous control.
Our experimental procedure is as follows. For each environment, we randomly select 10 policies from the logged checkpoints of 20 independent runs of TD3, conditioned on the fact that the policy has a left-tail probability which is greater than zero. These are policies that we know are prone to poor returns following an update. For each policy, we compute the post-update return distribution by collecting trajectories in the environment after a single update to the original policy. According to this procedure, we identify two trajectories drawn from the neighborhood around the policy: a successful trajectory, characterized by a return within  $10\%$  of the mean of the post-update distribution, and a failing trajectory, characterized by a return of less than  $50\%$  of the mode of the post-update distribution, as in the left-tail probability.
Note that the LTP is not properly defined for a small number of policies achieving negative return, that appear as points on the right border of the scatter plot.
Our goal is to understand the differences between these successful and failing trajectories in order to explain how long-tail returns occur. To this end, Figure 3 depicts two views of the trajectory data. For each environment, the left subplot considers a single pair of successful/failing trajectories corresponding to one of the chosen policies, and plots the reward per timestep earned in these two trajectories. These results suggest that the failing policies which make up the tail of the post-update distribution are capable of collecting similar rewards to the successful policies, yet are prone to missteps which result in episode termination (as in walker2d) or transition to a low-reward, quasi-absorbing state (as in halfcheetah). Figure 4 shows an example of such a misstep in walker2d.
We present a broader view of these observations through the right subplots, per-environment, in Figure 3. Here, we plot each of the trajectory pairs as a parametric curve in time. For both the successful and failing trajectories, we compute the return up to time  $t$ ,  $R_{\leq t} = \sum_{i=1}^{t} r(s_i, a_i)$ . Then, for each value of  $t$ , we plot  $R_{\leq t}$  for the successful and failing trajectories as a point on the curve, allowing us to visualize the simultaneous evolution of both trajectories.
Figure 4: The trajectory of a successful (top) and failing (bottom) policy, both coming from the same post-update distribution in walker2d. They exhibit a similar gait until right before the failure.
We assume that  $R_{\leq t + 1} = R_{\leq t}$  when the length of one trajectory exceeds the other, that is, no additional reward is collected after the trajectory terminates. The resulting visualization reveals several notable findings. First, we show that nearly all trajectory pairs begin by following the line  $y = x$ , indicating that the respective policies collect rewards at almost exactly the same rate. Next, we observe that many curves rapidly diverge from this line to horizontal, indicating that the failing trajectory suddenly starts collecting little to no reward, while the successful trajectory continues. In walker2d, these divergences reflect sudden terminations of the episode, represented by horizontal lines. In half cheetah, which does not terminate, we see that instead the failing agent gets stuck in low-reward absorbing states, but is sometimes able to recover and go back to collecting reward at the same rate as the successful trajectory. We include similar visualizations for the hopper and ant environments in Appendix A.8, which support the same conclusions.
Taken together, these results demonstrate that some policies exist on the edge of failure, where a slight update can trigger
the policy to take actions which push it out of its stable gait and into catastrophe. Indeed, when we compare the gaits learned by policies of high left-tail probability to those which are more well-behaved under updates, we observe that the behaviors of the former are qualitatively more unstable (Figure 1, with more examples in Appendix A.11).
4 Navigating Return Landscapes
In the previous section, we took a fine-grained look at the return landscape, using post-update return distributions to characterize the neighborhood of different policies learned by deep RL algorithms. We now consider this landscape on a more global scale, specifically how the agent's return changes as one interpolates between different policies.
4.1 Connectivity in the Return Landscape
For our analysis, we use 200 policies generated by different runs of TD3. From these we select pairs of policies with different post-update return distributions, as measured by their standard deviation or left-tail probability, but similar mean. Consider two sets of policy parameters  $\theta_{1}$  and  $\theta_{2}$ , for which the post-update return distribution implied by  $\theta_{1}$  has higher LTP than the implied by  $\theta_{2}$ . We linearly interpolate these two to form a family of parameters  $\theta = \alpha \theta_{1} + (1 - \alpha)\theta_{2}, \alpha \in [0,1]$ . For each such  $\theta$ , we then record the return  $R(\theta)$  obtained by a single simulation with the corresponding policy.
In Figure 5, we show the result of this interpolation for six pairs of policies in the hopper and walker2d environments, in two distinct cases. In the first case, the two policies have been produced by the same run of TD3 (i.e., starting from the same initialization and history of batches); in the second case, the two policies have been generated by independent repetitions of the algorithm. The plot shows interesting information about the global structure of the return landscape: the interpolation
Figure 5: Return of the policies obtained by linear interpolation of the parameters of policies of approximately the same level of return in the hopper and walker2d environments. The neighborhoods traversed transition from being noisy to being smooth; policies from the same run are connected by paths with no valleys of low performance in the return landscape, even if separated by hundreds of thousands of updates (i.e., at least  $1 \times 10^5$  steps for all pairs of policies from the same run).