SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 11: A scatter plot showing mean return and standard deviation, skewness or left-tail probability of the post-update return distribution of policies produced by three popular deep RL algorithms on the halfcheetah Brax task.
Figure 12: A scatter plot showing mean return and standard deviation, skewness or left-tail probability of the post-update return distribution of policies produced by three popular deep RL algorithms on the hopper Brax task.
Figure 13: A scatter plot showing mean return and standard deviation, skewness or left-tail probability of the post-update return distribution of policies produced by three popular deep RL algorithms on the walker2d Brax task.
Figure 14: A scatter plot showing mean return and standard deviation, skewness or left-tail probability of the post-update return distribution of policies produced by two popular deep RL algorithms on the quadruped-walk DeepMind Control task.
Figure 15: A scatter plot showing mean return and standard deviation, skewness or left-tail probability of the post-update return distribution of policies produced by two popular deep RL algorithms on the quadruped-run DeepMind Control task.
our paper: we evaluate 10 policies evenly-spaced across training, and perform 1000 independent updates to each policy. We find that post-update return distributions computed for these environments still exhibit a remarkable degree of variation, as captured by their standard deviation (see Figure 21). At the same time, the shape of the resulting distributions can be quite different compared to the ones observed in robotic locomotion tasks, and, thus, it is not necessarily described in a rich way by metrics such as the LTP (see Figure 20).
Figure 16: A scatter plot showing mean return and standard deviation, skewness or left-tail probability of the post-update return distribution of policies produced by two popular deep RL algorithms on the humanoid-stand DeepMind Control task.
Figure 17: A scatter plot showing mean return and standard deviation, skewness or left-tail probability of the post-update return distribution of policies produced by two popular deep RL algorithms on the humanoid-walk DeepMind Control task.
Figure 18: A scatter plot showing mean return and standard deviation, skewness or left-tail probability of the post-update return distribution of policies produced by two popular deep RL algorithms on the humanoid-run DeepMind Control task.
We also run an interpolation experiment on the ALE to investigate whether the connectivity phenomenon studied in Section 4.1 generalizes to discrete-control environments. We produce quantitative results on the phenomenon with the following procedure. For each one of the four games, we sample a set of at least 20 pairs of policies from same runs and 20 pairs of policies from different runs of the algorithm. Then, we linearly interpolate between policies in the pairs, producing 50 intermediate policies, and randomly perturb them using Gaussian noise with standard deviation 0.0003 to obtain an estimate of the mean of their (random) post-update return distribution. Then, for each pair of policy, we measure how frequently the return collapses in between the two extremes, by counting
Figure 19: Proportion of return collapses when interpolating between randomly-sampled policies produced by either the same or different runs in the ALE (QBert, Breakout, Ms . Pacman, Seaquest). Far fewer return collapses are observed when interpolating between policies produced by the same run. Results are aggregated over environments with  $95\%$  bootstrapped C.I..
Figure 20: Histograms of the post-update return distribution for policies obtained by PPO on 4 Atari environments. These distributions were selected for illustrative purposes to demonstrate that 1) returns can vary significantly after a single policy update (see also Figure 21) and 2) the distributions obtained can exhibit varied profiles.
how many times it becomes less than  $10\%$  of the minimum return of the two original policies. We then average this Below-Threshold Proportion across pairs, and across environments using riable [1]. Figure 19 shows that the phenomenon, properly quantified, is still present when using a very different class of environments (discrete-action, game-based).
A.7 Behavior cloning
While the results in the main paper demonstrate that policies learned by commonly-used RL algorithms are sensitive to updates, we were curious to understand whether behavior cloning produces policies which occupy less noisy neighborhoods of the return landscape. To do so, we conducted a set of additional experiments on 4 Brax environments. The protocol was as follows: For each environment, we consider 10 independent training runs of TD3, and 5 policies distributed evenly throughout the run. For each of these policies, we train a new agent using behavior cloning on the data logged up until the collection time of the teacher policy, replacing the actions in the dataset with the actions of the teacher policy, for 1 million gradient steps. We log 10 policies throughout each training run of behavior cloning. To compute the post-update return distribution for the policies obtained by behavior cloning, we used one additional gradient step on the MSE-based BC objective, and 1000 samples.
In Figure 22 (left), we compare statistics of the post-update return distributions for all pairs of policies  $(\pi_{i,j}^{BC}, \pi_i^{TD3})$  where policy  $\pi_{i,j}^{BC}$  is obtained by behavior cloning  $\pi_i^{TD3}$ . We compute the Pearson correlation coefficient of statistics of the post-update return distributions of these policies: between the mean of each pair, and between the LTPs of each pair. We find that the means are highly correlated and that the learned BC policies are comparable in performance to their teacher policy. We additionally show that correlation in the LTP is much more variable across environments – in general, cloning a policy of high or low LTP does not always lead to a cloned policy of the same LTP.
But does training policies by BC produce policies occupying less noisy neighborhoods of the return landscape, which would have correspondingly lower LTP overall? In Figure 22 (right), we show the mean LTP across policies and environments, along with its  $95\%$  bootstrapped confidence interval following the recommendations from riable [1]. Our results demonstrate that BC does not produce fundamentally more stable policies, as measured by the LTP.
Figure 21: Scatter plots of the mean and standard deviation of the post-update return distribution of policies learned by PPO across four Atari environments. In the majority of cases, the post-update return distribution exhibits significant variability relative to its mean, indicating that learned policies in Atari are also sensitive to single updates.
corrmean	corrLTP
ant	0.994	0.763
halfcheetah	0.995	0.976
walker2d	0.973	0.218
hopper	0.753	0.449
Figure 22: Comparing pairs of policies obtained by behavior cloning a given TD3 policy. Left: BC learns policies which are well-correlated in mean return with the cloned policy, measured by Pearson's correlation coefficient. The correlation in LTP is more variable across environments, indicating that cloning a given policy does not always produce a new policy of the same LTP. Right: Average LTP of policies learned by TD3 and BC, aggregated across environments. Overall, BC learns policies of similar LTP to policies learned by TD3. Results are aggregated over environments with  $95\%$  bootstrapped confidence interval.
A.8 Complete results on comparing successful and failing trajectories
In Figure 23, we include similar results to Figure 3 for all the Brax environments we study.
Figure 23: A visualization of how failures occur on four Brax tasks. The plots show the simultaneous evolution of returns for 10 trajectory pairs, of one successful and one failing drawn from the same post-update policy/trajectory distribution. We observe that these policies from the same neighborhood behave similarly (diagonal segments of the curve) until the failing policy makes a sudden misstep and collects low rewards (horizontal segments). At divergence, we observe that the failing policy can terminate, collect no reward, or even collect negative reward, yet it sometimes recovers and goes back to matching the performance of the successful policy.
A.9 Justification of rejection procedure
In Algorithm 1, we proposed a procedure that rejects gradient updates based on estimates of the CVaR of the resulting policies' post-update returns. Since our goal with this procedure is ultimately to decrease the LTP of an existing policy, one might rightfully question why CVaR was our heuristic of choice. Our reasoning for this choice was simply that the LTP can be minimized by arbitrarily poor policies: that is, policies that only achieve "low enough" returns will have very low LTP, because they cannot substantially fail relative to their existing performance. Such an issue can be circumvented by comparing the CVaR of the post-update returns: among all policies that have the same left-tail
walker2d
Figure 24: Experimental results for Algorithm 1 on 4 Brax domains. Each empty circle denotes the starting policy, while the filled circle at the end of the arrow shows the policy obtained after application of Algorithm 1. X's show other policies obtained by TD3, for context.
probability, the CVaR is still able to order the policies by the quality of the returns in the left tail. To further justify the use of CVaR as a rejection heuristic, we visualized how different CVaR heuristics order policies with respect to their post-update return distributions. Figure 25 illustrates how the  $\alpha$ -CVaR ranks policies. Note that  $\alpha$ -CVaR in this instance is effectively measuring the mean among the post-update returns below the  $\alpha$ -quantile of the post-update return distribution, so 0-CVaR would measure the worst-case post-update return and 1.0-CVaR is equivalent to the mean of the post-update returns.
As expected, we see that ranking by the mean is relatively agnostic to the LTP of the policies, but for  $\alpha < 1$ , ranking by CVaR is effectively trading off larger mean post-update returns for lower LTP. Notably, the gradient of colors (rankings) in the 0.01-CVaR column is distinctly "top-left to bottom-right", indicating that only policies with both high returns and low LTP are ranked favorably. The 0.1-CVaR column is fairly similar, but the gradient is more vertical - that is, it is more forgiving with respect to LTP than the 0.01-CVaR heuristic, but it still clearly prefers policies with lower LTP among those with similar levels of return. Finally, the gradient in the rightmost column is exactly vertical, meaning that the ranking is agnostic to LTP.
Consequently, we argue that CVaR is a reasonable heuristic for identifying robust policies without neglecting their performance level. In our experiments, we found that the 0.1-CVaR heuristic for rejecting gradient updates struck the best balance between robustness and performance.
A.10 Additional post-update-CVaR rejection results
In Figure 24 we include the results from our experiments with Algorithm 1 in four Brax domains for policies found by TD3.
A.11 Visualization of behaviors
We uploaded some visualization of the behaviors on a website (https://sites.google.com/view/on-return-landscapes/home): policies of similar mean for the post-update return but different standard deviation exhibit different behaviors.
Figure 25: Rankings of policies by the  $\alpha$ -CVaR of their post-update returns. Points with darker colors are ranked higher, and stars denote the highest ranked policy.
A.12 Evidence of multi-scale structure
To investigate whether the shape of a neighborhood heavily depends on the size of the steps used for exploring it, we "zoom-in" on the landscape plot from Figure 1 by using the same granularity but on a smaller window. We apply this process twice, with the results in Figure 27 giving evidence for scale-independence.
Rejection rates by environment
Figure 26: Frequency of rejections during execution of Algorithm 1 across environments. Bars depict the mean across starting policies and runs, and the error bars represent standard error.
Figure 27: Multiscale version of the plot in Figure 1. Note the similarity between the original and the more granular versions, giving evidence of the independence of the general shape of a neighborhood from the resolution at which it is visualized.