SlowGuess/NeurIPS_2023_OCR huggingface dataset:
process traverses different parts of the landscape, highlighting a transition between a noisy part of the landscape to an inherently smoother one. Interestingly, the interpolations between policies from the same run and from different runs exhibit very different qualities. When interpolating between policies of different runs, the process traverses entire regions of the landscape of poor return, until the point in which it gets to the neighborhood of the second policy. By contrast, when interpolating between policies from the same run, the transition from a noisy to a smooth landscape happens without encountering any valley of low return – even when these policies are separated by hundreds of thousands of gradient steps in training. This is particularly surprising given that  $\theta$  is a high-dimensional vector containing all of the weights of the neural network, and there is no a priori reason to believe that interpolated parameters should result in policies that are at all sensible.
Figure 6: Proportion of return collapses when interpolating between randomly-sampled policies produced by either the same or different runs in Brax. Far fewer return collapses are observed when interpolating between policies produced by the same run. Results are aggregated over all environments with  $95\%$  bootstrapped C.I and 500 pairs of policies.
To further quantify the phenomenon, we want to measure the proportion of return collapses encountered when interpolating between policies. We use the following experimental design. We sample for each environment a set of 500 pairs of policies from the same runs and a set of 500 pairs of policies from different runs. Then, we linearly interpolate between policies in the pairs, producing 100 intermediate policies, and randomly perturb them using Gaussian noise with standard deviation  $3 \times 10^{-4}$  to obtain an estimate of the mean of their (random) post-update return distribution. Then, for each pair of policies, we measure how frequently the return collapses in between the two extremes, by counting how many times it becomes less than  $10\%$  of the minimum return of the two original policies. We then average this Below-Threshold Proportion across pairs, and across environments using riable [1].
Figure 6 shows that there is on average almost no drop in return when interpolating among policies from the same run. We additionally report similar results on four ALE games in Appendix A.6.
We hypothesize this might be interpreted as each individual run of the algorithm specializing on a different family of behaviors, for which, due to the geometry of the return landscape, interpolation between policy parameters does not have any disrupting effect. This result can be interpreted as being related to linear mode connectivity [17, 18], a phenomenon observed in supervised learning, for which different points in the loss landscape of neural networks can be connected by near-constant-loss paths. In other words, it appears there is typically no barrier of low average return separating policies generated from the same run, even when those policies feature very different levels of stability. The existence of such a phenomenon in the RL setting is far from certain: the optimization objective is non-stationary and the evaluation metric (the return instead of the loss) depends on an environment and multiple forward passes from a neural network.
Figure 7: LTP reduction over 40 gradient steps without rejections (TD3) and with rejections (Algorithm 1). Data is aggregated over starting policies, environments, and five independent runs for each starting policy. We see that Algorithm 1 is strictly superior to TD3 with respect to LTP reduction. Results are aggregated over environments with  $95\%$  bootstrapped confidence interval.
Algorithm 1: Post-Update-CVaR Rejection
4.2 Stabilizing Policies by Navigating the Landscape
Overall, Figure 5 demonstrates the existence of paths in the return landscape which are able to increase the level of stability of a given policy, but are not necessarily followed in a spontaneous way by typical policy optimization algorithms. In the absence of a desirable end policy to interpolate towards, we would like to understand if it is possible to find similar stabilizing paths (as measured by the LTP), given a starting policy inhabiting a noisy neighborhood of the return landscape. We conjecture that this is feasible by filtering the policy updates produced by an algorithm: In particular, we propose to reject gradient updates that lead to policies with less favorable post-update return distributions.
In our procedure, which is outlined in Algorithm 1, we use the CVaR as a heuristic to compare the stability of post-update return distributions<sup>3</sup>, as it is effectively a measure of the left tail mean [38]. Our procedure works as follows: starting with a given policy, we use TD3 to interact with the environment, maintain a replay buffer, and compute updates to the policy and critic parameters. However, before applying a proposed update, we first
Input: Initial policy parameter  $\theta$ , CVaR level  $\alpha$ , policy update function  $u$ , number of MC samples  $N$ , tolerance  $\delta$   
while True do  
 $\{R_{\theta}^{(k)}\}_{k=1}^{N} \stackrel{\mathrm{iid}}{\sim} \mathcal{R}(\theta)\{\text{Post-update return samples}\}$ $B \gets$  random minibatch  
 $\pmb{\theta}' \gets \mathfrak{u}(\pmb{\theta}, B)$ $\{R_{\pmb{\theta}''}^{(k)}\}_{k=1}^{N} \stackrel{\mathrm{iid}}{\sim} \mathcal{R}(\pmb{\theta}')$   
if CVaR $_{\alpha}(\{R_{\pmb{\theta}''}^{(k)}\}_{k=1}^{N}) \geq (1-\delta) \text{CVaR}_{\alpha}(\{R_{\pmb{\theta}}^{(k)}\}_{k=1}^{N})$  then  
 $\pmb{\theta} \gets \pmb{\theta}'$   
end if  
end while
estimate the post-update return distributions of the post-update policies by sampling TD3 updates from random minibatches of the replay buffer and evaluating the returns of the corresponding policies. If the estimate of the post-update return CVaR is not sufficiently high relative to that of the post-update return distribution of the current policy, the update is rejected, so that the networks and the replay buffer are reverted to the state that they were in before the update was computed. In our experiments, we study the effect that such a rejection mechanism has on the evolution of the LTP by comparing the trajectories induced by this procedure without the ability to reject (i.e., regular TD3) and with the ability to reject.
In Figure 7, we show the improvement in LTP that this algorithm induces when applied to the same policy, aggregated across Brax tasks, using at least 10 policies per environment, after only forty gradient steps. We additionally present scatter plots demonstrating the effect of applying Algorithm 1 to individual policies in Appendix A.10. Our results demonstrate that this rejection procedure can be an effective tool for reducing the LTP of an existing policy.
Reliability of deep RL. The goal to avoid catastrophic drops in performance was at the core of the development of foundational methods in deep RL based on conservative updates [41, 42]. Previous work also studied the development of safer algorithms for learning and exploration, both from the theoretical and the empirical standpoints [25, 30, 32, 37, 48]. Our work focuses on understanding the landscape visited by commonly employed policy optimization algorithms and shows that it is
See Appendix A.9 for further justification.
possible to relatively easily move from parts of the landscape that induce dangerous behaviors to safer policy parameter vectors. On a higher-level, the sensitivity of deep RL algorithms to stochasticity and hyperparameters, and the extreme variability of results across seeds has been the object of previous studies [2, 11, 20], which mostly focused on proposing more reliable evaluation metrics. Previous work [9] also explicitly advocated for measuring the stability of deep RL algorithms over different axes and using a diverse set of metrics. Our paper proposes a complementary perspective, based on return landscapes and on a distributional view on them. Our procedure which leverages the directions proposed by a policy optimization algorithm to improve the LTP of a policy is related to previous work based on rejection/backtracking strategies [25, 40].
Return and loss landscapes. Return landscapes have been previously investigated at a coarser level under the name of reward surfaces/landscapes. In particular, they have been employed for studying the alignment of the gradient directions suggested by policy optimization algorithms to directions of improvement in the actual environment [23] and investigating performance degradation as a long-term optimization danger in such algorithms [43]. Our study of return landscapes with a distributional view in an otherwise fully deterministic setting sheds new light both on the landscape itself and on how it can be leveraged to characterize individual policies. More generally, the investigation of the return that policies collect in an environment is related to the study of the loss landscape of neural networks in supervised learning, for which different techniques have been proposed [28]. Those techniques, together with RL-specific tools, have been employed to explore the loss landscapes of RL algorithms, by visualizing them [5], probing their interaction with entropy regularization [3] or larger neural networks [35]. Our discovery of how policies from the same run are connected by simple paths in parameter space is related to (linear) mode connectivity, which shows a similar behavior in the landscapes of neural networks trained in supervised learning tasks [12-14, 17, 18]. Finally, our work is related to distributional RL [6], but we specifically focus on the post-update return distribution as opposed to the distribution of returns under a given policy.
In this paper, we have investigated return landscapes in continuous control tasks, as traversed by deep RL algorithms. We demonstrated the existence of noisy neighborhoods of the landscape, where a single update to the policy parameters produces a wide-breadth of post-update returns. By taking a distributional view on these neighborhoods, we revealed the existence of neighborhoods of similar mean return, yet different statistics, which correspond to qualitatively different agent behaviors. We studied the characteristics of failing policies and trajectories in such neighborhoods and attributed their subpar performance to sudden collapses in trajectory reward, rather than overall degradation in the policy. By focusing on linear paths through the global policy landscape, we showed that the landscape exhibits macro-scale variations which extend beyond specific local neighborhoods, and that policies from the same run can be surprisingly connected by linear paths with no valleys of low return. Finally, we demonstrated a simple procedure which discovers paths towards smoother regions of the landscape, starting from a trained policy.
Our results suggest that some of the previously-observed reliability issues in deep reinforcement learning agents for continuous control may be due to the fundamental structure of the return landscape for neural network policies. In particular, while the return of policy in a given neighborhood may be adequate, the distributional structure of the neighborhood characterizes additional dimensions of policy quality: How stable is this policy? What kind of behavior has the agent learned? Is it safe to perform further optimization of this policy? These nuances indicate the potential utility of a landscape-inspired approach to the design of reliable deep RL algorithms.
In addition, our study of parameter interpolation on the return landscape reveals new curiosities surrounding the training behavior of deep reinforcement learning agents. It appears that many policies from the same run fall within a single basin of the return landscape; we conjecture that this may correspond to the algorithm "specializing" on one particular behavior. Our demonstration of regions of lower and higher variability in returns along such paths further supports the possibility of robustifying existing policies, yet also raises the question of whether there are significantly different behaviors separated by barriers of low return, and whether our algorithms can find them. As they are beyond the scope of this paper, we reserve such questions for future work.
The authors thank Jesse Farebrother, Georg Ostrovski, David Meger, Rishabh Agarwal, Josh Greaves, Max Schwarzer and Pablo Castro for insightful discussions and useful suggestions on the early draft, the Mila community for creating a stimulating research environment, and the Digital Research Alliance of Canada for computational resources. This work was partially supported by CIFAR, Fonds de recherche du Québec (FRQNT) and Gruppo Ermenegildo Zegna.
[1] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C. Courville, and Marc G. Bellemare. Deep reinforcement learning at the edge of the statistical precipice. In Neural Information Processing Systems, 2021.
[2] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C. Courville, and Marc G. Bellemare. Deep reinforcement learning at the edge of the statistical precipice. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 29304-29320, 2021.
[3] Zafarali Ahmed, Nicolas Le Roux, Mohammad Norouzi, and Dale Schuurmans. Understanding the impact of entropy on policy optimization. In International Conference on Machine Learning, 2018.
[4] Igor Babuschkin, Kate Baumli, Alison Bell, Surya Bhupatiraju, Jake Bruce, Peter Buchlovsky, David Budden, Trevor Cai, Aidan Clark, Ivo Danihelka, Claudio Fantacci, Jonathan Godwin, Chris Jones, Tom Hennigan, Matteo Hessel, Steven Kapturowski, Thomas Keck, Iurii Kemaev, Michael King, Lena Martens, Vladimir Mikulik, Tamara Norman, John Quan, George Papamakarios, Roman Ring, Francisco Ruiz, Alvaro Sanchez, Rosalia Schneider, Eren Sezener, Stephen Spencer, Srivatsan Srinivasan, Wojciech Stokowiec, and Fabio Viola. The DeepMind JAX Ecosystem, 2020.
[5] Recep Yusuf Bekci and Mehmet Gumus. Visualizing the loss landscape of actor critic methods with applications in inventory optimization. arXiv preprint arXiv:2009.02391, 2020.
[6] Marc G Bellemare, Will Dabney, and Mark Rowland. Distributional reinforcement learning, 2022.
[7] Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents (extended abstract). In International Joint Conference on Artificial Intelligence, 2012.
[8] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018.
[9] Stephanie CY Chan, Samuel Fishman, Anoop Korattikara, John Canny, and Sergio Guadarrama. Measuring the Reliability of Reinforcement Learning Algorithms. In International Conference on Learning Representations, 2019.
[10] Herman Chernoff. Estimation of the mode. Annals of the Institute of Statistical Mathematics, 16(1):31-41, 1964.
[11] Cédric Colas, Olivier Sigaud, and Pierre-Yves Oudeyer. A hitchhiker's guide to statistical comparisons of reinforcement learning algorithms. In Reproducibility in Machine Learning, ICLR 2019 Workshop, New Orleans, Louisiana, United States, May 6, 2019. OpenReview.net, 2019.
[12] Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In International conference on machine learning, pages 1309-1318. PMLR, 2018.
A.1 Hyperparameters and implementation
For our analyses, we run TD3 and SAC for one million environment steps, with the default hyperparameters reported in Table 1, and collected checkpoints at  $5 \times 10^{4}$ ,  $15 \times 10^{4}$ ,  $25 \times 10^{4}$ ,  $35 \times 10^{4}$ ,  $45 \times 10^{4}$ ,  $55 \times 10^{4}$ ,  $65 \times 10^{4}$ ,  $75 \times 10^{4}$ ,  $85 \times 10^{4}$ ,  $95 \times 10^{4}$  steps for 20 different seeds. We run 20 seeds of PPO for 60 million environment steps with the hyperparameters in Table 1, to obtain a comparable level of performance, being that PPO is notoriously less sample-efficient than the two other algorithms [19]. We use equally-spaced checkpoints, at approximately  $6 \times 10^{5}$ ,  $12 \times 10^{5}$ ,  $18 \times 10^{5}$ ,  $24 \times 10^{5}$ ,  $30 \times 10^{5}$ ,  $36 \times 10^{5}$ ,  $42 \times 10^{5}$ ,  $48 \times 10^{5}$ ,  $54 \times 10^{5}$ ,  $60 \times 10^{5}$  environment steps.
We acknowledge the Python community [34, 46] for developing the core set of tools that enabled this work, including JAX [4, 8], Jupyter [26], Matplotlib [22], numpy [33, 45], pandas [36, 47], and SciPy [24]. Our implementations of TD3 and SAC are based on jaxr1 [27] and our implementation of PPO is based on CleanRL [21].
Our code is available at https://github.com/nathanrahn/return-landscapes.
A.2 Details of return landscape visualization
To generate the return landscapes in Figure 1, we start from a given policy of parameters  $\pmb{\theta}_0$  (produced, in that case, by SAC), and update it two times independently to obtain  $\pmb{\theta}'$  and  $\pmb{\theta}''$ . Then, we plot  $R(\pmb{\theta}), \pmb{\theta} = \alpha (\pmb{\theta}' - \pmb{\theta}_0) + \beta (\pmb{\theta}'' - \pmb{\theta}_0) + \pmb{\theta}_0, \alpha \in [-3,3], \beta \in [-3,3]$ , combining the two gradient directions into other directions to explore the neighborhood. The resulting directions are equivalent to the ones that would be obtained by oversampling or undersampling some elements in a larger batch composed of elements from both the two sampled updates.
A.3 Interpolation between policies
We now specify the details of the experimental protocol used for Figure 5 and report additional results. In Figure 9, we report results of interpolation for the other two Brax environments, ant and half cheetah, missing from Figure 5 for space reasons. We generate the checkpoints for these plots to be policies with similar level of return but different standard deviation of the post-update return distribution. A list of the checkpoints and seeds from TD3 used for producing these plots is shown in Table 2. While the view provided in Figure 5 and Figure 9 provides explicit visual cues on the variability of the neighborhood, it does not leverage directly the post-update return distribution to characterize the neighborhood of the policies obtained by interpolation. To provide a more detailed view, we estimate the post-update return distribution of an unstructured update, by sampling, given a policy of parameters  $\theta$ , 1000 perturbations from an isotropic Gaussian distribution  $\mathcal{N}(\theta, 0.0003)$  and evaluating the resulting policies. Figure 8 shows the mean and the standard deviation (as an error band) of the post-update return distribution of the interpolated policies from the same runs shown in Table 2. This alternate view confirms the results: linearly interpolating between policies of similar mean for the post-update return distribution produces paths of approximately constant (or smoothly increasing/decreasing) mean post-update return, even when the standard deviation of their post-update return distributions is different.
A.4 Instabilities in states other than the starting one
In our analyses, we only investigated the return landscape by computing the return on a fixed initial state. Even if the characteristics of the return from other states naturally propagate back to the initial state, it is natural to wonder whether the failures analyzed in our paper also happen naturally in other states. To investigate this question, we compute the LTP of the post-update return distribution for 3 runs of TD3, when starting from one of the states collected by the algorithm during training. In Figure 10, for the ant and walker2d tasks, we show for each of these states, identified by their collection time (index in the replay buffer) the LTP averaged over all the checkpoints collected during training. The results indicate that, regardless of some variation across seeds, there appears not to be any special pattern in the presence of failures over states grouped by time, and that the tail of the post-update distributions is long even when measured from states other the initial one.
Figure 8: Mean and standard deviation, displayed as error band, of the (noise-based) post-update return distributions for the four different Brax environments. The mean of the post-update return distribution remains approximately constant when linearly interpolating between checkpoints of similar level of return.
Figure 9: Results on linear interpolation among policies in the ant and halfcheetah.
Figure 10: Left-tail probability computed on different states, from a replay buffer for 3 seeds of TD3, averaged across all checkpoints.
Table 1: Hyperparameters for TD3, SAC and PPO.
TD3
Parameter	Setting
Discount factor	0.99
Minibatch size	256
Optimizer (all)	Adam
Optimizer (all): learning rate	0.0003
Optimizer (all): β1	0.9
Optimizer (all): β2	0.999
Optimizer (all): ε	0.00015
Networks (all): activation	ReLU
Networks (all): n. hidden layers	2
Networks (all): hidden units	256
Replay Buffer Size	10^6
Updates per step	1
Target update period	1
τ (EMA coefficient)	0.995
Exploration noise	0.1
Actor Delay	2
Policy Noise	0.2
Noise Clip	0.5
SAC
Parameter	Setting
Discount factor	0.99
Minibatch size	256
Optimizer (all)	Adam
Optimizer (all): learning rate	0.0003
Optimizer (all): β1	0.9
Optimizer (all): β2	0.999
Optimizer (all): ε	0.00015
Networks (all): activation	ReLU
Networks (all): n. hidden layers	2
Networks (all): hidden units	256
Initial Temperature	1
Replay Buffer Size	10^6
Updates per step	1
Target update period	1
τ (EMA coefficient)	0.995
PPO
Parameter	Setting
Discount factor	0.99
Optimizer (all)	Adam
Optimizer (all): learning rate	0.0026
Optimizer (all): β1	0.9
Optimizer (all): β2	0.999
Optimizer (all): ε	0.00001
Networks (all): activation	ReLU
Networks (all): n. hidden layers	2
Networks (all): hidden units	256
Parallel Envs	2048
Steps to update	16
GAE λ	0.95
Num. Updates	4
Max Grad. Norm	1
A.5 Scatter plots of statistics of post-update return distribution
We now show similar "maps" of the landscape to Figure 2 for all of the Brax environments we study. Results are shown in Figure 11, Figure 12, and Figure 13. We also include similar results for the quadruped-walk, quadruped-run, humanoid-stand, humanoid-walk, and humanoid-run tasks from DeepMind Control Suite [44] in Figures 14, 15, 16, 17, and 18. We believe that this type of scatter plot can provide a different perspective, compared to the one only focused on training curves, to study the interaction between a particular algorithm and an environment.
A.6 Discrete control
Despite the fact that the main focus of our paper is on continuous control, we also run small-scale experiments to understand if the tools we introduced generalize to discrete-action environments. In particular, we leverage four games from the standard ALE benchmark: Qbert, MsPacman, Breakout, and Seaquest [7]. We run PPO on these environments for 5 runs and for 10 million steps each, and measure post-update return distribution statistics using the same protocol employed in the rest of
Table 2: List of checkpoints used for Figure 5, Figure 9 and Figure 8.
Same Run
Seed	Ckpt 1	Ckpt 2
ant
1	550000	150000
6	850000	650000
11	650000	250000
halfcheetah
3	250000	950000
6	250000	350000
13	250000	550000
hopper
2	450000	250000
15	550000	150000
17	950000	750000
walker2d
1	750000	250000
4	950000	850000
4	550000	450000
Diff. Run
Seed 1	Ckpt 1	Seed 2	Ckpt 2
ant
14	850000	4	750000
11	750000	4	550000
2	650000	4	350000
halfcheetah
10	950000	7	450000
7	750000	16	750000
1	250000	6	350000
hopper
7	350000	4	450000
1	250000	2	150000
9	550000	6	250000
walker2d
14	350000	17	450000
1	550000	10	950000
17	350000	3	250000