Identifying Policy Gradient Subspaces
Authors: Jan Schneider
Pierre Schumacher
Simon Guist
Le Chen
Daniel Haeufle
Bernhard Schölkopf
Dieter Büchler
Venue: ICLR 2024 poster (2024)
Keywords: reinforcement learning
policy gradients
gradient subspaces
Primary area: reinforcement learning
TL;DR: We investigate the potential of gradient subspaces in deep reinforcement learning.

Abstract: Policy gradient methods hold great potential for solving complex continuous control tasks. Still, their training efficiency can be improved by exploiting structure within the optimization problem. Recent work indicates that supervised learning can be accelerated by leveraging the fact that gradients lie in a low-dimensional and slowly-changing subspace. In this paper, we conduct a thorough evaluation of this phenomenon for two popular deep policy gradient methods on various simulated benchmark tasks. Our results demonstrate the existence of such gradient subspaces despite the continuously changing data distribution inherent to reinforcement learning. These findings reveal promising directions for future work on more efficient reinforcement learning, e.g., through improving parameter-space exploration or enabling second-order optimization.

Published as a conference paper at ICLR 2024
IDENTIFYING POLICY GRADIENT SUBSPACES
Jan Schneider Pierre Schumacher
Simon Guist
Le Chen'
Daniel Häufle Bernhard Schölkopf
Dieter Büchler
1 Max Planck Institute for Intelligent Systems, Tübingen, Germany Hertie Institute for Clinical Brain Research, Tübingen, Germany 3 Institute for Computer Engineering, University of Heidelberg, Germany
ABSTRACT
Policy gradient methods hold great potential for solving complex continuous con- trol tasks. Still, their training efficiency can be improved by exploiting structure within the optimization problem. Recent work indicates that supervised learning can be accelerated by leveraging the fact that gradients lie in a low-dimensional and slowly-changing subspace. In this paper, we conduct thorough evaluation of this phenomenon for two popular deep policy gradient methods on various sim- ulated benchmark tasks. Our results demonstrate the existence of such gradient subspaces despite the continuously changing data distribution inherent to rein- forcement learning. These findings reveal promising directions for future work on more efficient reinforcement learning, e.g., through improving parameter-space exploration or enabling second-order optimization.
INTRODUCTION
Deep reinforcement learning (RL) has marked significant achievements numerous challeng- ing problems, ranging from Atari games (Mnih et al., 2013) to various real robotic challenges, such as contact-rich manipulation (Gu et al., 2017: Kalashnikov et al., 2018), complex plan- ning problems (Everett et al., 2018; Ao et al., 2022), and hard-to-control dynamic tasks (Cheng et al., 2023; Kaufmann et al., 2023). Despite these notable successes, deep RL methods are often brittle due to the use of function approximators with large numbers of parameters and persistently changing data distributions - a setting notoriously hard for optimization. Deep RL, in its vanilla form, operates under limited prior knowledge and structural information about the problem, conse- quently requiring large numbers of interactions with the environment to reach good performance.
For supervised learning (SL), Gur-Ari al. (2018) demonstrated that the gradients utilized for neu- ral network optimization reside in a low-dimensional, slowly-changing subspace. Based on this insight, recent works introduce more structured optimization procedures for SL by identifying and harnessing these gradient subspaces. Exploiting this structure enables the optimization to be carried out in a deduced-dimensienn performance (Li et al., 2018; Gressmann et al., 2020; Larsen et al., 2021; Li et al., 2022a).
subspace, yielding enhanced efficiency with minimal, if any, loss in
Despite the benefits of subspace methods in SL., their adoption deep RL has remained limited. A straightforward way to transfer these principles is to find lower-dimensional subspaces in policy gradient approaches (Peters & Schaal, 2008). Policy gradient (PG) methods estimate the gradient of the RL objective (0) to update the policy's parameters 0 using some form of stochastic gradient descent (SGD). Since most SL approaches using subspaces operate at the level of the SGD optimization, PG algorithms would be natural choice to leverage the knowledge about subspaces from SL in the RL context. Nevertheless, in RL, such methods have been explored primarily within the realm of evolutionary strategies (Maheswaranathan al., 2019), representation learning (Le Lan et al., 2023), and transfer learning (Gaya et al., 2022). A possible explanation is the constantly changing data distribution of RL due continual exploration that intuitively seems to hinder the identification of gradient subspaces. The limited body of studies using subspaces in PG algorithms underlines the need for more profound discussion in this domain.
*Correspondence jan. schnei derêtuebi .gw..ngg.dee

---

Published as a conference paper at ICLR 2024
This paper conducts comprehensive empirical evaluation of gradient subspaces in the context of PG algorithms, assessing their properties across various simulated RL benchmarks. Our experi- ments reveal several key findings: (i) there exist parameter-space directions that exhibit significantly larger curvature compared to other parameter-space directions, (ii) the gradients live in the subspace spanned by these directions, and (iii) the subspace remains relatively stable throughout the RI train- ing. Additionally, we analyze the gradients of the critic- an integral part of the PG estimation in actor-critic methods- and observe that the critic subspace often exhibits less variability and retains a larger portion of its gradient compared to the actor subspace. We also test the robustness of PG subspaces regarding mini-batch approximations of the gradient that are used in practice during train- ing and evaluate similar mini-batch approximation of the Hessian. Lastly, we explore the extent to which the variation the data distribution influences the aforementioned subspace analysis by conducting experiments with both an on-policy as well as an off-policy algorithm, the latter of which reuses previously collected data for training.
By shedding light gradient subspaces in deep RL, this paper provides insights that can potentially enhance RL performance by advancing parameter-space exploration or enabling second-order opti- mization. We begin by reviewing existing literature subspace approaches in Section 2, followed by a recapitulation of the RL preliminaries in Section 3 as a foundation for the analysis of gradient subspaces in RL in Section 4. Section 5 concludes this work with discussion of the results and implications of our work. The code for our experiments is available on the project website.
RELATED WORK
Numerous works have studied the use of gradient subspaces in SL. These works can be roughly divided into informed and random subspace approaches. In the following, we give overview of these papers and highlight works that investigate related concepts in RL.
Informed subspaces in supervised learning Informed subspace methods identifyy the subspace from some part of the training process. For instance, Gur-Ari et al. (2018) show that the gradients used for neural network optimization lie in low-dimensional subspace of the parameter-space di- rections with the highest curvature in the loss. Furthermore, they show that this subspace changes slowly throughout the training. Recent works have highlighted numerous benefits of these insights. Li et al. (2022a) apply principal component analysis on the network parameters at previous check- points and use the top principal components as subspace. They apply SGD and BFGS algorithms in the subspace and demonstrate superior learning performance compared to training in the original parameter space. Similarly, Tuddenham et al. (2020) propose to use a BFGS method in a low- dimensional subspace defined by the full-batch gradients with respect to individual classification labels. Sohl-Dickstein et al. (2014) construct second-order Taylor approximation of individual loss terms efficiently in a low-dimensional subspace that captures recently observed gradients. With this approximation, the overall loss is a sum of quadratic functions that can be optimized in closed form. Chen al. (2022) utilize insights about the low-dimensional subspaces to learn optimizer that scales to high -igh-dimensoonal parameter spaces. Traditionally, Iearned optimizers are limited in scale by the high output dimensionality of the network. By outputting parameter updates in the low-dimensional subspace, Chen et al. circumvent this challenge and efficiently learn to optimize large neural networks. Gauch al. (2022) apply the knowledge of low-dimensional gradient sub- space to few-shot adaptation by identifying a suitable subspace on the training task and utilizing it for efficient finetuning on the test task. Likewise, Zhou et al. (2020) tackle differential privacy by identifying a gradient subspace on a public dataset and projecting gradients calculated from the pri- vate dataset into this subspace. Adding noise the subspace instead of the parameter space lowers the error bound of differential privacy to be linear in the number of subspace dimensions instead of the parameter dimensions. Li al. (2022b) tackle an overfitting problem of adversarial training by identifying gradient subspace in an early training phase before overfitting occurs. By projecting later gradients into this subspace, they enforce that the training dynamics remain similar to the early training phase, which mitigates the overfitting problem.
Random subspaces in supervised learning Gressmann et al. (2020) describe different modifica- tions to improve neural network training in random subspaces, like resampling the subspace reg- ularly and splitting the network into multiple components with individual subspaces. They utilize these subspaces reduce the communication overhead in a distributed learning setting by send-
2

---

Published as a conference paper at ICLR 2024
ing the parameter updates low-dimensional subspace coefficients. Li et al. (2018) quantify the difficulty of a learning task by its intrinsic dimensionaliny, i.c., the dimensionality of the random subspace needed to reach good performance. Aghajanyan et al. (2021) apply a similar analysis to language models and find that the effectiveness of fine-tuning language models is tied to a low in- trinsic dimensionality. Based on these findings, Hu al. (2021) propose method for fine-tuning large models efficiently via low-rank parameter updates, Larsen et al. (2021) compare informed and random subspaces and find that with informed subspaces, typically significantly fewer dimensions are needed for training. Furthermore, they note the benefit of first training in the original space for some time before starting subspace training, as this increases the probability of the subspace intersecting with low-loss regions in the parameter space.
Subspace approaches to reinforcement learning
Only a handful of works use subspace methods for RL. Tang.a et al. (2023) identify a parameter subspace through singular value decomposition of the policy parameters' temporal evolution. They propose to cut off the components of the update step that lie outside this subspace and additionally elongate the update in the directions corresponding to the largest singular values. In contrast to our work, they focus solely on the policy network, consider only off- off-policy learning algorithms, and do not establish connection to the curvature of the learning objective. Li et al. (2018) quantify the difficulty of RL tasks by their intrinsic dimensionality. To that end, the authors optimize the value function of Deep Q-Networks (DQN) (Mnih et al., 2013) as well as the population distribution a distribution over the policy parameters - of evolutionary strategies (ES) (Salimans et al., 2017) in random subspaces. Maheswaranathan et al. (2019) define a search distribution for ES that is elongated along an informed subspace spanned by surrogate gradients. While the authors do not evaluate their approach for RL, they highlight the relevance to RL due to the method's 's ability to use surrogate gradients generated from learned models, similar to the use of the critic in actor-critic methods. Gaya et al. (2022) enable improved generalization in online RL by learning subspace of policies. Recently, they extended their method to scale favorably to continual learning settings (Gaya et al., 2023). Both methods explicitly learn a subspace of policy parameters, while our work investigates the natural training dynamics of PG methods.
PRELIMINARIES
This section introduces the mathematical background and notation used throughout the paper. Fur- thermore, we briefly describe the two RL algorithms that we will analyze in Section 4.
3.1 MATHEMATICAL BACKGROUND AND NOTATION
For given objective function (0), we use g 00 (0) E R" denote the gradient of model with 8a
respect to its parameters 0 and H 3o0 [(0) E R to denote the corresponding Hessian matrix. We use U, to denote the ith largest eigenvector of H. Note that we use "ith largest eigenvector" as shorthand for "eigenvector with respect to the ith largest eigenvalue". Since H is symmetric, all eigenvectors are orthogonal to each other, and we assume ||::|| = 1.
In this work, we investigate projections of gradients into power-dimessonne subspaces, i.e., map- pings from R" to & with is < n. These mappings are defined by a projection matrix P E Rx g E RR denotes the projection of g into the subspace and P 9 E R. is the mapping of back to the original dimensionality that minimizes the projection error ||g ğ||- Here, p+ denotes the pseudoinverse of P. If the projection matrix is semi-orthogonal, i.e., the columns are orthogonal and norm one, the pseudoinverse simplifies to the transpose p+ = pT' The matrix of the k largest eigenvectors P (UI...., " is one example of such a semi-orthogonal matrix.
3.2 REINFORCEMENT LEARNING
We consider tasks formulated as Markov decision processes (MDPs), defined by the tuple (S, A. p. r). Here, S and A are the state and action spaces, respectively. The transition dynam- ics p: S x A x S + j0, oc) define the probability density of evolving from one state to another. At each timestep the agent receives a scalar reward r: S x A R. A stochastic policy, To (a, | st), defines a mapping from state s, to a probability distribution over actions a, RL aims to find an optimal policy n², maximizing the expected cumulative return, discounted by 7 E [0, 1].
3

---

Published as a conference paper at ICLR 2024
The value function V* (s) represents the expected (discounted) cumulative reward from state following policy я. and the action value function Q= a) denotes the expected (discounted) cu- mulative reward for taking action a in state and then following The advantage function (s, a) Q as V* (s) quantifies the relative benefit of taking an action a in state s over the average action according policy
RL algorithms generally can be divided into two styles of learning. On-policy methods, like Prox- imal Policy Optimization (PPO) (Schulman et al., 2017), only use data generated from the current policy for updates. In contrast, oof-policy algorithms, such as Soft Actor-Critic (SAC) (Haarnoja al., 2018) leverage data collected from different policies, such as old iterations of the policy,
3.2.1 1
PROXIMAL POLICY OPTIMIZATION
On-policy PG methods typically optimize the policy no via an objective such as J(0)
En. (a |s)4 Em (0) with r0( = me(a, sc) and A, being an estimator of the [s.
advantage function at timestep t and Fold denoting the policy before the update (Kakade Lang- ford, 2002). However, optimizing this objective can result in excessively large updates, leading to instabilities and possibly divergence. Proximal Policy Optimization (PPO) (Schulman et al., 2017) is an on-policy actor-critic algorithm designed to address this issue by clipping the probability ratio re (e()) to the interval [1 [1-6, 1 + e). which removes the incentive for moving (0) outside the interval, resulting in the following actor loss.
J PPO
actor (0) = Emaaa min (O)())A,,
4.)]
(1)
The advantage estimation Å. 6, + + (+ (+))))
with 6, v. + VV (a
Vo uses Iearned value function Va. which acts as critic. The hyperparameter À determines the trade-off between observed rewards and estimated values. The critic is trained to minimize the mean squared eπror between the predicted value and the discounted sum of future episode rewards V.turgt
1+T
TT.
JPPO (d) E [(V)()) V. Vtargett
3.2.2 SOFT ACTOR-CRITIC
Soft Actor-Critic (SAC) (Haarnoja al., 2018) policy gradient algorithm that integrates the maximum entropy reinforcement learning framework with the actor-critic approach. As such, it op- timizes a trade-off between the expected return and the policy's 's entropy. It is an off off-policy algorithm and, as such, stores transitions in replay buffer D, which it samples from during optimization.
To that end, SAC modifies the targets for the learned Q-function Qù include a term that incen- tivizes policies with large entropy Q: = Fı + (8 - log (a(+ S1+1)), resulting in the following critic loss.
J SAC (Ø) = ED critic
2 (1)
(3)
Note that SAC, its original formulation, trains an additional value function and a second Q- function, but we omitted these details for brevity. The algorithm then trains the actor to minimize the KL-divergence between the policy and the exponential of the learned Q-function.
exp(Q 22
JSCC (0) Ep DKL. T
st))
Zo (st) denotes the normalization make the right side of the KL-divergence a proper distribution. Optimizing this loss increases the probability of actions with high value under the Q-function.
4 GRADIENT SUBSPACES IN POLICY GRADIENT ALGORITHMS
In Section 2, we have highlighted several works from SL that utilize cow-dimensional gradient sub- spaces for improving the learning performance. Naturally, we would like to transfer these benefits
4

---

Published as a conference paper at ICLR 2024
to policy gradient algorithms. However, the training in RL is significantly less stationary than in the supervised setting (Bjorck et al., 2022). As the RL agent changes, the data distribution shifts since the data is generated by the agent's interactions with its environment. Furthermore, the value of state also depends on the agent's ' behavior in future states. Thus, the targets for the actor and critic networks change constantly. These crucial differences between SI. and RL underscore the need to analyze to which extent insights about gradient subspaces transfer between these settings.
The analysis presented in this section focuses on two policy gradient algorithms: PPO (Schulman et al.. 2017) and SAC (Haarnoja et al., 2018), which are popular instantiations of on-policy and off-policy RL. We apply the algorithms to twelve benchmark tasks from OpenAI Gym (Brockman et al., 2016). Gym Robotics (Plappert et al., 20), and the DeepMind Control Suite (Tunyasuvu- nakool et al., 2020). Our code builds upon the algorithm implementations of Stable Baselines3 (Raf- fin et al., 2021). The learning curves are displayed in Appendix A. We ran each experiment for 10 random seeds and plot the mean and standard deviation for the results in Sections 4.2 and 4.3. Due to space constraints, we show the analysis results only for selected tasks and present detailed results for all twelve tasks in Appendix B. Moreover, we conduct an evaluation of the impact of the RL algorithm's 's hyperparameters on the gradient subspace in Appendix C.
For the following analyses, we calculate Hessian eigenvectors of the loss with respect to the net- work parameters via the Lanczos method (Lehoueg et al., 1998) since it is an efficient method for estimating the top eigenvectors that avoids explicitly constructing the Hessian matrix. Since we can only estimate the Hessian from data, estimates for the eigenvectors of the true Hessian, similar to how Ilyas et al. (2020) approximate the true policy gradient. For PPO, we collect 10 on-policy samples. This would, however, not be faithful to the diverse distribution of off-policy data that SAC uses for training. To match this data distribution for the analysis, we save the replay buffer during training and use the data of the complete replay buffer for estimating the Hessian. Note that the replay buffer also has capacity of 10 samples but is not completely filled at the beginning of training.
large number of state-action pairs to obtain precise
As mentioned in Section 3, SAC and PPO each train two different networks, an actor and critic. We, therefore, conduct our analysis for each network individually. To verify that there exist high- curvature directions spanning a subspace that stays relatively stable throughout the training and that contains the gradient, we check three conditions:
i) Some parameter-space directions exhibit significantly larger curvature in the actor/critic loss than other directions.
ii) The actor/critic gradient mainly lies in the subspace spanned by these directions.
iii) The subspaces for the actor and critic networks change slowly throughout the training.
4.1 THE LOSS CURVATURE IS LARGE IN A FEW PARAMETER-SPACE DIRECTIONS
The Hessian matrix describes the curvature of a function, with the eigenvectors being the directions of maximum and minimum curvature. The corresponding eigenvalues describe the magnitude of the curvature along these directions. Therefore, we verify condition i) by plotting the spectrum of Hessian eigenvalues for the actor and critic loss of PPO with respect to the network parameters in Figure 1. The plots show that there are a few large eigenvalues for both the actor and critic loss. All remaining eigenvalues are distributed close to zero. These plots confirm that there are few directions with significantly larger curvature; in other words, the problem is ill-conditioned.
4.2 THE GRADIENT LIES IN THE IIGIN-CURVATURE SUBSPACE
To verify condition ii) that the high-curvature subspace contains the gradients of the respective loss, we measure how well these gradients can be represented in the subspace. Let Pe E R be the seti-orthogonal matrix that projects into the high- high-curvature subspace. Pt consists row-wise of the k largest Hessian eigenvectors. We compute the relative projection error, i.e., the relative difference between the original gradient g RD and the projected gradient ğ = Pr Pkg that is the result of mapping into the high-curvature subspace and back into the original space. The fraction of the gradient that can be represented in the subspace is then given by
lg
Shace (Pr, g) = 1-
(5)
9 2
5

---

Published as a conference paper at ICLR 2024
100
Elgenvaluu
1000
500
0
0
1000
-500
-100
4000
Index
Index
(a) Finger-spin, actor
(b) Finger-spin, critic
(c) Walker2D, actor
(d) Walker2D, critic
Figure 1: The spectrum of the Hessian eigenvalues for PPO the tasks Finger-spin (a, b) and Walker2D (c, d). The Hessian is estimated from 10 state-action pairs. For both the actor (a, c) and critic (b, d) loss, there is small number of large eigenvalues, while the bulk of the eigenvalues is close to zero. This finding shows that there is small number of high-curvature directions in the loss landscapes, which is in accordance with results from SL.
which simplifies to the following gradient subspace fraction criterion of Gur-Ari al. (2018).
Strae Pk: 9) || gll
|1912
We derive this equality in Appendix D. Note that 0 I Strae (trae(Pk. I holds, where Stree (Pa ( = 1 implies that the subspace perfectly contains the gradient, while Strac(P,,)) 0 means that the gradient lies entirely outside the subspace. Due to the normalization by [[l]l this criterion invariant to the scale of the gradient, which enables comparing gradient subspaces of different models and models at different stages of the training.
To evaluate how the gradient subspace fraction evolves over time, we evaluate the criterion at check- points every 50,000 steps during the RL training. To compactly visualize this data, we split the training into three phases: initial, training, and convergence, and for each phase, we average the results of all timesteps of that phase. Since the algorithms require different numbers of timesteps for solving each of the tasks and reach different performance levels, we define the following heuristic criterion for the training phases. We first smooth the learning curves by averaging over sliding window and compute the maximum episode return Rmax over the smoothed curve. Next, we calcu- late the improvement relative to the episode return Rinit of the initial policy at each timestep & of the smoothed learning curve as
R(()) Rinit
(t Rmax Riui
We then define the end of the initial phase the first timestep which the agent reaches 10% of the total improvement, i.e., (n(()= 2 0.1. Similarly, we define the start of the convergence phase as the first timestep at which the agent reaches 90% of the total improvement, i.e., AR(() 2 0.9.
We choose A = 100 as subspace dimensionality since this subspace already largely captures the gradients, and the largest 100 eigenvectors can still be calculated reasonably efficiently with the Lanczos method. Appendix B displays results for different subspace sizes. With the tuned hyperpa- rameters from RL Baselines3 Zoo that we use for training, the PPO actor and critic usually contain around 5,000 parameters, and the SAC actor and critic around 70,000 and 140,000 parameters (2 Q-networks 70,000 parameters), respectively. Hence, the subspace dimensionality is around 2% the size of the parameters for PPO and around 0. 0.44% and 0.07% for SAC.
We consider a precise approximation of the true gradient computed with 10 state-action pairs for PPO and the full replay buffer for SAC. We denote this approximation precise gradient and the low-sample gradient used during regular RI. training as mini-batch gradient. denote the Hessian estimated on the large dataset as precise Hessian and the estimate from 2048 samples as mini-bated Hessian. We choose 2048 samples for the mini-batch Hessian since that is the amount of data that PPO with default hyperparameters collects for the policy updates. Hence, this is a realistic setting for estimating the subspace during training.
similar manner,
Figure 2 shows the value of the gradient subspace fraction (Pk. g) for PPO and SAC four different tasks, divided into the three training phases. Note that for an uninformed random pro- jection, the gradient subspace fraction would be k/n in expectation, i.e., the ratio of the original

---

Published as a conference paper at ICLR 2024
phase
Training phase
Convergence phase
Precise gradient, pred Hessian
Mini-batch gradient. precise Hessian
Mini- Mini-batch gradient, mini-batch Hessian
Gradient subspace fractoon
1.0
0.8
0.6
0.4
0.2
PPO
SAC
PPO
SAC
PPO
SAC
PPO
SAC
Fingerspin
LunarLanderConénuuus
Walker2D
(a) Actor
Gradient subspace fraction
1.0
0.8
0.6
0.2
0.0
SAC
PPO
SAC
PPO
SAC
Ant
Anger-spin
nunartaaderCantnuuus
Walker2D
(b) Critic
Figure 2: The fraction Stree of the gradient that lies within the high high-curvature subspace spanned by the 100 largest Hessian eigenvectors. Results are displayed for the actor (top) and critic (bottom) of PPO and SAC on the Ant, Finger-spin, JunarlanderrContiuuuss and Walker2D tasks. The results demonstrate that significant fraction of the gradient lies within the high-curvature subspace. but the extent to which the gradient is contained in the subspace depends on the algorithm, task, and training phase. For both algorithms, the gradient subspace fraction is significantly higher for the critic than for the actor. Furthermore, the quantity is also often larger for SAC's actor than for PPO's, s,, particularly in the early stages of the training. Even with mini-batch estimates for the gradient and Hessian, the gradient subspace fraction is considerable.
and subspace dimensionalities (0.02 for PPO and 0.0014 for SAC's actor and 0.0007 for its critic). The results in Figure 2 show significantly higher gradient subspace fraction, which means that the gradients computed by PPO and SAC lie to a large extent in the high-curvature subspace. We observe that the fraction of the gradient in the subspace considerably higher for the critic than for the actor. Furthermore, the gradient subspace fraction is also often higher for SAC's actor than for PPO's. This finding is particularly significant since the subspace size corresponds to a significantly lower percentage of the parameter dimensionality for SAC than for PPO. We hypothesize that the effect is caused by the off-po nature of SAC. In the off-policy setting, the training distribution for the networks changes slowly since the optimization reuses previous data. In this regard, SAC closer than PPO to the supervised learning setting, where the data distribution is fixed and for which Gur-Ari al. (2018) report high gradient subspace fractions. Still, the subspace fraction for PPO is significant, considering that the dimensionality of the subspace is merely 2% of the origi- nal parameter space. Furthermore, for PPO, the subspace fraction often improves after the initial phase. Similarly, Gur-Ari et al. (2018) report for the supervised learning setting that the gradient starts evolving the subspace only after some initial steps. However, for the SAC actor, this trend appears to be reversed, with the gradient subspace fraction being highest in the initial steps.
Moreover, the precise gradient, computed with large number of samples, tends to lie better in the subspace than the mini-batch gradient. The noise resulting from the low-sample approximation seems to perturb the gradient out of the subspace. However, since the difference typically small, the gradient subspace is still valid for the low-sample gradient estimates used during RL training. Lastly, even the subspace identified with the mini-batch Hessian captures the gradient to significant extent. This property is crucial since it implies that do not need access to the precise Hessian, which costly to compute and might require additional data. Instead, we can already obtain a reasonable gradient subspace from the mini-batch Hessian.
7

---

Published as a conference paper at ICLR 2024
PPO, actor
PPO, critic
SAC, actor
SAC, critic
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
subspace
0.6
0.6
0.6
0.6
0.4
0.4
0.2
0.2
0.2
02
0.0
0.0
0.0
0.0
f2
x10
x10*
62
x10
12
x10
(a) Ant
(b) Finger-spin
(c) Lunarl ConterCont.
(d) Walker2D
Figure Evolution of the overlap between the high-cur highccurvature subspace identified at an early timestep 4 = 100,000 and later timesteps for the actor and critic of PPO and SAC. While the overlap between the subspaces degrades as the networks are updated, remains considerable even after 400,000 timesteps. indicating that the subspace remains similar, even under significant changes in the network parameters and the data distribution. This finding implies that information about the gradient subspace at earlier timesteps can be reused at later timesteps.
4.3 THE HIGH-CURVATURE SUBSPACE CHANGES SLOWLY THROUGHOUT THE TRAINING
So far, we have verified that the gradients of the actor and critic losses optimized by PPO and SAC lie to a large extent in the subspace spanned by the top eigenvectors of the Hessian with respect to the current parameters. However, even though there are relatively efficient methods for computing the top Hessian eigenvectors without explicitly constructing the Hessian matrix, calculating these eigenvectors at every step would be computationally expensive. Ideally, we would like to identify a subspace once that remains constant throughout the training. In practice, however, the gradient subspace will not stay exactly the same during the training, but it changes slowly, it possible to reuse knowledge from earlier timesteps and update the subspace at lower frequency. To that end, we investigate condition iii) by calculating the subspace overlap, defined by Gur-Ar et al. (2018). and 12 is defined
The subspace overlap between timesteps t
Soverlap (p!)),,(()))
Σ
(8)
where of is the ith largest eigenvector at timestep t. p(() (01 (0))......0) denotes the pro- jection matrix from the full parameter space to the high-curvature subspace, identified at timestep t. Similar to Equation (5), the criterion measures how much of the original vector is preserved during the projection into the subspace. For the subspace overlap, however, we use the projection matrix at timestep & not to project the gradient but rather project the Hessian eigenvectors that span the high-curvature subspace identified at a later timestep 12 of the training. This criterion, thus, mea- sures how much the gradient subspace changes between these timesteps. Note that we assume the eigenvectors (t) v. to be normalized to one and therefore do not normalize by their length.
Gur-Ari et al. (2018) showed in the supervised setting that the gradient subspace stabilizes only after some initial update steps. Therefore, we choose the timestep t at which we initially identify the subspace as L = 100,000 since this is still relatively early in the training, but the gradient subspace should already have stabilized reasonably well. We evaluate the subspace overlap criterion every 10,000 timesteps until timestep 500,000. This interval covers a significant portion of the training and showcases the extent to which the subspace changes under significant differences in the network parameters and the data distribution. For the sake of completeness and to further highlight the influence of the data distribution on the subspace, we showcase the subspace overlap over the entire duration of the training in Appendix E. As in Section 4.2, we use k 100 as subspace dimensionality and refer to Appendix B for the evaluation of different subspace sizes. The analysis results in Figure 3 show that the subspace overlap reduces the further apart the two timesteps & and t2 are, but in all cases, the subspace overlap remains significantly above zero, implying that information of previous subspaces can be reused at later timesteps. If the two timesteps are close to each other, the overlap is considerable. Similar to the gradient subspace fraction in Section 4.2, the subspace overlap is often more pronounced for the critic than the actor, particularly for SAC.
8

---

Published as a conference paper at ICLR 2024
5 CONCLUSION
In this work, we showed that findings from the SL literature about gradient subspaces transfer the RL setting. Despite the continuously changing data distribution inherent to RL, the gradients of the actor and critic networks of PPO and SAC lie in oww-imensional, slowly-changing subspace of high curvature. We demonstrated that this property holds for both on-policy and off-policy learning, even though the distribution shift in the training data is particularly severe the on-policy setting.
5.1 HIGH-CURVATURE SUBSPACES EXPLAIN CLIFFS IN REWARD LANDSCAPES
Sullivan et al. (2022) investigate visualizations of the reward landscapes around policies optimized by PPO. Reward landscapes describe the resulting cumulative rewards over the space of policy pa- rameters. They observe empirically that these landscapes feature "cliffs" in policy gradient direction. When changing the parameters this direction, the cumulative reward increases for small steps but drops sharply beyond this increase. In random directions, these cliffs do not seem occur.
The results from Section 4.2 constitute a likely explanation of this phenomenon. The cliffs that the authors describe can be interpreted as signs of large curvature in the reward landscape. Our analysis demonstrates that the policy gradient is prone to lie in a high-curvature direction of the policy loss. Sullivan al. investigate the cumulative reward, which is different from the policy loss that analyze in this work. However, one of the fundamental assumptions of policy gradient methods is that there is a strong link between the policy loss and the cumulative reward. Therefore, high curvature in the loss likely also manifests in the cumulative reward. There is no such influence for random directions, so the curvature in gradient direction larger than in random directions.
5.2 POTENTIAL OF GRADIENT SUBSPACES IN REINFORCEMENT LEARNING
Leveraging properties of gradient subspaces has proven beneficial in numerous works in SI., e.g.- (Li al.. 2022a; Chen et al., 2022; Gauch et al., 2022: Zhou et al., 2020; Li al., 2022b). The analyses in this paper demonstrate that similar subspaces can be found in popular policy gradient algorithms. the following, we outline two opportunities for harnessing the properties of gradient subspaces and bringing the discussed benefits to RL.
Optimization in the subspace While the network architectures used in reinforcement learning are often small compared to the models used in other fields of machine learning, the dimensionality of the optimization problem still considerable. Popular optimizers, like Adam (Kingma & Ba, 2014 4), typically rely only on gradient information, as computing the Hessian at every timestep would be computationally very demanding in high dimensions. However, in Section 4.1, we have seen that the optimization problem is ill-conditioned. Second-order methods, like Newton's 's method, are known to be well-suited for ill-conditioned problems (Nocedal & Wright, 1999). With the insights this paper, it seems feasible reduce the dimensionality of the optimization problems in RL algorithms by optimizing in the low-dimensional subspace instead of the original parameter space. The low dimensionality of the resulting optimization problems would enable computing and inverting the Hessian matrix efficiently and make second-ord -onder optimization methods feasible.
Guiding parameter-space exploration The quality of the exploration actions significantly im- pacts the performance of RL algorithms (Amin et al., 2021). Most RL algorithms explore by ap- plying uncorrelated noise the actions produced by the policy, However, this often leads ineffi- cient exploration, particularly in over-actuated systems, where correlated actuation is crucial (Schu- macher et al., 2022). A viable alternative is to apply exploration noise the policy parameters instead (Rückstiess et al., 2010; Plappert et al., 2018b). This approach results in more directed exploration and can be viewed as exploring strategies similar to the current policy.
In Section 4, we observed that the gradients utilized by policy gradient methods predominantly lie within a small subspace of all parameter-space directions. As typical parameter-space exploration does not consider the properties of the training gradient when inducing parameter noise, only a small fraction of it might actually push the policy parameters along directions that are relevant to the task. Considering that the optimization mostly occurs in restricted subspace, it might be advantageous to limit exploration to these directions. Sampling parameter noise only in the high-curvature subspace constitutes one possible way of focusing exploration informative parameter-space directions.

---

Published as a conference paper at ICLR 2024
6 REPRODUCIBILITY
We applied our analyses to proven and publicly available implementations of the RL algorithms from 3aable-Baellnne33 (Raffin et al., 2021) on well-known, publicly available benchmark tasks from (Brockman et al., 2016; Plappert et al., 2018b; Tunyasuvunakool al., 2020). Further exper- imental details like the learning curves of the algorithms and the fine-grained analysis results for displayed in Appendices A and B, respectively. To facilitate reproducing results, we make our code, as well as the raw analysis data, including hyperparmeter settings and model checkpoints, publically available on the project website.
the entire training
REFERENCES
Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the ef- fectiveness of language model fine-tuning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume 1: Long Pupers), pp. 7319-7328, 2021.
Susan Amin, Maziar Gomrokchi, Harsh Satija, Herke van Hoof, and Doina Precup. A survey of exploration methods in reinforcement learning. arXiv preprint 1eXiv:2109.00111 2021.
Yunke Ao, Le Chen, Florian Tschopp, Michel Breyer, Roland Siegwart, and Andrei Cramariuc. Uni- fied data collection for visual-inertial calibration via deep reinforcement learning. International Conference Robotics and Automation, pp. 1646-1652. 1652. IEEE, 2022.
Johan Bjorek, Carla Gomes, and Kilian Q Weinberger. high variance unavoidable RL? A case study in continuous control. In International Conference on Learning Representations, 2022.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl Gym. arXiv preprint 1 2016.
Xuxi Chen, Tianlong Chen, Yu Cheng, Weizhu Chen, Ahmed Awadallah, and Zhangyang Wang- Scalable learning to optimize: A learned optimizer can train big models. European Conference Computer Vision, pp. 389-405. Springer, 2022.
Xuxin Cheng, Kexin Shi, Ananye Agarwal, and Deepak Pathak. Extreme parkour with legged robots. arXiv preprint arXiv:2309. 14341, 2023.
Michael Everett, Yu Fan Chen, and Jonathan P How. Motion planning among dynamic, decision- making agents with deep reinforcement learning. In International Conference on Intelligent Robots and Systems, pp. 3052-3052 IEEE, 2018.
Martin Gauch, Maximilian Beck, Thomas Adler, Dmytro Kotsur, Stefan Fiel, Hamid Eghbal-zadeh, Johannes Brandstetter, Johannes Kofler, Markus Holzleitner, Werner Zellinger, et al. Few-shot learning by dimensionality reduction in gradient space. Conference on Lifelong Learning Agents, pp. 1043-1064. PMLR, 2022.
Jean- ManntBapistt Gaya, Laure Soulier, and Ludovic Denoyer. Learning subspace of policies for online adaptation in reinforcement learning. In International Conference of Learning Representations, 2022.
Jean- Baptiste Gaya, Thang Doan, Lucas Caccia, Laure Soulier, Ludovic Denoyer, and Roberta Raileanu. Building subspace of policies for scalable continual learning. In International Con- ference of Learning Representations, 2023.
Frithjof Gressmann, Zach Eaton- Rossnn and Carlo Luschi. Improving neural network training in low dimensional random bases. Advances in Neural Information Processing Systems, 33:12140- 12150, 2020.
Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In International Conference on Robotics and Automation, pp. 3389-3396. IEEE, 2017.
10

---

Published as a conference paper at ICLR 2024
Guy Gur-Ari, Daniel A Roberts, and Ethan Dyer. Gradient descent happens in tiny subspace. arXiv preprint arXiv:1812.04754 2018.
Tuomas Haarooja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critics Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Confer- ence on Machine Learning, pp. 1861-1870. PMLR, 2018.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021.
Andrew Ilyas, Logan Engstrom, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. closer look at deep policy gradients. In International Con- ference Learning Representations, 2020.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In International Conference on Machine Learning, pp. 267-274, 2002.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan. Vincent Vanhoucke, al. QT-Opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv preprint arXiv:/800 10293, 2018.
Elia Kaufmann, Leonard Bauersfeld, Antonio Loquercio, Matthias Müller, Vladlen Koltun, and Davide Scaramuzza. Champion-level drone racing using deep reinforcement learning. Nature, 620:982-983 2023.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.69880 2014.
Brett W Larsen, Stanislav Fort, Nic Becker, and Surya Ganguli. How many degrees of freedom do we need to train deep networks: A loss landscape perspective. In International Conference on Learning Representations, 2021.
Charline Le Lan, Joshua Greaves, Jesse Farebrother, Mark Rowland, Fabian Pedregosa, Rishabh Agarwal, and Mare G Bellemare. A novel stochastic gradient descent algorithm for learning principal subspaces. In International Conference on Artificial Intelligence and Statistics, pp. 1703-1718. PMLR, 2023.
Richard B Lehoueq. Danny C Sorensen, and Chao Yang. ARPACK users' guide: Solution of large- scale eigenvalue problems with implicitly restarted Arnoldi methods. SIAM, 1998.
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. In International Conference on Learning Representations, 2018
Tao Li, Lei Tan, Zhehao Huang, Qinghua Tao, Yipeng Liu, and Xiaolin Huang. Low dimensional trajectory hypothesis is true: DNNs can be trained in tiny subspaces. Transactions on Pattern Analysis and Machine Intelligence, 45(3):3411-3422 2022a.
IEEE Conference Computer Vision and Pattern Recognition, pp. 14409--1448.,.1 2022b. Tao Li, Yingwen Wu, Sizhe Chen, Kun Fang, and Xiaolin Huang. Subspace adversarial training. In
Niru Maheswaranathan, Luke Metz, George Tucker, Dami Choi, and Jascha Sohl-Dickstein. Guided evolutionary strategies: Augmenting random search with surrogate gradients. In International Conference on Machine Learning. pp. 4264 4273. PMLR, 2019.
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing Atari with deep reinforcement learning. arXiv preprint 112.5:131 2013.
Jorge Nocedal and Stephen J Wright. Numerical optimization. Springer, 1999.
Supratik Paul, Vitaly Kurin, and Shimon Whiteson. Fast efficient hyperparameter tuning for policy gradient methods. Advances in Neural Information Processing Systems, 32, 2019.

---

Published as a conference paper at ICLR 2024
Jan Peters and Stefan Schaal. Reinforcement learning of motor skills with policy gradients. Neural networks, 21(4):682-665 09776882677722 2008.
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Pow- ell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, al. Multi-goal reinforce- ment learning: Challenging robotics environments and request for research. arXiv preprint Ntt1::1802009464 2018a.
Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. In International Conference on Learning Representations, 2018b.
Antonin Raffin.
RL Baselines3
Zoo.
com
tl-baseelness--oos 2020.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dor- mann. Stable-Baselines: Reliable reinforcement learning implementations. The Journal of Ma- chine Learning Research, 22(1):12348. 12355, 2021.
Thomas Rückstiess, Frank Sehnke, Tom Schaul, Daan Wierstra, Yi Sun, and Jürgen Schmidhuber. Exploring parameter space in reinforcement learning. Paladyn, Journal of Behavioral Robotics, 1:14-24, 2010.
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as a scalable alternative reinforcement learning. arXiv preprint arXiv:1703.03880 2017.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Pierre Schumacher, Daniel Haeufle, Dieter Büchler, Syn Schmitt, and Georg Martius. DEP-RL: Embodied exploration for reinforcement learning in overactuated and musculoskeletal systems. In International Conference on Learning Representations, 2022.
Jascha Sohl-Dickstein, Ben Poole, and Surya Ganguli. Fast large-scale optimization by unifying stochastic gradient and quasi-Newton methods. In International Conference on Machine Learn- ing, pp. 604-612. PMLR, 2014.
Ryan Sullivan, Jordan K Terry, Benjamin Black, and John Dickerson. Cliff diving: Exploring reward surfaces in reinforcement learning environments. In International Conference on Machine Learning, pp. 20744-20776. PMLR, 2022.
Hongyao Tang, Min Zhang, and Jianye Hao. The ladder in chaos: simple and effective im- provement to general DRL algorithms by policy path trimming and boosting. arXiv preprint arXiv:2303.01391, 2023.
Mark Tuddenham, Adam Prügel-Benned Bennett, and Jonathan Hare. Quasi-Newton's method in the class gradient defined high-curvature subspace. arXiv preprint arXiv:2012.01938, 2020.
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm_control: Software and tasks for continuous control. Software Impacts, 6:100022, 2020.
Yingxue Zhou, Steven Wu, and Arindam Banerjee. Bypassing the ambient dimension: Private SGD with gradient subspace identification. In International Conference Learning Representations, 2020.
12

---

Published as a conference paper at ICLR 2024
A
LEARNING CURVES
PPO
SAC
Cumulative reward
6000
1000
200
500
0
0.0
0.5
1.0
1.5
2.0
2.5 3.0
0.00
0.25
0.50
0.75
1.00
0.0
0.5
1.0
1.5 2.0
2.5
3.0
(a) Ant
(b) Ball_in.cup
(c) BipedalWalker
Cumulative reward
1000
15000
10000
20
500
5000
-40
0.0
0.1
0.2
0.3
0.4 0.5
0.00
0.25
0.50
0.75
1.00
0.0
0.5
1.0
1.5
2.0
2.5
3.0
(d) FetchReach
(e) Finger-spin
(f) HalfCheetah
Cumulaative reward
4000
250
-500
2000
0
-1000
-250
1500
0.0
0.5
1.0
1.5
0.00
0.25
0.50
0.75
1.00
0.0
0.1
0.2
0.3
(g) Hopper
(h) Lunarl anderContinuous
(i) Pendulum
Cumulative reward
400
6000
200
4000
-40
-60
0.0
0.3
0.5
0.5 1.5 2.5 Environment steps (millions) (k) Swimmer
1.0 2.0 3.0 Environment steps (millions)
0.0
Environment steps (millions)
(j) Reacher
(I) Walker2D
Figure 4: Learning curves for PPO and SAC on tasks from OpenAl Gym (Brockman et al., 2016), Gym Robotics (Plappert et al., 2018a), and the DeepMind Control Suite (Tunyasuvunakool al., 2020). We the algorithm implementations of Stable Baselines3 (Raffin al., 2021) with tuned hyperparameters from RL Baselines3 Zoo (Raffin, 2020) for the Gym tasks and hyperparam- eters tuned by random search over 50 configurations for the Gym Roboties and DeepMind Control Suite tasks. Results are averaged over ten random seeds; shaded areas represent the standard devia- tion across seeds.
13

---

Published as a conference paper at ICLR 2024
B DETAILED ANALYSIS RESULTS FOR ALL TASKS
Precise gradient; EV Mini-batch gradient; 1 EV
Precise gradient:
Precise gradient;
Mini-batch gradient: 10 EVs
Mini-batch gradient: 100 EVs
0.8
0.8
0.8
0.8
0.4
0.4
0.4
0.4
0.0
0.0
Ant, actor
(b) Ant, critic
Ball in cup, actor
(d) Ball in cup, critic
M
0.8
0.8
0.8
0.8
0.2
0.2
0.2
0.2
15
2.0
2.5
3.0
0.0
15
2.0
2.5 3.0
0.0
0.0
0.1
02
0.4
0.5
0.0
0.1
0.2
0.4
0.5
(e) Bip. Walker, actor
(l) Bip. Walker, critic
(g) FetchReach, actor
(h) FetchReach, critic
0.6
0.6
0.6
02
0.2
0.2
0.2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
b.a
0.5
1.0
155
2.0
25
x.n
0.0
0.5
1.0
1.5
2.0
2.5 3.0
(i) Finger-spin, actor
(j) Finger-spin, critic
(k) HalfCheetthh, actor
(l) HalfChectah, critic
1.0
0.6
0.6
0.6
0.6
0.0
0.0
0.5
1.0
1.5
0.0
0.0
0.5
1.0
1.5
0.0
02
0.4
0.6
08
1.
9.0
0.2
0.4
0.6
08
1.0
(m) Hopper, actor
(n) Hopper, critic
(o) Lunarr_aneer actor
(p) LunarL.ander critic
1.0
1.0
1.0
1.0
pitspacc
0.4
0.4
0.4
0.4
0.0
0.0
0.0
0.0
0.3
0.1
0.5
0.2
(q) Pendulum, actor
(r) Pendulum, critic
(s) Reacher, actor
(t) Reacher, critic
1.0
1.0
1.0
0.8
0.8
0.8
Man
0.4
0.4
0.4
0.4
0.0
0.0.0
0.0
Ervironment steps (millions)
Environmat ent steps (millinss
Environment steps imillionst
Environment (millions)
(u) Swimmer, actor
(v) Swimmer, critic
(w) Walker2D, actor
(x) Walker2D, critic
Figure 5: The evolution of the fraction of the gradient that lies within the high-curvature subspace throughout the training for PPO on all tasks. Evaluation of gradient subspaces with different num- bers of eigenvectors. Results for the actor and critic.
14

---

Published as a conference paper at ICLR 2024
Precise gradient; 1 EV
Precise gradient: 10 EVs
Precise gradient: 100 EVs
Mini-batch gradient; 1 EV
Mini-batch gradient: 10 EVs
Mini-batch gradient; 100 EVs
subrpoce
0.8
0.8
0.6
0.6
0.2
0.2
0.2
0.2
0.0
05
1.0
15
2.0
2.5
3.0
0.0
1.5
2.0
2.5 3.0
0.0
02
0.4
0.6
0.8
1.0
0.0.0
022
0.4
as
1.0
(a) Ant, actor
(b) Ant, critic
(c) Ball_in cup, actor
(d) Ball cup, critic
2.0
1.0
M
1.0
0.6
0.6
0.6
0.6
0.2
0.2
0.0
0.5
1.0
15
20
25
3.0
0.0
0.0
0.5
1.0
15
20 25 3.0
0.0
0.1 0.2
0.3
044
0.5
0.0
0.1
0.2
0.3
0.4
0.5
(e) Bip. Walker, actor
(l) Bip. Walker, critic
(g) FetchReach, actor
(h) FetchReach, critic
1.0
1.0
1.0
1.0
0.6
0.6
0.6
0.6
0.0
0.2
04
066
0.
1.0
0.0
0.2
0.4
0.8
1.0
0.0
1.5
2.0
255
s.0
0.0
1.5
2.0
25
300
(i) Finger-spin, actor
(j) Finger-spin, critic
(k) HalfCheetah, actor
1.0 (I) HalfCheetah, critic
1.0
1.0
1.0
0.8
0.4
0.4
0.4
0.4
0.0
0.0
0.0
02
0.4
1.
0.0
0.2
0.4
1.0
(m) Hopper, actor
(n) Hopper, critic
(o) LunarLander, actor
(p) LunarLander, critic
0.8
0.8
0.8
0.8
0.4
0.4
0.4
0.4
0.2
0..0
0.0
0.0
0.0
(q) Pendulum, actor
Pendulum, critic
Reacher, actor
Reacher, critic
tnc.
0.8
0.8
0.8
0.8
0.2
0.2
0.2
0.2
1.5
20
2.5
3.0
0.0
1.5 20 25 3.0
0.0
0.5
1.0
1.5
20
25
3.0
0.0
0.5
1.0
1.5
2.0
25 3.0
Ervironment (millions)
Environment staps
Environment steps
Environment steps
(u) Swimmer, actor
(v) Swimmer, critic
(w) Walker2D, actor
(x) Walker2D, critic
Figure 6: The evolution of the fraction of the gradient that lies within the high-curvature subspace throughout the training for SAC all tasks. Evaluation of gradient subspaces with different num- bers of eigenvectors. Results for the actor and critic.
15

---

Published as a conference paper at ICLR 2024
EV
10 EVs
100 EVs
1.0
1.0
1.0
0.8
0.8
0.8
0.8
Subopace
0.0
0.0
0.0
0.0
x10'
X10
x10*
x10
(a) Ant, actor
(h) Ant, critic
(c) Ball Sall_in_cupp. actor
(d) Ball _in_cup, critic
1.0
1.0
1.0
1.0
0.4
0.4
02
0.2
0.2
0.2
0.0
0.0
0.0
0.0
x10
x10
(e) Bip. Walker, actor
(I) Bip. Walker, critic
(g) FetchReach, actor
(h) FetchReach. critic
1.0
1.0
1.0
1.0
0.4
0.4
0.4
0.4
ww
0.2
02
0.2
x1o
(i) Finger-spin, actor
Finger-spin, critic
(k) HalfCheetah, actor
(I) HalfCheetah, critic
0.6
0.6
0.6
0.4
0.4
0.4
0.4
2
2
3
(m) Hopper, actor
(n) Hopper, critic
(0) LunarLander, actor
(p) LunarLandes critic
0.8
0.8
0.8
0.6
0.6
0.6
0.6
3
2
3
2
3
(q) Pendulum, actor
Pendulum, critic
(s) Reacher, actor
(1) Reacher, critic
Subopaceeveripp
0.8
0.8
0.8
0.8
0.6
0.6
0.6
0.6
0.0
0.0
0.0
0.0
x10
X10
x10
x10
(u) Swimmer, actor
(v) Swimmer, critic
(w) Walker2D, actor
(x) Walker2D, critic
Figure 7: The evolution of the overlap between the high-curvature subspace identified timestep t = 100,000 and later timesteps for PPO on all tasks. Evaluation of gradient subspaces with different numbers of eigenvectors. Results for the actor and critic.
16

---

Published as a conference paper at ICLR 2024
1 EV
10 EVs
100 EVs
1.0
1.0
1.0
0.8
0.8
0.8
0.8
Subopace
0.2
0.0
0.0
0.0
0.0
x10
X10
x10*
x10*
(a) Ant, actor
(h) Ant, critic
(c) Ball Ball_in_cupp_ actor
(d) Ball _aal__i_ccupp__ critic
1.0
1.0
1.0
1.0
0.4
0.4
02
0.2
0.2
0.2
0.0
0.0
0.0
0.0
x10
x10
(e) Bip. Walker, actor
(I) Bip. Walker, critic
(g) FetchReach, actor
(h) FetchReach, critic
1.0
1.0
1.0
1.
0.4
0.4
0.4
0.4
0.2
0.
02
x10
(i) Finger-spin, actor
Finger-spin, critic
(k) HalfCheetah, actor
(I) HalfCheetah, critic
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
3
2
3
2
3
(m) Hopper, actor
(n) Hopper, critic
(o) LunarLander, actor
(p) LunarLander, critic
0.8
0.8
0.8
0.6
0.6
0.6
0.6
2
3
2
3
(q) Pendulum, actor
Pendulum, critic
(S) Reacher, actor
(1) Reacher, critic
Subopaceeveripp
0.8
0.8
0.8
0.8
0.6
0.6
0.6
0.6
0.0
0.0
0.0
0.0
x10
X10
x11
x10
(u) Swimmer, actor
(v) Swimmer, critic
(w) Walker2D, actor
(x) Walker2D, critic
Figure 8: The evolution of the overlap between the high-curvature subspace identified timestep t = 100,000 and later timesteps for SAC on all tasks. Evaluation of gradient subspaces with different numbers of eigenvectors. Results for the actor and critic.
17

---

Published as a conference paper at ICLR 2024
C IMPACT OF SUBOPTIMAL HYPERPARAMETERS
Hyperparameters typically have a significant impact on the learning performance of policy gradient algorithms (Paul et al., 2019). In Section 4.2, we analyzed the gradient subspace for training runs with tuned hyperparameter configurations. However, which hyperparameters work well for given problem is typically not known priori. Finding good hyperparameters often involves running nu- merous RI. trainings, which might be infeasible when training on real-worke tasks. For our insights to be valuable for such real- world settings, it is crucial that suitable gradient subspaces can also be identified for training runs with suboptimal hyperparameters. Therefore, in this section, we investi- gate the robustness of the gradient subspace with respect to suboptimal hyperparameters. To get set of hyperparameters that are realistic but potentially suboptimal, we sample hyperparameter con- figurations from the ranges that found frequently in the tuned hyperparameters of RL Baselines3 Zoo (Raffin, 2020). Particularly, we draw the samples from the sets given in Table 1. Note that these also the bounds we use when sampling configurations for hyperparameter tuning.
Algorithm
Hyperparameter
Values
Logscale
learning.rate
[10-5, 10-1]
yes
batch.size
(32, 64, 256,
no
n_steps
{256, 512, 1024, 2048, 4096)
n.epochs
(5, 10, 20}
no
PPO
gamma
[0.9, 1]
gae.lambda
[0.9,
no
clip-range
{0.1,
no
ent_coef
[10-8, 10-2]
yes
net_arch
{(64, 64), (128, 128), (256, 256)
no
learning.rate
[10-5, 10-
yes
batch-size
{32, 64, 128, 256}
no
train_freq
[1,4, 4, 16, 32, 64)
no
SAC
gamma
[0.9, 1]
tau
{0.005, 0.01, 0.02]
no
learningsstarts
{100, 1000, 10000}
net-arch
{(64, 64), (128, 128), (256, 256))
Table I: Sets from which we draw random hyperparameter configurations for PPO and SAC. We refer each hyperparameter by its name in Stable Baselines3 (Raffin et al., 2021). Logscale means that we first transform the interval into logspace and draw the exponent uniformly from the trans- formed range. All other hyperparameters are drawn uniformly. These sets reflect common hyperpa- rameter choices from RL Baselines3 Zoo (Raffin, 2020).
We sampled total of 100 hyperparameter configurations per algorithm and task. Figure 9 dis- plays the distribution over the maximum cumulative reward that these hyperparameter configurations achieve. As expected, the variance in the performance is large across the hyperparameter configura- tions. While some configurations reach performance levels comparable the tuned configuration, most sampled configurations converge to suboptimal behaviors.
To display the values of the gradient subspace fraction and the subspace overlap compactly for all configurations, we compress the results for each training run to single scalar. For the gradient subspace fraction, we compute the mean the criterion over the timesteps. Regarding the subspace overlap, we choose the early timestep 4 100,000 in accordance to the experiments in Section 4.3. However, taking the mean over the remaining timesteps would not be faithful to how the subspace would typically be utilized in downstream applications. Such methods would likely update the gradient subspace after a given number of gradient steps instead of identifying it once and using for the rest of the training. The rate of this update would likely depend on the exact way that the
18

---

Published as a conference paper at ICLR 2024
1,000
6000
800
Maximum mwand
Moximum
400
2000
200
Finger-spin (PPO)
Fingerspin (SAC)
Walker20 (PPO)
Walker20 (SAC)
Figure Violin plot of the maximum cumulative rewards achieved by 100 agents trained with random hyperparameters on Finger-spin (left) and Walker2D (right). The black dots mark the per- formance of the individual random configurations, and the red cross signifies the performance of the tuned hyperparameters averaged over 10 random seeds. The random hyperparameters result in agents of vastly different maximum performance. While some of the random hyperparameters achieve performance comparable to the tuned agents, the bulk of random hyperparameters stagnate at suboptimal performance levels.
1.0
Gradient subspace Traction
0.8
0.6
0.4
0.2
0.0
Finger-spin (PPO)
singer-spin (SAC)
Walker20 (PPO)
Wal ker20 (SAC)
(a) Actor
Gradient subspace traction
Finger-sii (PPO)
Finger-spin (SAC)
WalkerDD (PPO)
Walker20 (SAC)
(b) Critic
Figure 10: Violin plot of the mean gradient subspace fraction throughout the training of 100 agents with random hyperparameters. The red cross marks the quantity for the high- perfopeeforming hyperparameters used throughout the paper (averaged over 10 random seeds). For PPO's s actor, the variance in the gradient subspace fraction is significantly higher than for SAC's. The critic's Aitic's gradient subspace fraction is very high across all hyperparameter configurations.
19

---

Published as a conference paper at ICLR 2024
50,000)
1.0
0.8
0.6
Finger-spin
Finger-spin (SAC)
Walker2D (PPO)
Walker2D
(a) Actor
-00-500000000 (0.000)
1.0
0.8
0.6
0.4
0.2
0.0
Finger-spin (PPO)
Finger-sin (SAC)
Walker20 (PPO)
Walker2D (SAC)
(b) Critic
Figure 11: Violin plot of the mean subspace overlap for 100,000 and t2 150,000 of 50 agents with random hyperparameters. The red cross marks the same quantity for the high-performing hyperparameters used throughout the paper (averaged 10 random seeds). For both tasks and networks, the variance of the subspace overlap is large across the hyperparameter configurations.
subspace is utilized. For this analysis, we choose l2 150,000, so that the timestep difference of l2 - 4 50,000 steps is significantly shorter than the total length of the training, but the number of gradient steps is large enough that the subspace could change.
The results for the gradient subspace fraction and the subspace overlap for random hyperparam- eters are visualized in Figure 10 and Figure 11, respectively. Figure 10a shows for SAC's actor that the gradient is well contained in the subspace for most random hyperparameter configurations. For PPO's actor, the spread in the gradient subspace fraction is significantly higher. As shown in Figure 10b, the gradient subspace fraction for the critie is very high for all hyperparameter config- urations. These results suggest that gradient subspaces can be utilized in SAC independently of the hyperparameter configuration, while PPO's actor might require more considerations.
Figure 11 shows that there is a significant spread in the subspace overlap for the random hyperpa- rameter configurations, which indicates that potential downstream applications might need to update the gradient subspace more frequently, depending on the hyperparameters.
20

---

Published as a conference paper at ICLR 2024
D DERIVATION OF THE GRADIENT SUBSPACE FRACTION OBJECTIVE
In this section, we derive Equation (6). Recall that Ps (vi....,,2) ux) E R is the matrix of the k largest Hessian eigenvectors, which is semi-orthogonal. We use 9 = P& IR" denote the projection of the gradient g into the subspace and back to its original dimensionality. In the following derivation, we drop the subscript of matrix Ps for ease of notation.
119 gll
Strac (P,9)
(9)
|19/1
||P* Pg 9|1
(10)
119112
|| Pg - g/1
(11)
||912
P2 Pg 9)t (P" Pg 9)
(12)
g² g
Pg - g) ( o )
(PT
(13)
g'g g
g PT PPT Pg 2g T PT Pg + g
(14)
g" g
9 P Pg 2g
pd Pg + g
(15)
g gTg 9
g g g pT Pg
(16)
9 9
9² 9 T
p Pg
(17)
g" 9
g² PT Pg
(18)
g" g (Pg)
(19)
g"gg
|Pol2 11g| |
(20)
Note that we used the fact that the pscudo-inverse of a semi-orthogonal matrix P is equal to its transpose p+ ps in step (11). Furthermore, we used the property PpT 1 of semi-orthogonal matrices in step (15).
E SUBSPACE OVERLAP FOR THE ENTIRE TRAINING
In Section 4.3, we showed the subspace overlap for a range of 400,000 timesteps. Figure 12 visual- izes the subspace overlap criterion with 4 100,000 for all future timesteps tasks that require training for 3 million steps. While in practical applications of gradient subspaces, the subspace would likely be updated multiple times during training, this visualization highlights the influence of the data distribution on the subspace. The plots show a small drop in the subspace overlap for SAC at 1.1 million steps. Since the replay buffer has size of million samples, this marks the point at which the original data from timestep f is completely replaced by new data collected by updated policies. Since the networks' training data is sampled from the replay buffer, this drop indicates that this change in the data distribution has negative effect the subspace overlap. The effect is gen- erally more pronounced for the critic than the actor because the actor's 's subspace overlap degrades faster and is already at relatively low level at the mark of 1.1 million timesteps. For PPO, there is no such drop in the subspace overlap since the algorithm does not use experience replay and instead collects new data for every update.
21

---

Published as a conference paper at ICLR 2024
PPO, actor
PPO, critic
SAC, actor
SAC, critic
1.0
1.0
Subspace overlap
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
x10
x10
(a) Ant
(b) HalfCheetah
1.0
1.0
Subspace overlap
0.8
0.8
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
t2
x10
t2
x10%
(c) Swimmer
(d) Walker2D
Figure 12: Evolution of the subspace overlap between the early timestep f = 100,000 (marked by the dashed gray line) and all future timesteps of the training. Results for the actor and critic of PPO and SAC. For SAC, small drop in the subspace overlap is visible in all plots at around 1.1 million timesteps. This marks the timestep at which the data in the replay buffer is replaced completely by new data, indicating that the data distribution affects the subspace overlap.
22


Reviews:
[{"review_id": "hlLmtcp2f1", "reviewer": "Reviewer_u1A7", "summary": "**EDIT: After the rebuttal, I am raising my score. The authors have promised that they have included additional explanations in the paper (and will add the rest) along with some experiments that they are running, and have modified the abstract to clearly specify the scope of this paper. With these additions, I feel that this paper can be a worthwhile addition to this conference.**\n\nThe paper empirically analyzes two popular policy gradient methods (PPO and SAC) on standard RL benchmark tasks and demonstrates (for the first time, to the best of my knowledge) that the actor and critic gradients lie in a tiny subspace (containing about 1% of the original network parameters) and that this subspace changes very slowly. Similar knowledge, as the paper discusses, about the gradients of neural networks in supervised learning have already resulted in various methods to improve training. This paper has the potential to result in similar advances for RL.\n\nI propose a weak accept for this paper because \n(0) The findings seem novel and can have a positive impact on PG algorithm development\n(1) I find the empirical methods of this paper ad-hoc and not well justified; and\n(2) Many empirical details are missing (although, maybe they are all minor details, and I'm over-estimating their importance, especially since the authors released their code);\n\nNote to the authors: I have a limited experience in deep learning, and as such my knowledge of most of the references and the methods employed in this paper is very limited. So if you feel that I don't understand the significance of your analysis, I would appreciate if you could point it out in your rebuttal (and finally in the revised paper). Thanks!", "questions": "I would really appreciate if the authors could clarify the following points (please refer to the Weaknesses section for details):\n\n1(b) Is my understanding correct here? Would these conclusions continue to hold for randomly chosen hyperparameters?\n\n2(a) Why can Eq. 5 helpful? How would the projections on random subspaces look like?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "**Originality and significance:** The paper (empirically) shows that for the first time that policy gradients (and the critic gradients) of the widely popular deep RL methods (PPO and SAC) lie in a very tiny subspace and this subspace remains somewhat stable during the course of learning. This result is interesting in itself as it gives us additional insights into the deep RL methods. Further, such a result can have significant implications on developing policy gradient methods, as outlined by the authors: Since deep learning methods often have a large number of parameters, this paper's insights can speed up existing RL methods by restricting optimization to a small fraction of those parameters. It can also result in better exploration techniques.\n\n**Quality and clarity:** The paper does a great job of outlining existing research: it's literature survey and references to relevant works is very thorough. The main ideas of the paper themselves are presented in an easy to follow manner, complemented by clear graphs and intuitive explanations. The writing itself is free from any grammatical mistakes (which is uncommon in the papers I have reviewed; so that is nice!).", "weaknesses": "# Major issues (these affect my score significantly):\n\nThe major weakness of the paper is a lack of rigor and concrete results. \n\n## 1. There is a very weak link between the experimental results and the claims made in the paper: ##\n\n(a) For example, see the following claims of the paper:\n- Section 1 (last paragraph): \"(i) parameter-space directions with significantly larger curvature exist in PG\": --> why can we say that the curvature is significantly larger? Figure 1 is highly qualitative in nature. In fact, Figure 2 is essentially qualitative as well. (And larger than what?)\n- Section 1 (last paragraph): \"(iii) the subspace is sufficiently stable to be useful for training\" --> how do the experiments justify the sufficiency? As far as I understand, Figure 3 only shows that the magnitude of the largest eigenvector projected to the subspace is, say, 0.4 for most part of the learning. What does 0.4 mean? How does that imply sufficient stability for learning based on methods that make updates in this subspace?\n- Section 1 (last paragraph): \"observe that the value function subspace often exhibits less variability and retains a larger portion of its gradient compared to the PG subspace\" --> inconclusive (Figure 2 shows very similar trends for actor and critic)\n\n(b) The results for the paper are specifically for well-tuned PPO and SAC agents on specific RL benchmarks, while the conclusions are drawn for general results. For instance, can we be sure that the subspace would also exist for a random hyperparameter configuration? The existence of subspace is even more important for a wide range of hyperparameters, since we won't know the optimal choice apriori for a random problem. Would the subspace be as restricted (for instance a dimensionality of 0.1% of the total parameter values) for a random SAC agent?\n\n(c) Section 4.3 (paragraph 3): \"Similar to the gradient subspace fraction results from Section 4.2, the subspace overlap is more pronounced for the critic than the actor.\" --> why? Are the scales comparable for the actor and critic graphs in Figure 3? Is this \"extra\" overlap really useful for designing algorithms? (Maybe it is, but without any explanations or additional experiments, this information is just speculative.)\n\n## 2. There is no rigorous motivation about various metrics used: ##\nIt seems that the paper adopted the metrics from Gur-Ari et al. (2018) for its experiments. However, these metrics seem somewhat arbitrary, and it is unclear how they relate to the usefulness/existence of the gradient subspace.\n\n(a) Gradient subspace fraction (Eq. 5) seems uninformative. Why showing that the gradient norm in the projected subspace is similar to the original gradient norm helpful? For instance, from the [Johnson-Lindenstrauss lemma](https://en.wikipedia.org/wiki/Johnson%E2%80%93Lindenstrauss_lemma) we know that any random projection matrix will ensure that the projected gradient norm is not too far away from the actual gradient norm. My point being, the metric in Eq. 5 doesn't seem to be useful right away. Maybe one way to make it more informative would be to establish a scale (so compare the norm of the projected gradient to the Hessian eigenvector subspace and the norm of the gradient projected to random subspaces).\n\n(b) Subspace overlap (Eq. 7): This metric is very complicated and I don't clearly see why it helps with showing the stability of the subspace and what that means for restricting training there. Maybe additional references could help with this.\n\n## 3. Missing details about the implementations: ##\n\nI give some examples below:\n- Section 4.3 (the equation for S_overlap): what is $k$ in the experiments? Does $k$ change with $t$ in the graph?\n- Section 4.3: \"where $v_i$ is the ith largest eigenvector at timestep t\" --> how is the largest eigenvector determined? Is it by the size of the corresponding eigenvalue?\n- Section 4 (paragraph 2): This section is unclear and doesn't provide sufficient details for reproduction (I understand the code does that, but without some details, the paper's results are very difficult to reason about). For instance, SAC usually has multiple Q networks; which ones do the experiments analyze?\n- Page 6, Figure 2: I was not able to understand the difference between the three categories: \"Estimated/true\" gradient, \"estimated/true\" Hessian, and why does that matter (probably Hessian is used to find the projection matrix $P_k$; if so, please explicitly mention it somewhere)? For instance, why do we care about estimated gradients? I understand that estimated Hessian can be important because we will use it to identify the policy gradient subspace and then use it downstream in algorithms. Some more justification about this could be nice.\n- Page 6 (first paragraph): Why was 2%, 0.14%, and 0.07% chosen? These seem highly arbitrary choices. How do the trends change when these numbers are changed?\n- Page 5: cutoffs for Equation 6 seem arbitrary.\n\n# Minor issues / suggestions (these do not affect my score as much; please ignore them if you don't agree):\n- The abstract seems incomplete/misleading in its present form, and it overclaims the contributions of the paper. For instance, it says \"we demonstrate the existence of such gradient subspaces for policy gradient algorithms.\" While this is not technically wrong to say that, the paper only empirically shows these subspaces, and that too for a very limited class of algorithms, i.e. deep policy gradient methods (in fact just two algorithms PPO and SAC). This could be easily rectified, for instance, by including this line from the introduction: \"This paper conducts a comprehensive empirical evaluation of gradient subspaces in the context of PG algorithms, assessing their properties across various simulated RL benchmarks.\" Further, the line \"Our findings reveal promising directions for more efficient reinforcement learning, e.g., through improving parameter-space exploration or enabling second-order optimization.\" seems to be suggesting that the paper also introduces methods for exploiting these subspaces, whereas that is just suggestion for future research.\n- The appendices are not properly referenced anywhere in the paper. At the very least, each section of the appendix should be referenced at the appropriate place in the main paper, explaining what additional details are there. Currently, appendices are a dump of graphs with no accompanying textual explanations (other than the caption). \n- Do you think the \"LoRA: Low-Rank Adaptation of Large Language Models\" (https://arxiv.org/abs/2106.09685) paper could be added to the related works section as well?\n- In Section 3.1: can $\\gamma$ be 1? Is the setting episodic? \n- In Section 3.1 (paragraph 2): The phrase, \"expected cumulative reward\" could be replaced by \"expected (discounted) cumulative reward\"\n- In Section 3.1 (paragraph 2): \"advantage function ... and can also be defined as\" --> why say \"can also be defined as\"? Isn't that the definition of advantage function? Maybe rephrase the sentence..\n- Section 3.2, first line: Maybe give a reference for the objective function $J(\\theta)$? \n- Section 3.2, first line: Also, the definition of the expectation is unclear: in particular, in its current form, the expression $J = \\mathbb{E}[ \\pi(a_t | s_t) \\hat A_t]$ seems to depend on the timestep $t$. That shouldn't be the case..\n- Section 3.2, first paragraph: What \"estimator of advantage function\" is being used? It is not specified.  \n- Section 3.2, first paragraph (above equation 2): what is the target value for the critic?\n- Section 3.3 (above Eq. 4): \"exponential of the learned Q-function\" --> isn't saying something like a Gibbs distribution or softmax distribution more appropriate?\n- Figure 2: since the \"true gradient\" is just the gradient computed using more state-action pairs, maybe calling it \"true\" is not accurate?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oGTOQ9s37i", "reviewer": "Reviewer_qF4u", "summary": "The papers verifies experimentally the existence of gradient subspace in reinforcement learning in the on-policy algorithm PPO and off-policy algorithm SAC.", "questions": "- Could you elaborate in more detail the motivation of identifying gradient subspace in comparison to parameter subspace, for the goal of guiding parameter-space exploration? This seems to be overlapping with Gaya et al. 2023's claimed benefit of identifying parameter subspace and it is not clear there what would be the benefits of using gradient subspace instead of parameter subspace in that case.\n- Is it possible to experimentally verify in a realistic example that the methods of Gaya et al. fail and the methods in current paper succeed?\n\n\n=============================\n\nPost-rebuttal: I thank the authors for the clarification on their contribution. I have raised my score accordingly.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Current literature on identifying gradient subspace focus on supervised learning, and related work in RL focus on identifying parameter subspace rather than gradient subspace. Therefore, this work is the first to identify gradient subspace in the context of RL and is informative for training RL algorithms. Project codes are provided for reproducibility.", "weaknesses": "It is unclear to which extent the contribution of identifying gradient subspace comparing to existing works in RL that focus on identifying parameter subspace (e.g. Gaya et al 2023 in the references) is significant, since both approaches have the same goal of improving training efficiency of policy parameters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fj8rBmCgzd", "reviewer": "Reviewer_CUg2", "summary": "Previous literature demonstrated that the gradient in supervised learning could be dominated by some of several high-curvature subspaces. Inspired by this, the authors investigate whether that is true in policy gradient methods and find that similar phenomena appears in both PPO and SAC by checking the gradient subspace fraction. Moreover, the authors demonstrate that these high-curvature subspaces remain stable across the training process with empirical results.", "questions": "One of the perspectives to make the paper stronger and more convincing is to show the unique conclusion of the policy gradient since this is an extension of the gradient subspace in the supervised learning setup. I could imagine more in-depth discussion in the paper could be helpful. For instances:\n\n- Investigate and explain why the PPO/on-policy method has a much lower gradient fraction in Figure 2, and how much of this is due to the data distribution shift.\n\n - Have more experiments/configurations in Figure 2 to make the conclusion and discussion more sound, for instance, whether the PPO low gradient fraction is consistent across more RL tasks or not.\n\n - Have demonstrative experiments to explore one of the potential RL applications (using the insight obtained from this experiment) to make it more convincing that these insights will be helpful for RL training.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is clearly presented and easy-to-follow, the motivation and the methodology are clearly described.\n- The work has demonstrated that the subspace exists in policy gradient methods as well, similar to what people have discovered in the supervised learning setup. This is done by relatively comprehensive experiments including different approaches to estimate the policy gradient and Hessian matrix and consider both the policy model and the critic model.\n- The authors also discuss how to leverage the insights to improve RL algorithms such as subspace-based optimization or parameter-space exploration.", "weaknesses": "One of the perspectives to make the paper stronger and more convincing is to show the unique conclusion and domain-specific insight for policy gradient learning, since most of the conclusions actually come from the gradient subspace paper under a supervised learning setup.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AryShRBFJc", "reviewer": "Reviewer_862E", "summary": "The authors investigate the *existence* of gradient subspaces in policy gradient algorithms. More specifically, they conduct an empirical campaign on two relevant methods (i.e., PPO and SAC) to verify that there exist directions with high curvature spanning a subspace that stays relatively stable throughout the training and that contains the gradient. To this end, the authors propose several numerical setups that verify such an existence. All experiments are conducted both for the actor and the critic gradients. Finally, the authors discuss how the existence of these subspaces could be used in practice to advance the state-of-the-art in policy search algorithms.", "questions": "None.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The existence of gradient subspaces has gathered consistent attention in the supervised learning community. This work aims to empirically show that such subspaces exist in RL as well. Given the numerous challenges that are introduced in RL, the existence of gradient subspaces is not trivial. In this sense, the work done by the authors aims at filling this gap within the literature, and, therefore, I retain it to be interesting for the community. \n\n- The paper is well-written. The main concepts and idea are easy to understand.", "weaknesses": "1. The contribution of the paper is focused on the **empirical existence** of gradient subspaces. Although the authors, in Section 5, discuss how the existence of this sub-space could be leveraged in practice, it remains an open question to provide empirical evidence of these eventual benefits.\n2. **Limited experimental campaign**. It has to be noticed that the contributions of this paper are only empirical. As a consequence, I would have expected more experiments to prove the empirical existence of these sub-spaces. Results are limited to 6 domains (i.e., Finger-spin, Ball_in_cup, Ant, HalfCheetah, Pendulum, Walker2D). I invite the authors to expand the set of domains considered.  \n3. **Clarity and writing (minor)**. I would encourage the authors to introduce and follow a more rigorous and formal definition of the elements that are used in the paper (e.g., subspace and so on). All the discussion is, indeed, rather informal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)