The Ladder in Chaos: Improving Policy Learning by Harnessing the Parameter Evolving Path in A Low-dimensional Space
Authors: Hongyao Tang
Min Zhang
Chen Chen
Jianye HAO
Venue: NeurIPS 2024 poster (2024)
Keywords: Reinforcement Learning
Policy Optimization
Primary area: reinforcement_learning
TL;DR: This paper reveals the parameter evolving path of policy network in a low-dimensional space and propose a general method to leverage the path for better learning performance.

Abstract: Knowing the learning dynamics of policy is significant to unveiling the mysteries of Reinforcement Learning (RL). It is especially crucial yet challenging to Deep RL, from which the remedies to notorious issues like sample inefficiency and learning instability could be obtained. In this paper, we study how the policy networks of typical DRL agents evolve during the learning process by empirically investigating several kinds of temporal change for each policy parameter. In popular MuJoCo and DeepMind Control Suite (DMC) environments, we find common phenomena for TD3 and RAD agents: (1) the activity of policy network parameters is highly asymmetric and policy networks advance monotonically along a very limited number of major parameter directions; (2) severe detours occur in parameter update and harmonic-like changes are observed for all minor parameter directions. By performing a novel temporal SVD along the policy learning path, the major and minor parameter directions are identified as the columns of the right unitary matrix associated with dominant and insignificant singular values respectively. Driven by the discoveries above, we propose a simple and effective method, called Policy Path Trimming and Boosting (PPTB), as a general plug-in improvement to DRL algorithms. The key idea of PPTB is to trim the policy learning path by canceling the policy updates in minor parameter directions, and boost the learning path by encouraging the advance in major directions. In experiments, we demonstrate that our method improves the learning performance of TD3, RAD, and DoubleDQN regarding scores and efficiency in MuJoCo, DMC, and MinAtar tasks respectively.

The Ladder in Chaos:
Improving Policy Learning by Harnessing the Parameter Evolving Path in A Low-dimensional Space
Hongyao Tangl:2; Min Zhang', Chen Chen', Jianye Hao Cooleege of Intelligence and Computing, Tianjin University Mila,, Université de Montréal
3 Department of Automation, Tsinghua University
Abstract
Knowing the learning dynamics of policy is significant to unveiling the mysteries Reinforcement Learning (RL). It is especially crucial yet challenging to Deep RL, from which the remedies to notorious issues like sample inefficiency and learning instability could be obtained. In this paper, study how the policy networks of typical DRL agents evolve during the learning process by empirically investigating several kinds of temporal change for each policy parameter. In popular MuJoCo and DeepMind Control Suite (DMC) environments, we find common phenomena for TD3 and RAD agents: (1) the activity of policy network parameters is highly asymmetric and policy networks advance monotonically along a very limited number of major parameter directions; (2) severe detours occur in parameter update and harmonic-like changes are observed for all minor parameter directions. By performing novel temporal SVD along the policy learning path, the major and minor parameter directions are identified as the columns of the right unitary matrix associated with dominant and insignificant singular values respectively. Driven by the discoveries above, we propose a simple and effective method, called Policy Path Trimming and Boosting (PPTB), as general plug-in in improvement to DRL algorithms. The key idea of PPTB is to trim the policy learning path by canceling the policy updates in minor parameter directions, and boost the learning path by encouraging the advance in major directions. In experiments, we demonstrate that our method improves the learning performance of TD3, RAD, and DoubleDQN regarding scores and efficiency in MuJoCo, DMC, and MinAtar tasks respectively.
Introduction
Deep Reinforcement Learning (DRL) is far from well understood, although its great potential has been demonstrated with a lot achievements in different practical problems [Badia et al., 2020 Shah et al.] 2022 Fawzi et al.) 2022 Degrave et al., [022 OpenAl 2022]. Consistent efforts are made to gain better understanding of the learning dynamics of RL agents. Different from the studies with tabular [Sutton and Barto 1988 and linear approximation [Ghosh and Bellemare, (020), understanding the learning dynamics of DRL agents challenging. The difficulty comes from complex interplay between Deep Learning models and RL algorithms, and further escalates when only limited online interactions or offline logged data are considered.
Recently, there have been few works that study the learning dynamics of DRL agents from different perspectives. major stream of works among them studies the co-learning dynamics between representation and DRL functions (usually the value network) [Dabney et al.) 2021 Kumar et al.
"Corresponding author: Hongyao Tang (aanghoogyao@qqueee...i.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

2021 Lyle et al. 2022 Nikishin et al., 2022 Tang et al. 2022b]. The focus of this stream is that the DRL agent sver-shapes its representation toward early experiences and objectives, and becomes less capable of learning for the later learning process. This degradation finally leads to myopic convergence or even divergence. A related work [Sokar et al. [023] presents a phenomenon called Dormant Neuron in DRL. It reveals that the neurons of typical value networks gradually become inactive during the learning process, leading to the loss of network expressivity. From another angle, phenomenon called Policy Churn is discovered by [Schaul et al. 2022]. It shows that the greedy policy of a typical value network changes on about 10% of the states after only a single update. These discoveries provide useful insights for understanding the learning behavior of the DRL agent, based on which new methods can be proposed to improve the learning performance of DRL.
In this paper, we aim to unveil the learning dynamics of the policy network during the training process of typical DRL agents. Specifically, we focus on how the parameters of the policy network evolve along the policy learning path, i.e., the sequence of historical policy networks. Taking MuloCo [Brockman et al., [016] and DeepMind Control Suite (DMC) Tassa et al.) 2018] as typical DRL environments, we conduct a series of empirical investigations on the policy learning path of TD3 Fujimoto et al. 2018 and RAD [Laskin et al. 2020 agents respectively. From the perspective of accumulated absolute parameter change and novel temporal SVD view, we present four commonly observed phenomena on the policy learning path. The main message from the phenomena is that the policy networks of DRL agents advance nearly monotonically along only very few major parameter directions and show harmonic-like oscillations on almost all other minor ones.
This drives us to ask the question: can we make the DRI. agent focus on the policy learning along major directions and suppress the oscillation in minor directions? To this end, we propose new method called Policy Path Trimming and Boosting (PPTB). The key idea of PPTB is to trim the policy learning path by canceling the policy parameter updates in minor dimensions, and boost the learning path by encouraging the advance in major directions. The method is simple and general plug-in improvement for DRL agents, which can be implemented with only a few lines of code modification to most DRL methods. Finally, we evaluate the effectiveness of PPTB based on TD3, RAD, and Dou [van Hasselt et al. 2016 in MuJoCo, DMC, and MinAtar Young and Tian 2019 environments.
Key contributions of this work are summarized below: (1) We propose a temporal analysis view to study the policy learning path of typical DRL. agents. (2) We present four common phenomena from our empirical investigations, which reveal the policy evolving path in low-dimensional space. (3) We propose an easy-to-implement and general improvement for general DRL algorithms, which is demonstrated to improve the learning performance of TD3, RAD, and DoubleDQN regarding scores and efficiency in MuJoCo, DMC, and MinAtar environments.
2 Preliminaries
Markoy Decision Process (MDP) Consider a Markov Decision Process (MDP) P A. P, AS R. 7, Po. T), defined with state set S, action set A, the transition function
+ R. the reward function R : S x A + R, the discounted factor y E [0. 1). the initial state distribution Po and the horizon T. The agent interacts with the MDP by performing its policy π S - P(A) that defines the distribution over all actions for each state. The objective of RL agent is to optimize its policy to maximize the expected discounted cumulative reward J(z) = E. Eto Y'ri), where Sq ~ Po (So). at (st).
8141 ~ P (s141 at) and rt = R (st, at). The state-action value function Q" is defined as Q* a)= = E, Σ 2 Tr so = s, ao a for all s,
Deep Reinforcement Learning (DRL) With function approximation based on deep neural net- works, an RL agent is able to deal with large and continuous state-action space. Conventionally, Q" can be approximated by Qs with parameters ó typically through minimizing Temporal Dif-
ference loss (Sttoon and Barto 1988]. i.e., L(Ø)= 2 (s, - Eae~z(s)) (r d'))]² parameterized policy ne, with parameters 0. can be updated by taking the gradient of the ob- jective, i.e., 0' <- 0 + V)J with learning rate n. Therefore, starting from a initial pol- icy oh, the DRL agent proceeds along the learning process and obtains a sequence of policies {Te, T J. For two representative DRL algorithms, Deterministic Policy Gradient (DPG) theo-
2

---

rem Silver et al. 2014] is often used to update a deterministic policy with the gradient: (T)( =
Benpto V V Qo a) [== where pa is the discounted state distribution under policy Te; Soft Actor-Critic (SAC) [Harrojj al. 2018 updates a stochastic policy with the gradient: VaJ(TT) E Egri'n Vo log ( + (V. a log Th (a]s)) V VaQo(s, Vo sam (with noise e and implicit function fo for re- ce-parameterization) based on the maximum maximum-entropy objective (TTe) Ex (2 n'T, + {+( where & is the temperature of entropy term.
3 Phenomena on DRL Policy Learning Path
In this section, conduct empirical investigations on the typical DRL policy learning process. In specific, we use official codes of TD3 [Fujimoto et al. 2018 and RAD [Laskin et al., 2020) for OpenAl MuJoCo [Brockman al. 2016 continuous control environments and visual-input
DeepMind Control (DMC) Suite [Tassa et al. 2018) environments respectively. Both of them use
policy architecture of two-layer MLP (where the output layer is viewed as Layer 3). Note that we exclude the convolution layers for image representation in RAD here and focus on the policy part [Chung et al. 2019]. All the experimental details can be found in Appendix D
First of all, we call the sequence of policies {T1, TTi.TT..... .T...... Tnl or policy parameters {01. ,,,....,,8 0,) obtained during the learning process of typical DRL algorithm as policy learning path or policy parameter path throughout this paper. The policy parameters 0g is an m-dimensional vector 0; = 10 2. 0mmm Rm In the following, investigate how policy network parameters evolve along the policy learning path from different angles.
3.1 Policy Parameter Change and Detour
We start by tracking the absolute change amount of each parameter in policy network. Given any j E {1,2,-.. ..), .), we first introduce three quantities for our following investigations.
Accumulated Parameter Change: (10: - Σi= [1+11,5 Bt,5-
Final Parameter Change: A 1/ ((0:) 10n. B1.91-
Parameter Update Detour Ratio: r.th ([0:) > 10: 14113-01:1 (n.. 01.
We ATt A r as abbreviations when the context is clear and write (A))), (1.5t)) as the vectors consisting of the quantities from all parameters. It can be derived that the larger r is, the more detours there are in the evolving path of parameter j. For each environment, we run TD3 RAD for three trials and collect the policies along the learning process at intervals. We then calculate vectors (Af"), (1 and plot their CDF histograms in layer-wise manner (i.e., Layer 1,2,3). The results are shown in Fig. [] The results are similar in almost all other MuJoCo and DMC environments, and the complete figures can be found in Appendix D.I
We begin by assessing the magnitude of accumulated changes in the policy parameters. CDF histograms of vector (Ar))) (as shown in left panels of Fig. (a), (b)) indicate that a certain portion of parameters (ranging from 10% to 40% across environments) in the second and output layers experience minor accumulated changes. In contrast, the cumulative changes for nearly all parameters in the first layer are significantly higher, with thresholds around 0.7 for Hopper and 0.6 for walker- walk. This observation marks initial key finding.
Phenomenon 1.1 (Parameter Change Asymmetry): There is significant discrepancy in the change amount among policy parameters. The second and the output layers have similar patterns which differ from that of the first layer.
Further discussion about Phenomenon 1.1 can be found in Appendix D.I
Moving forward, we examine how much each policy parameter detours from its initial value to its
final value. The CDF histograms of (1-5ul) (as shown in the right panels of Fig. (a) and (b)) indicate that a significant proportion of parameters exhibit substantial detour ratio, which we identify as the second phenomenon.
3

---

(a) MuJoCo/Hopper
(b) DMC/valkerwwalk
Figure I: Policy Parameter Change and Detour on policy paths obtained by TD3 on MuJoCo/Hop-
per and RAD on DMC/walker-wee walk. We use the colors = to denote Layer 1,2,3 respectively. Each subfigure contains: (left) the CDF histogram of the elements in the vectors (A JELayer x). and (right) the CDF histograms of the elements in the vectors (r (Eyyaer a.) for k E 21,2,33. Only upper 80% values according to (A,)) are taken to plot the histogram for (r)))) for meaningful analysis; extreme elements in (r))))) are neglected for clarity. See Phenomenon 1.1 and 1 for conclusions.
Myyr Pa008
(a) Singular Value Information Curves
(b) SVD Left Unitary Matrix (U) Investigations
Figure 2: Temporal SVD Analysis of policy paths obtained by RAD on DMC/walkerwwalk (a) Curves of (53 against various threshold | for the three layers and three periods (i.e., early, middle, later) of the learning process for temporal view. X-axis represents D(3) and y-axis represents different (b) The second and third panels depict the curves of u.,k. with each curve illustrating the evolving path of the k-th coordinate. The fourth and fifth panels display the curves of detour ratio (r."d) and final change (A) The X-axis represents the index of the singular values.
Phenomenon 2.1 (Parameter Update Detour): There are severe detours in policy parame- ter update. All the three layers show similar patterns.
It can be easy to consider that such detours oscillations in policy parameter updates can be mainly attributed to noisy policy gradients. This also reveals the correlation between policy parameters where most of them can not be updated independently.
3.2 Temporal SVD Analysis of Policy Learning Path
The two phenomena observed above naturally raise further questions: 1) The asymmetry of parameter change shown by Phenomenon 1.1 indicates the imbalanced importance of parameters. Can rule the important parameters or parameter directions off from the less important ones? 2) For the detour shown by Phenomenon 2.1, in their detour behaviors? And do important parameters or parameter directions detour less?
identify the difference among parameters parameter directions
In this section, we make use of temporal viewpoint on policy path. We introduce Temporal Singular Value Decomposition (SVD) as our main tool for the following investigations. Given policy path (0, ......, Jn} with 0, 10.., ......... m, where m (2 10a usually) is the dimenionality of

---

policy parameters. We then stack the parameter vectors along the policy path and form a temporal policy parameter matrix, for which standard SVD can be performed:
01
01,1
1
02
22.1
...
22.
U1.1
u1,
V1.1
V1,
: 0n
=
On.1
On
Un,1
Un.d
0d
Ud,1 Ud.m
x
where U, V are left and right unitary matrices, the singular values are indexed in decreasing order with d = min(n, m). For the convenience of expression, we use u4,, for the i-th row vector of U and use u, for the j-th column vector (and the same way for V The temporal SVD offers us an angle to view how policy evolves in lower-dimensional space: we can now take the row vectors of U as new d-dimensional coordinates for policies along the policy path (i.e., Wi, for the i-th policy) regarding the scaling vector Diag( (E) and the parameter subspace spanned by the row vectors of V i.e. Span( [vi,,, 12,....... }) Rd. In the following, also call {vr,.} SVD directions. For any index d, it is obvious that Va,j is the evolving path of the j-th new coordinate.
To answer the questions listed at the beginning of this subsection, we introduce two more quantities:
Singular Value Information Amount for dimensionality number k E (1....,d} d): ag
Σ/ %i Li-1 o,
SVD Major Dimensionality regarding an information threshold B E 1): D(3) min(k E {I.....d) d} : 04 2 B).
Note that d = [1) and D (0.99) recovers approximate rank used in few recent works Yang et al. 2020
Kumar et al. 2021 Lyle et al. 2022]. Additionally, a smaller D(3) implies fewer dominant SVD directions when 3 is held constant.
We also use the policy path data obtained by TD3 on MuloCo and RAD on DMC as in Sec. 3.1 First, we plot D(33) against various threshold candidates 3 E {0.5, 0.6, 0.7, 0.8, 0.85, 0.9, 0.95, 0.99). } In addition to layer-wised manner, we consider three periods (i.c., early, middle, and later) of learning process for temporal view. The results for DMC/walker-wal are shown in Fig. 2(a) The results are similar in almost all other MuJoCo and DMC environments and the complete figures can found in Appendix D.2
The following observations can be made: (i) Within the range from 0.5 0.8, we observe that ((3)) for all three layers in both environments consistently remain near O. This observation suggests that the policy learning path primarily proceeds within a low-dimensional subspace of the entire parameter space, particularly within the dimensions spanned by the most dominant SVD directions. This provides additional insights into Phenomenon 1. elucidating which parameters undergo more significant changes. (ii) The curves of the second layer show more pronounced steepness near 3=0 O, indicating that the parameters in this layer evolve within a subspace characterized by fewer dominant SVD directions compared to those in other layers. This observation also matches the results Fig. as the second layer is often more over-parameterized than the other layers. (iii) As training advances from period to period the curve's 's steepness around 0 correspondingly increases, revealing a growing concentration on a diminishing number of dominant singular values. The higher concentration of later periods is casy to understand since policy improvement gradually becomes slower. We can summarize these observations as our third phenomenon.
Phenomenon 1.2 (Singular Value Information Concentration): The singular value infor- mation is highly concentrated on the first several singular values. The concentration is more evident for the second layer and later periods.
Further, regarding the second question raised in the beginning, expect to find some correlation between the policy parameter path in the low-dimensional subspace obtained by SVD and the improvement of policy performance. To this end, in Figure 2(b) examine the evolving path of the k-th coordinate by plotting coordinate vectors u.,k (panels 2, 3). We further present the curves of
detour ratio (r.mdd) (panel 4) and final change (A (Tt) (panel 5) with respect to the singular value index
5

---

Policy Parameter Space (0)
Original Poliey Path
Trimmed and Boosted Policy Path
Mejor Direction
Minor Direction
Trimming in
Boosting in
Local Minor Direction
Local Major Direction
Figure 3: A conceptual illustration of Policy Path Trimming and Boosting (PPTB). The 2D view of exemplary policy learning path in policy parameter space (left) and the corresponding policy learning path improved by PPTB (right).
Í. These results are against the policy performance curve (panel 1), and we focus on only the period with significant policy improvement, i... the left of the blue dashed vertical line.
According to the results in Fig. 2(b) we observe the final phenomenon.
Phenomenon 2.2 (Policy Path in Major and Minor SVD Directions): The extent of the detour becomes increasingly pronounced the coordinate index k increases (from the major SVD directions to the minor ones). Specially, ux,1 is almost monotonic while u,,A shows harmonic-wave changes with overall increasingly higher frequencies for k 2
is surprising to observe Phenomenon 2.2. Intuitively, it indicates that the policy parameter path proceeds monotonically along one major direction and oscillates in other directions with frequencies inversely proportional to singular values. This empirically supports our hypothesis that important (i.e., major) parameter directions detour less while insignificant directions detour severely. Since typical policy gradients are derived regarding approximate value estimates, we suggest that the dynamics of policy parameters may be closely related recent studies on the learning dynamies of value function [yle et al. 2021 2022]. We leave it as a major direction of future work.
have observed that the policy parameter path evolves mainly in low-dimensional parameter subspace (spanned by (U,,,)). }). This is commonly seen in our empirical investigations for typical DRL algorithms in popular environments with proprioceptive observations. A natural idea is: why not let the agent focus on the policy update in the parameter subspace, by following the major SVD directions and neglecting the insignificant SVD directions? Moreover, somewhat excitingly, it seems that Ua,1, Ua,z has a strong correlation to policy performance. We are curious about whether it is possible leverage the correlation boost the learning process. In the next section, we study on these points along with the proposal of Policy Path Trimming and Boosting.
For brief summary, till
visual
Policy Path Trimming and Boosting
Driven by the phenomena discovered in the previous section, propose a simple and effective method, called Policy Path Trimming and Boosting (PPTB), as a general plug-in improvement to DRL algorithms. In the following, we introduce the details of the two components of PPTB, i.e. Policy Path Trimming (Sec. 4.1 and Policy Path Boosting (Sec. and then the general implementation of DRL with PPTB (Sec. 4.3
4.1 Policy Path Trimming
As summarized in Phenomenon 1.2, we have observed that the policy path mainly evolves in low- dimensional parameter subspace with large proportion of singular value information concentrated in the first several singular values. Our first idea is to truncate the parameter change in minor SVD directions and only keep the change in major ones based on the Temporal SVD of the policy path. We call this method as Policy Path Trimming (PPT).

---

Given a policy path {01, 12..0..... ....... } and the number of major directions to remain (usually < min(n, For policy with original parameters 0, PPT reconstructs policy parameters by only taking into consideration the first SVD directions:
U1
0,
[1,p1
Wjrre
(I)
rr
Ure
1
V [[mm))
Note that we use : in subscripts and [+--] denote the slices of vector and matrix respectively.
conceptual illustration of PPT is shown in Fig. As shown by the red arrows and cross, PPT trims the policy parameter update in minor SVD directions and enforces the policy path proceeds in major directions, i.e., in the subspace Spaa((1)].. U'......, vr }). Intuitively, this suppresses the detours and oscillations of parameter update (recall the results in Fig. 2(b).. Therefore, we expect PPT to improve the efficiency of policy learning process in this sense.
One may worry about whether the trimmed policy parameters 0f still ensure an effective policy. For sanity check, we compare policy performance between 0g and 0' regarding different choices of Tr. We refer the readers to Table 4 to in Appendix D.3 We found that small (5 128) is sufficient to ensure a valid recovery of policy performance, while increasing rt shows no significant difference. Somewhat surprisingly, reconstruction, which supports the effectiveness of PPT to some extent. In addition, PPT can be viewed particular way to do network resetting to alleviate the primacy bias (Nikishin et al. 2022 More discussion on this can be found in Appendix B
found r, (S 64) often achieves improved performance after
4.2 Policy Path Boosting
In addition suppressing the parameter change in minor directions, are interested in accelerating policy learning by leveraging the dynamics pattern of major SVD directions as noted in Phenomenon 2.2. In particular, we propose boost the change of u.,1, 1, Ux since they show near monotonic changes corresponding to the improvement of policy performance in Fig. 2(b) We call this method as Policy Path Boosting (PPB).
For a policy with original parameters 0g among the policy path {0,, 02,....... 0n). PPB modifies 0n by increasing 1n Un, along the temporal direction follows with boosting coefficient ps:
un Un + Pb (un,. W1,-)
0, = [ūn,1 ún, Un.3
un ΣVT
concat 1:a
1:2
a..
PPB only modifies 1,, Un while keeps other parts unchanged. As illustrated by the green arrows in Fig. 3 Intuitively, PPB boosts the policy parameter update in the first two major directions to accelerate the improvement of policy performance.
4.3 DRL with PPTB
Now, we are ready to propose PPTB as a combination of PPT and PPB. Formally,
6 concat (..1:2 04,3.r,) ]E[1 r [1 rz]
(3)
Although perform PPTB for any policy arbitrary policy path, in practice consider the policy path that consists of the historical policies within a recent window and the current policy, and we perform PPTB for the current policy.
For almost all off off-the-shelf policy- based DRL algorithms, PPTB can be implemented and incorporated by adding the following three steps to the conventional policy update scheme: (1) Initialize a policy buffer and store parameters of the current policy at intervals along the policy learning process; (2) On certain occasions, retrieve the recent policy parameter path and perform Temporal SVD; then do policy path trimming and boosting for the current policy. (3) Load the consequent policy parameters processed by PPTB back to the current policy.
Apparently, PPTB

---

In our practical implementation used by our experiments, we only make the modifications of about 10-line core codes in the official implementations of TD3 [Fujimoto et 2018 and RAD [Laskin et al. 2020). We refer the readers to Algorithm | in the appendix for pytorch-like pseudocode.
Experiments
5.1
Performance Evaluation of PPTB
Setups To evaluate the performance of our proposed method PPTB, we consider the continuous control environments in OpenAI MuJoCo [Brockman et al., 2016] and DeepMind Control Suite (DMC) TTassa et al., 2018]. which cover both proprioceptive and visual observation. We use TD3 [Fujimoto et al. 2018] and RAD [Laskin et al. 2020] as the base algorithms for MuJoCo and DMC respectively, due to their simplicity and effectiveness. We use the official codes of TD3 and
RAD and modify them according to Algorithm [ to implement PPTB with no other change to the original implementation. Our code is available in httht:://ggitte thubbbb.. com/ bluecontra/PPT3
Moreover, we also evaluate the performance PPTB for value-based RL, although the greedy policy is implicitly derived from the value networks. Specifically, we use three tasks from MinAtar (Young and Tian 2019] and use DoubleDQN [van Hasselt et al. 2016) as the value-based RL baseline. More implementation details are provided in Appendix
Results for AC Methods We train the agent for I million timesteps and evaluate the agent every 5k timesteps for each agent-environment configuration. We run each configuration with six random seeds. We consider two evaluation metries: (1) Score: the best average evaluation returns (across multiple runs) over the course of learning, which is used by TD3 (Fujimoto al. 2018 : (2) AUC: the mean of average evaluation returns over the course of learning, which is also used in (Kumar
al. 2022]. The former cares about effectiveness while the latter measures efficiency and stability, which is also significant to the practical use of DRL algorithms.
For MuloCo environments, we report Score and AUC for 1M time step training; for DMC, report for 100k, 500k as usually done in prior works [Laskin et al. [002). For the convenience of comparing across different return scales, we report the comparative improvement (/") and the aggregate results by normalizing with random-agent baseline as O and the DRL base algorithm (i.e., TD3 or RAD) as The score of the random-agent baseline is omitted. We report the means and standard deviation errors across six independent trials.
The results for MuloCo and DMC reported in Table I and 2 respectively. The results show the overall improvements brought by PPTB for both TD3 and RAD, indicating its compatibility and effectiveness. The improvement is obvious in AUC, which means PPTB improves the learning efficiency and stability of the base algorithms. Somewhat surprisingly, we can observe that PPTB outperforms the base algorithms by large margin in several environments, e.g., Walker2d and Ant in MuJoCo, and Walker-walk in DMC. To some extent, this reveals the potential of studying and modifying the policy learning path to achieve general improvements DRL agents.
Table 1: Performance evaluation of Policy Path Trimming and Boosting (PPTB) for TD3 Fujimoto et al. 2018 in four MuJoCo environments. The learning curves are in Figure 10
ENVIRONMENTS
METRICS
TD3
PPTB
SCORE
10548 + 357
11103 # (5.12%
HALFCHEETAH
AUC
7981 4: 304
8689 1:: 76 (8.55%
SCORE
3394 + 59
3420 + 54 (0.77%
HOPPER
AUC
2005
2249 114 (12.28%
SCORE
3406 1: 436
4385 + 164 (28. (88.76%
WALKER2D
AUC
1977 + 353
2805 85 (41.92%
SCORE
4177 1: 451
5147 1: 449 (22.81%
ANT
AUC
2624 +:
3542 392 (34.02% 7)
SCORE
1.0
1.1436
AGGREGATE
AUC
1.0
1.2419
8

---

Table 2: Performance evaluation of Policy Path Trimming and Boosting (PPTB) for RAD Laskin et al. 2020 in four DeepMind Control (DMC) environments. The learning curves are in Figure 11
ENVIRONMENTS
METRICS
RAD (100K)
RAD-PPTB (100K)
RAD (500K)
RAD-PPTB (500k)
SCORE
553 : 71
592 # 55 (7.09%/)
898 +: 55
970 + (8.04% /)
FINGER-SPIN
AUC
201
254 +: 40 (26.76%
695
767 (10.40%
SCORE
210 = 48
349 +: 35 (86.87%
/)
923 +
940 + (1.94% ()
WALKER-WALK
AUC
104 22
148 + 15 (81.48% //)
583 30
672 11 (16.69%; /
SCORE
235 15
268 + 18 (30.84% /
836
860 + (3.38% /)
CARTPOLE-SWINGUP
AUC
158 + 6
189 + 2 (103.33% /
530 +: 18
572 #: 20 (10.44% 2)
SCORE
360
394 + 10 (10.11%
574 ::
605 = 10 (5.63%
CHERTAH-RUN RUN
AUC
194 5
207 + 13 (7.64%/
428 +: 10
452 + 6 (5. (5.94%
SCORE
1.0
1.3372
1.0
1.0474
AGGREGATE
AUC
1.0
1,5480
1.0
1.1086
Results for Value-based Methods We run 3M time steps for SpaceInvader and Breakout, and 5M time steps for Seaquest. We reported the means and standard errors of the final episode returns across six seeds in 3
Table Performance evaluation regarding Score of Policy Path Trimming and Boosting (PPTB) for DoubleDQN van Hasselt al., 2016 in three MinAtar environments.
The results show that PPTB also improves the learning performance of DoubleDQN somewhat significantly in MinAtar tasks with discrete ac- tions. Note that for DoubleDQN, value net- works are learned to derive the greedy actions. Intuitively, the learning dynamics of the value learning path and the policy learning path in AC methods are different as typical TD learning adopts bootstrapping and semi-gradient (Sutton and Barto 1988). We view our experimental results here as preliminary evidence of the effectiveness of PPTB for value-based methods, which also inspired us further investigate this point in the future.
ENVIRONMENTS DOUBLEDON
DOUBLEDON
SPACEINVADER
77.63 + 6.85
99.08 + 6.76
SEAQUEST
11.00 + 1.70
25.43 +: 4.38
BREAKOUT
28.85 4.01
32.33 +: 2.92
5.2 Investigation on Policy Learning Path of Behavior Clone
The policy learning path is influenced by multiple factors, including, policy learning algorithm, distribution of training data, network architecture, optimizer, etc. To expand our study on typical online RL bit, we investigate the policy learning path of Behavior Cloning (BC) with Adam and vanilla SGD on aaffcheetah-mediun--ehhhe additional setting provides two comparison angles, i.e., RL v.s. BC, Adam v.s. SGD (where using BC and the same offline dataset fixes the influence of data distribution). Similar to our investigation in
an offline dataset in D4RL [Fu et al., 2020]. This
Sec. 3] we perform parameter analysis and Temporal SVD analysis the logged policy paths through learning under different configurations. The results are presented in Figure and 12
For the parameter analysis, we can observe that BC and RL show different distributions of network parameter update amount. Compared with the RL cases reported in Figure 5[a) the parameter update amount is more Gaussian distribution like for BC (Adam) in Figure (aa In addition, by comparing the parameter analysis of BC (Adam) and BC (SGD), somewhat surprisingly, we can observe that the asymmetry of parameter update amount turns out be more severe in BC (SGD). The momentum mechanism used in Adam leads to a relatively more evenly distributed network parameter update amount, while SGD has large number of minor updates (also reflected by the flat curve in Figure 4(d) the first column) and a long tail in distribution.
For the Temporal SVD analysis, we can observe that BC (Adam), BC (SGD) and the RL cases show different patterns in terms of temporal SVD evolvement. BC policies show wavelet-like oscillations (Figure 4 where the amplitude of the oscillations seems to decrease throughout training; while the oscillations of RL are more Harmonic-wave-liie (Figure ((a)).
These observations reveal that low-dimensional policy paths can exist in many policy learning problems with different features influenced by multiple factors. We expect further studies in this direction in the future.

---

(a) Parameter Analysis of BC (Adam)
(b) Temporal SVD Analysis of BC (Adam)
C1, 00000
caange De.3
(c) Parameter SVD Analysis of BC (SGD)
(d) Temporal SVD Analysis of BC (SGD)
Figure 4: The parameter and Temporal SVD analysis for Behavior Cloning with Adam and SGD
optimizers in D4RL halfcheetah- mecirtt--mddiu-rpplaayy
6 Related Works
In the topic of general Deep Learning, there have been a large number of efforts devoted consistently during the past decade for the purpose of understanding the learning dynamics and behaviors of deep neural networks, e.g., Neural Tangent Kernel (NTK) [Jacot et al.] 2018) and Lazy Training [Chizat iet al. 2018). Recently, there have been few works that study the learning dynamics of DRL agents from different perspectives. Among them, major stream of works study the co-learning dynamics between representation and RI. functions (i.e., value network or policy network) [Kumar et al., 2021 Lyle et af. 2022 Nikishin et al., 2022]. The focus of this stream that the DRL. agent over-shapes its representation towards early experiences and objectives (e.g., TD targets of early policies) and becomes less capable for the later learning process. Such a degradation becomes more severe and detrimental gradually due to the non-stationary learning nature, finally leading to myopic convergence or even divergence. In this work, we study the learning dynamics of typical DRL policy networks, mainly from the angle of policy network parameters. To the best of our knowledge, we are almost the first to study how the parameters of practical DRL policy networks evolve.
To our knowledge, there are some works that conduct their study from the angle of the path of learning. Dabney et al. [021] takes inspiration from the study in [Bellemare et al., 2019] and proposes the concept of the value improvement path. With this concept, a new method for representation learning is proposed by approximating the value functions along the path. From another angle, [Tang et al. [2022a] studies the generalization along the policy learning path. Through learning a low- -iweennnioonal policy embedding, an improved generalized policy iteration (Sutton and Barto 1988] that leverages policy generalization is proposed. Closely related to our work, a very recent work [Schneider et al. 2024 demonstrates the existence of gradient subspaces deep policy gradient algorithms with experimental investigations. In this paper, we investigate the dynamics of policy network parameters from the perspective of Temporal SVD. Moreover, we propose a new method that leverages the parameter evolving path in the oww-diensioonal space, which demonstrates the potential for improving learning. In essence, the policy learning path is a special scenario of stochastic gradient descent (SGD) in DRL, thus this work is related to studies the dynamics of SGD or optimization (Gur-Ari al. 2018
Gressmann et al. 2020 Li et al. 2023 in more general speaking.
7 Conclusion
In this paper, we aim to unveil the learning dynamics of policy network over the training course of typical DRL agents, mainly from an empirical perspective. Focusing on the concept of policy learning path, we conduct series of empirical investigations and summarize four common phenomena, revealing that the policy learning path of typical DRL agents evolves mainly in a low-dimensional space with monotonic and waggling changes in major and minor parameter directions respectively. Driven by our discovery, we propose simple method, called Policy Path Trimming and Boosting (PPTB), as a plug-in improvement to general DRL algorithms. We demonstrate the effectiveness of PPTB based on TD3 and RAD in a few MuJoCo and DMC environments.
10

---

Acknowledgements
This work is supported by the National Natural Science Foundation of China (Grant Nos. 92370132).
References
Jordan T. Ash and Ryan P. Adams. On warm-starting neural network training. NeurlPS, 2020.
A. Badia, B. Piot, S. Kapturowski, P. Sprechmann, A. Vitvitskyi, Z. Guo, and C. Blundell. Agent57: Outperforming the atari human benchmark. In ICML, volume 119, pages 507-517, 2020.
Marc G. Bellemare, Will Dabney, Robert Dadashi, Adrien Ali Taiga, Pablo Samuel Castro, Nicolas Le Roux, Dale Schuurmans, Tor Lattimore, and Clare Lyle. A geometric perspective on optimal representations for reinforcement learning. In NeurlPS, pages 1360-437) 2019.
G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and Zaremba. Openai gym. arXiv preprint, 1606v: 2016.
L. Chizat, Oyallon, and F. R. Bach. On lazy training differentiable programming. In NeurIPS, 2018.
W. Chung, S. Nath, A. Joseph, and M. White. Two-timescale networks for nonlinear value function approximation. In ICLR, 2019.
W. Dabney, A. Barreto, M. Rowland, R. Dadashi, J. Quan, M. G. Bellemare, and D. Silver. The value-improvemmet path: Towards better representations for reinforcement learning. AAAI, pages 7160-7168. 2021.
Degrave, Felici, J. Buchli, M. Neunert, B. D. Tracey, F. Carpanese, T. Ewalds, Hafner, A. Ab- dolmalcki, D. de Las Casas, C. Donner, L. Fritz, C. Galperti, A. Huber, J. Keeling, M. Tsimpoukelli, Kay, A. Merle, J. Moret, S. Noury, F. Pesamosca, D. Pfau, O. Sauter, C. Sommariva, S, Coda, B. Duval, A. Fasoli, Kohli, K. Kavukcuoglu, D. Hassabis, and M. A. Riedmiller. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897):4444119 2022.
A. Fawzi, M. Balog, A. Huang, T. Hubert, B. Nomera-Paredee M. Barekatain, A. Novikov, F, J. R. Ruiz, J. Schrittwieser, G. Swirszez, D. Silver, D. Hassabis, and P. Kohli. Discovering faster matrix multiplication algorithms with reinforcement learning. Nature, 610(7930):47-53, 2022.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: datasets for deep data-driven reinforcement learning. arXiv preprint, arXiv:2004.02119, 2020.
S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In /CML, volume 80, pages 1582-1591, 2018
D. Ghosh and M. G. Bellemare. Representations for stable off-policy reinforcement learning. In ICML, 2020.
Frithjof Gressmann, Zach Eaton-Rosen, and Carlo Luschi. Improving neural network training in low dimensional random bases. In NeurIPS, 2020.
Guy Gur-Ari, Daniel A. Roberts, and Ethan Dyer. Gradient descent happens in a tiny subspace. arXiv preprint, arXiv:1812.04755 2018.
Haamoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In ICML,, volume 80, pages 1856-1865, 2018.
A. Jacot, C. Hongler, and F. Gabriel. Neural tangent kernel: Convergence and generalization neural networks. In NewrIPS, pages 8580-8589, 2018
A. Kumar, R. Agarwal, D. Ghosh, and Levine. Implicit undae--aaaamtttriizatonn inhibits data- efficient deep reinforcement learning. In ICLR, 2021.

---

A. Kumar, R. Agarwal. T. Ma, A. Courville, G. Tucker, and S. Levine. DR3: value- deep reinforcement learning requires explicit regularization. ICLR, 2022.
M. Laskin, K. Lee, A. Stooke, L. Pinto, P. Abbeel, and A. Srinivas. Reinforcement learning with augmented data. In NeurlPS, 2020.
Tao Li, Lei Tan, Zhehao Huang, Qinghua Tao, Yipeng Liu, and Xiaolin Huang. Low dimensional trajectory hypothesis is true: Dnns can be trained in tiny subspaces. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3):3411 1-3420, 2023.
C. Lyle, M. Rowland, G. Ostrovski, and W. Dabney, On the effect of auxiliary tasks on representation dynamics. In AISTATS, volume 130, pages 1-9, 2021.
C. Lyle, M. Rowland, and W. Dabney. Understanding and preventing capacity loss in reinforcement learning. In ICLR, 2022.
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. A. Riedmiller, A. Fidjeland, G. Ostrovski. S. Petersen, C. Beattie, Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforce- ment learning. Nature, 518(7540):529
2015.
E. Nikishin, M. Schwarzer. P. D'Oro, P. Bacon, and A. C. Courville. The primacy bias in deep reinforcement learning. In ICML, volume 162 of Proceedings of Machine Learning Research, pages 16828-16847, 2022.
OpenAl. Chatgpt: Optimizing language models for dialogue, 2022. URL httph:ttp///. htttps//openati com/ hhtg//hhttt//
T. Schaul, A. Barreto, J. Quan, and G. Ostrovski. The phenomenon of policy churn. arXiv preprint, arXiv:2206.0003 00730, 2022.
Jan Schneider, Pierre Schumacher, Simon Guist, Le Chen, Daniel F. B. Häufle, Bernhard Schölkopf, and Dieter Büchler. Identifying policy gradient subspaces. arXiv preprint, arXiv:2401.66600 2024.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.0637777 2017.
D. Shah, B. Osinski, B. Ichter, and S. Levine. Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action. arXiv preprint, arXiv:2207.04429, 2022.
D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. A. Riedmiller. Deterministic policy gradient algorithms. In ICML, pages 387-395, 2014.
G. Sokar, R. Agarwal, P. Castro, and U. Evei. The dormant neuron phenomenon in deep reinforce- ment learning. arXiv preprint, arXiv:2302.12900 /2902, 2023.
R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. IEEE Transactions Neural Nenworks, 16:285-286, 1988.
H. Tang, Z. Meng. J. Hao, C. Chen, D. Graves, D. Li, C. Yu, H. Mao, W. Liu, Yang, W. Tao, and L. Wang. What about inputting policy in value function: Policy representation and policy-extended value function approximator. In AAAI, pages 8441 14149,444,4 2022a.
Hongyao Tang and Glen Berseth. Improving deep reinforcement learning by reducing the chain effect of value and policy churn. arXiv preprint, 20%1:22409 2024.
Y. Tang, Z. D. Guo, H. Richemond, Á. Pires, Chandak, R. Munos, M. Rowland, M. G. Azar, C. Le Lan, C. Lyle, A. György, S. Thakoor, W. Dabney, B. Piot, D. Calandriello, and M. Valko. Understanding self-predictive learning for reinforcement learning. arXiv preprint, arXiv:2212.03319, 2022b.
Y. Tassa, Y. Doron, A. Muldal, T. Erez, Li, D. de Las Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq, T. P. Lillicrap, and M. A. Riedmiller. Deepmind control suite. arXiv preprint, 1811v01801.00690, 2018.
12

---

Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double learning. AAAI, 2016.
Y. Yang, G. Zhang, Z. Xu, and D. Katabi. Harnessing structures for value-based planning and reinforcement learning. In ICLR, 2020.
Kenny Young and Tian Tian. Minatar: An atari-inspired testbed for more efficient reinforcement learning experiments. arXiv preprint, arXiv:1903.33176, 2019.
13

---

A Limitations
In this paper, we only provide empirical investigations the phenomena we present in Sec. Although we observe relatively consistent results for TD3 and RAD across a variety of MuloCo and DMC environments, which demonstrates the generality some degree, we have theoretical support for the observed phenomena at present. A rudimentary thought on this point is to study the learning dynamics of (TT)) especially of VoQo(s, a). This is because the policy path is the accumulation of policy gradient while policy gradients are significantly determined by the landscape of Q-network, which learns concurrently along the process. We believe that recent studies on the
learning dynamics of the value function and representation [Lyle et al., 2021 2022 can be inspiring
references.
Still from the empirical perspective, our work is not complete in the sense that we only consider TD3 and RAD (i.e., SAC inside). On-policy policy-based DRL algorithms like PPO (Schulman et al. [017]] may have a very different policy path. Besides, the empirical investigation for more value-based DRL algorithms beyond DQN [Mnih al. 2015] is missing in this paper. For typical value-based DRL algorithms, we can also treat the value network as a special form of policy and perform PPTB in the same way. Although we provided some preliminary results of applying PPTB to DoubleDQN in our experiments, this is kind of reckless since the natures of policy and value function are different, nor are their learning dynamics. A recent phenomenon called Policy Churn discovered in (Schaul et al. 2022 Tang and Berseth 2024] reveals that the greedy policy induced by a typical value network changes its actions on about 10% states after one update, We leave in-depth studies on the value function path in the future.
For methodology, propose very simple methods and we believe that there is great potential for more sophisticated methods to be proposed. First, we use standard SVD as the main tool in both our empirical investigation and our methods. We use no acceleration for SVD or other more advanced alternatives to obtain the major and minor directions of the policy path. Moreover, we use fixed dimensionalities and intervals for policy path trimming and boosting. This paper contains no attempts to design adaptive approaches or propose principled algorithms (which may rely on the advances in theoretical results). In addition, smarter hyperparameter selection methods are expected. One another limitation is that our proposed method is not evaluated in sparse- reward environments.
B
More Discussion
Policy Path Trimming in the View of Primacy Bias The policy path trimming (PPT) method proposed in this work can be viewed as a special way to do network resetting to alleviate the primacy bias [Nikishin et al. 2022]. The resetting effect of PPT in the minor update directions is achieved by (1) obtaining the transformed parameter space via performing temporal SVD with the historical policies collected in sliding window, (2) and then (re-)setting (or dropping) the left singular vector value (i.e.. u) to zero. More concretely, this differs from the resetting method in [Nikishin et al.. 2022 at two points:
Resetting to random initialization v.s., Resetting to Zero in the transformed parameter space: The resetting method in [Nikishin et al.) 2022) resets the network parameters to a set of randomly (re- jinitialized parameters. In contrast, value (i.e., u) to zero for the minor temporal SVD directions, i.e., resetting the parameters zero in the transformed parameter space. One thing to note is that resetting the left singular vector value (i.e., u) to zero does not mean that the network parameters (in the original parameter space) are necessarily zero, because the basis of space is different. A more in-depth analysis of the correlation between them is worth further study in the future.
method resets the left singular vector
Global resetting v.s., Local resetting: The resetting method in Nikiishn et al. 2022 is global, as the network parameters are reset to a set of randomly (re- innttaiite parameters. Differently, we do resetting according to the temporal SVD parameter space based on a sliding window that includes recent historical policies (thus being local). In this sense, this similar to Shrink-and- Perturb [Ash and Adams, 2020). which can be viewed as soft version of the resetting method, that shrinks the network parameters and adds random parameters with coefficient.
14

---

Moreover, from the perspective of plasticity, our method periodically rolls back the plasticity of the network in the directions that are orthogonal the ones that represent the effective knowledge learned so far.
The Connection between PPTB and Momentum- based Optimization The idea momentum adopted in modern optimizers is essentially make use of the information of historical gradients. In this sense, we can establish a connection between momentum and PPTB proposed in our work as the key idea of PPTB is to make use of the latent structure of the historical policies along the policy update path.
Momentum- based optimizers use different kinds of aggregation quantities of the historical gradients (e.g., M4+1 = 3m, + Vt). In our context, PPTB differs at two points: (1) not all the temporal SVD directions are used, and directions with large oscillations are trimmed (note that both SVD and cigendecomposition are techniques that capture the features of curvature); (2) policy path boosting can be viewed type of momentum conducted in the major temporal SVD directions. Another difference to notice is that, both our analysis and experimental evaluation are done with Adam which already leverages momentum (except for the analysis for SGD). Thus, findings based on the practical optimization of a momentum- based optimizer.
We believe that further study this connection with formal description is worthwhile in the future.
C Experiment Details
Hyperparameter Setting For the maintenance of the policy path, we use a FIFO buffer with the size of 2k and Ik for MuJoCo and DMC respectively, and save the policy parameters every 25 mini- batch gradient updates of the policy network. For the hyperparameter of PPTB, the number of major directions to keep (i.e., Y2 in Sec. 4.1 is searched in the set {8, 16, 32, 64) for each environment. For PPB, we simply always boost the first two SVD directions (i.e., 1,,1, Ua, as described in Sec. 4.2 and choose the boosting coefficient ps from the set {0.1, 1.0].
For the experiments Ataat we implemented DoubleDQN- PPTB based on the official code of by adding several lines of code to collect the historical network parameters of DoubIDQN
MinAtar 2 and perform PPTB intervals through learning. We made no change to the default hyperparameters of DoubleDQN. For DoubleDON-PPTR we r2 64 and pa 0.1.
Compute Resource We use Nvidia 3080ti GPU and V100 GPU for our experiments. For instance, each training trial of MuJoCo experiment requires less than 4G memory and about 8G GPU memory.
Note that the main computation cost added by PPTB is the calculation of SVD. In practice, we perform PPBT at sparse intervals (e.g.. tp 2 1000) for the policy parameter matrix with size [2000, = 1e5), and each SVD computation takes less than second with torch. torch. Linalg.sve The practical computation cost in wall-clock time is less than 5% of the total training time, which is worth it compared to the benefits SVD brings. Moreover, the temporal SVD operations remain scalable as matrix size increases, thanks the use of efficient linear algebra libraries.
PyTorch Code Implementation of PPTB We provide pytorch-like like pseudocode of PPTB imple- mentation in Algorithm|] | where the slices of vector and matrix are also in pytorch-like style. In our practical implementation used by our experiments, we only make the modifications of about 10-line core codes in the official implementations of TD3 [Fujimoto et al. 2018 and RAD Laskin et al.. 2020
Code implementation of Baseline Algorithms We build our experiment codes based on the official implementation GitHub for both TD3 (Fujimoto et al. 2018 and RAD [Laskin al. 2020 We do not change the default algorithm implementation or the recommended hyperparameters.
httpp://gtthhbbooo///ee////////e bttpee//ggccabb.cmm//tjjjuua/// http:://ggttabbb.cc////alaaaaaaa
15

---

Algorithm 1: DRL with PPTB (Psedudocode in a PyTorch-like style)
# env: environment
# agent:
policy-based DRL agent
d.p: policy parameter buffer with size k
Tb:
dimensionality hyperparams of PPTB where Yt 2 Tb (see Eq and
# lp intervals of storing policy and performing PPTB where lp % tg =
d_p lagent.policy, moliccyparamm
Initialize
for range (max_ inaa.iitttacti)osssepp))) # Typical agent-env interaction (omitted)
agent learn
1) Store policy parameters at intervals If % L, ==0
a_p.appen (agent. palint. policy.params, params)
d_p d_p[[kk:]
# 2) Perform Temporal SVD and PPTB
If % ip == 0
sgl, vh = SVD(d_p)
u_b (u[-1] [00]) ph u[ u_tb concat concat(u_ (u_b[:n.), u(-l, param_tb = (u_tb sgl(:n)) dot (vh([:))))
3) Apply modified params to agent agent. agent.policy load (params_tb)
D Additional Experimental Results
D.1 Empirical Investigation on Policy Parameter Change Amount
Further discussion about Phenomenon 1.1. One thing to note that the results reported in Fig. are based on the policy paths of entire learning process, thus being global view. For a more local view (i.e., a window of recent learning process), the phenomenon of parameter change asymmetry becomes more obvious and more parameters have minor and even no change (i.e., dead parameters). We hypothesize that this could be some practical evidence of lazy training of neural network (Chizat et al. 2018). Concretely, the phenomenon indicates that a lot of parameters have insignificant gradients VaQ(s, a),,a) Vono (s))==)(s) during learning process.
16

---

HalfCheetah-v4
Icapeet PCcy contenttr Parce sertent
3apparel fat ctaces and conmante sncuer inppanel aae aets posmanee srcourt
(b) Hopper-v4
(c) Walker2d-v4
ol thange
(d) Ant-v4
(e) Humanoid-v4
Figure 5: Complete results empirical investigation on policy parameter change amount MuloCo environments.
17

---

(a)
cartpole-swingup
(b) finger-spin
tay0
(C) walker-walk
Papprts in pene anccaa
Iapaeneeee vato dtate and someerr charge sowct whppeeet IE C'RCA Mitraa semmmntr www.aaa..ooooo.... Checuasporrt
theige
(d)
hopper-stand
(c) chectab-run
Figure 6: Complete results of empirical investigation on policy parameter change amount in DMC environments.
18

---

D.2 Empirical Investigation on Policy Learning Path by Temporal SVD
Lever2 Peroo:: Logan
Layen2 Terke
Layan2 Nerdd
Layers Avodl
Layer wead
(a) cartpole-swingup
pelcy
weee Jeent me aEC
Leard
Lowe1 Arred
Leyer, Teredl
Leyer3 Perod2
Love1 Nondd
(b) finger-spin
entrraado NATH s'eeea brevdry
wutto with wwweeatioon wwwdere
willer with interattee vynverin
Layer Nrtd1
Loye: 2rod2
Leven Perod3
Tnden sigguer vilum
kele diiggaier valee
Td o'anga oike
(c) walker- walk
Laver Felecl
Lever3 fetes2
Levec Perod3
Laperc Pobell
Lipact
(d) hopper-stand
Cheetafo SVD ilomatis of hemiing
cheessh-ouu SVD iffrmad polic bamsing
chearat cur 2wD.rromazon bariing
Aecue of Mormottn SOOPE
Layit Polic
Lsper2
Lnverz Peree:
(c) chectah-run
Figure 7: Complete results of the empirical investigation SVD Information of policy learning path in DMC environments.
19

---

(a) HalfChectah-vd
(b) Hopper-v4
Walker2d-v4
(d) Ant-v4
Figure Complete results of the empirical investigation SVD left unitary matrix of policy learning path in MuJoCo environments.
(a) cartpole-swingup
(b) walker walk
Figure 9: Complete results of the empirical investigation on SVD left unitary matrix policy learning path in DMC environments.
20

---

D.3
Empirical Investigation on Temporal SVD Reconstruction of DRL Policies
Table 4: Statistics for Temporal SVD reconstruction of RAD policies in cartpole-wingup (the first period) with varying number of major dimensions kept.
OF MAJOR DIM.
(VG((A
AVG((
MAX((AR(0))1.))
MIN( (0:)))
24.24 + 65
43.83
53.90
313.15
-85.48
26.68 = 61.70
37.45 55.83
278.79
-71.03
14.47 1: 50.29
34.78 39.10
168.46
-106.50
13.86 + 45.97
32.91
34.96
157.25
-94.24
15.06 + 38.30
27.69 30.45
125.26
-52.95
32
8.98 + 41.06
29.47 29.97
129.39
-72.51
64
2.15 41.69
28.19
30.78
125.08
-132.06
128
-2.04 1 45.76
27.95
36.30
120.36
-202.92
Table 5: Statistics for Temporal SVD reconstruction of RAD policies in cartpole-wingup (the second period) with varying number of major dimensions kept.
No.
AVG((A.(0))))
AVG([ AA (0.)1))
MAX((Ax(0.)3)
MIN([Ax(0.)J
18.10 1 41.86
21.01 40.47
265.96
-12.61
8.37 + 24.39
9.70 23.89
167.57
-13.24
4.71 21.28
10.07
19.33
129.82
-54.63
8
5.03 1: 24.63
8.28 23.74
164.32
-53.45
16
4.42 18.25
6.96 17.44
123.94
-27.75
32
5.48 + 35.52
10.54 = 34.36
238.01
-77.67
64
3.41 +: 18.15
7.35
16.94
116.21
-32.73
128
3.02 11,10
5.51
10.09
65.10
-31.63
Table 6: Statistics for Temporal SVD reconstruction of RAD policies in finger-spin (the first period) with varying number of major dimensions kept.
No.
AVG((
AVG([[An(0)])b))
MAX((Ax(0.)))
(IN()A((O)).))
-61.70 94.46
62.70 93.80
10.00
-393.67
2.28 + 13.87
7.48 11.91
70.00
-15.67
0.77 18.79
16.54
51.33
-106.33
0.09 + 12.15
6.49 10.27
28.33
-62.67
16
-5.06 1. 25.28
9.07 24.13
24.33
-158.00
32
-0.83 7.45
5.77 4.80
20.00
-20.67
64
-0.24 9.50
5.15 7.99
18.67
-52.00
1.55
10.02
5.91 8.24
52.33
-22.33
21

---

Table 7: Statistics for Temporal SVD reconstruction of RAD policies in finger-spin (the second period) with varying number major dimensions kept.
No. OF MAJOR DIM.
AVG({Ax(0)))))
AVG([[An(0.])))
MAX((Ax(0.))))
MIN([Ax(0.)))
-10.68 26.53
19.89 = 20.54
45.33
-90.00
-5.94 30.71
11.67
26.67
-202.00
-0.95 + 10.37
6.05 8.48
23.00
-51.00
0.03 10.39
6.04 8.46
26.00
-54.33
16
-0.81 1 10.20
5.81 8.42
27.33
-47.00
32
-0.47 + 10.63
5.99 8.80
28.33
-53.67
-4.90 + 29.45
9.85 28.
24.67
-197.00
128
-16.57 69.42
22.17 67.84
23.00
-407.67
Table 8: Statistics for Temporal SVD reconstruction of RAD policies walker-walk (the first period) with varying number of major dimensions kept.
No.
Av))(Ax(0:)))
AVG([AAa())])b)
MAX((Ax((.)).)
(IN({A2(0:)3))
7.20 1: 116.51
84.63
80.40
285.06
-298.17 17
61.12 + 82.60
72.36 72.96
347.76
-58.92
41.79 = 86.74
67.88 68.29
283.30
-211.50
8
32.80 1 77.18
55.46 62.91
261.98
-144.36
16
25.53 + 82.42
62.40 59.59
233.16
-177.25
32
24.22 1 86.47
61.09 65.81
255.32
-262.19
64
20.31 + 77.88
51.96 61.47
298.37
-225.65
12.57 + 74.66
56.62 50.27
240.83
-156.34
Table 9: Statistics for Temporal SVD reconstruction of RAD policies in walker-walk (the second period) with varying number of major dimensions kept.
No. OF MAJOR DIM.
AVG([Ax(0.)))
AVG([A{{()]}a))
MAX((Ax((.))))
MIN({Ax(0:))))
15.03 = 52.61
38.35 39.02
155.56
-149.09
21.23 1: 37.68
30.37 30.80
145.45
-37.95
4
21.81 + 34.98
28.14 30.
113.82
-25.32
14.12 = 44.51
31.12 34.81
134.84
-163.47
16
6.16 +: 46.68
32.26 34.30
100.79
-169.74
32
1.55 62.75
35.26 $1.93
101.41
-266.15
64
11.91 + 38.39
28.99 = 27.85
125.69
-75.02
128
14.11 +: 39.72
28.39 31.15
124.47
-116.62
22

---

(a)
Ant-y4
(b)
Walker2d-v4
(c) Hopper-v4
(d) HalsCheetah-v4
Figure 10: The learning curves for TD3 (Gray) and TD3-PPTB (Red) in four MuJoCo tasks, with means and standard errors across 6 seeds.
702
003
200
300
300
109
Figure 11: The learning curves for RAD (Gray) and RAD-PPTB (Red) in four DMC tasks, with means and standard errors across 6 seeds.
(a) Parameter Analysis of BC (Adam)
BALLOR
430 36c1a.13
(b) Temporal SVD Analysis of BC (Adam)
(c) Parameter SVD Analysis of BC (SGD)
(d) Temporal SVD Analysis of BC (SGD)
Figure 12:
The parameter and Temporal SVD analysis for Behavior Cloning with Adam and SGD optimizers in D4RL hulfcheetaf-medimm-eee the asymmetry of parameter update amount is less observed in BC (Adam) but more severe in BC (SGD); the pattern of temporal SVD evolvement is different for BC (Adam) and BC (SGD).
Compared with the RL cases reported in our paper,
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims in the abstract and introduction summarize contributions and are supported by our experiment results Section and Section 5
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Section A
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide sufficient experiment and implementation details in Section 5 and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
25

---

Answer: [Yes]
Justification:
Our code is available in
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c
assionolicyyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ //ippe.cc/pullic/guuii wwwti//gguie///eeeeceoocc CoodeSubiiiie ssi wsionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide necessary experimental details in Section 5 and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The results in our paper are accompanied by error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
26

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the details of compute resources in Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpss/r//ouui...cooo/ol////tttt
Answer: [Yes]
Justification:
This work conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper study policy network evolvement in a low-dimensional space, which can have positive impact general for better understanding the learning dynamics of DRL algorithms. No specific real-world application is concerned and we do not see specific societal impact of this paper.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
27

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All the environments, data and codes of baseline methods used this paper are publicly available Github. We cited the original papers and provided the URLs the assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
28

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide the necessary details for implementing our proposed method in this paper. We will provide a README document alongside the code release after the review process.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
29

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "0cBTg6oaD9", "reviewer": "Reviewer_zpVk", "summary": "The authors study how parameters evolve in Deep RL. They perform SVD on updates and find that parameters advance along a small number of directions. They then propose a method to trim the policy learning path by focusing the updates on these major directions. They show that their methods improve performance in MuJoCo and DMC.", "questions": "1. Do you think this would work equally well in sparse(r) reward settings? My intuition is that it would not, since the early parameter updates likely do not contain significant information about the reward. The continuous control tasks evaluated are *particularly dense*. Also, doesn't pruning the noisier directions harm exploration?\n\n2. Related to the above, why do you believe this method does not exacerbate issues of primacy bias mentioned in the paper? The related work section's first paragraph does not really compare and contrast to the prior works, just mentions them.\n\n3. Do you think these results apply beyond just RL? I see no reason why this phenomenon is RL-specific. \n\n4. Related to Q3: Doesn't the fact that these methods use momentum (e.g. Adam) make this phenomenon obviously true? Does this phenomenon persist when using plain SGD? What about when you observe the *gradient updates* as opposed to the *parameter updates*?\n\n5. Can you include the plots mentioned above? It would really help with my understanding of the paper.", "limitations": "The authors address the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The work is a well-written exploration of an interesting and novel perspective on RL parameter updates. The authors are extremely thorough and clear with their investigation and show strong results on common benchmarks. Furthermore, their method seems very simple to implement, which is valuable to the community.", "weaknesses": "Concerns with the paper:\n\n1. The paper only investigates extremely dense reward settings. Intuitively, the conclusions of this paper should not apply to sparse(r) reward settings, which are arguably much more interesting in RL. (See Q1 below).\n\n2. The results of the paper do not seem like they would be at all specific to RL. I'm not convinced that the observed phenomenon is not just a simple byproduct of using an optimizer with momentum. It would be good to have results that show that this phenomenon does not occur or help in supervised learning tasks (where I would imagine there is significantly more literature on this topic), and/or that this phenomenon still occurs when studying agents trained with plain SGD.\n\n3. The significance of the results are unclear (see below) \n\nPossible improvements:\n\n1. It could be neat to replicate the empirical investigations when using your new method PPTB. Does PPTB actually address the issues presented and dampen parameter updates? \n\n2. The results are not easy to read. It is hard to tell which results are statistically significant (I would recommend using standard error!) and also plotting standard RL training curves with the appropriate error regions. As-is, it's very hard to tell whether this method actually helps or not. In particular, it seems as though the error regions often overlap in Table 2, for example.\n\n3. The writing is often vague. In the abstract, the authors write: \"we study how the policy networks of typical DRL agents evolve during the learning process by empirically investigating several kinds of temporal change for each policy parameter\". This vague sentence conveys little information. The authors also use the word \"asymmetry\" in the abstract without explaining what they mean (asymmetric with respect to what?).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: “Do you think these results apply beyond just RL? I see no reason why this phenomenon is RL-specific. Doesn't the fact that these methods use momentum (e.g. Adam) make this phenomenon obviously true? Does this phenomenon persist when using plain SGD? What about when you observe the gradient updates as opposed to the parameter updates?”\n\nWe appreciate the reviewer for pointing out this. We agree that it will be interesting and valuable to extend our experimental investigation to more general deep learning settings.\n\nTo this end, we provide additional results in Figure 11 of the one-page pdf to present additional investigation for **Behavior Cloning** in D4RL halfcheetah-medium-replay **with Adam and SGD optimizers**.\n\nCompared with the RL cases reported in our paper, the parameter update amount is more Gaussian-disribution like for BC (Adam). This indicates the difference in policy parameter update dynamics between BC (i.e., supervised learning) and RL.\nSomewhat surprisingly, the asymmetry of parameter update amount turns out to be more severe in BC (SGD). This is a bit counterintuitive because Adam exhibits less asymmetry than SGD. \n\nMoreover, we found that BC (Adam), BC (SGD) and the RL cases reported in our paper show different patterns in terms of temporal SVD evolvement. \n\nIn this paper, we start from and focus more on online RL and we defer the study across different learning paradigms in the future. \n\n\n> Q2: “why do you believe this method does not exacerbate issues of primacy bias mentioned in the paper?”\n\nIn fact, the policy path trimming (PPT) method proposed in our paper can be viewed as a special way to do network resetting to alleviate the primacy bias as in [Nikishin et al., 2022].\n\nConcretely, our method differs from the vanilla network resetting method in [Nikishin et al., 2022] on two points: (1) we do reset in a transformed subspace (with temporal SVD) rather than the original space; (2) we only reset the parameter update in the minor directions and maintain the learned information in the major directions.\n\nFrom the perspective of plasticity, **our method periodically rolls back the plasticity of the network in the directions that are orthogonal to the ones that represent the effective knowledge learned so far**.\n\nWe appreciate the reviewer’s insightful comments and we believe that more study can be done in the future by taking into consideration both policy update subspace and the plasticity loss problem.\n\n\n> Q3: “Do you think this would work equally well in sparse(r) reward settings? My intuition is that it would not, since the early parameter updates likely do not contain significant information about the reward. The continuous control tasks evaluated are particularly dense. Also, doesn't pruning the noisier directions harm exploration?”\n\nWe appreciate the reviewer for pointing out this inspiring point. First, we consider that the asymmetry in terms of parameter update amount and the concentration of temporal SVD information should be more severe in sparse-reward settings, as the self-distillation mechanism can dominate when the reward signal contains little information [1].\n\nSecond, our method does not aim to address the exploration problem and thus it is less likely to improve the learning performance especially when the baseline algorithm fails with very sparse rewards. \n\nHowever, we do not think that our method necessarily harms exploration, because the trimming happens at a very sparse interval and moreover the effect of the parameter update of the minor directions in exploration is unclear yet. One possible understanding is that the periodic trimming rolls back the plasticity (as discussed in Q3) and could encourage learning new behaviors with the parameters in these minor directions. \n\nWe are also running additional experiments in several sparse-reward DMC tasks. We will present the results for this if we are able to finish this before the discussion stage ends.\n\n\n\n---\n\nReference:\n\n[1] DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularization. 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eMchcAPipT", "reviewer": "Reviewer_dZpR", "summary": "The authors examine the trajectories of policy learning in continuous control reinforcement learning tasks. \n\nThey begin by measuring how directly parameters go to their destination and observe large detours and differing update behaviour for different layers. They then examine the singular value decomposition of the different training checkpoints and observe some strong common update trends. \n\nThey propose Policy Path Trimming and Boosting (PPTB), where policy boosting is boosting the gradients towards the strongest singular values, and trimming trims the updates from the smaller singular values.", "questions": "* Can you comment more on the relationship between your work and other, less compute intensive methods, that aim to allow more uniform convergence among the different data eigenvalues such as momentum? Is there an explicit connection here? Have you thought about that?", "limitations": "* The authors, to their credit, provide an extensive discussion of the limitations of their work in Appendix A. I agree largely with the points in that section and enjoyed their contextualisation of their work there.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The paper describes their method and analysis well. I generally found it easy to read and understand what points they were trying to convey.\n* The proposed method does seem to improve performance on the tasks presented.", "weaknesses": "* The environments and methods investigated, given the authors resutls are purely empirical, are not diverse enough to draw any real conclusions. The authors focus on continuous control environments without considering discrete environments such as Atari, or more diverse network architectures such as  recurrent networks or transformers, or different methods such as PPO or DQN. These results hold only for MLPs or CNNs trained on continuous control tasks, which just isn't a convincing enough setting to warrant acceptance.\n* The proposed method does not seem particularly practical. Computing SVD is both time intensive and requires storing a wide range of previous parameters. It therefore requires a lot more compute, which is why such methods are typically not used. Compared to approximations such as momentum-based optimisation [1], which are also aimed at achieving more uniform convergence, this time among eigenvalues of the data matrix. Although their method improves performance, they have not convincingly demonstrated that the compute couldn't be better used elsewhere, for example by training a bigger network or by sweeping hyperparameters more effectively.\n\n\n\n[1] Goh, \"Why Momentum Really Works\", Distill, 2017. http://doi.org/10.23915/distill.00006", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: “Computing SVD is both time intensive and requires storing a wide range of previous parameters. It therefore requires a lot more compute, which is why such methods are typically not used”\n\nWe apologize for causing the reviewers to misunderstand the memory consumption and computational overhead of our method. \n\nWe would like to further clarify this point in the following:\n\n**[Computation cost]** We would like to clarify that (1) **each SVD operation in our experiments takes roughly 1 second** by using **torch.linalg.svd**, and (2) the SVD operation is **not performed at every time step but rather at sparse intervals (e.g., $t_p \\ge 1000$)**. The practical computation cost in wall-clock time is **less than 5%** of the total training time, which is worth it compared to the benefits SVD brings. \n\nMoreover, the temporal SVD operations remain scalable as matrix size increases, thanks to the use of efficient linear algebra libraries. This scalability has been validated by numerous successful cases, such as the widely adopted large model fine-tuning technique, LoRA (Low-Rank Adaptation) [1]. The effective application of LoRA clearly demonstrates that SVD is not a significant issue in terms of memory overhead and computational cost.\n\n**[Memory cost]** As described in Appendix B, PPBT does not require storing all historical policies but instead stores policies at sparse intervals. In our implementation, the size of the stored policy parameter matrix is $[2000, \\approx 1e6]$, making the memory overhead entirely acceptable.\n\nWe have clarified this point in our draft.\n\n\n\n\n\n\n> Q2: Can you comment more on the relationship between your work and other, less compute intensive methods, that aim to allow more uniform convergence among the different data eigenvalues such as momentum? Is there an explicit connection here? Have you thought about that?\n\n\nIn Q1, We provide the response to the misunderstanding on the “compute-intensive” point.\n\nWe are more than willing to discuss the relationship between our work and any concrete related work provided by the reviewer in the discussion stage.\n\n\n---\n\nReference:\n\n[1] LoRA: Low-Rank Adaptation of Large Language Models. 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2lG2eMp5pB", "reviewer": "Reviewer_iaGj", "summary": "This paper investigates the evolution of parameters over time during policy optimization with TD3 and RAD. By analysing the SVD of a matrix containing parameters over time, the authors find that there are a few directions in which the parameters move consistently and many with more oscillations. Using this insight, an algorithm is proposed which boosts updates in the top estimated directions and reduces updates in the uninmportant ones. Evaluations on standard deep RL benchmark tasks demonstrate the utility of the approach.", "questions": "_Clarification questions_\n\n\n- Line 157: What is $\\alpha_k$? Is it Singular Value Information Amount? \n\n\n- Fig 2 b) 4th and 5th figure from left. Interesting findings. So basically, the detour ratio is smaller for the \nAgain, I wonder if that has more to do with noise or curvature.\nPerhaps there is less noise in these directions, \n\n\t- In the 3rd fig from the left, it's hard to see much since the black curve masks everything else. Consider using different colors or some transparency. It would be nice to see how the paths get increasingly noisy.\n\n- Policy Path Trimming: To clarify the algorithm, does it project the current parameters into the space identified by the top singular vectors? \n\tThe description of the algorithm could be improved in Sec. 4.1. where it is introduced. While the intuition is described, the exact mechanism that is implemented is not explained.\n\n- Similarly, in section 4.2, the Policy Path Boosting could be described a bit more clearly. Eq. 2 in particular is a bit confusing since  \n$\\hat{u}_{n,*}$ is updated but then only \n\n$\\hat{u}_{n,i}$ $(i=1,2)$ are actually used. \n\nThe phrase in line 241 \"PPB modifies $\\theta_n$ by increasing $u_{n,1}, u_{n,2}$ along the temporal direction...\" sounds  overly complicated. Perhaps rephrasing it to something like \"PPB moves the parameter further in the direction of previous updates along the first two main directions $u_{n,1}$ and $u_{n,2}$\" could be simpler. \n\n_Suggestions and broader questions_\n- By constraining the parameter evolution path to focus on the previous main directions, would the effect of \"primacy bias\" or related phenomena be even stronger? Could we be losing out on performance due to prematurely committing to certain update directions? \n\n- Line 270. The evaluation metric \"SCORE\" should no longer be used since the maximum over runs introduces overestimation bias and leads to less reliable estimatse. See [1] for better evaluation practices and [2] for arguments against using the max. \n\n- I wonder if the detours are mainly due to noise or curvature of the objective. One way to test this would be to increase or decrease the minibatch size, which can control the variance of the updates. Then, by inspecting the effect on the detour ratio or cumulative parameter movement, we could guess the relative impact of noise and curvature. \n\n- Fig.1 is a CDF plot. I think a histogram or a box plot might be easier to interpret than CDF plots since you need to look at differences in a CDF plot to identify where most of the probability mass is.\n\n- An ablation study for the two components of the algorithm would be a valuable addition. It is not clear if both pieces are necessary right now or how important they are.\n\n- I would also be curious to know what would happen if we took the estimated space \n\n- Here is a paper that could be interested to read [3] where the authors show that, even if you constrain neural network parameters to a random subspace, as long as the dimension of that subspace is not too small, you can recover the same performance as the original network. \nThe idea of Policy Path Trimming could be interpreted as a more intelligent approach which estimates the constrained subspace instead of choosing a random one. \n\n- Another line of research (e.g. see [4]) has observed that the Hessian contains only a few large eigenvalues in neural network training, which may be related to the ideas discussed in the paper since there would be some interplay between the curvature of the loss surface and directions of updates over time. \n\n- In the matrix of parameters over time, consider using $t \\in {1,...,T}$ to index the rows instead of $n$ so it's a little easier for the reader to remember which dimension corresponds to what. Alternatively, if $t$ is reserved for environment timesptes, $\\tau$ could be used as a substitute. \n\n\n\n[1] \"Deep Reinforcement Learning at the Edge of the Statistical Precipice\" Agarwal et al.\n\n[2] \"Deep Reinforcement Learning that Matters\" Henderson et al. \n\n[3] \"Measuring the Intrinsic Dimension of Objective Landscapes\" Li et al. \n\n[4] \"An Investigation into Neural Net Optimization via Hessian Eigenvalue Density\" Ghorbani et al.", "limitations": "These are discussed.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "This paper uncovers some nice insights into the training process of deep RL agents. \nAs far as I know, I have not seen the SVD used to study parameter evolution over time, and I find it is an interesting application of that tool. The proposed algorithm is conceptually simple and easy to implement, making it easy to add to a variety of existing algorithms.\nIn terms of impact, the identified phenomena may extend past RL and could be true of deep neural network training more generally, potentially giving widespread impact. \n\nThe paper is well-organized, with the different sections flowing nicely into each other. Generally, the paper was easy to follow and experiments chosen appropriately to make the intended arugments.", "weaknesses": "My main concern is about the evaluation of the algorithms. \nFor example, reporting the max return in the evaluation of algorithm or using the standard deviation across runs, whereas the standard error or boostrapped confidence intervals would be more appropriate. (see Questions)\n\nAlso, the following design choice is confusing to me:\n- Fig. 1 caption. \"Only\tupper 80% values according to $\\nabla^{apc}_j$ are taken to plot...for meaningful analysis\" \n\tCould you elaborate why this decision was made? Does it have to do with many of the values being close to zero?\n\nThe improvement in the performance is generally fairly modest. I think this is fine given that main contribution to me is identifying behaviours in the parameter evolution. I think expanding a bit more on the analysis could be interesting. \n\nIn terms of clarity, some details could be expanded upon more in the main text. See Questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: the evaluation of the algorithms. For example, reporting the max return in the evaluation of algorithm or using the standard deviation across runs, whereas the standard error or boostrapped confidence intervals would be more appropriate\n\nTechnically, we are not reporting the “max return” in our evaluation. We report the **“maximum of the mean return”**. To be concrete, this is obtained by first (1) computing the curve of mean return evaluation across multiple runs, and then (2) taking the maximum of the mean evaluation curve. \n\nWe **follow the evaluation scheme presented in TD3 paper** (see “Max Average Return over 10 trials” in the caption of Table 1 in TD3 paper). \n\nThe rationality of this evaluation scheme is that, for online learning, it is possible to store checkpoints throughout learning. For the opposite case, in offline RL, the convention is to report the mean final score.\n\nIn Line 279, we wrote, “We report the means and standard deviation errors across six independent trials”. We meant to say “standard errors” rather than “standard deviation errors”. This is a mistake in writing. We have amended it in our revision.\n\n\nWe appreciate the reviewer for the suggestion of using other more reliable evaluation metrics. We will take the suggestion in our revision.\n\n\n\n> Q2:  “I think expanding a bit more on the analysis could be interesting”\n\nAs suggested by Reviewer zpVk, we provide additional results in Figure 11 of the one-page pdf to present additional investigation for **Behavior Cloning** in D4RL halfcheetah-medium-replay **with Adam and SGD optimizers**.\n\nCompared with the RL cases reported in our paper, the parameter update amount is more Gaussian-disribution like for BC (Adam). This indicates the difference in policy parameter update dynamics between BC (i.e., supervised learning) and RL.\nSomewhat surprisingly, the asymmetry of parameter update amount turns out to be more severe in BC (SGD). This is a bit counterintuitive because Adam exhibits less asymmetry than SGD. \n\nMoreover, we found that BC (Adam), BC (SGD) and the RL cases reported in our paper show different patterns in terms of temporal SVD evolvement.\n\nWe believe that these results are interesting to potential audiences of this paper and worth a more in-depth discussion.\n\n\n> Q3: “By constraining the parameter evolution path to focus on the previous main directions, would the effect of primacy bias or related phenomena be even stronger? Could we be losing out on performance due to prematurely committing to certain update directions?”\n\nIn fact, the policy path trimming (PPT) method proposed in our paper can be viewed as a special way to do network resetting to alleviate the primacy bias as in [Nikishin et al., 2022].\n\nConcretely, our method differs from the vanilla network resetting method in [Nikishin et al., 2022] on two points: (1) we do reset in a transformed subspace (with temporal SVD) rather than the original space; (2) we only reset the parameter update in the minor directions and maintain the learned information in the major directions.\n\nFrom the perspective of plasticity, **our method periodically rolls back the plasticity of the network in the directions that are orthogonal to the ones that represent the effective knowledge learned so far**.\n\nWe appreciate the reviewer’s insightful comments and we believe that more study can be done in the future by taking into consideration both policy update subspace and the plasticity loss problem.\n\n\n> Q4: \"Only upper 80% values according to $\\Delta_{j}^{\\text{apc}}$ are taken to plot...for meaningful analysis\" Could you elaborate why this decision was made? Does it have to do with many of the values being close to zero?\n\nThis is because many parameters have a very minor value (i.e., very close to zero) of accumulated parameter change (as revealed by the parameter update asymmetry phenomenon).\n\nWe have clarified this in our draft to eliminate the confusion.\n\n\n> Q5: The clarification questions\n\n- [Line 157: What is $\\alpha_k$? Is it Singular Value Information Amount?] $\\alpha$ is the Singular Value Information Amount for a dimensionality number $k$, which is **defined in Line 155**.\n\n- [Fig 2 b) 4th and 5th figure from left. Interesting findings. So basically, the detour ratio is smaller for the Again, I wonder if that has more to do with noise or curvature. Perhaps there is less noise in these directions] Yes, the detour ratio is smaller for the major directions. We consider that it is closely related to the curvature of the landscape of policy objective function.\n\n- [In the 3rd fig from the left, it's hard to see much since the black curve masks everything else. Consider using different colors or some transparency. It would be nice to see how the paths get increasingly noisy.] We appreciate the reviewer’s suggestion. We will replace the plot with the frequency information by Fourier transformation for a better presentation.\n\n- [More description for Policy Path Trimming and Policy Path Boosting] The conversion from the original parameter space to the subspace is done by performing temporal SVD. The corresponding left singular vectors are taken as the new coordinates in the subspace. This is described between Line 145-152. We have taken the writing suggestions provided by the reviewer and added more detailed descriptions in Section 4.1 and 4.2 to eliminate the confusion.\n\n\n> Q6: Other suggestions\n\nWe sincerely appreciate the reviewer for providing insightful comments along with very useful reference papers. \n\nDue to the time limit of the rebuttal stage, we may not be able to respond to each point. However, we are more than willing to discuss more in the discussion stage.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r2s6HrID68", "reviewer": "Reviewer_8fBA", "summary": "Off-policy actor-critic Deep RL has unstable and seemingly oscillatory learning dynamics, which are poorly understood. This paper looks closely at the trajectories taken by policy networks during training. An SVD analysis, performed over sequences of policy parameter snapshots, reveals near-monotonic parameter evolution along directions corresponding to the few dominant singular values, and--- indeed---oscillatory behaviour along the minor directions.\nThe authors propose an intuitive and mathematically sound way to remedy this pathology by only permitting the parameters to move in subspace spanned by a small number of the top directions, while retaining the policy performance. Additionally, by taking larger steps in the first two major directions, they are able to train more performant policies.", "questions": "- Line `434`: you say that you choose the boosting coefficient $p_b$ from a set of values. Do you mean that you search from this set for a good hyperpameter value, or do you randomly sample from this set at each training iteration?", "limitations": "The limitations section is well written, but one thing that isn't mentioned is that the method can struggle to scale up to larger network sizes (beyond the smallish networks used in this paper).\n\nOverall, I find this paper exciting, and I'm willing to substantially improve my rating if the authors address all the listed weaknesses (most crucially, number `3`).", "rating": 4, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "**Significance**\n\nThe paper addresses an important problem. Deep RL researchers have long observed poor training dynamics in off-policy learning. These include divergence, performance collapses, and oscillatory behaviour (i.e. policies forgetting what was learnt, then recovering). The reasons behind these are poorly understood. While the paper does not quite advance our understanding of the problem, it does potentially identify some mechanistic signatures of the problem (oscillatory weight dynamics) and proposes a treatment for this symptom (suppressing movement in oscillation-afflicted subspaces). I think the proposed PPTB fix is unlikely to be a complete solution, mainly because the critic's behaviour is not yet studied, but it can be a stepping stone towards better behaved learning dynamics. \n\n**Clarity**\n\nThe paper is very clearly written, and I enjoyed reading it. Policy churn and oscillatory dynamics potentially stem from the combined actor-critic training dynamics, but the authors focus exclusively on the policy here. This is good, as it has kept the study focused and revealed interesting phenomena. \n\n**Originality**\n\nI am unaware of prior work which empirically inspects the policy learning dynamics by actually plotting out the weights' evolution, as the authors have done.\n\n**Quality**\n\nThe idea of using temporal SVD upon the policy weights is quite sensible, and a treatment for the problem immediately pops out of the same tool; this fix also appears to be easily implementable in code, and seems to grant substantial performance boosts.", "weaknesses": "1. **Limited analysis.** The analysis is focused on TD3 and RAD (based on SAC here, which itself is quite close to TD3). I think this is insufficient. Seasoned RL practitioners will note that DDPG exhibits markedly more policy oscillations/collapses/recoveries than TD3: the introduction of the Clipped Double Q trick already substantially mitigates those effects. In this empirical study, I worry that by not including DDPG --- the simplest possible baseline --- the authors are not observing the problematic dynamics in their full glory, and perhaps not testing the full potential of PPTB. Right now, there is a risk that your observed phenomena are quirks of TD3-lineage methods.\n\n2. **Related work.** The observation that gradient descent moves the parameters along a few dominant directions, chiefly in a low-dimensional subspace, is not new. There is a body of deep learning literature around this phenomenon, which the authors don't cite right now. Here is one such paper, and there are more:\n - \"Gradient descent happens in a tiny subspace\" G Gur-Ari, DA Roberts, E Dyer \n\n3. **Potentially limited significance.** Following from the above point: the existence of dominant directions is unsurprising. The existence of oscillatory detours in the minor directions *could* be a novel finding regarding actor-critic RL, but we don't know that yet. To strengthen the paper, here's a test: do you also observe the same harmonics in basic supervised learning? (e.g. when training the actor network with an MSE behaviour cloning loss). I think this is a particularly important thing to investigate, because I'm concerned that it could change the story of your paper.\n\n4. **Lack of training curve comparisons.** For a method that expressly tries to curb oscillations, it is important that we see how the actual training curves look, instead of just a table with scores.\n\n5. **Limited evaluations.** 6 is not enough; it's common to report a minimum of 10 independent trials. Also: you have `Humanoid-v4` experiments in the paper, but don't report scores with PPTB on that env.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jbGT4eGUTD", "reviewer": "Reviewer_HTJS", "summary": "This paper investigates the evolving path of policy network parameters in deep reinforcement learning (DRL). The author conducts experiments on multiple tasks in MuJoCo using TD3 and on multiple tasks in DMC using RAD. The findings reveal significant discrepancies in the amount of change among policy parameters and severe detours in policy parameter updates. To address this, the author employs Temporal SVD to decompose the evolving path of policy parameters. Despite the large number of parameters, the learning dynamics of the policy network are found to be concentrated in a few primary directions, forming a low-dimensional space.\n\nBased on these insights, the author proposed a new method called Policy Path Trimming and Boosting (PPTB). This method improves the performance of DRL algorithms by canceling updates in secondary parameter directions and boosting progress in primary directions. Experiments conducted on TD3 and RAD agents in MuJoCo and DMC environments demonstrate that the PPTB method significantly outperformed the original methods in terms of score and AUC evaluation metrics, thereby substantially enhancing the performance of the DRL algorithm.\n\nThe author investigates the dynamics of policy network parameters from a novel perspective of Temporal SVD, and proposes an innovative method PPTB to enhance the performance of the DRL algorithm. The writing style of this paper is clear and easy to understand, with rigorous logic, and it is of great significance to algorithm optimization in the field of DRL.", "questions": "1. Wouldn't other DRL algorithms have similar phenomena when updating strategies?\n\n2. Does PPTB work with DRL methods other than TD3 and RAD?\n\n3. Can the performance of PPTB be tested in a deeper network?\n\n4. Can the performance of PPTB algorithm be tested in more experimental environments or different tasks? Are there more evaluation metrics besides score and AUC?", "limitations": "The author should enrich the experimental details and experimental content to demonstrate the versatility of the PPTB algorithm, or test the performance of the algorithm in more practical tasks. The article should appropriately discuss the limitations of the proposed algorithm, or provide some prospects for future improvements.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The author investigates the dynamics of policy network parameters from a novel perspective of Temporal SVD, and proposes an innovative method PPTB to enhance the performance of the DRL algorithm. The writing style of this paper is clear and easy to understand, with rigorous logic, and it is of great significance to algorithm optimization in the field of DRL.", "weaknesses": "The article does not clearly explain why only TD3 and RAD algorithms were selected. Wouldn't other algorithms have similar phenomena when updating strategies?\n\nThe PPTB experiment was only conducted on TD3 and RAD, and did not reveal the universality of PPTB on other DRL algorithms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: Why TD3 and RAD are selected?\n\nThe motivation for this work starts from the investigation of the dynamics of policy network parameters. Thus, we choose TD3 and RAD for the following reasons:\n- They are popular deep AC methods (note RAD is SAC-based), where an explicit policy network is trained. This ruled out the value-based RL methods.\n- TD3 and SAC are off-policy methods. They are usually more sample-efficient while less stable than on-policy algorithms like PPO.\n- RAD takes visual observations as input. Thus, our choices cover both proprioceptive inputs and image-based inputs.\n\n> Q2: Wouldn't other algorithms have similar phenomena when updating strategies?\n\n\nAs suggested by Reviewer zpVk, in Figure 11 of the one-page pdf, we can observe similar phenomena for Behavior Cloning with Adam or SGD optimizers, but with different concrete patterns.\n\n\nBesides, we observed similar phenomena for DDPG (omitted due to space limitation).\n\n\n> Q3: Other suggestions on the experiment\n\nWe appreciate the reviewer’s valuable suggestions on the experiment.\n\nWe will present the results for this if we are able to finish this before the discussion stage ends.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5NHmbvqYlW", "reviewer": "Reviewer_B4U6", "summary": "This paper first introduce some common phenomena for TD3 and RAD agents, and then introduce a novel deep reinforcement learning algorithm which perform a novel temporal SVD along the policy learning path called Policy Path Trimming and Boosting (PPTB). This algorithm offers us an angle to view how policy evolves in a lower-dimensional space and how to utilizes it in reinforcement learning part. The paper is well written and novel.", "questions": "1.Could the authors provide the learning curves of TD3 and TD3-PPTB? From my understanding, TD3-PPTB should only accelerate the training, I don't think TD3-PPTB would have better performance than TD3. I believe they should achieve the same converged result eventually.\n\n2.The author only provided results for 100k and 500k steps in the RAD results shown in Table 2. What about the results for 1000k or 2000k steps?\n\n3.Is the parameter P_b  fixed in each environment? Can you talk about how to determine the P_b in your experiment? If is not set properly, will it cause excessive oscillation during the training process? Additionally, in the experiment details, a 2000 or 1e5-dimensional neural network is too large for Mujoco training. Most papers indicate that commonly used model dimensions are 64 or 128.", "limitations": "1. This paper does not explain from a theoretical perspective why this approach would accelerate convergence.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.The paper is well-written.\n\n2.The author methodically analyzes the experimental results using specific indicators (Accumulated Parameter Change, Final Parameter Change, Parameter Update Detour Ratio), making the paper very comprehensible.", "weaknesses": "1.Although most of the figures in the paper are very clear, some are unclear and hard to understand, particularly the last two figures in Figure 2(b).\n\n2.Some of the reported improvements in AUC appear to be incorrect. For instance, (189 ± 2 (103.33%)) and (148 ± 15 (81.48%)) seem to contain errors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: The learning curves of TD3 and TD3-PPTB\n\nWe provide the missing learning curves in **Figure 9** (for RAD and RAD-PPTB) and **Figure 10** (for TD3 and TD3-PPTB) of the one-page pdf uploaded.\n\nWe hypothesize that the **higher convergence performance** in Ant and Hopper can be explained by **PPTB’s effect in preventing the plasticity loss** in the minor temporal SVD directions to some degree.\n\nIn fact, the policy path trimming (PPT) method proposed in our paper can be viewed as a special way to do network resetting to alleviate the primacy bias as in [Nikishin et al., 2022].\n\nConcretely, our method differs from the vanilla network resetting method in [Nikishin et al., 2022] on two points: (1) we do reset in a transformed subspace (with temporal SVD) rather than the original space; (2) we only reset the parameter update in the minor directions and maintain the learned information in the major directions.\n\nFrom the perspective of plasticity, our method periodically rolls back the plasticity of the network in the directions that are orthogonal to the ones that represent the effective knowledge learned so far.\n\n\n> Q2: “The author only provided results for 100k and 500k steps in the RAD results shown in Table 2. What about the results for 1000k or 2000k steps?”\n\nWe follow the evaluation setting as that in the original paper of RAD (see Table 1 in the original paper).\n\n> Q3: Some of the reported improvements in AUC appear to be incorrect. For instance, (189 ± 2 (103.33%)) and (148 ± 15 (81.48%)) seem to contain errors.\n\nAs described in Line 277-279, for each task, we report the comparative improvement by normalizing with a random-agent baseline as 0 and the DRL base algorithm (i.e., TD3 or RAD) as 1. \n\nThis resembles the convention used for Atari tasks, where the random agent is taken as 0 and the human performance is taken as 1.\n\n> Q4: “in the experiment details, a 2000 or 1e5-dimensional neural network is too large for Mujoco training. Most papers indicate that commonly used model dimensions are 64 or 128”\n\nFor the dimension of each hidden layer, we follow the convention and use 256 for TD3 and RAD. Thus **1e5 is the scale of the total number of network parameters** (and 2000 is the window size of the historical policies maintained), not the dimension of each hidden layer.\n\n> Q5: Can you talk about how to determine the P_b in your experiment? If is not set properly, will it cause excessive oscillation during the training process?\n\nIn our experiments, we found that choosing different $P_b$ may have a difference in the improvement achieved upon the baseline algorithm, but it is quite safe to choose among a relatively large range without incurring a clear performance drop. Choosing a large $P_b$ should be safer as it makes less impact on the parameter update in principle according to the temporal SVD.\n\nThis is also supported by our empirical results for the SVD parameter reconstruction shown in Table 3 to Table 8 in the appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate all the reviewers’ careful review and valuable comments. Please refer to the individual rebuttals for our responses.

&nbsp;

In the one-page pdf uploaded, we provide additional results:
- **[(Suggested by Reviewer zpVk) Additional empirical investigation on Behavior Cloning in D4RL halfcheetah-medium-replay with Adam and SGD optimizers]** Figure 11 presents the results for an extended investigation on Supervised Learning (BC) and RL, momentum-based optimizer (Adam) and vanilla optimizer (SGD). 
- **[(To Reviewer B4U6) The missing learning curves]** Please refer to Figure 9 and Figure 10.

&nbsp;

With these additional discussions and experimental results, we would like to emphasize that **our empirical study** and **the method for improving DRL agents in the policy subspace constructed with temporal SVD** have not been studied before to the best of our knowledge.




&nbsp;

Finally, we sincerely hope that our response can address the questions and concerns raised by the reviewers. We also hope that the reviewers can re-evaluate the value of our work based on the responses and the additional results provided in the one-page pdf. 
We are also more than willing to address any further questions or concerns during the discussion stage.

Meta-review: This paper finds that policy networks in some deep reinforcement learning (DRL) agents evolve in a simplified, low-dimensional space, with significant progress in a few key directions while showing oscillations in others. To improve learning efficiency, the authors propose a method called Policy Path Trimming and Boosting (PPTB), which focuses on the key directions and reduces unnecessary updates, leading to better performance in DRL tasks.

This is an insightful and valuable contribution to this venue. Although reviewers do not have any major concerns and generally appreciate the novelty and significance of the work, there are several ways the paper should be improved, as the reviewers suggested. Specifically, the connection with moment-based optimizers is a natural question that should be addressed. Moreover, there is a concern regarding the statistical significance, which the authors are suggested to take this into account given the context of the current state of the art in RL literature. Computational expense is another obvious concern that will naturally arise, given the work is based on an SVD approach, which is typically understood to be computationally expensive. The authors clarified during the rebuttal how the infrequent use of SVD leads to cheap computation, although this might still increase peak computation (spike in computation momentarily). The authors should address this concern early in the paper.
Decision: Accept (poster)