Title: The Ladder in Chaos: Improving Policy Learning by Harnessing the Parameter Evolving Path in A Low-dimensional Space
Venue: NeurIPS 2024
Accepted: yes
Keywords: Reinforcement Learning

Policy Optimization
Abstract: Knowing the learning dynamics of policy is significant to unveiling the mysteries of Reinforcement Learning (RL). It is especially crucial yet challenging to Deep RL, from which the remedies to notorious issues like sample inefficiency and learning instability could be obtained. In this paper, we study how the policy networks of typical DRL agents evolve during the learning process by empirically investigating several kinds of temporal change for each policy parameter. In popular MuJoCo and DeepMind Control Suite (DMC) environments, we find common phenomena for TD3 and RAD agents: (1) the activity of policy network parameters is highly asymmetric and policy networks advance monotonically along a very limited number of major parameter directions; (2) severe detours occur in parameter update and harmonic-like changes are observed for all minor parameter directions. By performing a novel temporal SVD along the policy learning path, the major and minor parameter directions are identified as the columns of the right unitary matrix associated with dominant and insignificant singular values respectively. Driven by the discoveries above, we propose a simple and effective method, called Policy Path Trimming and Boosting (PPTB), as a general plug-in improvement to DRL algorithms. The key idea of PPTB is to trim the policy learning path by canceling the policy updates in minor parameter directions, and boost the learning path by encouraging the advance in major directions. In experiments, we demonstrate that our method improves the learning performance of TD3, RAD, and DoubleDQN regarding scores and efficiency in MuJoCo, DMC, and MinAtar tasks respectively.

The Ladder in Chaos: Improving Policy Learning by Harnessing the Parameter Evolving Path in A Low-dimensional Space

Abstract

Knowing the learning dynamics of policy is significant to unveiling the mysteries of Reinforcement Learning (RL). It is especially crucial yet challenging to Deep RL, from which the remedies to notorious issues like sample inefficiency and learning instability could be obtained. In this paper, we study how the policy networks of typical DRL agents evolve during the learning process by empirically investigating several kinds of temporal change for each policy parameter. In popular MuJoCo and DeepMind Control Suite (DMC) environments, we find common phenomena for TD3 and RAD agents: (1) the activity of policy network parameters is highly asymmetric and policy networks advance monotonically along a very limited number of major parameter directions; (2) severe detours occur in parameter update and harmonic-like changes are observed for all minor parameter directions. By performing a novel temporal SVD along the policy learning path, the major and minor parameter directions are identified as the columns of the right unitary matrix associated with dominant and insignificant singular values respectively. Driven by the discoveries above, we propose a simple and effective method, called Policy Path Trimming and Boosting (PPTB), as a general plug-in improvement to DRL algorithms. The key idea of PPTB is to trim the policy learning path by canceling the policy updates in minor parameter directions, and boost the learning path by encouraging the advance in major directions. In experiments, we demonstrate that our method improves the learning performance of TD3, RAD, and DoubleDQN regarding scores and efficiency in MuJoCo, DMC, and MinAtar tasks respectively.

1 Introduction

Deep Reinforcement Learning (DRL) is far from well understood, although its great potential has been demonstrated with a lot of achievements in different practical problems [Badia et al., 2020, Shah et al., 2022, Fawzi et al., 2022, Degrave et al., 2022, OpenAI, 2022]. Consistent efforts are made to gain a better understanding of the learning dynamics of RL agents. Different from the studies with tabular [Sutton and Barto, 1988] and linear approximation [Ghosh and Bellemare, 2020], understanding the learning dynamics of DRL agents is challenging. The difficulty comes from the complex interplay between Deep Learning models and RL algorithms, and further escalates when only limited online interactions or offline logged data are considered.

Recently, there have been a few works that study the learning dynamics of DRL agents from different perspectives. A major stream of works among them studies the co-learning dynamics between representation and DRL functions (usually the value network) [Dabney et al., 2021, Kumar et al.,

2021, Lyle et al., 2022, Nikishin et al., 2022, Tang et al., 2022b]. The focus of this stream is that the DRL agent over-shapes its representation toward early experiences and objectives, and becomes less capable of learning for the later learning process. This degradation finally leads to myopic convergence or even divergence. A related work [Sokar et al., 2023] presents a phenomenon called Dormant Neuron in DRL. It reveals that the neurons of typical value networks gradually become inactive during the learning process, leading to the loss of network expressivity. From another angle, a phenomenon called Policy Churn is discovered by [Schaul et al., 2022]. It shows that the greedy policy of a typical value network changes on about 10% of the states after only a single update. These discoveries provide useful insights for understanding the learning behavior of the DRL agent, based on which new methods can be proposed to improve the learning performance of DRL.

In this paper, we aim to unveil the learning dynamics of the policy network during the training process of typical DRL agents. Specifically, we focus on how the parameters of the policy network evolve along the policy learning path , i.e., the sequence of historical policy networks. Taking MuJoCo [Brockman et al., 2016] and DeepMind Control Suite (DMC) [Tassa et al., 2018] as typical DRL environments, we conduct a series of empirical investigations on the policy learning path of TD3 [Fujimoto et al., 2018] and RAD [Laskin et al., 2020] agents respectively. From the perspective of accumulated absolute parameter change and a novel temporal SVD view, we present four commonly observed phenomena on the policy learning path. The main message from the phenomena is that the policy networks of DRL agents advance nearly monotonically along only very few major parameter directions and show harmonic-like oscillations on almost all other minor ones.

This drives us to ask the question: can we make the DRL agent focus on the policy learning along major directions and suppress the oscillation in minor directions? To this end, we propose a new method called Policy Path Trimming and Boosting (PPTB) . The key idea of PPTB is to trim the policy learning path by canceling the policy parameter updates in minor dimensions, and boost the learning path by encouraging the advance in major directions. The method is a simple and general plug-in improvement for DRL agents, which can be implemented with only a few lines of code modification to most DRL methods. Finally, we evaluate the effectiveness of PPTB based on TD3, RAD, and DoubleDQN [van Hasselt et al., 2016] in MuJoCo, DMC, and MinAtar [Young and Tian, 2019] environments.

Key contributions of this work are summarized below: (1) We propose a temporal analysis view to study the policy learning path of typical DRL agents. (2) We present four common phenomena from our empirical investigations, which reveal the policy evolving path in a low-dimensional space. (3) We propose an easy-to-implement and general improvement for general DRL algorithms, which is demonstrated to improve the learning performance of TD3, RAD, and DoubleDQN regarding scores and efficiency in MuJoCo, DMC, and MinAtar environments.

2 Preliminaries

Markov Decision Process (MDP) Consider a Markov Decision Process (MDP) ⟨S , A , P , R , γ, ρ 0 , T ⟩ , defined with a state set S , an action set A , the transition function P : S × A × S → R , the reward function R : S × A → R , the discounted factor γ ∈ [0 , 1) , the initial state distribution ρ 0 and the horizon T . The agent interacts with the MDP by performing its policy π : S → P ( A ) that defines the distribution over all actions for each state. The objective of an RL agent is to optimize its policy to maximize the expected discounted cumulative reward J ( π ) = E π [ ∑ T t =0 γ t r t ] , where s 0 ∼ ρ 0 ( s 0 ) , a t ∼ π ( s t ) , s t +1 ∼ P ( s t +1 | s t , a t ) and r t = R ( s t , a t ) . The state-action value function Q π is defined as Q π ( s, a ) = E π [ ∑ T t =0 γ t r t | s 0 = s, a 0 = a ] for all s, a ∈ S × A .

Deep Reinforcement Learning (DRL) With function approximation based on deep neural networks, an RL agent is able to deal with large and continuous state-action space. Conventionally, Q π can be approximated by Q ϕ with parameters ϕ typically through minimizing Temporal Difference loss [Sutton and Barto, 1988], i.e., L ( ϕ ) = 1 2 [ Q ϕ ( s, a ) -E a ′ ∼ π ( s ′ ) ( r + γQ ¯ ϕ ( s ′ , a ′ ) )] 2 . A parameterized policy π θ , with parameters θ , can be updated by taking the gradient of the objective, i.e., θ ′ ← θ + η ∇ θ J ( π θ ) with a learning rate η . Therefore, starting from a initial policy π θ 1 , the DRL agent proceeds along the learning process and obtains a sequence of policies { π θ 1 , π θ 2 , . . . } . For two representative DRL algorithms, Deterministic Policy Gradient (DPG) theo-

rem [Silver et al., 2014] is often used to update a deterministic policy with the gradient: ∇ θ J ( π θ ) = E s ∼ ρ π θ [ ∇ θ π θ ( s ) ∇ a Q ϕ ( s, a ) | a = π θ ( s ) ] , where ρ π θ is the discounted state distribution under policy π θ ; Soft Actor-Critic (SAC) [Haarnoja et al., 2018] updates a stochastic policy with the gradient: ∇ θ ˆ J ( π θ ) = E s ∼ ρ π θ [ ∇ θ log π θ ( a | s ) +( ∇ a log π θ ( a | s )) -∇ a Q ϕ ( s, a )) ∇ θ f θ ( ϵ ; s )) | a = f θ ( ϵ ; s ) ] (with noise ϵ and implicit function f θ for re-parameterization), based on the maximum-entropy objective ˜ J ( π θ ) = E π [ ∑ T t =0 γ t r t + ξ H ( π θ ( ·| s )] where ξ is the temperature of entropy term.

3 Phenomena on DRL Policy Learning Path

In this section, we conduct empirical investigations on the typical DRL policy learning process. In specific, we use official codes of TD3 [Fujimoto et al., 2018] and RAD [Laskin et al., 2020] for OpenAI MuJoCo [Brockman et al., 2016] continuous control environments and visual-input DeepMind Control (DMC) Suite [Tassa et al., 2018] environments respectively. Both of them use a policy architecture of two-layer MLP (where the output layer is viewed as Layer 3 ). Note that we exclude the convolution layers for image representation in RAD here and focus on the policy part [Chung et al., 2019]. All the experimental details can be found in Appendix D.

First of all, we call the sequence of policies { π 1 , π 2 , . . . , π n } or policy parameters { θ 1 , θ 2 , . . . , θ n } obtained during the learning process of a typical DRL algorithm as policy learning path or policy parameter path throughout this paper. The policy parameters θ i is an m -dimensional vector θ i = [ θ i, 1 , θ i, 2 , . . . , θ i,m ] ∈ R m . In the following, we investigate how policy network parameters evolve along the policy learning path from different angles.

3.1 Policy Parameter Change and Detour

We start by tracking the absolute change amount of each parameter in policy network. Given any j ∈ { 1 , 2 , · · · , m } , we first introduce three quantities for our following investigations.

Accumulated Parameter Change: ∆ apc j ( { θ i } n i =1 ) ≜ ∑ n -1 i =1 | θ i +1 ,j -θ i,j | .

Final Parameter Change: ∆ fpc j ( { θ i } n i =1 ) ≜ | θ n,j -θ 1 ,j | .

Parameter Update Detour Ratio: r pud j ( { θ i } n i =1 ) ≜ ∑ n -1 i =1 | θ i +1 ,j -θ i,j | | θ n,j -θ 1 ,j | .

We use ∆ apc j , ∆ fpc j , r pud j as abbreviations when the context is clear and write (∆ apc j ) , (∆ fpc j ) , ( r pud j ) as the vectors consisting of the quantities from all parameters. It can be derived that the larger r pud j is, the more detours there are in the evolving path of parameter j . For each environment, we run TD3 or RAD for three trials and collect the policies along the learning process at intervals. We then calculate vectors (∆ apc j ) , ( r pud j ) and plot their CDF histograms in a layer-wise manner (i.e., Layer 1,2,3). The results are shown in Fig. 1. The results are similar in almost all other MuJoCo and DMC environments, and the complete figures can be found in Appendix D.1.

We begin by assessing the magnitude of accumulated changes in the policy parameters. CDF histograms of vector (∆ apc j ) (as shown in left panels of Fig. 1 (a), (b)) indicate that a certain portion of parameters (ranging from 10% to 40% across environments) in the second and output layers experience minor accumulated changes. In contrast, the cumulative changes for nearly all parameters in the first layer are significantly higher, with thresholds around 0.7 for Hopper and 0.6 for walkerwalk. This observation marks our initial key finding.

Phenomenon 1.1 (Parameter Change Asymmetry): There is a significant discrepancy in the change amount among policy parameters. The second and the output layers have similar patterns which differ from that of the first layer.

Further discussion about Phenomenon 1.1 can be found in Appendix D.1.

Moving forward, we examine how much each policy parameter detours from its initial value to its final value. The CDF histograms of ( r pud j ) (as shown in the right panels of Fig. 1 (a) and (b)) indicate that a significant proportion of parameters exhibit substantial detour ratio, which we identify as the second phenomenon.

1.0

50000

0.8

40000

0.6

30000

0.4

20000

0.2

10000

Amount of elements in each bin

0.0

0

0.0

Hopper-v4: Policy parameter change amount

Hopper-v4: Policy parameter change amount

Layer 1

Layer 2

800000

700000

Layer 3

600000

500000

400000

300000

200000

100000

Amount of elements in each bin

0

Amount of elements in each bin

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

0.4

0.6

0.8

1.0

1.2

Amount of accu. parameter change (1e-3)

Amount of accu. parameter change (1e-3)

walker-walk: Policy parameter change amount

20000

Hopper-v4: Ratio of accu. and final parameter change amount

17500

15000

12500

10000

7500

5000

2500

0.0

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Amount of accu. parameter change (1e-3)

0.0

0.2

0.4

0.6

0.8

1.0

Percentage of elements in each bin

0

50

100

150

200

250

300

Ratio of accu. and final parameter change amount

0

2500

5000

7500

10000

12500

15000

17500

20000

Amount of elements in each bin

0

50

100

150

200

250

300

Ratio of accu. and final parameter change amount

0.0

0.2

0.4

0.6

0.8

1.0

Percentage of elements in each bin

0

50

100

150

200

250

300

Ratio of accu. and final parameter change amount

0

0

50

100

150

200

250

300

Ratio of accu. and final parameter change amount

0.0

0.2

0.4

0.6

0.8

1.0

Percentage of elements in each bin

(a) MuJoCo/Hopper

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Amount of accu. parameter change (1e-3)

0

100000

200000

300000

400000

500000

600000

700000

800000

Amount of elements in each bin

Layer 1

Layer 2

Layer 3

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Amount of accu. parameter change (1e-3)

0.0

0.2

0.4

0.6

0.8

1.0

Percentage of elements in each bin

0

100

200

300

400

Ratio of accu. and final parameter change amount

0.0

0.2

0.4

0.6

0.8

1.0

Amount of elements in each bin

1e6

0.2

0.4

0.6

0.8

1.0

1.2

Amount of accu. parameter change (1e-3)

Layer 1

Layer 2

Layer 3

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Amount of accu. parameter change (1e-3)

0.0

0.2

0.4

0.6

0.8

1.0

Percentage of elements in each bin

0

100

200

300

400

500

Ratio of accu. and final parameter change amount

0.0

0.2

0.4

0.6

0.8

1.0

Amount of elements in each bin

1e6

0

100

200

300

400

500

Ratio of accu. and final parameter change amount

0.0

0.2

0.4

0.6

0.8

1.0

Percentage of elements in each bin

(b) DMC/walker-walk

walker-walk: Ratio of accu. and final parameter change amount walker-walk: Ratio of accu. and final parameter change amount

500

0

200

400

600

800

Indices of singular values

0.5

0.6

0.7

0.8

0.9

1.0

Amount of acc. information amount

walker-walk: SVD information of policy learning path

Layer:1, Period:1

Layer:3, Period:1

0

200

400

600

800

Indices of singular values

0.5

0.6

0.7

0.8

0.9

1.0

Amount of acc. information amount

walker-walk: SVD information of policy learning path

Layer:1, Period:2

Layer:3, Period:2

0

200

400

600

800

Indices of singular values

0.5

0.6

0.7

0.8

0.9

1.0

Amount of acc. information amount

walker-walk: SVD information of policy learning path

Layer:1, Period:3

Layer:3, Period:3

(a) Singular Value Information Curves

0

10

20

30

40

50

Timestep (x 2e4)

0

200

400

600

800

1000

Average Return

0

200

400

600

800

1000

Time index

-0.04

-0.02

0.00

0.02

0.04

Value of

u

[

k

]

k

:1, det:1.00

k

:2, det:1.03

k

:3, det:1048.23

k

:4, det:3.02

k

:5, det:164.19

0

200

400

600

800

1000

Time index

-0.10

-0.05

0.00

0.05

0.10

Value of

u

[

k

]

walker-walk-svd-raw: SVD Singular Analysis

k

:50, det:74.35

k

:100, det:1588.96

k

:200, det:291.43

k

:500, det:860.40

k

:900, det:6780.46

0

200

400

600

800

1000

Index

i

of singular values

500

1000

1500

2000

2500

3000

Detour ratio of

u

[

i

]

0

200

400

600

800

1000

Index

i

of singular values

0.000

0.025

0.050

0.075

0.100

0.125

0.150

0.175

Total

u

[

i

] diff

(b) SVD Left Unitary Matrix (

U

) Investigations

Phenomenon 2.1 (Parameter Update Detour): There are severe detours in policy parameter update. All the three layers show similar patterns.

It can be easy to consider that such detours or oscillations in policy parameter updates can be mainly attributed to noisy policy gradients. This also reveals the correlation between policy parameters where most of them can not be updated independently.

3.2 Temporal SVD Analysis of Policy Learning Path

The two phenomena observed above naturally raise further questions: 1) The asymmetry of parameter change shown by Phenomenon 1.1 indicates the imbalanced importance of parameters. Can we rule the important parameters or parameter directions off from the less important ones ? 2) For the detour shown by Phenomenon 2.1, can we identify the difference among parameters or parameter directions in their detour behaviors ? And do important parameters or parameter directions detour less?

In this section, we make use of a temporal viewpoint on policy path. We introduce Temporal Singular Value Decomposition (SVD) as our main tool for the following investigations. Given a policy path { θ 1 , θ 2 , . . . , θ n } with θ i = [ θ i, 1 , θ i, 2 , . . . , θ i,m ] , where m ( ≥ 10 5 usually) is the dimenionality of

Hopper-v4: Policy parameter change amount walker-walk: Policy parameter change amount

walker-walk: Policy parameter change amount

Hopper-v4: Ratio of accu. and final parameter change amount

Hopper-v4: Ratio of accu. and final parameter change amount walker-walk: Policy parameter change amount

walker-walk: Ratio of accu. and final parameter change amount

Hopper-v4: Ratio of accu. and final parameter change amount walker-walk: Ratio of accu. and final parameter change amo

1.0

Percentage of elements in each bin

0.8

0.6

0.4

0.2

0.0

0

100

200

300

400

500

Ratio of accu. and final parameter change amount

policy parameters. We then stack the parameter vectors along the policy path and form a temporal policy parameter matrix, for which standard SVD can be performed:

where U, V are left and right unitary matrices, the singular values are indexed in decreasing order with d = min( n, m ) . For the convenience of expression, we use u i, ∗ for the i -th row vector of U and use u ∗ ,j for the j -th column vector (and the same way for V ⊤ ). The temporal SVD offers us an angle to view how policy evolves in a lower-dimensional space : we can now take the row vectors of U as new d -dimensional coordinates for policies along the policy path (i.e., u i, ∗ for the i -th policy) regarding the scaling vector Diag (Σ) and the parameter subspace spanned by the row vectors of V ⊤ , i.e., Span ( { v ⊤ 1 , ∗ , v ⊤ 2 , ∗ , . . . , v ⊤ d, ∗ } ) ⫅ R d . In the following, we also call { v i, ∗ } as SVD directions . For any index j < d , it is obvious that u ∗ ,j is the evolving path of the j -th new coordinate.

To answer the questions listed at the beginning of this subsection, we introduce two more quantities:

Singular Value Information Amount for a dimensionality number k ∈ { 1 , . . . , d } : a k = ∑ k i =1 σ i ∑ d i =1 σ i .

SVD Major Dimensionality regarding an information threshold β ∈ (0 , 1] : D ( β ) = min { k ∈ { 1 , . . . , d } : α k ≥ β } .

Note that d = D (1) and D (0 . 99) recovers approximate rank used in a few recent works [Yang et al., 2020, Kumar et al., 2021, Lyle et al., 2022]. Additionally, a smaller D ( β ) implies fewer dominant SVD directions when β is held constant.

We also use the policy path data obtained by TD3 on MuJoCo and RAD on DMC as in Sec. 3.1. First, we plot D ( β ) against various threshold candidates β ∈ { 0 . 5 , 0 . 6 , 0 . 7 , 0 . 8 , 0 . 85 , 0 . 9 , 0 . 95 , 0 . 99 } . In addition to a layer-wised manner, we consider three periods (i.e., early, middle, and later) of learning process for a temporal view. The results for DMC/walker-walk are shown in Fig. 2(a). The results are similar in almost all other MuJoCo and DMC environments and the complete figures can be found in Appendix D.2.

The following observations can be made: (i) Within the range of β from 0.5 to 0.8, we observe that D ( β ) for all three layers in both environments consistently remain near 0 . This observation suggests that the policy learning path primarily proceeds within a low-dimensional subspace of the entire parameter space, particularly within the dimensions spanned by the most dominant SVD directions. This provides additional insights into Phenomenon 1.1, elucidating which parameters undergo more significant changes. (ii) The curves of the second layer show a more pronounced steepness near β = 0 , indicating that the parameters in this layer evolve within a subspace characterized by fewer dominant SVD directions compared to those in other layers. This observation also matches the results in Fig. 1, as the second layer is often more over-parameterized than the other layers. (iii) As training advances from period 1 to period 3, the curve's steepness around β = 0 correspondingly increases, revealing a growing concentration on a diminishing number of dominant singular values. The higher concentration of later periods is easy to understand since policy improvement gradually becomes slower. We can summarize these observations as our third phenomenon.

Phenomenon 1.2 (Singular Value Information Concentration): The singular value information is highly concentrated on the first several singular values. The concentration is more evident for the second layer and later periods.

Further, regarding the second question raised in the beginning, we expect to find some correlation between the policy parameter path in the low-dimensional subspace obtained by SVD and the improvement of policy performance. To this end, in Figure 2(b), we examine the evolving path of the k -th coordinate by plotting coordinate vectors u ∗ ,k (panels 2, 3). We further present the curves of detour ratio ( r pud i ) (panel 4) and final change (∆ fpc i ) (panel 5) with respect to the singular value index

𝚯

)

Original Policy Path

𝜽

𝒕

Trimmed and Boosted Policy Path

Trimming in

Local Minor Direction

Major Direction

Minor Direction

Boosting in

Local Major Direction

𝜽

𝒕

"

i . These results are against the policy performance curve (panel 1), and we focus on only the period with significant policy improvement, i.e., the left of the blue dashed vertical line.

According to the results in Fig. 2(b), we observe the final phenomenon.

Phenomenon 2.2 (Policy Path in Major and Minor SVD Directions): The extent of the detour becomes increasingly pronounced as the coordinate index k increases (from the major SVD directions to the minor ones). Specially, u ∗ , 1 is almost monotonic while u ∗ ,k shows harmonic-wave-like changes with overall increasingly higher frequencies for k ≥ 2 .

It is surprising to observe Phenomenon 2.2. Intuitively, it indicates that the policy parameter path proceeds monotonically along one major direction and oscillates in other directions with frequencies inversely proportional to singular values. This empirically supports our hypothesis that important (i.e., major) parameter directions detour less while insignificant directions detour severely. Since typical policy gradients are derived regarding approximate value estimates, we suggest that the dynamics of policy parameters may be closely related to recent studies on the learning dynamics of value function [Lyle et al., 2021, 2022]. We leave it as a major direction of future work.

For a brief summary, till now we have observed that the policy parameter path evolves mainly in a low-dimensional parameter subspace (spanned by { v i, ∗ } ). This is commonly seen in our empirical investigations for typical DRL algorithms in popular environments with proprioceptive or visual observations. A natural idea is: why not let the agent focus on the policy update in the parameter subspace, by following the major SVD directions and neglecting the insignificant SVD directions ? Moreover, somewhat excitingly, it seems that u ∗ , 1 , u ∗ , 2 has a strong correlation to policy performance. We are curious about whether it is possible to leverage the correlation to boost the learning process . In the next section, we study on these points along with the proposal of Policy Path Trimming and Boosting.

4 Policy Path Trimming and Boosting

Driven by the phenomena we discovered in the previous section, we propose a simple and effective method, called Policy Path Trimming and Boosting ( PPTB ), as a general plug-in improvement to DRL algorithms. In the following, we introduce the details of the two components of PPTB, i.e., Policy Path Trimming (Sec. 4.1) and Policy Path Boosting (Sec. 4.2), and then the general implementation of DRL with PPTB (Sec. 4.3).

4.1 Policy Path Trimming

As summarized in Phenomenon 1.2, we have observed that the policy path mainly evolves in a lowdimensional parameter subspace with a large proportion of singular value information concentrated in the first several singular values. Our first idea is to truncate the parameter change in minor SVD directions and only keep the change in major ones based on the Temporal SVD of the policy path. We call this method as Policy Path Trimming (PPT).

Given a policy path { θ 1 , θ 2 , . . . , θ n } and the number of major directions to remain r t (usually ≪ d = min( n, m ) ). For a policy with original parameters θ i , PPT reconstructs policy parameters by only taking into consideration the first r t SVD directions:

Note that we use · , · in subscripts and [ · , · ] to denote the slices of vector and matrix respectively.

A conceptual illustration of PPT is shown in Fig. 3. As shown by the red arrows and cross, PPT trims the policy parameter update in minor SVD directions and enforces the policy path proceeds in major directions, i.e., in the subspace Span ( { v ⊤ 1 , ∗ , v ⊤ 2 , ∗ , . . . , v ⊤ r t , ∗ } ) . Intuitively, this suppresses the detours and oscillations of parameter update (recall the results in Fig. 2(b)). Therefore, we expect PPT to improve the efficiency of policy learning process in this sense.

One may worry about whether the trimmed policy parameters θ ′ i still ensure an effective policy. For sanity check, we compare policy performance between θ i and θ ′ i regarding different choices of r t . We refer the readers to Table 4 to 9 in Appendix D.3. We found that a small r t ( ≤ 128 ) is sufficient to ensure a valid recovery of policy performance, while increasing r t shows no significant difference. Somewhat surprisingly, we found r t ( ≤ 64 ) often achieves improved performance after reconstruction, which supports the effectiveness of PPT to some extent. In addition, PPT can be viewed as a particular way to do network resetting to alleviate the primacy bias [Nikishin et al., 2022]. More discussion on this can be found in Appendix B.

4.2 Policy Path Boosting

In addition to suppressing the parameter change in minor directions, we are interested in accelerating policy learning by leveraging the dynamics pattern of major SVD directions as noted in Phenomenon 2.2. In particular, we propose to boost the change of u ∗ , 1 , u ∗ , 2 since they show near monotonic changes corresponding to the improvement of policy performance in Fig. 2(b). We call this method as Policy Path Boosting (PPB).

For a policy with original parameters θ i among the policy path { θ 1 , θ 2 , . . . , θ n } , PPB modifies θ n by increasing u n, 1 , u n, 2 along the temporal direction as follows with a boosting coefficient p b :

PPB only modifies u n, 1 , u n, 2 while keeps other parts unchanged. As illustrated by the green arrows in Fig. 3, Intuitively, PPB boosts the policy parameter update in the first two major directions to accelerate the improvement of policy performance.

4.3 DRL with PPTB

Now, we are ready to propose PPTB as a combination of PPT and PPB. Formally,

Although we can perform PPTB for any policy on an arbitrary policy path, in practice we consider the policy path that consists of the historical policies within a recent window and the current policy, and we perform PPTB for the current policy.

Apparently, PPTB is algorithmic-agnostic. For almost all off-the-shelf policy-based DRL algorithms, PPTB can be implemented and incorporated by adding the following three steps to the conventional policy update scheme: (1) Initialize a policy buffer and store the parameters of the current policy at intervals along the policy learning process; (2) On certain occasions, retrieve the recent policy parameter path and perform Temporal SVD; then do policy path trimming and boosting for the current policy. (3) Load the consequent policy parameters processed by PPTB back to the current policy.

In our practical implementation used by our experiments, we only make the modifications of about 10-line core codes in the official implementations of TD3 [Fujimoto et al., 2018] and RAD [Laskin et al., 2020]. We refer the readers to Algorithm 1 in the appendix for a pytorch-like pseudocode.

5 Experiments

5.1 Performance Evaluation of PPTB

Setups To evaluate the performance of our proposed method PPTB, we consider the continuous control environments in OpenAI MuJoCo [Brockman et al., 2016] and DeepMind Control Suite (DMC) [Tassa et al., 2018], which cover both proprioceptive and visual observation. We use TD3 [Fujimoto et al., 2018] and RAD [Laskin et al., 2020] as the base algorithms for MuJoCo and DMCrespectively, due to their simplicity and effectiveness. We use the official codes of TD3 and RAD and modify them according to Algorithm 1 to implement PPTB with no other change to the original implementation. Our code is available in https://github.com/bluecontra/PPTB .

Moreover, we also evaluate the performance of PPTB for value-based RL, although the greedy policy is implicitly derived from the value networks. Specifically, we use three tasks from MinAtar [Young and Tian, 2019] and use DoubleDQN [van Hasselt et al., 2016] as the value-based RL baseline. More implementation details are provided in Appendix C.

Results for AC Methods We train the agent for 1 million timesteps and evaluate the agent every 5k timesteps for each agent-environment configuration. We run each configuration with six random seeds. We consider two evaluation metrics: (1) Score: the best average evaluation returns (across multiple runs) over the course of learning, which is used by TD3 [Fujimoto et al., 2018]; (2) AUC: the mean of average evaluation returns over the course of learning, which is also used in [Kumar et al., 2022]. The former cares about effectiveness while the latter measures efficiency and stability , which is also significant to the practical use of DRL algorithms.

For MuJoCo environments, we report Score and AUC for 1M time step training; for DMC, we report for 100k, 500k as usually done in prior works [Laskin et al., 2020]. For the convenience of comparing across different return scales, we report the comparative improvement ( ↗ ) and the aggregate results by normalizing with a random-agent baseline as 0 and the DRL base algorithm (i.e., TD3 or RAD) as 1. The score of the random-agent baseline is omitted. We report the means and standard deviation errors across six independent trials.

The results for MuJoCo and DMC are reported in Table 1 and 2, respectively. The results show the overall improvements brought by PPTB for both TD3 and RAD, indicating its compatibility and effectiveness. The improvement is more obvious in AUC, which means PPTB improves the learning efficiency and stability of the base algorithms. Somewhat surprisingly, we can observe that PPTB outperforms the base algorithms by a large margin in several environments, e.g., Walker2d and Ant in MuJoCo, and Walker-walk in DMC. To some extent, this reveals the potential of studying and modifying the policy learning path to achieve general improvements to DRL agents.

Results for Value-based Methods We run 3M time steps for SpaceInvader and Breakout, and 5M time steps for Seaquest. We reported the means and standard errors of the final episode returns across six seeds in Table 3.

The results show that PPTB also improves the learning performance of DoubleDQN somewhat significantly in MinAtar tasks with discrete actions. Note that for DoubleDQN, value networks are learned to derive the greedy actions. Intuitively, the learning dynamics of the value learning path and the policy learning path in AC methods are different as typical TD learning adopts bootstrapping and semi-gradient [Sutton

and Barto, 1988]. We view our experimental results here as preliminary evidence of the effectiveness of PPTB for value-based methods, which also inspired us to further investigate this point in the future.

5.2 Investigation on Policy Learning Path of Behavior Clone

The policy learning path is influenced by multiple factors, including, policy learning algorithm, distribution of training data, network architecture, optimizer, etc. To expand our study on typical online RL a bit, we investigate the policy learning path of Behavior Cloning (BC) with Adam and vanilla SGD on halfcheetah-medium-replay, an offline dataset in D4RL [Fu et al., 2020]. This additional setting provides two comparison angles, i.e., RL v.s. BC, Adam v.s. SGD (where using BC and the same offline dataset fixes the influence of data distribution). Similar to our investigation in Sec. 3, we perform parameter analysis and Temporal SVD analysis to the logged policy paths through learning under different configurations. The results are presented in Figure 4 and 12.

For the parameter analysis, we can observe that BC and RL show different distributions of network parameter update amount. Compared with the RL cases reported in Figure 5(a), the parameter update amount is more Gaussian distribution like for BC (Adam) in Figure 4(a). In addition, by comparing the parameter analysis of BC (Adam) and BC (SGD), somewhat surprisingly, we can observe that the asymmetry of parameter update amount turns out to be more severe in BC (SGD). The momentum mechanism used in Adam leads to a relatively more evenly distributed network parameter update amount, while SGD has a large number of minor updates (also reflected by the flat curve in Figure 4(d), the first column) and a long tail in distribution.

For the Temporal SVD analysis, we can observe that BC (Adam), BC (SGD) and the RL cases show different patterns in terms of temporal SVD evolvement. BC policies show wavelet-like oscillations (Figure 4(b)), where the amplitude of the oscillations seems to decrease throughout training; while the oscillations of RL are more harmonic-wave-like (Figure 8(a)).

These observations reveal that low-dimensional policy paths can exist in many policy learning problems with different features influenced by multiple factors. We expect further studies in this direction in the future.

(a) Parameter Analysis of BC (

Adam

)

(b) Temporal SVD Analysis of BC (

Adam

)

(c) Parameter SVD Analysis of BC (

SGD

)

(d) Temporal SVD Analysis of BC (

SGD

)

medium-replay: Policy parameter change amount

Layer 1

1.0

medium-replay: Policy parameter change amount

80'0

k:1, det:1.00

0.125

k:50, det:29.13

Amount of elements in each bin

20000

15000

Layer 2

Layer 3

0.8

0.6

Value of u[k]

0.04

0.02

0.06

k:2, det:2.32

k:3, det:44.14

k:4, det:3.35

k:5, det:10.60

Value of u[k]

0.100

0.075

0.050

0.025

k:200, det:1757.10

k:100, det:68.92

k:500,det:386.29

0.4

0.00

000°0

0.02

0.025

0005

0.2

0.04

0.050

0.06

-0.075

Amount of accu. parameter change (1e-3)

0.5

1.0

1.5

5

8.0

3.5

0.0

0.0

Amount of accu. parameter change (1e-3)

0.5

1.0

1.5

2.0

2.5

3.5

250500

75010001250

Time index

15001750

2000

250

500

750

Time index

1000

1250

Amount of elements in each bin

100000

medium-replay: Policy parameter change amount

1Layer 1

Layer 2

Percentage of elements in each bin

1.0

medium-replay: Policy parameter change amount

0.05

0.075

80000

Layer 3

0.00

0.050

Value of u[k]

Value of u[k]

0.025

60000

0.05

0.00

4000

k:1, det:1.00

0.025

k:50,det:395.39

0.10

k:2, det:1.00

k:100,det:1935.35

20000

o21

0.15

k:5, det:3.05

k:3, det:2.91

k:4, det:1.66

0.050

0.075

k:900, det:2525.98

k:500, det:1234.75

k:200,det:1156.98

0.00

Amount of accu. parameter change (1e-3)

0.02

0.040.060.080.10

0.12

0.0

000

Amount of accu. parameter change (1e-3)

0.02

0.04

0.06

0.08

0.10

0.12

250

500

750

Time index

1000

1250

150017502000

250500

750

Time index

10001250

1500

17502000

6 Related Works

In the topic of general Deep Learning, there have been a large number of efforts devoted consistently during the past decade for the purpose of understanding the learning dynamics and behaviors of deep neural networks, e.g., Neural Tangent Kernel (NTK) [Jacot et al., 2018] and Lazy Training [Chizat et al., 2018]. Recently, there have been a few works that study the learning dynamics of DRL agents from different perspectives. Among them, a major stream of works study the co-learning dynamics between representation and RL functions (i.e., value network or policy network) [Kumar et al., 2021, Lyle et al., 2022, Nikishin et al., 2022]. The focus of this stream is that the DRL agent over-shapes its representation towards early experiences and objectives (e.g., TD targets of early policies) and becomes less capable for the later learning process. Such a degradation becomes more severe and detrimental gradually due to the non-stationary learning nature, finally leading to myopic convergence or even divergence. In this work, we study the learning dynamics of typical DRL policy networks, mainly from the angle of policy network parameters. To the best of our knowledge, we are almost the first to study how the parameters of practical DRL policy networks evolve.

To our knowledge, there are some works that conduct their study from the angle of the path of learning. Dabney et al. [2021] takes inspiration from the study in [Bellemare et al., 2019] and proposes the concept of the value improvement path . With this concept, a new method for representation learning is proposed by approximating the value functions along the path. From another angle, Tang et al. [2022a] studies the generalization along the policy learning path. Through learning a low-dimensional policy embedding, an improved generalized policy iteration [Sutton and Barto, 1988] that leverages policy generalization is proposed. Closely related to our work, a very recent work [Schneider et al., 2024] demonstrates the existence of gradient subspaces in deep policy gradient algorithms with experimental investigations. In this paper, we investigate the dynamics of policy network parameters from the perspective of Temporal SVD. Moreover, we propose a new method that leverages the parameter evolving path in the low-dimensional space, which demonstrates the potential for improving learning. In essence, the policy learning path is a special scenario of stochastic gradient descent (SGD) in DRL, thus this work is related to studies on the dynamics of SGD or optimization [Gur-Ari et al., 2018, Gressmann et al., 2020, Li et al., 2023] in more general speaking.

7 Conclusion

In this paper, we aim to unveil the learning dynamics of policy network over the training course of typical DRL agents, mainly from an empirical perspective. Focusing on the concept of policy learning path, we conduct a series of empirical investigations and summarize four common phenomena, revealing that the policy learning path of typical DRL agents evolves mainly in a low-dimensional space with monotonic and waggling changes in major and minor parameter directions respectively. Driven by our discovery, we propose a simple method, called Policy Path Trimming and Boosting (PPTB), as a plug-in improvement to general DRL algorithms. We demonstrate the effectiveness of PPTB based on TD3 and RAD in a few MuJoCo and DMC environments.

References

Jordan T. Ash and Ryan P. Adams. On warm-starting neural network training. In NeurIPS , 2020.

Badia, B. Piot, S. Kapturowski, P. Sprechmann, A. Vitvitskyi, Z. Guo, and C. Blundell. Agent57: Outperforming the atari human benchmark. In ICML , volume 119, pages 507-517, 2020.

Marc G. Bellemare, Will Dabney, Robert Dadashi, Adrien Ali Taïga, Pablo Samuel Castro, Nicolas Le Roux, Dale Schuurmans, Tor Lattimore, and Clare Lyle. A geometric perspective on optimal representations for reinforcement learning. In NeurIPS , pages 4360-4371, 2019.

Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym. arXiv preprint, arXiv:1606.01540 , 2016.

Chizat, E. Oyallon, and F. R. Bach. On lazy training in differentiable programming. In NeurIPS , 2018.

Chung, S. Nath, A. Joseph, and M. White. Two-timescale networks for nonlinear value function approximation. In ICLR , 2019.

Dabney, A. Barreto, M. Rowland, R. Dadashi, J. Quan, M. G. Bellemare, and D. Silver. The value-improvement path: Towards better representations for reinforcement learning. In AAAI , pages 7160-7168, 2021.

Degrave, F. Felici, J. Buchli, M. Neunert, B. D. Tracey, F. Carpanese, T. Ewalds, R. Hafner, A. Abdolmaleki, D. de Las Casas, C. Donner, L. Fritz, C. Galperti, A. Huber, J. Keeling, M. Tsimpoukelli, J. Kay, A. Merle, J. Moret, S. Noury, F. Pesamosca, D. Pfau, O. Sauter, C. Sommariva, S. Coda, B. Duval, A. Fasoli, P. Kohli, K. Kavukcuoglu, D. Hassabis, and M. A. Riedmiller. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature , 602(7897):414-419, 2022.

Fawzi, M. Balog, A. Huang, T. Hubert, B. Romera-Paredes, M. Barekatain, A. Novikov, F. J. R. Ruiz, J. Schrittwieser, G. Swirszcz, D. Silver, D. Hassabis, and P. Kohli. Discovering faster matrix multiplication algorithms with reinforcement learning. Nature , 610(7930):47-53, 2022.

Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: datasets for deep data-driven reinforcement learning. arXiv preprint , arXiv:2004.07219, 2020.

Fujimoto, H. v. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In ICML , volume 80, pages 1582-1591, 2018.

Ghosh and M. G. Bellemare. Representations for stable off-policy reinforcement learning. In ICML , 2020.

Frithjof Gressmann, Zach Eaton-Rosen, and Carlo Luschi. Improving neural network training in low dimensional random bases. In NeurIPS , 2020.

Guy Gur-Ari, Daniel A. Roberts, and Ethan Dyer. Gradient descent happens in a tiny subspace. arXiv preprint, arXiv:1812.04754 , 2018.

Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In ICML , volume 80, pages 1856-1865, 2018.

Jacot, C. Hongler, and F. Gabriel. Neural tangent kernel: Convergence and generalization in neural networks. In NeurIPS , pages 8580-8589, 2018.

Kumar, R. Agarwal, D. Ghosh, and S. Levine. Implicit under-parameterization inhibits dataefficient deep reinforcement learning. In ICLR , 2021.

Kumar, R. Agarwal, T. Ma, A. Courville, G. Tucker, and S. Levine. DR3: value-based deep reinforcement learning requires explicit regularization. In ICLR , 2022.

Laskin, K. Lee, A. Stooke, L. Pinto, P. Abbeel, and A. Srinivas. Reinforcement learning with augmented data. In NeurIPS , 2020.

Tao Li, Lei Tan, Zhehao Huang, Qinghua Tao, Yipeng Liu, and Xiaolin Huang. Low dimensional trajectory hypothesis is true: Dnns can be trained in tiny subspaces. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(3):3411-3420, 2023.

Lyle, M. Rowland, G. Ostrovski, and W. Dabney. On the effect of auxiliary tasks on representation dynamics. In AISTATS , volume 130, pages 1-9, 2021.

Lyle, M. Rowland, and W. Dabney. Understanding and preventing capacity loss in reinforcement learning. In ICLR , 2022.

Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. A. Riedmiller, A. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforcement learning. Nature , 518(7540):529-533, 2015.

Nikishin, M. Schwarzer, P. D'Oro, P. Bacon, and A. C. Courville. The primacy bias in deep reinforcement learning. In ICML , volume 162 of Proceedings of Machine Learning Research , pages 16828-16847, 2022.

OpenAI. Chatgpt: Optimizing language models for dialogue, 2022. URL https://openai.com/ blog/chatgpt/ .

Schaul, A. Barreto, J. Quan, and G. Ostrovski. The phenomenon of policy churn. arXiv preprint, arXiv:2206.00730 , 2022.

Jan Schneider, Pierre Schumacher, Simon Guist, Le Chen, Daniel F. B. Häufle, Bernhard Schölkopf, and Dieter Büchler. Identifying policy gradient subspaces. arXiv preprint, arXiv:2401.06604 , 2024.

Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Shah, B. Osinski, B. Ichter, and S. Levine. Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action. arXiv preprint, arXiv:2207.04429 , 2022.

Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. A. Riedmiller. Deterministic policy gradient algorithms. In ICML , pages 387-395, 2014.

Sokar, R. Agarwal, P. S. Castro, and U. Evci. The dormant neuron phenomenon in deep reinforcement learning. arXiv preprint, arXiv:2302.12902 , 2023.

S. Sutton and A. G. Barto. Reinforcement learning: An introduction. IEEE Transactions on Neural Networks , 16:285-286, 1988.

Tang, Z. Meng, J. Hao, C. Chen, D. Graves, D. Li, C. Yu, H. Mao, W. Liu, Y. Yang, W. Tao, and L. Wang. What about inputting policy in value function: Policy representation and policy-extended value function approximator. In AAAI , pages 8441-8449, 2022a.

Hongyao Tang and Glen Berseth. Improving deep reinforcement learning by reducing the chain effect of value and policy churn. arXiv preprint, arXiv:2409.04792 , 2024.

Tang, Z. D. Guo, P. H. Richemond, B. Á. Pires, Y. Chandak, R. Munos, M. Rowland, M. G. Azar, C. Le Lan, C. Lyle, A. György, S. Thakoor, W. Dabney, B. Piot, D. Calandriello, and M. Valko. Understanding self-predictive learning for reinforcement learning. arXiv preprint, arXiv:2212.03319 , 2022b.

Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. de Las Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq, T. P. Lillicrap, and M. A. Riedmiller. Deepmind control suite. arXiv preprint, arXiv:1801.00690 , 2018.

Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double qlearning. In AAAI , 2016.

Yang, G. Zhang, Z. Xu, and D. Katabi. Harnessing structures for value-based planning and reinforcement learning. In ICLR , 2020.

Kenny Young and Tian Tian. Minatar: An atari-inspired testbed for more efficient reinforcement learning experiments. arXiv preprint , arXiv:1903.03176, 2019.

A Limitations

In this paper, we only provide empirical investigations on the phenomena we present in Sec. 3. Although we observe relatively consistent results for TD3 and RAD across a variety of MuJoCo and DMC environments, which demonstrates the generality to some degree, we have no theoretical support for the observed phenomena at present. A rudimentary thought on this point is to study the learning dynamics of ∇ θ J ( π θ ) , especially of ∇ a Q ϕ ( s, a ) . This is because the policy path is the accumulation of policy gradient while policy gradients are significantly determined by the landscape of Q -network, which learns concurrently along the process. We believe that recent studies on the learning dynamics of the value function and representation [Lyle et al., 2021, 2022] can be inspiring references.

Still from the empirical perspective, our work is not complete in the sense that we only consider TD3 and RAD (i.e., SAC inside). On-policy policy-based DRL algorithms like PPO [Schulman et al., 2017] may have a very different policy path. Besides, the empirical investigation for more value-based DRL algorithms beyond DQN [Mnih et al., 2015] is missing in this paper. For typical value-based DRL algorithms, we can also treat the value network as a special form of policy and perform PPTB in the same way. Although we provided some preliminary results of applying PPTB to DoubleDQN in our experiments, this is kind of reckless since the natures of policy and value function are different, nor are their learning dynamics. A recent phenomenon called Policy Churn discovered in [Schaul et al., 2022, Tang and Berseth, 2024] reveals that the greedy policy induced by a typical value network changes its actions on about 10% states after one update. We leave in-depth studies on the value function path in the future.

For methodology, we propose very simple methods and we believe that there is great potential for more sophisticated methods to be proposed. First, we use standard SVD as the main tool in both our empirical investigation and our methods. We use no acceleration for SVD or other more advanced alternatives to obtain the major and minor directions of the policy path. Moreover, we use fixed dimensionalities and intervals for policy path trimming and boosting. This paper contains no attempts to design adaptive approaches or propose principled algorithms (which may rely on the advances in theoretical results). In addition, smarter hyperparameter selection methods are expected. One another limitation is that our proposed method is not evaluated in sparse-reward environments.

B More Discussion

Policy Path Trimming in the View of Primacy Bias The policy path trimming (PPT) method proposed in this work can be viewed as a special way to do network resetting to alleviate the primacy bias [Nikishin et al., 2022]. The resetting effect of PPT in the minor update directions is achieved by (1) obtaining the transformed parameter space via performing temporal SVD with the historical policies collected in a sliding window, (2) and then (re-)setting (or dropping) the left singular vector value (i.e., u ) to zero. More concretely, this differs from the resetting method in [Nikishin et al., 2022] at two points:

Resetting to random initialization v.s., Resetting to Zero in the transformed parameter space : The resetting method in [Nikishin et al., 2022] resets the network parameters to a set of randomly (re-)initialized parameters. In contrast, our method resets the left singular vector value (i.e., u ) to zero for the minor temporal SVD directions, i.e., resetting the parameters to zero in the transformed parameter space. One thing to note is that resetting the left singular vector value (i.e., u ) to zero does not mean that the network parameters (in the original parameter space) are necessarily zero, because the basis of space is different. A more in-depth analysis of the correlation between them is worth further study in the future.

Global resetting v.s., Local resetting : The resetting method in [Nikishin et al., 2022] is global, as the network parameters are reset to a set of randomly (re-)initialized parameters. Differently, we do resetting according to the temporal SVD parameter space based on a sliding window that includes recent historical policies (thus being local). In this sense, this is similar to Shrink-and-Perturb [Ash and Adams, 2020], which can be viewed as a soft version of the resetting method, that shrinks the network parameters and adds random parameters with a coefficient.

Moreover, from the perspective of plasticity, our method periodically rolls back the plasticity of the network in the directions that are orthogonal to the ones that represent the effective knowledge learned so far.

The Connection between PPTB and Momentum-based Optimization The idea of momentum adopted in modern optimizers is essentially to make use of the information of historical gradients. In this sense, we can establish a connection between momentum and PPTB proposed in our work as the key idea of PPTB is to make use of the latent structure of the historical policies along the policy update path.

Momentum-based optimizers use different kinds of aggregation quantities of the historical gradients (e.g., m t +1 = βm t + ∇ t ). In our context, PPTB differs at two points: (1) not all the temporal SVD directions are used, and directions with large oscillations are trimmed (note that both SVD and eigendecomposition are techniques that capture the features of curvature); (2) policy path boosting can be viewed as a type of momentum conducted in the major temporal SVD directions. Another difference to notice is that, both our analysis and experimental evaluation are done with Adam which already leverages momentum (except for the analysis for SGD). Thus, our findings are based on the practical optimization of a momentum-based optimizer.

We believe that a further study on this connection with formal description is worthwhile in the future.

C Experiment Details

Hyperparameter Setting For the maintenance of the policy path, we use a FIFO buffer with the size of 2k and 1k for MuJoCo and DMC respectively, and save the policy parameters every 25 minibatch gradient updates of the policy network. For the hyperparameter of PPTB, the number of major directions to keep (i.e., r t in Sec. 4.1) is searched in the set { 8 , 16 , 32 , 64 } for each environment. For PPB, we simply always boost the first two SVD directions (i.e., u ∗ , 1 , u ∗ , 2 as described in Sec. 4.2) and choose the boosting coefficient p b from the set { 0 . 1 , 0 . 5 , 1 . 0 } .

For the experiments in MinAtar, we implemented DoubleDQN-PPTB based on the official code of MinAtar 2 by adding several lines of code to collect the historical network parameters of DoublDQN and perform PPTB at intervals through learning. We made no change to the default hyperparameters of DoubleDQN. For DoubleDQN-PPTB, we use r t = 64 and p b = 0 . 1 .

Compute Resource We use Nvidia 3080ti GPU and V100 GPU for our experiments. For instance, each training trial of MuJoCo experiment requires less than 4G memory and about 8G GPU memory.

Note that the main computation cost added by PPTB is the calculation of SVD. In practice, we perform PPBT at sparse intervals (e.g., t p ≥ 1000 ) for the policy parameter matrix with a size [2000 , ≈ 1 e 5] , and each SVD computation takes less than 1 second with torch.linalg.svd . The practical computation cost in wall-clock time is less than 5% of the total training time, which is worth it compared to the benefits SVD brings. Moreover, the temporal SVD operations remain scalable as matrix size increases, thanks to the use of efficient linear algebra libraries.

PyTorch Code Implementation of PPTB We provide a pytorch-like pseudocode of PPTB implementation in Algorithm 1, where the slices of vector and matrix are also in a pytorch-like style. In our practical implementation used by our experiments, we only make the modifications of about 10-line core codes in the official implementations of TD3 [Fujimoto et al., 2018] and RAD [Laskin et al., 2020].

Code implementation of Baseline Algorithms We build our experiment codes based on the official implementation on GitHub for both TD3 [Fujimoto et al., 2018] 3 and RAD [Laskin et al., 2020] 4 . We do not change the default algorithm implementation or the recommended hyperparameters.

D Additional Experimental Results

D.1 Empirical Investigation on Policy Parameter Change Amount

Further discussion about Phenomenon 1.1. One thing to note is that the results reported in Fig. 1 are based on the policy paths of entire learning process, thus being a global view. For a more local view (i.e., a window of recent learning process), the phenomenon of parameter change asymmetry becomes more obvious and more parameters have minor and even no change (i.e., dead parameters). We hypothesize that this could be some practical evidence of lazy training of neural network [Chizat et al., 2018]. Concretely, the phenomenon indicates that a lot of parameters have insignificant gradients ∇ a Q ( s, a ) ∇ θ π θ ( s ) | a = π ( s ) during learning process.

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Amount of accu. parameter change (1e-3)

0

20000

40000

60000

80000

Amount of elements in each bin

HalfCheetah-v4: Policy parameter change amount

Layer 1

Layer 2

Layer 3

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Amount of accu. parameter change (1e-3)

0.0

0.2

0.4

0.6

0.8

1.0

Percentage of elements in each bin

HalfCheetah-v4: Policy parameter change amount

0

50

100

150

200

250

300

Ratio of accu. and final parameter change amount

0

2500

5000

7500

10000

12500

15000

17500

20000

Amount of elements in each bin

HalfCheetah-v4: Ratio of accu. and final parameter change amount

0

50

100

150

200

250

300

Ratio of accu. and final parameter change amount

0.0

0.2

0.4

0.6

0.8

1.0

Percentage of elements in each bin

HalfCheetah-v4: Ratio of accu. and final parameter change amount

(a) HalfCheetah-v4

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Amount of accu. parameter change (1e-3)

0

10000

20000

30000

40000

50000

Amount of elements in each bin

Hopper-v4: Policy parameter change amount

Layer 1

Layer 2

Layer 3

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Amount of accu. parameter change (1e-3)

0.0

0.2

0.4

0.6

0.8

1.0

Percentage of elements in each bin

Hopper-v4: Policy parameter change amount

0

50

100

150

200

250

300

Ratio of accu. and final parameter change amount

0

2500

5000

7500

10000

12500

15000

17500

20000

Amount of elements in each bin

Hopper-v4: Ratio of accu. and final parameter change amount

0

50

100

150

200

250

300

Ratio of accu. and final parameter change amount

0.0

0.2

0.4

0.6

0.8

1.0

Percentage of elements in each bin

Hopper-v4: Ratio of accu. and final parameter change amount

(b) Hopper-v4

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Amount of accu. parameter change (1e-3)

0

20000

40000

60000

80000

Amount of elements in each bin

Walker2d-v4: Policy parameter change amount

Layer 1

Layer 2

Layer 3

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Amount of accu. parameter change (1e-3)

0.0

0.2

0.4

0.6

0.8

1.0

Percentage of elements in each bin

Walker2d-v4: Policy parameter change amount

0

50

100

150

200

250

300

Ratio of accu. and final parameter change amount

0

2500

5000

7500

10000

12500

15000

17500

20000

Amount of elements in each bin

Walker2d-v4: Ratio of accu. and final parameter change amount

0

50

100

150

200

250

300

Ratio of accu. and final parameter change amount

0.0

0.2

0.4

0.6

0.8

1.0

Percentage of elements in each bin

Walker2d-v4: Ratio of accu. and final parameter change amount

(c) Walker2d-v4

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

Amount of accu. parameter change (1e-3)

0

10000

20000

30000

40000

50000

60000

70000

80000

Amount of elements in each bin

Ant-v4: Policy parameter change amount

Layer 1

Layer 2

Layer 3

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

Amount of accu. parameter change (1e-3)

0.0

0.2

0.4

0.6

0.8

1.0

Percentage of elements in each bin

Ant-v4: Policy parameter change amount

0

50

100

150

200

250

300

Ratio of accu. and final parameter change amount

0

2500

5000

7500

10000

12500

15000

17500

20000

Amount of elements in each bin

Ant-v4: Ratio of accu. and final parameter change amount

0

50

100

150

200

250

300

Ratio of accu. and final parameter change amount

0.0

0.2

0.4

0.6

0.8

1.0

Percentage of elements in each bin

Ant-v4: Ratio of accu. and final parameter change amount

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Amount of accu. parameter change (1e-3)

0

20000

40000

60000

80000

100000

120000

140000

Amount of elements in each bin

Humanoid-v4: Policy parameter change amount

Layer 1

Layer 2

Layer 3

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Amount of accu. parameter change (1e-3)

0.0

0.2

0.4

0.6

0.8

1.0

Percentage of elements in each bin

Humanoid-v4: Policy parameter change amount

0

50

100

150

200

250

300

Ratio of accu. and final parameter change amount

0

5000

10000

15000

20000

25000

30000

35000

Amount of elements in each bin

Humanoid-v4: Ratio of accu. and final parameter change amount

0

50

100

150

200

250

300

Ratio of accu. and final parameter change amount

0.0

0.2

0.4

0.6

0.8

1.0

Percentage of elements in each bin

Humanoid-v4: Ratio of accu. and final parameter change amount

(e) Humanoid-v4

0.0

0.1

0.2

0.3

0.4

Amount of accu. parameter change (1e-3)

0

100000

200000

300000

400000

500000

Amount of elements in each bin

cartpole-swingup: Policy parameter change amount

Layer 1

Layer 2

Layer 3

0.0

0.1

0.2

0.3

0.4

Amount of accu. parameter change (1e-3)

0.0

0.2

0.4

0.6

0.8

1.0

Percentage of elements in each bin

cartpole-swingup: Policy parameter change amount

0

100

200

300

400

500

Ratio of accu. and final parameter change amount

0

200000

400000

600000

800000

Amount of elements in each bin

cartpole-swingup: Ratio of accu. and final parameter change amount

0

100

200

300

400

500

Ratio of accu. and final parameter change amount

0.0

0.2

0.4

0.6

0.8

1.0

Percentage of elements in each bin

cartpole-swingup: Ratio of accu. and final parameter change amount

(a) cartpole-swingup

0.0

0.2

0.4

0.6

0.8

Amount of accu. parameter change (1e-3)

0

100000

200000

300000

400000

500000

600000

700000

Amount of elements in each bin

finger-spin: Policy parameter change amount

Layer 1

Layer 2

Layer 3

0.0

0.2

0.4

0.6

0.8

Amount of accu. parameter change (1e-3)

0.0

0.2

0.4

0.6

0.8

1.0

Percentage of elements in each bin

finger-spin: Policy parameter change amount

0

100

200

300

400

500

Ratio of accu. and final parameter change amount

0

200000

400000

600000

800000

Amount of elements in each bin

finger-spin: Ratio of accu. and final parameter change amount

0

100

200

300

400

500

Ratio of accu. and final parameter change amount

0.0

0.2

0.4

0.6

0.8

1.0

Percentage of elements in each bin

finger-spin: Ratio of accu. and final parameter change amount

(b) finger-spin

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Amount of accu. parameter change (1e-3)

0

100000

200000

300000

400000

500000

600000

700000

800000

Amount of elements in each bin

walker-walk: Policy parameter change amount

Layer 1

Layer 2

Layer 3

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Amount of accu. parameter change (1e-3)

0.0

0.2

0.4

0.6

0.8

1.0

Percentage of elements in each bin

walker-walk: Policy parameter change amount

0

100

200

300

400

500

Ratio of accu. and final parameter change amount

0.0

0.2

0.4

0.6

0.8

1.0

Amount of elements in each bin

1e6

walker-walk: Ratio of accu. and final parameter change amount

0

100

200

300

400

500

Ratio of accu. and final parameter change amount

0.0

0.2

0.4

0.6

0.8

1.0

Percentage of elements in each bin

walker-walk: Ratio of accu. and final parameter change amount

(c) walker-walk

0.0

0.2

0.4

0.6

0.8

1.0

Amount of accu. parameter change (1e-3)

0

100000

200000

300000

400000

500000

Amount of elements in each bin

hopper-stand: Policy parameter change amount

Layer 1

Layer 2

Layer 3

0.0

0.2

0.4

0.6

0.8

1.0

Amount of accu. parameter change (1e-3)

0.0

0.2

0.4

0.6

0.8

1.0

Percentage of elements in each bin

hopper-stand: Policy parameter change amount

0

100

200

300

400

500

Ratio of accu. and final parameter change amount

0

200000

400000

600000

800000

Amount of elements in each bin

hopper-stand: Ratio of accu. and final parameter change amount

0

100

200

300

400

500

Ratio of accu. and final parameter change amount

0.0

0.2

0.4

0.6

0.8

1.0

Percentage of elements in each bin

hopper-stand: Ratio of accu. and final parameter change amount

(d) hopper-stand

0.0

0.5

1.0

1.5

2.0

Amount of accu. parameter change (1e-3)

0

100000

200000

300000

400000

500000

600000

700000

800000

Amount of elements in each bin

cheetah-run: Policy parameter change amount

Layer 1

Layer 2

Layer 3

0.0

0.5

1.0

1.5

2.0

Amount of accu. parameter change (1e-3)

0.0

0.2

0.4

0.6

0.8

1.0

Percentage of elements in each bin

cheetah-run: Policy parameter change amount

0

100

200

300

400

500

Ratio of accu. and final parameter change amount

0

200000

400000

600000

800000

Amount of elements in each bin

cheetah-run: Ratio of accu. and final parameter change amount

0

100

200

300

400

500

Ratio of accu. and final parameter change amount

0.0

0.2

0.4

0.6

0.8

1.0

Percentage of elements in each bin

cheetah-run: Ratio of accu. and final parameter change amount

(e) cheetah-run

D.2 Empirical Investigation on Policy Learning Path by Temporal SVD

0

200

400

600

800

Indices of singular values

0.5

0.6

0.7

0.8

0.9

1.0

Amount of acc. information amount

cartpole-swingup: SVD information of policy learning path

Layer:1, Period:1

Layer:3, Period:1

0

100

200

300

400

500

600

700

800

Indices of singular values

0.5

0.6

0.7

0.8

0.9

1.0

Amount of acc. information amount

cartpole-swingup: SVD information of policy learning path

Layer:1, Period:2

Layer:3, Period:2

0

100

200

300

400

500

600

700

800

Indices of singular values

0.5

0.6

0.7

0.8

0.9

1.0

Amount of acc. information amount

cartpole-swingup: SVD information of policy learning path

Layer:1, Period:3

Layer:3, Period:3

(a) cartpole-swingup

0

200

400

600

800

Indices of singular values

0.5

0.6

0.7

0.8

0.9

1.0

Amount of acc. information amount

finger-spin: SVD information of policy learning path

Layer:1, Period:1

Layer:3, Period:1

0

200

400

600

800

Indices of singular values

0.5

0.6

0.7

0.8

0.9

1.0

Amount of acc. information amount

finger-spin: SVD information of policy learning path

Layer:1, Period:2

Layer:3, Period:2

0

200

400

600

800

Indices of singular values

0.5

0.6

0.7

0.8

0.9

1.0

Amount of acc. information amount

finger-spin: SVD information of policy learning path

Layer:1, Period:3

Layer:3, Period:3

(b) finger-spin

0

200

400

600

800

Indices of singular values

0.5

0.6

0.7

0.8

0.9

1.0

Amount of acc. information amount

walker-walk: SVD information of policy learning path

Layer:1, Period:1

Layer:3, Period:1

0

200

400

600

800

Indices of singular values

0.5

0.6

0.7

0.8

0.9

1.0

Amount of acc. information amount

walker-walk: SVD information of policy learning path

Layer:1, Period:2

Layer:3, Period:2

0

200

400

600

800

Indices of singular values

0.5

0.6

0.7

0.8

0.9

1.0

Amount of acc. information amount

walker-walk: SVD information of policy learning path

Layer:1, Period:3

Layer:3, Period:3

(c) walker-walk

0

200

400

600

800

Indices of singular values

0.5

0.6

0.7

0.8

0.9

1.0

Amount of acc. information amount

hopper-stand: SVD information of policy learning path

Layer:1, Period:1

Layer:3, Period:1

0

200

400

600

800

Indices of singular values

0.5

0.6

0.7

0.8

0.9

1.0

Amount of acc. information amount

hopper-stand: SVD information of policy learning path

Layer:1, Period:2

Layer:3, Period:2

0

200

400

600

800

Indices of singular values

0.5

0.6

0.7

0.8

0.9

1.0

Amount of acc. information amount

hopper-stand: SVD information of policy learning path

Layer:1, Period:3

Layer:3, Period:3

(d) hopper-stand

0

200

400

600

800

Indices of singular values

0.5

0.6

0.7

0.8

0.9

1.0

Amount of acc. information amount

cheetah-run: SVD information of policy learning path

Layer:1, Period:1

Layer:3, Period:1

0

200

400

600

800

Indices of singular values

0.5

0.6

0.7

0.8

0.9

1.0

Amount of acc. information amount

cheetah-run: SVD information of policy learning path

Layer:1, Period:2

Layer:3, Period:2

0

200

400

600

800

Indices of singular values

0.5

0.6

0.7

0.8

0.9

1.0

Amount of acc. information amount

cheetah-run: SVD information of policy learning path

Layer:1, Period:3

Layer:3, Period:3

(e) cheetah-run

0

25

50

75

100

125

150

175

200

Timestep (x 5e3)

0

2000

4000

6000

8000

10000

Average Return

0

250

500

750

1000

1250

1500

1750

2000

Time index

-0.04

-0.03

-0.02

-0.01

0.00

0.01

0.02

0.03

Value of

u

[

k

]

k

:1, det:1.00

k

:2, det:2.86

k

:3, det:6.26

k

:4, det:5.61

k

:5, det:15.09

0

250

500

750

1000

1250

1500

1750

2000

Time index

-0.10

-0.05

0.00

0.05

0.10

Value of

u

[

k

]

HalfCheetah-v4-svd-raw-all-layer: SVD Singular Analysis

k

k

:100, det:145.27

k

:200, det:80.78

k

:500, det:1243.99

k

:900, det:4820.73

0

250

500

750

1000

1250

1500

1750

2000

Index

i

of singular values

500

1000

1500

2000

2500

3000

Detour ratio of

u

[

i

]

0

250

500

750

1000

1250

1500

1750

2000

Index

i

of singular values

0.000

0.025

0.050

0.075

0.100

0.125

0.150

0.175

0.200

Total

u

[

i

] diff

(a) HalfCheetah-v4

0

25

50

75

100

125

150

175

200

Timestep (x 5e3)

0

500

1000

1500

2000

2500

3000

3500

Average Return

0

250

500

750

1000

1250

1500

1750

2000

Time index

-0.03

-0.02

-0.01

0.00

0.01

0.02

0.03

Value of

u

[

k

]

k

:1, det:1.01

k

:2, det:2.76

k

:3, det:5.68

k

:4, det:5.94

k

:5, det:12.89

0

250

500

750

1000

1250

1500

1750

2000

Time index

-0.075

-0.050

-0.025

0.000

0.025

0.050

0.075

Value of

u

[

k

]

Hopper-v4-svd-raw-all-layer: SVD Singular Analysis

k

:50, det:61.12

k

:100, det:413.95

k

:200, det:302.60

k

:500, det:6191.30

k

:900, det:5484.34

0

250

500

750

1000

1250

1500

1750

2000

Index

i

of singular values

500

1000

1500

2000

2500

3000

Detour ratio of

u

[

i

]

0

250

500

750

1000

1250

1500

1750

2000

Index

i

of singular values

0.00

0.05

0.10

0.15

0.20

0.25

Total

u

[

i

] diff

(b) Hopper-v4

0

25

50

75

100

125

150

175

200

Timestep (x 5e3)

0

500

1000

1500

2000

2500

3000

3500

Average Return

0

250

500

750

1000

1250

1500

1750

2000

Time index

-0.04

-0.03

-0.02

-0.01

0.00

0.01

0.02

0.03

Value of

u

[

k

]

k

:1, det:1.01

k

:2, det:2.76

k

:3, det:5.98

k

:4, det:5.64

k

:5, det:13.06

0

250

500

750

1000

1250

1500

1750

2000

Time index

-0.075

-0.050

-0.025

0.000

0.025

0.050

0.075

Value of

u

[

k

]

Walker2d-v4-svd-raw-all-layer: SVD Singular Analysis

k

:50, det:34.94

k

:100, det:128.01

k

:200, det:581.98

k

:500, det:1006.59

k

:900, det:5085.69

0

250

500

750

1000

1250

1500

1750

2000

Index

i

of singular values

500

1000

1500

2000

2500

3000

Detour ratio of

u

[

i

]

0

250

500

750

1000

1250

1500

1750

2000

Index

i

of singular values

0.000

0.025

0.050

0.075

0.100

0.125

0.150

0.175

0.200

Total

u

[

i

] diff

(c) Walker2d-v4

0

25

50

75

100

125

150

175

200

Timestep (x 5e3)

500

1000

1500

2000

2500

3000

Average Return

0

250

500

750

1000

1250

1500

1750

2000

Time index

-0.04

-0.03

-0.02

-0.01

0.00

0.01

0.02

0.03

Value of

u

[

k

]

k

:1, det:1.02

k

:2, det:2.68

k

:3, det:5.29

k

:4, det:5.24

k

:5, det:12.32

0

250

500

750

1000

1250

1500

1750

2000

Time index

-0.075

-0.050

-0.025

0.000

0.025

0.050

0.075

Value of

u

[

k

]

Ant-v4-svd-raw-all-layer: SVD Singular Analysis

k

:50, det:31.29

k

:100, det:87.96

k

:200, det:310.83

k

:500, det:1998.12

k

:900, det:8157.07

0

250

500

750

1000

1250

1500

1750

2000

Index

i

of singular values

500

1000

1500

2000

2500

3000

Detour ratio of

u

[

i

]

0

250

500

750

1000

1250

1500

1750

2000

Index

i

of singular values

0.00

0.05

0.10

0.15

0.20

Total

u

[

i

] diff

cartpole-swingup-svd-raw: SVD Singular Analysis

0

10

20

30

40

50

Timestep (x 2e4)

0

200

400

600

800

1000

Average Return

0

200

400

600

800

1000

Time index

-0.04

-0.02

0.00

0.02

0.04

Value of

u

[

k

]

k

:1, det:1.00

k

:2, det:1.00

k

:3, det:26.07

k

:4, det:3.13

k

:5, det:38.88

0

200

400

600

800

1000

Time index

-0.15

-0.10

-0.05

0.00

0.05

0.10

0.15

Value of

u

[

k

]

k

:50, det:98.28

k

:100, det:246.05

k

:200, det:828.31

k

:500, det:8732.38

k

:900, det:2349.42

0

200

400

600

800

1000

Index

i

of singular values

500

1000

1500

2000

2500

3000

Detour ratio of

u

[

i

]

0

200

400

600

800

1000

Index

i

of singular values

0.000

0.025

0.050

0.075

0.100

0.125

0.150

0.175

Total

u

[

i

] diff

(a) cartpole-swingup

0

10

20

30

40

50

Timestep (x 2e4)

0

200

400

600

800

1000

Average Return

0

200

400

600

800

1000

Time index

-0.04

-0.02

0.00

0.02

0.04

Value of

u

[

k

]

k

:1, det:1.00

k

:2, det:1.03

k

:3, det:1048.23

k

:4, det:3.02

k

:5, det:164.19

0

200

400

600

800

1000

Time index

-0.10

-0.05

0.00

0.05

0.10

Value of

u

[

k

]

walker-walk-svd-raw: SVD Singular Analysis

k

:50, det:74.35

k

:100, det:1588.96

k

:200, det:291.43

k

:500, det:860.40

k

:900, det:6780.46

0

200

400

600

800

1000

Index

i

of singular values

500

1000

1500

2000

2500

3000

Detour ratio of

u

[

i

]

0

200

400

600

800

1000

Index

i

of singular values

0.000

0.025

0.050

0.075

0.100

0.125

0.150

0.175

Total

u

[

i

] diff

(b) walker-walk

D.3 Empirical Investigation on Temporal SVD Reconstruction of DRL Policies

(a) Ant-v4

(b) Walker2d-v4

(c) Hopper-v4

(d) HalfCheetah-v4

= TD3-PPTB

EG1

8168-(01 = [01 =

TD3 = TD3-PPTB

81dd-001 = 001 =

5000

4000

3000

10000

4000

3000

2500

8000

3000

2000

6000

2000

2000

1500

4000

1000

1000

1000

500

2000

200k

400k

600k

800k

Step

1M

0

200k

400k

600k

800k

global_step

200k

400k

600k

800k

global_step

0

0

200k

400k

600k

800k

Step

1M

1000

finger-spin

1000

walker-walk

1000

cartpole-swingup

1000

cheetah-run

900

900

900

900

800

800

800

800

Return

700

700

700

700

600

Re

600

Re

600

600

Average

500

Average

500

Average

500

Average

D

500

400

400

400

400

300

300

300

300

200

100

RAD

RAD-PPTB

200

100

RAD

RAD-PPTB

100

200

RAD

RAD-PPTB

100

200

RAD

RAD-PPTB

0

0.0

0.2

Time Steps (1e6)

0.4

0.6

0.8

1.0

1e6

0

0.0

0.2

Time Steps (1e6)

0.4

0.6

0.8

1.0

1e6

0-

0.0

0.2

Time Steps (1e6)

0.4

0.6

0.8

1.0

1e6

0

0.0

0.2

Time Steps (1e6)

0.4

0.6

0.8

1.0

1e6

(a) Parameter Analysis of BC (

Adam

)

(b) Temporal SVD Analysis of BC (

Adam

)

(c) Parameter SVD Analysis of BC (

SGD

)

medium-replay: Policy parameter change amount

medium-replay: Policy parameter change amount

medium-replay: Ratio of accu.and final parameter change amounthedium-replay: Ratio of accu.and final parameter change amount

Layer 1

Layer 2

1.0

1.0

Layer 3

1500

0.6

00052

9.6

0000

0.4

15000

专

5000

0.2

1000

0.0

Amount of accu.parameter change (1e-3）

0.5

0.0

Amount of accu. parameter change (1e-3)

ST

Ratio of accu.andfinal parameter change amount

Ratio of accu.and final parameter change amount

250

300

k:2, det:2.32

k:1, det:1.00

0.00-

k:50,det:29.13

k:100, det:68.92

scto

Value of u[k]

0.02

k:3, det:44.14

k:5, det:10.60

k:4, det:3.35

Value of u[k]

0.050

0.25

0.075

k:900.det:6522.56

k:200, det:1757.10

k:500,det:386.29

000

0.25-

0.00-

1500

000

0.02

0.025

1000

0.050

0.050

0.025

0.075

500

75010001250

Time index

150017502000

500

75010001250

Time index

15001750

250

500

Index i of singular values

750100012501500

17502000

250

Indexiof singular values

750100012501500

17502000

medium-replay: Policy parameter change amount

medium-replay: Policy parameter change amount

medium-replay: Ratio of accu.and final parameter change amountedium-replay:Ratio of accu.and final parameter change amount

Layer 1

.0

Amount of elements in each bin

Layer 2

80000

Layer 3

mount of elements ineach

6000

0.6

6000

0.6

4000

40000

20000

Amount of accu. parameter change (1e-3)

0.02

0.08

0.10

0.12

Amount of accu. parameter change (1e-3)

1.04

0.12

Ratio of accu. and final parameter change amount

20

0.0

Ratio of accu. and final parameter change amount

0.075

0500

2500

0C0

0.00

20

Value of u[k]

Value of u[k]

0.025

0.25

0.05

0.00-

1500

0.10

k:1, det:1.00

k:2, det:1.00

k:3, det:2.91

0.050

0.025

k:50,det:395.39

k:100,det:1935.35

k:200,det:1156.98

Jnox

1000

0.15-

0.10-

0.15

75010001250150017502000

Time index

k:5, det:3.05

k:4, det:1.66

0.075

250500

k:500, det:1234.75

k:900, det:2525.98

7501001250

Time index

150017502000

250

50075010001250150017502000

Index i of singular values

9.00

500

Index i of singular values

x010001201500

17502000

(d) Temporal SVD Analysis of BC (

SGD

)

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims in the abstract and introduction summarize our contributions and are supported by our experiment results in Section 3 and Section 5.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations in Section A.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide sufficient experiment and implementation details in Section 5 and Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Our code is available in https://github.com/bluecontra/PPTB . Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide necessary experimental details in Section 5 and Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The results in our paper are accompanied by error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the details of compute resources in Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This work conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper study policy network evolvement in a low-dimensional space, which can have a positive impact in general for better understanding the learning dynamics of DRL algorithms. No specific real-world application is concerned and we do not see a specific societal impact of this paper.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All the environments, data and codes of baseline methods used in this paper are publicly available on Github. We cited the original papers and provided the URLs to the assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide the necessary details for implementing our proposed method in this paper. We will provide a README document alongside the code to release after the review process.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper first introduce some common phenomena for TD3 and RAD agents, and then introduce a novel deep reinforcement learning algorithm which perform a novel temporal SVD along the policy learning path called Policy Path Trimming and Boosting (PPTB). This algorithm offers us an angle to view how policy evolves in a lower-dimensional space and how to utilizes it in reinforcement learning part. The paper is well written and novel.\n\nSTRENGTHS:\n1.The paper is well-written.\n\n2.The author methodically analyzes the experimental results using specific indicators (Accumulated Parameter Change, Final Parameter Change, Parameter Update Detour Ratio), making the paper very comprehensible.\n\nWEAKNESSES:\n1.Although most of the figures in the paper are very clear, some are unclear and hard to understand, particularly the last two figures in Figure 2(b).\n\n2.Some of the reported improvements in AUC appear to be incorrect. For instance, (189 ± 2 (103.33%)) and (148 ± 15 (81.48%)) seem to contain errors.\n\nQUESTIONS:\n1.Could the authors provide the learning curves of TD3 and TD3-PPTB? From my understanding, TD3-PPTB should only accelerate the training, I don't think TD3-PPTB would have better performance than TD3. I believe they should achieve the same converged result eventually.\n\n2.The author only provided results for 100k and 500k steps in the RAD results shown in Table 2. What about the results for 1000k or 2000k steps?\n\n3.Is the parameter P_b  fixed in each environment? Can you talk about how to determine the P_b in your experiment? If is not set properly, will it cause excessive oscillation during the training process? Additionally, in the experiment details, a 2000 or 1e5-dimensional neural network is too large for Mujoco training. Most papers indicate that commonly used model dimensions are 64 or 128.\n\nLIMITATIONS:\n1. This paper does not explain from a theoretical perspective why this approach would accelerate convergence.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the evolving path of policy network parameters in deep reinforcement learning (DRL). The author conducts experiments on multiple tasks in MuJoCo using TD3 and on multiple tasks in DMC using RAD. The findings reveal significant discrepancies in the amount of change among policy parameters and severe detours in policy parameter updates. To address this, the author employs Temporal SVD to decompose the evolving path of policy parameters. Despite the large number of parameters, the learning dynamics of the policy network are found to be concentrated in a few primary directions, forming a low-dimensional space.\n\nBased on these insights, the author proposed a new method called Policy Path Trimming and Boosting (PPTB). This method improves the performance of DRL algorithms by canceling updates in secondary parameter directions and boosting progress in primary directions. Experiments conducted on TD3 and RAD agents in MuJoCo and DMC environments demonstrate that the PPTB method significantly outperformed the original methods in terms of score and AUC evaluation metrics, thereby substantially enhancing the performance of the DRL algorithm.\n\nThe author investigates the dynamics of policy network parameters from a novel perspective of Temporal SVD, and proposes an innovative method PPTB to enhance the performance of the DRL algorithm. The writing style of this paper is clear and easy to understand, with rigorous logic, and it is of great significance to algorithm optimization in the field of DRL.\n\nSTRENGTHS:\nThe author investigates the dynamics of policy network parameters from a novel perspective of Temporal SVD, and proposes an innovative method PPTB to enhance the performance of the DRL algorithm. The writing style of this paper is clear and easy to understand, with rigorous logic, and it is of great significance to algorithm optimization in the field of DRL.\n\nWEAKNESSES:\nThe article does not clearly explain why only TD3 and RAD algorithms were selected. Wouldn't other algorithms have similar phenomena when updating strategies?\n\nThe PPTB experiment was only conducted on TD3 and RAD, and did not reveal the universality of PPTB on other DRL algorithms.\n\nQUESTIONS:\n1. Wouldn't other DRL algorithms have similar phenomena when updating strategies?\n\n2. Does PPTB work with DRL methods other than TD3 and RAD?\n\n3. Can the performance of PPTB be tested in a deeper network?\n\n4. Can the performance of PPTB algorithm be tested in more experimental environments or different tasks? Are there more evaluation metrics besides score and AUC?\n\nLIMITATIONS:\nThe author should enrich the experimental details and experimental content to demonstrate the versatility of the PPTB algorithm, or test the performance of the algorithm in more practical tasks. The article should appropriately discuss the limitations of the proposed algorithm, or provide some prospects for future improvements.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nOff-policy actor-critic Deep RL has unstable and seemingly oscillatory learning dynamics, which are poorly understood. This paper looks closely at the trajectories taken by policy networks during training. An SVD analysis, performed over sequences of policy parameter snapshots, reveals near-monotonic parameter evolution along directions corresponding to the few dominant singular values, and--- indeed---oscillatory behaviour along the minor directions.\nThe authors propose an intuitive and mathematically sound way to remedy this pathology by only permitting the parameters to move in subspace spanned by a small number of the top directions, while retaining the policy performance. Additionally, by taking larger steps in the first two major directions, they are able to train more performant policies.\n\nSTRENGTHS:\n**Significance**\n\nThe paper addresses an important problem. Deep RL researchers have long observed poor training dynamics in off-policy learning. These include divergence, performance collapses, and oscillatory behaviour (i.e. policies forgetting what was learnt, then recovering). The reasons behind these are poorly understood. While the paper does not quite advance our understanding of the problem, it does potentially identify some mechanistic signatures of the problem (oscillatory weight dynamics) and proposes a treatment for this symptom (suppressing movement in oscillation-afflicted subspaces). I think the proposed PPTB fix is unlikely to be a complete solution, mainly because the critic's behaviour is not yet studied, but it can be a stepping stone towards better behaved learning dynamics. \n\n**Clarity**\n\nThe paper is very clearly written, and I enjoyed reading it. Policy churn and oscillatory dynamics potentially stem from the combined actor-critic training dynamics, but the authors focus exclusively on the policy here. This is good, as it has kept the study focused and revealed interesting phenomena. \n\n**Originality**\n\nI am unaware of prior work which empirically inspects the policy learning dynamics by actually plotting out the weights' evolution, as the authors have done.\n\n**Quality**\n\nThe idea of using temporal SVD upon the policy weights is quite sensible, and a treatment for the problem immediately pops out of the same tool; this fix also appears to be easily implementable in code, and seems to grant substantial performance boosts.\n\nWEAKNESSES:\n1. **Limited analysis.** The analysis is focused on TD3 and RAD (based on SAC here, which itself is quite close to TD3). I think this is insufficient. Seasoned RL practitioners will note that DDPG exhibits markedly more policy oscillations/collapses/recoveries than TD3: the introduction of the Clipped Double Q trick already substantially mitigates those effects. In this empirical study, I worry that by not including DDPG --- the simplest possible baseline --- the authors are not observing the problematic dynamics in their full glory, and perhaps not testing the full potential of PPTB. Right now, there is a risk that your observed phenomena are quirks of TD3-lineage methods.\n\n2. **Related work.** The observation that gradient descent moves the parameters along a few dominant directions, chiefly in a low-dimensional subspace, is not new. There is a body of deep learning literature around this phenomenon, which the authors don't cite right now. Here is one such paper, and there are more:\n - \"Gradient descent happens in a tiny subspace\" G Gur-Ari, DA Roberts, E Dyer \n\n3. **Potentially limited significance.** Following from the above point: the existence of dominant directions is unsurprising. The existence of oscillatory detours in the minor directions *could* be a novel finding regarding actor-critic RL, but we don't know that yet. To strengthen the paper, here's a test: do you also observe the same harmonics in basic supervised learning? (e.g. when training the actor network with an MSE behaviour cloning loss). I think this is a particularly important thing to investigate, because I'm concerned that it could change the story of your paper.\n\n4. **Lack of training curve comparisons.** For a method that expressly tries to curb oscillations, it is important that we see how the actual training curves look, instead of just a table with scores.\n\n5. **Limited evaluations.** 6 is not enough; it's common to report a minimum of 10 independent trials. Also: you have `Humanoid-v4` experiments in the paper, but don't report scores with PPTB on that env.\n\nQUESTIONS:\n- Line `434`: you say that you choose the boosting coefficient $p_b$ from a set of values. Do you mean that you search from this set for a good hyperpameter value, or do you randomly sample from this set at each training iteration?\n\nLIMITATIONS:\nThe limitations section is well written, but one thing that isn't mentioned is that the method can struggle to scale up to larger network sizes (beyond the smallish networks used in this paper).\n\nOverall, I find this paper exciting, and I'm willing to substantially improve my rating if the authors address all the listed weaknesses (most crucially, number `3`).", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the evolution of parameters over time during policy optimization with TD3 and RAD. By analysing the SVD of a matrix containing parameters over time, the authors find that there are a few directions in which the parameters move consistently and many with more oscillations. Using this insight, an algorithm is proposed which boosts updates in the top estimated directions and reduces updates in the uninmportant ones. Evaluations on standard deep RL benchmark tasks demonstrate the utility of the approach.\n\nSTRENGTHS:\nThis paper uncovers some nice insights into the training process of deep RL agents. \nAs far as I know, I have not seen the SVD used to study parameter evolution over time, and I find it is an interesting application of that tool. The proposed algorithm is conceptually simple and easy to implement, making it easy to add to a variety of existing algorithms.\nIn terms of impact, the identified phenomena may extend past RL and could be true of deep neural network training more generally, potentially giving widespread impact. \n\nThe paper is well-organized, with the different sections flowing nicely into each other. Generally, the paper was easy to follow and experiments chosen appropriately to make the intended arugments.\n\nWEAKNESSES:\nMy main concern is about the evaluation of the algorithms. \nFor example, reporting the max return in the evaluation of algorithm or using the standard deviation across runs, whereas the standard error or boostrapped confidence intervals would be more appropriate. (see Questions)\n\nAlso, the following design choice is confusing to me:\n- Fig. 1 caption. \"Only\tupper 80% values according to $\\nabla^{apc}_j$ are taken to plot...for meaningful analysis\" \n\tCould you elaborate why this decision was made? Does it have to do with many of the values being close to zero?\n\nThe improvement in the performance is generally fairly modest. I think this is fine given that main contribution to me is identifying behaviours in the parameter evolution. I think expanding a bit more on the analysis could be interesting. \n\nIn terms of clarity, some details could be expanded upon more in the main text. See Questions section.\n\nQUESTIONS:\n_Clarification questions_\n\n\n- Line 157: What is $\\alpha_k$? Is it Singular Value Information Amount? \n\n\n- Fig 2 b) 4th and 5th figure from left. Interesting findings. So basically, the detour ratio is smaller for the \nAgain, I wonder if that has more to do with noise or curvature.\nPerhaps there is less noise in these directions, \n\n\t- In the 3rd fig from the left, it's hard to see much since the black curve masks everything else. Consider using different colors or some transparency. It would be nice to see how the paths get increasingly noisy.\n\n- Policy Path Trimming: To clarify the algorithm, does it project the current parameters into the space identified by the top singular vectors? \n\tThe description of the algorithm could be improved in Sec. 4.1. where it is introduced. While the intuition is described, the exact mechanism that is implemented is not explained.\n\n- Similarly, in section 4.2, the Policy Path Boosting could be described a bit more clearly. Eq. 2 in particular is a bit confusing since  \n$\\hat{u}_{n,*}$ is updated but then only \n\n$\\hat{u}_{n,i}$ $(i=1,2)$ are actually used. \n\nThe phrase in line 241 \"PPB modifies $\\theta_n$ by increasing $u_{n,1}, u_{n,2}$ along the temporal direction...\" sounds  overly complicated. Perhaps rephrasing it to something like \"PPB moves the parameter further in the direction of previous updates along the first two main directions $u_{n,1}$ and $u_{n,2}$\" could be simpler. \n\n_Suggestions and broader questions_\n- By constraining the parameter evolution path to focus on the previous main directions, would the effect of \"primacy bias\" or related phenomena be even stronger? Could we be losing out on performance due to prematurely committing to certain update directions? \n\n- Line 270. The evaluation metric \"SCORE\" should no longer be used since the maximum over runs introduces overestimation bias and leads to less reliable estimatse. See [1] for better evaluation practices and [2] for arguments against using the max. \n\n- I wonder if the detours are mainly due to noise or curvature of the objective. One way to test this would be to increase or decrease the minibatch size, which can control the variance of the updates. Then, by inspecting the effect on the detour ratio or cumulative parameter movement, we could guess the relative impact of noise and curvature. \n\n- Fig.1 is a CDF plot. I think a histogram or a box plot might be easier to interpret than CDF plots since you need to look at differences in a CDF plot to identify where most of the probability mass is.\n\n- An ablation study for the two components of the algorithm would be a valuable addition. It is not clear if both pieces are necessary right now or how important they are.\n\n- I would also be curious to know what would happen if we took the estimated space \n\n- Here is a paper that could be interested to read [3] where the authors show that, even if you constrain neural network parameters to a random subspace, as long as the dimension of that subspace is not too small, you can recover the same performance as the original network. \nThe idea of Policy Path Trimming could be interpreted as a more intelligent approach which estimates the constrained subspace instead of choosing a random one. \n\n- Another line of research (e.g. see [4]) has observed that the Hessian contains only a few large eigenvalues in neural network training, which may be related to the ideas discussed in the paper since there would be some interplay between the curvature of the loss surface and directions of updates over time. \n\n- In the matrix of parameters over time, consider using $t \\in {1,...,T}$ to index the rows instead of $n$ so it's a little easier for the reader to remember which dimension corresponds to what. Alternatively, if $t$ is reserved for environment timesptes, $\\tau$ could be used as a substitute. \n\n\n\n[1] \"Deep Reinforcement Learning at the Edge of the Statistical Precipice\" Agarwal et al.\n\n[2] \"Deep Reinforcement Learning that Matters\" Henderson et al. \n\n[3] \"Measuring the Intrinsic Dimension of Objective Landscapes\" Li et al. \n\n[4] \"An Investigation into Neural Net Optimization via Hessian Eigenvalue Density\" Ghorbani et al.\n\nLIMITATIONS:\nThese are discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors examine the trajectories of policy learning in continuous control reinforcement learning tasks. \n\nThey begin by measuring how directly parameters go to their destination and observe large detours and differing update behaviour for different layers. They then examine the singular value decomposition of the different training checkpoints and observe some strong common update trends. \n\nThey propose Policy Path Trimming and Boosting (PPTB), where policy boosting is boosting the gradients towards the strongest singular values, and trimming trims the updates from the smaller singular values.\n\nSTRENGTHS:\n* The paper describes their method and analysis well. I generally found it easy to read and understand what points they were trying to convey.\n* The proposed method does seem to improve performance on the tasks presented.\n\nWEAKNESSES:\n* The environments and methods investigated, given the authors resutls are purely empirical, are not diverse enough to draw any real conclusions. The authors focus on continuous control environments without considering discrete environments such as Atari, or more diverse network architectures such as  recurrent networks or transformers, or different methods such as PPO or DQN. These results hold only for MLPs or CNNs trained on continuous control tasks, which just isn't a convincing enough setting to warrant acceptance.\n* The proposed method does not seem particularly practical. Computing SVD is both time intensive and requires storing a wide range of previous parameters. It therefore requires a lot more compute, which is why such methods are typically not used. Compared to approximations such as momentum-based optimisation [1], which are also aimed at achieving more uniform convergence, this time among eigenvalues of the data matrix. Although their method improves performance, they have not convincingly demonstrated that the compute couldn't be better used elsewhere, for example by training a bigger network or by sweeping hyperparameters more effectively.\n\n\n\n[1] Goh, \"Why Momentum Really Works\", Distill, 2017. http://doi.org/10.23915/distill.00006\n\nQUESTIONS:\n* Can you comment more on the relationship between your work and other, less compute intensive methods, that aim to allow more uniform convergence among the different data eigenvalues such as momentum? Is there an explicit connection here? Have you thought about that?\n\nLIMITATIONS:\n* The authors, to their credit, provide an extensive discussion of the limitations of their work in Appendix A. I agree largely with the points in that section and enjoyed their contextualisation of their work there.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study how parameters evolve in Deep RL. They perform SVD on updates and find that parameters advance along a small number of directions. They then propose a method to trim the policy learning path by focusing the updates on these major directions. They show that their methods improve performance in MuJoCo and DMC.\n\nSTRENGTHS:\nThe work is a well-written exploration of an interesting and novel perspective on RL parameter updates. The authors are extremely thorough and clear with their investigation and show strong results on common benchmarks. Furthermore, their method seems very simple to implement, which is valuable to the community.\n\nWEAKNESSES:\nConcerns with the paper:\n\n1. The paper only investigates extremely dense reward settings. Intuitively, the conclusions of this paper should not apply to sparse(r) reward settings, which are arguably much more interesting in RL. (See Q1 below).\n\n2. The results of the paper do not seem like they would be at all specific to RL. I'm not convinced that the observed phenomenon is not just a simple byproduct of using an optimizer with momentum. It would be good to have results that show that this phenomenon does not occur or help in supervised learning tasks (where I would imagine there is significantly more literature on this topic), and/or that this phenomenon still occurs when studying agents trained with plain SGD.\n\n3. The significance of the results are unclear (see below) \n\nPossible improvements:\n\n1. It could be neat to replicate the empirical investigations when using your new method PPTB. Does PPTB actually address the issues presented and dampen parameter updates? \n\n2. The results are not easy to read. It is hard to tell which results are statistically significant (I would recommend using standard error!) and also plotting standard RL training curves with the appropriate error regions. As-is, it's very hard to tell whether this method actually helps or not. In particular, it seems as though the error regions often overlap in Table 2, for example.\n\n3. The writing is often vague. In the abstract, the authors write: \"we study how the policy networks of typical DRL agents evolve during the learning process by empirically investigating several kinds of temporal change for each policy parameter\". This vague sentence conveys little information. The authors also use the word \"asymmetry\" in the abstract without explaining what they mean (asymmetric with respect to what?).\n\nQUESTIONS:\n1. Do you think this would work equally well in sparse(r) reward settings? My intuition is that it would not, since the early parameter updates likely do not contain significant information about the reward. The continuous control tasks evaluated are *particularly dense*. Also, doesn't pruning the noisier directions harm exploration?\n\n2. Related to the above, why do you believe this method does not exacerbate issues of primacy bias mentioned in the paper? The related work section's first paragraph does not really compare and contrast to the prior works, just mentions them.\n\n3. Do you think these results apply beyond just RL? I see no reason why this phenomenon is RL-specific. \n\n4. Related to Q3: Doesn't the fact that these methods use momentum (e.g. Adam) make this phenomenon obviously true? Does this phenomenon persist when using plain SGD? What about when you observe the *gradient updates* as opposed to the *parameter updates*?\n\n5. Can you include the plots mentioned above? It would really help with my understanding of the paper.\n\nLIMITATIONS:\nThe authors address the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-21"}