Title: No Representation, No Trust: Connecting Representation, Collapse, and Trust Issues in PPO
Venue: NeurIPS 2024
Accepted: yes
Keywords: proximal policy optimization

plasticity loss

trust region

feature rank collapse

regularization
Abstract: Reinforcement learning (RL) is inherently rife with non-stationarity since the states and rewards the agent observes during training depend on its changing policy.
Therefore, networks in deep RL must be capable of adapting to new observations and fitting new targets.
However, previous works have observed that networks trained under non-stationarity exhibit an inability to continue learning, termed loss of plasticity, and eventually a collapse in performance.
For off-policy deep value-based RL methods, this phenomenon has been correlated with a decrease in representation rank and the ability to fit random targets, termed capacity loss.
Although this correlation has generally been attributed to neural network learning under non-stationarity, the connection to representation dynamics has not been carefully studied in on-policy policy optimization methods.
In this work, we empirically study representation dynamics in Proximal Policy Optimization (PPO) on the Atari and MuJoCo environments, revealing that PPO agents are also affected by feature rank deterioration and capacity loss.
We show that this is aggravated by stronger non-stationarity, ultimately driving the actor's performance to collapse, regardless of the performance of the critic.
We ask why the trust region, specific to methods like PPO, cannot alleviate or prevent the collapse and find a connection between representation collapse and the degradation of the trust region, one exacerbating the other.
Finally, we present Proximal Feature Optimization (PFO), a novel auxiliary loss that, along with other interventions, shows that regularizing the representation dynamics mitigates the performance collapse of PPO agents.
Code and run histories are available at https://github.com/CLAIRE-Labo/no-representation-no-trust.

No Representation, No Trust: Connecting Representation, Collapse, and Trust Issues in PPO

Abstract

Reinforcement learning (RL) is inherently rife with non-stationarity since the states and rewards the agent observes during training depend on its changing policy. Therefore, networks in deep RL must be capable of adapting to new observations and fitting new targets. However, previous works have observed that networks trained under non-stationarity exhibit an inability to continue learning, termed loss of plasticity, and eventually a collapse in performance. For off-policy deep value-based RL methods, this phenomenon has been correlated with a decrease in representation rank and the ability to fit random targets, termed capacity loss. Although this correlation has generally been attributed to neural network learning under non-stationarity, the connection to representation dynamics has not been carefully studied in on-policy policy optimization methods. In this work, we empirically study representation dynamics in Proximal Policy Optimization (PPO) on the Atari and MuJoCo environments, revealing that PPO agents are also affected by feature rank deterioration and capacity loss. We show that this is aggravated by stronger non-stationarity, ultimately driving the actor's performance to collapse, regardless of the performance of the critic. We ask why the trust region, specific to methods like PPO, cannot alleviate or prevent the collapse and find a connection between representation collapse and the degradation of the trust region, one exacerbating the other. Finally, we present Proximal Feature Optimization (PFO), a novel auxiliary loss that, along with other interventions, shows that regularizing the representation dynamics mitigates the performance collapse of PPO agents. Code and run histories are available at https://github.com/CLAIRE-Labo/no-representation-no-trust .

1 Introduction

Reinforcement learning (RL) agents are inherently subject to non-stationarity as the states and rewards they observe change during learning. Therefore, neural networks in deep RL must be capable of adapting to new inputs and fitting new targets. However, previous works have observed that networks trained under non-stationarity exhibit an inability to continue learning, termed loss of plasticity, and a collapse in performance (Dohare et al., 2021; Abbas et al., 2023; Kumar et al., 2023; Dohare et al., 2023a,b). Kumar et al. (2021); Lyle et al. (2022) connect this phenomenon to representation dynamics and show that value networks in off-policy value-based RL algorithms exhibit a decrease in the rank of their representations, termed feature rank collapse, and a decrease in their ability to regress to arbitrary targets, called capacity loss. Although this deterioration in representation is more generally attributed to neural networks trained under non-stationarity (Lyle et al., 2023), the connection to representation dynamics has been overlooked in on-policy policy optimization methods. In particular, Proximal Policy Optimization (PPO) (Schulman et al., 2017), one of the most popular policy optimization methods, makes several minibatch updates over non-stationary data, unlike vanilla policy gradient methods, and optimizes a surrogate loss that depends on a moving old

policy. This raises the question of how much PPO agents are impacted by the same representation degradation attributed to non-stationarity. Dohare et al. (2021, 2023a,b) have shown that PPO agents lose plasticity throughout training but have only diagnosed it as a collapse in performance or as an Adam optimization issue. Igl et al. (2021) have shown that non-stationarity affects the generalization of PPO agents (learning speed when training episodes are very different otherwise performance at test time on novel episodes) but not necessarily training, and no connection was made with the feature rank and capacity measures used in the recent value-based works. One crucial outstanding question is why the trust region embedded in methods like PPO is unable to prevent the deterioration in policy by constraining its update. To address these gaps, we present the following contributions:

We provide the first study of feature rank and capacity loss in on-policy policy optimization, revealing that PPO agents in the Arcade Learning Environment (Bellemare et al., 2013) and MuJoCo (Todorov et al., 2012) environments are subject to representation collapse.

Wedraw connections between representation collapse, performance collapse, and trust region issues in PPO, showing that PPO's clipping becomes ineffective under poor representations and fails to prevent performance collapse, which is irrecoverable due to loss of capacity. We further isolate the breakdown of the trust region in a theoretical setting.

We corroborate these connections by performing interventions that regularize nonstationarity and representations and result in a better trust region and mitigation of performance collapse, incidentally giving insights on sharing an actor-critic trunk.

Wepropose Proximal Feature Optimization (PFO), a new regularization on the representation of the policy that regularizes the change in pre-activations. PFO strengthens our analysis by addressing the representation issues and mitigating performance collapse.

We open source our code providing a comprehensive and reproducible codebase for studying representation dynamics in policy optimization and a large database of run histories with extensive logging for further investigation on this topic.

2 Background

Reinforcement Learning (Sutton & Barto, 2018) We formalize our RL setting with the finitehorizon undiscounted Markov decision process, describing the interaction between an agent and an environment with finite 2 sets of states S and actions A , and a reward function r : S × A × S → R . An initial state S 0 ∈ S is sampled from the environment, then at each time step t ∈ { 0 , . . . , t max -1 } , the agent observes the state S t ∈ S , picks an action A t ∈ A according to its policy π : S → ∆( A ) with probability π ( A t | S t ) , 3 observes the next state S t +1 ∈ S sampled from the environment and receives a reward R t +1 . = r ( S t , A t , S t +1 ) . We denote by G t . = ∑ t max -1 k = t R k +1 the return after the action at time step t . The goal of the agent is to maximize its expected return J ( π ) . = E π [ ∑ t max -1 t =0 R t +1 ] = E π [ G 0 ] over the induced random trajectories. We discuss the choice of this setting in Appendix A.1.

Actor-Critic Agent We consider on-policy deep actor-critic agents which train a policy (or actor) network π ( · ; θ ) also denoted π θ , and a value (or critic) network ˆ v ( · ; w ) that approximates the return of π θ at every state. At every training stage, the agent collects a batch of samples, called rollout, with its current policy π θ , and both networks are trained with gradient descent on this data. The critic is trained to minimize the Euclidean distance to an estimator of the returns (e.g., G t ). We use λ -returns computed with the Generalized Advantage Estimator (GAE) (Schulman et al., 2015b). The actor is trained with the Proximal Policy Optimization (PPO) (Schulman et al., 2017).

Proximal Policy Optimization PPO-Clip, the most popular variant of PPO algorithms (Schulman et al., 2017), optimizes the actor by repeatedly maximizing the objective in Equation 1 at each rollout.

The objective is defined for some small hyperparameter ϵ ; π old is the last π θ of the previous optimization phase, used to collect a training batch (rollout) after each optimization phase; Ψ t is an estimator of the advantage of π old (e.g., Ψ t = G t -ˆ v ( S t ; w old ) ); we use the GAE in our experiments. An optimization phase consists of maximizing the objective with minibatch gradient steps over multiple epochs on the training batch. We refer to PPO-Clip as PPO and provide a pseudocode in Algorithm 1.

Intuitively, PPO aims to maximize the policy advantage E π old [ ∑ t max -1 t =0 π θ ( A t | S t ) π old ( A t | S t ) Ψ t ] defined by Kakade & Langford (2002), which participates in a lower bound to the improvement of π θ given that it is close to π old (Schulman et al., 2015a, see Theorem 1). In this regard, a gradient step on L CLIP π old ( θ ) would increase (resp. decrease) the probability of actions at states yielding positive (resp. negative) advantage until the ratio between the policies for those actions reaches 1 + ϵ (resp. 1 -ϵ ) at which point the gradient at those samples becomes null. This is a heuristic to ensure a trust region that keeps policies close to each other, resulting in policy improvement.

Non-stationarity in deep RL and PPO The actor and the critic networks are both subject to nonstationarity in deep RL. As the agent improves, it visits different states, shifting the distribution of states which makes the networks' input distribution non-stationary. This also holds for the targets to fit the critic, which change as the returns of the policy change. Unlike vanilla policy gradient (Sutton et al., 1999), and A2C (Mnih et al., 2016), PPO's objective is optimized by performing multiple epochs of minibatch gradient descent on the current collected batch, potentially making the networks more likely to be impacted by previous training rollouts. In this sense, increasing the number of epochs in PPO can cause the agent to 'overfit' more to previous experience.

Feature rank As done in most works studying feature dynamics in deep RL (Lyle et al., 2022; Kumar et al., 2021), we refer to the activations of the last hidden layer of a network (the penultimate layer) as the features or representation learned by the network. On a batch of N samples, this gives a matrix of dimensions N × D denoted by Φ , where D < N is the width of the penultimate layer. Several measures of the rank of this matrix have been used to quantify the 'quality' of the representation (Kumar et al., 2021; Gulcehre et al., 2022; Lyle et al., 2022; Andriushchenko et al., 2023). Their absolute values differ significantly, but their dynamics are often correlated. We track all of the different rank metrics in our experiments, compare them in Appendix E, and use the approximate rank in our main figures for its connection to principal component analysis (PCA). Given a threshold δ ∈ R and the singular values ⟨ σ i (Φ) , . . . , σ D (Φ) ⟩ of Φ in decreasing order, the approximate rank of Φ is min k { ∑ k i =1 σ 2 i (Φ) ∑ D j =1 σ 2 j (Φ) > 1 -δ } which corresponds to the smallest dimension

of the subspace recovering (1 -δ )% of the variance of Φ . We use δ = 0 . 01 i.e. the reconstruction recovers 99% of the variance as done by Andriushchenko et al. (2023); Yang et al. (2020). We refer to this metric as feature rank with reference to the rank of the feature matrix when there is no ambiguity.

Capacity loss Target-fitting capacity (Lyle et al., 2022) is computed on checkpoints of a network undergoing some training to measure the evolution of its ability to fit some chosen target independent from its training. It is a concrete metric to evaluate plasticity. Given a fixed target (distribution over inputs and outputs) and a fixed optimization budget, a checkpoint's capacity loss is the loss from fitting the checkpoint to the target at the end of the optimization budget. Usually, the capacity of a deep RL agent is measured by its ability to fit the outputs of a model initialized randomly from the same distribution as the agent on a fixed rollout collected by this target random model (Lyle et al., 2022; Nikishin et al., 2023). We follow this practice. The data would in expectation be from the same distribution as the agent's initial checkpoint. To fit the critic, we use an L 2 loss on the outputs of the models. To fit the actor, we use a KL divergence between the target and the checkpoint (forward KL).

3 Deteriorating representations, collapse, and loss of trust

It is well-known that non-stationarity in deep RL can be a factor causing issues in representation learning. However, most of the observations have been made in value-based methods showing that value networks are prone to rank collapse, harming their expressivity, and in turn, the performance of the agent (Lyle et al., 2022; Kumar et al., 2022). Non-stationarity has been shown to impact PPO's generalization Igl et al. (2021) and performance in the long run or in a continual learning setting (Dohare et al., 2021, 2023a,b), but no evidence of representation deterioration was shown. Our motivation is to reuse the tools that showed that value-based methods are prone to representation collapse but in policy optimization methods for the first time. We focus on PPO for its popularity and its non-stationarity which is impacted and can be controlled by multi-epoch optimization.

Furthermore, a crucial question for PPO, compared to most value-based alternatives, is how the regularization implicit in PPO through its trust region interacts with representation and performance collapse. Intuitively it should prevent rapid degradation of the policy.

Experimental setup We begin our experiments by training PPO agents on the Arcade Learning Environment (ALE)(Bellemare et al., 2013) for pixel-based observations with discrete actions and on MuJoCo (Todorov et al., 2012) for continuous observations with continuous actions. To keep our experiments tractable, we choose the Atari-5 subset recommended by Aitchison et al. (2023) and add Gravitar to include at least one sparse-reward hard-exploration game from the taxonomy presented by Bellemare et al. (2016). For MuJoCo, we train on Ants, Half-Cheetahs, Humanoids, and Hoppers, which have varying complexity and observation and output sizes. We use the same model architectures and hyperparameters as popular implementations of PPO on ALE and MuJoCo (Raffin et al., 2021; Huang et al., 2022b); these are also the architectures and hyperparameters used by Schulman et al. (2017) in the original implementation of PPO; they do not include normalization layers. For MuJoCo we further adopt a parameterization of the output action distribution using a TanhNormal 4 with both its mean and variance depending on the state representation as done by Haarnoja et al. (2018); Andrychowicz et al. (2021). As we study the connection between performance and representation dynamics this is a more natural choice than using the commonly implemented state-independent variance which would be independent of representation dynamics. The ALE models use ReLU activations (Nair & Hinton, 2010) and the MuJoCo ones tanh; we also experiment with ReLU on MuJoCo. We use separate actor and critic models for both environments unless specified in Section 4. Details on the performance metrics and tables of all environment parameters, model architectures, and algorithm hyperparameters are presented in Appendix B. Observing that the previous findings on the feature dynamics of value-based approaches (Gulcehre et al., 2022; Lyle et al., 2022) apply to the critic of PPO as well since the loss function is the same, we focus on studying the feature dynamics of the actor unless stated otherwise in the text or figures.

We vary the number of epochs as a tool to control the effects of non-stationarity, which gives the agent a more significant number of optimization steps per rollout while not changing the optimal target it can reach due to clipping, as opposed to changing the value of ϵ in the trust region for example. 5 We keep the learning rate constant throughout training and use the same learning rate for all the epoch configurations. 6 To understand the feature dynamics, we measure different metrics that are proposed in the literature, including feature rank, number of dead neurons (Gulcehre et al., 2022), capacity loss (Lyle et al., 2022), and penultimate layer pre-activation norm. Previous work has monitored feature norm values as well (Abbas et al., 2023; Lyle et al., 2024); however, in our case, we found that as the neurons in the policy network die, the feature norm might be stable while the pre-activation norm blows up. All the metrics are computed on on-policy rollouts except for the capacity loss.

We run five seeds per hyperparameter configuration and report mean curves with min/max shaded regions unless specified otherwise. All curves, except for capacity loss, are smoothed using an exponentially weighted moving average with a coefficient of 0 . 05 .

3.1 PPO suffers from deteriorating representations

Deteriorating representation How do the representation metrics of a PPO agent such as the feature rank and the capacity loss , evolve during training? Are they subject to the same decline observed by Kumar et al. (2021); Lyle et al. (2022) in value-based methods? Does it affect performance?

As illustrated in Figure 1 with ALE/Phoenix as an example, we observe a consistent increase in the norm of the pre-activations of the feature layer of the policy network. Learning curves for all the ALE games and MuJoCo tasks considered can be found in Appendix D. The increase in feature norm is present in all the games/tasks considered in both environments, that is, with the two different model architectures and activation functions in the case of MuJoCo. We associate the rapid growth in the norm of the pre-activations with an eventual decline in the policy network's feature rank. We observe a rank decline in five out of six ALE games and seven out of eight MuJoCo tasks (four with ReLU and three with tanh). The same observations about the increasing norm of the pre-activations can be made about the critic network. However, its rank varies more with the sparsity of the reward: in

most environments, its rank experiences a significant deterioration after the policy's performance declines (not the policy's rank) and rewards become sparser, and in the sparse-reward game Gravitar, the critic's rank collapses before the policy. Furthermore, capacity loss is increasing for the critic, as observed in value-based plasticity studies (Lyle et al., 2022), and we also show that is the case for the actor, for which it explodes around rank collapse.

Worse consequences How does increasing the number of epochs per rollout to vary non-stationarity affect a PPO agent's representation? Does it degrade as observed in DQN and SAC agents when increasing the replay ratio (Nikishin et al., 2022; Kumar et al., 2022)?

Increasing the replay ratio in DQN and SAC deteriorates the agent's representation and, in turn, its performance (Kumar et al., 2022; D'Oro et al., 2023). This is commonly attributed to 'overfitting' to previous experience (Nikishin et al., 2022). Increasing the number of epochs in PPO is analogous, and a natural hypothesis is that this would accelerate the deterioration of the policy's representation. Figure 1 shows that increasing the number of epochs accelerates the increase of pre-activations norm and the decrease of the policy's feature rank. 7 In some cases, the rank eventually collapses, coinciding with the policy's performance collapse. We observe the performance collapse in three of the six ALE games and three of the four MuJoCo tasks.

0.0

0.5

1.0

Environment steps×10 8

0

2

4

6

8

Episode return

×10 3

ALE/Phoenix-v5

0.0

0.5

1.0

Environment steps×10 8

0

1

2

3

Feature rank policy (PCA)

×10 2

0.0

0.5

1.0

Environment steps×10 8

10 0

10 1

10 2

10 3

10 4

Norm preactivation policy

0.0

0.5

1.0

Environment steps×10 8

0

2

4

6

Capacity loss policy

0.0

0.5

1.0

Environment steps×10 8

0

1

2

3

4

Feature rank critic (PCA)

×10 1

0.0

0.5

1.0

Environment steps×10 8

0.0

0.5

1.0

1.5

2.0

Capacity loss critic

4 epochs

6 epochs

8 epochs

Characterizing the collapse The collapse we observe is distinct from the typical entropy collapse. Figure 2 shows that the policy reaches a high entropy. A high overall entropy can come from an average of high-entropy states with different action distributions or trivially from the same high-entropy distribution in all states. Our analysis reveals the latter, a zero policy variance across states. This corresponds to a collapsed representation where most neurons are inactive. 8 The output thus relies solely on the bias term, as linear weights act on a null feature vector, making actions near uniform across all states and collapsing performance on complex tasks.

0.00

0.25

0.50

0.75

1.00

Environment steps

×10 8

1.0

1.2

1.4

1.6

1.8

2.0

Entropy

ALE/Phoenix-v5

0.00

0.25

0.50

0.75

1.00

Environment steps

×10 8

0

1

2

3

4

5

Policy variance

0.00

0.25

0.50

0.75

1.00

Environment steps

×10 8

0

1

2

3

4

5

Dead neurons policy

×10 2

4 epochs

6 epochs

8 epochs

3.2 Collapsed representations create trust issues and unrecoverable loss

Intuitively, the heuristic trust region set by PPO-Clip should prevent sudden catastrophic changes and limit the rank collapse, which induces worse performance. However, empirically, it seems the trust region cannot mitigate the collapse. In this section, we seek to understand the interaction between the rank collapse and the trust region. We argue that as rank collapses, the clipping constraint becomes unreliable and unable to restrict learning. This is in line with previous works that have pointed out

that probability ratios during training can go beyond the clipping limits with PPO-Clip (Engstrom et al., 2020; Wang et al., 2020; Sun et al., 2022). We believe, however, that this behavior is systematic when rank collapses and does not merely happen occasionally.

Wang et al. (2020, Theorem 2) state that when the gradients of the unclipped samples align with the gradients of clipped samples, the clipped samples' ratios will have their probabilities continue to go beyond the clip limit. They claim this condition would hold in practice because of 'optimization tricks' or optimizer accumulated moments; however, there is no evidence that these factors induce the gradient alignment or that the alignment is present in practice. Our intuition is that representation degradation leads to alignment in the gradients and, therefore, a breakdown of the trust region constraint. This can create a snowball effect, preventing PPO-Clip from preventing representation collapse. We summarize this in two observations:

Loss of trust is extreme around poor representations The average of probability ratios outside the clipping limits (below 1 -ϵ in Figure 3) significantly diverges from the clipping limit around the collapse of the agent's representation. This gives one more reason why the PPO trust region can be violated. We isolate this in a toy setting and analyze it formally in the next section. We further show in Figure 4 scatter plots of the lowest average probability ratios in runs with their associated representation metrics. 9 We observe no significant correlation in the regions where the representation is rich (high rank, low pre-activation norm), but an apparent decrease of the average of probability ratios below 1 -ϵ is observed as the representation reaches poor values. Note that we characterize the collapsing regime by an extremely low rank, however, it is not straightforward to draw a line between low-rank representations beneficial for generalization and extremely low-rank representations causing aliasing as also acknowledged by Gulcehre et al. (2022), but for environments like Atari, our figures seem to draw the line at single-digit ranks, which can be related to the action space of dimension 8+.

0.0

0.5

1.0

Environment steps×10 8

0.00

0.25

0.50

0.75

1.00

1.25

1.50

Rank policy (PCA)

×10 2

0.0

0.5

1.0

Environment steps×10 8

0.0

0.2

0.4

0.6

0.8

1.0

Capacity loss policy

0.0

0.5

1.0

Environment steps×10 8

2

3

4

5

6

7

Episode return

×10 3

0.0

0.5

1.0

Environment steps×10 8

6.5

7.0

7.5

8.0

8.5

9.0

Avg of prob ratios < 1 -

ε

0.0

0.5

1.0

Environment steps×10 8

0

2

4

6

8

Policy variance

0.0

0.5

1.0

Environment steps×10 8

-1

0

1

2

3

4

5

PPO-Clip objective

ALE/NameThisGame-v5, 4 epochs

ALE/NameThisGame-v5, 6 epochs

ALE/NameThisGame-v5, 8 epochs

0

1

2

3

4

5

Dead neurons policy

×10 2

4.5

5.0

5.5

6.0

6.5

7.0

7.5

8.0

Avg of prob ratios < 1 -

ε

ALE/Phoenix-v5

Kendall: -0.11, Spearman: -0.16

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Feature rank policy (PCA)

×10 2

4.5

5.0

5.5

6.0

6.5

7.0

7.5

8.0

Avg of prob ratios < 1 -

ε

Kendall: 0.08, Spearman: 0.11

10 2

10 3

Feature preactivation norm

4.5

5.0

5.5

6.0

6.5

7.0

7.5

8.0

Avg of prob ratios < 1 -

ε

Kendall: -0.28, Spearman: -0.40

Loss of plasticity makes performance collapse unrecoverable The persistent decrease in performance overlaps with a monotonic decrease in policy variance and PPO objective. It appears that as

the policy loses its ability to distinguish state, it can also ascend the PPO objective less and less at each batch (recall: after collecting a batch, the loss starts around zero with a normalized advantage, and through minibatch updates, the clipped policy advantage is ascended). Intuitively, this is implied by a loss of plasticity or a collapse in entropy (no new actions to learn from). As seen in Section 2 the entropy does not collapse, and measuring the capacity loss in Figure 3 shows that the decrease in objective gain is associated with a significant increase in capacity loss, implying loss of plasticity.

Connecting the dots Hence, around collapse, the representation of the policy is getting so poor that it is impacting its ability to distinguish and act differently across states; the trust region cannot prevent this catastrophic change as it also breaks down with a poor representation; finally, the policy's plasticity is also becoming so poor that the agent cannot recover by optimizing the surrogate objective.

Implications and discussion The causal connection we draw between the representation dynamics, the trust region, and the performance primarily holds around the collapse regime and not necessarily throughout training. However, this does not mean that one should only be concerned about the link when performance is starting to deteriorate. The representations don't collapse all of a sudden; they deteriorate throughout training until they reach collapse. Thus, mitigating representation degradation should happen throughout training and not only when around the collapsing regime. In addition, the connection gives important insights into the failure mode of the popular PPO-Clip algorithm, whose trust region is highly dependent on the representation quality, and more generally about trust-region methods which only constrain the output probabilities.

3.2.1 A toy setting to understand the effects of rank collapse on trust region

We present a toy example that illustrates how a collapsed representation bypasses the clipping set by PPO and cannot satisfy the trust region it seeks to set. PPO constructs a trust region around the policy π θ ( ·| s ) of the agent evaluated at a given state s , enforcing (in an approximate way) that the update computed on state s can not move the policy π θ ( ·| s ) outside of the trust region. However, the constraint does not capture how updates computed on another state s ′ affect the policy's probability distribution over the current state s . The underlying assumption is that updates computed on different states are, at least in expectation, approximately orthogonal to each other, and they do not interact. Therefore, restricting the update of the current state is sufficient to keep the policy within the region.

In our case, however, one can show that as the rank collapses or the neurons die, the representations corresponding to different states become more colinear. 10 Therefore, the gradients also become more colinear. In the extreme case, when the rank collapses to 1, or there is only one neuron alive, all representations are exactly colinear; therefore, all gradients are also. This means that even though clipping prevents the policy π θ ( ·| s ) on the current state s from changing due to the update of that state ∇ L ( π θ ( ·| s )) , π θ ( ·| s ) will still change and move outside of the trust region due to the updates on other states s ′ . Leading to the trust region constraint being ineffective and not constraining the learning process in any meaningful sense. This gives a clear situation where the theorem of Wang et al. (2020) holds and can easily be analyzed as below without resorting to the theorem for an end-to-end proof or to get a better intuition.

Formal statement of the toy setting

Let us consider a batch containing two state-action pairs ( x, a 1 ) and ( y, a 1 ) with sampled probabilities π old ( a 1 | x ) and π old ( a 1 | y ) and positive estimated advantages A ( x, a 1 ) , A ( y, a 1 ) > 0 . Let ϕ ( x ) , ϕ ( y ) ∈ R be fixed 1-dimensional representations of x , and y that can be seen as the output of the (frozen) penultimate layer of a policy network with collapsed representation (all but one dead neuron), and let α ∈ R such that ϕ ( y ) = αϕ ( x ) . Let θ = [ θ 1 , θ 2 ] , be the last layer of the network, computing the logits of two actions, a 1 and a 2 , that are

0.0

0.5

1.0

1.5

2.0

Minibatch

×10 1

1.00

1.05

1.10

1.15

1.20

1.25

Ratio

πθ

(

a

1|

x

)/

π

old

(

a

1|

x

)

πθ

(

a

1|

y

)/

π

old

(

a

1|

y

)

1+

ε

0.0

0.5

1.0

1.5

2.0

Minibatch

×10 1

0.95

1.00

1.05

1.10

1.15

Ratio

πθ

(

a

1|

x

)/

π

old

(

a

1|

x

)

πθ

(

a

1|

y

)/

π

old

(

a

1|

y

)

1+

ε

fed into a softmax to compute the probabilities. I.e., π θ ( a i | s ) = e θ i ϕ ( s ) e θ 1 ϕ ( s ) + e θ 2 ϕ ( s ) . Consider PPO

minibatch updates alternating between ( x, a 1 ) and ( y, a 1 ) . Ideally, the PPO loss increases π θ ( a 1 | s ) at gradients on ( x, a 1 ) until it reaches the clip ratio and similarly on ( y, a 1 ) . However, we show in Appendix C that a gradient step in ( x, a 1 ) also affects π θ ( a 1 | y ) and depending on α will increase it past its clipped ratio, or decrease it below its initial value. Essentially, when α ≥ 0 , a gradient on ( x, a 1 ) increases θ new 1 therefore increasing both π θ new ( a 1 | x ) and π θ new ( a 1 | y ) . The same holds for a gradient on ( y, a 1 ) , causing one state to reach the clip limit first depending on α > 1 but still have the other keep pushing its probability upwards. However, when α ≤ 0 , a gradient on ( x, a 1 ) increases θ new 1 therefore increasing π θ new ( a 1 | x ) but decreasing π θ new ( a 1 | y ) . For a gradient on ( y, a 1 ) it is the opposite: θ new 1 decreases therefore π θ new ( a 1 | x ) decreases and π θ new ( a 1 | y ) increases, causing each state to reduce the probability of the other, and depending on α < 1 one of the probabilities will dominate and push the other one down. Figure 5 shows the evolution of the probabilities when simulating the updates empirically.

4 Intervening to regularize representations and non-stationarity

Having observed that PPO is affected by a frequent representation degradation that impacts its trust region heuristic and causes its performance to collapse, we turn to study interventions that aim at regularizing the representation of the policy network or reducing the non-stationarity in the optimization. We investigate whether these interventions improve the representation metrics we track and if in turn, this affects performance. We choose simple interventions that do not apply modifications to the models during training (e.g., resetting or adding neurons) or require significantly more memory (e.g., maintaining separate copies of the models). We perform interventions on the games/tasks where the collapse is the most significant. We are interested in the state of the agent at the end of the training budget. We record the performance and representation metrics for each run as averages over the last 5% of training progress. We measure the excess ratio at a timestep as the average probability ratio above 1 + ϵ divided by the average probability ratio below 1 -ϵ at that timestep. This metric gives an idea of how much the policy exceeds the trust region. Its average value is computed over the last 5% of training progress where the ratios are non-trivial, giving the same window at the end of training as the other metrics when there is no collapse, otherwise a window before total collapse covering 5% of training progress, as after collapse, the model does not change anymore and the ratios are trivially within the 1 + ϵ and 1 -ϵ limits. We give additional details on the computation of these aggregate metrics and the interventions performed in Appendix B.

PFO: Regularizing features to mitigate trust-region issues The motivation for our first intervention and our proposed regularization method comes from our observation that the norm of the preactivation features is consistently increasing, which can be linked to the trust-region issues discussed in Section 3. We seek to mitigate this effect in a way that is analogous to the PPO trust region, by extending the trust region to the feature space. We apply an L 2 loss on the difference between the pre-activated features of the optimized policy and the policy that collected the batch, as a way to keep the preactivations of the network during an update within a trust region. We apply this regularization to the pre-activations and not the activations, as dead neurons cannot propagate gradients, and even when they do, depending on the activation function, do so with a low magnitude. The regularization is an additional loss/penalty added to the overall loss. We term this loss the Proximal Feature Optimization (PFO) loss. With ϕ θ ( s ) as the pre-activation of the penultimate layer of the actor π θ given a state s ,

We apply two versions of PFO: one on only the penultimate layer's pre-activations and one on all the pre-activations until the penultimate layer. In the scope of this work, we do not tune the coefficient of PFO; we pick the closest power of 10 that sets the magnitude of this loss to a similar magnitude of the clipped PPO objective tracked on the experiments without intervention. This gives a coefficient of 1 for ALE, 1 for MuJoCo with tanh, and 10 with ReLU. The goal is not necessarily to obtain better performance but to see if PFO improves the representations learned by PPO and if, in turn, it affects its trust region and performance. As shown in Figure 6, the regularization of PFO effectively brings the norm of the preactivation down, the number of dead neurons down, the capacity loss down, and the rank up. This coincides with a significant decrease in the excess probability ratio, especially in the upper tail. More importantly, we also see a significant increase in the lower tail of the returns where no collapse in performance is observed anymore on ALE/NameThisGame and ALE/Phoenix, with a

slight increase in the upper tail showing that PFO can increase performance. Among the interventions we have tried, PFO provided the most consistent improvements in representation and trust region.

Mean

2

4

6

×10 3

Adam equal betas

Share actor and critic features

Regularize all preactivations

Regularize last preactivation

No invervention

Episode return

2.0

2.5

5.0

Excess ratio

0

2

4

×10 2

Dead neurons policy

10 2

10 3

10 4

Norm preactivation policy

0

2

×10 2

Feature rank policy (PCA)

0.0

0.5

1.0

Capacity loss policy

Phoenix

2.5

5.0

7.5

×10 3

Adam equal betas

Share actor and critic features

Regularize all preactivations

Regularize last preactivation

No invervention

Episode return

1.5

2.0

2.5

Excess ratio

0

2

4

×10 2

Dead neurons policy

10 2

10 3

10 4

Norm preactivation policy

0

2

×10 2

Feature rank policy (PCA)

0

1

2

Capacity loss policy

Mean

Median

Outliers

NameThisGame

0.5

1.0

×10 3

Adam equal betas

Share actor and critic features

Regularize all preactivations

Regularize last preactivation

No invervention

Episode return

1.5

2.0

2.5

3.695

×10 2

Excess ratio

0

2

4

×10 2

Dead neurons policy

10 2

10 4

Norm preactivation policy

0

2

×10 2

Feature rank policy (PCA)

0.0

2.5

5.0

Capacity loss policy

Mean

Median

Outliers

Gravitar

Median

Outliers

Sharing the actor-critic trunk In deep RL, the decision to use the same feature network trunk for both the actor and the critic is not trivial. Depending on the complexity of the environment, it can significantly change the performance of a PPO agent (Andrychowicz et al., 2021; Huang et al., 2022a). We, therefore, attempt to draw a connection between sharing the feature trunk, the resulting representation, and its effects on the PPO objective. In this intervention, we make the actor and the critic share all the layers except their respective output layers and backpropagate the gradients from both the value and policy losses to the shared trunk. Figure 6 shows that the value loss acts as a regularizer, which decreases the feature rank and, depending on the reward's sparsity, gives two distinct effects. In dense-reward environments such as ALE/Phoenix and ALE/NameThisGame, the ranks are concentrated at low but non-trivial values: the upper tail significantly decreases compared to the baselines while the lower tail increases. This coincides with a lower feature norm, lower excess probability ratio, and, in turn, a high tail for the returns. It also increases performance in some cases. However, the opposite is true in the sparse-reward environment Gravitar: the rank completely collapses, and the feature norms and excess ratios are very high, collapsing the model's performance. This is consistent with the observations made in the plasticity works studying value-based methods: they show that sparse rewards deteriorate the rank of the value network, and we show that when shared in an actor-critic architecture they, in turn, deteriorate the policy. It is important to note that this distinction using the reward sparsity holds when comparing environments from the same family (e.g., ALE), but may not hold otherwise (e.g., comparing an ALE and a MuJoCo environment). We provide training curves showing the difference in the evolution of the feature rank when sharing the actor-critic trunk in Appendix D. To further strengthen this observation we run an intervention on ALE/Phoenix (a dense reward environment), with a reward mask randomly masking a reward with 90% chance, comparing the effects of sharing the actor-critic trunk. As expected, while with dense rewards, sharing the trunk is beneficial in ALE/Phoenix (Appendix Figure 21), with the sparse reward, the opposite is true: sharing the trunk is detrimental (Appendix Figure 35).

Adapting Adam Asadi et al. (2023) argue that as the targets of the value function change with the changing policy rollouts, the old moments accumulated by Adam become harmful to fit the new targets and find that resetting the moments of Adam helps performance in DQN-like algorithms. As the PPO objective creates a dependency on the previous policy, and more generally, in the policy gradient, the advantages change with the policy, the same argument about Adam moments can be made for PPO. Furthermore, Dohare et al. (2023b); Lyle et al. (2023) advocate for decaying the second moment of Adam faster than its default decay of 0.999 when training under non-stationarity

and set it to match the decay of the first moment. Therefore, we experiment with both resetting Adam's moments after each batch collection (to avoid tuning its frequency) and setting the second moment to decay at the (smaller) default decay of the first moment for both the actor and the critic; the moments are thus only accumulated over the epochs on the same batch in the former and over shorter batch sequences in the latter. We observe in Figure 6 and Appendix D that these interventions reduce the feature norm and increase the feature rank on ALE, which also reduces the excess probability ratio and, in some cases, improves performance; however, they are not sufficient to prevent collapse and, like sharing the actor-critic trunk, result in poor performance on ALE/Gravitar.

5 Related Work

Our work is complementary to various other works studying the plasticity and representation dynamics of neural networks trained under non-stationarity. Kumar et al. (2023) provide a comprehensive comparison and categorization of methods used to mitigate plasticity loss in continual supervised learning tasks and their effects on representations. Our work provides insights into the transferability of some of these solutions to RL and tools to evaluate their impact on trust region methods. Sokar et al. (2023) provide an alternative characterization of plasticity loss in RL using dormant neurons and observe an increase in dormant neurons for non-stationary objectives. Abbas et al. (2023) study representation metrics such as feature norms and observe a decrease of the norm due to dying neurons. Like in the work of Lyle et al. (2022), both studies only include value-based methods. In this work, we study dead units and capacity loss as Lyle et al. (2022) and provide corroboration of the dying units phenomenon in policy optimization methods and, taking the dying neurons out of the equation, find that the norm of preactivations actually blows up.

Other feature regularizations similar to PFO have been studied in value-based offline RL. Kumar et al. (2022) propose DR3, which counteracts an implicit regularization in TD learning by minimizing the dot product between the features of the estimated and target states. Ma et al. (2023) propose Representation Distinction (RD) which tries to avoid unwanted generalization by minimizing the dot product between the features of state-action pairs sampled from the learned policy and those sampled from the dataset or an OOD policy. Both are related to PFO as the methods directly tackle an undesired feature learning dynamic, but there is no motivation for DR3 or RD in online RL, and PFO is conceptually different. The implicit regularization that DR3 counteracts is not present in on-policy RL as shown by Kumar et al. (2022) in the SARSA experiment, and PFO differs from DR3 as it extends a trust region rather than counteracts an implicit bias. Similarly, the overestimation studied by Ma et al. (2023) in the vicious backup-generalization cycle is broken by on-policy data, and RD regularizes state features between the learned policy and the dataset policy, not consecutive policies.

6 Conclusion and Discussion

Conclusion In this work, we provide evidence that the representation deterioration under nonstationarity observed by previous work in value-based methods generalizes to PPO agents in ALE and MuJoCo with their common model architectures and is connected to performance collapse. This brings a novel perspective to previous works that showed that PPO agents lose plasticity throughout training. We show that this is particularly concerning for the heuristic trust region set by PPO-Clip, which fails to prevent collapse as it becomes less effective when the agent's representation becomes poor. Finally, we present Proximal Feature Optimization (PFO), a simple novel auxiliary loss based on regularizing the evolution of features that mitigates representation degradation and, along with other interventions, shows that controlling representation mitigates performance collapse.

Limitations and open questions In this work, we study the common architecture and optimizer of PPO agents in ALE and MuJoCo consisting of relatively small models without normalization layers, weight decay, or memory (e.g., not using Transformers and RNNs). Despite our best attempts, as with any other empirical machine learning work, the generalization of our results to other settings is not fully known. Still, this work should raise awareness about the representation collapse phenomenon observed in PPO and encourage future work to monitor representations when training PPO agents, as it can help diagnose performance collapse. We have focused on simple interventions that regularize non-stationarity and representations to highlight the effects of non-stationarity and the connection between representation, trust region, and collapse, but exploring interventions on plasticity is also valuable, as these may also influence the same dynamics. We believe further studies to analyze this problem, both empirically and particularly theoretically, to understand the reasons driving representation deterioration to be valuable. We hope that our study encourages work in this direction.

Acknowledgments and Disclosure of Funding

We extend our gratitude to the reviewers for their valuable insights, which significantly enhanced the clarity and rigor of this work. We are particularly grateful to the area chair for their guidance which shaped the final version of this paper. We thank the SCITAS team at EPFL for the access to the beta testing phase of their new cluster. We are also grateful to Vincent Moens for his support with the TorchRL library. Finally, we thank nimble.ai for their generous gift to the CLAIRE Lab, which supported D.P.'s Master's project.

References

Zaheer Abbas, Rosie Zhao, Joseph Modayil, Adam White, and Marlos C. Machado. Loss of plasticity in continual deep reinforcement learning. In Sarath Chandar, Razvan Pascanu, Hanie Sedghi, and Doina Precup (eds.), Proceedings of The 2nd Conference on Lifelong Learning Agents , volume 232 of Proceedings of Machine Learning Research , pp. 620-636. PMLR, 22-25 Aug 2023. URL https://proceedings.mlr.press/v232/abbas23a.html .

Matthew Aitchison, Penny Sweetser, and Marcus Hutter. Atari-5: Distilling the arcade learning environment down to five games. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pp. 421-438. PMLR, 23-29 Jul 2023. URL https://proceedings.mlr.press/ v202/aitchison23a.html .

Maksym Andriushchenko, Dara Bahri, Hossein Mobahi, and Nicolas Flammarion. Sharpness-aware minimization leads to low-rank features. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems , volume 36, pp. 47032-47051. Curran Associates, Inc., 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/ 92dd1adab39f362046f99dfe3c39d90f-Paper-Conference.pdf .

Marcin Andrychowicz, Anton Raichuk, Piotr Sta´ nczyk, Manu Orsini, Sertan Girgin, Raphaël Marinier, Leonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, Sylvain Gelly, and Olivier Bachem. What matters for on-policy deep actor-critic methods? a large-scale study. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id= nIAxjsniDzg .

Kavosh Asadi, Rasool Fakoor, and Shoham Sabach. Resetting the optimizer in deep rl: An empirical study. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems , volume 36, pp. 72284-72324. Curran Associates, Inc., 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/ file/e4bf5c3245fd92a4554a16af9803b757-Paper-Conference.pdf .

Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems , volume 29. Curran Associates, Inc., 2016. URL https://proceedings.neurips.cc/paper_ files/paper/2016/file/afda332245e2af431fb7b672a68b659d-Paper.pdf .

Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research , 47: 253-279, 2013.

Albert Bou, Matteo Bettini, Sebastian Dittert, Vikash Kumar, Shagun Sodhani, Xiaomeng Yang, Gianni De Fabritiis, and Vincent Moens. TorchRL: A data-driven decision-making library for pytorch. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=QxItoEAVMb .

Shibhansh Dohare, Richard S Sutton, and A Rupam Mahmood. Continual backprop: Stochastic gradient descent with persistent randomness. arXiv preprint arXiv:2108.06325 , 2021.

Shibhansh Dohare, J Fernando Hernandez-Garcia, Parash Rahman, A Rupam Mahmood, and Richard S Sutton. Maintaining plasticity in deep continual learning. arXiv preprint arXiv:2306.13812 , 2023a.

Shibhansh Dohare, Qingfeng Lan, and A. Rupam Mahmood. Overcoming policy collapse in deep reinforcement learning. In Sixteenth European Workshop on Reinforcement Learning , 2023b. URL https://openreview.net/forum?id=m9Jfdz4ymO .

Pierluca D'Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Luc Bacon, Marc G Bellemare, and Aaron Courville. Sample-efficient reinforcement learning by breaking the replay ratio barrier. In The Eleventh International Conference on Learning Representations , 2023. URL https: //openreview.net/forum?id=OpC-9aBBVJe .

Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep rl: A case study on ppo and trpo. In International Conference on Learning Representations , 2020. URL https://openreview.net/ forum?id=r1etN1rtPB .

Caglar Gulcehre, Srivatsan Srinivasan, Jakub Sygnowski, Georg Ostrovski, Mehrdad Farajtabar, Matthew Hoffman, Razvan Pascanu, and Arnaud Doucet. An empirical study of implicit regularization in deep offline RL. Transactions on Machine Learning Research , 2022. ISSN 2835-8856. URL https://openreview.net/forum?id=HFfJWx60IT .

Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905 , 2018.

Shengyi Huang, Rousslan Fernand Julien Dossa, Antonin Raffin, Anssi Kanervisto, and Weixun Wang. The 37 implementation details of proximal policy optimization. The ICLR Blog Track 2023 , 2022a.

Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research , 23(274):1-18, 2022b. URL http://jmlr.org/papers/v23/21-1342.html .

Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung, Pulkit Agrawal, and Phillip Isola. The low-rank simplicity bias in deep networks. Transactions on Machine Learning Research , 2023. ISSN 2835-8856. URL https://openreview.net/forum?id=bCiNWDmlY2 .

Maximilian Igl, Gregory Farquhar, Jelena Luketina, Wendelin Boehmer, and Shimon Whiteson. Transient non-stationarity and generalisation in deep reinforcement learning. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id= Qun8fv4qSby .

Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning , ICML '02, pp. 267-274, San Francisco, CA, USA, 2002. Morgan Kaufmann Publishers Inc. ISBN 1558608737.

G. Kendall. A new measure of rank correlation. Biometrika , 30(1/2):81-93, 1938. ISSN 00063444. URL http://www.jstor.org/stable/2332226 .

Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit under-parameterization inhibits data-efficient deep reinforcement learning. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=O9bnihsFfXU .

Aviral Kumar, Rishabh Agarwal, Tengyu Ma, Aaron Courville, George Tucker, and Sergey Levine. DR3: Value-based deep reinforcement learning requires explicit regularization. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id= POvMvLi91f .

Saurabh Kumar, Henrik Marklund, and Benjamin Van Roy. Maintaining plasticity in continual learning via regenerative regularization, 2023.

Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in reinforcement learning. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=ZkC8wKoLbQ7 .

Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney. Understanding plasticity in neural networks. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pp. 23190-23211. PMLR, 23-29 Jul 2023. URL https://proceedings.mlr.press/ v202/lyle23b.html .

Clare Lyle, Zeyu Zheng, Khimya Khetarpal, Hado van Hasselt, Razvan Pascanu, James Martens, and Will Dabney. Disentangling the causes of plasticity loss in neural networks. arXiv preprint arXiv:2402.18762 , 2024.

Yi Ma, Hongyao Tang, Dong Li, and Zhaopeng Meng. Reining generalization in offline reinforcement learning via representation distinction. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems , volume 36, pp. 40773-40785. Curran Associates, Inc., 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/ 802a4350ca4fced76b13b8b320af1543-Paper-Conference.pdf .

Marlos C Machado, Marc G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research , 61:523-562, 2018.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature , 518(7540):529-533, 2015.

Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning , volume 48 of Proceedings of Machine Learning Research , pp. 1928-1937, New York, New York, USA, 20-22 Jun 2016. PMLR. URL https: //proceedings.mlr.press/v48/mniha16.html .

Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10) , pp. 807-814, 2010.

Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pp. 16828-16847. PMLR, 17-23 Jul 2022. URL https://proceedings.mlr.press/v162/ nikishin22a.html .

Evgenii Nikishin, Junhyuk Oh, Georg Ostrovski, Clare Lyle, Razvan Pascanu, Will Dabney, and Andre Barreto. Deep reinforcement learning with plasticity injection. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems , volume 36, pp. 37142-37159. Curran Associates, Inc., 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/ 75101364dc3aa7772d27528ea504472b-Paper-Conference.pdf .

Chris Nota and Philip S. Thomas. Is the policy gradient a gradient? In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems , AAMAS '20, pp. 939-947, Richland, SC, 2020. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9781450375184.

Fabio Pardo, Arash Tavakoli, Vitaly Levdik, and Petar Kormushev. Time limits in reinforcement learning. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International

Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pp. 4045-4054. PMLR, 10-15 Jul 2018. URL https://proceedings.mlr.press/v80/pardo18a. html .

William H. Press, Saul A. Teukolsky, William T. Vetterling, and Brian P. Flannery. Numerical Recipes 3rd Edition: The Art of Scientific Computing . Cambridge University Press, USA, 3 edition, 2007. ISBN 0521880688.

Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research , 22(268):1-8, 2021. URL http://jmlr.org/papers/v22/20-1364.html .

Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In 2007 15th European Signal Processing Conference , pp. 606-610, 2007.

John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning , volume 37 of Proceedings of Machine Learning Research , pp. 1889-1897, Lille, France, 07-09 Jul 2015a. PMLR. URL https://proceedings.mlr.press/ v37/schulman15.html .

John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438 , 2015b.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evci. The dormant neuron phenomenon in deep reinforcement learning. In International Conference on Machine Learning , pp. 32145-32168. PMLR, 2023.

Spearman. The proof and measurement of association between two things. The American Journal of Psychology , 100(3/4):441-471, 1987. ISSN 00029556. URL http://www.jstor. org/stable/1422689 .

Mingfei Sun, Vitaly Kurin, Guoqing Liu, Sam Devlin, Tao Qin, Katja Hofmann, and Shimon Whiteson. You may not need ratio clipping in ppo. arXiv preprint arXiv:2202.00079 , 2022.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In S. Solla, T. Leen, and K. Müller (eds.), Advances in Neural Information Processing Systems , volume 12. MIT Press, 1999. URL https://proceedings.neurips.cc/paper_files/paper/1999/file/ 464d828b85b0bed98e80ade0a5c43b0f-Paper.pdf .

Csaba Szepesvári. Algorithms for reinforcement learning . Springer Nature, 2022.

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , pp. 5026-5033. IEEE, 2012. doi: 10.1109/IROS.2012.6386109.

Mark Towers, Jordan K. Terry, Ariel Kwiatkowski, John U. Balis, Gianluca de Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Arjun KG, Markus Krimmel, Rodrigo Perez-Vicente, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Andrew Tan Jin Shen, and Omar G. Younis. Gymnasium, March 2023. URL https://zenodo.org/record/8127025 .

Yuhui Wang, Hao He, and Xiaoyang Tan. Truly proximal policy optimization. In Ryan P. Adams and Vibhav Gogate (eds.), Proceedings of The 35th Uncertainty in Artificial Intelligence Conference , volume 115 of Proceedings of Machine Learning Research , pp. 113-122. PMLR, 22-25 Jul 2020. URL https://proceedings.mlr.press/v115/wang20b.html .

Yuzhe Yang, Guo Zhang, Zhi Xu, and Dina Katabi. Harnessing structures for value-based planning and reinforcement learning. In International Conference on Learning Representations , 2020. URL https://openreview.net/forum?id=rklHqRVKvH .

A Additional background

A.1 Reinforcement Learning

The undiscounted formulation presented in the background (Section 2) has also been used by Schulman et al. (2015b) and does not limit the use of a discount factor to discount future rewards; for that purpose, as we consider a finite-horizon setting, we can assume that discounting would already be present in the reward which depends on time through the state. This allows to isolate the discount factor γ for the purpose of the value estimation with GAE which serves as a trade-off between the bias and the variance in the estimator, in addition to λ used for the λ -returns that combine multiple n -step returns. More importantly, this also allows us to reuse the policy gradient and PPO losses without discount factors, as the deep RL community is used to them while avoiding their incorrect use in the discounted setting as pointed out by Nota & Thomas (2020). In any case, our results can also be translated to the discounted setting using a biased gradient estimator (missing a discount factor), being the typical setting considered in deep RL works.

B Experiment details

B.1 Code and run histories

Our codebase is publicly available at https://github.com/CLAIRE-Labo/ no-representation-no-trust . It includes the development environment distributed as a Docker image for GPU-accelerated machines and a Conda environment for MPS-accelerated machines, the training code, scripts to run all the experiments, and the notebook that generated the plots. The codebase uses TorchRL (Bou et al., 2024) and provides a comprehensive toolbox to study representation dynamics in policy optimization. We also provide modified scripts of CleanRL (Huang et al., 2022b) to replicate the collapse observed in this work and ensure it is not a bug from our novel codebase.

The code repository contains links to the Weights&Biases (W&B) project with all of our run histories, a summary W&B report of the runs, and a W&B report with the replication with CleanRL.

Runs are fully reproducible on the same acceleration device on which they were run. In particular, we have reproduced our results on three different clusters with the same NVIDIA GPU device.

B.2 Additional details on our experimental setup

We conduct experiments on an environment with pixel-based observations and discrete actions and an environment with continuous observations and actions, each with a different model architecture. For the discrete action case, we use the Arcade Learning Environment (ALE)(Bellemare et al., 2013) with the specification recommended by Machado et al. (2018) in v5 on Gymnasium (Towers et al., 2023). That is, with a sticky action probability of 0.25 as the only form of environment stochasticity, using only the game-over signal for termination (as opposed to end-of-life signals) with the default maximum of 108 × 10 3 environment frames per episode and reporting performance over training episodes (i.e., with sampling according to the policy distribution as opposed to taking the mode action). We train all models for 100 million environment frames. We use standard algorithmic choices to make our setting and results relevant to previous work. This includes taking only the sign of rewards (clipping) and frame skipping. We use a frame skip of 3, as opposed to the standard value of 4, due to limitations in the ALE-v5 environment, which does not implement frame pooling. 11 We use the standard architecture of Mnih et al. (2015) consisting of convolutional layers followed by linear layers, all with ReLU activations, and no normalization layers. We also use Mnih et al. (2015)'s standard observation transformations with a resizing to 84x84, grayscaling, and a frame stacking of 4.

For the continuous case, we use MuJoCo (Todorov et al., 2012) with v4 on Gymnasium (Towers et al., 2023) with the default maximum of 1,000 environment frames to mark episode termination. Similarly to Atari, we report performance as the average episode return over training episodes. We

train all models for 5 million environment frames. We standardize the observations (subtract mean and divide by standard deviation) according to an initial rollout of 4,000 environment steps (at least four episodes). The standardization parameters are kept the same to avoid adding non-stationarity. We use the same architecture as Schulman et al. (2017), with only linear layers, tanh activations, and no normalization layers. We also experiment with ReLU activations. The network outputs a mean and a standard deviation (with softplus), both conditioning on the observation independently for each action dimension, which are then used to create a TanhNormal distribution, similarly to Haarnoja et al. (2018).

To measure the capacity loss of a checkpoint, we use the same optimization hyperparameters used to train the checkpoint, i.e. the same batch size and learning rate. The optimizer is reconstructed from its initial state (loading the optimizer state is also a valid design choice). The dataset sizes and fitting budgets for capacity are listed in Tables 2 and 3.

We provide a high-level pseudocode for PPO in Algorithm 1 and list all hyperparameters considered in Tables 2 and 3.

Proximal Feature Regularization With a coefficient c PFO , the PFO loss is added to the overall loss ( -L CLIP π old ( θ ) + c PFO L PFO π old ( θ ) -c H H ( θ ) + c V F L V F ( w )) optimized with autograd over multiple minibatch epochs.

B.3 Additional details on metrics used in the figures

Training curves A point in the training curves in Figures 1, 2, 3, before aggregating seeds and smoothing, corresponds to an average value over the last batch collected at the time of logging for metrics available at every batch (feature rank, entropy, etc.) or the latest batch where the metric was available at the time of logging for the episodic return (as it's only available when episodes finish, and it requires multiple batches to finish an episode). E.g., in Figure 1 on ALE, a feature rank corresponds to the average feature rank over all the states in the last batch collected at the time of logging and is logged every 0.1% of the batches (i.e. every 6,144 env steps); A return corresponds to the average return across all workers that had episodes finished in the latest batch containing finished episodes at the time of logging.

Figure 4 A window of size 1% of training progress represents approximately 1 million training steps on ALE and 50,000 training steps on MuJoCo We average the metrics per window and then take the 20 windows with the lowest average probability ratios below 1 -ϵ . The probability ratios in a run can be trivially within the 1 -ϵ region after the model collapses, resulting in less than 20 points if the model collapses before 20% of the training progress. When all runs give 20 points, we can observe 300 points in total per scatter plot.

Environment

Figure 6 A window of size 5% of training progress represents approximately five million training steps in ALE and captures at least five episodes per environment so in total at least 40 episodes. For MuJoCo this represents approximately 256,000 training steps and captures at least 128 episodes per environment so in total at least 256 episodes.

When a model collapses, it typically doesn't change anymore so its optimization trivially gives ratios within the clipping limits (no value above 1 + ϵ and below 1 -ϵ is logged). In that case, we are more interested in the evolution of the excess ratio before the ratios become trivial. Therefore, the upper limit of the 5% of training progress is taken such that it is the latest timestep where there are at least 10 non-trivial ratios, i.e. 10 logged excess ratios. This coincides with a window before the collapse of the model capturing the values we are interested in. Note that when a model collapses this window may not coincide with the window used to report the other metrics such as the average return, however, these other metrics typically do not change after a collapse, so it is more robust to capture them at the end of training rather than looking for an arbitrary window after the collapse. We give training curves similar to Figure 1 with the interventions performed.

In MuJoCo, with continuous action distributions the ratios diverge to infinity and 0 before collapse therefore to get meaningful plots, we clip average probability ratios above 1 + ϵ and below 1 -ϵ to 10 12 and 10 -12 , respectively, before computing the average excess ratio.

We group the different epoch configurations of an intervention on the same environment, giving 15 runs per boxplot (three epochs with five seeds each). The right (resp. left) whiskers are determined by the highest (resp. lowest) observed datapoint below Q3 + 1.5 IQR (default of Matplotlib). The outliers are points outside of the whiskers.

B.4 Statistical significance

Stochasticity in our experiments arises from network initialization, environment transitions (e.g., sticky actions in ALE), agent action sampling, and minibatch sampling for optimization. A seed fully controls the sequence of randomness in a run with the same hyperparameter configuration. We repeat each configuration with five seeds using the same collection of seeds, resulting in the same initialization of the networks and environments for a given seed across configurations. This form of repeated measures allows us to compare the configurations with lower variance as they share the same initial conditions, hence requiring a lower number of seeds.

In Figures 1 and 2, we aggregate the five runs of each experiment into mean curves with min/max shaded areas. The use of min/max error bars allows us to demonstrate the full range of observed outcomes, although it may result in shaded areas that overlap more than with other types of error bars. Most of the claims we make based on those figures do not rely on non-overlapping shaded areas and are instead stronger when the max or min boundaries are consistent with the observation made

(min boundary of feature norm increasing, max boundary of feature rank decreasing). Otherwise, we made comparative claims when shaded areas did not overlap (feature rank decreasing faster with more epochs and more non-stationarity).

Figure 3 displays individual seeds to zoom on single-run dynamics around collapse. It is used for an illustrative purpose to provide intuition and does not depend on the number of runs or statistical aggregation of results. The main claim made with the intuition (breakdown of the trust region) is backed by Figure 4, which includes 300 points per plot per environment, subsampled from 15 training curves per environment.

To evaluate the effects of the interventions in Figure 6, we show boxplots to give a complete idea of the distribution of the data which is formed by grouping the different configurations in the same environment. Each boxplot contains 15 runs. We make claims such as preventing collapse using the tails and medians and claims about lower excess ratio and higher rank using the interquartile range. Without a clear intuition about the distribution of combined configurations per environment, we consider this approach appropriate for comparing interventions.

In summary, we believe our experimental design provides a balanced tradeoff between statistical significance and richness of claims. The computational cost of running more seeds may not yield proportionately valuable insights.

B.5 Hardware and runtime

The experiments in this project took a total of ~11,300 GPU hours on NVIDIA V100 and A100 GPUs (ALE) and ~25,500 CPU hours (MuJoCo). A run on ALE takes around 10 hours on an A100 and 16 hours on a V100. A run on MuJoCo takes around 5 hours on 6 CPUs.

Training

Environment every 0.1% (~100,000 env steps)

Capacity every 2.5% (41 times in total)

Environment

C Toy setting derivation details

The derivatives of the softmax probability π θ ( a 1 | s ) with respect to θ 1 and θ 2 are as follows:

The update rule for each parameter θ i in θ with SGD is θ new i = θ i + η ∂L ∂θ i where η is the learning rate. Therefore, given the partial derivatives, the updated values for θ 1 and θ 2 after taking a gradient step are (if the probability is still inferior to 1 + ϵ , otherwise the gradient is 0)

Hence,

Let α ≥ 0 and without loss of generality, let's take α ≥ 1 . After a gradient step on x one has

And after a gradient step on y :

Let α ≤ 0 and without loss of generality, let's take α ≤ 1 , after a gradient step on x one has

And after a gradient step on y :

D Main paper figures on all environments

0.0

0.5

1.0

Environment steps×10 8

0

2

4

6

8

Episode return

×10 3

ALE/Phoenix-v5

0.0

0.5

1.0

Environment steps×10 8

0

1

2

3

Feature rank policy (PCA)

×10 2

0.0

0.5

1.0

Environment steps×10 8

10 0

10 1

10 2

10 3

10 4

Norm preactivation policy

0.0

0.5

1.0

Environment steps×10 8

0

2

4

6

Capacity loss policy

0.0

0.5

1.0

Environment steps×10 8

0

1

2

3

4

Feature rank critic (PCA)

×10 1

0.0

0.5

1.0

Environment steps×10 8

0.0

0.5

1.0

1.5

2.0

Capacity loss critic

4 epochs

6 epochs

8 epochs

0.0

0.5

1.0

Environment steps×10 8

2

4

6

Episode return

×10 3

ALE/NameThisGame-v5

0.0

0.5

1.0

Environment steps×10 8

0.0

0.5

1.0

1.5

Feature rank policy (PCA)

×10 2

0.0

0.5

1.0

Environment steps×10 8

10 0

10 1

10 2

10 3

10 4

Norm preactivation policy

0.0

0.5

1.0

Environment steps×10 8

0.0

0.5

1.0

1.5

2.0

2.5

Capacity loss policy

0.0

0.5

1.0

Environment steps×10 8

2

4

6

8

Feature rank critic (PCA)

0.0

0.5

1.0

Environment steps×10 8

0

2

4

6

Capacity loss critic

4 epochs

6 epochs

8 epochs

0.0

0.5

1.0

Environment steps ×10 8

0.0

0.5

1.0

1.5

2.0

2.5

Episode return

×10 4

ALE/BattleZone-v5

0.0

0.5

1.0

Environment steps ×10 8

0.0

0.5

1.0

1.5

2.0

2.5

Feature rank policy (PCA)

×10 2

0.0

0.5

1.0

Environment steps ×10 8

10 0

10 1

10 2

10 3

Norm preactivation policy

0.0

0.5

1.0

Environment steps ×10 8

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Capacity loss policy

0.0

0.5

1.0

Environment steps ×10 8

0.5

1.0

1.5

2.0

Feature rank critic (PCA)

×10 1

0.0

0.5

1.0

Environment steps ×10 8

0

2

4

6

8

Capacity loss critic

×10 -3

4 epochs

6 epochs

8 epochs

0.0

0.5

1.0

Environment steps ×10 8

0.00

0.25

0.50

0.75

1.00

1.25

Episode return

×10 3

ALE/Gravitar-v5

0.0

0.5

1.0

Environment steps ×10 8

0.0

0.5

1.0

1.5

2.0

2.5

Feature rank policy (PCA)

×10 2

0.0

0.5

1.0

Environment steps ×10 8

10 0

10 1

10 2

10 3

10 4

Norm preactivation policy

0.0

0.5

1.0

Environment steps ×10 8

0.0

0.2

0.4

0.6

0.8

1.0

Capacity loss policy

0.0

0.5

1.0

Environment steps ×10 8

0.25

0.50

0.75

1.00

1.25

1.50

Feature rank critic (PCA)

×10 1

0.0

0.5

1.0

Environment steps ×10 8

0

2

4

6

8

Capacity loss critic

×10 -3

4 epochs

6 epochs

8 epochs

0.0

0.5

1.0

Environment steps ×10 8

-2.5

-2.0

-1.5

-1.0

-0.5

0.0

Episode return

×10 1

ALE/DoubleDunk-v5

0.0

0.5

1.0

Environment steps ×10 8

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Feature rank policy (PCA)

×10 2

0.0

0.5

1.0

Environment steps ×10 8

10 0

10 1

10 2

10 3

10 4

Norm preactivation policy

0.0

0.5

1.0

Environment steps ×10 8

0

1

2

3

4

5

6

Capacity loss policy

0.0

0.5

1.0

Environment steps ×10 8

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Feature rank critic (PCA)

×10 1

0.0

0.5

1.0

Environment steps ×10 8

0

1

2

3

Capacity loss critic

×10 -3

4 epochs

6 epochs

8 epochs

0.0

0.5

1.0

Environment steps ×10 8

0.0

0.5

1.0

1.5

Episode return

×10 4

ALE/Qbert-v5

0.0

0.5

1.0

Environment steps ×10 8

0.0

0.5

1.0

1.5

2.0

Feature rank policy (PCA)

×10 2

0.0

0.5

1.0

Environment steps ×10 8

10 0

10 1

10 2

10 3

Norm preactivation policy

0.0

0.5

1.0

Environment steps ×10 8

0.0

0.5

1.0

1.5

Capacity loss policy

0.0

0.5

1.0

Environment steps ×10 8

0

1

2

3

4

5

6

Feature rank critic (PCA)

×10 1

0.0

0.5

1.0

Environment steps ×10 8

0.00

0.25

0.50

0.75

1.00

1.25

Capacity loss critic

4 epochs

6 epochs

8 epochs

0

2

4

Environment steps ×10 6

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Episode return

×10 3

Hopper-v4

0

2

4

Environment steps ×10 6

0

1

2

3

4

5

Feature rank policy (PCA)

×10 1

0

2

4

Environment steps ×10 6

10 1

10 2

Norm preactivation policy

0

2

4

Environment steps ×10 6

10 -2

10 0

10 2

10 4

10 6

10 8

Capacity loss policy

0

2

4

Environment steps ×10 6

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Feature rank critic (PCA)

×10 1

0

2

4

Environment steps ×10 6

0

1

2

3

4

5

Capacity loss critic

×10 4

10 epochs

15 epochs

20 epochs

0

2

4

Environment steps ×10 6

-1

0

1

2

3

4

5

Episode return

×10 3

HalfCheetah-v4

0

2

4

Environment steps ×10 6

1

2

3

4

5

Feature rank policy (PCA)

×10 1

0

2

4

Environment steps ×10 6

10 1

Norm preactivation policy

0

2

4

Environment steps ×10 6

10 -1

10 1

10 3

10 5

Capacity loss policy

0

2

4

Environment steps ×10 6

0.50

0.75

1.00

1.25

1.50

1.75

Feature rank critic (PCA)

×10 1

0

2

4

Environment steps ×10 6

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Capacity loss critic

×10 4

10 epochs

15 epochs

20 epochs

0

2

4

Environment steps ×10 6

2

4

6

8

Episode return

×10 2

Humanoid-v4

0

2

4

Environment steps ×10 6

4.0

4.5

5.0

5.5

Feature rank policy (PCA)

×10 1

0

2

4

Environment steps ×10 6

10 1

Norm preactivation policy

0

2

4

Environment steps ×10 6

10 -1

10 0

10 1

10 2

10 3

10 4

Capacity loss policy

0

2

4

Environment steps ×10 6

0

1

2

3

4

Feature rank critic (PCA)

×10 1

0

2

4

Environment steps ×10 6

0

1

2

3

Capacity loss critic

×10 3

10 epochs

15 epochs

20 epochs

0

2

4

Environment steps ×10 6

-1

0

1

2

3

4

Episode return

×10 3

Ant-v4

0

2

4

Environment steps ×10 6

2

3

4

5

6

Feature rank policy (PCA)

×10 1

0

2

4

Environment steps ×10 6

10 1

Norm preactivation policy

0

2

4

Environment steps ×10 6

10 -1

10 1

10 3

10 5

Capacity loss policy

0

2

4

Environment steps ×10 6

0

1

2

3

Feature rank critic (PCA)

×10 1

0

2

4

Environment steps ×10 6

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Capacity loss critic

×10 4

10 epochs

15 epochs

20 epochs

0

2

4

Environment steps ×10 6

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Episode return

×10 3

Hopper-v4

0

2

4

Environment steps ×10 6

0

1

2

3

4

Feature rank policy (PCA)

×10 1

0

2

4

Environment steps ×10 6

10 1

10 2

10 3

Norm preactivation policy

0

2

4

Environment steps ×10 6

10 -1

10 2

10 5

10 8

10 11

Capacity loss policy

0

2

4

Environment steps ×10 6

2

4

6

Feature rank critic (PCA)

0

2

4

Environment steps ×10 6

0

1

2

3

4

5

6

Capacity loss critic

×10 3

10 epochs

15 epochs

20 epochs

0

2

4

Environment steps ×10 6

-1

0

1

2

3

4

5

Episode return

×10 3

HalfCheetah-v4

0

2

4

Environment steps ×10 6

0

1

2

3

4

5

Feature rank policy (PCA)

×10 1

0

2

4

Environment steps ×10 6

10 1

10 2

10 3

10 4

Norm preactivation policy

0

2

4

Environment steps ×10 6

10 0

10 3

10 6

10 9

10 12

Capacity loss policy

0

2

4

Environment steps ×10 6

0.2

0.4

0.6

0.8

1.0

1.2

Feature rank critic (PCA)

×10 1

0

2

4

Environment steps ×10 6

0

2

4

6

Capacity loss critic

×10 3

10 epochs

15 epochs

20 epochs

0

2

4

Environment steps ×10 6

0.2

0.4

0.6

0.8

1.0

1.2

Episode return

×10 3

Humanoid-v4

0

2

4

Environment steps ×10 6

0

1

2

3

4

5

Feature rank policy (PCA)

×10 1

0

2

4

Environment steps ×10 6

10 1

10 3

10 5

10 7

10 9

Norm preactivation policy

0

2

4

Environment steps ×10 6

10 0

10 3

10 6

10 9

10 12

Capacity loss policy

0

2

4

Environment steps ×10 6

2

4

6

8

Feature rank critic (PCA)

0

2

4

Environment steps ×10 6

0.00

0.25

0.50

0.75

1.00

1.25

Capacity loss critic

×10 20

10 epochs

15 epochs

20 epochs

0

2

4

Environment steps ×10 6

0

1

2

3

Episode return

×10 3

Ant-v4

0

2

4

Environment steps ×10 6

3.0

3.5

4.0

4.5

5.0

Feature rank policy (PCA)

×10 1

0

2

4

Environment steps ×10 6

10 1

Norm preactivation policy

0

2

4

Environment steps ×10 6

10 -1

10 0

10 1

10 2

10 3

Capacity loss policy

0

2

4

Environment steps ×10 6

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Feature rank critic (PCA)

×10 1

0

2

4

Environment steps ×10 6

0

1

2

3

4

Capacity loss critic

×10 3

10 epochs

15 epochs

20 epochs

0.00

0.25

0.50

0.75

1.00

Environment steps

×10 8

1.0

1.2

1.4

1.6

1.8

2.0

Entropy

ALE/Phoenix-v5

0.00

0.25

0.50

0.75

1.00

Environment steps

×10 8

0

1

2

3

4

5

Policy variance

0.00

0.25

0.50

0.75

1.00

Environment steps

×10 8

0

1

2

3

4

5

Dead neurons policy

×10 2

4 epochs

6 epochs

8 epochs

0.00

0.25

0.50

0.75

1.00

Environment steps

×10 8

0.50

0.75

1.00

1.25

1.50

1.75

Entropy

ALE/NameThisGame-v5

0.00

0.25

0.50

0.75

1.00

Environment steps

×10 8

0

2

4

6

8

Policy variance

0.00

0.25

0.50

0.75

1.00

Environment steps

×10 8

2.5

3.0

3.5

4.0

4.5

5.0

Dead neurons policy

×10 2

4 epochs

6 epochs

8 epochs

0.00

0.25

0.50

0.75

1.00

Environment steps

×10 8

1.0

1.5

2.0

2.5

3.0

Entropy

ALE/BattleZone-v5

0.00

0.25

0.50

0.75

1.00

Environment steps

×10 8

0

1

2

3

Policy variance

0.00

0.25

0.50

0.75

1.00

Environment steps

×10 8

1

2

3

4

5

Dead neurons policy

×10 2

4 epochs

6 epochs

8 epochs

0.00

0.25

0.50

0.75

1.00

Environment steps

×10 8

2.0

2.2

2.4

2.6

2.8

Entropy

ALE/Gravitar-v5

0.00

0.25

0.50

0.75

1.00

Environment steps

×10 8

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Policy variance

0.00

0.25

0.50

0.75

1.00

Environment steps

×10 8

1

2

3

4

5

Dead neurons policy

×10 2

4 epochs

6 epochs

8 epochs

0.00

0.25

0.50

0.75

1.00

Environment steps

×10 8

1.5

2.0

2.5

Entropy

ALE/DoubleDunk-v5

0.00

0.25

0.50

0.75

1.00

Environment steps

×10 8

0.0

0.5

1.0

1.5

Policy variance

0.00

0.25

0.50

0.75

1.00

Environment steps

×10 8

2

3

4

5

Dead neurons policy

×10 2

4 epochs

6 epochs

8 epochs

0.00

0.25

0.50

0.75

1.00

Environment steps

×10 8

0.50

0.75

1.00

1.25

1.50

1.75

Entropy

ALE/Qbert-v5

0.00

0.25

0.50

0.75

1.00

Environment steps

×10 8

0.0

0.2

0.4

0.6

0.8

1.0

Policy variance

0.00

0.25

0.50

0.75

1.00

Environment steps

×10 8

1

2

3

4

5

Dead neurons policy

×10 2

4 epochs

6 epochs

8 epochs

0

2

4

Environment steps

×10 6

0.0

0.5

1.0

1.5

2.0

2.5

Entropy

×10 8

Hopper-v4

0

2

4

Environment steps

×10 6

10 -5

10 -3

10 -1

10 1

10 3

Policy variance

0

2

4

Environment steps

×10 6

0

2

4

6

Dead neurons policy

×10 1

10 epochs

15 epochs

20 epochs

0

2

4

Environment steps

×10 6

0.0

0.2

0.4

0.6

0.8

1.0

Entropy

×10 5

HalfCheetah-v4

0

2

4

Environment steps

×10 6

10 -1

10 1

10 3

10 5

Policy variance

0

2

4

Environment steps

×10 6

0.0

0.5

1.0

1.5

2.0

2.5

Dead neurons policy

10 epochs

15 epochs

20 epochs

0

2

4

Environment steps

×10 6

-1.0

-0.8

-0.6

-0.4

-0.2

0.0

Entropy

×10 2

Humanoid-v4

0

2

4

Environment steps

×10 6

10 -2

10 -1

10 0

10 1

10 2

Policy variance

0

2

4

Environment steps

×10 6

-4

-2

0

2

4

Dead neurons policy

10 epochs

15 epochs

20 epochs

0

2

4

Environment steps

×10 6

0

2

4

6

Entropy

×10 6

Ant-v4

0

2

4

Environment steps

×10 6

10 -1

10 1

10 3

10 5

Policy variance

0

2

4

Environment steps

×10 6

-4

-2

0

2

4

Dead neurons policy

10 epochs

15 epochs

20 epochs

0

2

4

Environment steps

×10 6

0.0

0.5

1.0

1.5

2.0

2.5

Entropy

×10 12

Hopper-v4

0

2

4

Environment steps

×10 6

10 -2

10 0

10 2

10 4

10 6

Policy variance

0

2

4

Environment steps

×10 6

0

1

2

3

4

Dead neurons policy

×10 1

10 epochs

15 epochs

20 epochs

0

2

4

Environment steps

×10 6

0.0

0.5

1.0

1.5

2.0

Entropy

×10 13

HalfCheetah-v4

0

2

4

Environment steps

×10 6

10 -1

10 2

10 5

10 8

10 11

10 14

Policy variance

0

2

4

Environment steps

×10 6

0

1

2

3

Dead neurons policy

×10 1

10 epochs

15 epochs

20 epochs

0

2

4

Environment steps

×10 6

0.0

0.5

1.0

1.5

Entropy

×10 24

Humanoid-v4

0

2

4

Environment steps

×10 6

10 0

10 4

10 8

10 12

10 16

10 20

Policy variance

0

2

4

Environment steps

×10 6

0

1

2

3

4

Dead neurons policy

×10 1

10 epochs

15 epochs

20 epochs

0

2

4

Environment steps

×10 6

-2.5

-2.0

-1.5

-1.0

-0.5

0.0

0.5

Entropy

×10 1

Ant-v4

0

2

4

Environment steps

×10 6

10 -2

10 -1

Policy variance

0

2

4

Environment steps

×10 6

0

2

4

6

Dead neurons policy

10 epochs

15 epochs

20 epochs

0.0

0.5

1.0

Environment steps×10 8

0.00

0.25

0.50

0.75

1.00

1.25

1.50

Rank policy (PCA)

×10 2

0.0

0.5

1.0

Environment steps×10 8

0.0

0.2

0.4

0.6

0.8

1.0

Capacity loss policy

0.0

0.5

1.0

Environment steps×10 8

2

3

4

5

6

7

Episode return

×10 3

0.0

0.5

1.0

Environment steps×10 8

6.5

7.0

7.5

8.0

8.5

9.0

Avg of prob ratios < 1 -

ε

0.0

0.5

1.0

Environment steps×10 8

0

2

4

6

8

Policy variance

0.0

0.5

1.0

Environment steps×10 8

-1

0

1

2

3

4

5

PPO-Clip objective

ALE/NameThisGame-v5, 4 epochs

ALE/NameThisGame-v5, 6 epochs

ALE/NameThisGame-v5, 8 epochs

0

2

4

Environment steps ×10 6

0

1

2

3

4

5

Rank policy (PCA)

×10 1

0

2

4

Environment steps ×10 6

10 -2

10 0

10 2

10 4

10 6

10 8

Capacity loss policy

0

2

4

Environment steps ×10 6

0.0

0.5

1.0

1.5

2.0

2.5

Episode return

×10 3

0

2

4

Environment steps ×10 6

0

1

2

3

4

5

6

7

Avg of prob ratios < 1 -

ε

0

2

4

Environment steps ×10 6

10 -2

10 -1

10 0

10 1

10 2

Policy variance

0

2

4

Environment steps ×10 6

-1.00

-0.75

-0.50

-0.25

0.00

0.25

0.50

0.75

1.00

PPO-Clip objective

HalfCheetah-v4, 10 epochs, seed=64

Hopper-v4, 10 epochs, seed=25

Hopper-v4, 10 epochs, seed=7

Hopper-v4, 10 epochs, seed=64

0

1

2

3

4

5

Dead neurons policy

×10 2

4.5

5.0

5.5

6.0

6.5

7.0

7.5

8.0

Avg of prob ratios < 1 -

ε

ALE/Phoenix-v5

Kendall: -0.11, Spearman: -0.16

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Feature rank policy (PCA)

×10 2

4.5

5.0

5.5

6.0

6.5

7.0

7.5

8.0

Avg of prob ratios < 1 -

ε

Kendall: 0.08, Spearman: 0.11

10 2

10 3

Feature preactivation norm

4.5

5.0

5.5

6.0

6.5

7.0

7.5

8.0

Avg of prob ratios < 1 -

ε

Kendall: -0.28, Spearman: -0.40

3.0

3.5

4.0

4.5

5.0

Dead neurons policy

×10 2

4

5

6

7

8

Avg of prob ratios < 1 -

ε

ALE/NameThisGame-v5

Kendall: -0.27, Spearman: -0.38

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

Feature rank policy (PCA)

×10 2

4

5

6

7

8

Avg of prob ratios < 1 -

ε

Kendall: 0.25, Spearman: 0.34

10 2

10 3

10 4

Feature preactivation norm

4

5

6

7

8

Avg of prob ratios < 1 -

ε

Kendall: -0.30, Spearman: -0.42

1

2

3

4

5

Dead neurons policy

×10 2

5.0

5.5

6.0

6.5

7.0

7.5

8.0

Avg of prob ratios < 1 -

ε

ALE/BattleZone-v5

Kendall: -0.41, Spearman: -0.58

0.0

0.5

1.0

1.5

2.0

2.5

Feature rank policy (PCA)

×10 2

5.0

5.5

6.0

6.5

7.0

7.5

8.0

Avg of prob ratios < 1 -

ε

Kendall: -0.01, Spearman: 0.00

10 1

10 2

10 3

Feature preactivation norm

5.0

5.5

6.0

6.5

7.0

7.5

8.0

Avg of prob ratios < 1 -

ε

Kendall: -0.52, Spearman: -0.70

1

2

3

4

5

Dead neurons policy

×10 2

7.4

7.5

7.6

7.7

7.8

7.9

8.0

8.1

Avg of prob ratios < 1 -

ε

ALE/Gravitar-v5

Kendall: -0.32, Spearman: -0.47

0.0

0.5

1.0

1.5

2.0

2.5

Feature rank policy (PCA)

×10 2

7.4

7.5

7.6

7.7

7.8

7.9

8.0

8.1

Avg of prob ratios < 1 -

ε

Kendall: 0.15, Spearman: 0.25

10 2

10 3

Feature preactivation norm

6.0

6.5

7.0

7.5

8.0

Avg of prob ratios < 1 -

ε

Kendall: -0.21, Spearman: -0.33

2.5

3.0

3.5

4.0

4.5

5.0

Dead neurons policy

×10 2

4

5

6

7

8

Avg of prob ratios < 1 -

ε

ALE/DoubleDunk-v5

Kendall: -0.26, Spearman: -0.38

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Feature rank policy (PCA)

×10 2

4

5

6

7

8

Avg of prob ratios < 1 -

ε

Kendall: -0.09, Spearman: -0.13

10 1

10 2

10 3

Feature preactivation norm

4

5

6

7

8

Avg of prob ratios < 1 -

ε

Kendall: -0.43, Spearman: -0.60

1

2

3

4

5

Dead neurons policy

×10 2

6.5

7.0

7.5

8.0

8.5

Avg of prob ratios < 1 -

ε

ALE/Qbert-v5

Kendall: 0.18, Spearman: 0.26

0.0

0.5

1.0

1.5

2.0

Feature rank policy (PCA)

×10 2

6.5

7.0

7.5

8.0

8.5

Avg of prob ratios < 1 -

ε

Kendall: -0.13, Spearman: -0.18

10 1

10 2

10 3

Feature preactivation norm

4

5

6

7

8

Avg of prob ratios < 1 -

ε

Kendall: -0.38, Spearman: -0.56

0

1

2

3

4

5

6

Dead neurons policy

×10 1

0.0

0.5

1.0

1.5

2.0

Avg of prob ratios < 1 -

ε

Hopper-v4

Kendall: -0.50, Spearman: -0.63

0

1

2

3

4

5

Feature rank policy (PCA)

×10 1

0.0

0.5

1.0

1.5

2.0

Avg of prob ratios < 1 -

ε

Kendall: 0.70, Spearman: 0.78

10 2

3×10 1

4×10 1

6×10 1

Feature preactivation norm

0.0

0.5

1.0

1.5

2.0

Avg of prob ratios < 1 -

ε

Kendall: -0.47, Spearman: -0.61

0.0

0.5

1.0

1.5

2.0

2.5

Dead neurons policy

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

Avg of prob ratios < 1 -

ε

HalfCheetah-v4

Kendall: 0.07, Spearman: 0.08

1

2

3

4

5

Feature rank policy (PCA)

×10 1

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

Avg of prob ratios < 1 -

ε

Kendall: 0.00, Spearman: 0.01

3×10 1

4×10 1

6×10 1

Feature preactivation norm

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

Avg of prob ratios < 1 -

ε

Kendall: 0.15, Spearman: 0.23

-4

-2

0

2

4

Dead neurons policy

1.25

1.50

1.75

2.00

2.25

2.50

2.75

3.00

Avg of prob ratios < 1 -

ε

Humanoid-v4

5.0

5.2

5.4

5.6

5.8

Feature rank policy (PCA)

×10 1

1.25

1.50

1.75

2.00

2.25

2.50

2.75

3.00

3.25

Avg of prob ratios < 1 -

ε

Kendall: -0.26, Spearman: -0.38

2×10 1

3×10 1

Feature preactivation norm

1.25

1.50

1.75

2.00

2.25

2.50

2.75

3.00

Avg of prob ratios < 1 -

ε

Kendall: -0.23, Spearman: -0.37

-4

-2

0

2

4

Dead neurons policy

0

1

2

3

4

5

Avg of prob ratios < 1 -

ε

Ant-v4

2

3

4

5

6

Feature rank policy (PCA)

×10 1

0

1

2

3

4

5

Avg of prob ratios < 1 -

ε

Kendall: -0.03, Spearman: 0.08

2×10 1

3×10 1

4×10 1

Feature preactivation norm

-1

0

1

2

3

4

5

Avg of prob ratios < 1 -

ε

Kendall: -0.67, Spearman: -0.87

1

2

3

4

Dead neurons policy

×10 1

-0.25

0.00

0.25

0.50

0.75

1.00

1.25

Avg of prob ratios < 1 -

ε

Hopper-v4

Kendall: -0.46, Spearman: -0.58

0.25

0.50

0.75

1.00

1.25

1.50

1.75

Feature rank policy (PCA)

×10 1

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

Avg of prob ratios < 1 -

ε

Kendall: 0.46, Spearman: 0.53

10 2

10 3

Feature preactivation norm

-0.25

0.00

0.25

0.50

0.75

1.00

1.25

Avg of prob ratios < 1 -

ε

Kendall: -0.43, Spearman: -0.54

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Dead neurons policy

×10 1

-1

0

1

2

3

4

5

Avg of prob ratios < 1 -

ε

HalfCheetah-v4

Kendall: -0.46, Spearman: -0.68

0

1

2

3

4

Feature rank policy (PCA)

×10 1

0

1

2

3

4

5

Avg of prob ratios < 1 -

ε

Kendall: 0.53, Spearman: 0.67

10 1

10 2

10 3

10 4

Feature preactivation norm

0

1

2

3

4

5

Avg of prob ratios < 1 -

ε

Kendall: -0.24, Spearman: -0.35

0

1

2

3

4

Dead neurons policy

×10 1

-1.0

-0.5

0.0

0.5

1.0

1.5

2.0

2.5

Avg of prob ratios < 1 -

ε

Humanoid-v4

Kendall: -0.30, Spearman: -0.41

0

1

2

3

4

5

Feature rank policy (PCA)

×10 1

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Avg of prob ratios < 1 -

ε

Kendall: 0.42, Spearman: 0.55

10 2

10 3

10 4

10 5

10 6

10 7

10 8

10 9

Feature preactivation norm

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

2.0

2.5

Avg of prob ratios < 1 -

ε

Kendall: -0.45, Spearman: -0.58

0

1

2

3

4

5

6

Dead neurons policy

1.5

2.0

2.5

3.0

3.5

4.0

Avg of prob ratios < 1 -

ε

Ant-v4

Kendall: -0.04, Spearman: -0.06

3.4

3.6

3.8

4.0

4.2

4.4

4.6

4.8

Feature rank policy (PCA)

×10 1

1.5

2.0

2.5

3.0

3.5

4.0

Avg of prob ratios < 1 -

ε

Kendall: -0.21, Spearman: -0.33

10 1

6×10 0

Feature preactivation norm

1.5

2.0

2.5

3.0

3.5

4.0

Avg of prob ratios < 1 -

ε

Kendall: -0.23, Spearman: -0.33

2

4

6

×10 3

Adam equal betas

Share actor and critic features

Regularize all preactivations

Regularize last preactivation

No invervention

Episode return

2.0

2.5

5.0

Excess ratio

0

2

4

×10 2

Dead neurons policy

10 2

10 3

10 4

Norm preactivation policy

0

2

×10 2

Feature rank policy (PCA)

0.0

0.5

1.0

Capacity loss policy

Mean

Median

Outliers

Phoenix

2.5

5.0

7.5

×10 3

Adam equal betas

Share actor and critic features

Regularize all preactivations

Regularize last preactivation

No invervention

Episode return

1.5

2.0

2.5

Excess ratio

0

2

4

×10 2

Dead neurons policy

10 2

10 3

10 4

Norm preactivation policy

0

2

×10 2

Feature rank policy (PCA)

0

1

2

Capacity loss policy

Mean

Median

Outliers

NameThisGame

0.5

1.0

×10 3

Adam equal betas

Share actor and critic features

Regularize all preactivations

Regularize last preactivation

No invervention

Episode return

1.5

2.0

2.5

3.695

×10 2

Excess ratio

0

2

4

×10 2

Dead neurons policy

10 2

10 4

Norm preactivation policy

0

2

×10 2

Feature rank policy (PCA)

0.0

2.5

5.0

Capacity loss policy

Mean

Median

Outliers

Gravitar

Mean

0

1

×10 3

Adam equal betas

Share actor and critic features

Regularize all preactivations

Regularize last preactivation

No invervention

Episode return

10 17

10 21

Excess ratio

0.0

2.5

5.0

×10 1

Dead neurons policy

10 2

Norm preactivation policy

0

2

4

×10 1

Feature rank policy (PCA)

10 3

10 6

Capacity loss policy

Hopper

0.5

1.0

×10 3

Adam equal betas

Share actor and critic features

Regularize all preactivations

Regularize last preactivation

No invervention

Episode return

10 8

10 17

Excess ratio

0

1

2

×10 1

Dead neurons policy

10 2

Norm preactivation policy

2

4

6

×10 1

Feature rank policy (PCA)

10 3

10 6

Capacity loss policy

Mean

Median

Outliers

Humanoid

0.0

2.5

5.0

×10 3

Adam equal betas

Share actor and critic features

Regularize all preactivations

Regularize last preactivation

No invervention

Episode return

10 8

10 17

Excess ratio

0

2

4

×10 1

Dead neurons policy

10 2

Norm preactivation policy

0.0

2.5

5.0

×10 1

Feature rank policy (PCA)

10 3

10 7

Capacity loss policy

Mean

Median

Outliers

HalfCheetah

-2.5

0.0

2.5

×10 3

Adam equal betas

Share actor and critic features

Regularize all preactivations

Regularize last preactivation

No invervention

Episode return

10 8

10 17

Excess ratio

0

2

4

×10 1

Dead neurons policy

10 2

Norm preactivation policy

2.5

5.0

×10 1

Feature rank policy (PCA)

10 3

10 7

Capacity loss policy

Mean

Median

Outliers

Ant

Median

Outliers

Mean

0

5

×10 2

Adam equal betas

Share actor and critic features

Regularize all preactivations

Regularize last preactivation

No invervention

Episode return

10 17

10 21

Excess ratio

0.0

2.5

5.0

×10 1

Dead neurons policy

10 1

10 2

10 3

Norm preactivation policy

0

1

2

×10 1

Feature rank policy (PCA)

10 3

10 7

10 11

Capacity loss policy

Hopper

0.0

0.5

1.0

×10 3

Adam equal betas

Share actor and critic features

Regularize all preactivations

Regularize last preactivation

No invervention

Episode return

10 8

10 17

Excess ratio

0.0

2.5

5.0

×10 1

Dead neurons policy

10 3

10 6

10 9

Norm preactivation policy

0

2

×10 1

Feature rank policy (PCA)

10 4

10 10

10 16

Capacity loss policy

Mean

Median

Outliers

Humanoid

-2

0

2

×10 3

Adam equal betas

Share actor and critic features

Regularize all preactivations

Regularize last preactivation

No invervention

Episode return

10 8

10 17

Excess ratio

0

2

4

×10 1

Dead neurons policy

10 2

10 4

Norm preactivation policy

0

2

4

×10 1

Feature rank policy (PCA)

10 5

10 11

Capacity loss policy

Mean

Median

Outliers

HalfCheetah

-2.5

0.0

2.5

×10 3

Adam equal betas

Share actor and critic features

Regularize all preactivations

Regularize last preactivation

No invervention

Episode return

10 8

10 17

Excess ratio

0.0

2.5

5.0

×10 1

Dead neurons policy

10 2

10 4

Norm preactivation policy

0

2

4

×10 1

Feature rank policy (PCA)

10 4

10 8

Capacity loss policy

Mean

Median

Outliers

Ant

Median

Outliers

0.0

0.5

1.0

Environment steps×10 8

0

2

4

6

8

Episode return

×10 3

ALE/Phoenix-v5,

4 epochs

ALE/Phoenix-v5,

4 epochs

ALE/Phoenix-v5,

4 epochs

ALE/Phoenix-v5,

4 epochs

ALE/Phoenix-v5,

4 epochs

ALE/Phoenix-v5,

4 epochs

0.0

0.5

1.0

Environment steps×10 8

0

1

2

3

4

Feature rank policy (PCA)

×10 2

0.0

0.5

1.0

Environment steps×10 8

10 0

10 1

10 2

10 3

10 4

Norm preactivation policy

0.0

0.5

1.0

Environment steps×10 8

0

1

2

3

4

Capacity loss policy

0.0

0.5

1.0

Environment steps×10 8

0.0

0.2

0.4

0.6

0.8

1.0

Feature rank critic (PCA)

×10 2

0.0

0.5

1.0

Environment steps×10 8

0

1

2

3

Capacity loss critic

Adam equal betas

No intervention

Regularize last preactivation

Regularize all preactivations

Share actor and critic features

0.0

0.5

1.0

Environment steps×10 8

1

2

3

4

5

Episode return

×10 3

ALE/Phoenix-v5,

6 epochs

ALE/Phoenix-v5,

6 epochs

ALE/Phoenix-v5,

6 epochs

ALE/Phoenix-v5,

6 epochs

ALE/Phoenix-v5,

6 epochs

ALE/Phoenix-v5,

6 epochs

0.0

0.5

1.0

Environment steps×10 8

0

1

2

3

4

Feature rank policy (PCA)

×10 2

0.0

0.5

1.0

Environment steps×10 8

10 0

10 1

10 2

10 3

10 4

Norm preactivation policy

0.0

0.5

1.0

Environment steps×10 8

0.0

0.2

0.4

0.6

0.8

1.0

Capacity loss policy

0.0

0.5

1.0

Environment steps×10 8

0

2

4

6

8

Feature rank critic (PCA)

×10 1

0.0

0.5

1.0

Environment steps×10 8

0

1

2

3

4

Capacity loss critic

Adam equal betas

No intervention

Regularize last preactivation

Regularize all preactivations

Share actor and critic features

0.0

0.5

1.0

Environment steps×10 8

0

1

2

3

4

5

Episode return

×10 3

ALE/Phoenix-v5,

8 epochs

ALE/Phoenix-v5,

8 epochs

ALE/Phoenix-v5,

8 epochs

ALE/Phoenix-v5,

8 epochs

ALE/Phoenix-v5,

8 epochs

ALE/Phoenix-v5,

8 epochs

0.0

0.5

1.0

Environment steps×10 8

0

1

2

3

4

Feature rank policy (PCA)

×10 2

0.0

0.5

1.0

Environment steps×10 8

10 0

10 1

10 2

10 3

10 4

Norm preactivation policy

0.0

0.5

1.0

Environment steps×10 8

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Capacity loss policy

0.0

0.5

1.0

Environment steps×10 8

0

2

4

6

8

Feature rank critic (PCA)

×10 1

0.0

0.5

1.0

Environment steps×10 8

0

2

4

6

Capacity loss critic

Adam equal betas

No intervention

Regularize last preactivation

Regularize all preactivations

Share actor and critic features

0.0

0.5

1.0

Environment steps×10 8

2

4

6

8

Episode return

×10 3

ALE/NameThisGame-v5,

4 epochs

ALE/NameThisGame-v5,

4 epochs

ALE/NameThisGame-v5,

4 epochs

ALE/NameThisGame-v5,

4 epochs

ALE/NameThisGame-v5,

4 epochs

ALE/NameThisGame-v5,

4 epochs

0.0

0.5

1.0

Environment steps×10 8

0

1

2

3

Feature rank policy (PCA)

×10 2

0.0

0.5

1.0

Environment steps×10 8

10 0

10 1

10 2

10 3

Norm preactivation policy

0.0

0.5

1.0

Environment steps×10 8

0.0

0.5

1.0

1.5

2.0

2.5

Capacity loss policy

0.0

0.5

1.0

Environment steps×10 8

0.0

0.5

1.0

1.5

2.0

2.5

Feature rank critic (PCA)

×10 1

0.0

0.5

1.0

Environment steps×10 8

0.0

0.5

1.0

1.5

2.0

2.5

Capacity loss critic

Adam equal betas

No intervention

Regularize last preactivation

Regularize all preactivations

Share actor and critic features

0.0

0.5

1.0

Environment steps×10 8

2

4

6

8

Episode return

×10 3

ALE/NameThisGame-v5,

6 epochs

ALE/NameThisGame-v5,

6 epochs

ALE/NameThisGame-v5,

6 epochs

ALE/NameThisGame-v5,

6 epochs

ALE/NameThisGame-v5,

6 epochs

ALE/NameThisGame-v5,

6 epochs

0.0

0.5

1.0

Environment steps×10 8

0

1

2

3

Feature rank policy (PCA)

×10 2

0.0

0.5

1.0

Environment steps×10 8

10 0

10 1

10 2

10 3

10 4

Norm preactivation policy

0.0

0.5

1.0

Environment steps×10 8

0.0

0.5

1.0

1.5

2.0

Capacity loss policy

0.0

0.5

1.0

Environment steps×10 8

0

1

2

3

4

Feature rank critic (PCA)

×10 1

0.0

0.5

1.0

Environment steps×10 8

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Capacity loss critic

Adam equal betas

No intervention

Regularize last preactivation

Regularize all preactivations

Share actor and critic features

0.0

0.5

1.0

Environment steps ×10 8

1

2

3

4

5

6

Episode return

×10 3

ALE/NameThisGame-v5,

8 epochs

ALE/NameThisGame-v5,

8 epochs

ALE/NameThisGame-v5,

8 epochs

ALE/NameThisGame-v5,

8 epochs

ALE/NameThisGame-v5,

8 epochs

ALE/NameThisGame-v5,

8 epochs

0.0

0.5

1.0

Environment steps ×10 8

0

1

2

3

Feature rank policy (PCA)

×10 2

0.0

0.5

1.0

Environment steps ×10 8

10 0

10 1

10 2

10 3

10 4

Norm preactivation policy

0.0

0.5

1.0

Environment steps ×10 8

0.00

0.25

0.50

0.75

1.00

1.25

1.50

Capacity loss policy

0.0

0.5

1.0

Environment steps ×10 8

0

1

2

3

4

Feature rank critic (PCA)

×10 1

0.0

0.5

1.0

Environment steps ×10 8

0.0

0.5

1.0

1.5

Capacity loss critic

Adam equal betas

No intervention

Regularize last preactivation

Regularize all preactivations

Share actor and critic features

0.0

0.5

1.0

Environment steps ×10 8

0.00

0.25

0.50

0.75

1.00

1.25

1.50

Episode return

×10 3

ALE/Gravitar-v5,

4 epochs

ALE/Gravitar-v5,

4 epochs

ALE/Gravitar-v5,

4 epochs

ALE/Gravitar-v5,

4 epochs

ALE/Gravitar-v5,

4 epochs

ALE/Gravitar-v5,

4 epochs

0.0

0.5

1.0

Environment steps ×10 8

0

1

2

3

Feature rank policy (PCA)

×10 2

0.0

0.5

1.0

Environment steps ×10 8

10 0

10 1

10 2

10 3

10 4

Norm preactivation policy

0.0

0.5

1.0

Environment steps ×10 8

0

2

4

6

Capacity loss policy

0.0

0.5

1.0

Environment steps ×10 8

0.0

0.5

1.0

1.5

2.0

Feature rank critic (PCA)

×10 2

0.0

0.5

1.0

Environment steps ×10 8

0

1

2

3

4

Capacity loss critic

Adam equal betas

No intervention

Regularize last preactivation

Regularize all preactivations

Share actor and critic features

0.0

0.5

1.0

Environment steps×10 8

0

2

4

6

Episode return

×10 2

ALE/Gravitar-v5,

6 epochs

ALE/Gravitar-v5,

6 epochs

ALE/Gravitar-v5,

6 epochs

ALE/Gravitar-v5,

6 epochs

ALE/Gravitar-v5,

6 epochs

ALE/Gravitar-v5,

6 epochs

0.0

0.5

1.0

Environment steps×10 8

0

1

2

3

Feature rank policy (PCA)

×10 2

0.0

0.5

1.0

Environment steps×10 8

10 0

10 1

10 2

10 3

Norm preactivation policy

0.0

0.5

1.0

Environment steps×10 8

0

1

2

3

4

5

6

Capacity loss policy

0.0

0.5

1.0

Environment steps×10 8

0.0

0.5

1.0

1.5

2.0

Feature rank critic (PCA)

×10 2

0.0

0.5

1.0

Environment steps×10 8

0

1

2

3

Capacity loss critic

Adam equal betas

No intervention

Regularize last preactivation

Regularize all preactivations

Share actor and critic features

0.0

0.5

1.0

Environment steps×10 8

0

2

4

6

8

Episode return

×10 2

ALE/Gravitar-v5,

8 epochs

ALE/Gravitar-v5,

8 epochs

ALE/Gravitar-v5,

8 epochs

ALE/Gravitar-v5,

8 epochs

ALE/Gravitar-v5,

8 epochs

ALE/Gravitar-v5,

8 epochs

0.0

0.5

1.0

Environment steps×10 8

0

1

2

3

Feature rank policy (PCA)

×10 2

0.0

0.5

1.0

Environment steps×10 8

10 0

10 1

10 2

10 3

10 4

Norm preactivation policy

0.0

0.5

1.0

Environment steps×10 8

0.0

0.2

0.4

0.6

0.8

1.0

Capacity loss policy

0.0

0.5

1.0

Environment steps×10 8

0.0

0.5

1.0

1.5

Feature rank critic (PCA)

×10 2

0.0

0.5

1.0

Environment steps×10 8

0.0

0.5

1.0

1.5

2.0

Capacity loss critic

Adam equal betas

No intervention

Regularize last preactivation

Regularize all preactivations

Share actor and critic features

0

2

4

Environment steps×10 6

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Episode return

×10 3

Hopper-v4

10 epochs

Hopper-v4

10 epochs

Hopper-v4

10 epochs

Hopper-v4

10 epochs

Hopper-v4

10 epochs

Hopper-v4

10 epochs

0

2

4

Environment steps×10 6

0

1

2

3

4

5

Feature rank policy (PCA)

×10 1

0

2

4

Environment steps×10 6

10 1

10 2

Norm preactivation policy

0

2

4

Environment steps×10 6

0.0

0.5

1.0

1.5

2.0

Capacity loss policy

×10 8

0

2

4

Environment steps×10 6

0

1

2

3

4

Feature rank critic (PCA)

×10 1

0

2

4

Environment steps×10 6

0

1

2

3

4

5

6

Capacity loss critic

×10 4

Adam equal betas

No intervention

Regularize last preactivation

Regularize all preactivations

Share actor and critic features

0

2

4

Environment steps×10 6

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Episode return

×10 3

Hopper-v4

15 epochs

Hopper-v4

15 epochs

Hopper-v4

15 epochs

Hopper-v4

15 epochs

Hopper-v4

15 epochs

Hopper-v4

15 epochs

0

2

4

Environment steps×10 6

0

1

2

3

4

5

Feature rank policy (PCA)

×10 1

0

2

4

Environment steps×10 6

10 1

10 2

Norm preactivation policy

0

2

4

Environment steps×10 6

0.0

0.5

1.0

1.5

2.0

Capacity loss policy

×10 8

0

2

4

Environment steps×10 6

0

1

2

3

4

Feature rank critic (PCA)

×10 1

0

2

4

Environment steps×10 6

0

1

2

3

4

5

Capacity loss critic

×10 4

Adam equal betas

No intervention

Regularize last preactivation

Regularize all preactivations

Share actor and critic features

0

2

4

Environment steps×10 6

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Episode return

×10 3

Hopper-v4

20 epochs

Hopper-v4

20 epochs

Hopper-v4

20 epochs

Hopper-v4

20 epochs

Hopper-v4

20 epochs

Hopper-v4

20 epochs

0

2

4

Environment steps×10 6

0

1

2

3

4

5

Feature rank policy (PCA)

×10 1

0

2

4

Environment steps×10 6

10 1

10 2

Norm preactivation policy

0

2

4

Environment steps×10 6

0.0

0.5

1.0

1.5

2.0

Capacity loss policy

×10 8

0

2

4

Environment steps×10 6

0

1

2

3

4

Feature rank critic (PCA)

×10 1

0

2

4

Environment steps×10 6

0

1

2

3

4

5

Capacity loss critic

×10 4

Adam equal betas

No intervention

Regularize last preactivation

Regularize all preactivations

Share actor and critic features

0

2

4

Environment steps×10 6

0.0

0.5

1.0

1.5

2.0

Episode return

×10 3

Humanoid-v4

10 epochs

Humanoid-v4

10 epochs

Humanoid-v4

10 epochs

Humanoid-v4

10 epochs

Humanoid-v4

10 epochs

Humanoid-v4

10 epochs

0

2

4

Environment steps×10 6

0

1

2

3

4

5

6

Feature rank policy (PCA)

×10 1

0

2

4

Environment steps×10 6

10 1

10 2

Norm preactivation policy

0

2

4

Environment steps×10 6

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Capacity loss policy

×10 8

0

2

4

Environment steps×10 6

0

1

2

3

4

5

Feature rank critic (PCA)

×10 1

0

2

4

Environment steps×10 6

0.0

0.2

0.4

0.6

0.8

1.0

Capacity loss critic

×10 4

Adam equal betas

No intervention

Regularize last preactivation

Regularize all preactivations

Share actor and critic features

0

2

4

Environment steps ×10 6

0.2

0.4

0.6

0.8

1.0

Episode return

×10 3

Humanoid-v4

15 epochs

Humanoid-v4

15 epochs

Humanoid-v4

15 epochs

Humanoid-v4

15 epochs

Humanoid-v4

15 epochs

Humanoid-v4

15 epochs

0

2

4

Environment steps ×10 6

0

1

2

3

4

5

6

Feature rank policy (PCA)

×10 1

0

2

4

Environment steps ×10 6

10 1

10 2

Norm preactivation policy

0

2

4

Environment steps ×10 6

0.00

0.25

0.50

0.75

1.00

1.25

Capacity loss policy

×10 8

0

2

4

Environment steps ×10 6

0

1

2

3

4

5

Feature rank critic (PCA)

×10 1

0

2

4

Environment steps ×10 6

0.00

0.25

0.50

0.75

1.00

1.25

1.50

Capacity loss critic

×10 4

Adam equal betas

No intervention

Regularize last preactivation

Regularize all preactivations

Share actor and critic features

0

2

4

Environment steps ×10 6

2

4

6

8

Episode return

×10 2

Humanoid-v4

20 epochs

Humanoid-v4

20 epochs

Humanoid-v4

20 epochs

Humanoid-v4

20 epochs

Humanoid-v4

20 epochs

Humanoid-v4

20 epochs

0

2

4

Environment steps ×10 6

0

1

2

3

4

5

6

Feature rank policy (PCA)

×10 1

0

2

4

Environment steps ×10 6

10 1

10 2

Norm preactivation policy

0

2

4

Environment steps ×10 6

0.00

0.25

0.50

0.75

1.00

1.25

1.50

Capacity loss policy

×10 8

0

2

4

Environment steps ×10 6

0

1

2

3

4

5

Feature rank critic (PCA)

×10 1

0

2

4

Environment steps ×10 6

0

2

4

6

8

Capacity loss critic

×10 3

Adam equal betas

No intervention

Regularize last preactivation

Regularize all preactivations

Share actor and critic features

0

2

4

Environment steps×10 6

0

2

4

6

Episode return

×10 3

HalfCheetah-v4

10 epochs

HalfCheetah-v4

10 epochs

HalfCheetah-v4

10 epochs

HalfCheetah-v4

10 epochs

HalfCheetah-v4

10 epochs

HalfCheetah-v4

10 epochs

0

2

4

Environment steps×10 6

0

1

2

3

4

5

Feature rank policy (PCA)

×10 1

0

2

4

Environment steps×10 6

10 1

10 2

Norm preactivation policy

0

2

4

Environment steps×10 6

0.0

0.5

1.0

1.5

2.0

Capacity loss policy

×10 8

0

2

4

Environment steps×10 6

0.5

1.0

1.5

2.0

Feature rank critic (PCA)

×10 1

0

2

4

Environment steps×10 6

0

1

2

3

Capacity loss critic

×10 4

Adam equal betas

No intervention

Regularize last preactivation

Regularize all preactivations

Share actor and critic features

0

2

4

Environment steps ×10 6

0

2

4

6

Episode return

×10 3

HalfCheetah-v4

15 epochs

HalfCheetah-v4

15 epochs

HalfCheetah-v4

15 epochs

HalfCheetah-v4

15 epochs

HalfCheetah-v4

15 epochs

HalfCheetah-v4

15 epochs

0

2

4

Environment steps ×10 6

0

1

2

3

4

5

Feature rank policy (PCA)

×10 1

0

2

4

Environment steps ×10 6

10 1

10 2

Norm preactivation policy

0

2

4

Environment steps ×10 6

0.00

0.25

0.50

0.75

1.00

1.25

1.50

Capacity loss policy

×10 9

0

2

4

Environment steps ×10 6

0.0

0.5

1.0

1.5

2.0

Feature rank critic (PCA)

×10 1

0

2

4

Environment steps ×10 6

0

1

2

3

4

5

Capacity loss critic

×10 4

Adam equal betas

No intervention

Regularize last preactivation

Regularize all preactivations

Share actor and critic features

0

2

4

Environment steps ×10 6

-1

0

1

2

3

4

5

Episode return

×10 3

HalfCheetah-v4

20 epochs

HalfCheetah-v4

20 epochs

HalfCheetah-v4

20 epochs

HalfCheetah-v4

20 epochs

HalfCheetah-v4

20 epochs

HalfCheetah-v4

20 epochs

0

2

4

Environment steps ×10 6

0

1

2

3

4

5

6

Feature rank policy (PCA)

×10 1

0

2

4

Environment steps ×10 6

10 1

10 2

Norm preactivation policy

0

2

4

Environment steps ×10 6

0.00

0.25

0.50

0.75

1.00

1.25

1.50

Capacity loss policy

×10 9

0

2

4

Environment steps ×10 6

0.0

0.5

1.0

1.5

2.0

2.5

Feature rank critic (PCA)

×10 1

0

2

4

Environment steps ×10 6

0

1

2

3

4

Capacity loss critic

×10 4

Adam equal betas

No intervention

Regularize last preactivation

Regularize all preactivations

Share actor and critic features

0

2

4

Environment steps×10 6

0

1

2

3

4

5

Episode return

×10 3

Ant-v4

10 epochs

Ant-v4

10 epochs

Ant-v4

10 epochs

Ant-v4

10 epochs

Ant-v4

10 epochs

Ant-v4

10 epochs

0

2

4

Environment steps×10 6

0

1

2

3

4

5

Feature rank policy (PCA)

×10 1

0

2

4

Environment steps×10 6

10 1

10 2

Norm preactivation policy

0

2

4

Environment steps×10 6

0.0

0.5

1.0

1.5

2.0

2.5

Capacity loss policy

×10 3

0

2

4

Environment steps×10 6

0

1

2

3

4

5

Feature rank critic (PCA)

×10 1

0

2

4

Environment steps×10 6

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Capacity loss critic

×10 4

Adam equal betas

No intervention

Regularize last preactivation

Regularize all preactivations

Share actor and critic features

0

2

4

Environment steps ×10 6

-2

-1

0

1

2

3

Episode return

×10 3

Ant-v4

15 epochs

Ant-v4

15 epochs

Ant-v4

15 epochs

Ant-v4

15 epochs

Ant-v4

15 epochs

Ant-v4

15 epochs

0

2

4

Environment steps ×10 6

0

1

2

3

4

5

6

Feature rank policy (PCA)

×10 1

0

2

4

Environment steps ×10 6

10 1

10 2

Norm preactivation policy

0

2

4

Environment steps ×10 6

0.00

0.25

0.50

0.75

1.00

1.25

Capacity loss policy

×10 8

0

2

4

Environment steps ×10 6

0

1

2

3

4

5

Feature rank critic (PCA)

×10 1

0

2

4

Environment steps ×10 6

0

1

2

3

4

Capacity loss critic

×10 4

Adam equal betas

No intervention

Regularize last preactivation

Regularize all preactivations

Share actor and critic features

0

2

4

Environment steps×10 6

-2

-1

0

1

2

3

Episode return

×10 3

Ant-v4

20 epochs

Ant-v4

20 epochs

Ant-v4

20 epochs

Ant-v4

20 epochs

Ant-v4

20 epochs

Ant-v4

20 epochs

0

2

4

Environment steps×10 6

1

2

3

4

5

6

Feature rank policy (PCA)

×10 1

0

2

4

Environment steps×10 6

10 1

10 2

Norm preactivation policy

0

2

4

Environment steps×10 6

0.0

0.5

1.0

1.5

2.0

2.5

Capacity loss policy

×10 9

0

2

4

Environment steps×10 6

0

1

2

3

4

5

Feature rank critic (PCA)

×10 1

0

2

4

Environment steps×10 6

0

1

2

3

4

Capacity loss critic

×10 4

Adam equal betas

No intervention

Regularize last preactivation

Regularize all preactivations

Share actor and critic features

0

2

4

Environment steps×10 6

0.0

0.5

1.0

1.5

2.0

2.5

Episode return

×10 3

Hopper-v4

10 epochs

Hopper-v4

10 epochs

Hopper-v4

10 epochs

Hopper-v4

10 epochs

Hopper-v4

10 epochs

Hopper-v4

10 epochs

0

2

4

Environment steps×10 6

0

1

2

3

Feature rank policy (PCA)

×10 1

0

2

4

Environment steps×10 6

10 1

10 2

10 3

Norm preactivation policy

0

2

4

Environment steps×10 6

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Capacity loss policy

×10 10

0

2

4

Environment steps×10 6

2

4

6

8

Feature rank critic (PCA)

0

2

4

Environment steps×10 6

0.0

0.2

0.4

0.6

0.8

1.0

Capacity loss critic

×10 4

Adam equal betas

No intervention

Regularize last preactivation

Regularize all preactivations

Share actor and critic features

0

2

4

Environment steps×10 6

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Episode return

×10 3

Hopper-v4

15 epochs

Hopper-v4

15 epochs

Hopper-v4

15 epochs

Hopper-v4

15 epochs

Hopper-v4

15 epochs

Hopper-v4

15 epochs

0

2

4

Environment steps×10 6

0

1

2

3

4

Feature rank policy (PCA)

×10 1

0

2

4

Environment steps×10 6

10 1

10 2

10 3

Norm preactivation policy

0

2

4

Environment steps×10 6

0

2

4

6

8

Capacity loss policy

×10 9

0

2

4

Environment steps×10 6

0.2

0.4

0.6

0.8

1.0

1.2

Feature rank critic (PCA)

×10 1

0

2

4

Environment steps×10 6

0

2

4

6

8

Capacity loss critic

×10 3

Adam equal betas

No intervention

Regularize last preactivation

Regularize all preactivations

Share actor and critic features

0

2

4

Environment steps×10 6

0.0

0.5

1.0

1.5

2.0

2.5

Episode return

×10 3

Hopper-v4

20 epochs

Hopper-v4

20 epochs

Hopper-v4

20 epochs

Hopper-v4

20 epochs

Hopper-v4

20 epochs

Hopper-v4

20 epochs

0

2

4

Environment steps×10 6

0

1

2

3

4

Feature rank policy (PCA)

×10 1

0

2

4

Environment steps×10 6

10 1

10 2

10 3

Norm preactivation policy

0

2

4

Environment steps×10 6

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Capacity loss policy

×10 11

0

2

4

Environment steps×10 6

0.2

0.4

0.6

0.8

1.0

Feature rank critic (PCA)

×10 1

0

2

4

Environment steps×10 6

0

2

4

6

Capacity loss critic

×10 3

Adam equal betas

No intervention

Regularize last preactivation

Regularize all preactivations

Share actor and critic features

0

2

4

Environment steps ×10 6

0.2

0.4

0.6

0.8

1.0

1.2

Episode return

×10 3

Humanoid-v4

10 epochs

Humanoid-v4

10 epochs

Humanoid-v4

10 epochs

Humanoid-v4

10 epochs

Humanoid-v4

10 epochs

Humanoid-v4

10 epochs

0

2

4

Environment steps ×10 6

0

1

2

3

4

5

Feature rank policy (PCA)

×10 1

0

2

4

Environment steps ×10 6

10 1

10 3

10 5

10 7

10 9

Norm preactivation policy

0

2

4

Environment steps ×10 6

0

1

2

3

4

Capacity loss policy

×10 17

0

2

4

Environment steps ×10 6

0.25

0.50

0.75

1.00

1.25

Feature rank critic (PCA)

×10 1

0

2

4

Environment steps ×10 6

0.0

0.5

1.0

1.5

2.0

Capacity loss critic

×10 17

Adam equal betas

No intervention

Regularize last preactivation

Regularize all preactivations

Share actor and critic features

0

2

4

Environment steps×10 6

0.0

0.5

1.0

1.5

Episode return

×10 3

Humanoid-v4

15 epochs

Humanoid-v4

15 epochs

Humanoid-v4

15 epochs

Humanoid-v4

15 epochs

Humanoid-v4

15 epochs

Humanoid-v4

15 epochs

0

2

4

Environment steps×10 6

0

1

2

3

4

5

Feature rank policy (PCA)

×10 1

0

2

4

Environment steps×10 6

10 1

10 3

10 5

10 7

10 9

Norm preactivation policy

0

2

4

Environment steps×10 6

0

2

4

6

Capacity loss policy

×10 14

0

2

4

Environment steps×10 6

2

4

6

8

Feature rank critic (PCA)

0

2

4

Environment steps×10 6

0

2

4

6

8

Capacity loss critic

×10 17

Adam equal betas

No intervention

Regularize last preactivation

Regularize all preactivations

Share actor and critic features

0

2

4

Environment steps ×10 6

2

4

6

8

Episode return

×10 2

Humanoid-v4

20 epochs

Humanoid-v4

20 epochs

Humanoid-v4

20 epochs

Humanoid-v4

20 epochs

Humanoid-v4

20 epochs

Humanoid-v4

20 epochs

0

2

4

Environment steps ×10 6

0

1

2

3

4

5

Feature rank policy (PCA)

×10 1

0

2

4

Environment steps ×10 6

10 1

10 3

10 5

10 7

10 9

Norm preactivation policy

0

2

4

Environment steps ×10 6

0

1

2

3

4

5

Capacity loss policy

×10 13

0

2

4

Environment steps ×10 6

0.25

0.50

0.75

1.00

1.25

Feature rank critic (PCA)

×10 1

0

2

4

Environment steps ×10 6

0.00

0.25

0.50

0.75

1.00

1.25

Capacity loss critic

×10 20

Adam equal betas

No intervention

Regularize last preactivation

Regularize all preactivations

Share actor and critic features

0

2

4

Environment steps ×10 6

0

2

4

Episode return

×10 3

HalfCheetah-v4

10 epochs

HalfCheetah-v4

10 epochs

HalfCheetah-v4

10 epochs

HalfCheetah-v4

10 epochs

HalfCheetah-v4

10 epochs

HalfCheetah-v4

10 epochs

0

2

4

Environment steps ×10 6

0

1

2

3

4

5

Feature rank policy (PCA)

×10 1

0

2

4

Environment steps ×10 6

10 1

10 2

10 3

10 4

Norm preactivation policy

0

2

4

Environment steps ×10 6

0

2

4

6

Capacity loss policy

×10 11

0

2

4

Environment steps ×10 6

0.25

0.50

0.75

1.00

1.25

1.50

Feature rank critic (PCA)

×10 1

0

2

4

Environment steps ×10 6

0.00

0.25

0.50

0.75

1.00

1.25

Capacity loss critic

×10 4

Adam equal betas

No intervention

Regularize last preactivation

Regularize all preactivations

Share actor and critic features

0

2

4

Environment steps ×10 6

-1

0

1

2

3

4

5

Episode return

×10 3

HalfCheetah-v4

15 epochs

HalfCheetah-v4

15 epochs

HalfCheetah-v4

15 epochs

HalfCheetah-v4

15 epochs

HalfCheetah-v4

15 epochs

HalfCheetah-v4

15 epochs

0

2

4

Environment steps ×10 6

0

1

2

3

4

5

Feature rank policy (PCA)

×10 1

0

2

4

Environment steps ×10 6

10 1

10 2

10 3

10 4

Norm preactivation policy

0

2

4

Environment steps ×10 6

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Capacity loss policy

×10 12

0

2

4

Environment steps ×10 6

0.25

0.50

0.75

1.00

1.25

1.50

Feature rank critic (PCA)

×10 1

0

2

4

Environment steps ×10 6

0

1

2

3

4

5

Capacity loss critic

×10 3

Adam equal betas

No intervention

Regularize last preactivation

Regularize all preactivations

Share actor and critic features

0

2

4

Environment steps×10 6

-2

0

2

4

Episode return

×10 3

HalfCheetah-v4

20 epochs

HalfCheetah-v4

20 epochs

HalfCheetah-v4

20 epochs

HalfCheetah-v4

20 epochs

HalfCheetah-v4

20 epochs

HalfCheetah-v4

20 epochs

0

2

4

Environment steps×10 6

0

1

2

3

4

5

Feature rank policy (PCA)

×10 1

0

2

4

Environment steps×10 6

10 1

10 2

10 3

10 4

Norm preactivation policy

0

2

4

Environment steps×10 6

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Capacity loss policy

×10 15

0

2

4

Environment steps×10 6

0.5

1.0

1.5

Feature rank critic (PCA)

×10 1

0

2

4

Environment steps×10 6

0

1

2

3

4

5

6

Capacity loss critic

×10 3

Adam equal betas

No intervention

Regularize last preactivation

Regularize all preactivations

Share actor and critic features

0

2

4

Environment steps×10 6

-2

0

2

4

Episode return

×10 3

Ant-v4

10 epochs

Ant-v4

10 epochs

Ant-v4

10 epochs

Ant-v4

10 epochs

Ant-v4

10 epochs

Ant-v4

10 epochs

0

2

4

Environment steps×10 6

0

1

2

3

4

5

Feature rank policy (PCA)

×10 1

0

2

4

Environment steps×10 6

10 1

10 2

10 3

Norm preactivation policy

0

2

4

Environment steps×10 6

0

1

2

3

4

Capacity loss policy

×10 10

0

2

4

Environment steps×10 6

0.2

0.4

0.6

0.8

1.0

Feature rank critic (PCA)

×10 1

0

2

4

Environment steps×10 6

0.0

0.5

1.0

1.5

Capacity loss critic

×10 5

Adam equal betas

No intervention

Regularize last preactivation

Regularize all preactivations

Share actor and critic features

0

2

4

Environment steps×10 6

-3

-2

-1

0

1

2

3

Episode return

×10 3

Ant-v4

15 epochs

Ant-v4

15 epochs

Ant-v4

15 epochs

Ant-v4

15 epochs

Ant-v4

15 epochs

Ant-v4

15 epochs

0

2

4

Environment steps×10 6

0

1

2

3

4

5

Feature rank policy (PCA)

×10 1

0

2

4

Environment steps×10 6

10 1

10 2

10 3

Norm preactivation policy

0

2

4

Environment steps×10 6

0.0

0.2

0.4

0.6

0.8

1.0

Capacity loss policy

×10 11

0

2

4

Environment steps×10 6

0.2

0.4

0.6

0.8

1.0

1.2

Feature rank critic (PCA)

×10 1

0

2

4

Environment steps×10 6

0

2

4

6

8

Capacity loss critic

×10 4

Adam equal betas

No intervention

Regularize last preactivation

Regularize all preactivations

Share actor and critic features

0

2

4

Environment steps ×10 6

-3

-2

-1

0

1

2

Episode return

×10 3

Ant-v4

20 epochs

Ant-v4

20 epochs

Ant-v4

20 epochs

Ant-v4

20 epochs

Ant-v4

20 epochs

Ant-v4

20 epochs

0

2

4

Environment steps ×10 6

0

1

2

3

4

5

Feature rank policy (PCA)

×10 1

0

2

4

Environment steps ×10 6

10 1

10 2

10 3

10 4

Norm preactivation policy

0

2

4

Environment steps ×10 6

0

2

4

6

8

Capacity loss policy

×10 10

0

2

4

Environment steps ×10 6

0.25

0.50

0.75

1.00

1.25

1.50

Feature rank critic (PCA)

×10 1

0

2

4

Environment steps ×10 6

0

2

4

6

8

Capacity loss critic

×10 4

Adam equal betas

No intervention

Regularize last preactivation

Regularize all preactivations

Share actor and critic features

0

1

2

Environment steps×10 8

2

4

6

Episode return

×10 3

ALE/Phoenix-v5

0

1

2

Environment steps×10 8

0

1

2

3

Feature rank policy (PCA)

×10 2

0

1

2

Environment steps×10 8

10 0

10 1

10 2

10 3

10 4

Norm preactivation policy

0

1

2

Environment steps×10 8

1.0

1.2

1.4

1.6

1.8

2.0

Entropy

Baselines

PFO

0

1

2

Environment steps×10 8

2

3

4

5

6

Episode return

×10 3

ALE/NameThisGame-v5

0

1

2

Environment steps×10 8

0.0

0.5

1.0

1.5

Feature rank policy (PCA)

×10 2

0

1

2

Environment steps×10 8

10 0

10 1

10 2

10 3

10 4

Norm preactivation policy

0

1

2

Environment steps×10 8

0.6

0.8

1.0

1.2

1.4

1.6

1.8

Entropy

Seed = 7

Seed = 25

Seed = 64

0

1

2

Environment steps×10 8

0

2

4

6

8

Episode return

×10 3

ALE/Phoenix-v5

0

1

2

Environment steps×10 8

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Feature rank policy (PCA)

×10 2

0

1

2

Environment steps×10 8

10 0

10 1

10 2

10 3

10 4

Norm preactivation policy

0

1

2

Environment steps×10 8

0.5

1.0

1.5

2.0

Entropy

ε

= 0.05

ε

= 0.1

ε

= 0.2

0.0

0.5

1.0

Environment steps×10 8

0.5

1.0

1.5

2.0

2.5

3.0

Episode return

×10 3

ALE/Phoenix-v5

0.0

0.5

1.0

Environment steps×10 8

0

1

2

3

Feature rank policy (PCA)

×10 2

0.0

0.5

1.0

Environment steps×10 8

0.0

0.5

1.0

1.5

2.0

2.5

Feature rank critic (PCA)

×10 2

0.0

0.5

1.0

Environment steps×10 8

1.2

1.4

1.6

1.8

2.0

Entropy

Separate networks

Shared trunk

E Measuring and comparing rank dynamics

Several matrix rank approximations have been used in the deep learning literature, and more specifically the deep RL literature, to measure the rank of the representation of features learned by a deep network. In complement to the background presented in section 2, we give here all the rank metrics we have tracked in this work and their correlations, showing that although their absolute values differ, their dynamics tend to describe the same evolution.

E.1 Definitions of different rank metrics

Essentially, the main difference between the rank metrics considered in the literature is whether they apply a relative thresholding of the singular values or an absolute one. Their implementation can be found under src/po_dynamics/modules/metrics.py in our codebase.

Referring by Φ the N × D matrix of representations as in Section 2, and letting δ = 0 . 01 be the threshold, and ⟨ σ i (Φ) , . . . , σ D (Φ) ⟩ the singular values of Φ in decreasing order, the different rank definitions are as follows.

Effective rank (Roy & Vetterli, 2007) A relative measure of the rank. Let H ( p 1 , . . . , p k ) denote the Shannon entropy of a probability distribution over k events and ∥ σ ∥ 1 be the sum of the singular values. Let ˜ σ i (Φ) = σ i (Φ) ∥ σ ∥ 1 be the normalized singular values. The effective rank is

This rank measure has also been used in deep learning by Huh et al. (2023).

Approximate rank (PCA) A relative measure of the rank. Intuitively this rank measures the number of PCA values that together explain 99% of the variance of the matrix. This can also be viewed as the lowest-rank reconstruction of the feature matrix with an error lower than 1%. 12 It is also used in RL by Yang et al. (2020).

srank (Kumar et al., 2021) A relative measure of the rank. This is a relative thresholding of the singular values, similar to the approximate rank but with no connection to low-rank reconstruction or variance of the feature matrix.

Feature Rank (Lyle et al., 2022) An absolute measure of the rank. The number of singular values of the normalized Φ that are larger than a threshold δ .

PyTorch rank An absolute measure of the rank. This is the rank computed by torch.linalg.matrix_rank and torch.linalg.matrix_rank . Let ϵ be the smallest difference possible between points of the data type of the singular values, i.e. for torch.float32 that is 1 . 19209 e -7 . This rank is computed as follows.

It also appears in Press et al. (2007) in the discussion of SVD solutions for linear least squares.

E.2 Correlations between the rank metrics

We compute various correlation coefficients and distance measures between the rank metrics. To compute a correlation/distance on a pair of rank metrics ( X,Y ) , we take for each training run the set { ( x t , y t ) t ∈ { 0 , ..., T }} of coinciding values of the curves of the two rank metrics during the run that had T logged steps, compute the correlation/distance on this set, and average the correlation/distance values across all considered runs. We also compute the worst correlation/distance between each rank metric pair for a worst-case analysis. We separate the average values and worst-case values by environment (ALE vs. MuJoCo) for a more granular analysis. We consider all the runs without the interventions and exclude a few runs where the models collapse since the beginning of training, giving constant trivial ranks, as these result in undefined or trivial correlation coefficients.

We compute Kendall's τ coefficient (Kendall, 1938), Spearman's ρ coefficient (Spearman, 1987), the Pearson correlation coefficient, and a normalized L2-distance computed as √ ∑ T t =1 ( x t -y t ) 2 √ T × L where L is the width of the feature layer considered (i.e., 512 for ALE and 64 for MuJoCo).

Results We visualize the correlation/distance between the pairs of ranks as heatmaps annotated with averages and standard deviations. Overall, the metrics are highly correlated with average correlation the coefficients varying between 0.99 and 0.51. Individually, no rank metric correlates significantly more on average with the other metrics. Interestingly, from the average correlations, we clearly see two consistent clusters of stronger correlations between the relative rank metrics (approximate rank (PCA) and Effective rank (Roy & Vetterli, 2007)) and absolute rank metrics (Feature Rank (Lyle et al., 2022) and PyTorch rank). The srank (Kumar et al., 2021) which is technically a relative metric, but with a weak normalization rationale, correlates more with the relative metrics on MuJoCo with tanh but more with the absolute metrics on ALE and MuJoCo with ReLU.

Vetterli

PCA

PyTorch

Vetterli

PCA

Pytorch

1.00

0.94

0.73

0.73

0.69

0.82

0.94

1.00

0.72

0.72

0.68

0.81

0.73

0.72

1.00

0.94

0.92

0.86

0.73

0.72

0.94

1.00

0.92

0.86

0.69

0.68

0.92

0.92

1.00

0.84

0.82

±0.14

0.82

0.81

±0.15

0.81

0.86

±0.13

0.86

0.86

±0.13

0.86

0.84

±0.15

0.84

Vetterli

PCA

PyTorch

Vetterli

PCA

Pytorch

1.00

0.99

0.85

0.85

0.82

0.90

0.99

1.00

0.82

0.83

0.79

0.89

0.85

0.82

1.00

0.99

0.98

0.93

0.85

0.83

0.99

1.00

0.97

0.93

0.82

0.79

0.98

0.97

1.00

0.91

0.90

±0.09

0.90

0.89

0.89

0.93

±0.08

0.93

0.93

±0.08

0.93

0.91

0.91

Vetterli

PCA

PyTorch

Vetterli

PCA

Pytorch

1.00

0.99

0.88

0.89

0.86

0.92

0.99

1.00

0.84

0.85

0.81

0.90

0.88

0.84

1.00

0.99

0.99

0.94

0.89

0.85

0.99

1.00

0.98

0.94

0.86

0.81

0.99

0.98

1.00

0.93

0.92

±0.07

0.92

0.90

±0.09

0.90

0.94

±0.08

0.94

0.94

±0.07

0.94

0.93

±0.09

0.93

Pearson

Vetterli

PCA

PyTorch

Vetterli

PCA

Pytorch

1.00

0.99

0.82

0.78

0.75

0.87

0.99

1.00

0.83

0.79

0.76

0.87

0.82

0.83

1.00

0.95

0.93

0.91

0.78

0.79

0.95

1.00

0.96

0.90

0.75

0.76

0.93

0.96

1.00

0.88

0.87

±0.12

0.87

0.87

±0.11

0.87

0.91

±0.08

0.91

0.90

±0.11

0.90

0.88

±0.12

0.88

1-distance

0.70

0.75

0.80

0.85

0.90

0.95

1.00

0.800

0.825

0.850

0.875

0.900

0.925

0.950

0.975

1.000

0.825

0.850

0.875

0.900

0.925

0.950

0.975

1.000

0.75

0.80

0.85

0.90

0.95

1.00

Vetterli

PCA

PyTorch

Vetterli

PCA

Pytorch

1.00

0.93

0.84

0.58

0.52

0.77

0.93

1.00

0.85

0.57

0.51

0.77

0.84

0.85

1.00

0.66

0.57

0.78

0.58

0.57

0.66

1.00

0.78

0.72

0.52

0.51

0.57

0.78

1.00

0.68

0.77

±0.21

0.77

0.77

±0.22

0.77

0.78

±0.17

0.78

0.72

±0.18

0.72

0.68

±0.21

0.68

Vetterli

PCA

PyTorch

Vetterli

PCA

Pytorch

1.00

0.98

0.93

0.67

0.61

0.84

0.98

1.00

0.94

0.67

0.60

0.84

0.93

0.94

1.00

0.71

0.63

0.84

0.67

0.67

0.71

1.00

0.79

0.77

0.61

0.60

0.63

0.79

1.00

0.72

0.84

±0.19

0.84

0.84

±0.19

0.84

0.84

±0.16

0.84

0.77

±0.14

0.77

0.72

±0.17

0.72

Vetterli

PCA

PyTorch

Vetterli

PCA

Pytorch

1.00

0.99

0.91

0.73

0.66

0.86

0.99

1.00

0.88

0.68

0.62

0.83

0.91

0.88

1.00

0.87

0.77

0.88

0.73

0.68

0.87

1.00

0.86

0.83

0.66

0.62

0.77

0.86

1.00

0.78

0.86

±0.16

0.86

0.83

±0.18

0.83

0.88

±0.08

0.88

0.83

±0.13

0.83

0.78

±0.16

0.78

Pearson

Vetterli

PCA

PyTorch

Vetterli

PCA

Pytorch

1.00

0.99

0.95

0.94

0.94

0.97

0.99

1.00

0.95

0.95

0.94

0.97

0.95

0.95

1.00

0.99

0.99

0.98

0.94

0.95

0.99

1.00

1.00

0.98

0.94

0.94

0.99

1.00

1.00

0.97

0.97

±0.03

0.97

0.97

±0.03

0.97

0.98

±0.02

0.98

0.98

±0.03

0.98

0.97

±0.03

0.97

1-distance

0.6

0.7

0.8

0.9

1.0

0.60

0.65

0.70

0.75

0.80

0.85

0.90

0.95

1.00

0.65

0.70

0.75

0.80

0.85

0.90

0.95

1.00

0.95

0.96

0.97

0.98

0.99

1.00

Vetterli

PCA

PyTorch

Vetterli

PCA

Pytorch

1.00

0.78

0.58

0.38

0.49

0.65

0.78

1.00

0.59

0.44

0.48

0.66

0.58

0.59

1.00

0.70

0.81

0.74

0.38

0.44

0.70

1.00

0.73

0.65

0.49

0.48

0.81

0.73

1.00

0.70

0.65

±0.25

0.65

0.66

±0.23

0.66

0.74

±0.17

0.74

0.65

±0.25

0.65

0.70

±0.22

0.70

Vetterli

PCA

PyTorch

Vetterli

PCA

Pytorch

1.00

0.87

0.65

0.46

0.57

0.71

0.87

1.00

0.69

0.53

0.58

0.73

0.65

0.69

1.00

0.79

0.89

0.80

0.46

0.53

0.79

1.00

0.80

0.72

0.57

0.58

0.89

0.80

1.00

0.77

0.71

±0.22

0.71

0.73

±0.20

0.73

0.80

±0.14

0.80

0.72

±0.22

0.72

0.77

±0.19

0.77

Vetterli

PCA

PyTorch

Vetterli

PCA

Pytorch

1.00

0.94

0.63

0.48

0.53

0.71

0.94

1.00

0.69

0.54

0.56

0.74

0.63

0.69

1.00

0.82

0.92

0.81

0.48

0.54

0.82

1.00

0.80

0.73

0.53

0.56

0.92

0.80

1.00

0.76

0.71

±0.24

0.71

0.74

±0.21

0.74

0.81

±0.15

0.81

0.73

±0.21

0.73

0.76

±0.21

0.76

Pearson

Vetterli

PCA

PyTorch

Vetterli

PCA

Pytorch

1.00

0.99

0.96

0.93

0.94

0.96

0.99

1.00

0.96

0.93

0.94

0.97

0.96

0.96

1.00

0.97

0.98

0.97

0.93

0.93

0.97

1.00

0.98

0.96

0.94

0.94

0.98

0.98

1.00

0.97

0.96

±0.03

0.96

0.97

±0.03

0.97

0.97

±0.02

0.97

0.96

±0.03

0.96

0.97

±0.03

0.97

1-distance

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.5

0.6

0.7

0.8

0.9

1.0

0.5

0.6

0.7

0.8

0.9

1.0

0.94

0.95

0.96

0.97

0.98

0.99

1.00

Vetterli

PCA

PyTorch

Vetterli

PCA

Pytorch

1.00

0.84

0.20

0.17

0.16

0.47

0.84

1.00

0.16

0.11

0.08

0.44

0.20

0.16

1.00

0.74

0.73

0.57

0.17

0.11

0.74

1.00

0.55

0.51

0.16

0.08

0.73

0.55

1.00

0.50

0.47

±0.41

0.47

0.44

±0.44

0.44

0.57

±0.37

0.57

0.51

0.51

0.50

±0.39

0.50

Vetterli

PCA

PyTorch

Vetterli

PCA

Pytorch

1.00

0.95

0.27

0.23

0.21

0.53

0.95

1.00

0.20

0.15

0.12

0.49

0.27

0.20

1.00

0.88

0.88

0.64

0.23

0.15

0.88

1.00

0.68

0.59

0.21

0.12

0.88

0.68

1.00

0.58

0.53

±0.41

0.53

0.49

±0.45

0.49

0.64

0.64

0.59

0.59

0.58

±0.40

0.58

Vetterli

PCA

PyTorch

Vetterli

PCA

Pytorch

1.00

0.95

0.57

0.58

0.35

0.69

0.95

1.00

0.33

0.44

0.10

0.56

0.57

0.33

1.00

0.89

0.95

0.75

0.58

0.44

0.89

1.00

0.73

0.73

0.35

0.10

0.95

0.73

1.00

0.63

0.69

±0.28

0.69

0.56

±0.39

0.56

0.75

±0.29

0.75

0.73

±0.22

0.73

0.63

±0.39

0.63

Pearson

Vetterli

PCA

PyTorch

Vetterli

PCA

Pytorch

1.00

0.96

0.64

0.54

0.53

0.73

0.96

1.00

0.60

0.50

0.49

0.71

0.64

0.60

1.00

0.90

0.87

0.80

0.54

0.50

0.90

1.00

0.85

0.76

0.53

0.49

0.87

0.85

1.00

0.75

0.73

±0.23

0.73

0.71

±0.25

0.71

0.80

±0.17

0.80

0.76

±0.22

0.76

0.75

±0.22

0.75

1-distance

0.2

0.4

0.6

0.8

1.0

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.5

0.6

0.7

0.8

0.9

1.0

Vetterli

PCA

PyTorch

Vetterli

PCA

Pytorch

1.00

0.87

0.65

0.07

0.04

0.52

0.87

1.00

0.71

0.07

0.02

0.54

0.65

0.71

1.00

0.09

0.05

0.50

0.07

0.07

0.09

1.00

0.13

0.27

0.04

0.02

0.05

0.13

1.00

0.25

0.52

±0.45

0.52

0.54

0.54

0.50

±0.41

0.50

0.27

±0.41

0.27

0.25

±0.42

0.25

Vetterli

PCA

PyTorch

Vetterli

PCA

Pytorch

1.00

0.95

0.78

0.09

0.04

0.57

0.95

1.00

0.84

0.09

0.03

0.58

0.78

0.84

1.00

0.10

0.06

0.56

0.09

0.09

0.10

1.00

0.13

0.28

0.04

0.03

0.06

0.13

1.00

0.25

0.57

±0.47

0.57

0.58

±0.48

0.58

0.56

±0.44

0.56

0.28

±0.40

0.28

0.25

±0.42

0.25

Vetterli

PCA

PyTorch

Vetterli

PCA

Pytorch

1.00

0.96

0.71

0.10

0.04

0.56

0.96

1.00

0.73

0.09

0.03

0.56

0.71

0.73

1.00

0.43

0.15

0.60

0.10

0.09

0.43

1.00

0.17

0.36

0.04

0.03

0.15

0.17

1.00

0.28

0.56

0.56

0.56

±0.47

0.56

0.60

±0.32

0.60

0.36

0.36

0.28

±0.41

0.28

Pearson

Vetterli

PCA

PyTorch

Vetterli

PCA

Pytorch

1.00

0.98

0.92

0.91

0.91

0.95

0.98

1.00

0.92

0.91

0.91

0.94

0.92

0.92

1.00

0.99

0.99

0.96

0.91

0.91

0.99

1.00

0.99

0.96

0.91

0.91

0.99

0.99

1.00

0.96

0.95

±0.04

0.95

0.94

±0.04

0.94

0.96

±0.04

0.96

0.96

±0.05

0.96

0.96

±0.05

0.96

1-distance

0.2

0.4

0.6

0.8

1.0

0.2

0.4

0.6

0.8

1.0

0.2

0.4

0.6

0.8

1.0

0.92

0.94

0.96

0.98

1.00

Vetterli

PCA

PyTorch

Vetterli

PCA

Pytorch

1.00

0.31

-0.51

-0.53

-0.54

-0.06

0.31

1.00

-0.52

-0.60

-0.60

-0.08

-0.51

-0.52

1.00

0.12

0.33

0.08

-0.53

-0.60

0.12

1.00

0.13

0.02

-0.54

-0.60

0.33

0.13

1.00

0.06

-0.06

±0.69

-0.06

-0.08

±0.72

-0.08

0.08

±0.64

0.08

0.02

±0.65

0.02

0.06

±0.66

0.06

Vetterli

PCA

PyTorch

Vetterli

PCA

Pytorch

1.00

0.41

-0.68

-0.70

-0.71

-0.14

0.41

1.00

-0.68

-0.74

-0.75

-0.15

-0.68

-0.68

1.00

0.09

0.38

0.02

-0.70

-0.74

0.09

1.00

0.06

-0.06

-0.71

-0.75

0.38

0.06

1.00

-0.00

-0.14

-0.14

-0.15

±0.81

-0.15

0.02

±0.72

0.02

-0.06

±0.71

-0.06

-0.00

±0.74

-0.00

Vetterli

PCA

PyTorch

Vetterli

PCA

Pytorch

1.00

0.56

-0.65

-0.72

-0.72

-0.11

0.56

1.00

-0.66

-0.75

-0.74

-0.12

-0.65

-0.66

1.00

-0.08

0.43

0.01

-0.72

-0.75

-0.08

1.00

-0.06

-0.12

-0.72

-0.74

0.43

-0.06

1.00

-0.02

-0.11

±0.82

-0.11

-0.12

±0.84

-0.12

0.01

±0.72

0.01

-0.12

±0.71

-0.12

-0.02

±0.75

-0.02

Pearson

Vetterli

PCA

PyTorch

Vetterli

PCA

Pytorch

1.00

0.99

0.94

0.89

0.91

0.95

0.99

1.00

0.93

0.89

0.91

0.94

0.94

0.93

1.00

0.91

0.97

0.95

0.89

0.89

0.91

1.00

0.93

0.92

0.91

0.91

0.97

0.93

1.00

0.94

0.95

±0.05

0.95

0.94

±0.05

0.94

0.95

±0.04

0.95

0.92

±0.04

0.92

0.94

±0.04

0.94

1-distance

-6

-4

-2

0

2

4

6

8

10

-0.6

-0.4

-0.2

0.0

0.2

0.4

0.6

0.8

1.0

-0.6

-0.4

-0.2

0.0

0.2

0.4

0.6

0.8

1.0

0.90

0.92

0.94

0.96

0.98

1.00

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Using the numbered contributions and claims in the introduction, we provide evidence for claims 1, 2, 3, 4, and 5 in Sections 3.1, 3.2, 4, 4, and Appendix B.1, respectively.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Limitations are discussed in Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Proofs relative to Section 3.2.1 are presented in Appendix C.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We list all the experimental details in Appendix B and provide a fully reproducible codebase.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide a fully reproducible codebase with instructions in Appendix B.1. Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We describe our experimental setup in Section 3 and give details in Appendix B. Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We discuss statistical significance in Appendix B.4.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We discuss computing resources in Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The paper addresses a fundamental algorithmic issue with no specific application. The impact of any application should be discussed specifically when introducing the application.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The codebase implements already existing algorithms and training curves do not pose such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All the assets used have been cited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We introduce a codebase and run histories documented in Appendix B.1.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing or research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing or research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the phenomenon of feature collapse in the popular on-policy RL algorithm PPO. It demonstrates that increasing the sample reuse (number of epochs) in PPO deteriorates the feature rank and plasticity. It also finds that the clipping operation for the conservative update in PPO does not prevent this feature collapse. Furthermore, it shows that feature collapse leads to the degradation of the trust region, and vice versa. Finally, the paper proposes several techniques to prevent feature collapse, including feature regularization, network sharing, and Adam hyperparameter adjustments, though these techniques do not fully resolve the issue.\n\nSTRENGTHS:\n- To the best of my knowledge, this is the first work to investigate the phenomenon of feature collapse in on-policy algorithms.\n- The experimental design is strong. They appropriately select benchmarks: Atari for discrete action spaces and MuJoCo for continuous action spaces. Additionally, the experiments are extensive and well-conducted.\n- The connection between feature collapse and trust region is intriguing and distinguishes this work from previous studies on feature collapse in value-based methods.\n\nWEAKNESSES:\n- They demonstrated that training policy and value networks for more epochs leads to feature collapse, but this is not typical in PPO settings. It is uncommon to see PPO run for 6 or 8 epochs (Figure 1, 2, and 3).\n- While they attempt to explain feature collapse in the critic networks, the explanation is insufficient and requires further elaboration  (Section 3.1).\n- While they emphasize that the focus of the paper is not on improving the performance of PPO, they do not provide a clear solution to mitigate feature collapse in PPO (Figure 6).\n\nQUESTIONS:\n- In Section 3.1 and Figures 7, 8, and 9, there appears to be no clear correlation between the number of epochs and the plasticity loss of the critic network, which is counterintuitive since it contrasts with findings from value-based plasticity studies. Could you explain more about this?\n- In Section 3.2, you state that the clipped sample’s ratio will have their probabilities continue to go beyond the clip limit in PPO. How about replacing the clipping operation with a KL penalty? This technique was proposed in the original PPO paper [1], and it has shown no significant difference in performance compared to the clipping operation in some environments [2]. What impact do you think replacing the clipping operation with a KL penalty would have on feature collapse?\n- Regarding PFO, instead of applying regularization in the feature space to ensure the current representation does not deviate from previous representations, how about applying regularization in the weight space using EWMA update?\n- It would be beneficial to investigate Phasic Policy Gradient (PPG) [2]. It trains the actor network to predict the value of the old policy, which is similar to PFO. It would be interesting to analyze feature collapse in PPG to see if similar patterns emerge as observed in PFO.\n\n[1] John Schulman et al., Proximal Policy Optimization Algorithms, arXiv 2017 \\\n[2] Karl Cobbe et al., Phasic Policy Gradient, ICML 2020\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper examines the loss of plasticity and its connection representation collapse in policy networks in online reinforcement learning (as opposed to previously studied value networks in offline reinforcement learning). The paper establishes the problem in the Atari game domain including the growth in the norm of feature pre-activations of the policy network. The analysis isolates and illustrates the problem of increasing representation state collinearity and representation collapse in the trust region of a policy in a toy domain. To mitigate representation collapse the paper proposes a feature regularization loss term based on these pre-activation features of the policy network.\n\nThe evaluation compares methods to mitigate policy network representation collapse including the new loss, sharing the actor and critic feature trunk, and modifying the Adam optimizer (by resetting moments or faster decay of second moments) in the ALE and MuJoCo environments. The results show the loss has does not impact episode returns but does mitigate loss of plasticity.\n\nSTRENGTHS:\n# originality\nGood\n- Investigating plasticity loss in offline RL is established, but has not been extended to online RL.\n- The analysis provides new insights on trust region optimization.\n\n# quality\nGood\n- The toy domain isolates the effect of interest to strengthen the empirical results and motivate the new regularization term.\n- the experiments show modest evidence for improvements and report variation in results over runs.\n\n# clarity\nMostly good\n- The exposition on trust region weaknesses and collapse could be clearer and make the main points sooner. The rest of the text was good.\n\n# significance\nGood\n- Loss of plasticity is an important problem, particularly in the case of online RL.\n- The paper will be of interest to researchers in continual learning and RL communities; both theoretical (for the trust region points) and practical (for the regularization term) researchers.\n\nWEAKNESSES:\nThe paper would benefit from outlining the trust region insight sooner as it is a core idea that only gets explained in detail on page 7 of the paper.\n\nThe results on the regularization term are a bit confusing and mixed. The narrative would benefit from clarifying the aspects that remain unresolved or ambiguous that merit further attention. The Gravitar analysis of sparse reward environments was helpful. But I was not really sure what to make of the effects on episode returns compared to representation collapse. Some way of teasing these effects out more cleanly would help.\n\nQUESTIONS:\n- line 143: \"do not include normalization layers\"\n\t- Does this help plasticity preservation? (even if only to cite prior work on the effects)\n- lines 151-153: \"We vary the number of epochs as a way to control the effect of non-stationarity, which gives the agent a more significant number of optimization steps per rollout while not changing the optimal target it can reach due to clipping, as opposed to changing the value of ε in the trust region for example.\"\n\t- It might be interesting to contrast the effects of varying $\\epsilon$ as well.\n- line 280 (minor): $\\pi_\\theta(a_i | s)$\n\t- $a_i$ should be $a_1$ in this case\n- The toy setting was good demonstration of the problems with trust region constraints. It may help to introduce this sooner as I was confused about the trust region claims being made until that point.\n- Figure 6 (minor)\n\t- The caption only mentions \"Top\" and \"Bottom\", omitting the middle row of \"NameThisGame\".\n\nLIMITATIONS:\nYes. The paper acknowledges remaining opportunities to relate the findings on representation collapse to other research on plasticity loss. And to incorporate more complex model architectures (like memory).\n\nThere is no need for additional comments on potential societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses non-stationarity in RL and its impact on deep learning networks, focusing on PPO. It identifies that networks in PPO, like those in off-policy methods, suffer from representation rank deterioration, leading to performance collapse. The authors propose Proximal Feature Optimization (PFO), a new regularization on the policy representation that regularizes preactivation changes, which can mitigate performance collapse and improve the agent’s performance.\n\nSTRENGTHS:\n- The work exhibits a coherent structure, and the author adeptly elucidates the underlying motivation. The author adeptly elucidates the shortcomings of PPO and offers enough reasons. This paper is well-written.\n- Section 3's Q1 and Q2 effectively describe the potential problems of PPO, and I concur with the underlying motivation.\n- The paper not only identifies problems but also suggests practical interventions, such as PFO and sharing the actor-critic trunk, to regularize representations and reduce non-stationarity, showing promising improvements in performance.\n- The authors provide details of the experiment design, and it's clear.\n\nWEAKNESSES:\nThe authors claim to open data and code; however, I could not locate them. Therefore, I apologize if I overlooked their presence. Refer to the question section for other weaknesses.\n\nQUESTIONS:\n- Please provide a more detailed explanation on Eq.2. I don't understand the meaning of $\\phi$, and should the formula be $s_t$ instead of $S_t$?\n- In section 4, the authors introduce an intervention that allows the actor and critic to share all the layers. The experiment results illustrate that this can bring some improvements but not significantly. I am more curious about whether the reason for this is the sparsity of reward or the environment itself. I think the latter is more likely, and we can verify this by setting up a control experiment with different reward functions in the same environment.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work provides an empirical study of the feature rank deterioration and loss of plasticity of the Proximal Policy Optimization (PPO) algorithm on Atari and Mujoco tasks. Then links the deterioration of the performance to representation collapse and hence the break of the trust region. From there, the authors propose an auxiliary loss to PPO to maintain the representation rank.\n\nSTRENGTHS:\nThe study is very interesting and brings some new insight into how PPO works in practice. The experiments are conducted in clear logic and the analysis and observations are all novel and interesting.\n\nWEAKNESSES:\nFigures can be plotted with better quality, don't overlap the labels (in Figure 3), and maybe set the titles to the quantity of interest. \n\nI understand that several works are measuring the feature rank at the last policy/critic layer. But why that is sufficient evidence that the policy is losing rank? For instance, if my action space is binary, then the policy's last layer could learn to be rank-2. More information might be stored in previous layers. From another perspective, isn't it expected that the policy learns to lose rank, by compressing state information to only correlate with actions?\n\nThe proposed Proximal Feature Optimization objective is a bit confusing. First it should be a $\\ell_2$ norm? What does it mean by $(\\phi - \\phi_\\text{old})^2$. Secondly, from a high-level, isn't this just a regularization loss as in continual learning that prevents forgetting? But based on my knowledge it will exacerbate the the loss of plasticity. If you want to preserve rank, should not you add some reconstruction loss?\n\nQUESTIONS:\nCould the authors address my questions in the weakness section?\n\nFrom Figure 6, does it mean that the proposed method does not consistently improve the loss of plasticity?\n\nIf the main goal is to prevent the preactivation norm from going larger, why not just regularize it to be smaller?\n\nLIMITATIONS:\nUes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the phenomenon of loss of trainability and performance collapse with PPO agents in standard RL benchmarks. The same phenonema from other settings is found to hold with PPO agents and, moreover, increasing the number of training epochs exacerbates the effect. \nFurther investigation finds that PPO's trust-region constraint is not maintained when the feature rank decreases and a mechanism for this finding is explained. \nThe authors propose a regularizer, PFO, which softly constrains the change in features is shown to mitigate trainability loss throughout the learning process.\n\nSTRENGTHS:\nThe paper investigates the topic of trainability loss in a less-explored context, on-policy policy optimization, and focusing on a popular algorithm, PPO. Better understanding this algorithm and how it interacts with neural networks could lead to widespread impact as the research direction is developed.\n\nA wide variety of experiments are conducted with some interesting findings. To me, the most intriguing insight is the connection between low feature ranks and increased violations of PPO's trust region. The investigation is fairly thorough, examining various metrics in tandem throughout the training process. Experiments seem to be conducted and reported in a reasonable fashion.\nI appreciate the simple, clear example to demonstrate how a poor representation may lead to the clipping constraint being ineffective (line 270-...).\n\nWEAKNESSES:\nCurrently, the paper feels a bit disorganized and it is a bit difficult to follow the train of thought. There are many different experiments done and placing more focus on the most important could help streamline the paper. For example, the section at line 193 feels a bit out of place since this line of investigation is not mentioned earlier. Another example is that the research questions Q1. and Q2. do not mention the trust-region although it is a large part of the paper.\n\nAdditionally, various phrases (or claims) in the paper could use more support or evidence. These are discussed more specifically in the Questions section. \n\nI am also concerned about the link between feature rank and trust-region violation, a central point of the paper. \nIn Fig.4, the correlation between the prob ratios and dead neurons or prob ratios and feature rank seem weak. There is only a substantial dip in the prob ratios once the feature rank is near zero or the number of dead units is very high. In other words, only when the agent is doing very poorly can we see a strong relationship, which may limit the applicability of the finding. Ideally, one would avoid this regime since performance is poor to start with.\n\nQUESTIONS:\nI would be willing to revise my score based on answers to the following.\n\n_Clarification questions_\n- Line 308. Could you clarify how the average probability ratio is computed?\n\tAs written, my interpretation is that, given some window of updates, you consider all ratios above $1+\\epsilon$ and take their mean. Then, you do the same for the ratios less than $1-\\epsilon$. Finally you divide the first mean ratio by the second mean ratio. Is this correct?\n\tIf so, this seems potentially misleading because the means are computed conditionally, considering only ratios that are above 9or below) the clipping ratios. Then, we cannot tell what the mean ratio is overall, and whether there are many violations of the trust-region occurring. Moreover, ratios can be misleading since they may overemphasize small probabilities in the denominator. \n\n\tI would suggest also reporting how often the clipping criteria is violated. E.g. compute on collected states which proportion of them have a ratio outside the clipping region after a round of updates on a batch of trajectories.\n\tPerhaps it would also be useful to consider ratios less than 1 or greater than 1 seperately instead of computing a ratio.\n\n\n- Line 164 Using the term \"plasticity loss\" as a metric is confusing because it also refers to the overall phenomena of neural networks being unable to learn additional tasks after training on other tasks. \nDoes this term refer to the capacity to fit random labels? In that case, I would use a term that is more descriptive e.g. random label trainability (RLT). \nAlso, how are these random labels generated in this context? In the original paper, a regression task was made since a value-based algorithm was used. With a policy-based algorithm, how is the plasticity evaluation task set up?\n\n\n- Fig.1 Why does the preactivation norm plateau at 10^4? \n\n- Which data is used to produce the box plots? The caption for Fig. 6 mentions \"A boxplot includes 15 runs with the different epochs\". Does this mean the box plot conains data across training times and epochs? \n\n- In the box plots, how are outliers determined? We can see they lie outside the interquartile range indicated by the \"whiskers\" but what additional criteria is there? \n\n\n- How is the PFO regularizer related to adding a regularizer in parameter space? Say $\\ell_2$ regularization between the current parameters and the previous parameters? Basically, this seems similar in spirit to adding a target network to value-based methods. \n\n\n- Line 94-99 (and elsewhere) \"PPO introduces additional non-stationarity...\": I do not think it would be fair to see PPO has more nonstationarity than other policy gradient methods. REINFORCE and PPO both have nonstationarity due to the changing policies. REINFORCE can also be interpreted as a trust-region-like method. Since REINFORCE is essentially gradient ascent on the expected return objective, we can view it as minimizing a linear approximation with a quadratic constraint (see paragraph containing eq. 5.5 in [4]). In this light, by doing a single optimization step before changing the surrogate objective, we could say REINFORCE has _more_ nonstationarity.\n\n\n- Line 257. \"The underlying assumption...approximately orthogonal...\" Could you give some more support to this claim?\n\tAs mentioned above, with state aggregation, this clipping strategy could be effective without having orthogonal features. \n\n\n- Line 8-9: \"...policy optimization methods which are often thought capable of training indefinitely\". I would disagree with this claim. Deep RL algorithms have often be thought to be high-variance and unstable so I do not think it would be common to believe they can train indefinitely. \nI suggest replacing this phrase and simply mention the phenonmenon has been underexplored in the context of on-policy policy optimization.\n\n\n- The parameter norm has been linked to plasticity/trainability loss in neural networks. Is there a reason why the paper measures feature norms instead? \n\n\n\n\n\n_Broader questions_\n- The performance degradation from training more epochs could be attributed to difficulties of offline RL. As the number of epochs is increased, the more the procedure ressembles offlne RL---a setting in which standard RL algorithms are not suitable for. What do you think of this?\n\n\n- I find the proposed solution (Proximal Feature Optimization) to be slightly unsatisfying, Generally, I can see how using regularization on the policy and features can mitigate the performance collapse but would this be simply delaying the problem? If we train for longer, would we still expect to see the same performance degradation? \n\n\n- Interestingly, the value network is not the problem. It is the policy network that suffers from representation collapse. Do you have any thoughts about this? I think this could be an interesting avenue of investigation.\n\n\n_Suggestions_\n\n- I would suggest removing or summarizing the paragraph at line 193 into a single sentence. \n\tIt is clear that the policy would be the same for all states if the features are constant. This would imply the policy is uniform. Currently the paragraph seems to explain this in a roundabout manner when a concise explanation suffices. \n\n- A suggestion would be to try the Atari environemnts identified in [1] to benefit from plasticity injection. On many Atari games, little or no benefit was found, so it may be more meaningful to focus on those where there is more evidence for trainability loss.  \n\n\n- I would be careful about mentioning that low feature rank necessarily means that the representation is poor. Successful training of deep neural networks often involves a reduction in feature rank and the relationship between rank and performance can be complex [3].\n\n\n- As a sidenote: even if there is a low dimensional representation, this does not necessarily imply the clipping criteria will be violated. For example, if we use state aggregation, then multiple states are clustered into one aggregate state. Then, the policy will be exactly the same for all these states, so if the policy exits the clipping region for one those states, there will no longer be any incentive to update the policy any further on any of them. \n\n\n- A minor suggestion is to use stable rank [2] as a measure of the rank instead of effective rank since it is a continuous quantity in the singular values. \n\n\n\n\n[1] \"Deep Reinforcement Learning with Plasticity Injection\" Nikishin et al. \n\n[2] https://nickhar.wordpress.com/2012/02/29/lecture-15-low-rank-approximation-of-matrices/\n\n[3] \"An Empirical Study of Implicit Regularization in Deep Offline RL\" Gulcehre et al. \n\n[4] https://www.stat.cmu.edu/~ryantibs/convexopt-S15/scribes/05-grad-descent-scribed.pdf\n\nLIMITATIONS:\nThese are discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a series of experimental studies to diagnose the learning issues of PPO under non-stationarity in Atari-5 and MuJoCo tasks. Based on the results, this paper establishes a connection among feature rank/norm, plasticity loss and trust region violation and learning performance. To mitigate the issues in feature representation, this paper proposes Proximal Feature Optimization (PFO) and demonstrates its effectiveness in mitigating the learning issues under non-stationarity. Besides, the effects of sharing network parameters and adapting Adam are also evaluated in the same context.\n\nSTRENGTHS:\n- Most existing works on plasticity loss or learning under non-stationarity focus on value-based RL or off-policy AC methods. In contrast, this paper focuses on on-policy algorithms, mainly PPO, which helps the RL community gain a better understanding of this problem.\n- The experimental studies contains the results with multiple metrics, i.e., feature rank, feature norm, plasticity loss, dead neuron, excess ratio. These results will be a useful reference to the audiences.\n- This paper reveals the connection among factors like feature rank, plasticity loss, excess rate, and learning performance.\n\nWEAKNESSES:\n- The writing is almost clear. The empirical results and conclusions can be better organized and made prominent.\n- The proposed method PFO is closely related to DR3 (Kumar et. al, 2022) and RD [1], both of which propose regularizing the feature representation. More discussions are necessary.\n- Although PFO are demonstrated to mitigate the feature rank, plasticity loss, excess ratio in Section 4, it does not bring clear improvement in terms of learning performance like episode return (according to Figure 18 to Figure 27). This cripples the significance of PFO and also the potential causal connection between the learning issues investigated and the learning performance.\n\n---\n\nReference:\n\n[1] Reining Generalization in Offline Reinforcement Learning via Representation Distinction. NeurIPS 2023\n\nQUESTIONS:\n1. The epsiode return curves in Figure 24 and Figure 26 (i.e., for MuJoCo Hopper) look strange. Is there any explanation to the collapse? A regular implementation of PPO should work in Hopper when the epoch num is 10.\n2. The authors mentioned 4 MuJoCo tasks and 5 Atari games in the Experimental Setup paragraph. However, it seems only 3 Atari games (Phoenix, NameThisGame, Gravitar) and 2 MuJoCo tasks (i.e., Humanoid, Hopper) are used for the evaluation in Section 4. Are there more results on the remaining tasks/games?\n\nLIMITATIONS:\nThe limitations are discussed in Section 5. However, one major limitation of this work is that the results in this paper do not provide sufficient support for the point that \"mitigating the learning issues in feature rank, plasticity loss of PPO can improve the learning performance in terms of episode return\".", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-15"}