Understanding and Preventing Capacity Loss in Reinforcement Learning
Authors: Clare Lyle
Mark Rowland
Will Dabney
Venue: ICLR 2022 Spotlight (2022)
Keywords: Reinforcement learning
representation learning

Abstract: The reinforcement learning (RL) problem is rife with sources of non-stationarity that can destabilize or inhibit learning progress.
We identify a key mechanism by which this occurs in agents using neural networks as function approximators: \textit{capacity loss}, whereby networks trained to predict a sequence of target values lose their ability to quickly fit new functions over time.
We demonstrate that capacity loss occurs in a broad range of RL agents and environments, and is particularly damaging to learning progress in sparse-reward tasks. We then present a simple regularizer, Initial Feature Regularization (InFeR), that mitigates this phenomenon by regressing a subspace of features towards its value at initialization, improving performance over a state-of-the-art model-free algorithm in the Atari 2600 suite. Finally, we study how this regularization affects different notions of capacity and evaluate other mechanisms by which it may improve performance.

UNDERSTANDING AND PREVENTING CAPACITY Loss IN REINFORCEMENT LEARNING
Clare Lyle
Mark Rowland & Will Dabney DeepMind
Department of Computer Science
University of Oxford"
ABSTRACT
The reinforcement learning (RL) problem is rife with sources of non-stationarity, making notoriously difficult problem domain for the application of neural net- works. We identify a mechanism by which non-stationary prediction targets can prevent learning progress in deep RL agents: capacity loss, whereby networks trained on sequence of target values lose their ability to quickly update their predictions over time. We demonstrate that capacity loss occurs in range of RI. agents and environments, and is particularly damaging to performance in sparse- reward tasks. We then present a simple regularizer, Initial Feature Regularization (InFeR), that mitigates this phenomenon by regressing subspace of features to- wards its value initialization, leading to significant performance improvements in sparse-reward environments such Montezuma's Revenge. We conclude that preventing capacity loss is crucial to enable agents maximally benefit from the learning signals they obtain throughout training.
INTRODUCTION
Deep reinforcement learning has achieved remarkable successes a variety of tasks (Mnih et al. 2013 Moravčík et al. 2017; Silver et al. 2017] AAbreu et al. 2019). but its impressive performance mirrored by brittleness and sensitivity to seemingly innocuous design choices (Henderson) al. 2018). In sparse-reward environments in particular, even different random seeds of the same algorithm can attain dramatically different performance outcomes. This presents stark contrast to supervised learning, where existing approaches are reasonably robust to small hyperparameter changes, random seed inputs, and GPU parallelisation libraries. Much of the brittleness of deep RL. algorithms has been attributed to the non-stationary nature of the prediction problems to which deep neural networks are applied in RL tasks. Indeed, naive applications of supervised learning methods the RL problem may require explicit correction for non- staiiooatio and bootstrapping in order to yield similar improvements (Bengio et 2020 Raileanu et al.) 2020).
We hypothesize that the non-stationary prediction problems agents face in RL may be driving force in the challenges described above. RL agents must solve sequence of similar prediction tasks as they iteratively improve their value function accuracy and their policy (Dabney et al.) 2021). Solving each subproblem (at least to the extent that the agent's policy is improved) in this sequence necessary to progress to the next subproblem. Ideally, features learned to solve one subproblem would enable forward transfer to future problems. However, prior work on both supervised and reinforcement learning (Ash Adams 2020; Igl al., 2021 Fedus al., (0200) suggests that the opposite is true: networks trained on sequence of similar tasks are prone overfitting, exhibiting negative transfer.
The principal thesis of this paper is that over the course of training, deep RL agents lose some of their capacity to quickly fit new prediction tasks, and in extreme cases this capacity loss prevents the agent entirely from making learning progress. present rigorous empirical analysis of this
"Correspondence ala cl.re. 1 yycccca uckuk.uk

---

phenomenon which considers both the ability of networks to learn new target functions via gradient- based optimization methods, and their ability to linearly disentangle states' feature representations. We confirm that agents' ability to fit new target functions declines over the course of training in several environments from the Atari suite (Bellemare et al. 2013) and non-stationary reward pre- diction tasks. We further find that the ability of representations to linearly distinguish different states, a proxy for their ability to represent certain functions, quickly diminishes in sparse-reward environments, leading representation collapse, where the feature outputs for every state in the environment inhabit a low-dimensional or possibly even zero subspace. Crucially, we find evi- dence that sufficient capacity is a necessary condition in order for agents to make learning progress. Finally, we propose simple regularization technique, Initial Feature Regularization (InFeR), to prevent representation collapse by regressing a set of auxiliary outputs towards their value under the network's ' initial parameters. We show that this regularization scheme mitigates capacity loss in a number of settings, and also enables significant performance improvements a number of RL tasks.
One striking take-away from our results is that agents trained so-called 'hard exploration' games such Montezuma's Revenge attain significant improvements over existing competitive base- lines without using smart exploration algorithms, given a suitable representation learning objec- tive. This suggests that the poor performance of deep RL agents in sparse-reward environments is not solely due to inadequate exploration, but rather also in part due to poor representation learn- ing. Investigation into the interplay between representation learning and exploration, particularly in sparse-reward reward settings, thus presents a particularly promising direction for future work.
2 BACKGROUND
We consider the reinforcement learning problem wherein agent interacts with environment formalized by a Markov Decision Process .M- = (X' A. R, P.7)). where X' denotes the state space, A the action space, R the reward function, P the transition probability function, and y the discount factor. We will be primarily interested in value-based RL, where the objective is to learn the value function QF X' x A - R associated with some (possibly stochastic) policy π X - (A), defined as Q (I, a)
PE > R(rk. ax) x, ag al. In particular, we are interested in learning the value function associated with the optimal policy n² which maximizes the expected discounted sum of rewards from any state.
In Q-Learnng (Watkins & Dayan 1992). the agent performs updates to minimize the distance be- tween a predicted action-value function Q and the bootstrap target defined as
TQ(=, a) = B[[(ro, ao) + max Q(21, a' [=0 = 4g = a]
(1)
a'EA
In most practical settings, updates are performed with respect to sampled transitions rather than on the entire state space. The target can be computed for a sampled transition (æ,, a,, Tt, T241) as TQ(2,, rt + max, (x)++1)a).
When deep neural network is used function approximator (the deep RL setting), Q is defined to be the output of a neural network with parameters 6, and updates are performed by gradient descent on sampled transitions 7 = (xt, at, (t+1) A number of tricks are often used to improve stability: the sample-based objective is minimized following stochastic gradient descent based on minibatches sampled from a replay buffer of stored transitions, and a separate set of parameters 0 is used compute the targets Qz(4141, at+1) which is typically updated more slowly than the network's 's online parameters. This yields the following loss function, given sampled transition
(rp(Qo.T) (R441 + max - 2
In this work we will be interested in how common variations on this basic learning objective shape agents' learning dynamics, in particular the dynamics of the learned representation, or features. We will refer to the outputs of the final hidden layer of the network (i.e. the penultimate layer)
2

---

Target-fiting error over iterative tasks
amidar
pong
End beud
17800 MIP
10aM
Moan squared error
20.2
10-1
10-4
10"
DON
10
08-D0N
20
Target Baralions
Millions
frames
Networks trained to fit se- quence of different targets on MNIST data see increasing error new target func- tions with the number of tasks.
Figure 1:
Figure 2: Networks see reduced ability to fit new targets over the course of training in two demon- strative Atari environments.
as its features, denoted Φn (x). Our choice of the penultimate layer is motivated by prior literature studying representations RL (Ghosh & Bellemare| 2020 Kumar et al. 2021), although many works studying representation learning consider the outputs of earlier layers well. In general, the features of a neural network are defined to be the outputs of whatever layer is used to compute additional representation learning objectives.
3 CAPACITY LOss
Each time value-based RL agent discovers new source reward in its environment or, in the case of temporal difference methods, updates its value estimate, the prediction problem it needs to solve changes. Over the course of learning, such agent must solve long sequence of target prediction problems as its value function and policy evolve. Studies of neural networks in supervised learning suggest that this sequential fitting of new targets may be harmful to network's s ability to adapt to new targets (Achille et al.) 2018 see Section 5) for further details, e.g.). This presents a significant challenge to deep RL agents undergoing policy improvement, for which it is necessary to quickly make significant changes to the network's ' predictions even late in the training process. In this section, show that training on a sequence of prediction targets can lead to reduced ability to fit targets in deep neural networks, a phenomenon that term capacity loss. Further, we show that agent's inability to quickly update its value function to distinguish states presents a barrier to performance improvement in deep RL agents.
3.1 TARGET-HITTING CAPACITY
The parameters of neural network not only determine the network's current outputs, but also influ- ence how these outputs will evolve over time. A network which outputs zero because its final-layer weights are zero will evolve differently from one whose ReLU units are fully saturated at zero de- spite both outputting the same function Maas et al. (2013) time adapting to new targets. It this capacity to fit new targets that is crucial for RL agents to ob- tain performance improvements, and which frames our perspective on representation learning. We interested in identifying when agent's current parameters perform gradient updates that meaningfully change its predictions based new reward information in the environment or evolving bootstrap targets, notion formalized in the following definition.
particular, will have a much easier
flexible enough allow it
Definition (Target-fitting capacity). Let Px (X) be some distribution over inputs X and Pr distribution over a family of real-valued functions with domain X. Let N (go. 0o) represent the pairing of neural network architecture with some initial parameters du. and O correspond to optimization algorithm for supervised learning. We measure the target-fitting capacity of N under the optimizer O to fit the data generating distribution D = (Px,PF) as follows:
C(NOOD) Es~Ps [Ea Px [[gor (x) f1x)) -
where O' O(0o, Ox,0, Px,
(3)
3

---

Our definition of capacity measures the ability of a network to reach a new set of targets within lim- ited optimization budget from its current parameters and optimizer state. The choice of optimization budget and target distribution are left as hyperparameters, and different choices result in different notions of capacity. In reinforcement learning we ultimately care about the network's ' ability to fit its Bellman targets quickly, however the ability on its own will not necessarily be a useful measure: for example, a network which can only output the zero function will attain low Bellman error immedi- ately on sparse-reward environment, but will fail produce useful updates improve the policy. Our evaluations of this measure will use target functions that are independent of the current network parameters to avoid these pathologies; the effect of this choice is explored further in Appendix [.2
The process of training a neural network to fit a set of labels must by necessity change some prop- erties of the network. Works studying the information bottleneck principle (Tishby & Zaslavsky, 2013), for example, identify a compression effect of training on the latent representation, where inputs with similar labels are mapped to similar feature vectors. This compression can benefit gen- eralization the current task, but in the face of the rapidly-changing nature of the targets used in value iteration algorithms may harm the learning process by impeding the network's ability to fit new targets. This motivates two hypotheses. First: that networks trained to iteratively fit a sequence of dissimilar targets will lose their capacity to fit new target functions (Hypothesis 1), and second: the non-stationary prediction problems in deep RL also result in capacity loss (Hypothesis 2).
To evaluate Hypothesis 1, we construct a series toy iterative prediction problems on the MNIST data set, a widely-used computer vision benchmark which consists of images of handwritten digits and corresponding labels. We first fit series of labels computed by randomly initialized neural network fe: we transform input-label pairs (z, y) from the canonical MNIST dataset to (æ,, fo(x)), where fo( is the network output. To generate new task, we simply reinitialize the network. Given a target function, we then train the network for fixed budget from the parameters obtained at the end of the previous iteration, and repeat this procedure of target initialization and training 30 times. We use a subset of MNIST inputs of size 1000 to reduce computational cost. In F see that the networks trained on this task exhibit decreasing ability to fit later target functions under a fixed optimization budget. This effect is strongest in the smaller networks, matching the intuition that solving tasks which are more challenging for the network will result in greater capacity loss. We consider two other tasks in Appendix B.2 obtaining similar results, well as a wider range of architectures. We find that sufficiently over-parameterized networks (on the order of one million parameters for a task with one thousand data points) exhibit positive forward transfer, however models which are not over-parameterized relative to the task difficulty consistently exhibit increasing error as the number of targets trained on grows. This raises a question concerning our second hypothesis: are the deep neural networks used by value-based RL agents on popular benchmarks in the over- or under- parameterized regime?
we train an agent's 's network checkpoints sampled over the course of training to fit randomly generated target functions. We provide full details of this procedure in Ap- pendix We generate target functions by randomly initializing neural networks with new parame- ters, and use the outputs of these networks as targets for regression. We then load initial parameters from an agent checkpoint at some time f, sample inputs from the replay buffer, and regress on the random target function evaluated on these inputs. We then evaluate the mean squared error after training for fifty thousand steps. We consider DQN (Mnih et al. 2015). a QR-DQN (Dabney al. 2018), and a Rainbow agent (Hessel et al. 2018). We observe in all three cases that as training pro- gresses agents' checkpoints on average get modestly worse at fitting these random targets in most environments; due to space limitations we only show two representative environments where this phenomenon occurs in Figure 2 and defer the full evaluation to Appendix C.3
To evaluate Hypothesis 2,
3.2 REPRESENTATION COLLAPSE AND PERFORMANCE
The notion of capacity in Definition measures the ability of a network to eventually represent a given target function. This definition reflects the intuition that capacity should not increase over time.
4

---

montezuma, revenge
Train upicode return
600
montezuma revenge
Feature rank
400
400
Feature rank
400
Train apicods
10
200
230
200
-10
RC- DQN
DDQN
(R-DDN
50
100
150
200
50
100
150
200
0
50
100
150
200
50
100
150
200
Millions of frames
Millions of Tramns
Millions frames
Killions of frames
Figure 3: Feature rank and performance over the course of training for Montezuma's Revenge (left) and Pong (right). We observe that feature rank is higher for environments and auxiliary tasks which provide denser reward signals than for sparse reward problems.
However, deep RL agents must quickly update their predictions in order to make efficient learning progress. We present an alternate measure of capacity that captures this property which we call the feature rank, as it corresponds to an approximation of the rank of feature embedding. Intuitively, the feature rank measures how easily states can be distinguished by updating only the final layer of the network. This approximately captures a network's ability to quickly adapt to changes in the target function, while being significantly cheaper estimate than Definition [T
Definition 2 (Feature rank). Let X - R.d be feature mapping. Let Xn C X be set of states in X sampled from some fixed distribution P. Fix & 2 O, and let ((X,) E Raxd denote the matrix whose rows are the feature embeddings of states x E X,. Let SVD(M) denote the multiset of singular values of matrix M. The feature rank of & given input distribution P is defined to be
p P.c) = lim Ex.~plle E SVD (Xx))
(4)
n-+o0
for which a consistent estimator can be constructed as follows, letting X X, X =
0, lKo E SVD ( -EHE
(5)
The numerical feature rank (henceforth abbreviated to feature rank) is equal to the dimension of the subspace spanned by the features when E and the state space X' is finite, and its estimator is equal the numerical rank (Golub et al.) 1976 Meier & Nakatsukasa (02T] of the sampled feature matrix. For 0, it throws away small components of the feature matrix. We show that p is well-defined and that P.. is consistent estimator in Appendix A. Our analysis of the feature rank resembles that Kumar et al. (2021), but differs in two important ways: first, our estimator does not normalize by the maximal singular value. This allows us to more cleanly capture representation collapse, where the network features, and thus also their singular values, converge to zero. Second, we are interested the capacity agents with unlimited opportunity to interact with the environment, rather than in the data-limited regime. We compare our findings on feature rank against the srank used in prior work in Appendix B.2
In our empirical evaluations, train a double DQN (DDQN) agent, quantile regression (QRDQN) agent, and a double DQN agent with an auxiliary random cumulant prediction task (RC DQN) (Dab-) ney et al. 202I), on environments from the Atari suite, then evaluate Pn with n = 5000 on agent checkpoints obtained during training. We consider two illustrative envirouments: Montezuma'ss Revenge (sparse reward), and Pong (dense reward), deferring two additional environments to Ap- pendix We run random seeds on each environment aggnt combination.
We visualize agents' feature rank and performance in Figure 3 Non-trivial prediction tasks, ei- ther value prediction in the presence of environment rewards auxiliary tasks, lead to higher feature rank. In Montezuma's 's Revenge, the higher feature rank induced by RC DQN corresponds to higher performance, but this auxiliary loss can have a detrimental effect on learning progress in complex, dense-reward games presumably due to interference between the random rewards and the true learning objective. Unlike in target-fitting capacity, we only see a consistent downward trend in sparse-reward environments, where a number of agents, most dramatically QR-DQN, exhibit representation collapse. We discuss potential mechanisms behind this trend in Appendix A.2
5

---

(b)
sparse pong
20-
500
0.5
howling
centipeds
104
400
grovtarr
mortizuma
100
PBb
500
400
900
1.01
private_rv
Train return
208
Human-
100
0.5
6.0
100
200
300
400
500
Millions
Effective Bank
Figure 4: (a): Agent capacity vs human-normalized score in games where Rainbow does not achieve superhuman performance. While feature rank does not appear to solely determine agent perfor- mance, there is positive correlation between feature rank and human-normalized score. Bottom row contains Rainbow agents trained with the regularizer presented in Equation 6 (b) An 'unlucky' seed from our evaluations on the sparsified version of Pong, where learning progress occurs only after the agent recovers from representation collapse.
Figure 4a reveals a correlation between learning progress and feature rank for the Rainbow agent (Hessel et al. 2018) trained on challenging games in the Atari 2600 suite where it fails to achieve human-level performance; this trend also reflected for other agents described in the next section. The points the scatterplot largely fall into two clusters: those with low feature rank, which attain less than half of the average human score, and those with high feature rank, which tend to attain higher scores. Having a sufficiently high feature rank thus appears to be necessary condition for learning progress, as demonstrated by the learning curves shown in Figure Fp which highlights an unlucky agent trained on a variant of Pong (described in Appendix C.2 which experienced represen- tation collapse, and only solved the task after it had overcome this collapse. However, high feature rank does not appear to be sufficient for learning progress. Other properties an agent, such as its ability perform accurate credit assignment, the stability of its update rule, the suitability of its optimizer, its exploration policy, and countless others, must be appropriately tuned given task in order for progress to occur. Simply mapping inputs to a relatively uniform distribution in feature space will not overcome failures in other components of the RL problem. An agent must be able to both collect useful learning signals from the environment and effectively update its predictions in response to those signals order to make learning progress. This section has shown that at least in some instances poor performance can be attributed to the latter property.
4 INFER: MITIGATING CAPACITY LOSS WITH FEATURE REGULARIZATION
The previous section showed that capacity loss occurs in deep RL agents trained with online data, and in some cases appears to be bottleneck to performance. We now consider how it might be mit- igated, and whether explicitly regularizing the network to preserve its initial capacity improves per- formance in environments where representation collapse occurs. Our approach involves a function- space perspective on regularization, encouraging networks to preserve their ability to output linear functions of their features initialization.
4.1 INFER: FEATURE-SPACE REGULARIZATION
Much like parameter regularization schemes seek to keep parameters close to their initial values, we wish to keep a network's 's ability to fit new targets close to its initial value. We motivate our approach with the intuition that a network which has preserved the ability to output functions it could easily fit initialization should be better able to adapt to new targets. To this end, we will regress set network outputs towards the values they took at initialization. Our method, Initial Feature Regularization (InFeR), applies la regularization penalty the output-space level by regressing

---

(a)
(b)
Roralive Rog Esuon Tesk
(c)
1e3
Qucbi (1)))))
Mean squared error
0.5
o(x; 8t)
alter 5k steps
0.5
Return
o(x; Ag.
git (x::)
0.0
10
150
200
X
Target Fteration
Millions of frames
Regression
InFeR
DDON+InFet
DOON
Rainbow-lnifet
Rainbow
(d)
Human momannNorrmalized Return Atari
%6 Improvement
(VS Ra (boo)
400
Capped Normalized Return Normmlizzdd Return
sxteraice
batth spacy anverdn
ting fiohllaee
attartii
oor kumg Eamas craay chimterr
haver
Kargeu Srgparee This gu for
pittail
rolocaa
ytur gurntt proei aftee
video titat CarkTaae
ice thrtakcte
besin /ittr
aerteedd del doleenetee
pricatt Tye glooni aborr cleppet berank
part
Figure 5: (a) Visualization of InFeR. (b) Analysis of the effect of InFeR on capacity loss. (c) Effect of InFeR on performance in Montezuma's Revenge with respect to Rainbow and Double DQN baselines. (d) Performance of InFeR relative Rainbow on all 57 Atari games.
set of auxiliary network output heads match their values initialization. Similar perspectives have been used to prevent catastrophic forgetting in continual learning (Benjamin et al.) 2019).
In our approach, illustrated in Figure 5 we begin with a fixed deep Q-learning neural network with parameters 8, and modify the network architecture by adding auxiliary linear prediction heads gi on top of the feature representation Φo. We take a snapshot of the agent's parameters at initialization 0o and use the outputs of the k auxiliary heads under these parameters as auxiliary prediction targets. We then compute the mean squared error between the outputs of the heads under the current parameters (z; 0,) and their outputs at initialization (z; 0o). This approach has the interpretation of amplifying and preserving subspaces of the features that were present at initialization. In practice, we find that scaling the auxiliary head outputs by a constant 3 increases this amplification effect. This results the following form of our regularization objective, where we let B denote the replay buffer sampling scheme used by the agent:
&
(8,0:R(,,;; = En-B Σ ())())))- B9: (o)))
We evaluate the effect of incorporating this loss in both DDQN Van Hasselt et al. 2016) and Rain- bow (Hessel et al. 2018) agents, and include the relative performance improvement obtained by the InFeR agents over Rainbow on 57 games from the Atari 2600 suite Figure 5 deferring the com- parison to DDQN, where the regularizer improved performance slightly on average but only yielded significant improvements sparse- reward games, to the appendix. We observe a net improvement over the Rainbow baseline by incorporating the InFeR objective, with significant improvements in games where agents struggle to obtain human performance. The evaluations in Figure are for k = 10 heads with 3 = 100 and a = 0.1, and we show the method's robustness to these hyperpa- rameters in Appendix C.I We further observe in Figure that the InFeR loss reduces aarget-fitting enror on the stationtt MNIST prediction task described in the previous section. We show in Appendix C.2 that InFeR tends to increase the feature rank of agents trained on the Atari domain over the entire course of training; we study the early training period in Appendix C.3
7

---

The striking improvement obtained in the sparse-reward Montezuma's Revenge environment begs the question of whether such results can be replicated in other RI agents. We follow the same experimental procedure as before, but now use the DDQN agent, see Figure 5 We find that adding InFeR to the DDQN objective produces a similar improvement as does adding it to Rainbow, leading the DDQN agent, which only follows an extremely naive e-greedy exploration strategy and obtains zero reward all points in training, to exceed the performance of the noisy networks approach taken by Rainbow in the last 40 million training frames. This leads two intriguing conclusions: first, that agents which are explicitly regularized to prevent representation collapse can make progress in sparse reward problems without the help of good exploration strategies; and second, that this form of regularization yields significantly larger performance improvements in the presence of additional algorithm design choices that are designed to speed up learning progress.
4.2 UNDERSTANDING How INFER WORKS
While InFeR improves performance on average across the Atari games, its improvements are con- centrated principally games where the baseline rainbow agent performs significantly below the human baseline. It further slows down progress in a subset of environments such as Asteroids and Jamesbond. We now investigate two hypothesized mechanisms by which this regularizer may shape the agent's 's representation, in the hopes of explaining this differential effect on performance. Hy- pothesis 1: InFeR improves performance by preserving a random subspace of the representation that the final linear layer can use to better predict the value function. The effect of the regularizer on other aspects of the representation learning dynamics does not influence performance. Hypothesis 2: The InFeR loss slows down the rate which the learned features at every layer of the network can drift from their initialization in function space, improving the learning dynamics of the entire network to prevent feature collapse and over-fitting to past targets. The precise subspace spanned by the auxiliary weights is not directly useful to value function estimation.
To evaluate Hypothesis 1, we concatenate the outputs of a randomly initialized network to the feature outputs of the network used learn the Q-function, and train a linear layer on top of these joint learned and random features. If Hypothesis I were true, then we would expect this architecture to perform comparably to the InFeR agents, as the final linear layer has access to randomly initialized feature subspace. Instead, Figure 0 shows that the performance of the agents with access to the random features to be comparable to that of the vanilla Rainbow agents, confirming that the effect of InFeR on earlier layers is crucial to its success.
We now consider Hypothesis 2. InFeR limits the degrees of freedom with which network can collapse its representation, which may reduce the flexibility of the network make the changes necessary to fit new value functions, slowing down progress in environments where representation collapse is not a concern. In such cases, increasing the dimension of the layer to which we apply InFeR should give the network more degrees of freedom to fit its targets, and so reduce the per- formance gap induced by the regularization. We test this hypothesis by doubling the width of the penultimate network layer and comparing the performance InFeR and Rainbow on games where
phoenix
montezuma, revenge
berzerk
asteroids
hero
ameshand
Roum
100
200
102
200
100
200
0.0
200
100
200
100
20c
Million frames
Millon of fromes
Millons of frames
Millins frames
Millons of frames
Millins frames
RainboweR8
Rainbow
Raiboweeteick
Doublekainbow
Doublekajtbcee sineeeeeeokk
Rainbow
Rairbow-IiFFR
Figure 6: Left: agent performance does not improve over baseline when random features are added to the representation. Right: doubling the width of the neural network narrows the performance gap in games which InFeR under- mndeorperformed relative to Rainbow.
8

---

InFeR hurt performance in the original network. We refer to this agent as DoubleRainbow. We see in Figure 6 that increasing the network's size reduces, eliminates, or in some cases reverses the performance gap induced by InFeR in the smaller architecture. We therefore conclude that the principal mechanism by which InFeR affects performance is by regularizing the entire network's learning dynamics.
5 RELATED WORK
Suitably designed auxiliary tasks have been shown improve performance and encourage learned representations satisfy desirable properties in wide range of settings (Jaderberg al., 2017 Veeriah al., 2019 Gelada et al. 2019 Machado et al. 2018), with further insight given by prior analysis of the geometry (Bellemare et al. 2019) and stability (Ghosh & Bellemare 2020) of value functions in RL. Our analysis of linear algebraic properties of agents' representations is comple- mented by prior works which leverage similar ideas to analyze implicit
(Ku- mar et al. 2021) and spectral normalization (Gogianu et al.) 2021) in deep RL agents, and by the framework proposed by Lyle et al. (202T) to study learning dynamics in deep RL agents. In contrast prior work, which treats the layers of the network which come before the features black box, explicitly study the properties and learning dynamics of the whole network.
A separate line of work has studied the effect of interference between sub-tasks in both reinforcement 202T and supervised learning settings Igl et al.
learning (Schaul et al. 2019: Teh et al., 2017
Beck et al. 2021 Of particular interest has been catastrophic forgetting, with prior work proposing novel training algorithms using regularization
(Sharkey & Sharkey 1995 Ash & Adams 2020
(Kirkpatrick et al., 2017 Bengio al. 2014 Lopez-Paz Ranzato 2017) or distillation (Schwarz)
2017) approaches. Methods which involve re- network have seen particular success reducing interference between tasks in deep reinforcement learning (Igl et al. 2021 Tch et al. [017 Rusu et al., 2016, Fedus et al. 2220)). A closer relative of our approach is that of Benjamin et al. 2019), which also applies a function- space regularization approach, but which involves saving input-output pairs into a memory bank with the goal of mitigating catastrophic forgetting. Unlike prior work, InFeR seeks to maximize performance on future tasks, works without task labels, and incurs minimal, fixed computational cost independent of the number of prediction problems seen during training.
al. 2018 Silver & Mercer 2002
Li Hoiem
initializing
CONCLUSIONS
This paper has demonstrated fundamental challenge facing deep RL agents: loss of the capacity to distinguish states and represent target functions the course of training. We have shown that this phenomenon is particularly salient in sparse-reward settings, in some cases leading to complete collapse of the representation and preventing the agent from making learning progress. Our analysis revealed a number of nuances to this phenomenon, showing that larger networks trained on rich learning signals are more robust to capacity loss than smaller networks trained to fit sparse targets. To address this challenge, we proposed a regularizer to preserve capacity, yielding improved per- formance across a number of settings in which deep RL agents have historically struggled match human performance. Further investigation into this method suggests that it is performing form of function space regularization the neural network, and that settings where it appears the task reduces performance are actually instances of under- parameterization relative to the difficulty of the environment. Particularly notable is the effect of incorporating InFeR in the hard exploration game of Montezuma's 's Revenge: its success here suggests that effective representation learning can allow agents to learn good policies in sparse-reward environments even under naive exploration strategies. Our findings open up a number of exciting avenues for future work in reinforcement learning and beyond to better understand how preserve plasticity in non-stationary prediction tasks.

---

ACKNOWLEDGEMENTS
Thanks to Georg Ostrovski, Michael Hutchinson, Joost van Amersfoort, Daniel Guo, Diana Borsa, Anna Harutyunyan, Razvan Pascanu, Caglar Gulcehre, Srivatsan Srvinivasan, and Remi Munos for helpful discussions and feedback on early versions of this paper. CL supported by an Open Philanthropy Al Fellowship.
10

---

REFERENCES
Miguel Abreu, Luis Paulo Reis, and Nuno Lau. Learning run faster in a humanoid robot soccer environment through reinforcement learning. In Robot World Cup, pp. 3-15. Springer, 2019.
Alessandro Achille, Matteo Rovere, and Stefano Soatto. Critical learning periods in deep networks. In International Conference on Learning Representations, 2018.
Jordan Ash and Ryan P Adams. On warm-starting neural network training. In Neural Information Processing Systems (NeurIPS), 2020.
Nathan Beck, Durga Sivasubramanian, Apurva Dani, Ganesh Ramakrishnan, and Rishabh Iyer. Ef- fective evaluation of deep active learning image classification tasks. arXiv, 2021.
Mare Bellemare, Will Dabney, Robert Dadashi, Adrien Ali Taiga, Pablo Samuel Castro, Nicolas Le Roux, Dale Schuurmans, Tor Lattimore, and Clare Lyle. A geometric perspective on optimal representations for reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2019.
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Emmanuel Bengio, Joelle Pineau, and Doina Precup. Correcting momentum in temporal difference learning. In NeurlPS Deep Reinforcement Learning Workshop, 2020.
Yoshua Bengio, Mehdi Mirza, Ian Goodfellow, Aaron Courville, and Xia Da. An empirical inves- tigation of catastrophic forgeting in gradient-based neural networks. In International Conference on Learning Representations (ICIR), 2014.
Ari Benjamin, David Rolnick, and Konrad Kording. Measuring and regularizing networks in func- tion space. In International Conference on Learning Representations (ICLR), 2019.
Will Dabney, Mark Rowland, Mare Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI Conference Artificial Intelli- gence, volume 32, 2018.
Will Dabney, André Barreto, Mark Rowland, Robert Dadashi, John Quan, Mare Bellemare, and David Silver. The value-improvement path: Towards better representations for reinforcement learning. In AAAI Conference Artificial Intelligence, 2021.
William Fedus, Dibya Ghosh, John D Martin, Marc G Bellemare, Yoshua Bengio, and Hugo Larochelle. On catastrophic interference in Atari 2600 games. arXiv, 2020.
Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Mare Bellemare. DeepMDP: Learning continuous latent space models for representation learning. In International Conference on Machine Learning (ICML), 2019.
Dibya Ghosh and Mare G Bellemare. Representations for stable off off-policy reinforcement learning. In International Conference on Machine Learning (ICML), 2020.
Florin Gogianu, Tudor Berariu, Mihaela Rosca, Claudia Clopath, Lucian Busoniu, and Razvan Pas- canu. Spectral normalisation for deep reinforcement learning: an optimisation perspective. arXiv preprint arXhv:2105.05246, 2021.
Gene Golub, Virginia Klema, and Gilbert W Stewart. Rank degeneracy and least squares problems. Technical report, STANFORD UNIV CA DEPT OF COMPUTER SCIENCE, 1976.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
11

---

Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In AAAI conference on artificial intelligence, 2018.
Maximilian Igl, Gregory Farquhar, Jelena Luketina, Wendelin Boehmer, and Shimon Whiteson. Transient non-stationarity and generalisation in deep reinforcement learning. In International Conference on Learning Representations (ICLR), 2021.
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukeuoglu. Reinforcement learning with unsupervised auxiliary tasks. In International Conference on Learning Representations (ICLR), 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka BarbbkkaBaawiskka, Demis Hass- abis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgettt in neural networks. Proceedings of the National Academy of Sciences, 114(13):3511-3556, 2017.
Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit under-parameterizatiien inhibits data-efficient deep reinforcement learning. In International Conference on Learning Rep- resentations (ICLR), 2021.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40( (0(12):2935-299) 2017.
David Lopez-Paz. and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. In Neural Information Processing Systems (NIPS), 2017.
Clare Lyle, Mark Rowland, Georg Ostrovski, and Will Dabney. On the effect of auxiliary tasks on representation dynamics. In Artificial Intelligence and Statistics (AISTATS), 2021.
Andrew L Maas, Awni Y Hannun, Andrew Y Ng, et al. Rectifier nonlinearities improve neural network acoustic models. In Proceedings of the International Conference on Machine Learning. Citeseer, 2013.
Marlos C. Machado, Clemens Rosenbaum, Xiaoxiao Guo, Miao Liu, Gerald Tesauro, and Murray Campbell. Eigenoption discovery through the deep successor representation. In International Conference on Learning Representations (ICLR), 2018
Maike Meier and Yuji Nakatsukasa. Fast randomized numerical rank estimation. arXiv e-prints, pp. arXiv-2105, 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518((550))559955333 2015.
Matej Moravčík, Martin Schmid, Neil Burch, Viliam Lisý, Dustin Morrill, Nolan Bard, Trevor Davis, Kevin Waugh, Michael Johanson, and Michael Bowling, DeepStack: Expert-level artificial intelligence in heads-up no-limit poker. Science, 356(6337):508 513, 2017. ISSN 0036-8075.
John Quan and Georg Ostrovski. DQN Zoo: Reference implementations of DQN-based agents, 2020. URL
com /wtam//deeiin/////ee haapinddddqppaaa zoo
Roberta Raileanu, Maxwell Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus. Automatic data augmentation for generalization in reinforcement learning. arXiv, 2020.
12

---

Andrei A. Rusu, Sergio Gomez Colmenarejo, Çaglar Gülgehre, Guillaume Desjardins, James Kirk- patrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distil- lation. In International Conference on Learning Representations (ICLR), 2016.
Tom Schaul, Diana Borsa, Joseph Modayil, and Razvan Pascanu. Ray interference: source of plateaus in deep reinforcement learning. arXiv, 2019.
Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Baabska-Barwinska, Yee Whye
Teh, Razvan Pascanu, and Raia Hadsell. Progress compress: A scalable framework for contin- ual learning. In International Conference Machine Learning (ICML), 2018.
Noel E Sharkey and Amanda JC Sharkey. An analysis of catastrophic interference. Connection Science, 1995.
Daniel Silver and Robert E Mercer. The task rehearsal method of life-long learning: Overcoming impoverished data. In Conference of the Canadian Society for Computational Studies of Intelli- gence, pp. 90-101, 2002.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui. Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of Go without human knowledge. Nature, 35007676):354359, 2017.
Yee Whye Teh, Victor Bapst, Wojeiech M Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning, In Neural Information Processing Systems (NIPS), 2017.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 ieee information theory workshop (itw), pp. 1-5. IEEE, 2015.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double Q- learning. AAAI Conference on Artificial Intelligence, 2016.
Vivek Veeriah, Matteo Hessel, Zhongwen Xu, Richard Lewis, Janarthanan Rajendran, Junhyuk Oh, Hado van Hasselt, David Silver, and Satinder Singh. Discovery of useful questions as auxiliary tasks. In Neural Information Processing Systems (NeurIPS), 2019.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3-4):279-22 222.2,,, 1992.
13

---

A THEORETICAL RESULTS
A. ESTIMATOR CONSISTENCY
We here show that our estimator of the agent's feature rank is consistent. First recall
d(r.)d(a:))
Vn
Vn
n
The following property of the expected value holds
Es-rlo(z))(2)) = Σ
(8)
It is then straightforward to apply the strong law of large numbers. To be explicit, we consider an element of M = EEoo J. Mag.
Σ _ (2s))) 3 Maj
[[((())((2) )=y]= M, =[(=)))()))))
Since we have convergence any Mij. we get convergence of the resulting matrix to M. Because the singular values of are the eigenvalues of M and the eigenvalues are continuous functions of that matrix, the eigenvalues of M, converge to those of M almost surely. Then for almost all values of e, the threshold estimator , Specifically, the estimator will be convergent for all values of c which are not eigenvalues of M itself.
[{ <}] will converge to ((seec (M); e).
A.2 FEATURE DYNAMICS
We apply similar analysis to that of Lyle et al. (2021) to better understand the effect of sparse-reward environments on representation collapse. To do so, we consider the setting where 4 are features and
wy a linear function approximator which jointly parameterize a value function V/2 = (I)( wy). We will be interested in studying a continuous-time approximation to learning, where the discrete- time expected updates
T - t a Vo. IVV, R" Wt + ne + VV Vi r)V, + RT translated into a continuous-time flow, described by the following equations.
(10)
(11)
0,4, a(> (w,, R"
(12)
0,w, 3 [(> I) + +R),
(13)
where p E XxX is the matrix of state-transine probabilities under and R E R is the vector of expected rewards.
One of the key take-aways of prior works that under certain assumptions, a tabular value function following continuous-ting time TD dynamics will converge its limiting value V along the principal components of the environment's transition matrix. In the function-approximation case described complex. However, it turss out that under
above, the dynamics of the features e are somewhat certain training regimes, we can obtain similar convergence results for the features. We therefore turn our attention to ensemble prediction, where M linear prediction 'heads', each using separate weight vector with (m = 1,..., ...,, M) are all trained to regress on the TD targets using the shared feature representation the state as input, resulting in the following dynamics.
M
0. M =a Σ
(RT +y p ah w." 10mh-0p - - w/"' (w
(14)
m=1
0:wg = MT T + 7 MM winh v,
(15)
14

---

We now restate the result of Lyle et al. (2021) regarding the behaviour of the representation in the limit of many ensemble heads.
Theorem (Lyle et al., 2021). For M N, les (o J:20 be the solution to Equation| 14 with each " for m = ,..... M initialised independently from N (00, d and fixed throughout training (3 = 0). We consider two settings: first, where the learning rate scaled as M and 03 1 for all M, and second where oh MT and the learning rate is equal 1. These two settings yield the following dynamics, respectively:
lim 0.0.4 E (I-""""")0 and
(16)
lim 0, - M-20
(I- -
(1-p""))0 Rd R Te~
(17)
The corresponding limiting trajectories for fixed initialisation t Rxxd are therefore given respectively by
lim D, F =exp(-t( 7 pP and -oc
(18)
lim T. exp( t([ y (PT)) - (I- + (1 - - RTETT - N(,,
M-00
(19)
One important corollary of this result occurs in sparse-reward environments under sub-optimal poli- cies, where R = 0. In this case, we see that the representation converges precisely to the zero vector.
Corollary 1. Let M,, (w) be defined as in Theorem 1 Then if R = 0, the feature represen- tation converges to the zero vector for every state, independent of whether the learning rate a scaled as 1 or the linear weight initialization variance scales as In particular:
lim lim D MM 0. t-000 M
(20)
As result, we have that the feature rank of & will also tend to zero
Ve > 0
lim lim Ho E SVD M No > etl Eo.
(21)
1000 M-000
Proof. The proof of this result follows from a straightforward application of Theorem 1 setting
R = 0 and letting t + xo. We can obtain an analogous result for the srank of &.dd when pr diagonalizable by noting that for any eigenvector v, of P", the value of v o. 2; evolves as e eexp(-tA) for some constant e that depends on D.M In this case, we obtain a limiting value for the srank so long as px corresponds to an ergodic Markov chain.
The setting of this result is distinct from that of deep neural network representation dynamics, as neural networks use discrete optimization steps, finite learning rates, and typically do not leverage linear ensembles. However, we emphasize two crucial observations that suggest the intuition devel- oped in this setting may be relevant: first, in sparse reward environments the representation will be pushed to zero along dimensions spanned by the linear weights used to compute outputs. Once suffi- ciently many independent weight vectors are being used to make predictions, this effectively forces every dimension of the representation to fit the zero vector output. We would therefore expect rep- resentation collapse to be particularly pronounced in the QR-DQN agents trained on sparse-reward environments, in this setting we obtain many independently initialized heads all identically trying to fit the zero target.
Second, in the presence of ReLU activations and stochastic optimization, the trajectories followed by the learned features in deep neural networks run the risk of getting 'trapped' in negative values. If these features would normally tend to small values close to zero (as we would expect in agents following similar dynamics to those obtained in Theorem TT. this increases the risk of unit saturation, where the representation may get trapped in bad local minima. This appears to be what happens in the QR-DQN agents trained on sparse-reward environments such as Montezuma's Revenge.
15

---

B SEQUENTIAL SUPERVISED LEARNING
B.1 DETAILS: TARGET-FITTING CAPACITY NON-STATIONARY MNIST
In addition our evaluations in the Atari domain, we also consider a variant of the MNIST dataset in which the labels change over the course of training.
Inputs and Labels: We use 1000 randomly sampled input digits from the MNIST dataset and assign either binary random targets.
Distribution Shift: We divide training into N = 30 or N = 10 iterations depending on the structure the target function. In each iteration, a target function randomly sampled, and the network's s parameters obtained the end of the previous iteration are used the initial values for new optimization run. We use the Adam (Kingma & Ba 2013) optimizer with learning rate 1 e-3, and train minimize the mean squared error between the network outputs and the targets for either 3000 5000 steps depending on the nature of the target function.
Architecture: we use standard fully-connected architecture with ReLU activations, and vary with width and depth of the network. The parameters at the start of the procedure are initialized following the defaults in the Jax Haiku library.
We note that the dataset sizes, training budgets, and network sizes in the following experiments are all relatively small. This was chosen to enable short training times and decrease the computational budget necessary too replicate the experiments. The particular experiment parameters were selected to be the fastest and cheapest settings in which we could observe the capacity loss phenomenon, while still being nontrivial tasks. In general, we found that capacity loss is casiest to measure in 'sweet spot' where the task for given architecture is simple enough for a freshly-initializee network to attain low loss, but complex enough that the network cannot trivially solve the task. the findings of the following section, we see how some of the larger architectures don't exhibit capacity loss on "casier' target functions, but do on more challenging ones that exhibit less structure. This suggests that replicating these results in larger networks will be achievable, but will require re- re-tuning the task difficulty to the larger network's 's capacity.
B.2 ADDITIONAL EVALUATIONS
We expand on the MNIST target-fitting task shown in the main paper by considering how network size and target function structure influences capacity loss.
Random-MNIST (smooth) this task uses the images from the MNIST dataset as inputs. The goal is perform regression the outputs of randomly initialized, fixed neural network. We use small network for this task, consisting of two width-30 fully connected hidden layers with ReLU activations which feed into a final linear layer which outputs a scalar. Because the network outputs are small, we scale them by 10 so that it is not possible to get a low loss by simply predicting the network's bias term. This task, while randomly generated, has some structure: neural networks tend to map similar inputs to similar outputs, and so the inductive bias of the targets will match that of the function approximator we train on them.
Hash-MNIST (mon-smooth) uses the same neural network architecture as the previous task to generate targets, however rather than using the scaled network output as the target, we multiply the output by le3 and feed it into sine function. The resulting targets no longer have the structure induced by the neural network. This task amounts to memorizing set of labels for the input points.
ThNIS (sparse) replaces the label of an image with binary indicator variable indicating whether the label is smaller than some threshold. To construct a sequence of tasks, we set the threshold at iteration i be equal to i. This means that at the first iteration,
16

---

Target-fitting error: non-smooth
32x2 MLP
32x4 MLP
256x2 MLP
30
0.30
200
0.4
30
0.25
squared
0.3
20
25
Rark
0.20
20
0.15
100
15
Mean
Mean
0.1
10
0.1
15
50
0.05
10
0.0
0.0
0.00
10
:0
10
256x4 MLP
1024x2 MLP
1024x4 MLP
0.05
0.20
000
0.04
200
0.15
0.08
0.03
150
0.10
400
0.04
400
0.02
100
0.05
200
200
0.30
0.00
0.00
10
200
Feature num.
Feature
Figure 7: Mean squared eπrоr the end of training on each iteration of the hash-MNIST task. Target-fitting error increases over time in smaller networks, but increasing the depth width of the network slows down capacity loss, enabling positive transfer in the largest networks we studied.
the labels of the form (x, 0) for all inputs At the second iteration, they are of the form (xx 8(y < 1)), where is the digit in the image and so on.
We consider MLP networks varying widths and depths, noting that the network architecture used generate the random targets is fixed and independent of the approximating architecture. We are interested in evaluating whether factors such as target function difficulty, network parameterization, and number of target functions previously fit influence the network's 's ability to fit future target func- tions. Our results are shown in Figure 7 g a We visualize srank and feature rank of the features output at the network's penultimate layer, in addition to the loss obtained at the end of each iteration.
B.3 EFFECT OF INFER TARGET-FITTING CAPACITY IN MNIST
In addition our study of the Atari suite, we also study the effect of InFeR the non-stationary MNIST reward prediction task with a fully-connected architecture; Figure I0 We find that significantly mitigates the decline in target-fitting capacity demonstrated in Figure
C ATARI EVALUATIONS
We now present full evaluations of many of the quantities described paper, along with a study of the sensitivity of InFeR its hyperparameters, We use the same training procedure for all of the figures in this section, loading agent parameters from checkpoints to compute the quantities shown.
C.I HYPERPARAMETER SENSITIVITY OF INFER IN DEEP REINFORCEMENT LEARNING AGENTS
We report results of hyperparameter sweeps over the salient hyperparameters relating to InFeR, so as to assess the robustness of the method. For both the DDQN and Rainbow agents augmented with InFeR, we sweep over the number of auxiliary predictions (1, 5, 10, 20), the cumulant scale used in the predictions (10, 100, 200), and the scale of the auxiliary loss (0.01, 0.05, 0.1, 0.2). We consider
17

---

Target-fitting error: smooth
32x2 MLP
32x4 MLP
256x2 MLP
30
150
0.4
0.15
125
0.3
squared
Mean squared
100
Rank
0.2
10
0.4
15
75
10
0.05
50
0.1
0.0
0.0
0.00
10
10
20
10
256x4 MLP
1024x2 MLP
1024x4 MLP
0.20
0.125
0.10
400
0.100
Moan squared
0.08
0.15
300
0.075
100
0.10
200
0.030
0.04
50
0.05
0.025
0.02
0.00
0.00
10
200
Feature
Feature srank
Figure 8: Mean squared error after 2c3 training steps on the random-MNIST task. Target-fitting error increases over time under-parameterizzed networks, but increasing the depth width of the network slows down capacity loss, enabling positive transfer in the largest network studied.
Target-fitting error: sparse
32x2 MLP
32x4 MLP
256x2 MLP
0.010
18
Mesit squered etror
Mean squared error
0.004
0.015
0.008
squered
0.006
25
Rank
125
0.004
20
0.002
100
0.005
15
0.001
75
10
50
0.000
0.000
0.030
0.0
2.5
5.0
7.5
0.0
2.5
5.0
7.5
0.0
2.5
5.0
7.5
256x4 MLP
512x2 MLP
512x4 MLP
140
600
0.036
150
0.004
Mean squared exion
0.004
Mean squared error
0.005
100
Mean squared
0.003
Pank
0.003
Fank
0.004
100
0.002
0.000
0.003
0.002
30
0.001
100
0.000
0.000
0.000
0.0
2.5
5.0
7.5
0.0
2.5
5.0
7.5
7.5
Feature
rank
Feature
Figure 9: Mean squared error after 2e3 training steps on the threshold-MNIST task. Target-fitting increases over time in under-parameterized networks, but increasing the depth width of the network slows down capacity loss, enabling positive transfer in the largest network studied.
18

---

Torget-fiting eiror on hash prediction task over time
Target- fitiing error on threshold prediction task over time
MLP MarNCII Lost
MLE Rastorn Lecs
MUP Metwor boss
AUP Sapvars Loss capth
MLR Barsaar
VAR Medaort Loss
Target-fttting prediction
WiS WIZ 2
anl read dediiw
toel deslled
Figure 10: Effect adding InFeR to the regression objective in random reward prediction problem on the non-stationary MNIST environment studied previously. We see that the InFeR objective pro- duces networks that consistently outperform those trained with standard regression objective, exhibiting minimal capacity loss in comparison to the same network architecture trained on the same sequence of targets.
202
0.20
100
Fam
0..00
0.01
0.05
0.15
0.29
100
200
20
AUL stale
camulant scale
num_aix
Figure 11: Hyperparameter sweeps for the DIQN+InFeR agent. Each contour plot shows average capped normalizd score at the end training marginalized over all hyperparameters not shown on its axes.
the capped human- normmazzed return across four games (Montezuma's 's Revenge, Hero, James Bond, and MsPacman), and run each hyperparameter configuration with 3 seeds. Results are shown in Figure I for the DDQN agent; we compare performance as each pair of hyperparameters varies (averaging across the other hyperparameter, games, and seeds, and the last five evaluation runs of each agent). Corresponding results for Rainbow are given in Figure 12
200
0.20
Rum, MIX
1.00
0..0
0.05
0.01
sux.scil
saaklsnt_ccak
num_auX
Figure 12: Hyperparameter sweeps for the Rainbow+InFeR agent. Each contour plot shows average capped human-normalized score at the end of training marginalized over all hyperparameters not shown on its axes.
19

---

deess
denos
530-
500
430
423
400
200
300
102-
.10
AC-DoN
60-
13300
169
100
-2
100
59
130
703
1000
100
150
Figure 13: Feature rank and performance of RI. agents on demonstrative Atari environments.
We train a Rainbow agent (Hessel et al. 2018) with the same architecture and hyper- parameters as are described in the open-source implementation made available by Quan Os- trovski (2020). We additionally add InFeR, as described in Section 4 with 10 heads, gradient weight 0.1 and scale 100.
Agent:
Training: We follow the training procedure found in the Rainbow implementation mentioned above. We train for 200 million frames, with 500K evaluation frames interspersed every IM training frames. We save the agent parameters and replay buffer every 10M frames to estimate feature dimension and target-fitting capacity.
C.2 FEATURE RANK
We first extend the results shown in Figure 3 to two additional games: Seaquest, and sparsified version of Pong in which the agent does not receive negative rewards when the opponent scores. In these settings, we stored agent checkpoints once every 10M frames in each 200M frame trajectory, and used 5000 sampled inputs from the agent's replay buffer to estimate the feature rank, using the cutoff = 0.01. Results are shown in Figure
We further evaluate the evolution of feature rank in agents trained on all 57 games in the arcade learning environment. We find that the decline in dimension after the first checkpoint at 10M frames shown across the different agents in the selected games also occurs more generally in Rainbow agents across most environments in the Atari benchmark. We also show that in most cases adding InFeR mitigates this phenomenon. Our observations here do not show a uniform decrease in feature rank or a uniformly beneficial effect InFeR. The waters become particularly muddied in settings where neither the Rainbow nor Rainbow+InFeR agent consistently make learning progress such as in tennis, solaris, and private eye. is outside the scope of this work to identify precisely why the agents do not make learning progress in these settings, but does not appear to be due to the type of representation collapse that can be effectively prevented by InFeR.
Procedure. We compute the feature rank by sampling n = 50000 transitions from the replay buffer and take the set of origin states as the input set. We then compute a n X d matrix whose row i is given by the output of the penultimate layer of the neural network given imput S.. We then take the singular value decomposition of this matrix and count the number of singular values greater than 0.01 to get an estimate of the dimension of the network's representation layer.
In most games, this decline in dimension holds even in the online RL setting where the agent's 's improving policy presumably leads it to observe more diverse set of states over time, which under fixed represen- tation would tend to increase the numerical rank of the feature matrix. This indicates that even in the face of increasing state diversity, agents' representations face strong pressure towards degeneracy. is worth noting, however, that the agents in dense-reward games do tend to see their feature rank increase significantly early in training: this is presumably due to the network initially learning to disentangle the visually similar states that yield different bootstrap targets.
a decline in feature rank after the first checkpoint 10M frames. Strikingly,
20

---

alien
amidar
assault
asterix
asteroids
512
500
512
512
505
450
400
atlantis
bank helst
battle
beam rider
_eam__ider
berzerk
500
500
525
500
510
500
400
bowling
baxing
breakout
centipede
pompper_coomand
300
500
200
450
475
450
crazy_clime climber
defender
Lemon_attace
double_dume
enduro
510
500
500
510
500
lahing_derby
freeway
frostbite
gopher
gravitar
500
400
500
510
490
508
hero
ice hockey
jamesbond
kangaroo
krull
510
500
500
450
475
490
400
kung_fu fu_ master
montezuma revenge
uss_pacman
name_ this_game
phoenix
512
500
512
500
475
450
250
511
508
pitfall
pong
private_ eyye
qbert
riverai
300
400
510
500
510
200
400
300
490
508
road runner
seaquest
skiing
520
500
510
500
500
505
450
475
space iivvadrrsdeesss
star _aa__g
surround
tennis
time_pilot
512
500
100
500
450
510
tutankham
_p_n_ down
venture
sideo_pinball
wizard of _f_ wor
500
500
500
500
475
400
250
475
400
yars_revee revenge
zaxxon
100
200
100
200
100
200
Millions of frames
Millions frames
Millions of frames
500
500
InFoR
400
450
100
200
100
200
Millions frames
MI Illoss of frames
Figure 14: feature rank of agent representations over the course of training all 57 games in the Atari benchmark. We compare Rainbow against Rainbow+InFeR, Rainbow+InFel does not uniformly prevent decreases feature rank across all games, but on average it has beneficial effect on preserving representation dimension.
21

---

C.3 TARGET-RITTING CAPACITY
In this section we examine the target-fitting capacity of neural networks trained with DQN, QR- DQN, and Rainbow over the course of 50 million environment frames five games in the Atari benchmark (amidar, montezuma's revenge, pong, bowling, and hero). Every 1 million training frames we save a checkpoint of the neural network weights and replay buffer. For each checkpoint, we generate random target network by initializing network weights with a new random seed. We then train the checkpoint network predict the output of this random target network for 10000 mini-batch updates (batch size of 32) under mean squared error loss, for states sampled from the first 100, 000 frames in the checkpoint's 's replay buffer. Furthermore, we repeat this for 10 seeds used to initialize the random target network weights.
The results of this experiment are shown in Figure 15 (in orange), where the solid lines show means and shaded regions indicate standard deviations over all seeds (both agent seeds (5) and target fitting seeds (10), for a total of 50 trials). We also show srank and feature rank of the features output at the network's penultimate layer for each of the checkpointed networks used for target fitting. These computed using the network features generated from 1000 states sampled randomly from that checkpoint's 's replay buffer. For feature rank, averages and standard deviations only over the agent seeds.
wellow of hera CR--ON
of
Miku of fanee pong 128-D2M0
2rdidar (QE-DOM)
bos bog 10R -O11
inceroour revegg (OR-DOM)
00005
L2000
Mf forns
Loss
Feature num. rank
Feature srank
Figure 15: Mean squared error, after 10000 training steps for the target-fitting on random network targets. We also show the corresponding feature rank of the pre-trained neural network (before target-fitting)
C.4 PERFORMANCE
We provide full training curves for both Rainbow and Rainbow+InFeR on all games in Figures 16 17 (capped Homan-normalized performance), and 18 & 19 (raw evaluation score). We also provide evaluation performance curves for DDQN and InRRR agents in Figure 20
22

---

allen
amidar
assault
asterix
asteroids
1.0
0.5
0.5
atlantis
bank _eist
battle_zone
beam_rider rider
berzerk
1.0
0.5
bowling
boxing
breakout
centipede
chopper_ command
0.25
1.0
0.75
0.50
0.00
0.5
0.25
crazy_c cliiberr
defender
demon_atee attacck
double dunk
enduro
1.0
1.0
1.0
0.5
0.5
0.5
fishing_derfe derby
freeway
frostbite
gapher
gravitar
1.0
0.5
0.5
0.5
0.0
hero
ice_hooe hocceyy
jamesbond
kangaroo
kruil
1.0
0
kung_ fu master
montezuma_revengge
ms_pacman
name_
name_this_gane this_game
phoenix
0.5
0.75
0.50
0.5
0.25
pitfall
pong
private_
qbert
riverraid
0.025
1.-
0.0
0.000
0.5
0.025
_an__rnner
robotank
seaquest
skiing
solaris
1.0
0.5
-0.5
0.00
space_imee jooce_invaders
star star_gunner
surround
tennis
time_pilot
1.0
0.5
tutankham
down
venture
video video_pinbal pinball
wizard of wor
1.0
0.5
_aar_revenne
zaxxon
0
Millions
100
200
Millions
100
200
Millions
100
200
Rainbow
Rainbow+Infedd
100
200
100
20D
Millions
Figure 16: Full evaluation of capped human-normalized performance Atari benchmarks for the default Rainbow architecture.
23

---

alien
amidar
assault
asterix
asteroids
0.5
0.5
0
atlantis
bank heist
battle_zone zone
beam beam_rider rider
berzerk
1.0
0
bowling
boxing
breakout
centipede
chopper_commare
le-1
2.5
5
crazy_climbal climber
defender
demon_ard attack
double __nk
enduro
0.5
0.5
-1
fishing_derye derby
freeway
frostbite
gopher
gravitar
1.0
0.5
0.5
hero
ice hockey
jamesbond
kangaroo
krull
1.0
1.0
kung_f fu_master
montezumaa_evvee
mp__pacman
name_ this_game
phoenix
1.0
1.0
2.5
7.5
5.0
0.5
0.5
0.5
2.5
-25
pitfall
pong
private eye
qbert
riverraid
2.5
1.0
0-
0.0
0.5
road_runner runner
robotank
seaquest
skiing
solaris
1
1.0
2.5
0.5
0.0
space invaders
star_gunner
surround
tennis
time_pilot
1.0
1
0.5
tutankham
up___down
venture
video_pinbal
wizard of wor _or
1.0
0.5
yars_reveeee revenge
zaxxon
Mi Ilions frames
100
200
Millions frames
100
200
100 Millians frames
200
1.0
Rainbow
0.5
Rainbow+inFeRR
100
200
100
200
Millions
Millions frames
Figure 17: Full evaluation of capped human noraa-normalized performance on Atari benchmarks in the double-width Rainbow architecture.
24

---

alien
amidar
asterix
asteroids
2-
atlantis
bank bank_heist heist
battle_zone
beam beam_rider rider
berzerk
1e5
1e3
lef
le1
1ei
2
bowling
boxing
breakout
centipede
chopper command
1e1
1c2
1e2
crazy_climber climmer
defender
@emon_attace
double_dunk dunk
double
enduro
1e5
1e5
1e5
lel
1e3
-2
fishing_de berbing_debby
freeway
frostbite
gopher
gravitar
1e2
lel
1o4
1c5
1e3
hero
ice ce__hokey
amesbond
kangaroo
krull
1ef
1el
lef
let
1e3
5.0
kung_fu fu master 1e4
montezuma nentezuma_revenge le3
pacman
name, this saime
phoenix
le3
1c4
le5
2
5.0
1.5
2.5
2.5
0.5
0.0
pitfall
pong
private_eye
qbert
riverraid
1e3
le1
le5
1c4
le4
0.0
road
robotank
seaquest
skiing
solaris
les
1e3
0-
space invaders
star_ _ar_gunner
surround
tennis
time_pilot
lel
lc4
2.5
0.0
tutankham
up_n_doww
venture
hideo_pinbba
wizard of _or wor
1+2
1a3
yars yaar_raereeeeeeaaee revenge
zaxxon
100
200
100
200
100 Millians frames
200
le5
le4
Mi llions frames
Millions frames
Rainbow
Rainbow+inFeR
100
200
100
200
Millions
Millions frames
Figure 18: Full evaluation of raw scores Atari benchmarks for the default Rainbow architecture.
25

---

alien
amidar
assault
asterix
asteroids
0
0
atlantis
bank heist
battle zone
beam beam_rider rider
berzerk
1e6
1e3
led
lel
1e4
0.0
bowling
boxing
breakout
centipede
peppper_commane
1e1
1o2
1e2
1e3
5.0
2.5
crazy climber
defender
demon attack le5
double __dnk
enduro
le5
1e5
lel
1e3
1-
fishing_derfs derby
freeway
frostbite
gopher
gravitar
102
le1
1o4
1e5
1e3
hero
ice_ hockey
amesbond
kangaroo
krull
lef
lel
lef
le1
1e3
7.5
kung_f fu ju_matter
montezuma revengee e33
papamaa
name_ this this_game
phoenix
1e3
1e5
5.0
1.5
5.0
2.5
2.5
1.0
pitfall
pong
private eye
qbert
riverraid
1e2
1e5
1e4
2.5
0
2.5
0.0
road_runner runner
robotank
seaquest
skiing
solaris
1e5
7.5
5.0
0.0
space invaders
star_gunee gunnerr
surround
tennis
time_pilee
1e4
lei
lel
1e4
2.5
0.0
tutankham
up_ uu_n_de down
venture
@ideo_pinball
wizard of wor
1e2
1e5
1e3
le5
165
yars yars_revenge revenge
1e4
zaxxon
Mi Millions frames
100
200
Millions frames
100
200
100 Millians frames
200
Rainbow
Rainbow+lnFeR
100
200
100
200
Millions
Millions
Figure 19: Full evaluation raw scores on Atari benchmarks for the double-width Rainbow archi- tecture.
26

---

alien
amidar
assault
asterix
asteroids
le3
1e3
1e3
e4
le3
5-
2
0.0
atlantis
bank heist
battle_zone zone
beam_ beam_rider
berzerk
1e6
1e5
1e4
1e4
103
2.5
0-
0.0
bowling
boxing
breakout
centipede
chopper_commene command
lel
le2
le2
1e3
les
5-
crazy climber
crazy_climber
defender
demon_attain attack 145
double dunk
enduro
145
1e4
lel
1e3
2.5
0.0
@ihhiggderda
freeway
frostbite
gopher
gravitar
le2
lel
le3
le4
1e3
2.5
2.5
0.0
0.0
hero
hockey
ameebond
kangaroo
krull
1e4
1e4
5.
kung master
montezuma_
ms pacman
name this hhi__aamee
phoenix
le4
5.0
0
pitfall
pong
private_ey
qbert
riverraid
2.5
le5
2
0-
0
-5
road runner
robotank
seaquest
skiing
solaris
1e4
lel
le4
1e3
2.5
-3
space incvaderresess
star _ar_ggnner
surround
tennis
time_pilot
1.44
1e5
le1
le1
1e4
2.5
0.0
-1
-2.5
tutankham
up n_ down
venture
video pinball Lidea_pinballl
wizard of wor Led
le2
1e4
le3
1e5
2.5
0.0
pars_revenge
zaxxon
100
200
100 Millions framnes
200
100
200
1.64
1o4
DOQN DOON+InFeR
100
200
100
200
Millions of frames
Millions frames
Figure 20: Evaluations of the effect InFeR on performance of Double DQN agent. Overall we do not see as pronounced an improvement in Rainbow, but note that the average human- normalized score over the entire benchmark is nonetheless slightly higher for the InFeR agent, and that the performance improvement obtained by InFeR in Montezuma's Revenge is still significant in this agent.
27


Reviews:
[{"review_id": "qu6bvA2uu-a", "reviewer": "Reviewer_KLqH", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a very well written and very insightful work. I took great pleasure and interest reading it.\nThe writing is flawless, the structure is very clear and the presentation of ideas unfolds naturally.\nThe contributions are significant, their evaluation is convincing, both in the method and in the results, and the discussion has an appreciable depth.\n\nI identify two weaknesses. First, some evaluation aspects are omitted, and I question their effect on the mechanism of InFeR. Second, some parts of the discussion could maybe be condensed to develop other aspects more in-depth, both in the main text or as references to the appendix. The following remarks list the aspects upon which I would like to see more details and/or an improved discussion in the paper.\n\nThe introduction of network capacity and, more importantly, the characterization of a network's effective dimension are well-developed ideas, with enough nuance with other approaches from the literature to be relevant contributions in themselves. In a sense, the effective dimension seems very similar in spirit to a VC dimension. Could the authors elaborate a little on this (both here and in the paper)?\n\nIn Figure 3, I fail to understand why the initial effective dimension of the QR-DQN network is always the smallest, except for the sparse pong task. This seems very counter-intuitive and should be justified.\n\nThe idea to regularize against a set of projections of the initial features is equivalent to defining a set of corresponding auxiliary tasks. I would appreciate seeing more details on how these $g_i$ projections are designed in the first place. Are they random projections? Is there a rationale in their construction? How dependent on their choice is the overall training?\n\nOne common remark on Deep RL papers is the very low number of seeds used to assess the statistical significance of experiments. This paper makes no exception: the canonical number of 3 seeds is used. I am well aware of the compute cost issue and I don't see this number of seeds (despite the associated low statistical confidence) as a reason to cross out these results. Nevertheless, the presented experiments display large variance in the results. Specifically, the performance gain of InFeR seems to come along an strong increase in performance variance. It seems unavoidable to discuss the causes for this variance. Is there a link with the construction of $g_i$? Or the variance of gradients for the ADP loss in sparse rewards environments? Is it related to the games InFeR is tested on? What do the corresponding policies actually do (a video would definitely be nice)? Do they act well consistently? I believe the discussion could be quite deeper on this question.\n\nIn the testing of hypothesis 1, concatenating with a fully random network is a rather radical comparison point. First, I believe the way this network is initialized might have an impact, and this should be at least mentioned explicitly (if not discussed). Second, I believe an in-between solution would consist in adding lateral connections, in a fashion similar to that of progressive neural networks. In a sense, this can be compared to adding random projections after each layer. This might provide a few more insights regarding the claim that the effect of InFeR on earlier layers is crucial to its success.\n\nAn interesting aspect which I would have liked to see is the evolution of the effective dimension at different depths of the network. Does representation collapse occur abruptly at a certain depth? Is it caused by early or deep layers in the network? Does it manifest within convolutional layers? Do shared weights (under-parameterization) act as a regularizer? By the way, what is the network architecture used? Classical DQN-style 3 convolutional layers + 2 fully connected ones (with the appropriate variations for QR-DQN)? What would happen with deeper networks such as the IMPALA architecture? \n\nMinor remark:  \nPage 2. I believe the $(X_t, A_t, R_t, X_{t+1})$ tuple of equation 2 was actually introduced as $(x_t, a_t, r_t, w_{t+1})$ a few lines above. Making this consistent would help.", "summary_of_the_paper": "This paper proposes a new regularization method for deep reinforcement learning, that is designed to prevent representation collapse and thus retain the ability of the network to fit functions that evolve in time, such as the sequence of Q-functions generated by approximate dynamic programming methods. This method consists in regularizing the network's features towards a set of initial linear transformations of their (the features) values at initialization.", "summary_of_the_review": "Overall this is a solid contribution on the question of preventing capacity loss in deep RL. The ideas are novel and the point of view developed on representation collapse, and the proposed way to prevent it, is a significant contribution to the community. A number of points could fruitfully be discussed, both in the main text and in the appendix, which would increase the practical impact of the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "qkpB38yAN2D", "reviewer": "Reviewer_vuGv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. The paper tackles a super important question which is the stability of training in RL. \n\n2. The high computational effort in running all Atari games for 200M frames.\n\nWeaknesses:\n\n3. The paper has no theoretical result, and even its capacity definition is thrown away quickly for a more practical measure. \n\n4. Despite the heavy experimentations, I wasn't convinced by some of the plots, and some seem overbearing:\n- Figure 1: Why use 3 algorithms to show the same point.\n- Figure 2: Not sure I understood what is shown here besides the difference between activation functions - is this the point of the figure? because it seems a bit outside the scope of the paper.\n- Figure 3: Does it make sense that at Frame count=0 not all plots start roughly at the same point - there wasn't training yet so why do we see a difference in the dimension? Also, we see the effective dimension can grow larger, if this is the case why would a collapse occur at all? Since the point in this Figure is to show a connection to sparsity, why not include a plot of the sparsity as well (like num steps where R!=0 divided by num steps)\n- Figure 5 (b): The results for Robotank doesn't really follow the theme of the paper - both methods reach the same score even though one loses capacity. For Montezuma there is no capacity loss.\n- Figure 5(c): The improvement does not look very consistent across games.\n\n5. The proposed method of Infer looks a bit naive to me - why preserve random functionals and not for example an autoencoder of the state or some other meaningful feature. This method forces the network to keep random functions of the state instead of learning the underlying structure. You can also try to regularize for the rho-hat directly using some relaxation.\n", "summary_of_the_paper": "The authors tackle a convergence issue in RL where the networks lose expressiveness during the training and are unable to regain it as the policy improves, ultimately leading to bounded performance. To solve this issue the authors propose adding a regularization term that forces the trained networks to regress to a random function of their initial output.", "summary_of_the_review": "While the paper tackles an important problem, the proposed solution and it's empirical exhibition are insufficient in my view for acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "fDttjpL0hSP", "reviewer": "Reviewer_Mqcv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "==========================\n\nStrengths:\n- novelty: this problem is not adequately studied in the RL setting, so I believe the authors are making some interesting contribution\n- interesting empirical analysis and insights are provided. \n- empirical result: the result on Montezuma's revenge show that the proposed method (which is very simple) can improve performance a lot, which seems to support the claims of the paper. \n- writing quality: good\n\n==========================\n\nWeaknesses: \n\nMajor concern:\n\nFigure 4 (b) I'm a bit concerned whether this figure actually supports the argument that agent gradually lose capacity during training. Authors claim that this figure shows that learning happens only after agent *recovers* from representation collapse. Note that Figure 1, 2 show that agent gradually lose capacity during training. But figure 4 (b) shows this agent's capacity is \"collapesed\" right from the very beginning of trainining, and then later this capacity is *significantly increased*. To make it more clear: \n\n1) Figure 1, 2 seem to show capacity (measured with MSE) starts *high* then gradually get *lower*\n\n2) Figure 4 (b) seems to show capacity (measured with effective rank) starts \"low\" then can get \"high\"\n\nSo it seems to me these 2 conclusions are actually... disagreeing with each other? Would be great if you can address this concern.\n\nAn potentially interesting experiment is... if InFeR is added to middle of training that sparse pong unlucky seed, should we see the capacity immediately increase? To be specific, I'm talking about comparing 2 curves, the first curve trains that unlucky seed as usual, the second curve uses the same seed, but when trains to sth like 50M frames, InFeR is added. In this case, will we see InFeR immediately increase the effective rank, and potentially immediately after bring performance up? \n\n\nOther concerns and comments:\n\n- One thing seems a bit lacking to me is comparison to other methods that improve representation learning. Have you try compare InFeR with any other methods that center around representation learning? For example, contrastive learning methods like CURL, will they achieve the same capacity-preserving effect? What is the advantage of using InFeR compared to other representation learning methods? \n\n- A disucssion on the potential negative effect of the proposed regularization scheme would be good: in some environments the performance dropped a bit after adding INFER, what could be the reason? And if the initial values are simply not good enough, will this method lead to performance drop? \n\n- Page 4 when authors explain Figure 2, \"We further consider a ‘sparse-reward’ version of MNIST...\" after this there is one sentence to explalin the MNIST experiment setting, and this seems a bit too concise. I can still get what you mean, but would really prefer you try to expand the explanation a bit and maybe give an example or schematic, you can put this in the appendix if needed, it would be really helpful for readers to understand it fast and prevent potential confusion. \n\n- page 1 in introduction, some very recent results on data augmentation (mainly DrQ and DrQv2, and this is also used in CURL, temporal augmented contrast etc.) actually find that data augmentation can boost RL performance quite robustly. Perhaps you want to consider rewrite this part slightly? ", "summary_of_the_paper": "Summary of paper:\n- The authors identify a problem of \"capacity loss\" during RL training with neural networks, this is the problem of agents gradually losing the ability to fit new functions while trainig. \n- The authors argue this is a key factor that hinders learning and is most prominent in sparse reward environments, propose 2 different measures of capacity loss and conduct empirical analysis to show this phenomenon exist in some Atari tasks.  \n- The authors present a number of empirical analysis on this problem, and propose a simple regularization scheme to mitigate this issue, and presented some analysis on the effect of this scheme, giving interesting insights. \n- The proposed method improve performance especially on Montezuma's revenge when no specialized exploration techniques are used, supporting the paper's claims", "summary_of_the_review": "Overall I think the paper makes some interesting and novel contributions, the problem of capacity loss is rarely studied in the RL setting, and research in this direction might help us better understand the difficulty in DRL training. Though I currently do have some concerns. I look forward to reading the authors' rebuttal. I will consider increase my score if my concerns are fully addressed. \n\n========================================================\n\npost-rebuttal\n\nI now increase my score to 6 since I feel a lot of the concerns are indeed addressed in the response. However the authors should go through the paper carefully and try to make things more clear in the paper, would be nice to incorporate some of these reviewer discussions into the paper or appendix. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Ohl6Cgx2j78", "reviewer": "Reviewer_zrcc", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Very nice paper that is well corroborated and is clearly presented in logical order\n- Solid experiments, good job!\n- Great analysis in Section 4.2\n\nWeaknesses:\n- Needs to better logically connect 3.1 and 3.2.  Why is effective dimension representative of/correlated to capacity? A good mathematical proof would work, ideally in Appendix B.\n- INFER seems very similar to RND exploration, except that the L2 error is moved from intrinsic reward to the loss function. Ablate on this! (such as Rainbow + RND)\n- Novelty of problem. This problem has also been shown in supervised learning, where neural networks become less adaptable over time, and I believe in several other works in RL, such as in Kumar's prior work. The paper, especially in introduction, present it like it is a new problem. Good to downplay this.\n- Figure 3 and Figure 4 make the same message (correlation between effective dimension and performance). Recommend combining both figures and move some repeated information (such as 4(b)) to the Appendix.\n-In Appendix, show how robust INFER is against hyperparameter tuning. It looks like beta=100 is a pretty arbitrary value.\n-Paper seems to be rooted in Atari environments. What prevents extending this to continuous control environments such as Mujoco?\n\nQuestions:\n- In Figure 2 (MNIST experiments) (clarification), is the label y_i = 1[y<i] a mask applied over the random network output?", "summary_of_the_paper": "Summary:\n- Over course of training, deep RL agents experience \"capacity loss\", where networks are unable to quickly fit new functions. This problem is further exacerbated by non-stationary predictions (such as bootstrapping) over the course of training.\n- To prove this problem, the author runs two experiments (Atari and MNIST), and show that training error against a randomly-initialized network increases over time. They also define effective dimension (rank of feature space) and show that effective dimension is tightly correlated with performance.\n- The authors introduce INFER, which regularizes Q-value loss with error w.r. to randomly initialized network, and evaluate this on Atatri-57, showing most benefits in sparse reward environments (Montezuma's revenge)", "summary_of_the_review": "Overall:\nThis is a well written paper which gives strong evidence that capacity loss/representational collapse is indeed a root problem for deep RL. As long as the authors address the weaknesses, I am willing to bump this up from weak accept to strong accept.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper analyzes the learning behavior of deep networks inside RL algorithms, and proposes an interesting hypothesis: that many of the observed difficulties in deep RL methods stem from _capacity loss_ of the trained network (that is, the network loses the ability to adapt quickly to fit new functions). As the paper points out, some of these difficulties have popularly been attributed to other causes (such as difficulties in exploration) or to less-specific causes (such as reward sparsity: the paper proposes that capacity loss mediates observed problems due to sparsity). 

The paper investigates its hypothesis two ways: first by attempting to measure how capacity varies over time during training of existing deep RL methods, and second by proposing a new regularizer to attempt to preserve capacity. These experiments are set up well, and their results are convincing &mdash; while there is likely no perfect way to measure or preserve capacity, the methods chosen here make sense. 

This is a strong paper: it proposes a creative, appealing, and interesting hypothesis about an important problem (difficulties in training deep RL methods), and conducts a well-designed evaluation. We expect and hope that it will inspire interesting follow-on work.

We thank the authors for their thorough and helpful participation in the discussion period, including updates to improve the clarity of the paper.
Decision: Accept (Spotlight)