An empirical study of implicit regularization in deep offline RL
Authors: Caglar Gulcehre
Srivatsan Srinivasan
Jakub Sygnowski
Georg Ostrovski
Mehrdad Farajtabar
Matthew Hoffman
Razvan Pascanu
Arnaud Doucet
Venue: Accepted by TMLR ()

Abstract: Deep neural networks are the most commonly used function approximators in offline reinforcement learning. Prior works have shown that neural nets trained with TD-learning and gradient descent can exhibit implicit regularization that can be characterized by under-parameterization of these networks. Specifically, the rank of the penultimate feature layer, also called effective rank, has been observed to drastically collapse during the training. In turn, this collapse has been argued to reduce the model's ability to further adapt in later stages of learning, leading to the diminished final performance. Such an association between the effective rank and performance makes effective rank compelling for offline RL, primarily for offline policy evaluation. In this work, we conduct a careful empirical study on the relation between effective rank and performance on three offline RL datasets : bsuite, Atari, and DeepMind lab. We observe that a direct association exists only in restricted settings and disappears in the more extensive hyperparameter sweeps. Also, we empirically identify three phases of learning that explain the impact of implicit regularization on the learning dynamics and found that bootstrapping alone is insufficient to explain the collapse of the effective rank. Further, we show that several other factors could confound the relationship between effective rank and performance and conclude that studying this association under simplistic assumptions could be highly misleading.


Published in Transactions on Machine Learning Research (12/2022)
An empirical study of implicit regularization in deep offline RL
Caglar Gulcehre, Srivatsan Srinivasan", Jakub Sygnowski, Georg Ostrovski,
Mchrdad Farajtabar, Matt Hoffman, Razvan Pascanu, Arnaud Doucct DeepMind
Reviewed on OpenReview: Attps://
//:::///pppe openreview. net/ forum? id= HFf
Abstract
Deep neural networks are the most commonly used function approximators offline re- inforcement learning. Prior works have shown that neural nets trained with TD-learning and gradient descent exhibit implicit regularization that be characterized by under- parameterization of these networks. Specifically, the rank of the penultimate feature layer, also called effective rank, has been observed to drastically collapse during the training. In turn, this collapse has been argued to reduce the model's ability to further adapt in later stages of learning, leading to the diminished final performance. Such an association between the effective rank and performance makes effective rank compelling for offline RL, primarily for offline policy evaluation. In this work, we conduct careful empirical study on the re- lation between effective rank and performance and DeepMind lab. We observe that a direct association exists only in restricted settings and disappears in the more extensive hyperparameter sweeps. Also, we empirically identify three phases of learning that explain the impact of implicit regularization on the learning dynamics and found that bootstrapping alone is insufficient to explain the collapse the effective rank. Further, we show that several other factors could confound the relationship between effective rank and performance and conclude that studying this association under simplistic assumptions could be highly misleading.
three offline RL datasets bsuite, Atari,
1 Introduction
The use of deep networks function approximators in reinforcement learning (RL), referred to as Deep Reinforcement Learning (DRL), has become the dominant, paradigm in solving complex tasks. Until recently, most DRL literature focused on online-RL paradigm, where agents must interact with the environment to explore and learn. This led to remarkable results on Atari (Miih el all, 201)), Go (Silver el all, 2017),
StarCraft II (Vinyals et all 2019), Dota 2 (Berner et all, 201)), and robotics (Andrychowicz et all 2020). Unfortunately, the need to interact with the environment makes these algorithms unsuitable and unsafe for many real-world applications, where any action taken can have serious ethical or harmful consequences be costly. In contrast, in the oflline RI paradigm eFu all, 2020; Fujimoto et. alj, 201;; Guleehre el all, 2020: Levine et all, 2021), also known as batch RL (Eemstt et all 2005 Lange et all, 2012), agents learn from a fixed datasct previously logged by other (possibly unknown) agents. This ability makes offline RL more applicable the real world.
Recently, Kumar all (20211) showed that offline RL methods coupled with TD learning losses could suffer from an effective rank collapse of the penultimate layer's activations, which renders the network to become umder-paraneterivel. They further demonstrated significant fraction of Atari games, where the collapse of effective rank collapse corresponded to performance degradation. Subsequently, Kumar et all (202
*Indicates joint, first authors.

---

Published in Transactions on Machine Learning Research (12/2022)
explained the rank collapse phenomenon by analyzing TD-learning loss with bootstrapped targets in the kernel and linear regression setups. In these simplified scenarios, bootstrapping leads to self-dist illation, causing severe pudeer-prrametrizaiio and poor performance, as also observed and analyzed by Mohahi et al (2020). Nevertheless, Huh of all (2021) studied the rank of the representations in a supervised learning setting (image classification Lasks) and argued that low rank leads to better performance. Thus the low-rank representations could act as an implicit regularizer.
BeamRder, 05 Pearson
DamonAtteck Kendall
p-0.21
DoublDunk, Kendel 30,
100
150
75
100
50-
50
-50
IceHockeey Kondall r.0.00, Pearson p-0.08
MsPacman, Kondill r:0.36, Pearson p-0.15
Kendall Pearson p.0.0
Effeciive Rank (end of traiinn)
260
160
Hyperperametor Sweap
150
Narrow
100
150
Variants
BCO
100
60
50
DON SAM
DQN large minibetch
sweep
ReadRummer Kendall T Pearson p.00.0
Rocotank, Kendall Pearson gr0.02
Zaxaron Kendali T 05.08. Pearson p:
150
150
60
20000
40030
20
40
2000 4000
6000
8000
Episode Returns (end of training)
Figure 1: [Atari] The rank and the performance Correlation between effective rank and agent's performance towards the end of training in different Atari games. We report the regression lines for the narrow sweep, which covers only single offline RI. algorithm, with a small minibatch size (32) and a learning rate sweep similar to the hyperparameter sweep defined in RL Unplugged (Gmlechre of all 2020), whercas in the broad sctting, we included more data from different models and a larger hyperparameter sweep. In the narrow setup, there a positive relationship bet ween the effective rank and the agent's 's performance, but that relationship disappears the broad data setup and almost reverses.
broad narrow hyperparameter sweep:
Typically, in machine learning, we rely on empirical evidence to extrapolate the rules or behaviors of our learned system from the experimental data. Often those extrapolations are done based on a limited number of experiments due to constraints on computation and time. Unfortunately, while extremely useful, these extrapolations might not always generalize well across all settings. While Kumar el. all (2u121a) do not concretely propose causal link between the rank and performance of the system, extrapolate the results (agents performing poorly when their rank collapsed) to the existence of such causal link, which herein we refer to as rank collapse hypothesis. In this work, we do careful largo-scale empirical analysis of this potential causal link using the ofllin RL setting (also used by Kumar el all (2020a) and also the Tandem RL (Ostrovski et all, anzi) setting. The existence of this causal link would be beneficial for offline RI, as controlling the rank of the model could improve the performance (sce the regularization term explored in Kumar et all (2020a)) or as we investigate here, the effective rank could be used for model selection in settings where offline evaluation proves to be elusive.
might be tempted to
2

---

Published in Transactions on Machine Learning Research (12/2022)
Key Observation 1: The effective rank and the performance are correlated in restricted settings, but this correlation disappears when we increase the range of hyperparameters and the types of models that
evaluating (Figure m) a.
"This is because other factors like hyperparameters and architecture can confound the rank of the penultimate layer; unless the those factors controlled carefully, the conclusions drawn from the experiments based on the rank can be misleading.
Instead, we show that different factors affect a network's s rank without affecting its performance. This finding indicates that unless all these factors of variations are controlled many of which we might still be unaware of the rank alone might a misleading indicator of performance.
A deep Q network exhibits three phases during training. We show that rank can be used identify different stages of learning in Q-lcarning if the other factors controlled carcfully. We believe that our study, similar to others (e.g. Dlll Z0T7), re-emphasizes the imporlance of critically judging our understanding of the behavior of neural networks based on simplified mathematical models or empirical evidence from a limited set of experiments.
Key Obscrvation 2: Doep Q-lcarning approaches goes through three phases of learning: i) sim- ple behaviors, ii) complex behaviors, iii) ander--prrameteriattion (Figure 2 These phases be identified by the effective rank and performance
given task 10
"The first two phases of learning happen during the training of all models tested. At times, the third phase of the learning could potentially lead to the agent losing its representation capacity and the ensuing poor performance.
Episode Returns
Effective Rank
Training Steps
Training Sleps
Phasos of Learning
Phose 1: Simple
Phose 2: Complex
Phase 3:
Beheviors
Behaviors
Underparameerizaiinn
Figure 2: Lifespan of learning in deep Q-learning: The plot on the left-hand side illustrates the evolution of the effective rank, and the plot on the right-hand hand side demonstrates the evolution of the performance during training. In the first phase, the model learns casy-to-learn behaviors that simplistic by nature and ignore many factors of environmental variations. The effective rank collapses to minimal value in the first phase since the model does not need large capacity to learn the simple behaviors. In phase 2, the model learns more complex behaviors that we identify as those that obtain large returns when the policy is evaluated in the environment. Typically, in supervised lcarning, phase followed by overfitting. However, in offline RL (specifically the TD-learning approaches that we tried here), we observed that often followed by phase 3.
underfitting/ hwnpiittiinnnnne--enneeee parameterization
We organize the rest of the paper and our contributions in the order we introduce in the paper as follows:
Section presents the related work, and Section 5 describes experimental protocol.
In Sections H. 8 R. / and B we study the extent of impact of different interventions such as architec- tures, loss functions and optimization on the causal link between the rank and agent performance. Some interventions in the model, such introducing auxiliary loss (e.g. CURL (Laskin all,
3

---

Published in Transactions on Machine Learning Research (12/2022)
2020),, SAM (Foret et all 20221) the activation function) can increase the effective rank but does not necessarily improve the performance. This finding indicates that the rank of the penultimate layer is not enough to explain an agent's performance. We also identify the settings where the rank strongly correlates with the performance, such as DQN with ReLU activation and many learning sleps over a fixed dataset.
In Section I we show that a deep Q network goes through three stages of learning, and those stages can be identified by using rank the hyperparameters of the model are controlled carefully.
Section describes the main outcomes of our investigations. Particularly, we analyze the impact of the interventions described earlier and provide counter-examples that help contradiet the runk collapse hypothesis, establishing that the link between rank and performance can be affected by several other confounding factors of variation.
In Section I., we ablate and compare the robustness of BC and DQN models with respect to random perturbation introduced only when evaluating the agent in the environment. We found out that the offline DQN agent is more robust than the behavior cloning agent which has higher effective rank.
Section A presents the summary our findings and its implications for offline RL, along with potential future rescarch directions.
2 Background
2.1
Effective rank and implicit under-regularization
The choice of architecture and optimizer can impose specific implicit biases that prefer certain solutions over others. The study of the impact of these implicit biases on the generalization of the neural networks is often referred to as implicit regularization. There is a plethora of literature studying different sources of implicit rogularization such as initialization of parameters (Glorot and Bengio, 2010 Li and Liang 2018 He of all, 2015), architecture (Liel All 2017 Huang el all. 2020), slochaslicily (Keskar el all 2016 Sagun el al, 2007)), Dherin, 2021). The rank of the feature matrix of neural network source of implicit regularization has been active of study, specifically in the context of supervised learning (Arror et all, 2019 Hnh et all, 2021; Peeniigtoon and Worah, 2017 Sanyal et all, 208 Daneshmand et. all 2020; |Martin and Mahoney 2021).. In this work, we study the phenomenon of implicit. regularization through the effective rank of the last hidden layer of the network, which we formally define below.
and optimázation (Smith ef al, 2021 Barrett
Definition: Effective Rank
Recently, Kumar at (2020a) studied the impact of the effective rank on generalization in the general RL context. With a batch size of N and D units in the feature layer, they proposed the elfective rank formulation presented Equation M for a feature matrix & E NXD where N 2 D that uses a threshold value with the singular values (9) in descending order i.e. (1 2 (2( 2 ... We provide the specific implementation of the effective rank we used throughout this paper in Appendix A.
( E (
k
21 21-6
effective ranks (4) min k
(1)
Terminology and Assumptions. Note that the threshold value 8 throughout this work has been fixed to 0.01 similar to Kumar et all (2020a). Throughout this paper, we use the term effective rank to describe the rank of the last hidden layer's features only, unless stated otherwise. This choice is consistent with prior work (Kumar et all 2020a) as the last layer acts as a representation bottleneck to the output layer.
Kumar (2021a) suggested that the effective rank of the Deep RL models trained with TD-learning objectives collapses because of i) implicit regularization, happening due to repeated iterations over the dataset.
4

---

Published in Transactions on Machine Learning Research (12/2022)
with gradient descent; ti) self-distillation effect emerging due to bootstrapping losses. They supported their hypothesis with both theoretical analysis and empirical results. The theoretical analysis provided in their paper assume simplified setting, with infinitesimally small learning rates, batch gradient descent and linear networks, in line with most theory papers on the topic. We focus our experiments to use the effective rank definition by Kumnr el all (2021)) to make our results more comparable. However, in our preliminary experiments we compared different rank measures and they perfectly correlated to each other on Atari.
2.2 Auxiliary losses
Additionally, we ablate the effect of using auxiliary loss agent's performance and rank. Specifically, we chose the 'Contrastive Unsupervised Representations for Reinforcement Learning' (CURL) (Laskin ct all, 2020) loss that is designed for use in a contrastive self-supervised learning setting within standard RL algorithms:
L(s, a, Lg(s,,,r,0) a. + X LCURL (8, 8,0).
(2)
Here, LQ refers to standard RL losses and the CURL loss (S, 3,6) is the contrastive loss between the features of the current observation and randomly augmented observation & as described in Laskin
et al (212T). Besides this, we also ablate with Sharpuess-Awar Minimization (SAM) (Foret et all, 2021) an approach that seeks parameters that have a uniformly low loss in its neighborhood, which leads to flatter local minima. We chose SAM as means to better understand whether the geometry of loss landscapes help inform the correlation between the effective rank and agent performance in offline RI. algorithms. In our experiments, we focus on analyzing mostly the deep Q-lcarning (DQN) algorithm (Lmih all, 2115) simplify the experiments and facilitate deeper investigation into the rank collapse hypothesis.
2.3 Deep offline RL
Online RL requires interactions with an environment to learn using random exploration. However, online interactions with an environment can be unsafe and unethical in the real-world (Qulac-Arnold el all, 20MM) Offline RL methods do not suffer from this problem because they can leverage offline data to learn policies that enable the application of RL in the real world (Menick et all, 2022; Sh all, 2021: Konyushkova et all, (02I). Here, we focused on offline RL due to its importance in real-world applications, and the previous works showed that the implicit regularization effect is more pronounced in the offline RL (Kumar all 2020at 20218).
Some of the early examples of offline RL algorithms are least-squares temporal difference methods (Baadike and Barto, [MM Lagoudakis and Parr, 201)) and fitted Q-itcration (Ernst ef all, 2005 Ricdmillan 20.15
Recently, value-based approaches to offline RL have been quite popular.
Value-based approaches typically lower the value estimates for unseen state-action pairs, either through regularization (IKumar et all, 20207) or uncertainty (Agarwal et all 2027). One could also include R- BVE (Gulcehre et all, 2021) in this category, although it regularizes the Q function only the rewarding transitions to prevent learning suboptimal policies. Similar to R-BVE, Mathien all all (2I2T) have also shown that the methods using single-step of policy improvement work well on tasks with very large action space and low state-action coverage. In this paper, due to their simplicity and popularity, we mainly study action value-based methods: offline DQN (Agarwal et all 2020) and offline R2D2 (Gullehree el all, 202T). Morcover, we also sparingly use Batched Constrained Deep Q-learning (BCQ) algorithm (Fujimoto all, 2013), another popular oflline RL algorithm that uses the behavior policy to constrain the actions taken by the target network.
Most offline RL approaches we explained here rely on pessimistic value estimates (Jin ct all 2021; Xio aL 2021; Guicehre el. all, 2021; Kumar el all, 20208). Mainly because oflline RL datasets lack exhaustive exploration, and extrapolating the values to the states and actions not seen in the training set can result in extrapolation errors which can be catastrophic with TD-lcarning (Fujimoto et all, 2018: Kumar of all, 2019). On the other hand, in online RL, it is common practice to have inductive biases to keep optimistic value functions to encourage exploration (Machado el all 20
5

---

Published in Transactions on Machine Learning Research (12/2022)
We also experiment with the tandem RL setting proposed by Ostrovski et all (2122T) which employs two independently initialized online (active) and offline (passive) networks in training loop where only the online agent explores and drives the data generation process. Both agents perform identical learning updates on the identical sequence of training batches in the same order. Tandem RL is a form of offine RL. Still, unlike the traditional offiie RL setting on fixed datasels; in the landem RI, the behavior policy can change over time, which can make the learning non-stationary. We are interested in this setting because the agent does not necessarily reuse the same data repeatedly, which was pointed in Lvle (212T) a potential cause for the rank collapse.
3 Experimental setup
To test and verify different aspects of the rank collapse hypothesis and its potential impact on the agent performance,
large number of experiments bsuite (Osband ct all, 201)), Atari (Bellemard all 203) and DeepMind lab (Beathie (UTG) environments. In all these experiments, we use the experimental protocol, datasets and hyperparameters from Gulcehre et all (2020) unless stated otherwise. We provide the details of architectures and their default hyperparameters in Appendix A.
bsuite We run ablation experiments on bsuite a fully offline setting with the same offline dataset. as the one used in Guicehre all (202T).. We use a DQN agent (as a representative TD Learning algorithm) with multi-layer feed-forward networks to represent the value function. bsuite provides with small playground environment which lets test certain hypotheses which are computationally prohibilive in other domains (for e.g.: computing Hessians) with respect to terminal features and an agent's generalization performance.
To test whether some of our observations are also true with higher-dimensional input features such as images, we run experiments on the Atari dataset. Once again, we use an offline DQN agent representative TD-learning algorithm with a convolutional network as function approximator. On Atari, conducted large-scale experiments on different configurations:
Atari
(a) Small-scale experiments:
DDNN-25-2MM Offline DQN on Atari with minibatch size 256 trained for 2M gradient steps with four different learning rates: [3e - 5, le 4, 4, 5e ---], 4]. We ran these experiments observe if our observations hold in the default training scenario identified in RL Unplugged (Gulcehre el. all,
202)).
DQN-32-100M: Offline DQN on Atari with minibatch size of 32 trained for 100M gradient steps with three different learning rates: x 10-5, x 10 10=4]. We ran those experiments to explore the effects of reducing the minibatch size.
(b) Large-scale experiments:
Long run learning rate sweep (DQN-256-20MB) Offline DQN trained for 20M gradients steps with 12 different learning rates evenly spaced in log-space between 10-2 and 10-5 trained on minibatches of size 256. The purpose of these experiments is to explore the effect of wide range of learning rates and longer training on the effective rank.
Long run interventions (DQN-interventions)) Offline DQN trained for 20M gradient steps on minibatches of size 256 with 128 different hyperparameter interventions on activation functions, dataset size, auxiliary losses etc. The purpose of these experiments to understand the impact of such interventions on the effective rank in the course of long training.
DeepMind Lab While bsuite and Atari present relatively simple fully observable tasks that require no memory, DeepMind lab tasks (Gulechre et all, (021) are more complex, partially observable tasks where is very difficult lo obtain good coverage in the datasel even after collecting billions of transitions. We specifically conduct our observational studies on the effective ranks on the DeepMind Lab dataset, which has data collected from a well-trained agent on the AockAvoid level. The dataset is collected by adding different levels of action exploration noise to well-trained agent in order get datasets with a larger coverage of the state-action space.

---

Published in Transactions on Machine Learning Research (12/2022)
B: Effective rank h: Observed confounders h: Performance
A: Hidden confounders
M: Model selection method
M
h
A
2
B
Figure 3: Structural causal model (SCM) of different factors that test: M represents the model selection method that we use to determine the h which denotes the observed confounders that chosen at the beginning of the training, including the task, the model architecture (including depth and number of units), learning rate and the number of gradient steps to train. B is the effective rank of the penultimate layer. À is the agent's performance, mcasured as episodic returns the agent attains after evaluating in the environment. A represents the unobserved confounders that change during training but may affect the performance, such as the number of dead units, the parameter norms, and the other underlying factors that influence learning dynamics. We test the effect of each factor by interventions.
We illustrate the structural causal model (SCM) of the interactions between different factors that we would like to test in Figure F. To explore the relationship between the rank and the performance, we intervene on h, which represents potential exogenous sources of implicit regularization, such architecture, dataset. size, and the loss function, including the auxiliary losscs. The interventions on will result in a randomized controlled trial (RCT.) A represents the unobserved factors that might affect performance denoted by À the effective rank 3 such activation norms and number of dead units. It is easy to justify the relationship between M, h, A and B. We argue that B is also confounded by A and h. We show the confounding effect of A on 3 with our interventions to beta via auxiliary losses or architectural changes that increase the rank but do not affect the performance. We aim to understand the nature of the relationship between these terms and whether SCM in the figure describes what we notice in our empirical exploration.
We overload the term performance of the agent to refer to episodic returns attained by the agent when it is evaluated online in the environment. In stochastic environments and datasets with limited coverage, offline RL algorithm's online evaluation performance and generalization abilities would generally correlate (Gulcchre et all 2020; Kumar et all, 2121c). The offline RL agents will need to generalize when they are evaluated in the environment due to,
Stochasticity in the initial conditions and transitions of the environment. For example, in the Atari environment, the stochasticity arises from sticky actions and on DeepMind lab, it arises from the ran- domization of the initial positions the lemons and apples.
Limited coverage: The coverage of the datasct is often limited. Thus, an agent is very likely to encounter states and actions that it has never seen in the training dataset.
4 Effective rank and performance
Based on the results of Kumar el all (20208), one might be templed to extrapolate a positive causal link between the effective rank the last hidden layer and the agent's performance measured as episodic returns attained when evaluated the environment. We explore this potentially interesting relationship on a larger scale by adopting a proof by contradiction approach. We evaluated the agents with the hyperparameter setup defined for the Atari datasets in RI. Unplugged (Gulcehre all 2220) and the hyperparameter sweep

---

Published in Transactions on Machine Learning Research (12/2022)
defined for DeepMind Lab (Grilehrre all 202T). For narrow set of hyperparameters this correlation exists observed in Figure M However, in both cases, we notice that a broad hyperparameter sweep makes the correlation between performance and rank disappear (see Figures D and DeepMind lab Figure in Appendix A In particular, we find hyperparameter scttings that lead to low (collapsed) ranks with high performance (on par with the best performance reported in the restricted hyperparameter range) and settings that lead to high ranks but poor performance. This shows that the correlation between effective rank and performance can not be trusted for offline policy selection. In the following sections, we further present specific ablations that help understand the dependence of the effective rank vs. performance correlation specific hyperparameter interventions.
4.1 Lifespan of learning with deep Q-networks
Empirically, we found effective rank sufficient to identify three phases when training offiine DQN agent with a ReLU activation function (Figure B). Although the effective rank may be sufficient to identify those stages, it still does not imply direct causal link between the effective rank and the performance as discussed in the following sections. Several other factors can confound effective rank, making it less reliable as a guiding metric for offline RL unless those confounders are carcfully controlled.
IcaHodkey
MsPacman
8
60
Effecllive Rank
20-
Learning
Phase Simple
Behaviors
01
Phase Complex Behaviors
-2.5
2500
Phase 3:
Epleode Retur
-5
2000
Underparametrization
-7.5
1500
-10
500
-12.5
5000
10000
15000
20060
5000
10000
15000
20000
Leamer Steps (x1000)
The three phases of learning: On IceHockey and MsPacman RL Unplugged Atari games illustrate the different phases of learning with the offline DQN agent using the learning rate of 0.0004. The blue region in the plots identifies the Phase 1, green region is the Phase 2, and red region is the Phase 3 of the learning. IceHockey is one of the games where the expert that generated the dataset performs quite poorly on it, thus the majority of the data is just random exploration data. It easy to see that in this phase, the Offline DQN performs very poorly and never manages to get out of the Phase The performance of the agent on IceHockey is poor and the effective rank of the network is low throughout the training. On MsPacman, we can observe all the three phases. The model transition into Phase 2 from Phase 1 quickly, and then followed by the under-fitting regime where the effective rank collapses the agent performs poorly.
Figure 4:
We identified three phases of learning according which stage they appear during training, the performance (returns obtained by the algorithm) when evaluated in the environment, and the effective rank:
Phase 1 (Simple behaviors): These behaviors emerge early in training and give rise to low rewards when evaluated in the environment (often close to a random policy.) The effective rank of the model first collapses to small value, sometimes to a singlo-digit value, and then gradually increascs. Let us note the term simple behaviors, here, refers to how easy/sing /ssype to learn behavior for an RL algorithm. We hypothesized that this could be due to the implicit. bias of the SGD to learn functions of increasing
8

---

Published in Transactions on Machine Learning Research (12/2022)
complexity over training iterations (Kalimeris et all, 2019); therefore, early in training, the network relies on simple behaviours that are myopic to most of the variation in the data. Hence the model would have a low rank. The rank collapse early at the beginming of the training happens very abruptly; just in a handful of gradient updates, the effective rank collapses to a single-digit number. However, this early rank collapse does not degrade the agent's 's performance. We call this rank collapse early in training as self-praning effect.
Phase 2 (Complex behaviors): These behaviors emerge later in training and achieve high rewards, often close to the best policy in the dataset when evaluated in the environment. In this phase, the effective rank of the model first increases and then usually flattens. The model starts to learn more complex behaviors that would achieve high returns when evaluated in the environment. We call these /dificcllty to learn these behaviors since they
behaviors as complex behaviors, because of the emerge in later during training and sensitive to the hyperparameters.
This is the last phase of the algorithm; in this phase, the effective rank collapses to a small value (often to 1), and the agent's performance collapses too. The third phase is called underfitting since the agent's performance usually drops, and the effective rank also collapses, which causes the agent to lose part of its capacity. This phase is not always observed (or the performance does not collapse with effective ran towards the end of the training) in all settings as demonstrate in our different ablations. Typically in supervised learning, Phase 2 is followed by over-fitting, but with offline TD-learning, could not find any evidence of over-fitting. We believe this phase is primarily due to the target Q-network needing to extrapolate over the actions not seen during the training and causing extrapolation errors as described by (Kumnar et 211 A piece of evidence to support this hypothesis is presented in Figure 24 in Appendix A.S. which suggests that the effective rank and the value error of the agent correlate well. In this phase, the low effective rank and poor performance are caused by a large number of dead RcLU units. Shin and Karniadmas (212T) also show that as the network has an increasing number of dead units, it becomes uuder-parametern and this could influence the agent's ' performance negatively.
Phase 3 (Underfitting/
DOindrrttingg/nndepraaeettenttte Undeeparaneterization))
It is possible to identify those three phases in many of the learning curves we provide in this paper, and our first two phases agree with the works on SGD's implicit bias of learning functions of increasing complexity (Kalimeris all 2019). Given a fixed model and architecture, whether it is possible to observe all these three phases during training fundamentally depends on:
Hyperparameters: The phases that an offline RL algorithm would go through during training depends on hyperparameters such as learning rate and the early stopping or training budget. For example, due carly stopping the model may just stop in the second phase, if the learning rate is loo small, since the parameters will move much slower the model may never get out of Phase 1. If the model is not large enough, it may never learn to transition from Phase into Phase 2.
2. Data distribution: The data distribution has very big influence on the phases the agent goes thorough, for example, if the dataset only contains random exploratory data and ORL method may fail to learn complex behaviors from that data and as a result, will never transition into Phase 2 from Phase 1.
3. Learning Paradigm: The learning algorithm, including optimizers and the loss function, influence the phases the agent would go through during the training. For example, we observed that, Phase 3 only happens with the offline TD-learning approaches.
It is possible to avoid phase three (underfitting) by finding the correct hyperparameters. We believe the third phase we observe might be duc to non-stationarity in RL losses (Igl ct all, 2020) due to bootstrapping or errors propagating through bootstrapped largets (Kumar el all, 2021) The underfitting regime only appears if the network is trained long enough. The quality and the complexity of the data that an agent learns from also plays key role in deciding which learning phases are observed during training. In Figure H. we demonstrate the different phases of learning on IceHockey and MsPacman games. On IceHockey, since the expert that generated the dataset has a poor performance on that game, the offline DQN is stuck in
9

---

Published in Transactions on Machine Learning Research (12/2022)
Phase 1, and did not managed to learn complex behaviors that would push it to Phase 2 but on MsPacMan, all the three phases are present. We provide learning curves for all the online policy selection Atari games across twelve learning rates in Appendix A.S.
Damontmack
looHockey
RoadRunner
58
comnt
dome
comer
Donta
ecness
comor
Canter
autee
comoe
comoo
Effectro (ond trainin)
Dossar
coour
exera
cocur
crou
creser
exeeca
15
conto
cxerts
Learning Rates
Trahing 1000000
Training Sleps: 20000000
Figure 5: [Atari]: Bar charts of effective ranks with respect to the learning rates after 1M and 20M learning steps. After 1M gradient steps, the ranks are distributed almost like a bell-shaped curve, indicating that high and low learning rates have low ranks (phases 1 and 3) while the learning rates in the middle are in phase and hence have the higher rank. After 20M learning steps, the mode of the distribution of the ranks skews towards the left, indicating low terminal ranks for many large learning rates. Namely, as we train the network longer, the terminal rank decreases, particularly for the large learning rates. The rank is low for the low learning rates because the model is stuck in Phase whereas the large learning rates get into Phase 3 quickly and thus the low terminal ranks.
Figure 5 shows the relationship between the effective rank and twelve learning rates. In this figure, the effect learning rale on the different phases learning is distinnuisea.le. For low learning rates, the ranks are low because the agent can never transition from Phase 1 Phase 2, and for large learning rates, the effective ranks low because the agent is in Phase 3. Therefore, the distribution of effective ranks and learning rales has a Gaussian-like shape, as depicted in the figure. The distribution of rank shifts towards low learning rates as we train the agents longer because slower models with low learning rates start entering Phase 2, and the models trained with large learning rates enter Phase 3.
4.2
The effect of dataset size
We can use the size of the dataset as a possible proxy metric for the coverage that the agent observes in the offline data. We uniformly sampled different proportions (from 5% of the transitions to the entire dataset)
10

---

Published in Transactions on Machine Learning Research (12/2022)
from the transitions in the RL Unplugged Atari benchmark dataset (Guicehree et all, 2120) to understand how the agent behaves with different amounts of training data and whether this is factor affecting the rank of the network.
Figure F shows the evolution of rank and returns over the course of training. The effective rank and perfor- mance collapse severely with low data proportions, such as when learning only on 5% of the entire dataset. subsampled. Those networks can never transition from phase to phase However, as the proportion of the dataset subsampled increases, the agents could learn more complex behaviors to get into phase 2. The effective rank collapses less severely for the larger proportions of the dataset, and the agents tend to perform considcrably better. In particular, we can see that in phase 1, an initial decrease of the rank correlates with an increase in performance, which we can speculate is due to the network reducing its reliance on spurious parts of the observations, leading to representations that generalize better across states. It worth noting that the ordering of the policies obtained by using the agents' performance does not correspond to the order- ing of the policies with respect the effective rank throughout training. For example, offline DQN trained the full dataset performs better than 50% of the dataset, while the agent trained using 50% of the data sometimes has higher rank. A similar observation be made for Zaxxon, at the end of the training, the network trained on the full dataset underperforms compared to the one trained on 50% of the data, even if the rank is the same or higher.
DsamRider
DemonArttack
Zarozan
Eftective Rank
100
120
90
00
60
40
30-
0
60
100
160
200
0
50
100
150
200
0
60
100
150
200
BeamnRider
Demck
Zaxzon
10000
Retums
4000
5000
1000
2000
2500
500
60
100
200
163
200
-50
Learner Slepetx 10000)
Fraction data -0.05 -.1.0.2-005--1
Figure 6: [Atari] Datasct size: Evolution of ranks and returns as we vary the fraction of data available for the agent to train on. We see that the agent which sees very little data collapses both terms of rank and performance. The agent which sees more of the data has good performance even while allowing some shrinkage of rank during the training.
5 Interactions between rank and performance
To understand the interactions and effects of different factors the rank and performance of offline DQN, we did several experiments and tested the effect of different hyperparameters on effective rank and the agent's performance. Figure E shows the causal graph of the effective rank, its confounders, and the agent's performance. Ideally, we would like to intervene in each node on this graph to measure its effect. As the rank a continuous random variable, it is not possible to directly intervene on effective rank. Instead, we emulate interventions on the effective rank by conditioning threshold /3 with r. In the control case (A(1)), we assume and for X(0), we would have T..
We can write the average treatment effect (ATE) (Holand (HSD for the effect of setting the effective rank to large value on the performance as:
ATE B, [[(A((1)) - B((((()))
(3)
11

---

Published in Transactions on Machine Learning Research (12/2022)
Let us note that this ATE(X, 7,) quantity doesn't ' necessarily measure the causal effect of 3 on X, since we know that the X can be confounded by A (hidden confounders.)
ReLU
ii) tanh
130
2000
iii) both
Figure 7: [Atari]: The correlation plot between the effective rank and the performance (measured in terms of episode returns by evaluating the agent in the environment) offline DQN on Asterix and Gravitar games over 256 different hyper-parameter configurations trained for 2M learning steps. There is a strong correlation with the ReLU function, but the correlation disappears for the network with tanh activation function. There is no significant correlation between effective rank and the performance on the Asterix game with the complete data. Still, a positive correlation exists on the Gravitar game. These results are not affected by the Simpson's 's paradox since the subgroups of the data when split into groups concerning activation functions, do not show a consistent correlation trend.
We study the impact of factors such as activation function, learning rate, data split, target update steps, and CURL and SAM losses on the Asterix and Gravitar levels. We chose these two games since Asterix is an easy-to-cxplore Atari game with relatively dense rewards, while Gravitar reward setting (BBelemaaee elall, 201G). In Table we present the results of intervening to 3 thresholded with different quantiles of the effective rank. Choosing network with high effective rank for ReLU network has statistically significant positive effect the agent's performance concerning different quantiles both Asterix and Gravitar. The agent's 's performance is measured in terms of normalized scores as described in Gulcehre el. all (2020). However, the results with the tanh activation function are mixed, and the effect of changing the rank does not have a statistically significant impact on the performance in most cases. In Figure a we show the correlations between the rank and performance the agent. The network with ReLU activation strongly correlates rank and performance, whereas tanh does not. The experimental data can be prone to Simpson's parador which implies that trend (corvelation in this case) may appear in several
hard-to-oxplore and sparse
12

---

Published in Transactions on Machine Learning Research (12/2022)
subgroups of the data, but it disappears or reverses when the groups are aggregated (Simpson, 1957)). This can lead to misleading conclusions. We divided our data into equal-sized subgroups for hyperparameters and model variants, but we could not find a consistent trend that disappeared when combined, as in Figure n Thus, our experiments do not appear to be affected by Simpson's 's paradox.
Table 1: Atari: The average treatment effect of having a network with a higher rank than concerning different quantiles. We report the average treatment effect (ATE), its uncertainty (using 95% confidence intervals using standard errors), and p-values for Asterix and Gravitar games. Higher effective rank scems to have smaller elfect on Asterix than Gravitar. Let us note that, Gravitar is sparse reward problem, and Asterix is a rense- one. Overall the effect is more prominent for ReLU than tanh, and with tanh networks, it is not statistically significant. We boldfaced the ATES where the effect is statistically significant (p < 0.05.) The lype column of the table indicates the activation functions used in the experiments we did the intervention. The "Combined" type corresponds to a combination of tanh and ReLU experiments.
quantile=0.25
quantile=0.5
quantile=0.75
quantile=0.95
Level
Type
ATE
p-value
ATE
p-value
ATE
p-value
ATE
p-value
Combined
0.019
0.010
0.001
0.007 = 0.013
0.207
0.005 1 0.019
0.324
+0.031 1 0.011
0.999
Asterix
ReLU
0.079
0.018
0.000
0.089 1 0.025
0.000
0.112 + 0.040
0.000
0.110 = 0.085
0.017
Tanh
-0.014 0,001
0.999
0.009
= 0.005
0.996
-0.005 = 0.006
0.886
0.020
0.017
0.023
Combined
0.954
0.077
0.000
0.569
0.098
0.000)
0.496 + 0.141
0.000
0.412
0.206
0.001
Gravitar
ReLUU
1.654
0.150
0.000
1.146
0.163
0.000
1.105 + 0.256
0.000
1.688 + 0.564
0.000
Tanh
0.028 1: 0.090
0.303
0.185
1 0.118
0.005
0.242 +: 0.167
0.009
0.054 4: 0.265
0.367
Table 2: [Atari] Average Treatment Effect of different interventions on Asterix and Gravitar: Quantity of interest Y is the terminal rank of the features. Activation function and learning rate influence the terminal feature ranks the most in our setup. We indicate them with boldface; changing the activation from ReLC to tanh improves the effective rank, whereas reducing the learning rate reduces the rank.
u Activation Function
Control (u)
Treatment T(u)
Yu()) Yo(u)
ReLU
tanh
66.97 + 7.13
Learning Rate
3 x 10
3 x 10-5
-54.15 = 7.54
Target Update Steps
2500
200
-15.23 + 8.21
SAM Loss Weight
0
0.05
-10.05 + 8.26
Level
Asterix
Gravitar
-9.41 + 8.25
CURL Loss Weight
0
0.001
8.09 - 8.27
Data Split
100%
5%
5.11 + 8.27
In this analysis, we set our control setting to be trained with ReLU activations, the learning rate of 3e, without any auxiliary losses, with training done the full dataset in the level Asterix. We present the Average Treatment Effect (ATE) (the difference in ranks between the inter- vention being present and absent in an experiment) of changing each of these variables in Table 2 We find that changing the activation function from ReLU to tanh dras- tically affects the effective ranks of the features, which is in syne with our carlier observations on other levels. In Figure E we present heatmap of ATEs where demon- strate the effective rank change when two variables are changed from original control simultaneously. Once again, the activation functions and learning rate significantly af- foet the terminal ranks. We also obscrve some interesting combinations that lead the model to converge to lower rank- for example, using SAM loss with dropout.
:25
SAM Mugrabi00)
71.5
22.3
100
waigh13.001)
Dispoct Hulvi
60
Turget Updeke Steak20o
702
.30
Leaning Dla
Dopost (3
hope yooas 24002 200
Lend Coveraa
600 wwgga00000
Figure 8: [Atari] ablations: Effects of different pairs of interventions on the control set. The control is on Asterix without dropout, CURL and SAM on the full dataset and with a target update period of 2500 steps.
13

---

750000 Published in Transactions on Machine Learning Research (12/2022)
These observations further reinforce our belief that different factors affect the phenomenon. The extent of rank collapse and mitigating rank collapse alone may never fully fix the agent's learning ability in different. environments.
6
The effect of activation functions
The severe rank collapse of phase 3 is apparent in our Atari models, which have simple convolutional neural networks with ReLU activation functions. When we study the same phenomenon on the SeckAvoid datasct, rank collapse does not seem to happen similarly. It is important to note here that to solve those tasks effectively, the agent needs memory; hence, all networks have a recurrent core and LSTM. Since standard LSTMs use t.mh(.) activations, investigating in this setting would help us understand the role of the choice of architecture on the behavior of the model's rank.
Figure III shows that the output features of the LSTM network the DeepMind lab dataset do not experi- ence any detrimental effective rank collapse with different exploration noise when use tanh(-) activation function for the cell. However, if we replace the tanh(-) activation function of the LSTM cell with ReLU or if we replace the LSTM with feed-forward MLP using ReLU activations (as seen in Figure ID]) the effective rank both cases, collapses to a small value at the end of training. This behavior shows that the choice of activation function has a considerable effect on whether the model's rank collapses Uhroughout training and, subsequently, its ability to learn expressive value functions and policies in the environment it is susceptible to enter phase of training.
Observation: Agents that have networks with ReL.U units tend to have dead units which causes the effective rank collapse phase of learning while other activations like tanh do not suffer similar collapsc.
The activation functions influence both the network's learning dynamics and performance. As noted by Pennington and Worah (2017), the activation function can influence the rank of each layer initialization. Figure g presents findings on bsuite levels. In general, the effective rank of the penultimate layer with ReLU activations collapses faster, while ELU and lamh(-) tend to maintain relatively higher rank than ReLU. As the effective rank goes down for the catch environment, the activations become sparser, and the units dic. We illustrate the sparsity of the activations with Gram matrices over the features the last hidden layer of a werdforward network trained on bsuite Figure 37 in Appendix A.E
cartpole
catch
mountain car
Feature Karnel Ranks
Activation Function
-ELU
-ReLU
-Sigmoid
201
-Tanh
10
290000 5
630000
750030
1000300
250000 500000
1000000
290000
500000
750000
10000C0
Learnr Steps
Figure 9: [bsuite] Effective ranks of different activation functions: The magnitude of drop in effective rank is more severe for ReLU and sigmoid activation functions than tanh.
7 Optimization
The influence of minibatch size and the learning rate on the learning dynamics is a well-studied phenomenon. Smith and Le (2017); Smith all (2124) argued that the ratio between the minibatch size and learning rate relates to implicit regularization of SGD and also affects the learning dynamics. Thus, it is evident that.
14

---

Published in Transactions on Machine Learning Research (12/2022)
Feedforward
USTM (ReLU)
LSTN (tanh))
Eltedliv Rankk
100-
7.5
150
75-
100
50
2.5
0
25
50
100
125
0
25
50
75
100
125
25
50
75
100
126
Foodforward
LISTM (RoLU)
LSTM (tanh)
3
Epleode Relume
2
10-
25
50
75
100
125
25
50
75
100
125
0
50 75
100
125
Learner Steps(x 1000)
Exploration =0.0 1.000..1-0.1-000
Figure 10: [DeepMind lab SeekAvoid] Activation functions: Evolution of ranks in a typical LSTM network with tanh activations at the cell of LSTM on DeepMind lab. We see that a typical LSTM network does not get into Phase 3. However, when we change the activation function of the cell gate from tanh ReLU the effective rank collapses to very small values. The effective rank collapses when the LSTM is replaced with feedforward network too in cases of ReLU activation.
those factors would impact the effective rank and performance. Here, we focus on a setup where we see the correlation ween effective rank and performance and investigate how it emerges.
Our analysis and ablations Table and Figure & illustrate that the learning rate one of the most promi- nent factors on the performance of the agent. general, there no strong and consistent correlation between the effective rank and the performance across different models and hyperparameter settings. However, in Figure D. we showed that the correlation exists in a specific setting with particular minibatch size and learning rate. Further, in Figure A we narrowed down that the correlation between the effective rank and the performance exists for the offline DQN with ReLU activation functions. Thus in this section, we focus on a regime where this correlation cxists with the offline DQN with ReLU and investigate how the learning rale and minibatches affect the rank. We ran several experiments to explore the relationship wet the minibatch size, learning rates, and rank. In this section, we report results on Atari in few different settings -ttari-DQN-256-MMM Atari-DQQ--32-00MM Atari-DQN-256-20X and Atari-BC-256-2M.
The dying ReLU problem is a well-studied issue in supervised learning (Glorot et all, 20TT; Gulcelre el all, 2016) where due to the high learning rates or unstable learning dynamics, the ReLU units can get stuck in the zero regime of the ReLU activation function. We compute the number of dead ReLU units of the penultimate layer of a network as the number units with zero activation value for all inputs. Increasing the learning rate increases the number of dead units as can be seen in Figures [ and I2 Even in BC, we observed that using large learning rates can cause dead ReLU units, rank collapse, and poor performance, as can be seen in Figure IK and hence this behavior is not unique to offline RL losses. Nevertheless, models with TD-learning losses have catastrophic rank collapses and many dead units with lower learning rates than BC. Let us note that the effective rank depends on the number of units (D) and the number of dead units (n) at a layer. It is easy to see that effective rank is upper-bounded by D-n- r,. In Figure I., we observe a strong correlation between the number of dead ReLU units of the penultimate layer of the ReLU network and its effective rank.
The pace of learning influences the number of dead units and the elfective rank: the larger the learning rates and the smaller minibatches are, the number of dead units increases, and the effective rank decreases. However, the performance only poor when the effective rank severely low.
Observation:
15

---

Published in Transactions on Machine Learning Research (12/2022)
BeamRi der
Demonklteck
ReadRRunee
Zasson
90
100
100-
100
20
Eftocivv
60
10
01
2300
5000
7500
10000
0
2500
5000
7500
10000
2000
5000
7000
10000
2500
5000
7500
10000
RoadRummer
2000
50000
40000
1500
30000
Episede
0000
20000
2000
10000
2500
5000
7500
10000
2900
9000
7600
10000
0
2500
5000
7500
10000
2600
6000
7503
10000
ReadRunner
5000
500
400
800
300
300
200
200
100
200
100
2300
5000
7500
10000
0
2600
5000
7500
10000
0
2500
5000
7500
10000
2500
5000
7500
10000
Learner Slepe//x 10000;
Leaminn Rata -3e 05-0.0001-0.0000
Figure 11: [Atari DQN-32-100M] Learning curves of offline DQN for 100M gradient steps of training with minibatch size of 32. Increasing the learning rate increases the number of dead units in the ReLU network. As result, the increased learning rate also causes more severe collapse well, which aligns very well with the number of dead units. We observed that this behavior can also happen with networks using saturating activation functions such as sigmoid.
Finally, we look into how effective ranks shape up towards the end of training. We test 12 different learning rates, trying to understand the interaction between the learning rales and the effective rank of the repre- sentations. We summarize our main results in Figure E where training offline DQN decreases the effective rank. The effective rank is low for both small and large lcarning rates. For higher learning rates, we have seen earlier, training for longer leads to many dead ReLU units, which in turn causes the effective rank to diminish, as seen in Figures ID and IZ Moreover, as seen in those figures, in Phase 1, the effective rank and the number of dead units are low, Thus, the rank collapse in Phase is not caused by the number of dead units. In Phase 3, the number of dead units is high, but the effective rank is drastically low. The drastically low effective rank caused by the network's large number of dead units in Phase 3. In Phase 3, we believe that both the under-pan aaaterapaatttrizattonn and the poor performance is caused by the number of dead units in ReLU DQN, which was shown to be the case by (Shin and Karniadaks 2021 in supervised learning.
Overall, we could only observe high correlation between the effective rank and the agent's performance, when we use ReLU activations in the network after long training. We also present more analysis on how controlling the loss landscape affects the rank vs performance in Appendix A.2 and A4
8
The effect of loss function
8.1 Q-learning and behavior cloning
Behavior cloning (BC) a method to learn the behavior policy from an offline dataset using supervised learning approaches (Pomerlcan (MS)). Policies learned by BC will learn to mimic the bohavior policy, and thus the performance of the learned BC agent is highly limited by the quality of the data the agent is trained on. We compare BC and Q-learning in Figure [R We confirm that with default hyperparameters, the BC
16

---

Published in Transactions on Machine Learning Research (12/2022)
Bearsiiiwer
DemrrsAterk
ReaeRunner
Zaceon
125
100
100
75
Effeciiva
60
60
20
0
100
200
300
100
200
300
0
100
200
300
0
100
200
300
12500
50000
6000
10000
40000
1500
7500
30000
5000
20000
1000
2000
2500
10000
0
100
200
300
100
200
800
0
100
200
300
100
200
800
Beamilider
DemonAttack
RoadRunner
Zarozon
500
500
500
400
400
400
1000
300
300
300
200
200
100
Slepe(x 10000)
Learning Rate -0.0001 -0.0000000000000000 0.000 -0.003
Figure 12: [Atari DQN-256-3M Learning curves of offline DQN for 3M gradient steps of training with minibatch size of 256. Increasing the minibatch size improves the performance of the network with larger learning rales.
RoadRunner
150
200
200
200-
100
Eflective
100
100
100
0
100
200
100
150
200
200
DeamRiddr
DemoonAtack
ReadRurner
Zasvon
10000
4000
1000
20000
2000
10000
100
10000)
Leaming
Figure 13: [Atari BC-256-2M] Learning Curves of BC After 2M gradient steps with mini-batch size of 256. For large enough learning rates the rank of BC agent also collapses. We hypothesize that the rank collapse is side effect of learning in general and not only due to TD-learning based losses.
agent's effective rank does not collapse at the end of training. In contrast, as shown by Kinar at (uizIa) DQN's effective rank collapses. DQN outperforms BC even rank is considerably lower, and during learning, the rank is not predictive of the performance (Figure lt] This behavior indicates the unreliability of effective rank for offline model selection.
17

---

Published in Transactions on Machine Learning Research (12/2022)
BearrRider Spearan som: 4.98
Derrnnttack Sceeman oon: 0087
DeobinCunk, Spearmn oom 4.96
120
100
102
20
Efockvv oainnn))
03
75
+3
+2.00
Readffurmen Spearmnn con 4.00
Rictolank, Spearman coT: -100
Zacoon Socerman car: 0.57
50
402 000
400
450
300 680
Dead Unit (end of training)
Figure 14: [Atari] Scatter plot for the correlation between the number of dead units and effective rank at the end of training and we observe that the effective rank strongly correlates with the number of dead units. Also, using larger learning rate increases the number dead units in the network.
ReaeRiiierr
DemresAteck
ReseRuncer
7wceenn
150
200
200
100
200
Eftectivv
100
100
01
50
100
150
200
50
100
150
200
0
50
160
150
200
0
90
100
150
200
1600
4000
1400
4000
1200
4000
20000
2000
2000
800
10000
830
50
100
150
200
100
150
200
90
100
150
200
100
150
200
Leamer Steas/x 10000)
Model-BC-DDN
Figure 15: [Atari] Offline DQN and Behavior Cloning (BC): We compare BC and DQN agents on the Atari datasct. We used the same architecture, dataset, and training protocols for both basclines. We used the hyperparameters defined in (Gulcehre el. al, (0200) for comparisons. The rank of the DQN agent is significantly lower and achieves higher returns than BC.
8.2 CURL: The effect of self-supervision
We study whether adding an auxiliary loss term proposed for CURL (Lasiin el all, 2020) (Equation z during training helps the model mitigate rank collapse. In all our Atari experiments, we use the CURL loss described in (Laskin of all 2020) without any modifications. Since the DeepMind lab tasks require memory, we apply a similar CURL loss to the features aggregated with a mean of the states over all timesteps. In all experiments, also sweep over the weight of the CURL loss X.
18

---

Published in Transactions on Machine Learning Research (12/2022)
BeamRRider
DemonAtteck
Zaxoon
150
100
100
Effcciiv
50
201
50
0
50
100
150
0
50
100
150
0
50
100
ReoimRader
DemonnAtackk
10009
1500
4000
Epicod
5000
1000
2500
2300
100
Learner Slepe(x 10000)
weight-0-
Figure 16: [Atari] Auxiliary losses: Evolution of ranks as we increase the weight of auxiliary losses. We see that a strong weight for auxiliary loss helps mitigate the rank collapse but prevents the model from learning useful represcntations.
Figure II shows the ranks and returns on Atari (for DeepMind lab results see Appendix A In Atari games, using very large weights for an auxiliary loss (> 1) prevents rank collapse, but they simultancously deteriorate the agent performance. We speculate, borrowing on intuitions from the supervised learning literature on the role of the rank as implicit. regularizer Huh el all (202I)), that in such scenarios large rank prevents the network from ignoring spurious parts of the observations, which affects its ability to generalize. On the other hand, moderate weights of CURL auxiliary loss do not significantly change the rank and performance of the agent. Previously Agarwal el al. (202T) showed that the CURL loss does not improve the performance of an RL agent Atari in statistically meaningful way.
On DeepMind lab games, we do not observe any rank collapse. In none of our DecpMind lab experiments agents enter into Phase after Phase and This due to the use of the tanh activation function for LSTM based on our investigation of the role of the activation functions in Section
9 Tandem RL
Kumar at (222 propose one possible hypothesis for the observed rank collapse as the re-use of the same transition samples multiple times, particularly prevalent in the offline RL sctting. A sctting in which this hypothesis can be tested directly is the 'Tandem RL' proposed in Ostrovski et all (212T)): here secondary ("passive") agent is trained from the data stream generated by architecturally equivalent, independently initialized baseline agent, which itself is trained in a regular, online RL fashion. The passive agent tends to under-perform the active agent, despite identical architecture, learning algorithm, and data stream. This setup presents a clean ablation setting which both agents use data in the same way (in particular, not. differing their re-use of data samples), and so any difference performance or rank of their represcntation cannot be directly attributable to the reuse of data.
In Figure M. summarize the results of a Tandem-DQN using Adam (Kingma and Ba, 2014 and RMSProp (Tilleman et all, 2012) optimizers. Despite the passive agent reusing data in similar fashion the online agent, we observe that it collapses to a lower rank. Besides, the passive agent's performance tends to significantly (in most cases, catastrophically: worse than the online agent that could not satisfactorily explained just by the extent of difference in their effective ranks alone. We think that the Q-learning algorithm seems be efficient enough to exploit the data generated by the active agent learn complex behaviors that would put the passive agent into Phase 2. We also noticed that, although Adam achieves
19

---

Published in Transactions on Machine Learning Research (12/2022)
broahout
space inaders
163
Effezive Rank
200
150
Optimier
100
100
-adam
100
100
60
60
80
50
passaive
50
100
150
200
30
100
100
200
50
100
150
200
50
900
150
200
Environment frames 1M)
20
40000
10000
Returns
3000
300000
7500
Optimizer -adam -maprop
200
20000
5000
Learner -astive passire
100
20
160
150
220
Environment Bremnss 1M)
Figure 17: Atari, tandem RL: We investigate the effect of the choice of optimizers between Adam and RMSProp rank and the performance of the models, both for the active (solid lines) and passive agents (dashed lines). We observe the rank of the passive agent is lower than the active agent both for Adam and RMSProp.
Foatue Komel (pacovc)
breskoou
SCMON inNaOUr
20c 1004
200
200
Algoritmm -BVE Carie
100
100
--leemming
00
60
01
Framees 1M)
Ineekoot
pang
SEMOO Inveeters
Epicodo Roaum (pasahe)
400
20
20000
1500
10.
15000
Alguritm Monte-Cari -Qileaming
200
10000
1000
10
-20
150 2000
100
1.50
2000
200
100 150
200
202
Frames (x. 1M)
Figure 18: Atari, Forked tandem RL: We evaluate different loss functions in the forked tandem setting, where the passive agent is forked from the online agent, and the online agent's parameters are frozen. Still, the parameters of the online agent are updated. We forked the passive agent after it had seen 50M frames during training which is denoted with dashed lines in the figure. We observe that using both BVE and MC return losses improves the agent's rank, but the agent's 's performance still poor.
better performance than RMSProp, the rank of the model trained with the Adam optimizer tends to be lower than the models trained with RMSProp.
In Figure IS we investigate the effect of different learning algorithms on the rank and the performance of an agent in the forked tandem RL setting. In the forked tandem setting, agent is firstly trained for fraction of its total training time. Then, the agent is 'orkedd' into active and passive agents, and they both start with the same network weights where the active agent is 'frozen' and not trained but continues to generate data from its policy to train the passive agent on this generated data for the remaining of the training time. Here, we see that the rank flat-lines when we fork the Q-learning agent, but the performance collapses dramatically, In contrast, despite the rank of BVE and an agent trained with on-policy Monte Carlo returns going up, the performance still drops. Nevertheless, the decline of performance for BVE and the agent trained with Monte Carlo returns is not as bad the Q-learning agent most Atari games.
20

---

Published in Transactions on Machine Learning Research (12/2022)
10 Offline policy selection
In Section I1, we found that with the large learning rale sweep selting rank and number dead units have high correlation with the performance. Offline policy selection (Paine et all, 2020) aims to choose the best policy without any online interactions purely from offline data. The apparent correlation between the rank and performance raises the natural question of how well rank performs as an offline policy selection approach. We did this analysis on DQN-256-20M experiments where we previously observed strong correlation wetween the rank and performance. We ran the DQN network described in DQN-256-20M, experimental setting until the end of the training and performed offine policy selection using the effective rank to select the best learning rate based on the learning rale that yields to maximum effective rank or minimum number of dead units.
Eflecive rank
Number of dead units
1.00
0.52
0/G11
0,500
Simple
0.504
3:534
0.00
0-423
464
1.000
Zano
Zanon
Pogje
Figure 19: Atari DQN-256-2002 20M:: This plot is depicting the simple regret of DQN agent with ReLU activations using effective rank and the number of dead units. On the left, we show the the simple regret to select the best. learning rate using the effective rank and on the right hand side, we show the simple regret. achieved based on the number of dead units.
Figure IN illustrates the simple regret on each Atari offline policy selection game. The simple regret be- tween the recommended policy measures how close an agent is to the best policy, and it is computed as described in Konyushkova et al. (202T)). . A simple re- gret of would mean that our offline policy selection mechanism successfully chooses the best policy, and would mean that it would choose the worst policy, After 2M training steps, the simple regret with the number of dead units as policy selection method is poor. In contrast, the simple regret achieved by selecting the agent with the highest effective rank is good. The mean simple regret achieved by using the number of dead units as an offline policy selec- tion (OPS) method 0.45 + 0.11, where the uncer- tainty +0.11 is computed as standard error across five seeds. In contrast, simple regret achieved by ing effective rank as an OPS method is 0.24 + 0.12. The ellective ranks for most. learning rales collapse as we train longer since more models enter Phase of lcarningthe number of dead units increases. Af- ter 20M of learning steps, the mean simple regret computed using effective rank the OPS method
74.074
-00
40
-00
Doman
Boadf Roadfurm
Zo000
BunnRo d
Poogan
Figure 20: Atari DQN-256-2002 Here, are de- picting the percentage of improvement by doing offline policy selection using rank individually vs online policy selection using median reward across nine Atari games to select the best learning rate. Using effective rank as the offline policy selection method performs relatively well when compared to doing online policy selection based median normalized score across Atari games.
21

---

Published in Transactions on Machine Learning Research (12/2022)
becomes 0.40 + 0.07, and the mean simple regret is 0.25 = 0.12 with the number of dead units for the OPS. Let us note that the 2M learning step is more typical in training agents on the RL Unplugged Atari dataset. The mumber of dead units becomes a good metric to do OPS when the network is trained for long (20M steps in our experiment), where the rank becomes drastically small for most learning rate configurations. However, effective rank seems to be a good metric for the OPS earlier in training. Nevertheless, without prior information about the task and agent, is challenging to conclude whether the number of dead units or effective rank would be an appropriate OPS metric. Other factors such the number of training steps, activation functions, and other hyperparameters confound the effective rank and number of dead units. Thus, we believe those two metries we tested are unreliable in general OPS settings without controlling those other extraneous factors.
In Figure 2L. we compare effective rank as an OPS method with respect to its percentage improvement over the online policy selection to maximize the median reward achieved by the each agent across nine Atari games. The effective rank on Zaxxon, BeamRider, MsPacman, Pooyan, Robotank, DoubleDunk perform competitively to the online policy selection. Effective rank may be complementary tool for networks with ReLU activation function and we believe it can be useful metric to monitor during the training in addition to number of dead units to have a better picture about the performance of the agent.
11 Robustness to input perturbations
Several works, such as Sanyal ct al (201)) suggested that low-rank models can be more robust to input per- turbations (specifically adversarial ones). It is difficult to just measure the effect of low-rank representations on the agent's performance since rank itself is not a robust measure, as it depends on different factors such as activation function and learning which in turn can effect the generalization of the algorithm independently from rank. However, it is casier to validate the antithesis, namely "Do more robust agents need to have higher effective rank than less robust agents?". We can easily test this hypothesis by comparing DQN and BC agents.
Robustness metric: We define the robustness metric p(p) based on three variables: i) the noise level p, ïi) the noise distribution d(p), and iii) the score obtained by agent when evaluated in the environment with noise level p: score[d(p)] for which score]d(O)] represents the average score achieved by the agent without. any perturbation applied on it. Then
define p(p) as follows:
plp) = score[d(0)] - score[d(p)] score [d(p)] score(d(O))
(4)
score [d()]]
In our experiments, we compare DQN and BC agents since we already know that BC has much larger ranks across all Atari games than DQN. We trained these agents on datasets without any data augmentation. The data augmentations are only applied on the inputs when the agent evaluated in the environment. We evaluate our agents on BeamRider, Icetlockey, MsPacman, DemonAttack, Robotank and RoadRunner games from RL Unplugged Atari online policy selection games. We excluded DoubleDunk, Zaxxon and Pooyan games since BC agent's performance on those levels poor and very close to the performance of random agent, so the robustness metrics on those games would not be very meaningful. On IceHockey, the results can be negative, thus we shifted the scores by 20 to make sure that they are non-negative. In our robustness experiments, we used the hyperparameters that were used in Gulrehre et all (202T) on Atari for both for DQN and BC.
Robustness to random shifts: In Figure Al we investigate the Robustness of DQN and BC agents with respect to random translation image perturbations applied to the observations described by Yarats et al (2020). We evaluate the agent on the Atari environment with varying degrecs of input scaling. We noticed that BC's performance deteriorates abruptly, whereas the performance of DQN, while also decreasing, is better than the BC one.
Robustness to random scaling: In Figure 2 we explore the Robustness of DQN and BC agents with respect to random scaling as image perturbation method applied to the observations that are feed into our
22

---

Published in Transactions on Machine Learning Research (12/2022)
0.0
0.8-
0.0
0.5
Robusiness score
0.2
0.2
0.0
502 0.4
Rideeark
Dranficer
1.0
1.0
1.0
a.
0.5
0.8
0.0
0.6
0.4
0.4.
02
0.2
0.0
0.2 0.06
0.0
0.00
The random ShIF
0:10
503
Figure 21: [Atari] Robustness to random shifts during evaluation: We measure the robustness of BC and DQN to random shilts on six Alari games from RL. Unplugged. The DQN and BC agents are trained offline without any perturbations on the inputs, only on the RL Unplugged datasets. However, during evaluation time perturbed the input images with 'random shift" data augmentation. Overall, DQN is more robust than BC to the eraluation-time random-scaling image perturbations that not during training. The difference is more pronounced on IceHockey and BeamRider games. DQN achieved mean AUC (over different games) of 2.042 and BC got 1.26.
locHocky
1.0
0.8
0.8
0.4
0.4
0.2
0.0
0.0
0.00
6.09
0.04
0.06
0.00
0.16
0.00
0.07
0.04
0.08
0.06
0.10
6.00
0.00
0.04
0.06
018
876
Robetark:
BeamFiser
ReadRunne
1.0
1.0
1.0
0.8
0.6
0.4
0.4
0.2
0.0
random
Figure 22: [Atari] Robustness to random scaling during evaluation: We measure the robustness of BC and DQN random scaling over inputs on six Atari games from RL Unplugged. The DQN and BC agents are trained offline without data augmentations over the RL Unplugged datascts. However, perturbed the observations during the evaluation with 'random scale" data augmentation. We observed that DQN& is more robust than BC to the evaluation-time random-scaling data augmentation. The difference is more pronounced in IceHockey, BeamRider, and Robotank games. DQN achieved mean AUC (over different games) of 1.60, and BC achioved 0.87.
23

---

Published in Transactions on Machine Learning Research (12/2022)
deep Q-network. We randomly scale the image inputs described in IChen ct al (2017). When evaluating the agent in an online enviromment, we test it with varying levels of image scaling. The results are again very similar to the random shift experiments where the DQN agent is more robust to random perturbations in the online environment.
According the empirical results obtained from those two experiments, is possible to see that DQN is more robust to the evaluation-time input perturbations, specifically random shifts and scaling than the BC agent. Thus, more robust representations do not necessarily require a higher effective rank.
12 Discussion
In this work, we empirically investigated the previously hypothesized connection between low effective rank and poor performance. We found that the relationship between effective rank and performance is not simple previously conjectured. We discovered that offline RL agent trained with Q-learning during training goes through three phases. The effective rank collapses to severely low values in the first phase we call this as the self-pruning phase- and the agent starts to learn basic behaviors from the dataset. Then in the second phase, the effective rank starts going up, and in the third phase, the effective rank collapses again. Several factors such as learning rate, activation functions and the number of training steps, influence the occurrence, persistence and the extent of severity of the three phases of learning.
In general, a low rank is not always indicative of poor performance. Besides strong empirical evidence, we propose hypothesis trying to explain the underlying phenomenon: not all features are useful for the task the neural network is trying to solve, and low rank might correlate with more robust internal representations that can lead to better generalization. Unfortunately, reasoning about what it means for the rank to be too low is hard in general, as the rank is agnostic to which direction of variations in the data are being ignored or to higher-order terms that hint towards a more compact representation of the data with fewer dimensions.
Our results indicate that an agent's effective rank and performance correlate in restricted settings, such as ReLU activation functions, Q-learning, and fixed architecture. However, as showed in our experiments, this correlation is primarily spurious in other settings since it disappears with simple modifications such as changing the activation function and the learning rate. We found several ways to improve the effective rank of the agent without improving the performance, such as using tanh instead of ReLU, an auxiliary loss (e.g., CURL), and the optimizer. These methods address the rank collapse but not the underlying learning deficiency that causes the collapse and the poor performance. Nevertheless, our results show that the dynamies of the rank and agent performance through learning are still poorly understood; we neod understand the relationship between those two factors. We also observed
more theoretical investigation in Tandem and offline RL settings that the rank collapses to minimal value early in training. Then there is unexplained variance between agents in the later stages of lcarning. Overall, the cause and rolc of this early rank collapse remain unknown, and we believe understanding its potential elfects is essential in understanding large-scale agents' practical learning dynamics. The existence of low-rank but high-performing policies suggest that our networks can be over-parameterizer for the tasks and parsimonious representations emerge naturally with TD-learning- Discarding the dead ReLU units might achieve more efficient inference. We believe this finding can give inspiration to new family of pruming algorithms.
bootstrapping losses and ReLU networks in the offline RL setting.
Acknowledgemments: We would like to thank Clare Lyle, Will Dabney, Aviral Kumar, Rishabh Agarwal, Tom le Paine, Mark Rowland and Yutian Chen for the discussions. We want to thank Mark Rowland, Rishabh Agarwal and Aviral Kumar for the feedback on the early draft version of the paper. We thank Sergio Gomez and Bjorn Winckler for their help with the infrastructure and the code-base at the inception of this project. We would like to thank the developers of Acme (Hoffman al.all, 2020). Jax (Bradbury of all, 201S) and the Deepmind JAX ecosystem (Babnschkin et all, 2020) for developing the software infrastructure that enabled our experiments.
24

---

Published in Transactions on Machine Learning Research (12/2022)
References
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective offline rein- forcement learning. In International Conference on Machine Learning, pages 104-114, PMLR, 2020.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 14:29304-29320 2021.
OpenAl: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pa- chocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulatioo.. The International Journal of Robotics Research, 39(1):3 20, 2020.
Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. Advances in Newral Information Processing Systems, 32:7413-77 14444, 2019.
Igor Babuschkin, Kate Baumli, Alison Bell, Surya Blupatiraju, Jake Bruce, Peter Buchlovsky, David Budden, Trevor Cai, Aidan Clark, Ivo Danihelka, Claudio Fantacci, Jonathan Godwin, Chris Jones, Ross Hemsley, Tom Hennigan, Matteo Hessel, Shaobo Hou, Steven Kapturowski, Thomas Keck, lurii Kemaev, Michael King, Markus Kunesch, Lena Martens, Hamza Merzic, Vladimir Mikulik, Tamara Norman, John Quan, George Papamakarios, Roman Ring, Francisco Ruiz, Alvaro Sanchez, Rosalia Schneider, Eren Sezener, Stephen Spencer, Srivatsan Srinivasan, Luyu Wang, Wojciech Stokowiec, and Fabio Viola. The DeepMind JAX Ecosystem, 2020. URL http: httpg//ttt....odeeeeeeeeeeeeeeee cow demm/mmd/mindjjx
David Barrett and Benoit Dherin. Implicit gradient regularization. In International Conference on Learning Representations, 2020.
Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Victor Valdés, Amir Sadik, et al. Deepmind lab. arXiv preprint arxiv:6612 038011 2016.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in Neural Information Processing Systems, 29, 2016.
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279. 2013.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. DotA 2 with large scale deep reinforcement learning. Xix preprinl arXiv: 291 06680, 2019.
James Bradbury, Roy Frostig, Peter Hawkins, Malthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Manderman-Milne, and Qiao Zhang, JAX: compos- able transformations of Python-NumPPy programs, 2018. URL http://github com/ homm/gogle/jja
Steven Bradtke and Andrew Barto. Linear least-squares algorithms for temporal difference learning. Machine Learning, 22:33-57, 03 1996.
Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arxiv: 106.05577,, 2017.
Hadi Daneshmand, Jonas Kohler, Francis Bach, Thomas Hofmann, and Aurelien Lucchi. Batch normalization provably avoids ranks collapse for randomly initialised deep networks. Advonces in Newral Information Processing Systems, 2020.
Yann Dauphin, Razvan Pascanu, Caglar Gulcchre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. arXiv preprint MxXi::146.200 2014.
25

---

Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. JMLR Workshop and Published in Transactions on Machine Learning Research (12/2022)
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In International Conference on Machine Learning, volume 70, pages 1019- 1028. PMLR, 2017.
Gabriel Dulac-Aroodd, Daniel Mankowitz, and Todd Hester. Challenges of real-world reinforcement learning. arXiv preprint 1rx: 1004.12911, 2019.
Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6:503-556, 2005.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2103. 09575, 2021.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Dalasets for deep data-
driven reinforcement learning. arXiv preprint arXiv:2044.07219, 2020.
Scott Fujimoto, Herke Van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. arXiv preprint Xr02 092..0977777 77, 2018
Scott Fujimoto, Edoardo Conti, Mohammad Ghavamzadch, and Joelle Pincau. Benchmarking batch deep reinforcement learning algorithms. arXiv preprint arXiv:1910.00 01700 2019.
Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via Hessian eigenvalue density. Xiv preprint arXiv:1901.1016: 2019.
International Conference on Artificial Intelligence and Statistics, pages
Conference Proceedings, 2010.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In International Conference on Artificial Intelligence and Statistics, pages 315 323. JMLR Workshop and Conference Pro- ceedings, 2011.
Caglar Gulcehre, Marcin Moczulski, Misha Denil, and Yoshua Bengio. Noisy activation functions. Tn International Conference on Machine Learning, pages 3059 3068. PMLR, 2016.
Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Tom Le Paine, Sergio Gómez Colmenarejo, Konrad Zolna, Rishabh Agarwal, Josh Merel, Daniel Mankowitz, Cosmin Paduraru, et al. RL unplugged: Benchmarks for offline reinforcement learning. Xiv preprinl arXiv:2006. 13888, 2020.
Caglar Gulcehre, Sergio Gómez Colmenarejo, Ziyu Wang, Jakub Sygnowski, Thomas Paine, Konrad Zolna, Yutian Chen, Matthew Hoffman, Razvan Pascanu, and Nando de Freitas. Regularized behavior value estimation. Xixiv preprint arXio:2103.090 09.007 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human- level performance on imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision, pages 1026-1034, 2015.
Matt Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Feryal Behbahani, Tamara Nor- man, Abbas Abdolmaleki, Albin Cassirer, Fan Yang, Kate Baumli, Sarah Henderson, Alex Novikov, Ser- gio Gómez Colmenarejo, Scrkan Cabi, Caglar Gulcchre, Tom Le Painc, Andrew Cowic, Ziyu Wang, Bilal Piot, and Nando de Freitas. Acme: A research framework for distributed reinforcement learning. arXiv preprint arXiv:2006. 00979, 2020.
Paul W Holland. Statistics and causal inference. Journal of the American, Statistical Association, 81(36)) 945-960, 1986.
Kaixuan Huang, Yuqing Wang, Molei Tao, and Tuo Zhao. Why do deep residual networks generalize better than deep feedforward networks?-g a neural tangent kernel perspective. arXiv preprint 2722:2002.00222 2020.
26

---

Published in Transactions on Machine Learning Research (12/2022)
Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung, Pulkit Agrawal, and Phillip Isola. The low-rank simplicity bias in deep networks. arXiv preprint arXiv:2103. 10427, 2021.
Maximilian igl, Gregory Farquhar, Jelena Luketina, Wendelin Boehmer, and Shimon Whiteson. Transient non-stationarity and generalisation in deep reinforcement learning. arXiv preprint arXiv:2006.068 2020.
Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline RL? In International Conference on Machine Learning, pages 5084-5096. PMLR, 2021.
Dimitris Kalimeris, Gal Kaplun, Preetum Nakkiran, Benjamin Edelman, Tristan Yang, Boaz Barak, and Haofeng Zhang. Sgd on neural networks learns functions of increasing complexity. Advances in neural information processing systems, 32, 2019.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXie preprint arXiv:1609.0m 04836, 2016.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Represenlations, 2014.
Ksenia Konyushkova, Yutian Chen, Thomas Paine, Caglar Gulcehre, Cosmin Paduraru, Daniel J Mankowitz, Misha Denil, and Nando de Freitas. Active offline policy selection. Advances in Neural Information Processing Systems, 34, 2021.
Aviral Kumar, Justin Fu, Matthow Soh, George Tucker, and Sergey Levine. Stabilizing off-policy Q-lcarning via bootstrapping error reduction. In Conference on Newral Information Processing Systems, pages 11761 11771, 2019.
Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit under-paranntteriation inhibits data-efficient deep reinforcement learning, arXiv preprint arXiv:2010. srXiv:2010.14498, 2020a.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-ccannig for offline reinforce- ment learning. Xixiv preprinl arXiv:2006.044 04779, 2020b.
Aviral Kumar, Rishabh Agarwal, Aaron Courville, Tengyu Ma, George Tucker, and Sergey Levine. Value- based deep reinforcement learning requires explicit regularization. In RL for Real Life Workshop & Over- porumeterization: Pitfalls and Opportunaties Workshop, ICML, 2021a. URL https://drive google..
File////E444Bbacppp-ssssisaapp aDVEzAFWV.m66/vzz aalpaaaaaa.aa.aaa.aaaaaaaaaaaaad
Aviral Kumar, Rishabh Agarwal, Tengyu Ma, Aaron Courville, George Tucker, and Sergey Levine. Dr3:
Value-basod deep reinforcement learning requires explicit regularization. arXio preprint arXiv:2112.447166 2021b.
Aviral Kumar, Anikait Singh, Stephen Tian, Chelsea Finn, and Sergey Levine. A workflow for offline model- free robotic reinforcement learning. arXiv preprint xrXi::2109. 10819, 2021c.
Michail G. Lagoudakis and Ronald Parr. Least-squares policy iteration. Journal of Machine Learning
Research, 4:1107-1149, 2003.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Marco Wiering and Martijn van Otterlo, editors, Reinforcement Learning: Mlale-of-the-Are pages 45-73. Springer Berlin Heidelberg, 2012.
Michael Laskin, Aravind Srinivas, and Picter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning, In International Conference on Machine Leurning, pages 5639 5650. PMLR, 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. Xix preprint arXi::0005.01643, 2020.
27

---

Published in Transactions on Machine Learning Research (12/2022)
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. arXiv preprint or orXi::7712.0000 2017.
Yuanzhi Li and Yingyu Liang. Learning overparameterized neural networks via stochastic gradient descent on structured data. arXiv preprint arXiv: 1804.0124. 2018.
Clare Lyle, Mark Rowland, Georg Ostrovski, and Will Dabney, On the effect of auxiliary lasks on represen- tation dynamics. In International Conference on Artificial Intelligence and Statistics, pages 1-9. PMLR, 2021.
Marlos C Machado, Sriram Srinivasan, and Michael Bowling. Domain-independent optimistic initialization for reinforcement learning. In Workshops at the Twenly- Ninth AAAI Conference on Artificial Intelligence, 2015.
Charles H Martin and Michael W Mahoney. Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning. Journal of Machine Learning Research, 22(165): 1-73, 2021.
Michael Mathieu, Sherjil Ozair, Srivatsan Srinivasan, Caglar Gulcehre, Shangtong Zhang, Ray Jiang, Tom Le Painc, Konrad Zolna, Richard Powell, Julian Schrittwicsor, et al. Starcraft ii unplugged: Large scale oflline reinforcement learning. In Deep RL Workshop NeurIPS 2021, 2021.
Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susamnah Young, Lucy Gamppell-Gillinghan, Geolfrey Irving, and Nat McAleese. Teaching language models to support answers with verified quotes. arXiv preprint arXiv:2203
2022.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Anlonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):522 533, 2015.
Hossein Mobahi, Mehrdad Farajtabar, and Peter L Bartlett. Self-distillation amplifies regularization in Hilbert space. arXiv preprint arXiv:2002.0571 057115, 2020.
Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepezvari, Satinder Singh, et al. Behaviour suite for reinforcement learning. ar preprint Xi:::1998.. 08568, 2019.
Georg Ostrovski, Pablo Samuel Castro, and Will Dabney. The difficulty of passive learning in deep reinforce- ment learning. Advances in Neural Information Processing Systems, 34, 2021.
Tom Le Paine, Cosmin Paduraru, Andrea Michi, Caglar Gulcehre, Konrad Zolna, Alexander Novikov, Ziyu Wang, and Nando de Freitas. Hyperparameter selection for offline reinforcement learning. arXiv preprint 2rXin:2007.09055 2020.
Jeffrey Pennington and Pratik Worah. Nonlinear random matrix theory for deep learning. Advances in Neural Information Processing Systems, 2017.
Dean A Pomerleau. ALVINN: An autonomous land vehicle in a neural network. In Conference on Neural Information Processing Systems, pages 305-312 305-313, 1989.
Martin Riedmiller. Neural fitted Q iteration first experiences with data efficient neural reinforcement lcarning method. In João Gama, Rui Camacho, Pavel B. Brazdil, Alípio Mário Jorgo, and Luis Torgo, editiors, European Conference on Machine Learning, pages 317-328, 2005.
Levent Sagun, Léon Bottou, and Yann LeCun. Singularity of the hessian in deep learning. arXie preprint arXin:1611.07446. 2016.
Levent Sagun, Utku Evei, V Ugur Guney, Yann Dauphin, and Leon Bottou. Empirical analysis of the hessian of over-parametrized neural networks. aixiv preprint arvi 1706. 0445,, 2017.
28

---

Published in Transactions on Machine Learning Research (12/2022)
Amartya Sanyal, Varun Kanade, Philip HS Torr, and Puneet K Dokania. Robustness via deep low-rank representations. arXiv preprint Xiv: 1800.184....0 804-07090, 2018.
Tianyu Shi, Dong Chen, Kaian Chen, and Zhaojian Li. Offline reinforcement learning for autonomous driving with safety and exploration enhancement. arXiv preprint orXiv:211000700 2021.
Yeonjong Shin and George Em Karniadakis. Trainability of relu networks and data-dependent initialization. Journal of Machine Learning for Modeling and Computing, 1(1), 2020.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioamnis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, and Adrian Bolton. Mastering the game of go without human knowl- edge. Nature, 550(7676):354 359, 2017.
Edward H Simpson.
The interpretation of interaction in contingency tables. Journal of the Royal Statistical Society: Series B (Methodological), 13(2):238-241) 1951.
Samuel L Smith and Quoc V Le. A Bayesian perspective on generalization and stochastic gradient descent. arXiv preprint AXxe:1710.0444 2017.
Samuel L Smith, Benoit Dherin, David GT Barrett, and Soham De. On the origin of implicit regularization in stochastic gradient descent. arXiv preprint arX1v:2101.11166 2021.
Tijmen Tieleman, GeofTrey Hinton, et al. Lecture 6.5-rmsprop: Divide the gradient by a rumning average of its recent magnitude. COURSERA: Newral networks for machine learning, 4(2):26 31, 2012.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double Q-learning. Tn Thirtieth AAAI conference on artificial intelligence, 2016.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czarnecki, Michaël Mathicu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7)22):350 354, 2019.
Tengyang Xie, Ching- An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellman--onnistent pessimism for offline reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.
Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixcls. In International Conference on Learning Representations, 2020.
A Appendix
A.1 The impact of depth
Pennington and Worah (2017 explored the relationship between the rank and the depth of neural network at initialization and found that the rank of each layer's feature matrix decreases proportionally to the index of a layer in a deep network. Here, we test the performance of a feedforward network on the bsuite task trained using Double Q-learning (Wan Hasselt ct all 2016) with 2, 8, and 16 layers to see the effect of the number of layers on the effective rank. All our networks use ReLU activation functions and use He initialization (Ho ot all 20B). Figure eK illustrates that the rank collapses as one progresscs from lower layers to higher layers proportionally at the end of training as well. The network's effective rank (rank of the penultimate layer) drops to a minimal value on all three tasks regardless of the network's 's number of layers. The last layer of network will act a bottleneck; thus, collapse of the effective rank would reduce the expressivity. Nevertheless, a deeper network with the same rank as a shallower one can learn to represent a larger class of functions (be less mnder-parametrizell
The agents exhibit poor performance when the effective rank collapses At that point, all the ReLU units die or become zero irrespective of input. Thus on bsuite, deeper networks -our and eight layered networks- performed worse than two layered MT.P.
29

---

Analyzing the eigenvalue spectrum of the Hessian is a common way to investigate the learning dynamics and the loss surface of deep learning methods (Ghorbani et al, 2ms3 Dauphin al, aue)). Understanding Hessian's loss landscape and eigenvalue spectrum can help design better optimization algorithms. Here, we analyze the eigenvalue spectrum of single hidden layer feedforward network trained on the bsuite Catch dataset from RT. Unplugged (Gulleerre et. all, 2020) to understand the loss-landscape of a network with a low effective rank compared to a model with higher rank at the end of the training. As established in Figure ELU activation function network has a significantly higher effective rank than the ReLU network. By comparing those two networks, we also look into the differences the eigenvalue spectrum of a network with high and low rank. Since the network and the inputs relatively low-dimensional, we computed the full Hessian over the dataset rather than a low-rank approxination. The eigenvalue spectrum of the Hessian with ReLU and the ELU activation functions is shown in Figure Ze] The rank collapse is faster for ReLU than ELU. After 900k gradient updates, the ReLU network concentrates 92% of the eigenvalues of Hessian around zero; this is due to the dead units in ReLU network (Glorot et all, 20II). On the other hand, the ELU network has a few very large eigenvalues after the same number of gradient updates. In Figure 28, we summarize the distribution of the eigenvalues of the Hessian matrices of the ELU and ReLU networks. As result, the Hcssian and the feature matrix of the penultimate layer of the network will be both low-rank. Moreover, in this case, the landscape of the ReLU network might be flatter than the ELU beyond the notion of wide basins (Sagun et all 2 This might mean that the ReLU network finds a simpler solution. Thus, the flatter landscape is confirmed by the simpler function learned and less capacity used at the end of the Published in Transactions on Machine Learning Research (12/2022)
Layar
Layer Not
Layer Not
8 Layer Net Layer
Effectiv Rank
Leyer Net
Effectio Rank
Layer Net 16 Layer Not
16 Layer Not
Rank
Network's Depth
Network's Depth
Network's Depth
LL)
(C)
(R)
Figure 23: [bsuite] The ranks and depths of the networks: The evolution of the ranks across different layers of deep neural networks. The figure on the left (L) is for the ranks of catch across different layers. The figure at the center (C) is for the ranks of mountain car across different layers. The figure on the right
(R) is for cartpole.
A.2 Spectral density of hessian for offline RL
training, which is induced by the lower rank representations.
A.3 DeepMind lab: performance vs the effective ranks
In Figure An, we show the correlation between the effective rank and the performance of R2D2, R2D2 trained with CURL and R2D2 trained with SAM. When we look at each variant separately or as an aggregate, we don't see a strong correlation between the performance and the rank of an agent.
A.4 SAM optimizer: does smoother loss landscape affect the behavior of the rank collapse?
We study the relationship between smoother loss landscapes and rank collapse. We use Sharpness Aware Minimization (SAM) (Eorel el all 2021) loss for palentially creating flatter loss surfaces in order to see if smoother loss landscapes affect the rank and performance dynamics differently. Figures 27 and es show the evolution of feature ranks and generalization performance Atari and DeepMind lab respectively. We do not observe a very clear relation between the extent of smoothing the loss and the feature ranks or generalization performance.
30

---

Published in Transactions on Machine Learning Research (12/2022)
10"
10
Leamer stepa =
sleps
9OCK
Learner Stepe 900k
16"
10
10
10
M MA M
Densky
Densty
10
10
10
10"
10
10
10
-0.1
0.0
0.1
0.2
0.3 Eigenvalue
0.4
0.5
0.6 0.7 10"
0.0
0.5
1.0
1.5
2.0 Eigenvllue
2.5
3.0
3.6
4.0
Figure 24: [bsuite Catch] spectral density of Hessian: The visualization of the spectral density of the full Hessian of network trained with 64 units using ReLU (left) and ELU (right) activation functions. The eigenvalues of the Hessian of the offline DQN with ReLU activation function are concentrated around 0 and most of them are less than 1. The eigenvalues of the ELU network also concentrate around 0 with few large outlier eigenvalues.
% near- evs
positive
% negative
ReLU DQN
ELU DQN
We visualize the percentages of positive, negative, and near-zero eigenvalues of the Hessian for offline DQN with ELU and ReLU activation functions on the catch dataset from bsuite. If the absolute value of an eigenvalue is less than 1e-7, we consider it near-zero eigenvalue. We can that for ELU network, near-zero, positive and negative eigenvalues are almost evenly distributed. However, with ReLU network majority of eigenvalues near-zero (90% of the evs are exactly zero), very few negative (2 %) and some positive eigenvalues (7.1 %).
Figure 25: [bsuite Catch] Hessian eigenvalues (evs) for offline DQN:
A.5 Effective rank and the value error
A curious relationship is between the effective rank and the value error, because potential for the rank collapse or Phase 3 with the TD-lcarning algorithms can be the errors propagating thorough the bootstrapped targets. Figure eu shows the correlation between the value error and the effective ranks. There is a strong anti-correlation between the effective ranks and the performance of the agents except on the levels where the expert agent that generated the dataset performs poorly at the end of the training (c.g. IceHockey.) This makes the hypothesis that the extrapolation error can cause rank collapse (or push the agent to Phase 3) more plausíble.
31

---

Published in Transactions on Machine Learning Research (12/2022)
Eps: C.O. Kondall m0.25, Pearson #10.30
Pearson Ess: 0.01, Kendall i 0.000, Pearson p.0.c6
Epsi D.1, Kendali v.0.3. Pearson p.0.41
Epsi Kendal I'0.14 Pearson p.0.13
Tom Jominal Eftedive Rarks
Varient CURL R2D2 SAM
Terminal Return
Figure 26; [DeepMind lab] The effective rank and the performance: Correlations between feature ranks and episode returns for different exploration noises on Deep.iind lab dataset. We include data from three models: R2D2, R2D2 trained with CURL, and R2D2 trained with SAM. We do not observe strong correlation between the effective rank and performance across different noise exploration levels in the data.
DeamFRider
DemonAllack
Zaxazon
Effective Rank
120
60-
40
50
100
150
0
50
100
150
0
50
100
150
BeamsRider
DemonAtiack
Zaxoxn
6000
Eploodo Retums
9000
1500
4000
1000
2000
0
50
100
150
0
100
150
0
50
100 150
Leamer Stepptx 10000)
SAM weight- 00tppttt0--00000.055----
Figure 27: [Atari] Sharpness Aware Minimization (SAM) Loss: Evolution ranks as we increase the weight of auxiliary losses. We see that some amount of weight on the SAM loss helps mitigale the extent of rank collapse but we observe no clear relationship with the agent performance.
A.6 CURL on DeepMind Lab
In Figure MI shows the CURL results DeepMind lab dataset. We couldn't find any consistent pattern across various CURL loss weights.
A.7 Learning rate evolution
In Figure 311, we also perform a hyperparameter selection by evaluating the model in the environment various stages during the training. As the offline DQN is trained longer, the optimal Icarning rate for the best. agent performance when evaluated online goes down. As increases the number of training steps of agent, need to change the learning rate accordingly since the number of training steps affects the best learning rate.
A.8 Learning curves long training regime and the different phases of learning
In this subsection, we investigate the effect changing learning rates on the effective rank and the perfor- mance of the agent RI Unplugged Atari online policy selection games. We train the offline DQN for
32

---

Published in Transactions on Machine Learning Research (12/2022)
0.0
0.01
0.1
0.26
200
150
150
150
Effectve
100
100
60
60
01
150
2000
90
100
130
200
50
700
150
200
100
150
12
6-
10-
20
30
100
150
200
10
150
20c
50
100
-50
200
30
100
150
200
Leamer Stepalx 1030)
Weigh! -Se 05 05-0.01
-0.005-0.05
Figure 28: [DeepMind lab-ScekAvoid Snapshot] Sharpness Aware Minimization (SAM) Loss We do not observe any rank collapse we continue training with the LSTM networks (because of the tanh activations discussed in Section F) used in DeepMind lab dataset over spectrum of different weights for the SAM loss.
00
ca0e p-135
23
0x/00
Figure 29: [Atari]: These plots shows the correlation between the value error and the effective rank for offline DQN agent trained for 20M steps on online policy selection games for Atari. There is an apparent anti-correlation between the effective rank and the value error. Namely, as the value error an agent when evaluated in the environment increases the effective rank decreases. The correlation is significant on most Atari levels except IceHockey where the expert agent that generated the dataset performs poorly.
20M learning steps which is ten times longer than typical offline Atari agents (Emlcehre et all 2021) We evaluated twelve learning rates in [10-2, 10-9 oqually spaced in logarithmic space. We show the effective rank learning and performance curves in Figure Ge. It easy to identify different phases of learning in most of those curves.
33

---

Published in Transactions on Machine Learning Research (12/2022)
6.01
0.1
025
200
Ellecivn Donnk
50
50
0.0
0.01
0,25
Epioods Doturns
7.5
33
25
Learnr Stepsix 1003)
CURL Luse Ww..h--0-0000----1- -000000--0000000000000000 00000000-00011111
Figure 30: (DeepMind lab-SeekAvoid) auxiliary losses Evolution of ranks as we increase the weight of auxiliary loss. While some auxiliary loss helps the model perform well, there is no clear correlation between rank and performance.
0.00030
0.00025
0.006231
Learning Rale
0.00020
0.00010
0.000066
0.00005
0.000019
1e+05 1.2+006 2.4e+06 3 6e+06
B.Be008
BerD6 7.26+06 8.4er06 9 0.8e4061.000 0.684061.086-077 1 20.071.326r00 12111330274401111111 4466667
Learning Steps
Figure 31: [Atari]: Evolution of the optimal learning rate found by online evaluations in the environment. As the model trained longer the optimal learning rate found by online evaluations goes down.
A.9 Effective rank and the performance
shows the correlation between the effective rank and the performance an agent trained with minibatch of size 32. On most Atari online policy selection games it is possible to see a very strong correlation games the correlation is not there. It seems like
Figure M%
but
Figure K depicts the correlation between the effective rank and the performance of DQN agent with ReLU activation function. There is a significant correlation on most Atari games. As we discussed carlier, long training setting with ReL.U activation functions where the effect of the rank the strongest on the performance.
A.10 Computation of feature ranks
Here, we present the Python code-stub that we used across our experiments (similar Kumar et all (20200) to compute the feature ranks of the pre-output. layer's features:
import numpy as np def compute_raae taapputerran_ffo_..._o_ wwwmffroottuuuue features (feature _ettturr__marix,,,ee_et_ fankdeetaa00.11) """Computes rank of the features based how many singular values significant. """ "".
34

---

Published in Transactions on Machine Learning Research (12/2022)
BeomiRider
Demon-ttack
Doubletunk
75
75
50
25
McPasmar
Fooyan
Learning Rate
-0.000 122887227
-0 0.00/33111300 304
-0 00D4528/225
Effeciiv
60-
0.000811331
0018480035
-0 0053366995
20-
-0.01
-6.5/9335 06
0-
RewdR.uneer
Robetank
Zaceon
125
02
100
60
40
50
20
15000
10000 Learner Sleps (x1000)
20000
BeamRider
DemonAttack
DoubieDunk
2000
-20
1300
1000
5000
-24
tockey
Episode Retum
-5
230
D00005550100 304
000022222477
- B.300443287623
-B 0.0008111 111
-10
002846036
1000
- 0053366996
-15
-0.01
- -..579335e.0
-20
ReadRRumer
Robetank
Zasson
40000
20000
2000
8000
10000
15000
20000
5000 10000
15000
20000
5000
10003
15000
20000
Learnzr Steps (x1000)
Effective rank and performance offline DQN agent across nine Atari online policy selection games. We train the offline DQN agent for 20M learning steps and evaluated 12 learning rates in [10-2, 10-5 equally spaced in logarithmic space. Let us note that in all games rank goes down early in the training (Phase 1), then goes up (phase 2) and for some learning rate the effective rank collapses (Phase 3). Correspondingly, the performance is low in the beginning of the training (Phase 1), goes up and stays high for while (Phase 2), and sometimes it performance collapses (Phase 3).
Figure 32: [Atari] Effective rank curves:
35

---

Published in Transactions on Machine Learning Research (12/2022)
Beamiiteer Spearman
DemonAtack Spearman
08r
DousteDurk, Spearman
0.44
1.16
150
1.10
100
1.05
0:
lattocceey Spieman coer
-0.80
MsPacmar, Speamian com:
6.89
Ponyan, Spearman com +0.55
Terminal Effective Fanks
-ee005
1.4
0.0001
0.0003
00-
50
RoadFurrer. Spesrmen com:
Spearmon
Zecion,
12
80
30-
600
2000
4000
8000
Episode relum
Figure 33: [Atari] The correlation between ranks and returns for DQN trained with minibatch size of 32. We ran each network with three learning rates and five different seeds but we only show the mean across those five seeds here. We very strong correlations some games, but that correlation is not consistent.
Uharss Kendall PR
Denuriaas KoeGE YOSK Peanee
Datach Kenda Y2.4A Peanoo
120
Vathercry Sonbel eant Phemee
McPaener Komit VOTT Paaningrtn
Praks VO33 Puann pr.
Leario Tades
DECASSSTISO
Precfurm: Nanaal 1.01t Perase p030
Poodeok Kenkil rodt. Parned
Zevert Kancal Paanoree
+000
6000
5000 7500
12000
Figure 34: [Atari] The correlation between ranks and returns for DQN (rained for network trained for 20M learning steps and 12 different learning rates with minibatch size of 256. We can that there is significant positive correlation between the rank and the learning rates for most online policy selection games.
36

---

Published in Transactions on Machine Learning Research (12/2022)
Alyper-parametess
bsuite
Atari
DeepMind lab
Training batch size
32
256
(episodes)
Rank calculation batch size
512
512
512
Num training steps
le6
2e6
2e4
Learning rale
3e-4
3e-5
1e-3
Optimizer
Adam
Adam
Adam
Feedforward hidden layer size
64
512
256
Num hidden layers
2
1
Activation
ReLU
ReLU
ReLU (tanh for LSTM gates)
Memory
None
None
LSTM
Discount
0.99
0.99
0.997
Table The default hyper-parameters used in our work across different domains.
sing_values values ww..inaag.ssvdfettuurr matrix compute_uv uv=Fesse)
cumsum = np. cumsum (sing__alues)s
nuclear_non norm = nu. (sing_ values)
approximate_rande taanttt__an____ee
tarrosiintte_rnntthhreeodddd threshold = 1.0 - rank_delta
threshold_creee crw.sedoseeeeeeoeee
cumsum approximate_rama taanktttrehood nuclear _norm)
sefective_rante Langevalues shape mp. (po_summthreehhod_coossss
return effective_rant.
A.11 Hyperparameters
Here, we list the standard set hyper-parameters that were used in different domains: bsuite, Atari, and DeepMind Lab respectively. These are the default eyper-parametes, which may differ when stated so in our specific ablation studies. For the DMLLAB task, (202T). For all the Atari tasks, use the same convolution torso that was used by Guicehre all (2020) which involves three layers of convolution with ReLU activations in between.
the same network that was used by Galechre of all
Layer 1 (chnee(,ha,,,,s=33,, kernel shape=[8, 8), stride=[4, 4))
Layer 2 Conv2l[channel=664, kernel
4), stride=(2, [2, 2))
Layer 3 Conv2d(channel==64, kerneellll Bermel_shape=Bi_e 3), stride= 1))
A.12 bsuite phase transitions and bottleneck capacity
We illustrate the phase transition of simple MLPs with ReLU activations. In Figure B5. we have network of size (64, bottleneck units, 64) where we vary the number of bottleneck units. In Figure Sin we have a network size (64, bottleneck units) where we vary the number of bottleneck units. both cases, having smaller number of bottleneck units reduces the performance of the mode and agents were able to solve the problem even when the penultimate layer's effective rank was small, With the larger learning rate the right handside figures (b), the effective ranks tend to be lower.
A.13 Activation sparsity on bsuite
In Figure 57. we show that the activations of the ReLU network becomes very sparse during the course of training. The sparsity of the ReLU units seems be significantly higher than the ELU units at the end of training.
37

---

Published in Transactions on Machine Learning Research (12/2022)
Aeturn cann scthtecture
Learning Gptrce fetun Canch actitacuree betlenek
wgg:Boottereekk Unit)
66
64
00
-02
02
-0.4
Steps (x1300)
Steps (x1500)
Learning 36-05- Buttlnnec Hilzen
techur 144, cetreredi usts
Learting pat 2,000 Eetterace ccter lover carch with coetuuu for bottlenac 291 6r)
00
100
200
J08
466
600
Stase (x1300)
500
6.23
Loarreg
Lost Eedien
wth autelebre Ecliooo Lnii
Loerro
1000 VEOe
anteo Eurtinnc LEpS 14)
Logu wegutBreeeeeec Lumbs)
10
20
20
100
200
400
SAN 600
200
b)
Figure 35: [bsuite] Catch dataset: Phase transition plots of network with hidden layers of size (64, bottleneck units, 64). On the y-axis, we show the loga (bottleneck units), and x-axis is the number of gradient steps. The figures on the left hand-side (a) is trained with the learning rate with 8e-5 and right hand-side (b) experiments are trained with learning rate of 3e-4. The first row shows the episodic returns. The second row shows the effective rank of the second layer (bottleneck units) and the third row is showing the penultimate layer's effective rank. The effective rank collapses much faster with the learning rate of 4e-3 than 5e-8. The low ranks still have good performance. The low bottleneck units causes the effective rank of the last layer to collapse faster. The performance of the network with the small number of bottleneck units is poor. The effective rank of the small number of bottleneck units is smaller.
38

---

Published in Transactions on Machine Learning Research (12/2022)
rate
last hidden ayer Units)
Leanning mite 0,0003 Epinede Seturn en Catch with archtrch on (64, unts)
1.00
6.90
0.50
6.02
0.00
-07
logat layer l0go
-0.50
102
200
Steps (x1500)
300
600
900
600
Sceps (x1500)
Learning pite - te-5- Lest hidden layer cans catch with arth texture (94, unis
logz (lgsle hidden layer uniba)
Leomling Nilden byee
900
100
200
300
400
525
500
Steps (x1500)
Steps (x1500)
a)
b)
Figure 36: [bsuite] Catch dataset: Phase transition plots of network with hidden layers of size (64, bottleneck units). On the y-axis, we show the logz (bottlencck units), and x-axis is the number of gradient steps. The figures on the left hand-side (a) is trained with the learning rate with 8e-5 and right hand-side (b) experiments are trained with learning rate 3e-4. The first row shows the episodic returns. The second row shows the effective rank of the second layer (bottleneck units). The effective rank collapses much faster with the lcarning rate of 4c-3 than 5c-8. The low ranks can still have good performance. The small number of bottleneck units causes the effective rank of the layer to collapse faster. The performance of the network with the small number of bottleneck units is poor. The effective rank of the small number of bottleneck units is smaller.
Initialization (step 0)
Convergence (step 900k)
ReLU
ELU
Figure 37: [bsuite Catch] Gram matrices of activations: Gram matrices of activations of two-layer MLP with ReLU and ELU activation functions. The activations of the ReLU units become sparser when compared to ELU units at the end of the training due to dead ReLU units.
39


Reviews:
[{"review_id": "mqf87XG2q1", "reviewer": "Reviewer_8ZFd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I found the presentation to be clear and engaging. In particular the authors were careful to distinguish between correlation and causation- clearly the latter is what we are interested in here, and so I appreciated that an explicit causal graph was hypothesized and tested. \n\nThe authors seem to have done a good job of conducting a comprehensive set of empirical tests, and make a convincing case that it would be misleading to claim that implicit underparametrization/rank collapse provides a monocausal explanation of the failures of deep Q-learning. I should say however that I am far from an expert in this field, and don't consider myself well qualified to judge the strength of the empirical evidence marshalled by the authors.\n\nThe authors suggest a three phase picture of Q learning- a first phase where \"simple\" behaviors are learned, a second phase where \"complex\" behaviors are learned, and a final phase of underparametrization. Given the subjective terminology, was unclear to me how exactly the authors defined these phases, and would have appreciated more details.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors conduct an extensive empirical investigation into the phenomenon of implicit under parametrization in offline (deep) Q learning (introduced by Lavine et al in 2020), i.e a situation where high capacity functional aproximators trained via TD learning are unable to fit the data because the final layer of activation \"collapses\" to a highly restricted subspace. They conclude that while one can indeed observe this behavior in some settings, the existence of implicit underparametrization is highly sensitive to network hyperparameters such as choice of activation function, learning rate, minibatch size etc. They conclude that extrapolating from simplified models (where one can analytically prove that implicit under parametrization occurs) and specific hyperparameter settings (where one can verify its occurence empirically) is misleading, and that the precise role of implicit underparametrization in the performance of deep RL methods is still unclear.  ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Given that the authors of the original implicit underparametrization paper proved that the phenomenon occurs in kernel regression, it would have been interesting to examine the behavior of wide (but finite) networks. I should emphasize that I don't view the inclusion of tests of this regime to be critical for securing my reccomendation for acceptance. ", "broader_impact_concerns": "I don't see any particular ethical concerns with the paper. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pyOwaGU4yq", "reviewer": "Reviewer_hEii", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. Investigation the relationship between rank of the penultimate layer of neural networks and agents' performance is an interesting topic, and may motivate more follow-up work.\n\n2. The authors did provide an extensive investigation on the topic from different angles, which is a good first attempt to tackle this problem.\n\n\nWeakness\n1. The current version is over empirical and hence it's difficult to draw any concrete conclusion. For example, on Key Observation 2 in Page 3, it's extremely difficult to verify that the three stages are necessary. Furthermore, the definitions for \"simple\" and \"complex\" are lacking and hence we cannot distinguish them with clear boundaries.\n\n2. As mentioned above, this submission cannot draw any conclusion based on the results shown. It's true that there is somewhat correlation between the effective rank and performance. However, we cannot know when it will or not will be the case from the provided results.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This submission provides an extensive empirical investigation on the relationship between effective rank and agents' performance in deep offline reinforcement learning. The authors proposed a few hypotheses on connecting ranks with different ingredient in deep neural networks. Experiments are provided to validate and discuss the hypotheses. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "It would be more convincing if the authors can provide either (1) theoretical justification on the relationship between rank and performance (2) more convincing empirical justification to draw some \"specific\" conclusion. A few detailed questions and comments are provided as follows.\n\n1. In the caption for Figure 5, the authors stated that \"the ranks are distributed almost like a Gaussian\". More rigor is necessary, instead of simple guesses. \n\n2. The authors mentioned in the caption of Figure 16 that \"auxiliary loss helps mitigate the rank collapse but prevents the model from learning useful representations\". Based on my understanding, there are no steps to measure the usefulness of representations and hence such a measure is necessary before drawing this conclusion.\n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "p4llr7pdHD", "reviewer": "Reviewer_5mXC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Below are my views on the strengths and weaknesses of the results. Note that I am not very familiar with RL and so am not very qualified to comment on the appropriateness or quality of the experiments.\n\nStrengths:\n- The paper identifies an interesting hypothesis and does a very thorough investigation of it.\n- The authors identify numerous hyperparameters that may affect the outcome and study their ablations.\n- The writing is fairly straightforward to understand.\n- In addition to the main objective of the work, there is also an interesting discussion and empirical investigation of training dynamics in RL.\n\nWeaknesses:\n- One important discussion that seems missing to me is an explanation for why Kumar et al. (2020) observed improved performance when controlling rank collapse.\n- W.r.t. the definition of effective rank: I understand that the authors are aiming to align with Kumar et al. (2020), but since this “effective rank” quantity is effectively an attempt to discount small singular values, I wonder if a better quantity to study—since it is parameter-free—would be the stable rank (c.f. the actual quantity used in Arora et al. (2018) and Sanyal et al. (2020)). Of course, these papers would argue instead for a smaller rather than a larger (stable) rank.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates the hypothesis that (effective) rank collapse causes poor performance in offline RL. Through an extensive set of experiments on offline RL benchmarks, the authors show that the results identified in past work only hold for a small range of hyperparameters, and so targeting it may not be a viable solution to improving performance. They also identify several interesting phases of offline RL training related to the rank and performance.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Changes that are critical for me to recommend acceptance:\n- Discussion: as discussed in the weaknesses, I believe the observation by Kumar et al. (2020) that controlling rank could improve performance should be better-discussed. Is this also a question of hyperparameter range, or do you believe there are confounding factors? \n\nChanges that would strengthen paper:\n- Figure 2: it’d be useful for the reader to know if the y-axes here start at zero (rank / returns) or not.\n- Section 2.2: I’m confused by the usage of the word “ablate” here. Usually it means to remove certain components to check their effects, but here it seems to just mean you use them? Perhaps writing here could be made more clear.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewer hEii raised some concerns about the submission being overly empirical and lacking theoretical grounding.  While I agree that it would have been much stronger had it included backing theory, I think the empirical contribution is of sufficient novelty and significance for publication, and the text is completely transparent about its empirical nature.  I therefore recommend acceptance.
Decision: Accept (poster)