DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularization
Authors: Aviral Kumar
Rishabh Agarwal
Tengyu Ma
Aaron Courville
George Tucker
Sergey Levine
Venue: ICLR 2022 Spotlight (2022)
Keywords: Q-learning
offline RL
regularization

Abstract: Despite overparameterization, deep networks trained via supervised learning are surprisingly easy to optimize and exhibit excellent generalization. One hypothesis to explain this is that overparameterized deep networks enjoy the benefits of implicit regularization induced by stochastic gradient descent, which favors parsimonious solutions that generalize well on test inputs. It is reasonable to surmise that deep reinforcement learning (RL) methods could also benefit from this effect. In this paper, we discuss how the implicit regularization effect of SGD seen in supervised learning could in fact be harmful in the offline deep RL setting, leading to poor generalization and degenerate feature representations. Our theoretical analysis shows that when existing models of implicit regularization are applied to temporal difference learning, the resulting derived regularizer favors degenerate solutions with excessive aliasing, in stark contrast to the supervised learning case. We back up these findings empirically, showing that feature representations learned by a deep network value function trained via bootstrapping can indeed become degenerate, aliasing the representations for state-action pairs that appear on either side of the Bellman backup. To address this issue, we derive the form of this implicit regularizer and, inspired by this derivation, propose a simple and effective explicit regularizer, called DR3, that counteracts the undesirable effects of this implicit regularizer. When combined with existing offline RL methods, DR3 substantially improves performance and stability, alleviating unlearning in Atari 2600 games, D4RL domains and robotic manipulation from images.

Published as a conference paper at ICLR 2022
DR3: VALUE-BASED DEEP REINFORCEMENT LEARNING REQUIRES EXPLICIT REGULARIZATION
Aviral Kumar
Rishabh Agarwal
Tengyu Mad
Aaron Courville
George Tucker² Sergey Levine
UC Berkeley
2 Google Research 3 MILA 4 Stanford University
aviralk@berkeley.edd
ABSTRACT
Despite overparameterization, deep networks trained via supervised learning are easy to optimize and exhibit excellent generalization. One hypothesis to explain this is that overparameterized deep networks enjoy the benefits of implicit regu- larization induced by stochastic gradient descent, which favors parsimonious so- lutions that generalize well on test inputs. It is reasonable to surmise that deep reinforcement learning (RL) methods could also benefit from this effect. In this paper, discuss how the implicit regularization effect of SGD seen in supervised learning could in fact be harmful in the offline deep RL setting, leading to poor generalization and degenerate feature representations. Our theoretical analysis shows that when existing models of implicit regularization are applied to temporal difference learning. the resulting derived regularizer favors degenerate solutions with excessive "aliasing". in stark contrast to the supervised learning case. We back up these findings empirically, showing that feature representations learned by a deep network value function trained via bootstrapping can indeed become degenerate, aliasing the representations for state-action pairs that appear on either side of the Bellman backup. To address this issue, we derive the form of this im- plicit regularizer and, inspired by this derivation, propose a simple and effective explicit regularizer, called DR3, that counteracts the undesirable effects of this implicit regularizer. When combined with existing offline RL methods, DR3 sub- stantially improves performance and stability, alleviating unlearning in Atari 2600 games, D4RL domains and robotic manipulation from images.
INTRODUCTION
Deep neural networks are overparameterized, with billions of parameters, which in principle should leave them vulnerable to overfitting. Despite this, supervised learning with deep networks still learn representations that generalize well. A widely held consensus is that deep nets find simple solutions that generalize due to various implicit regularization effects [6, 48, 4, 23, 47, 30). We may surmise that using deep neural nets in reinforcement learning (RL) will work well for the same reason, learning effective representations that generalize due to such implicit regularization effects. But is this actually the case for value functions trained via bootstrapping?
In this paper, we argue that, while implicit regularization leads to effective representations in super- vised deep learning, it may lead to poor learned representations when training overparameterized deep network value functions. In order to rule out confounding effects from exploration and non- stationary data distributions, we focus on the offline RL setting where deep value networks must be trained from a static dataset of experience. There is already evidence that value functions trained via bootstrapping learn poor representations: value functions trained with offline deep RL eventually degrade in performance [2, 28] and this degradation is correlated with the emergence of low-rank features in the value network [28]. Our goal to understand the underlying cause of the emer- gence of poor representations during bootstrapping and develop potential solution. Building the theoretical framework developed by Blanc et al. [6], Damian et al. [11], we characterize the im- plicit regularizer that arises when training deep value functions with TD learning. The form of this implicit regularizer implies that TD-learning would co-adapt feature representations at state-action tuples that appear either side of Bellman backup.
We show that this theoretically predicted aliasing phenomenon manifests in practice as feature co- adaptation, where the features of consecutive state-action tuples learned by the Q-value network

---

Published as a conference paper at ICLR 2022
become very similar in terms of their dot product (Section 3). This co-adaptation co-occurs with oscillatory learning dynamics, and training runs that exhibit feature co- co-adaptation typically converge to poorly performing solutions. Even when Q-values are not overestimated, prolonged training in offline RI can result in performance degradation as feature co-adaptation increases. To mitigate this co-adaptation issue, which arises result of implicit regularization, we propose explicit regularizer that we call DR3 (Section 4). While exactly estimating and cancelling the effects of the theoretically derived implicit regularizer is computationally difficult, DR3 provides a simple and tractable theoretically- shoretically-inspired approximation that mitigates the issues discussed above. In practice, DR3 amounts to regularizing the features at consecutive of their dot-product similarity. Empirically, we find that DR3 prevents previously noted pathologies such as feature rank collapse [28], gives methods that train for longer and improves performance relative to the base offline RL method employed in practice.
action pairs to be dissimilar in terms
Our first contribution is the derivation of the implicit regularizer that arises when training deep net value functions via TD learning, and empirical demonstration that it manifests as feature co-adaptation in the offline deep RL setting. Feature co-adaptation accounts at least in part for some of the challenges of offline deep RL, including degradation of performance with prolonged training. Second, we propose a simple and effective explicit regularizer for offline value-basee RL, DR3, which minimizes the feature similarity between state-action pairs appearing in bootstrapping update. DR3 is inspired by the theoretical derivation of the implicit regularizer, it alleviates co- adaptation and can be easily combined with modern offline RL methods, such as REM [2], CQL [27], and BRAC [49]. Empirically, using DR3 in conjunction with existing offline RL methods provides about 60% % performance improvement on the harder D4RL [18] tasks, and 160% and 25% stability gains for REM and CQL, respectively, on offline RL tasks in 17 Atari 2600 games. Additionally, we observe large improvements on image-basee robotic manipulation tasks [38].
2 PRELIMINARIES
The goal in RL is to maximize the long-term discounted reward in an MDP, defined as (S,A,R, P.7) [36], with state space S, action space A. reward function R(s, a). dynamics P(s' is, a) and a discount factor y E [0, 1). The Q-function Q as) for a policy π r[ass) is the expected sum of discounted rewards obtained by executing action at state and following (Tals) thereafter. (s, a) is the fixed point of Q(s, := Ris, a) + (()ss))
[Q(s', a()] We study the offline RL setting, where the algorithm must learn policy only using a given dataset D = [(s, a, s', r.)), generated from some behavior policy, π (als), without active data collection. The Q-function is parameterized with a neural net with parameters 0. We will denote the penultimate layer of the deep network (the learned features) (s, a). such that Qo a) w vTs a), where w E IRd, Standard deep RL methods [33, 24] convert the Bellman equation into a squared temporal difference (TD) error objective for Qe:
LTD(0)=
Σ (R(s, a) + + Q0 a') - Q0(8, a))² B,a.8'~D
where Qo is a delayed copy of same Q-network, referred to as the target network and a' is computed by maximizing the target Q-function at state s' for Q-learning (i.e., when computing Q*) and by sampling a' ~ (-|ss when computing the Q-value Q' of a policy n.
A major problem in offline RL is the issue of distributional shift between the learned policy and the behavior policy [29]. Since our goal is to study the effect of implicit regularization in TD-Icarning and not distributional shift, we build on top of existing offline RL methods in our experiments: CQL[[7], which penalizes erroneous Q-values during training, REM [2], which utilizes an ensemble of Q-functions, and BRAC [49], which applies a policy constraint. An overview of these methods is provided in Appendix E.
3 IMPLICIT REGULARIZATION IN DEEP RL VIA TD-LEARNING
While the "deadly-triad" [39] suggests that training value function approximators with bootstrap- ping off-policy can lead to divergence, modern deep RL algorithms have been able to successfully combine these properties [43]. However, making too many TD updates to the Q-function in offline deep RL is known to sometimes lead to performance degradation and unlearning, even for otherwise effective modern algorithms [17, 16, 2, 28]. Such unlearning is not typically observed when training overparameterized models via supervised learning. so what about TD learning is responsible for it? We show that one possible explanation behind this pathology is the implicit regularization induced
2

---

Published as a conference paper at ICLR 2022
Large feature products anie when cunpoff-ample actions are used TD-lcaming
festure products eventuuall correlate with divergent Qivolues
Large feature dos products arise when Q leterningg despito
companed SARSA, desp similar Quvallass
BARRA
DON
Supervaud
High dal prodacts
10000
Draalout
10"
4D0
1000
200
201
Oradlert
séc
Incorrect Qvaluess found eventually; dobepraducts increase
Upcilio $2.50)
120
300
Grediem Upditn (a
Breakoot
Dreakout
Seoup aod
Breokout
prcughouttrainng
Smir Qeeluss
Neerge-vatia Awerge Quaton
Nv
Awergge 0
0
8
100 200
133
200
800
200
400
800
600
Uppitos
Figure 1: Feature dot-products d(s, a) 6(s', a') increase during training when backing up from out-of- sample but in-distribution actions (TD-learning: lcft, Q-learning: right), though the average Q-value con- verges and stays relatively constant. Using only seen state-action pairs for backups (offiiee SARSA) or not performing Bellman backups (i.c., supervised regression) avoids this issue, with stable and relatively low dot products. Left: TD-learning with high feature dot products eventually destabilizes and produces incorrect Q-values, Right: DQN attains extremely large feature dot products, despite relatively stable trend Q-values.
by minimizing TD error a deep Q-network. Our theoretical results suggest that this implicit regularization "co-adapts" the representations of state-action pairs that appear in a Bellman backup (we will define this more precisely below). Empirically, this typically manifests "co-adapted" features for consecutive state-action tuples, even with specialized TD-learning algorithms that ac- count for distributional shift, and this turn leads to poor final performance both in theory and in practice. We first provide empirical evidence of this co-adaptation phenomenon in Section 3.1 (additional evidence in Appendix A) and then theoretically characterize the implicit regularization in TD learning, and discuss how it can explain the co-adaptation phenomenon in Section 3.2.
3.1 FEATURE CO-ADAPTATION AND HOW IT RELATES To IMPLICIT REGULARIZATION
In this section, we empirically identify a feature co-adaptation phenomenon that appears when training value functions via bootstrapping, where the feature representations of consecutive state- action pairs exhibit a large value of the dot product 6(s, a) "(s', a'). Note that feature co-adaptation may arise because of high cosine similarity or because of high feature norms. Feature co-adaptation appears even when there is no explicit objective to increase feature similarity.
Experimental setup. We ran supervised regression and three variants of approximate dynamic pro- gramming (ADP) on an offline dataset consisting of 1 1% of uniformly-sande buffer of DQN on two Atari games, previously used in Agarwal al. [2]. First, for comparison, we trained a Q-function via supervised regression to Monte-Carlo (MC) return estimates on the offline dataset to estimate the value of the behavior policy. Then, we trained variants of ADP which differ in the selection procedure for the action a' that appears in the target value in (T) (Equation 1). The offline SARSA variant aims to estimate the value of the behavior policy, QTa, and sets a' to the actual action observed the next time step in the dataset, such that (s', a') E D. The TD- learning variant also aims to estimate the value of the behavior policy, but utilizes the expectation of the target Q-value over actions a' sampled from the behavior policy na, a' ~ (-ss) We do not have access to the functional form of Ty for the experiment shown in Figure since the dataset corresponds the behavior policy induced by the replay buffer of an online DQN, so we train model for this policy using supervised learning. However, we see similar results comparing offfine SARSA and TD-learning on gridworld domain where we can access the exact functional form of the behavior policy in Appendix A.6.2. All of the methods so far estimate QT using different target value estimators. We also train Q-learning, which chooses the action a' to maximize the learned Q- function. While Q-learning learns a different Q-function, we can still compare the relative stability of these methods to gain intuition about the learning dynamics. In addition to feature dot products Φ(s, a)? dis', a'). we also track the average prediction of the Q-network over the dataset to measure whether the predictions diverge or are stable in expectation.
data from the replay
Observing feature co-adaptation empirically. As shown in Figure 1 (right), the average dot prod- uet (top row) between features at consecutive state-action tuples continuously increases for both Q-learning and TD-leaming (after enough gradient steps), whereas it flatlines and converges
3

---

Published as a conference paper at ICLR 2022
small value for supervised regression. We might at first think that this is simply of Q-learning failing to converge. However, the bottom row shows that the average Q-values do in fact converge to stable value. Despite this, the optimizer drives the network towards higher feature dot products. There is no explicit term in the TD error objective that encourages this behavior, indicating the pres- ence of some implicit regularization phenomenon. This implicit preference towards maximizing the dot products of features at consecutive state-action tuples is what we call "feature co-adaptation."
When does feature co-adaptation emerge? Observe in Figure (right) that the feature dot products for offline SARSA converge quickly and are relatively flat, similarly to supervised regression. This indicates that utilizing bootstrapped update alone is not responsible for the increasing dot products and instability, because while offline SARSA uses backups. it behaves similarly to supervised MC regression. Unlike offline SARSA, feature co-adaptation emerges for TD-learning, which is sur- prising as TD-eearning also aims to estimate the value of the behavior policy, and hence should match offiie SARSA expectation. The key difference that while offline SARSA always uti- lizes actions a' observed in the training dataset for the backup, TD-learning may utilize potentially unseen actions in the backup, even though these actions a' ~ (-(ss")) are within the distribution of the gett--genrratigg policy. This suggests that utilizing out-of-sample actions in the Bellman backup, even when they are not out-of distriibitt, critically alters the learning dynamics. This is distinct from the more common observation in offine RL., which attributes training challenges to distr--distribution actions [29], but not out-of-sample actions. The theoretical model developed in Section 3.2 will provide an explanation for this observation with a discussion about how feature co-adaption caused due to out-of-sample actions can be detrimental in offline RL.
3.2 THEORETICALL CHARACTERIZING IMPLICIT REGULARIZATION IN TD-LEARNING
Why does feature co-adaptation emerge in TD-learning and what do out-of-sample actions have to do with it? To answer this question, we theoretically characterize the implicit regularization effects in TD-learning. We analyze the learning dynamics of TD learning in the overparameterized regime, where there are many different parameter vectors 0 that fully minimize the training set temporal difference error. We base our analysis of TD learning on the analysis of implicit regularization in supervised learning, previously developed by Blanc et al. [6], Damian et al. [11].
Background. When training an overparameterized fo(x) via supervised regression using the squared loss, denoted by L, many different values of will satisfy L(Ø) = 0 on the training set due to overparameterriztion, but Blanc et al. [6] show that the dynamics of stochastic gradient de- scent will only find fixed points 0* that additionally satisfy a condition which can be expressed as (o = 0, along certain directions (that we will describe shortly). This function R(0) is referred to as the implicit regularizer. The noisy gradient updates analyzed in this model have the form:
0x+1 +- 02 v/ + MER, S ~ N (0, M).
Blanc et al. [6] and Damian et al. [11] show that some common SGD techniques fall into this frame- work, for example, when the regression targets in supervised learning are corrupted with A/ 1) label noise, then the resulting MM E. (o T and the induced implicit regular- izer R is given by R(0) = Σ. pll [[VaJa(x))]] Any solution 0* found by Equation 2 must satisfy (o 0 along directions E RI which lie in the null space of the Hessian of the loss VOL(0") at 8", v E Null(V3L((*)).. The intuition behind the implicit regularization effect is that along such directions in the parameter space, the Hessian is unable to contract 0₂ when running noisy gradient updates (Equation 2). Therefore, the only condition that the noisy gradient updates converge/stabilizee at 6" given by the condition that VR(@*) 0. This model corroborates empir- ical findings [34, 11] about the solutions found by SGD, which motivates our use.
Our setup. Following this framework, we analyze the fixed points of noisy TD-learning. We consider noisy geeudo-gradeent (or semi-gradient) TD updates with general noise covariance M:
0x+1 = 02 -
VeQ(s,, a.) (Qo(s, - -r)
tNEK, EN ~O (3)
(0)
We use deterministic policy at = π (s/) to simplify exposition. Following Damian et al. [1]], we can set the noise model M as M Σ, VoQ(s:, a.).s a;) or utilize different choice of
4

---

Published as a conference paper at ICLR 2022
M. but we will derive the general form first. Let H* denote a stationary point of the training TD error, such that the pseudo-gradin gradient g(0*) = O. Further, we denote the derivative of g(0) w.r.t. 0 as the matrix G(Ø) E I |o)),,01, and refer to as the pseudo-Hessian: although G(0) is not actually the second derivative of any well-defined objective, since TD updates are not proper gradient updates, as we will see it will play a similar role to the Hessian in gradient descent. For brevity, define G=G(0") 9 = g(0)), VG = VoG(0*) E [211 and let Xx(P) denote the i-th eigenvalue of matrix P. when arranged in decreasing order of its (complex) magnitude [A,(P)] (note that an eigenvalue can be complex).
Assumptions. To simplify analysis, assume that matrices G' and M (i.e., the noise covariance matrix) span the same n-dimensional basis in d-dimensional space, where d is the number of param- eters and n is the number of datapoints, and n < d due to overparameterization. We also require O" to satisfy technical criterion that requires approximate alignment between the eigenspaces of G and the gradient of the Q-function, without which noisy TD may not be stable 0*, We summarize all the assumptions in Appendix C, and present the resulting regularizer below.
Theorem 3.1 (Implicit regularizer at TD fixed points). Under the assumptions so far, a fixed point
a., of TD-learning, 0*, where Q (8;, a.) = vi + 1Q (s', a() for every as,,a, s() E DD is stable (atttractive) if: (I) satisfies Re( (G)) 0, vi and Re()(GG) 0 [[lmag(); > 0, and (2)
along directions v E Rdim((), .V Null(G), 0* is the stationary point of the implicit regularizer:
D
[D]
(DD(()= Σ VQo(s,, a.) E QQ a m tr (I[VQo(s), TI" VQ,
imspiicit regularizer for noisy GD supervised Teorning
additional ferm TD learning
where (s., a.) and (s', aL) denote state-action pairs that appear together in a Bellman update. [[]]]
denotes the stop-gradion functonn,, which does not pass partial gradients w.r.t. 9 into L. MM is the
fixed point of the discrete Lyapunov equation: ΣM == (1 ((--nG))E3s4 GG) + M.
proof of Theorem 3.1 is provided in Appendix C. Next, we explain the intuition behind this result and provide a proof sketch. To derive the induced implicit regularizer for a stable fixed point l of TD error, we study the learning dynamics of noisy TD learning (Equation 3) initialized at 8*, and derive conditions under which this noisy update would stay close to 0* with multiple updates. This gives rise to the two conditions shown in Theorem 3.1 which can be understood as controlling stability in mutually exclusive directions in the parameter space. If condition (1) is not satisfied, then even under- paranettrraeeeed TD will diverge away from 8*, since I - 1iG would be a non-contraction as the spectral radius, p(I G/G 2 1 in that case. Thus, 6% - 0" will grow not decrease in some direction. When (1) is satisfied for all directions in the parameter space, there are still directions where both the real and imaginary parts of the eigenvalue )₂ are O due overparameterization! In such directions, learning is governed by the projection of the noise under the tensor VG, which appears in the Taylor expansion of On- 0* around the point 0*:
OK+1 O2 - 11 g + G(Ox- 07) + V 0-, 0²,0, - 071) Sk; ek ~ ~40, M) (5)
= 1441 nG-nG) " " VVlu, ux + Eks
where we reparameterize in terms of 1/ 02 0r. The proof shows that 0- is stable if it is stationary point of the implicit regularizer RTD (condition (2)), which ensures that total noise (i.c., accumulated & over iterations k) accumulated by VG does not lead to a large deviation in 2i in directions where - T/G does not contract.
Interpretation of Theorem 3.1. While the choice of the noise model M will change the form of the implicit regularizer, in practice, the form of M is not known as this corresponds to the noise induced via SGD. We can consider choices of M for interpretation, but Theorem 3.1 is easy qualitatively interpret for M such that Ei, = I. In this case, we find that the implicit preference towards local minima of RTD(O) can explain feature co-adaptation. In this case, the regularizer is simpler:
(TD( Σ (s. 1 ~V a.) VI[[o a()]]-
To see why this the case, note that rank(G) S |D] < dim(0). and so some eigenvalues of G are
5

---

Published as a conference paper at ICLR 2022
Return degrades even starting from good salutions
Festure dot products increase returns decrease
Overall training error for COL/DQN is still low
Aater
Breelout
Adade
Astede
Brealout
200
Oveel Tranry Loap
www.o/owowooo..ooooooommoo 15
DQN
400
0.10
100
0.55
CQL
13
Updete io $2.50
160
Cradint Lidenn Jx2250
50
100
Goddent Upeitiee (2:50)
50
130
Grodaer Uptittt Da (2:90)
Figure 2:
Even when current offine RL algorithms are initialized at a high- nigh-performing checkpoint that attains
small feature dot products, feature dot products increase with further training and the performance degrades.
The first term is equal to the squared per-datapoint gradient norm, which is same as the implicit regularizer in supervised learning obtained by Blane et al. [6], Damian et al. [1I] with label noise. However, (TD(O) additionally includes a second term that is equal to the dot product of the gradient of the Q-function at the current and next states, VoQo a;) Qo a/), and thus this term is effectively maximized. When restricted to the last-layer parameters of a neural network, this term is equal to the dot product of the features at consecutive state-action tuples: Σ, VeQe (S;, a.) VoQ a() = Σ, ((s., a,) ((s), a,). The tendency to maximize this quan- tity to attain a local minimizer of the implicit regularizer corroborates the empirical findings of increased dot product in Section 3.1.
Explaining the difference between utilizing seen and unseen actions in the backup. If all state- action pairs (s). af) appearing on the right- hant--h of the Bellman update also appear in the dataset D, as in the case of offline SARSA (Figure 1), the preference to increase dot products will be balanced by the affinity to reduce gradient norm (first term of (TD(O) when Ei = I): for ex- ample, for offline SARSA, when (s(. a!) are permutations of (s., a:). RTD is lower bounded by (1 -1-27 (1- E. (x. and hence minimizing RTD(() (0) would minimize the feature norm instead of maximizing dot products. This also corresponds to the implicit regularizer we would obtain when training Q- Q-functions via supervised learning and hence, our analysis predicts that offline SARSA with in-sample actions (i.e., when (s', a') E D) would behave similarly to supervised regression.
However, the regularizer behaves very differently when unseen state-action pairs (s/, a!) appear only on the
of the backup. This happens with any algorithm where a' is not the dataset action, which is the case for all deep RL algorithms that compute target values by selecting a' according to the current policy. In this case, we expect the dot product of gradients at (s, a) and (s', a') to be large any attractive fixed point, since this minimizes (f). This is precisely a form of co-adaptation: gradients at sat-le state-action tuples are highly similar to gradients at observed state-action pairs measured by the dot product. This observation also supported by the analysis in Section 3.1. Finally, note that the choice of M is modelling assumption, and to derive our explicit regularizer, later the paper, we will make simplifying choice of M. However, we also empirically verify that a different choice of M, given by label noise, works well.
Why is implicit regularization detrimental to policy performance? To answer this question, we present theoretical and empirical evidence that illustrates the adverse effects of this implicit regularizer. Empirically, we ran two algorithms, DQN and CQL, initialized from a perf-performin Q-function checkpoint, which attains relatively small feature dot products (i.e., the second term of (T is small). Our goal is to see if TD updates starting from such a "good" initialization still stay around it diverge poorer solutions. Our theoretical analysis in Section 3.2 would predict that TD learning would destabilize from such a solution, since it would not be a stable fixed point. Indeed, as shown in Figure 2, the policy immediately degrades, and the the dot-product similarities start to increase. This even happens with CQL, which explicitly corrects for distributional shift confounds, implying that the performance drop cannot be directly explained by the typical out-of- distribution action explanations. To investigate the reasons behind this drop, we also measured the training loss function values for these algorithms (i.e., TD error for DQN and TD error + COL regularizer for CQL) and find in Figure 2 that the loss values are generally small for both CQL and DQN. This indicates that the preference to increase dot products is not explained by an inability to minimize TD error. In Appendix A.7, we show that this drop in performance when starting from good solutions can be effectively mitigated with our proposed DR3 explicit regularizer for both DQN and CQL. Thus we find that not only standard TD learning degrades from good solution in favor of increasing feature dot products, but keeping small dot products enables these algorithms to remain stable near the good solution.

---

Published as a conference paper at ICLR 2022
To motivate why co-adapted features can lead to poor performance in TD-learning, study the convergence of linear TD-learning on co-adapted features. Our theoretical result characterizes a lower bound on the feature dot products in terms of the feature norms for state-action pairs in the dataset DD which if satisfedd, will inhibit convergence:
Proposition 3.2 (TD-learning on co-adapted features). Assume that the features & [o(s, a)ls.a are used for linear TD-learning. Then, if E o(s, a) "o''''sss""" a') 2 3 Es.ED Ø(s. dsss a).
linear TD-learning using features 4 will not converge.
A proof of Proposition 3.2 is provided in Appendix D and it relies on a stability analysis of linear TD. While features change during training for TD-leaming with neural networks, and arguably linear TD is simple model to study consequences of co-adapted features, even in this simple linear setting, Proposition 3.2 indicates that T'D-learning may be non-convergent as a result of co-adaptation.
4 DR3: EXPLICIT REGULARIZATION FOR DEEP TD-LEARNING
Since the implicit regularization effects in TD- TD-learning can lead to feature co-adaptation, which in turn is correlated with poor performance, can we instead derive an explicit regularizer to alleviate this issue? Inspired by the analysis in the previous section, will propose an explicit regularizer that at- tempts to counteract the second term in Equation 4, which would otherwise lead to co-adaptation and poor representations. The explicit regularizer that offsets the difference between the two implicit reg- ularizers is given by: A = Σ; Irace [ V (s, a V Vo a{) T, which represents the second term of (O) Note that we drop the stop gradient on (s), a() in A (0)), as it performs slightly better in practice (Table 1) although as shown in that Table, the version with the stop gradient also significantly improves over the base method. The first term of (T) corresponds to the regularizer from supervised learning. Our proposed method, DR3, simply combines approxima- tions to (0) with various offline RL algorithms. For any offline RL algorithm, ALG, with objective CALa (0), the training objective with DR3 is given by: C(0) == A + (B), where cu the DR3 coefficient. See Appendix E.3 for details on how we tune cu in this paper.
Practical version of DR3. In order to practically instantiate DR3, we need choose particular noise model M. In general, it is not possible to know beforehand the "correct" choice of M (Fqua- tion 3), even in supervised learning, as this a complicated function of the data distribution, neural network architecture and initialization. Therefore, we instantiate DR3 with two heuristic choices of M: (i) M induced by label noise studied in prior work for supervised learning and for which we need to run a computationally heavy fixed-point computation for M, and (ii) simpler alternative that sets Σn = I. We find that both of these variants generally perform well empirically (Figure 6), and improve over the base offline RL method, and we utilize (ii) in practice due to low compu- tational costs. Additionally, because computing and backpropagating through per-example gradient dot products is slow, we instead approximate A(0) with the contribution only from the last layer parameters (i.e., E, V Qe(;,, a;) V (s',, a()), similarly to tractable Bayesian neural nets. As shown in Appendix A.5, the practical version of DR3 performs similarly the label-noise version.
Explicit DR3 regularizer:
Rexp (0) = Σ olss, a:) (6/s,, a().
RED
EXPERIMENTAL EVALUATION OF DR3
Our experiments aim evaluate the extent to which DR3 improves performance in offline RL in practice, and to study its effect on prior observations of rank collapse. To this end, we investigate if DR3 improves offline RL performance and stability on three offline RL benchmarks: Atari 2600 games with discrete actions [2], continuous control tasks from D4RL [18], and image-based robotic manipulation tasks [38]. Following prior work [18, 22]. we evaluate DR3 in terms of final offline RL performancee after a given number of iterations. Additionally, we report truining stability, which important in practice as offline RI. does not admit cheap validation of trained policies for model selection. To evaluate stability, prior work) and either report the average performance over the course of training or the final performance at the end of training. We expect that a stable method that does not unlearn with more gradient steps, should have better average performance, as compared to a method that attains good peak performance but degrades with more training. See Appendix E for further details.
train for large number of gradient steps (2-3x longer than
Offline RL on Atari 2600 games. We compare DR3 to prior offline RL methods on a set of offline Atari datasets of varying sizes and quality, akin to Agarwal et al. [2], Kumar al. [28]. We evaluated
7

---

Published as a conference paper at ICLR 2022
Table 1QM normalized average performance (training stability) across 17 games, with 95% CIs in parenthe- sis, after 6.5M gradient steps for the 1 % setting and 12.5M gradient steps for the 5%, 10% settings. Individual performances reported in Tables F.4-F9. DR3 improves the stability over both CQL. and REM.
Data
CQL
CQL + DR3
REM
REM + DR3
1%
43.7 (39.6, 48.6)
56.9 (52.5, 61.2)
4.0 (3.3, 4.8)
16.5 (14.5, 18.6)
5%
78.1 (74.5, 82.4)
105.7 (101.9, 110.9)
25.9 (23.4, 28.8)
60.2 (55.8, 65.1)
10%
59.3 (56.4, 61.9)
65.8 (63.3, 68.3)
53.3 (51.4, 55.3)
73.8 (69.3, 78)
Class Cope--C---rrnn Crawp 2800)
Connegenrree (211)
Norroldd Boog
1% Unitorm Replay
5% Unitorm Reslay
10% in Replay
18.0%)
78%
90%
50%
60%
25%
30%
0.0%
0%
0%%
50
75
200
250
1% Uniform Replay
Uniorm Roplay
10% Inttial Replay
Pla Openr Grarg (55-)
82%
120%
75%
00%
60%
20%
40%
28%
0%
0%
0%
100
Graciont 62
Grodknnt
Figure 4: Normalized performance across 17 Atari games for REM + DR3 (top), CQL DR3 (bottom). x-axis rep- resents gradient steps; no new data is collected. While nai'e REM suffers from degradation in performance with more training, REM DR3 not only remains generally stable with more training, but also attains higher final performance. CQL + DR3 attains higher performance than CQL. We report IQM with 95% stratified bootstrap CIs [3].
Coudket
Figure 3: Performance of DR3+ + COG on
two manipulation tasks using only 5% and 25% of the data used by Singh et al. [38] to make these more challenging. COG +
DR3 outperforms COG in training and at-
tains higher average and final performance.
on three datasets: (1) 1 1% and 5% samples drawn uniformly at random from DQN replay; (2) dataset with more suboptimal data consisting of the first 10% samples observed by an online DQN. Following Agarwal et al. [3], we report the interquartile mean (IQM) normalized scores across 17 games over the course of training in Figure 4 and report the IQM average performance in Table 1. Observe that combining DR3 with modern offline RL methods (CQL, REM) attains the best final and average performance across the 17 Atari games tested on, directly improving upon prior methods across all the datasets. When DR3 used conjunction with REM, it prevents severe unlearning and performance degradation with more training. CQL + DR3 improves by 20 % over CQL on final performance and attains 25% better average performance. While DR3 is not unequivocally "stable", as its performance also degrades relative to the peak it achieves (Figure 4), it is more stable relative to base offline RL algorithms. We also compare DR3 to the srank( (b) penalty proposed to counter rank collapse [28]. Directly taking median normalized score improvements reported by Kumar et al. [28), CQL + DR3 improves by over 2x (31.5% over naïve CQL relative to the srank penalty (14.1% %), indicating DR3's efficacy.
14),,)
Offiin RL on robotic manipulation from images. Next, we aim to evaluate the efficacy of DR3 on two image-based robotic manipulation tasks [38] (visual- ized on the right) that require composition of skills (e.g..
opening drawer, closing a drawer, picking an obstruc- tive object. placing an object, etc.) over extended horizons using only a sparse 0-1 reward. As shown in Figure 3, combining DR3 with COG improves over COG.
Offline RL on D4RL tasks. Finally, we evaluate DR3 in conjunction with CQL on the antmaze-v2 domain in D4RL [18]. To assess if DR3 is stable and able to prevent unlearning that eventually appears in CQL, we trained CQL+DR3 for 9x longer: 2M and 3M steps with 3x higher learning rate. This is different from prior works [18] that report performance at the end of 1M steps. Observe in Table 2. that CQL + DR3 outperforms CQL (statistical significance shown in Appendix A.8), indicating that DR3 significantly improves CQL. We also evalute DR3 on kitchen domains in D4RL in Appendix A.8, where we also find that DR3 improves CQL. Finally, we also compare CQL+DR3 and CQL. in terms of performance and stability on MuloCo tasks previously studied in Kumar et al.
8

---

Published as a conference paper at ICLR 2022
Table 2: Performance of CQL, CQL DR3 after 2M and 3M gradient steps with a learning rate of
3e-4 for the Q-function averaged over 3 seeds. This is training for 6x and 9x longer compared to CQL defaults. Observe that CQL + DR3 outperforms CQL at 2M and 3M steps. indicating is efficacy in preventing unlearning. We present the statistical significance of these results in Appendix A.8.
D4RL Task
CQL (2M)
CQL DR3 (2M)
CQL (3M)
CQL + DR3 (3M)
antmaze-umaze-
84.00 2.67
85.33 + 4.16
87.00 + 1.73
90.00 4.00
verse-vv
45.67 + 8.50
40.67 + 11.84
36,33 + 7.09
52.00 +: 11.26
aormaze-medidmm-llyyy-a
24.00 28.16
73,00 1 4.00
16,00 26.85
71.33 1.52
uctmare-mediimm-dieer
32.67 + 9.29
67.00 + 2.00
48.33 6.11
61.67 = 3.21
uctmaze-largepplat
3.33 2.51
28,00 = 4.35
0.33 + 0.57
26.33 11.93
lntmaeelaargedd
1.33 2.30
25.67 - 0.57
0.00 + 0.00
28.33 = 1.52
[28] in Appendix A.3. These tasks are constructed by uniformly subsampling transitions from the full-replay-v2 MuloCo datasets in D4RL and are much harder than the typical Gym-MuloCo tasks from Fu et al. [18] because succeeding
these tasks critically relies on estimating accurate Q- values for out-of-sample actions and all actions at certain states are out-of-sample. As shown in Appendix A.3, CQL+DR3 is significantly more stable, and does not unlearn with more training, unlike CQL whose performance degrades very quickly. We also evaluate DR3 in conjunction with BRAC [49]. a policy constraint method, and find that BRAC+DR3 improves over BRAC in 13.8 median normalized performance (Table F.2).
DR3 does not suffer from rank collapse. Prior work [28] has shown that implicit regularization can lead to a rank collapse issue in TD-learning, preventing Q-networks from using full capacity. To see DR3 addresses the rank collapse issue, we follow Kumar et al. [28] and plot the effec- tive rank of learned features with DR3 in Figure 5 (DQN, REM in Appendix A.4). While the value of the effective rank decreases during training with naive bootstrapping,
Figure 5: Trend of effective rank, (T) of features learned by the Q-function when trained with TD error (red, "Without DR3") and with TD er- ror + DR3 (blue, "With DR3") on three Atari games using the 5% dataset. Note that DR3 alleviates rank collapse, without explicitly aiming to.
find that rank of DR3 fea- tures typically does not collapse, despite no ex- plicit term encouraging this. Finally, we test the robusiness/sensiiivity of each layer in the learned Q-network to re- re-initialization [52] during training and find that DR3 alters the features to behave similarly to supervised learning (Figure A.2).
Comparing explicit regularizers for different choices of noise covariance M. Finally, we inves- tigate the behavior of different implicit regularizers derived via two choices of M in Equation and the corresponding explicit regularizers. While the ex-
COL + DR3 (labal naise)
COL DRS
DON DRO (lawel ralee)
CQL
DQN
Breakout
Seaquest
8000
Averagee Return
200
6000
plicit regularizer we use in practice is a simplifying choice that works well, another choice of M is the covariance matrix induced by label noise. which re-
4000
100
2000
quires explicit computation of E Observe in Fig- ure 6 that the explicit regularizer for our simplify- ing choice is not worse than the different choice of M. This justifies utilizing our simplified, heuristic choice of setting Ei, / in practice. Results on five Atari games are shown in Appendix A.S.
20
Gradiant Updates (a
Gradient Updatos
Figure 6: Comparing DR3 regularizers for simplifying choice of M and M induced by la- bel noise, with base CQL and DQN algorithms. Note that both of these penalties when applied over CQL improve performance.
Discussion. We characterized the implicit prefer- ence of TD-lcarning towards solutions that maximally co-adapt gradients (or features) at consecutive state-action tuples that appear in Bellman backup. This regularization effect is exacerbated when out-of-sample state-action samples are used for the Bellman backup and it can lead to poor policy performance. Inspired by the theory, we propose a practical explicit regularizer, DR3, that yields substantial improvements in stability and performance on a wide range of offline RL problems. We believe that understanding the learning dynamics of deep Q-learning will lead to more robust and stable deep RL algorithms and enable predicting such instability issues, well in advance, which can inspire cross cross-validane validation and model selection strategies. This is an important, open challenge in offline RI. for which existing off- off-policy evaluation techniques are not practically sufficient [19].

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENTS
We thank Dibya Ghosh, Xinyang Geng, Dale Schuurmans, Marc Bellemare, Pablo Castro and Ofir Nachum for informative discussions, discussions on experimental setup and for providing feedback on an early version of this paper. We thank the members of RAIL at UC Berkeley for their support and suggestions. We thank anonymous reviewers for feedback on an early version of this paper. This research is funded in part by the DARPA Assured Autonomy Program and in part, by compute resources from Microsoft Azure and Google Cloud. TM acknowledges support of Google Faculty Award, NSF IIS 2045685, the Sloan fellowship, and JD.com.
REFERENCES
[1] Joshua Achiam, Ethan Knight, and Pieter Abbeel. Towards characterizing divergence in deep q-learning. arXiv preprint orXiv:/903.006 2019.
[2] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. International Conference on Machine Learning (ICML), 2020.
[3] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Belle- mare. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems, 2021.
[4] Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparameterizatton. arXiv preprint arXiv: /80.00669,, 2018.
[5] Emmanuel Bengio, Joelle Pineau, and Doina Precup. Interference and generalization in tem- poral difference learning. arXiv preprint 2rXiv:2003.06330 2020.
[6] Guy Blane, Neha Gupta, Gregory Valiant, and Paul Valiant. Implicit regularization for deep neural networks driven by an ornstein mmsttin-uhlenneckk like process. In Conference on learning theory, pp. 483-513. PMLR, 2020.
[7] Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Nazanin Mohammadi Sepahvand, Edward Raff, Kanika Madan, Vikram Voleti, et al. Accounting for variance in machine learning benchmarks. Proceedings of Machine Learning and Systems, 2021.
[8] Niladri Chatterji, Behnam Neyshabur, and Hanie Sedghi. The intriguing role of module criticality in the generalization of deep networks. arXiv preprint arXiv: 2019.
[9] Jinglin Chen and Nan Jiang. Information-theoreiicc considerations in batch reinforcement learn- ing. ICML, 2019.
[10] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. arXiv preprint arXiv:2011. 10566, 2020.
[11] Alex Damian, Tengyu Ma, and Jason Lee. Label noise sgd provably prefers flat global mini- mizers. arXiv preprint arXiv:2 06530, 2021.
[12] Daniela Pucci De Farias. The linear programming approach to approximate dynamic program- ming: Theory and application. PhD thesis, 2002.
[13] Yaqi Duan, Zeyu Jia, and Mengdi Wang. Minimax- opttiaal off-policy evaluation with linear function approximation. In International Conference on Machine Learning, pp. 2701-2709. PMLR, 2020.
[14] Ishan Durugkar and Peter Stone. Td learning with constrained gradients. 2018.
[15] Amir-massoud Farahmand, Csaba Szepesvári, and Rémi Munos. Error propagation for ap- proximate policy and value iteration. In Advances in Neural Information Processing Systems (NIPS), 2010.
10

---

Published as a conference paper at ICLR 2022
[16] William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney. Revisiting fundamentals of experience replay. arXiv preprint a7Xiv:2007.06770, 2020.
[17] Justin Fu, Aviral Kumar, Matthew Soh, and Sergey Levine. Diagnosing bottlenecks in deep Qllaamnin algorithms. arXiv preprint arXiv:190..102555 2019.
[18] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
[19] Justin Fu, Mohammad Norouzi, Ofrr Nachum, George Tucker, ziyu wang, Alexander Novikov, Mengjiao Yang. Michael R Zhang, Yutian Chen, Aviral Kumar, Cosmin Paduraru. Sergey Levine, and Thomas Paine. Benchmarks for deep oollcc evaluation. International Con- ference Learning Representations, 2021. URL
iddkWWeeEeHvFF.
[20] Dibya Ghosh and Mare G Bellemare. Representations for stable off off-policy reinforcement learning. arXiv preprint arXiv:2007.05520, 2020.
[21] Jean-Base Bant-Batieen Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Ghesh- laghi Azar, al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733, 2020.
[22] Caglar Gulcehre, Ziyu Wang. Alexander Novikov, Tom Le Paine, Sergio Gómez Colmenarejo, Konrad Zolna, Rishabh Agarwal, Josh Merel, Daniel Mankowitz, Cosmin Paduraru, et al. RI unplugged: Benchmarks for offline reinforcement learning. 2020.
[23] Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Implicit regularization in matrix factorization. In Advances in Neural Information Processing Systems, pp. 6151-6159, 2017.
[24] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. CoRR, abs/1801.01290, 2018. URL / org/ abs/1801..0 01290.
01290.
[25] Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernl:: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems 31. 2018.
[26] Aviral Kumar, Abhishek Gupta, and Sergey Levine. Discor: Corrective feedback in reinforce- ment learning via distribution correction. arXiv preprint arXiv:2003.07300 2020.
[27] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint WwXiv:2006.04779 2020.
[28] Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit under- parameterization inhibits data-efficient deep reinforcement learning. In International Confer- Learning Representations, 2021. URL https: http:///pppeeeeeeeeeeee/e ntt.oorummtttttm
id=09bnihsFfxe
[29] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
[30] Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards explaining the regularization effect of initial large learning rate in training neural networks. In Advances in Neural Information Processing Systems, pp. 11674-1668. 2019.
[31] Hamid R. Maei, Csaba Szepesvári, Shalabh Bhatnagar, Doina Precup. David Silver, and Richard S. Sutton. Convergent temporal-ding cemporal-difference learning with arbitrary smooth function approximation. In Proceedings of the 22nd International Conference on Neural Information Processing Systems, 2009.
11

---

Published as a conference paper at ICLR 2022
[32] A Rupam Mahmood, Huizhen Yu, Martha White, and Richard S Sutton. Emphatic temporal- difference learning. arXiv preprint arXiv: 1FX.01507.00569, 2015.
[33] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep rein- forcement learning. Nature, 518(7540):529-5333 533, feb 2015. ISSN 0028-0836.
[34] Rotem Mulayoff and Tomer Michaeli. Unique properties of flat minima in deep networks. In International Conference on Machine Learning, pp. 7108-7118, PMLR, 2020.
[35] Tobias Pohlen, Bilal Piot, Todd Hester, Mohammad Gheshlaghi Azar, Dan Horgan, David Budden, Gabriel Barth-Maron, Hado Van Hasselt, John Quan, Mel Večerík, et al. Observe and look further: Achieving consistent performance on atari. arXiv preprint arXiv 11593, 2018.
[36] Martin L Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., 1994.
[37] Martin Riedmiller. Neural fitted q iteration-first experiences with data efficient neural re- inforcement learning method. European Conference on Machine Learning, pp. 317-328. Springer, 2005.
[38] Avi Singh, Albert Yu, Jonathan Yang, Jesse Zhang, Aviral Kumar, and Sergey Levine. Cog: Connecting new skills to past experience with offline reinforcement learning. arXiv preprint arXiv:2010.144 14500, 2020.
[39] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. Second edition, 2018
[40] Richard S. Sutton, A. Rupam Mahmood, and Martha White. An emphatic approach to the problem of off- off-policy remporal-difference learning. Mach. Learn. Res., 17(1):26032631, January 2016. ISSN 1532-4435.
[41] Yuandong Tian, Lantao Yu, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning with dual deep networks. arXiv preprint arXiv:2010.00578, 2020.
[42] Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dy- namies without contrastive pairs. arXiv preprint arXiv:2 2021.
[43] Hado van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil. Deep reinforcement learning and the deadly triad. ArXiv, abs/1812.02648,, 2018.
[44] Hado Van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil. Deep reinforcement learning and the deadly triad. arXiv preprint arXiv: 1 02648, 2018.
[45] Ruosong Wang, Dean Foster, and Sham M. Kakade. What are the statistical limits of offline [rl] with linear function approximation? In International Conference on Learning Representations, 2021. URL /tttps://openzeeee wttp:rrwwwwwi..eeeeeeeeeeee wttp://oorornidd0000/-eee---ee-e
[46] Ruosong Wang, Yifan Wu, Ruslan Salakhutdinov, and Sham M Kakade. Instabilities of offline rl with pre-trained neural representation. arXiv preprint arXiv:2103.0449 2021.
[47] Colin Wei, Jason Lee, Qiang Liu, and Tengyu Ma. Regularization matters: Generalization and optimization of neural nets vs their induced kernel. 2019.
[48] Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. arXiv preprint arXiv:2002.09277 2020.
[49] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learn- ing. arXiv preprint 1911 2019.
12

---

Published as a conference paper at ICLR 2022
[50] Tengyang Xie and Nan Jiang. Qo approximation schemes for batch reinforcement learning: A coretical comparison. 2020.
[51] Andrea Zanette. Exponential lower bounds for batch reinforcement learning: Batch rl can be exponentially harder than online rl. arXiv preprint arXiv:2012.08000 2020.
[52] Chiyuan Zhang, Samy Bengio, and Yoram Singer. Are all layers created equal? arXiv preprint 102119990..0199, 2019.
13

---

Published as a conference paper at ICLR 2022
Appendices
A ADDITIONAL VISUALIZATIONS AND EXPERIMENTS FOR DR3
In this section, we provide visualizations and diagnostic experiments evaluating various aspects of feature co-adaptation and the DR3 regularizer. We first provide more empirical evidence showing the presence of feature co-adaptation in modern deep offline RL algorithmss. We will also visualize DR3 inspired from the implicit regularizer term TD-learning alleviates rank collapse discussed in Kumar et al. [28]. We will compare the efficacies of the explicit regularizer induced for different choices of the noise covariance matrix M (Equation 4), understand the effect of dropping the stop gradient term in ou practical regularizer and finally, perform diagnostic experiments visualizing if the Q-networks learned with DR3 resemble more like neural networks trained via supervised learning, measured in terms of sensitivity and robustness to layer reinitialization [52].
A.1 MORE EMPIRICAL EVIDENCE OF FEATURE CO-ADAPTATION
In this section, we provide more empirical evidence demonstrating the existence of the feature co- adaptation issue in modern offline RL algorithms such as DQN and CQL. As shown below in Fig- ure A. while the average dataset Q-value for both CQL and DQN exhibit flatline trend, the dot product similarity for consecutive state-action tuples generally continues to increase throughout training and does not flatline. While DQN eventually diverges in Seaquest, the dot products increase with more gradient steps even before divergence starts to appear.
COL
DQN
Del Product Simtarity
Asterix
Breekout
Pong
Q'Bert
Seaquest
6000
1000
400
4000
1000
4000
2000
500
2000
5000
0
0
D-
0
60
100
0
50
100
50
100
50
100
0
50
100
Gradient Updales (x 62.5k)
Gradient Updales 82.5k)
Gradient Updatees 62.5k)
Gradient Updales (x 62 5k)
Gradient Updales 62.5k)
COL
DQN
Asterix
Breekout
Pong
QrBert
Seaquest
Average o- oovvauss
40
20-
200-
5
20
100-
0
0
100 200
0
100
200
0
100
200
100
200
100
200
Gradient Updatos (x 62.5K) Gradient Updates (x 62.5k)
Gradient Updatos 62.5k)
Gradiant Updalas (x 62.5k)
Gradient Updates (x 82.5k)
Figure A.I: Demonstrating feature co-adaptation on five Atari games with standard offline DQN and CQL, averaged over seeds. Observe that the feature dot products continue to rise with more training for both CQL and DQN, indicating the presence of co-adaptation. On the other hand, the average Q-values exhibit a converged trend, except on Seaquest. Further, note that the dot products continue to increase for CQL even though CQL explicitly corrects for out-of-distributis action inputs.
A.2 LAYER- LAYER-WISE STRUCTURE OF A Q-NETWORK TRAINED WITH DR3
To understand if DR3 indeed makes Q-networks behave as if they were trained via supervised learn- ing, utilizing the empirical analysis tools from Zhang al. [52]. we test the robusiness/eensiiivity of each layer in the learned network to re-initialization, while keeping the other layers fixed. This tests if particular layer is critical to the predictions of the learned neural network and enables us to reason about generalization properties [52, 8]. We ran CQL and REM and saved all the intermediate checkpoints. Then, as shown in Figure A.2, we first loaded a checkpoint (x-axis), and computed policy performance (shaded color; colorbar) by re-initializing given layer (y-axis) of the network to its initialization value before training for the same run.
Note in Figure A.2, that while almost all layers are absolutely critical for the base CQL algorithm, utilizing DR3 substantially reduces sensitivity to the latter layers in the Q-network over the course of training. This is similar to what Zhang et al. [52] observed for supervised learning, where the initial
14

---

Published as a conference paper at ICLR 2022
Seequeet
Breakout
Performance extremely sensitive to most layers, especially the latter layers.
CQL
Chackpont
Chackpaend
Sueeppool
Breekcout
CQL + DR3
Performance seneilive most leyers, especially the last layar.
30
Chockpoint
Bresiouu
12
REM
Performance quite senallive to the last leyer
10 90 50100 Checliudit
1022 X 00-70 30/30/0
Checkpotre
REM + DR3
Performenca the last layor.
30-4 50
Checkpolit
Figure A.2: CQL CQL DR3 and REM vs REM DR3. Average robustness of the learned Q- function to re-initializaaiion of all layers different checkpoints the course of training created based the protocol from Zhang et al. [52]. The colors in the heatmap indicate performance of the reinitialized checkpoint, normalized w.r.t the checkpoint without any change to layers. Note that while CQL and REM are more sensitive (i.e.. less robust) to reinitialization of all the layers especially the last layer, CQL + DR3 and REM + DR3 behave closer to supervised learning, in the sense that they are more robust to reinitialization of layers of the network, especially the last layer.
layers of network were the most critical, and the latter layers primarily performed near-random transformations without affecting the performance of the network. This indicates that utilizing DR3 alters the internal layers of Q-network trained with TD to behave closer to supervised learning.
A.3 RESULTS ON MuJoCo DOMAINS
In this section, provide the results of applying DR3 the MuloCo tasks shown in Figure A.5. (Appendix A) of Kumar et al. [28]. To briefly describe the setup, in these tasks we train on the three gym tasks (Hopper-v2, Ant-v2, Walker2d-v2) using 20% of the offline data, uniformly subsampled from the run of an online SAC agent, mimicking the setup from Kumar et al. [28]. Rather than retraining an SAC agent to collect data, we subsampled the Gym-MuJoCo www.ffulllla-aaaaaaa replay buffers from the latest D4RL [18]. In these cases we plot the srank, values, the feature dot products and the corresponding performance values with and without the DR3 regularizer for 4M steps (Kumar et al. [28] showed their plots for just under 4M steps) in Figure A.3.
Observe Figure A.3, that while the standard CQL algorithm performs poorly and suffers from performance degradation within about 1M-1.5M steps for Walker2d and Ant. CQL + DR3 is able to prevent the performance degradation and trains stably. Base CQL demonstrates oscillatory per- formance on Hopper, but CQL DR3 stabilizes the performace of CQL. This indicates that DR3 is effective on MuJoCo domains, and prevents the instabilities with CQL.
For details, the weight the CQL regularizer in this case is equal to 5.0 across all the tasks, and weight on the DR3 regularizer 0.01. We also attempted to tune the CQL coefficient for the baseline CQL algorithm within (1.0, 2.0, 5.0, 10.0, 20.0) to if it address the performance degradation
15

---

Published as a conference paper at ICLR 2022
Hopper-full-raphee 12 (20%)
wallewrad-full-eplaa--- (20%)
ant-full- 1a (20%)
D4RL Normalized Retum
120
100
100
100
80
60-
COL
COL
60
COL DR3 (Ours)
50-
COL DR3 (Ouns)
40
20
20
COL
-20
100
200
300
400
500
100
200
300
400
500
100
200
300
400
500
Environment Steps (x10k)
Environment Staps (x10k)
Enviranment Staps (x10k)
hopper rell-a-
1ed
walkerr26-ull-reppayyz2
1ec
ant-fulllreplay-w2
Feature doR products
25000
2.5
COL DR3 (Ours!
COL
COL
DR3 (Ours)
2.0
EqU
(Duns)
10000
5000
0.5
0.0
100
200
200
300
300
Steps (x10k)
Environment Steps (x10k)
Steps
hopper-full-replayyy (20%)
walker2d walkerzzd-iil-repplaay (20%)
ant-fullreepayyw22 (20%)
COL
100
(Qurs)
180
srank.00)
140
140
120
100
DR3 [Ours]
DR3 (Ours)
120
100
200
300
100 200
300
1000
2.00
300
400
500
Environment Steps (x10k)
Environment (x10k)
Environment Steps (x300)
Comparison of CQL and CQL + DR3 the offfine MuJoCo Gym domains, mimicking the setup of Kumar et al. [28]. The data is generated by randomly sampling 20% of the transitions of the D4RL [18] full-replay-22 datasets, which are collected via run of an online SAC agent. The performance shown in terms of the D4RL normalized score, where 0.0 denotes the performance of random policy and 100.0 denotes the performance of an expert online SAC policy. Observe that adding DR3 stabilizes the performance on Hopper, and prevents performance collapse on Walker2d and Ant. In addition note that the srank values attained by CQL + DR3 higher than base CQL and more importantly, the feature dot products are much smaller CQL. + DR3 compared to CQL..
Figure A.3:
issues, but did not find any difference in the collapsing behavior of base CQL. Our CQL bascline is therefore well-tuned, and DR3 improves the performance over this baseline.
A.4 RANK COLLAPSE IS ALLEVIATED WITH DR3
Asteris DRU
Suuqueat
Breakout
Pong
Qbert With 2013
560
With DR1
With D13
302
300
Wideost
210
151
200
120
1o
120
110
110
22
Updatac (o62.5k)
Updates
Figure A.4: Comparing the feature ranks for CQL and CQL DR3. Observe that utilizing DR3 success- fully alleviates the rank collapse issue noted in prior work without explicitly correcting for it.
Prior work [28] has shown that implicit regularization in TD-Jearning can lead to a feature rank collapse phenomenon in the Q-function, which hinders the Q-function from using its full represen- tational capacity. Such a phenomenon absent in supervised learning, where the feature rank does not collapse. Since DR3 is inspired by mitigating the effects of the term in the implicit regular- izer (Equation 4) that only appears in the case of TD-leamiing we wish to understand if utilizing DR3 also alleviates rank collapse. To do so, compute the effective rank sranks (Ø) metric of the features learned by Q-functions trained via CQL and CQL with DR3 explicit regularizer. As shown in Figure A.4, for the case of five Atari games, utilizing DR3 alleviates the rank collapse
16

---

Published as a conference paper at ICLR 2022
issue completely (i.e., the ranks do not collapse to very small values when CQL+ DR3 is trained for long). We do not claim that the ranks with DR3 are necessarily higher, and infact as we show below, higher srank of features may not always imply better solution. The fact that DR3 can prevent rank collapse is potentially surprising, because no term in the practical DR3 regularizer explicitly aims increase rank: feature dot products can be made smaller while retaining low ranks by simply rescaling the feature vectors. But, as we observe, utilizing DR3 enables learning features that do not exhibit collapsed ranks, thus we hypothesize that correcting for appropriate terms in (!) can address some of the previously observed pathologies in TD-learning.
Asteric
Brookout
Bert
Seoquest
Space Invaders
3000-
600.
1300-
500-
2000
1300-
4030
200-
1300
200
2030
300
$2.94
Gradient 12.50
Lipiliia (2.54)
Asterix
Breakout
Obert
Seaquest
Spacelivaders
450
1:0
390
220
3X
128
DON
DON DR3
250
DGN
DIN DD3
332
200
DGN
DON - DR3
DON DON + DR3
DON
DON + D33
26 180 355 25c
26
120
15. 250
220 125 200
59
320 326 244
53
332 353 20
Updaten
Updates
Upditees
Updates
Gradient Updates
Figure A.5: Performance and srank values for DQN and DQN DR3. Observe that the srank values increase for DQN DR3, while they collapse for DQN on Asterix, Seaquest and Spacelnvaders with training. Thus, DQN + DR3 does not suffer from sudden rank collapse. However, higher srank does not imply better return, and so while initially DQN does have a high rank, DQN + DR3 performs superiorly.
We now investigate the feature ranks of Q-network trained when DR3 is applied in conjunction with a standard DQN and REM [2] on the Atari domains. We plot the values of sranks (Ø). the feature dot products and the performance of the algorithm for DQN in Figure A.5 and for REM in Figure A.6. the case of DQN, we find that unlike the base DQN algorithm for which feature rank does begin to collapse with more training, the srank for DQN + DR3 is increasing. We also note that DQN+ + DR3 attains a better performance compared to DQN, throughout training.
REM
Astori
Breakoot
Or
Scoquesi
Space Imvders
10000-
Seger
3300
1200
30
300-0
1000
5030
150
150
250
100
200
160
150
250
50
130
130
200
53
200
Asterbx
Breakout
Obert
Seaquest
Spaceliwaders
45.0
wwwkko00..0 0.01
020
252-
236
242-
350
259
200
REM
D63
230
RE
REM
D83
230
REM REM4
132
209
REM
250
25- 200
Gradient Updates (4 02.5k)
Gradient Updates Oe 62.5k)
Gradient Updates (x 62.
Gradient Updates (x Sk)
Gradient Updates Ie 62.5k)
Comparing the performance and srank values for REM and REM + DR3. Observe that while REM + DR3 outperforms REM, the srank values attained by REM are much larger than REM + DR3, and none of these ranks have collapsed. Thus, while REM DR3 maintains non-collapsed features, for the case of REM, it reduces the value of srank and attains better performance. This does not contradict the observations from Kumar al. [28] as we discuss in the text.
Figure A.6:
17

---

Published as a conference paper at ICLR 2022
Table A.1: Normalized interquartile mean performance with 95% stratified bootstrap CIs [3] across 17 Atari games of REM. REM A' (Stop gradient in DR3), REM DR3 after 6.5M gradient steps for the 1% setting and 12.5M gradient steps for the 5%, 10% settings. Observe that REM (Ø) also improves over the base REM method significantly, by about 130%, even though A (Ø) is generally comparable and somewhat worse than the DR3 regularizer used in the main paper.
Data
REM
REM + (2)
REM+DR3
4.0 (3.3, 4.8)
15.0 (13.4, 16.6)
16.5 (14.5, 18.6)
5%
25.9 (23.4, 28.8)
55.5 (50.8, 59.8)
60.2 (55.8, 65.1)
10%
53.3 (51.4, 55.3)
67.7 (64.7,
73.8 (69.3. 78)
However, we note that the opposite trend is true for the case of REM: while REM + DR3 attains a better performance than REM, adding DR3 leads to a reduction in the srank value compared to base REM. At first glance, this might seem contradicting Kumar et al. [28], but this is not the case: to our understanding, Kumar et al. [28] establish a correlation between extremely low rank values (i.e., rank collapse) and poor performance, but this does not mean that all high rank features will lead to good performance. We suspect that since REM trains a multi-headed Q-function with shared features and randomized target values, it is able to preserve high-rank features, but this need not mean that these features are useful. In fact, as shown in Figure A.7, we find that the base REM algorithm does exhibit feature co-adaptation. This case is an example where the srank metric from Kumar et al. [28] may not indicate poor performance.
DR3
Dot Product Similmy
Asteni
Breakiout
Q"Bert
Seaquest
Invaders
10"
10
10'
10
10"
10
10"
10²
10'
10'
10
10'
100
200
0
100
200
100
200
0
100
200
100
200
Gradiant Updatos (x 62 62)5k)
Gradient Updates (x 62.5k)
Gradiont Updates (x 62 (2.5k)
Gradient Updatas (x 62.5k)
Gradiont Updates (x 62.5k)
Figure A.7: Feature dot products for REM and REM DR3 on log scale. REM does suffer from feature co-adaptation despite high-rank features.
A.5 INDUCED IMPLICIT REGULARIZER: THEORY AND PRACTICE
In this section, we compare the performance of our practical DR3 regularizer to the regularizers (Equation 4) obtained for different choices of M, such as M induced by noise, studied in previous work, and also evaluate the effect of dropping the stop gradient function from the practical version of our regularizer.
Empirically comparing the explicit regularizers for different noise covariance matrices, M. The theoretically derived regularizer (Equation 4) suggests that for given choice of M, the follow- ing equivalent of feature dot products should increase over the course of training:
(m))) = Σ trace [Ei (Q,s, VQ('s,a)))) (Generalized dot products) (A.1)
S,0D
We evaluate the efficacy of the explicit regularizer that penalizes the generalized dot products, A (8), in improving the performance of the policy, with the goal of identifying if our practical method performs similar to this regularizer on generalized dot products.. While Σi, must be explic- itly computed by running fixed point iteration for every parameter iterate 0 found during TD-learning which makes this method significantly computationally expensive", we evaluated it on five Atari games for 50 x 62.5k gradient steps as a proof of concept. As shown in Figure A.8, the DR3 penalty with the choice of M which corresponds to label noise, and the dot product DR3 penalty, which is our main practical approach in this paper generally perform similarly on these domains, attaining almost identical learning curves on 4/5 games, and clearly improving hints at the possibility of utilizing other noise covariance matrices to derive an explicit regularizer.
the base algorithm. This
II our implementation, we run 20 steps of the fixed- point computation of Y as shown in Theorem 3.1 for each gradient step the Q-function, and this increases the runtime to about 8 days for 50 iterations on a P100 GPU.
18

---

Published as a conference paper at ICLR 2022
Asterix
Breakout
Pong
Q'Bert
Seequeest
4000
Raweon Datum
4000-
10
10000
0-
20n0-
100
10
2000
-20
20
40
20
40
20
40
20 Grod Updistes
Goudiett Updaston 5kc
Gredient Upstnter (r 80 5a)
Crodert Uptiress (c
Goadhert Updatio (a
Figure A.8: Comparing the performance of explicit penalties for two different choices of covariance matrix M. Observe that in all the five games the DR3 regularizer derived for the choice of M from Blanc et al. [6] also leads to substantial increase in performance over the base algorithm, and in four of five games, DR3 (label-noise) works just well DR3.
Deriving more computationally efficient versions of the regularizer for general M and identifying the best choice of M are subject to future work.
Effect of stop gradient. Finally, we investigate the effect of utilizing a stop gradient in the DR3 regularizer. We run a variant of DR3: (o) = Ex... ő(s a) [[(s', a ']]]] with the stop gradient on the second term (s', a') and present comparison to the one without the stop gradient in Table A.1 for REM as the base offline method, averaged over 17 games. Note that this version of DR3, with the stop gradient, also improves upon the baseline offline RL method (i.e., REM) by 130% %. While this performs largely similar, but somewhat worse than the complete version without the stop gradient, these results do indicate that utilizing A (Ø) can also lead to significant gains in performance.
A.6 UNDERSTANDING FEATURE CO-ADAPTATION SOME MORE
In this section, present some more empirical evidence to understand feature co-adaptation. The three factors we wish to study are: (1) the effect of target update frequency on feature co-adaptation; (2) understand the trend in normalized similarities and compare these the trend in dot products; and (3) understand the effect of out-on-sample actions in TD-learning and compare it to offline SARSA on a simpler gridworld domain. We answer these questions one by one via experiments aiming to verify each hypothesis.
A.6.1 EFFECT OF TARGET UPDATE FREQUENCY ON FEATURE CO-ADAPTATION
We studied the effect of target update frequency on feature co-adaptation, on some gridworld domains from Fu et al. [17]. We utilized the gridl 6smoothobs environment, where the goal of the agent is to nav-
grid16smoothobs Standard FQI
grid16smoothobs CQL
Feature dex products (leg)
30'
N-5
Feature det products (og)
100
M-500 N=
207
3.0
igate from the center of 16 x 16 1.0
10
Ne5
N-10
gridworld maze to one its corners while avoiding obstacles and "lava" cells. The observations provided to the RL algorithm are given by high- dimensional random transformation of the (x. v) coordinates, smoothed over neighboring cells in the grid- world. We sampled an offline dataset of 256 transitions and trained Q- network with two hidden layers of size (1024, 1024) via fitted Q-iteration (FQI) [37].
10
No-200
10a
Ni-500
50
100
15.0
200
250
10a
160
150
200
250
Gradiant Updates
Gradient Updates
Figure A.9: Comparing the feature dot products for various target update delays. where smaller implies a faster update and larger corresponds to a slower target update. Observe that while slower updates to the target network may reduce co- adaptation, very slow target updates may still lead to excesssive co-adaptation.
We evaluated the feature dot products for Q-functions trained with a varying target update frequen- cies, given generically as: updating the target network using a hard target update once per N gradient steps, where N takes values N 5, 10, 50, 100, 200, 500, and present the results in Figure A.9 (left), averaged over random seeds. Thee feature dot products initially decrease from N = 5 to N 10, because the target network is updated slower, but then starts to rapidly increase when when the target network is slowed down further to 50 and N = 200 in one case and N = 500 in the other case.
19

---

Published as a conference paper at ICLR 2022
We also evaluated the feature dot products when using CQL the base offline RL algorithm. As shown in Figure A.9 (right), while CQL does reduce the absolute range of the feature dot products, slow target updates with N = 500 still lead to the highest feature dot products training progresses. Takeaway: While it is intuitive to think that a slower target network might alleviate co-adaptation, see that this is not the case empirically with both FQI and CQL, suggesting deeper question that is an interesting avenue for future study.
A.6.2 GRIDWORLD EXPERIMENTS COMPARING TD-LEARNING VS OFFLINE SARSA
To supplement the analysis in Section 3.1, we ran some experiments in the gridworld domains from Fu et al. [17]. In this case, we used the gr priddl Samoothsparde and gr gridls 6 Grandomsparse domains, which present challenging navigation tasks in a maze under a 0-1 sparse reward signal, provided at the end of the trajectory. Additionally, the observations available to the offline RI. agent do not consist of the raw (xt y) locations of the agent in the maze, but rather high-dimensional randomly chosen transformations of (x, y) in the case of gridl Srandomsparses which are additionally smoothed locally around a particular state to obtain gridl www.otssssarstt
Since our goal compare feature co-adaptation in TD-leaming and offline SARSA, we consider case where we evaluate a "mixed" behavior policy that chooses the optimal action with a probability of 0.7 at a given state, and chooses a random, suboptimal action with 0.3. We then generate a dataset of size 256 transitions and train offline SARSA and TD-Icarning on this data. While SARSA backups the next action observed in the offline dataset, TD-learning computes a full expectation of the Q-function Eg ( st) [Q(s' a')] under the behavior policy for computing Bellman backup targets. The behavior policy is fully known to the TD-learning agent. Our Q-network consists of two hidden layers of size (1024, 1024) as before.
We present the trends in the feature dot products for TD learning and of- fline SARSA in Figure A.10, aver-
grid1esmoothsparre: TD SARSA
griditrrandomspars:: TD SARSA
TD karning
2000
TD Teariinngg
Feature dek products
4000
offinee SARSA
Feature det products
Offline SARSA
aged over three seeds. Observe that
1500
3.000
the trends in the dot product values for TD-learning and offline SARSA closely follow each other for the ini- tial few gradient steps, soon, the dot products in TD-learning start grow- ing faster. In contrast, the dot prod- ucts for SARSA either saturate or start decreasing. The only difference between TD-learning and SARSA is the set of actions used to compute Bellman targets while the actions used for computing Bellman backup targets in SARSA are in-sample actions and are observed in the dataset, the actions used by TD- learning may be out-of-sample, but are still within the distribution of the data-generating behavior policy. This supports our empirical evidence in the main paper showing that out-of-sample actions can lead to feature co-adaptation.
1000
2000
1000
50
1.00
150
200
250
50
100
150
200
250
Gradient Updates
Gradienn Updates
Figure A.10: Comparing the feature dot products for TD- learning and offline SARSA, used compute the value of the behavior policy using dataset of size 256 on two gridworld do- mains. Observe that the feature dot products are higher in the case of TDDlcarnnng compared to offline SARSA.
A.6.3 FEATURE CO-ADAPTATION AND NORMALIZED FEATURE SIMILARITIES
Note that we characterized feature co-adaptation via the dot products of features. In this section, explore the trends in other notions of similarity, such as cosine similarity between Ø(s, a) and o(s', a') which measures the dot product of feature vectors at consecutive state-action tuples after normalization. Formally,
dss a) (ss' a')
cos (os, a), d(s's(s a')) [|o(s, a)ll2 |
We plot the trend in the cosine similarity with and without DR3 for five Atari games in Figure A.11 with CQL, DQN and REM, and for the three MuloCo tasks studied in Appendix A.3 in Figure A. We find that the cosine similarity is generally very high on the Atari domains, close to I, and not indicative of performance degradation. On the Ant and Walker2d MuloCo domains, find that
20

---

Published as a conference paper at ICLR 2022
DON
DON
DR3
Astorix
Breakout
Q'Bert
Seaquest
Space Invaders
Cosine Siritrrity
1.000
1.0
1.0
1.00
1.00
0.878
0.8
0.95
0.850-
0.80
0.90
0.825
100
200
0.6
100
200
0.8-
100
200
8.85
100
200
100
200
Gradient Updatees 62 5k)
Gradient Updates 62, 82.5k)
Gradient Updates (x 62.5k) Sk)
Gradient Updates (x 5k)
Gradient Updaltes 2k.
REM
REM+ DRS
Q*Bert
Seaquest
Space invaders
Casine Simburtty
1.00
1.00
1.00
0.98
0.95
0.90
0.98
0.90
0.8
0.96
0.96
100
200
100
200
0
100
200
100
200
100
200
Gradient Updates 62.5k)
Gradient Updates (x 62.5k)
Gradient Updates 62 62.5k]
Gradient Updates 62.5k)
Gradient Updates (x 62.5k)
COL
DR3
Asterix
Breakout
Q'Bert
Seaquest
Space Inwaders
Cosine Simiartyy
1,000
1.0
1.0
1000
-1.0 1.000
0.975
0.975
0.976
0.8-
0.9
0.950
0,950
0.860
0.925
0.6
1995
0.825
100
200
0
100
200
100
200
100
200
100
200
Gradient Updates (x
Gradient Updates
Gradient Updates 62
Gradient Updates 62
Gradient Updates (x 62.5K)
Figure A.11: Cosine similarities of DQN, DQN DR3, REM, REM DR3 and CQL, CQL + DR3. Note that DQN. REM and CQL attain close cosine similarities, and addition of DR3 does reduce the cosine similaritics of consecutive state-action features.
hopper-full--eplaay-2
walker2d hull-dd-uul--rela-----
ant-full-replay-v2
Feature cosine similarity
1.00
1.00
0.98
0.95
0.96
0.90
0.94
COL
COL
(Ours)
COL
DR3
CQL
DR3 (Ours)
100
200
300
400
500
300
200
300
400
500
100
200
300
400
500
Environment Steps (x10k)
Environment Steps (x10k)
Environment Steps (x10k)
Figure A.12: Cosine similarities of CQL. and CQL + DR3 on MuJoCo domains. Note that the cosine similarities of CQL. grow to and roughly stabilize for Ant and Walker2d, but start decreasing for Hopper. This happens despite the oscillatory trends in performance of CQL Hopper A.3. This means that low cosine similarity need not imply poor performance, and DR3 can improve performance even when cosine similarity of base CQL is decreasing. We also notice that DR3 does actually reduce cosine similarity.
the cosine similarity first rises close to I and roughly saturates there. On the Hopper domain, the cosine similarity even decreases over training. However we observe that the feature dot products are increasing for all the domains. Applying DR3 in both cases improves performance (as shown in earlier Appendix), and generally gives rise to reduced cosine similarity values, though it can also increase the cosine similarity values occasionally. Furthermore, even when the cosine similarities were decreasing for the base algorithm (e.g., in the case of Hopper), addition of DR3 reduced the feature dot products and helped improve performance. This indicates that both the norm and di- rectional alignment are contributors to the co-adaptation issue, which is what DR3 aims to fix and independently directional alignment does not indicate poor performance.
A.7 STABILITY OF DR3 FROM GOOD SOLUTION
In this appendix, study the trend of CQL + DR3 when starting learning from good initialization, which was studied in Figure 2. As shown in Figure A.13, while the performance for baseline CQL degrades significantly (from 5000 at initialization on Asterix, performance degrades to ~200 by
21

---

Published as a conference paper at ICLR 2022
Astartic
Breakout
Ms. Pac Man
Seaquest
Space Invaders
300
Avaoga Bearn.
5000
250
4000
200
4000
1500
2000
150
L1
50000
100
1000
50 Liputtine
75
Figure A.13: Running CQL DR3 and CQL in the setup of Figure 2 to evaluate the stability of CQL
DR3 when starting training from good solution. Observe that performance base CQL decays quickly from the good solution, but CQL + DR3 is relatively more stable.
Astarix
Broakout
Pac-Man
Seaquest
Invadors
Acsnoga baturn
3000
e0.
2500
4000
500
3000
2000
400-
300
1000
0-
1000
50
100
25
50
75
100
50
100
25
50
75
100
100
Gradien Uptatee DX 02 SK]
Gradient Uptinoss O1 (2..04)
Grad mm Updates 02 (k)
Grodient Updates io Ok)
Gradient Updattee
setup of Figure + DR3 when starting training from a good solution. Observe that the performance of base DQN decays quickly from the good solution, but DQN + DR3 is relatively more stable.
Figure A.14: Running DQN DR3 and DQN
evaluate the stability of DQN
100 iterations for base CQL), whereas the performance of DR3 only moves from 5000 44300 A similar trend holds for Breakout. This means that the addition of DR3 does stabilize the learning relative the baseline algorithm. Please note that are not claiming that DR3 is unequivocally stable, but that improves stability relative to the base method.
A.8 STATISTICAL SIGNIFICANCE OF DR3 AND FRANKA KITCHEN RESULTS
Table A.2: Performance of CQL, CQL. + DR3 after 2M gradient steps on Franka Kitchen domains averaged over seeds. This is training for 6x longer compared to CQL defaults. Observe that CQL + DR3 outperforms CQL at 2M steps, indicating is efficacy in preventing long term performance degradation.
D4RL Task
CQL.
CQL DR3
xitchen-mixet
27.67 + 12.66
37.00 = 11.53
xitchen-partial
20.67 + 15.57
40.67 + 4.04
Mitchen-complest
28.00 + 14.73
38.67 + 6.66
We present the results comparing CQL and CQL+DR3 on the Franka Kitchen tasks from D4RL in Table A.2. Observe that CQL+DR3 outperforms CQL, and test the statistical significance of these results, we analyze the probability of improvement of CQL+DR3 over CQL next.
AntMaze
Kitchen
0.5 0.6 0.7 0.8 0.9 1.0 P[CQL + DR3 > CQL]
Figure A.15:
Statistical significance of the results of CQL DR3 vs CQL (Table as measured by average probability of improvement [3], with stratified bootstrap confidence intervals for this statistic. Since the lower CI for this statistic is > 0.5, CQL + DR3 significantly improves over base CQL, and since the mean and upper CI are 2 0.75, this improvement is also meaningful.
In order to assess the statistical significance of our D4RL Antmaze and Kitchen results, we follow the recommendedations by Agarwal al. [3] for comparing deep RL. algorithms considering their
22

---

Published as a conference paper at ICLR 2022
statistical uncertainties. Specifically, computed the average probability of improvement [3] of CQL + DR3 over CQL on the antmaze and kitchen domains, and we find that DR3 does significantly and meaningfully improve over CQL on both the Kitchen and AntMaze domains. Before presenting the results, let us first descrihe the metric we compute.
Probability of improvement and statistical significance. For two given algorithms Alg and Alg2-
and runs X1,1, X2.2.".. ... , X2.m from Alg, and runs Y2,1, Yr.2, ... Yk.n from Alg, on task k, the probability of improvement of Alg, over Algz is given by P(Alg, > Alg2) = R Ek=1 P(Alg > Alg)). The probability of improvement on given task k, P(Alg > Algs) is computed using the Mann-Whiter Whitney U-statistic and is given by:
M
N
1, if < x,
P(Alg > Alga) MN
S(XA,s YE.)) where S(r, = =1
2. y
0, if >
Alg, leads to statistically significant improve over Alg, if the lower CI for P(Alg, > Algg) is larger than 0.5. Per the Neyman-Pearse Pearson statistical testing criterion in Bouthillier et al. [7], Alg leads to statistically meaningful improvement over Alg, if the upper confidence interval (CI) of P(Alg. > Alg2) is larger than 0.75.
Figure A.15 presents the value of P(CQL+ DR3 CQL) on the AntMaze and Kitchen domains at 2M gradient steps along with the 95% CI for this statistic. DR3 improves over CQL on the AntMaze domains with probability 0.83 with 95% CI (0.7, 0.96) and on the Kitchen domains with probability 0.8 with 95% CI (0.6, 1.0). These values pass the criterion of being both statistically significant and meaningful per the above definitions, implying that DR3 does significantly and meaningfully improve upon CQL on these domains.
B RELATED WORKS
In this section, we briefly review some extended related works, and in particular, try to connect feature ao-appptationn and implicit regularization to various interesting results pertaining to RL lower- bounds with function approximation and sulf-supe learning.
B.1 BRIEF SUMMARY OF RELATED WORK
Prior analyses of the learning dynamics in RL has focused primarily on analyzing error propaga- tion in tabular or linear settings [e.g., 9, 13, 50, 45, 46, 15, 12], understanding instabilities in deep RL [1, 26, 44] and deriving weighted TD updates that enjoy convergence guarantees [31, 32, 40], but these methods do not reason about implicit regularization or any form of representation learning. Ghosh & Bellemare [20] focuses on understanding the stability of TD-learning in underparameter- ized linear settings, whereas our focus on the overparameterized setting, when optimizing TD error and learning representations via SGD. Kumar et al. [28] studies the learning dynamics of Q-learning and observes that the rank of the feature matrix, Q drops during training. While this observation is related, our analysis characterizes the implicit preference of learning towards feature co-adaptation (Theorem 3. on out-of-sample actions as the primary culprit for aliasing. Addition- ally. while the goal of our work is not to increase srank (.), utilizing DR3 not only outperforms the srank (Ø) penalty in Kumar et al. [28] by more than 100 %2 but it also alleviates rank collapse, with no apparent term that explicitly enforces high rank values. Somewhat related to DR3, Durugkar & Stone [14], Pohlen et al. [35] heuristically constrain gradients of TD to prevent changes in target Q-values to prevent divergence. Contrary to such heuristic approaches, DR3 is inspired from the- oretical model of implicit regularization, and does not prevent changes in target values, but rather reduces feature dot products.
B.2 EXTENDED RELATED WORK
Lower-bounds for offline RL. Zanette [51] identifies hard instances for offline TD learning of linear value functions when the provided features are "aliased". Note that this work does not consider feature learning implicit regularization, but their hardness result relies heavily the fact the given linear features are aliased in a special sense. Aliased features utilized the hard instance inhibit learning along certain dimensions of the feature space with TD-style updates, necessitating an
23

---

Published as a conference paper at ICLR 2022
exponential sample size for near-accurate value estimation, even under strong coverage assumptions. combination of Zanette [51]'s argument, which provides a hard instance given aliased features, and our analysis, which studies the emergence of co-adapted/similar features in the offline deep RL setting, could imply that the co-adaptation can lead to failure modes from the hard instance, even on standard Offlin RI. problems, when provided with limited data.
Connections to self-supervised learning (SSL), Several modern self-supervised learning meth- ods [21, 10] can be viewwed as utilizing some form of bootstrapping where different augmentations of the same input (x + A X + Aug,) serve as consecutive state-action tuples that appear on two sides of the backup. If we may extrapolate our reasoning of feature co-adaptation it would suggest that performing noisy updates on a self-supervvsed bootstrapping loss will give us feature representations that are highly similar for consecutive state-action tuples, i.e., the representa- tions for o(x+ Aug,) Φ(x Auga) will be high. Intuitively, an easy way for obtaining high feature dot products is for o(-) to capture only that information which is agnostic to data augmentation, thus giving rise to features that are invariant transformations. This aligns with what has been shown in self-supervised learning [41, 42]. Another interesting point to note is that while such an explanation would indicate that highly co-adapted features are beneficial in SSL, such features can be adverse in value-basee based RL as discussed in Section 3.
this setting,
Preventing divergence in deep TD-learning. Finally, we discuss Achiam et al. [1] which proposes to pre-condition the TD-update using the inverse the neural tangent kernel [25] matrix so that the TD-update is always contraction, for every 0₂ found during TDD-lerniigg Intuitively, this can be overly restrictive in several cases: we do not need to ensure that TD always contracts, but that is eventually stabilizes at good solution over long periods of running noisy TD updates, Our implicit regularizer (Equation 4) derives this condition, and our secoretically-insire. DR3 regularizer shows that empirically, it suffices penalize the dot product similarity in practice.
C PROOF OF THEOREM 3.1
In this section, we will derive our implicit regularizer (TD(!) that emerges when performing TD updates with a stochastic noise model with covariance matrix M. We first introduce our notation that we will use throughout the proof, then present our assumptions and finally derive the regularizer. Our proof utilizes the analysis techniques from Blane et al. [6] and Damian et al. [11], which ana- lyze label-noise SGD for supervised learning, however key modifications need to be made to their arguments to account for non-symmetric matrices that emerge in TD learning. As a result, the form of the resulting regularizer is very different. To keep the proof concise, will appeal to lemmas from these prior works which will allow us to bound certain concentration terms.
C.I NOTATION
The noisy TD-learning update for training the Q-function is given by:
0x+1 = -6x-0 -n Σ VoQ(s,, a.) ()) - Qr.5) thEK. EK ~ M) (C.1)
==)(
where (0) denotes the parameter update. Note that g(0) is not a full gradient of a scalar objective, but it is form of "pseudo"-gradice" or "semi""ggrdien Let en denote an r.nddo noise that is added to each update. This noise is sampled from a zero-mean Gaussian random variable with covariance matrix M, i.c., (0, MM).
Let 0 denote a point in the parameter space such that in the vicinity of 0*, g(0) < 6, for a small enough 6. Let G(#) denote the derivative of g(0) w.r.t. #: G(l) Veg(f) and let VG(()) denote the third-order tensor (0). For notation clarity, let G = G(f*). VG = VG(0"). Let denote the signed TD error for a given transition (s,, a, s() E D 8t:
e = Q8- (s., a:) (r: + 7Qe- (s/, a)))
(C.2)
Since 0* is fixed point of the training TD error, e; 0. Following Blane et al. [6]. will assume that the learning rate in gradient descent, 77. is small and we will ignore terms that scale as + for 5 > 0. Our proof will rely on using reference Onsstenn-Ullenheck (OU) process which the TD
24

---

Published as a conference paper at ICLR 2022
parameter iterates will be compared to. Let Sa denote the k-th iterate of an OU process, which is defined as:
G4+1 - - + nJE EN M)
(C.3)
We will drop from Vo indicate that the gradient is being computed 0", and drop (st, a:) from Qs, a.) and instead represent as Q; for brevity; we will represent Q(s/, a.) as Q-- We assume that Q2Q, -Liisschitzz and Q L2-Lipschitz throughout the parameter space G.
C.2 PROOF STRATEGY
For given point l* to be an attractive fixed point of TDDlearning, our proof strategy would be to derive the condition under which it mimics given OU noise process, which we will show stays close to the parameter 0" This condition would then be interpreted as the gradient of a "induced" implicit regularizer. If the point 8" is not stationary point of this regularizer, we will show that the movement A is large when running the noisy TD updates, indicating that the regularizer, atleast part guides the dynamics isolate some terms that will give rise to the implicit regularizer, and bound the remaining terms us- ing contraction and concentration arguments. The contraction arguments largely follow prior work (though with key exceptions in handling contraction with asymmetric and complex eigenvalue trices), while the form of the implicit regularizer is different. Finally, we will interpret the resulting update over large timescales to show that learning is indeed guided by the implicit regularizer.
TD-Jearning. To show this, we would write out the gradient update,
C.3 ASSUMPTIONS AND CONDITIONS
Next, we present some key assumptions we will need for the proof. Our first assumption is that matrix G E ddxd is of maximal rank possible, which is equal to the number of datapoints and n < d, the dimensionality of the parameter space. Crucially, this assumption do not imply that G is of full rank it cannot be, because
in the overparameterized regime.
Assumption A1 (G spans n-dimensional basis.). Assume that the matrix G spans n-possible directions in the parameter space and hence, attains the maximal possible rank it can.
The second condition we require is that the matrices VQNTT and M share the same n- dimensional basis matrix G:
Assumption A2. Σ, VQ.VQ, M, and G span identical n-dimensional subspaces.
This is technical condition that is required. If this condition dynamics of noisy TD will not be contraction in certain direction in the parameter space and TD- learning will not stabilize at such a solution d* In fact, we will utilize a stronger version of this statement for TD-learning to converge, and we will discuss this shortly.
met, as we will show the learning
C.4 LEMMAS USED IN THE PROOF
Next, we present some lemmas that would be useful for proving the theoretical result.
Lemma C.1 (Expressions for the first and and second-order derivatives of g(0).). The following definitions and expansions apply to our proof:
G(O*) Σ V2Qie + Σ VQ.(Ve.-7VO))
VG(0")l. = 2 VV -VQ.-VVQ + Eir ( VQ + +&eee
Lemma C. presents a decomposition of the matrix G and the directional derivative of the third order tensor VG(v, v] in directions v and v. which will appear in the Taylor expansion layer. Note that at 8* since derives a fixed -eint recursion for the covariance matrix of the total noise accumulated in the OU- process with covariance matrix M and this will appear in our proof.
0, the first term in G(Ø*) and the third term in VG [[, vanish. Lemma C.2
25

---

Published as a conference paper at ICLR 2022
Lemma C.2 (Covariance of the random noise process (t). Let (b denote the OU process satisfying: Sk+1 = GG + TJE where EK ~ M), where M i O. Then, Sk+1 ~ (O, where Σ satisfies the discrete Lyapuñov equation:
EM = (1 - (1 M.
Proof. For the OU process. Ck+1 = (I nG) + nE,, since EY is Gaussian random variable, by induction so is SE+1, and therefore the covariance matrix of CR+1 is given by:
Ex+1 == (1 - Gf)) - 7/ + M,
(C.4)
Solving for the fixed point for Ex gives the desired expression.
In our proofs, we will require the following contraction lemmas to tightly bound the magnitude of some zero-mean terms that will appear in the noisy TD update under certain scenarios. Unlike the analysis in Damian et al. [11] and Blanc et al. [6] for supervised learning with label noise, where the contraction terms like -)) G are bounded by Kn intuitively because / G is contraction in the subspace spanned by matrix G. However, this not true for TD-learnnn directly since terms like (I - n S appear for different matrix S. Therefore, TD-learnnn will diverge from 8 unless matrices G and M have their corresponding eigenvectors assigned the top eigenvalues be approximately "aligned". We formalize this definition next, and then provide proof of the concentration guarantee.
Definition ((w; ali-annigm, Given a positive semidefinite matrix A, let A= AUA denote its eigendecomposition. Without loss of generality assume that the eiegenvalues are arranged in decreasing order, i.e., Vi > j, (i) S (J). Given another matrix B, let = AgABU denote its complex eigendecomposition, where eigenvalues in AB are arranged in decreasing order of their complex magnitudes, i.e., Vi > j, [An(i)] S [An())]. Then the matrix pair (A, B) is said to be (f) for a constant Co.
(w, Cal-aligned [U (i))
and if iii) I (o)
If two matrices are (w, Co)-aligned, this means that the corresponding eigenvectors when arranged in decreasing order of eigenvalue magnitude roughly align with each other. This condition would be crucial while deriving the implicit regularizer as it will quantify the rate of contraction of certain terms that define the neighborhood that the iterates noisy TD-leanning will lie with high prob- ability. We will operate in the setting when the matrix G and Σ, VQ, are (w, Cal-aligned with each other, and matrix M and G are also (w, Co)-aligned (note that we can consider w', (%), which will not change
bounds and therefore go for less notational clutter). Next utilize this notion of alignment to show a particular contraction bound that extends the weak contraction bound in Damian et al. [11].
Lemma C.3. Assume we are given a matrix G such that (A-(I- Po < 1 for all x such that A + O. Let G = H be the complex eigenvalue decomposition of G (since almost every matrix For positive semi definiei matrix S that (w, Co) Golgannge with G, if S = USASUğ & is its eigenvalue decomposition, the following contraction bound holds:
(WCo np
|I(T - (|)=
Proof. To prove this statement, we can expand (I - GG) using its eigenvalue decomposition only in the subspace that is jointly shared by G and M, and then utilize the definition of w-alignment to bound the terms.
- n
S|| = II(T -USUUUUSAUUU
(C.5)
|| U
(C.6)
|||7 - (U"A)"""USAAUUU
(C.7)
S w. ww11 (1A) -
(C.8)
< w Co
mmxx || -)(0)))))
mmxx
(C.9)
26

---

Published as a conference paper at ICLR 2022
Now need to solve for the inner maximization term. When A(i) is not complex for any i, the term above is S 1/mp using the result from Damian et al. [11]] but when A(i) is complex, this bound can only hold under certain conditions. To note when this quantity is bounded, expand |1 nz|* for some complex number x = r(cos 0 + sin 0):
1(1 mp cos 0) + cxp sin 0
(C.10)
Vi mpr cos 0)² 122 sin?
= (1 + 1/2 2np cos 0) 4/2
(C.11)
| |2| = (1 + rj2 - 2np cos 0) k/2,,
11
(C.12)
< 1
if S min Re(A(i)) and otherwise.
(C.13)
nyk
[A(i)
Plugging back the above expression in the bound above completes the proof.
The proof of Lemma C.3 indicates that unless the learning rate 1) and the matrix G are such that the [X, r S p < in directions spanned by matrix S. such an expression may not converge. This is expected since the matrix I - 7/GG' will not contract in directions of non-zero eigenvalues if the real part r cos is negative or zero. Additionally, we note that under Definition I, we can extend several weak-contraction bounds from Damian et al. [11] (Lemmas 9-14 in Damian et al. [11]) to our setting.
Next, Lemma C.4 shows that the OU noise iterates are bounded with high probability when Defini- tion holds:
Lemma C.4 (Cx is bounded with high probability). With probability atleast and under Defini-
tion 1, ||sl|| S nwv/1CC log 3 = O(VT).
Proof. To prove this lemma, we first bound the trace of the covariance matrix Σx+1 and then apply high probability bounds on the Martingale norm concentration. The trace of the covariance matrix Ex+1 can be bounded as follows (all the equations below are restricted to the dimensions of non-zero eigenvalues of G):
[2
Σ tr [r (U
(C.14)
jzk
tr - -UUUH
(C.15)
j
Σ tr - U nAPUM
(C.16)
J5k
Σ
nw Cotr [1T-nAp | nAP]
(C.17)
JSk
S Cn Σ n max([1 n/i13 D2D S 1n1 Cow?
(C.18)
J5k
Now, we can apply Corollary from Damian et al. [11] to obtain a bound on ||Sel| as with high probability, atleast 8. |KGull S /2tr(E) log = nww///C log J.
C.5 MAIN PROOF OF THEOREM 3.1
In this section, we present the main proof of Theorem 3.1. The proof involves two components: (1) the part where we derive the regularizer, and (2) bounding additional terms via concentration inequalities. Part (1) is specific to TD-learning, while a lot of the machinery for part (2) is directly taken from prior work [11] and Blanc et al. [6]. We focus part (1) here.
Our strategy is to analyze the learning dynamics of noisy TD updates that originate at 0*, In a small neighborhood around 0" we can expand the noisy TD update (Equation 3) using Taylor's expansion
27

---

Published as a conference paper at ICLR 2022
around 0r which gives:
0x+1 = 0k (g9(0k) (3)k) + ne Ek ~ (0,
(C.19)
g (n
1 G|Os - -0, 02 - 010 + +nE O(n - 0²1
= 0k+1 = 02 -
G(Ox - 0*)
(C.20)
Denoting VA == 02- 8" using the fact that [1((0))| S 6, we find that vs can be written as:
Vk+1 (I-n tEK + Givx 2d] mnl 1766)
(C.21)
Since the OU process SA stays in the vicinity of the point 0*, and follows a similar recursion to the one above, our goal would be to design a regularizer so that Equation C.21 closely follows the OU process. Thus, we would want to bound the difference between the variable V& and the variable Çt- denoted as ry to within small neighborhood:
Tk+1 Mk+1 - S6+1 (I - (J) (ux - (c) + Glu,, un] Omilex| 176).
We write down expression for Ta summing all the terms:
1) 2
- - pR]
-0 jgk
Σ J
nG [O(n] mx + r/6) (C.22)
Tk+1
term (a)
term (b)
Term (a) in the above equation is the one that can induce a displacement in rh as k increases and would be used to derive the regularizer, whereas term (b) primarily consists of terms that concentrate to O. We first analyze term (a) and then we will analyze the concentration terms later.
To analyze term (a), note that the term VGipk, vxl. by Lemma C.I, only depends Vi. via the covariance matrix V.4. So we will partition this term into two terms: (i) a term that utilizes the asymptotic covariance matrix of the OU process and (ii) errors due finite A: and stochasticity that will concentrate.
2
11 (1- pR]
(C.23)
jSk
Σ 1u-nGoa- jzk
E -u-nGy--iG(ppu. - -
(C.24)
isk
The first term is a "bias" term and doesn't 't concentrate to 0, and will give rise to the regularizer. We break this term using Lemma C.1 as:
TGIS",s" =2 =2 V - + Σ - VQ,
(C.25)
The regularizer (TD(O) is the function such that:
(o)r = Σ V' -V-))
(C.26)
RID(O) = Σ VQ,EVOQ - trace (E, VQ([VV111
(C.27)
where [[:]] denotes the stop gradient operator. the point 0 is stationary point the regularizer (T)( then Equations C.26 and C.27 imply that the first term of Equation C.25 must be 0. There- fore this case to show that 0" is attractive, we need to show that the other terms in Equations C.25, C.24 and term (b) in Equation C.22 concentrate around 0 and bounded in magnitude. The re- maining part of the proof shown in Appendix C.7 provides these details, but we first summarize the main takeaways in the proof to conclude the argument.
28

---

Published as a conference paper at ICLR 2022
C.6 SUMMARY OF THE ARGUMENT
We will show how concentrate terms in Equation C.26 besides the regularizer largely following the techniques from prior work, but we first summarize the entire proof. The overall update to the vector тж which measures the displacement between the parameter vector 0,- 0* and the OU- process Sa can be written as follows, and it is governed by the derivative of the implicit regularizer (modulo error terms):
7 2
[T-
+ O (Vitt 1 Ls Co)) (C.28)
Tk+1
JSK
An important detail to note here is that since the regularizer consists of EM and the size of MM (i.e, its eigenvalues), shown in Lemma C.4 depends one factor of 1j. So, effectively the first term in Equation C.28 does depend on two factors of r). Using Equation C.28, we can write the deviation between 0 and Og as:
Mk+1 Sk+1
? E -1-- 2
Vo O (Vnt polyo6, 22, Ls.w, Co)). (C.29)
jSk
The OU process ÇA converges to 8 in the subspace spanned by G, since the condition P < active in this subspace (if the condition that p(I n/G) < 1 in the subspace spanned by G is not true, then as Ghosh & Bellemare [20] show, TD can diverge). Now, given G satisfies this spectral radius condition, Çk would converge to 8" within a timescale of O
within this subspace, which Blanc al. [6] put the strength of the "mean-reversion" term. On the remaining directions (note that d > n2), the dynamics is guided by the regularizer, although with smaller weight of 17².
C.7 ADDITIONAL. PROOF DETAILS: CONCENTRATING OTHER TERMS
We first concentrate the terms in Equation C.25. The cumulative effect of the second term in Equa-
tion C.25 is given by:
Σ [T - G6 VQrtr [V - ~V
(C.30)
jsk
7)0) s $ E - 0 (22 (
(C.31)
Jšk
which follows from the fact that V²Q; is Lz-Lipschitz and using Lemma C.3 for contracting the remaining terms.
Next, we turn to concentrating the second term in Equation C.24. This term corresponds to the con- tribution of difference between the empirical covariance matrix v and the asymptotic covariance matrix T We expand this term below using the form of G from Lemma C.I, and bound it one by one.
Σ n)) -U-nC))-VV(((PA -CT. AEk
(C.32)
(V 1--- (x)") (VQ:-VQQ)) + (VikuaCa.22 + 7)0
jšk
(C.33)
Now, we that the term A+41 Uk+1V T can itself be written recursion:
A+t1 = (I- - TTO) + (T-n)G)sse + ecg (I (|)
+ +80 nM
(C.34)
Ax
B.
29

---

Published as a conference paper at ICLR 2022
Expanding the term Ak+1 in terms of summation over k, and plugging it into the expression from Equation C.33 we get
ΣE - isk
(C.35)
EE (-) + Bp) - n p-v-1
JSKPSi
Now by noting that and VQ, are (w, Co)-aligned, then so are GO and VQ,, we can finish the proof by repeating the calculations used by Damian et al. [11] (Appendix B, Equations 67-73) to bound the terms in Equation C.35 by (TvTE), but with an additional factor C
Term (b) in Equation C.22. When & is small enough, we can bound the term (b) using Ovynk)) similar Damian al. [11].
D PROOF OF PROPOSITION 3.2
In this section, we will prove Proposition 3.2. First, we refer to Proposition 3.1 in Ghosh Bellemare [20], which shows that TDD-earning is stable and converges if and only if the matrix Mg = 4 (4- -2') has eigenvalues with all positive real entries. Now note that if,
Σ (ss a) o(s a) S Σ
Ó(s' a') TT())))))))
(D.1)
s.a.s/
trace (Φ TT)) S ytrace TTT
(D.2)
Lrace
[ T (Φb 7b) < 0.
(D.3)
Since the of real matrix is the of real components of eigenvalues, if for given matrix
M, (raco(M) S O, then there exists atleast one eigenvalue A; such that Re(A,) < 0. If X; < 0, then the learning dynamics of TD would diverge, while if X; 0 for all i, then learning will not contract towards the TD fixed point. This concludes the proof of this result.
E EXPERIMENTAL DETAILS OF APPLYING DR3
In this section, we discuss the practical experimental details and hyperparameters in applying our method, DR3 various offline RL methods. We first discuss overview of the offline RL methods we considered in this paper, and then provide a discussion of hyperparameters for DR3,
E.1 BACKGROUND ON VARIOUS OFFLINE RL ALGORIHTMMS
In this paper, we consider four base offline RL algorithms that we apply DR3 on. These methods are detailed below:
REM. Random ensemble mixture [2] is an uncertainty-based offline RL algorithm uses multiple parameterized Q-functions to estimate the Q-values. During the Bellman backup, REM computes a random convex combination the target Q-values and then trains the Q-function to match this randomized target estimate. The randomized target value estimate provides robust estimate of target values, and delays unlearning and performance degradation that we typically see with standard DQN-style algorithms in the offline setting. For instantiating REM, follow the instantiation provided by the authors and instantiate a multi-headed Q-function with 200 heads, each of which serves as an estimate of the target value. These multiple heads branch off the last-but-one layer features of the base Q-network. The objective for REM is given by:
la I
min Ex Ex.anns'' 8
E Eap
(t. - MAx Σ Qu (s', a') (E.I)
where Iy denotes the Huber loss while PA denotes the probability distribution over the standard (K s)-sipllex
30

---

Published as a conference paper at ICLR 2022
CQL. Conservative Q-learning [27] is offline RL algorithm that learns a conservative value function such that the estimated performance of the policy under this learned value function lower- bounds its true value. CQL modifies the Q-function training to incorporate term that minimizes the overestimated Q-values in expectation, while maximizing the Q-values observed in the dataset, in addition to standard TD error. This CQL regularizer is typically multiplied by coefficient a, and we pick = 0.1 for all our Atari experiments following Kumar et al. [28] and a = 5.0 for all our kitchen and antmaze D4RL experiments. Using Te (s. a) denote the target values computed via the Bellman backup (we use actor-critic backup for D4RL experiments and the maxa' backup for standard Q-learning in our Atari experiments following Kumar et al. [27]), the objective for training CQL is given by:
log Σ exp(Q(s. - -...-.D (Q(s. 2 | a)) (s, a))
min
Eg-D
The deep Q-network utilized by us is a ReLU network with four hidden layers of size (256, 256, 256, 256) for the D4RL experiments, while for Atari we utilize the standard convolutional neural network from Agarwal et al. [2], Kumar et al. [28] with 3 convolutional layers borrowed from the nature DQN network and then a hidden feedforward layer of size 512.
BRAC. echavior-regularize actor-critic [49] is a policy-constraint based actor-critic offline RL algorithm which regularizes the policy to stay close to the behavior policy Ty to prevent the selection oout-of-distribus actions. In addition. BRAC subtracts this divergence estimate from the target Q-values when performing the backup, to specifically penalize target values that come from out-of- distribution action inputs the next state (s', a').
Q-function: min BR (r(s,,)) + +.aa (+b))) (aa) [Qo a²) + 3 log *t((a(())) - Qo(s, Qo(s,a))) a))
8
Policy: max EaDDannas(1)) [Qa(s, a) + ( - a log (a]s)].
(E.2)
COG. COG [38] is an algorithmic framework for utilizing large, unlabeled datasets of diverse be- havior to learn generalizable policies via offline RI. Similar to real-world scenarios where large unlabeled datasets are available alongside limited task-specific data, the agent is provided with two types of datasets. The task-specific dataset consists of behavior relevant for the task, but the prior dataset can consist of number of random or scripted behaviors being executed the same envi- ronment/setting. The goal in this task is to actually stitch together relevant and overlapping parts of different trajectories to obtain a good policy that can work from a new initial condition that was not seen in trajectory that actually achieved the reward. COG utilizes CQL the base offline RI. algorithm, and following Singh et al. [38], we fix the hyperparameter a = 1.0 in the CQL part for both base COG and COG DR3. All other hyperparameters including network sizes, etc are kept fixed as the prior work Singh et al. [38] as well.
E.2 TASKS AND ENVIRONMENTS USED
Atari 2600 games used. For all our experiments, we used the same set of 17 games utilized by Ku- mar al. [28] to test rank collapse. In the case of Atari, used the 5 standard games (ASTERIX, QBERT, PONG, SEAQUEST, BREAKOUT) for tuning the hyperparameters. a strategy followed by sev- eral prior works [22, 2, 28]. The 17 games we test on are: ASTERIX, QBERT, PONG, SEAQUEST, BREAKOUT, DOUBLE DUNK, JAMES BOND, MS. PACMAN, SPACE INVADERS, ZAXXON, WIZ- ARD OF WOR, YARS' REVENGE, ENDURO, ROAD RUNNER, BEAMRIDER, DEMON ATTACK, ICE HOCKEY.
Following Agarwal al. [3], we report interquartile mean (IQM) normalized scores across all runs as mean scores can be dominated by performance on a few outlier tasks while median is independent of performance on all except 1 task zero score on half of the tasks would not affect the median. IQM which corresponds to 25% trimmed mean and considers the performance middle 50% of the runs. IQM interpolates between mean and median, which correspond to 0% and almost 50% trimmed means across runs.
D4RL tasks used. For our experiments on D4RL, we utilize the Gym-MuJoCo-vd environments for evaluating BRAC, since BRAC performed somewhat reasonably these domains [18], whereas we use the harder AntMaze and Franka Kitchen domains for evaluating CQL, since these domains challenging for CQL [27].
31

---

Published as a conference paper at ICLR 2022
Table E.1: Hyperparameters used by the offline RL Atari agents in our experiments. Following Agarwal et al. [2], the Atari environments used by us are stochastic due to sticky actions, i.e., there a 25% chance at every time step that the environment will execute the agents previous action again, instead of the new action commanded. We report offline training results with same hyperparameters over random seeds of the offline dataset, game simulator and network initialization.
Hyperparameter
Setting (for both variations)
Sticky actions
Yes
Sticky action probability
0.25
Grey-scaling
True
Observation down-sampling
(84, 84)
Frames stacked
Frame skip (Action repetitions)
Reward clipping
[-1, 1]
Terminal condition
Game Over
Max frames per episode
108K
Discount factor
0.99
Mini-batch size
32
Target network update period
every 2000 updates
Training environment steps per iteration
250K
Update period every
4 environment steps
Evaluation
0.001
Evaluation steps per iteration
125K
-etwwork channels
32, 64, 64
Q networkk: filter size
8 x 8, 4 X 4, 3 x 3
Q Qettwork: stride
4. 2,
Q -entwork: hidden units
512
Robotic manipulation tasks from COG [38]. These tasks consist of a 6-DoF WidowX robot, placed in front of two drawers and larger variety of objects. The robot can open close drawer, grasp objects from inside the drawer or on the table, and place them anywhere in the scene. The task here consists of taking an object out of drawer. A reward of +1 is obtained when the object has been taken out, and zero otherwise. There are two variants of this domain: (1) in the first variant, the drawer starts out closed, the top drawer starts out open (which blocks the handle for the lower drawer), and an object starts out in front of the closed drawer, which must be moved out of the way before opening, and (2) in the second variant, the drawer is blocked by an object, and this object must be removed before the drawer can be opened and the target object can be grasped from the drawer. The prior data for this environment is collected from a collection of scripted randomized policies. These policies are capable of opening and closing both drawers with 40-50% success rates, can grasp objects in the scene with about a 70% success rate, and place those objects at random places in the scene (with slight bias for putting them in the tray).
E.3 THE DR3 REGULARIZER COFFFICIENT
We utilize identical hyperparameters of the base offline RI. algorithms when DR3 is used, where the base hyper-parameters correspond the ones provided in the corresponding publications. DR3 requires us to tune the additional coefficient co. that weights the DR3 explicit regularizer term. In order find this value on our domains, we followed the tuning strategy typically followed on Atari, where we evaluated four different values of co E (0.001, 0.01, 0.03. 0.3) on 5 games (ASTERIX, SEAQUEST, BREAKOUT, PONG and SPACEINVADERS on the 5% replay dataset settings, picked co that wprked best on just these domains, and used it to report performance on all 17 games, across all dataset settings (1 % replay and 10% initial replay) in Section 5. This protocol standard in Atari and has been used previously in Agarwal et al. [2]. Gulcehre et al. [22], Kumar et al. [28] in the context of offline RL. The value of the cocfficient found using this strategy was Go = 0.001 for REM and co 0.03 for CQL.
For CQL on D4RL, we ran DR3 with multiple values of co E (0.0001, 0.001, 0.01, 0.5, 1.0, 10.0). and picked the smallest value of co which did not lead to eventually divergent (either negatively diverging or positively diverging) Q-values, in average. For the antmaze domains, this corresponded co 0.001 and for the FrankaKitchen domains, this corresponded to co = 1.0.
32

---

Published as a conference paper at ICLR 2022
F COMPLETE RESULTS ON ALL DOMAINS
In this section, we present the results obtained by running DR3 on the Atari and D4RL. domains which were not discussed in the main paper due to lack of space. We first understand the effect of applying DR3 on BRAC [49], which was missing from the main paper, and then present the per-game Atari results.
Table F.:: Normalized interquartile mean (IQM) final performance (last iteration return) of CQL,, CQI. + DR3, REM and REM + DR3 after 6.5M gradient steps for the 1% setting and 12.5M gradient steps for the 5%, 10% settings. Intervals in brackets show 95% CIs computed using stratified percentile bootstrap [3]
Data
CQL
CQL DR3
REM
REM DR3
%
44.4 (31.0, 54.3)
61.6 71.5)
0.0 (.D.7,
13.1 (9.9. 18.3)
5%
89.6 (67.9.98.1)
100.2 (90.6, 102.7)
3.9 (3.1, 7.6)
74.8 (39.6, 84.4)
10%
57.4 (53.2, 62.4)
(62.8, 73.0)
24.9 (15.0, 29.1)
72.4 (65.7, 81.7)
Table F.2: Performance of DR3 when applied in conjunction with BRAC [49]. Note that DR3 attains larger final performance the end 2M steps of training) as well higher average performance (i.e. stability score) across all iterations of training.
Average Performance across Iterations BRAC
Final Performance
Task
BRAC DR3
BRAC
BRAC + DR3
halfcheetah-eeperrt-d
1.7 1,9
49.9 16.7
2.1 + 3.3
71.5 = 24.9
43.5 0,2
43.2 1: 0.2
45.1 1 0.8
44.9 0.6
wewhhhetah-mdiunnnt expertt-vO
5.5
24.8 9.3
6.7
halfcheetah--andoom--0
24.4
0.3
0.8
1..2
haffchetaa-mmediun-reelay--
0.3
44.1
45.0
4.5
hopper-cxpert-vo
15.7
21.8
1.06
5.3
hopper-medium--V
1.4
7.1
3.72
58.3
aogper--edduum-expeettww
4.7
+
31.7
4.9
hopper-random-vo
1..7
1.0
12.2
11.1
hopper-mediinn-eeplay-
0.3
30.3 0.8
3.2
36.1
walker2d-expert-vv0
24.5
54.0
60.6
walker2d-medium--v0
0.3
80.8 0.2
8.2
0.34
5.2
6.4 3.4
22.4
3
walker2d-random-wW
1.4
1.7 0.9
0.1
2.1
walker22-mediuu--eepay-000
26.1
47.4 + 4.1
11.7
38.7
33

---

Published as a conference paper at ICLR 2022
Table F.4: Mean evaluation returns per Atari game across 5 runs with standard deviations for 1 dataset. The coefficient for DR3 0.03 with CQL coefficient of 1.0. The average performance computed over 20 checkpoints spaced uniformly over training for 100 iterations where 1 iteration corresponds to 62,500 gradient updates.
Final Performance
Average Performance across Iterations CQL
Game
CQL
CQL + DR3
CQL DR3
Asterix
656.9 = 91.0
821.4 + 75.1
650.2 = 65.3
814.1 + 25.1
Breakout
23.9 3.8
32.0 3.2
23.8 + 0.5
32.8 + 3.1
Pong
16.7 + 1.7
14.2 + 3.3
15.7 2.0
15.1 + 2.3
Seaquest
449.0 11.0
446.6 26.9
474.5 + 30.3
456.1 + 17.0
Qbert
8033.8 1513.2
9162.7 993.6
1
9000.7 225.2
Sp SpaceInvaders
: 123.2
371.7
440.6 29.6
Zaxxon
813.4
1757.4 879.4
834.6
1634.0 673.9
YarsRevenge
=
16011.3 1409.0
1 613.6
RoadRunner
37000.7 1148.5
24928.7 7484.5
3
32063.3 1011.4
MsPacman
2
59.
2224.1 80.8
BeamRider
780.3 64.5
6 25.1
782.0 36.1
619.9 + 20.9
Jamesbond
5
460.5 102.0
524.6 118.5
484.2 89.4
Enduro
198.4
253.5 14.2
259.8 16.4
276.1 + 16,9
WizardOfWor
771.1 358.2
904.6 =43.7
833.7 + 168.4
935.2 174.4
IceHockey
-8.7 + 1.3
-7.8 0.9
-8.8
-7.9
DoubleDunk
11511:11
-1.0
-15.3 0.9
DemonAttack
1970.2 + 161.3
386.2 75.3
1338.8 + 298.4
414.0 + 46.0
Table F.5: Mean evaluation returns per Atari game across 5 runs with standard deviations for 5% dataset. The coefficient for DR3 0.03 with CQL coefficient of 0.1. The average performance computed over 20 checkpoints spaced uniformly over training for 200 iterations where 1 iteration corresponds to 62,500 gradient updates.
Final Performance
Average Performance Iterations CQL
Game
CQL
CQL + DR3
CQL DR3
Asterix
1798.2 + 168.6
3318.5 + 301.7
1812.7 + 64.0
3790.5 + 218.0
Breakout
94.1 44.4
166.0 23.1
105. = 10.4
196.5 = 4.4
Pong
13.1 + 4.2
17.9
15.2 1.3
17.4 1.2
Seaquest
1815.9 722.8
4
1382.3 + 258.1
3722.3 + 969.5
Qbert
10595.7 1648.5
9605.6 5
9552.0 925.6
10830.7 + 783.1
Spacelnvaders
758.9 56.9
1214.6
662.0
1323.7 = 94.4
Zaxxon
1
4
1508.8 437.5
3556.5 + 531.3
YarsRevenge
24036.7 33700.6
17124.742122
18339.8 + 1299.7
RoadRunner
40728.4
38432.6+1538
42338.44
41260.2 1 1008.6
MsPacman
+
2790.6+3355
2
3101.2 381.6
BeamRider
1897.6 473.7
785.8
2218.5
775.9 + 12.5
Jamesbond
118.84499
96.8
4.6
106.1 34.8
Enduro
764.3 168.7
938.5
797.7
923.2 1 40.3
WizardOfWor
943.2 + 380.3
612.0 343.3
1004.3 1004.331144 314.7
1007.4 313.2
IceHockey
-17.3 0.6
-15.0 + 0.7
-16.6 0.5
-12.0 0.3
DoubleDunk
-18.1 1 1.5
-16.2 + 1.7
-17.3 1.0
-16.0 1.6
DemonAttor Attack
4055.8 + 499.7
8517.4 + 1065.9
4062.4 465.8
8396.7 + 689,4
34

---

Published as a conference paper at ICLR 2022
Table F.6: Mean returns per Atari game across runs with standard deviations for initial 10 % dataset. The coefficient for DR3 is 0.03 with CQL coefficient 0.1. The average performance is computed over 20 checkpoints spaced uniformly over training for 200 iterations.
Final Performance
Average Performance across Iterations CQL DR3
Game
CQL
CQL + DR3
CQL
Asterix
2803.9 294.6
3906.2 521.3
2903.2 + 217.7
4692.2 + 377.0
Breakout
64.7 + 7.3
70.8 + 5.5
6
75.4 + 6.0
Pong
5.3 6.8
5.5 6.2
5.0
8.2
Seaquest
222.3 + 219.5
1313.0 220.0
704.9 254.5
1327.9 + 250.0
Qbert
4803.2 489.5
5395.3 1003.6
+
4708.5 463.0
SpaceInvaders
= 121.5
938.1 80.3
7
902.1
Zaxxon
101.6:4500
8
194.4:1855.1
#370.3
YarsRevenge
13076.2 2427.0
12413.9 2869.7
12493.2
12395.6 1044.2 1044.2
RoadRunner
45336.9 + 1366.7
45522.7 1068.1
44808.0
MsPacman
2459.5
2427.5 191.3
2528.1 149.2
2488.3 109.8
BeamRider
4200.7 470.2
3468.0 + 238.0
4
3344.3 + 289.0
Jameshond
25.4
89.7 + 15.6
+..11
111.7 10.9
Enduro
946.7 289.7
1160.2 81.5
1013.9
1136.2 32.5
WizardOfWor
520.4 451.2
764.7 = 250.0
4
792.2 101.3
IceHockey
-18.1
-16.0 + 1.3
+ 0.5
-
DoubleDunk
-21.2
-20.6 + 1.0
-20.6 0.3
-19.7 0.5
DemonAttack
4145.2 + 400.6
7152.9 + 723.2
4839.4 586.7
7278.5 + 701.3
Table F.7: Mean returns per Atari game across 5 runs with standard deviations for 1% dataset.
The coefficient for DR3 is 0.001 while multi-headed REM with 200 Q-heads [2]. The age performance is computed over 20 checkpoints spaced uniformly over training for 100 iterations.
Final Performance
Average Performance across Iterations
Game
REM
REM DR3
REM
REM DR3
Asterix
240.4 29.1
405.7 46.5
304.4 + 9.3
413.7 + 39.6
Breakout
0.7 0.7
14.3
6.3
10.3 + 1.1
Pong
-14.2
6.3
-12.1
-15.3 3.0
Seaquest
81.0 + 78.5
246.6 49.5
489.9 + 128.6
Qbert
239.6 = 133.2
4
255.5
471.0 116.5
SpaceInvaders
206.6 10.6-7744 77.6
188.6
422.7222.4
Zaxxon
534.9 #
2596.4 1726.4
1807.94477
707.7 577.4
YarsRevenge
5480.2 962.3 962.3
4018.8 987.8
7352.0 574.7
RoadRunner
0.0
6371.
14231,9 2406.0
MsPacman
698.8 + 129.5
1275.1 345.6
690.4
860.4 57.1
BeamRider
703.0+
522.9 42.2
745.5
592.2 27.7
Jameshond
157.6 65.0
53.3 12.1
88.8 + 27.2
Enduro
0.5 0.4
13244 16.1
21.7 4.0
197.5 + 19.1
WizardO/Wor
1663.7 417.8
552.1 1 253.1
1460.8 + 194.8
IceHockey
-16.7 0.9
-9.1 5.1
-12.1
-4,8 + 1.8
DoubleDunk
-21.8 1.0
-17.6 + 1.5
22.4
-17.1 1.6
DemonAttack
102.0 + 17.3
162.0 34.7
124.0 + 10.7
145.6 + 27.2
35

---

Published as a conference paper at ICLR 2022
Table F.8: Mean returns Atari game across 5 runs with standard deviations for the 5%
dataset. The coefficient for DR3 is 0.001 while we use multi-headed REM with 200 Q-heads [2]. The average performance is computed over 20 checkpoints spaced uniformly over training for 200 iterations.
Final Performance
Average Performance across Iterations REM
Game
REM
REM + DR3
REM DR3
Asterix
876.8 + 201.1
2317.0 = 838.1
958.9 50.9
1252.6 = 395.1
Breakout
15.2 4.9
33.4 = 4.0
16.3 3.4
17.7 = 2.4
Pong
7.5 5.2
-0.7 + 9.9
-4.7 + 3.0
-12.0 + 3.2
Seaquest
1276.0 417.3
2753.6 1119.7
1484.3 + 367.7
1602.0 603.7
Qbert
2421.4 = 1841.8
7417.0 2106.7
1330.7 4310
4045.8 = 898.9
Sp SpaceInvaders
23.
67.4
2
362.1 33.6
Zaxxon
6738.2
1
3630.7
346.1 = 512.1
YarsRevenge
16930.4 =
12936.5 = 1286.0
RoadRunner
15570.9
12795.66
2
33554.1 = 1880.4
MsPacman
1
2303.1 202.7
1147.7 126.1
1438.7 = 140.4
BeamRider
1922.5 589.1
674.8 + 21.4
886.9 82.1
698.3 21.5
Jamesbond
189.6
130.5 45.7
9.3
88.6 41.5
Enduro
172.7
583.9 108.7
236.8 11.
457.7 39.3
WizardOfWor
1281.3 = 66.7
1863.7 = 261.2
IceHockey
-9.7
+.1
-8.1 0.7
-4.1 - 1.5
DoubleDunk
-18.4
1.6
-19.6 1.0
-17.8 + 1.9
DemonAttace
507.7 + 120.1
5602.3 + 1855.5
581.6 + 207.0
1452.3 = 765.0
Table F.9; Mean returns per Atari game across 5 runs with standard deviations for initial 10 %
dataset. The coefficient for DR3 is 0.001 while we use multi-headed REM with 200 Q-heads [2]. The average performance is computed over 20 checkpoints spaced uniformly over training for 200
iterations.
Final Performance
Average Performance across Iterations REM
Game
REM
REM + DR3
REM DR3
Asterix
2254.7 + 403.6
5122.9 + 328.9
2684.6 + 184.4
3432.1 + 257.5
Breakout
81.2 13.9
96.8 = 21.2
63.5 + 4.6
62.4 + 6.1
Pong
8.8 + 3.1
7.6 11.1
2.6 + 2.1
-2.5 + 5.6
Seaquest
1540.2 + 354.6
9 605.9
1029.5 + 260.6
836.2 + 234.3
Qbert
4330.7 250.2
4126.2 495.7
3478.0 + 248.0
3494.7 + 380.3
SpaceInvaders
895.2 68.3
799.0 28.3
43.4
653.1 21.5
Zaxxon
100.788999 997.4
0.0
490.2 4
0.0
YarsRevenge
15199
1 2413.8
11508.5+
10977.7 + 1026.9
RoadRunner
4
49129.4
1887.9
41995.2 3799
1482.1
MsPacman
+ 393.0
2268.8 + 455.00
+
2126.6 +
BeamRider
2921.7 308.7
4
3727.5
2871.0
Jamesbond
197.8
304.5
149.0 120.5
Enduro
229.5:2000
832.5
584.6 85.3
801.8
WizardOfWor
606.5 1 823.2
823.2
920.0 497.0
838.3 = 343.7 343.7
926.3 318.5
IceHockey
.4.3 = 0.6
-5.9
-7.0 1.1
3.74
DoubleDunk
-17.7 #
-1.55 2.5
-16.9 1 0.5
-16.7 1.0
1.0
DemonAttor Attack
6097.9 4 1251.3
9674.7 + 1600.6
4649.1 4 514.6
5141.9 + 361.4
36

---

Published as a conference paper at ICLR 2022
Table F.10: Average returns across runs for the random agent and the average performance of the trajectories in the DQN (Nature) dataset. For Atari normalized scores reported in the paper, the random agent is assigned a score of 0 while the average DQN replay is assigned a score of 100. Note that the random agent scores are also evaluated Atari 2600 games with sticky actions.
Game
Random
Average DQN-Replay
Asterix
279.1
3185.2
Breakout
1.3
104.9
Pong
-20.3
14.5
Seaquest
81.8
1597.4
Qbert
155.0
8249.7
SpaceInvaders
149.5
1529.8
Zaxxon
10.6
1854.1
YarsRevenge
3147.7
21015.0
RoadRunner
15.5
38352.3
MsPacman
248.0
3108.8
BeamRider
362.0
4576.4
Jamesbond
27.6
560.3
Enduro
0.0
671.9
WizardO/Wor
686.6
1128.5
IceHockey
-9.8
-8.5
DoubleDunk
-18.4
-11.3
DemonAttad
166.0
4407.5
37

---

Published as a conference paper at ICLR 2022
F.1 PER-GAME LEARNING CURVES FOR ATARI GAMES
CO.
OCL + DR3
Adtari
Beam Rider
Broakout
5000
250
4000
3000
200
5000
2000
2000
100
1000
100
160
200
60
100
1.50
160
200
Graddent Updatee 60
Gradiet Updaatin BPP Sic)
Gredient Updittex
10000
000
-15-
000
5000
-20
400-
200
-25
50
100
160
200
60
100
150
200
50
100
150
200
Creecient Lipdatex (a 62 Sk)
Cradient Uppdattes (x BP Sic)
Gredient Updattes 5k)
Ioe Hockey
Ja mas Bond
Ms. Pao Man
Auoraga Boor
400
300
3000
-10
2000
-10-
100
1000
-20
0
150
Gradtent Updates 62.30
Grod Updates 02 BK)
Gradient Updattes 62.0K)
Pong
OrBert
Road Runner
50000
12500
Acompo Store
10000
8000
20000
-10-
2500
10000
100
100
60
100
150
200
160
200
Cradient Updattix
62.5k)
Cradlerd Uppdalle 62. Sk)
Gredient Updattes (2.555
2000
1500
2000
1000
100
150
200
60
100
150
200
50
160
150
200
Gradient Updates (x 62.2k)
Grad ent Updatees DX 02. 5K)
Gradlent Updates 02.00
Yard's Revenge
Zaxizon
25000
20000
Aootog
15000
2000
10000
5000
Grackent
62.5k)
Gradiert
Figure F.1: Per-game learning curves of CQL and CQL + DR3 on the 5% uniform replay dataset, for which the normalized average learning curve is shown in Figure 4. Note that CQL DR3 attains higher performance than CQL for a majority of games, and rises up a higher peak.
38

---

Published as a conference paper at ICLR 2022
REM
Auterik
Beam Rider
Breaicul
0000
4000
40
Aeange
3000
2000
20-
1000
1000
0
50
100
150
50
100
150
200
50
100
150
200
Gradient Updates (x 02 5kk
Gradient Updates (x 02 5k)
Gradient Updates (X 02 5k)
Demon Atlack
Doubre Durk
Enduro
10000-
8000
Aoraage Spore
6000
4000
155
2000
-20
-25
60
100
150
50
100
160
200
60
100
150
200
Credient Lipeistoss (x 02
Cradiet Updatee (x
Cracient Uppastees (a 62
low Horiwwy
Jammex Bond
Ms. Pacc-Mten
300
Avwrag
-10-
200
1030
-20-
00
100
150
50
100
150
200
60
100
150
200
Cradient Upphsoos (x 62
Cracient Updattos
Updastoos 62
Pong
OrBart
Rowd Runner
10000
10-
6000
20000
-10-
2000
-20-
50
100
180
50
100
150
50
100
1501
Updatoos
Gradient Updatios 62
Updatees
62
Beaquest
Space Invaders
Wizard Of War
4000
600
3000
2000
2000
2000
1000
200
1000
50
100
50
180
150
50
100
Updatos
Updates
Updates (X
Yard's Revenge
Zaxoon
8000
Neerage Spore
20000
6000
18000-
2000
0
Updates
Updates
Figure F.2: Per-game learning curves of REM and REM DR3 the 5% uniform replay dataset, for which normalized average learning curve shown Figure Note that REM + DR3 attains higher performance than REM for a majority of games.
39

---

Published as a conference paper at ICLR 2022
F.2
DOT PRODUCT SIMILARITIES FOR CQL+DR3 AND REM+DR3 ON 17 GAMES
COL
DRS
Astenx
Beam
Breakout
Das-pmdart aittwity
10"
10"
1o"
10"
10²
10
10
10"
10²
10
10"
10
2
3
Cradient Updanes 5k)
Gradiant Updates (x 62.5k)
Gredienn Updates (6.55))
Demon Attack
Double Dunk
Enduro
10"
10"
Del-produt mibrtt
10²
10
10
10"
10"
10
10"
10
10
Gradiem Updanes 62.5k)
Gradient Updates (x 82
Gradient Updates 02.62.2 (k)
Hockey
James
Ms. Pan-Man
10
10"
Delppootaat sistoohhssstttwwyyy sirtwiyy
10
10¹
10'
10"
10
10"
10
10"
10"
Gradient Updanes 62.5
Gradient Updattes (x
Gradienn Updates (2)
Pong
O'Bert
Road Runner
10
10"
Solprodctttmmo embtrrty
10
10²
10
10
10a
10
10"
10"
10"
10
2
Gradiem Updates 62.5k)
Gradient Updates (x
Gradient Updates 02. (k)
Seaquest
Space Invadera
Wizard Wor
10
Dol-ymduat simbaiity
10
10
10
10
10"
1o"
10
10'
10
10
10f
10"
Cradient Updanes 5k)
Gradiant Updatles (x 62.62.55
Gradiend Updates 62. (k)
Yandss Revenge
Zackon
10
10
Desepodtattt meletyy
10"
10"
10.
10"
100
10"
2
Grodiem Updates (x 62.5k)
Gradient Updates (x 02 5k]
Figure F3: Per-game feature dot products (in log scale) of CQL and CQL + DR3 on the 5% uniform replay dataset. Note that CQL+ + DR3 attains a smaller value of the feature dot product.
40

---

Published as a conference paper at ICLR 2022
REM
REM DR3
Aatarix
Bearn Rider
Breakout
Daloprobuut simibvrtt
10
10²
10"
10
10²
10
10"
10"
10"
10"
Gradient Updates 62.5% (225k)
Gradient Updates (x
Gradient Updates (2.5K)
Demon Attack
Double Dunk
Endura
10
Dekepodtart Dispostaattmmiee smiwey
10
10"
to
10"
10"
10'
10
10"
10'
10"
2 Cradient Updanes 5k)
Gradiant Updittes (x 62.62225
Gradient Updates 62 5k)
loa Hockey
James Band
Ms. Pac-Man
10
10"
Uatgrdduc dimbarty
10
10"
to"
10
1
10"
10²
10
10"
10
Grodiem Updates (x 62.50 5k)
Grodient Updates (x 02.82220 [k]
Gradient Updates 02 5k)
Pong
Q'Bart
Road Runnar
10 15
Dokprduct eimartty
1o
10
10
to'
10
10*
to
10
10"
10a
10
10"
10
Cradient Updanes 62. 5k)
Gradiont Updates (x 62.62.25 5k]
Gradient Updates (X 62 (H)
Seaquees
Space Invaders
Wizzrd War
10"
Dateprouutt ambrrtty
10
10
10
10
10
10"
10'
10
10
2
4
3
Gradiem Gradiem/Updates 62.5K)
Gradiont Updates (x 02 5k)
Gradiert Updotes by 02.52..00 (2255k)
Yars's Revenge
Zacozon
70"
10
Dasproduct atr
10""
10
10.
10'
10
10¹
2
4
Updaiin 62.61
Gradiant Updates (x 62.5k)
Figure F.4: Per-game feature dot products (in log scale) of REM and REM DR3 on the 5% uniform replay dataset Note that REM DR3 attains higher performance than REM for majority of games. Note that the dot products for REM+DR3 stabilize small, and decreases for majority of the training steps for number of games, or stabilize at a small value.
41


Reviews:
[{"review_id": "r6qTXvcKwp", "reviewer": "Reviewer_T3ZN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n---------\n\n-   Interesting theoretical analysis, while somewhat compromised to make\n    the algorithm tractable, still provides insight into the problem\n    addressed. It might be good to further show evidence that the noise\n    model is well-justified in the RL regime, which would justify the\n    leap from theory to empirics.\n-   Empirical analysis is comprehensive, clearly demonstrating the\n    problem and algorithms stability when the problem is addressed\n    (through the proposed regularizer).\n\nWeaknesses\n----------\n\n-   While the empirical analysis is comprehensive, the performance of\n    the algorithms with the DR3 regularizer in the experiments are not\n    completely compelling. Although the mean performance seems better,\n    the statistical significance is not there in Figure 3 and Table 2\n    (and Figure 6, Seaquest). The results in Atari across all games are\n    more impressive, but the aggregation and normalization across games\n    can hide idiosyncrasies.\n\nDetailed Comments\n-----------------\n\n-   Section 3, Experimental setup: For TD-Learning, it says an action is\n    sampled. Shouldn't this be an expectation over the policy (and hence\n    use every action, without sampling?). If all the actions are\n    included in the target through an expectation, the problem of\n    divergent action-values should be less pronounced if not absent. The\n    problem seems to be that some action-values are left dangling (i.e.\n    the action-values of actions that are not sampled). By including\n    them all through an expectation, any problematic danging\n    action-values should be corrected immediately. The problem remains\n    for Q-learning, but further analysis needs to ablate maximization\n    bias and other confounders.\n\n-   Sec 3.1, Paragraph 2: What is meant by learned behavior policy?\n    Shouldn't the behavior policy be given or fixed?\n\n-   I think there is a leap in reasoning by connecting states aliasing\n    and high dot-products. Dot products can increase between two vectors\n    without any aliasing. If the magnitude of both vectors increase then\n    the dot product will increase without any change in the angle\n    between the vectors.\n\n-   Increasing dissimilarity between successive states is intuitively\n    reasonable, but this can also be a hindrance if the successive\n    states are indeed similar. Is there any way to account for this?\n\n-   Shouldn't a target network prevent the representations from being\n    similar? The networks are different and hence the representations\n    will be a few gradient steps away.\n\n-   Section 2 Equation 1: While we do take the gradient of this\n    expression, it is not correct to say that this is a proper\n    optimization objective. It is not minimized in practice (and often\n    diverges). In addition, methods that actually minimize this\n    objective, such as residual-gradient, do not result in a good\n    control policy.\n\n-   Figure 2: The left and center plots communicate the issue clearly,\n    except that the training loss plot on the right is not reflective of\n    the problem. As you point out, bootstrapping algorithms do not\n    minimize any objective function as the gradient does not belong to\n    any objective function.", "summary_of_the_paper": "This paper provides empirical and theoretical evidence that value-based\nmethods that optimize TD errors with SGD have an implicit \"regularizer\" that\nincreases the dot-product of the representation at successive states.\nThe theoretical analysis shows that this can be offset by a term that\npenalizes large dot products, which motivates the proposed regularizer\n(DR3). The paper shows empirically that these large dot products are a\nsource of divergence and that training with the DR3 regularizer can\nstabilize learning.\n", "summary_of_the_review": "Overall, the paper raises and simultaneously addresses an interesting\nproblem for RL algorithms that employ function approximation. The paper\nis somewhat held back by the empirical analysis, specifically in the\nlack of statistical significance in several of their experiments. I am\nalso not sure of the correctness of the intuition that the dot-products\nresult in aliasing in reinforcement learning. The paper is still above\nthe acceptance threshold, but I hope that my comments below contribute\nto improving the paper.\n\n\nEdit: After discussion with the author and reading the back-and-forth between the authors and reviewer 5pgu, I have increased my score from 5 -> 8.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "jSFtiDyWsV", "reviewer": "Reviewer_oKwL", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The identified phenomenon is very critical: When the consecutive state-action pairs are encoded to have similar representations, the learned value functions, as well as corresponding policies, would perform poorly.\n2. This paper provides theoretical evidence to explain the feature co-adaptation phenomenon, which meanwhile explains why it is severe for out-of-sample next actions.\n3. The empirical evaluation strongly confirms the effectiveness of the proposed regularizer.\nWeakness:\n1. There are some typos, e.g., \"stableif\" in Theorem 3.1.\n2. The organization can be improved further. It is not easy to follow this story, and there are many preliminaries about offline RL that are indispensable for readers to understand the discussions.", "summary_of_the_paper": "This paper first empirically notices the feature co-adaptation phenomenon which will be exacerbated in TD-learning with out-of-sample next actions. The authors theoretically show that the implicit regularization of (noisy) SGD encourages maximizing the inner product between the encoded features of current and next state-action pairs, which is responsible for the observed phenomenon. Thus, the authors propose a novel explicit regularizer for offline TD-learning to encourage smaller inner products. Extensive experiments on both video game environments and robotics demonstrate the effectiveness of the proposed regularizer, with various offline TD-learning algorithms as the testbeds.", "summary_of_the_review": "This is a good paper which studies an important problem from both empirical and theoretical aspects and brings in sufficient novel stuff. I appreciate the contributions made by the authors and recommend this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "j471pVvkbdm", "reviewer": "Reviewer_7Brr", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is overall well-written and clear. The method proposed is simple, quite clearly motivated and the claims are quite well supported by the experiments. Here are a few elements/questions that might need some clarifications:\n- the dot products can increase either because of an increased co-adaptation or because the magnitude of the vector of features considered increases. How can be sure that it is due to a co-adaptation and not because of an increase in the magnitude of the features?\n- In Fig 3, 5% and 25% of the data used by Singh et al. (2020) are used to make these more challenging. Would it also be useful to provide the results for the case of 100% of the data? It seems that even when data is relatively abundant, it would be interesting to report the performance with and without the regularizer. The same comment applied for Figure where a plot with 100% of the data could also be of interest.\n- The authors mention that they will release scores for individual runs as well as open-source their code. Is it possible to have access to that during this rebuttal period?\n\nMinor remarks: \n- typo: \"(...) and the the dot-product (...)\"", "summary_of_the_paper": "This paper discusses how the implicit regularization effect of SGD could be harmful in the offline deep RL setting, due to degenerate feature representations, aliasing the representations for state-action pairs that appear on either side of the Bellman backup. To address this issue, the paper proposes a simple explicit regularizer (DR3) that counteracts the undesirable effects of this implicit regularizer. When combined with existing offline RL methods, DR3 has strong performance and stability (Atari 2600 games, D4RL domains and robotic manipulation from images).", "summary_of_the_review": "Paper that investigates a simple regularization technique for off-policy value function learning in deep RL. The idea is quite well motivated by showing a few insightful values through learning (not only the score). The empirical results are quite convincing, even though they could be more complete.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1eMOjLgKyxa", "reviewer": "Reviewer_5pgu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall the paper is well written, pedagogical, and the problem at stake is really interesting. But I find the presentation of experimental results misleading which makes me doubt about the performances of the proposed solution.\n\n### Major revisions/questions/remarks\n  * Roughly 3 papers out of 5 refer to non peered reviewed papers (arXiv)\n  * Which algorithm is used to learn the Q-value in Figure 5? I am assuming it is CQL since the results match the ones for CQL in Kumar et al. 2021 \n  * \"it also alleviates rank collapse, with no apparent term that explicitly increases rank\" (Section 5), and \"This is potentially suprising, because no term in the practical DR3 regularizer explicitly aims to increase rank\" (Appendix A.3):\n    * The only srank's behavior exposed in this paper are the ones observed while learning the Q-network with the CQL algorithm. What about DQN, REM, and BRAC? Is the alleviation of rank collapse is specific to CQL+DR3?\n    * Looking at the simplified loss the authors based their intuition on, composed of 3 (maybe 4?) terms:\n      * 1. TD error: $||r + {\\phi^\\prime}^\\top w$ - $\\phi^\\top w||^2$;\n      * 2. CQL OOD samples regularization: ${\\phi^\\prime}^\\top w$ (at least);\n      * 3. Implicit regularization: $||\\phi||^2 - \\[\\[{\\phi^\\prime}^\\top\\]\\] \\phi$;\n      * 4. Potential L2 regularization on weight: $||w||^2$ (at least);\n      * It looks like adding the explicit regularization is an explicit way to increase rank, even if this regularization is more computationally efficient than Kumar et al. 2020. Am I missing something?\n    * Plotting the norms of $\\phi$, $\\phi^\\prime$, and their cosine distance with and without DR3 could be quite informative\n  * Comparing Figure 6 (resp. A.4) to Figure 5 (resp. A.3) is misleading the reader:\n    * At first sight it looks like the performances are somehow related to the srank not shrinking\n    * But at 50x62.5K steps the srank of CQL is higher than CQL+DR3 for Breakout for example.\n    * Why the Average Return of Asterix is missing (Figure 6)?\n    * Looking further at the results reported in Kumar et al. 2021, the authors are missing what seems like the interesting region w.r.t. their claims. It is only after more than 50x62.5K gradient steps that the performance drops appear. Why did the authors clipped the plots if the experiments were done for at least 200x62.5K gradient updates. This could help conclude about the training stability the authors claim reporting in the paper (e.g. p. 2 \" giving rise to methods that train for longer, reach a better solution, and remain stable at this solution \").\n    * Nevertheless I notice what looks like a significative improvement w.r.t. the average return\n  * Same goes for Figure 4\n    * The x-axis' scale for 1% Uniform Replay is different from the other ones, why? It looks like CQL+DR3 suffers a performance drop and that with more gradient step the performances with or without the explicit regularization would be similar.\n    * Is that the case for the 1% Uniform Replay experiment at 200x62.5K gradient updates?\n  * \"we evaluate DR3 in conjunction with CQL on the harder D4RL domains (antmaze, kitchen domains)\" (Section 6), \"we use the harder AntMaze and Franka Kitchen domains for evaluating CQL, since these domains are challenging for CQL.\" (Appendix E.2)\n    * W.r.t. which metric those domains are harder than the other ones? From Kumar et al. 2021: \"These tasks are especially challenging, since they require composing parts of trajectories, precise long-horizon manipulation, and handling human-provided teleoperation data\". In this case, does the potentially poor performances are correlated to the co-adaptive phenomenon? Reporting sranks after 2M gradient steps would help conclude.\n    * While trying to put emphasis on \"stability\", what looks like the main focus of the paper, I was expecting the authors to focus on environments for which previous methods suffer from performance drop and the under-parametrization phenomenon such as Ant-v2, Hopper-v2, and Walker2d-v2 as shown in Kumar et al. 2021 or hopper-medium-v0 as shown in Kumar et al. 2020b\n  * I do not understand why the effect of DR3 on D4RL is discussed using BRAC and not CQL nor REM. This would have had more impact since the paper mainly focus on the effect of DR3 w.r.t. CQL and REM. \n  * Overall a rigorous statistical significance test, to highlight significatively better results in the tables, is missing\n\n### Minor revisions/questions/remarks\n  * Does light colored regions on the plots always refer to 95% CI?\n  * Does the under-parametrization phenomenon and the drop in performances are specific to over-parametrized NNs?\n  * What optimizer the authors are using to optimize their models? This could be interesting since the theory developed in the paper assume an SGD optimization scheme.\n  * It would be better if all the algorithms are referenced in the main paper, e.g. COG has only its attached reference in the appendix (?)\n  * Writing \"This protocol is standard in Atari\" seems a bit far-fetched when the references point only to Google related papers\n  * What does bold format mean in the paper tables?\n\n### Some typos\n  * $\\theta$ missing for some gradients\n  * p. 12 \"A eoretical comparison\"\n  * p. 13 \"in ou practical\"\n  * p. 16 \"(Equation'4)\"\"\n  * p. 22 \"atleast\"\n  * p. 24 \"wprked\"\n\nAuthors answering remarks and questions during the rebuttal period might lead to a higher grade. But, as is, important claims of the paper are not well-supported and the way the results are presented is suspicious.", "summary_of_the_paper": "The paper propose to study the under-parametrization/co-adaptation effect (observed while optimizing over-parametrized networks with SGD) arising in TD learning based RL algorithms which looks correlated to performance drops. They prove an implicit regularization effect, following previous works focussing on supervised learning, that could explained the co-adaptation phenomenon. They derive an explicit regularizer and show its effects on plethora of experiments.", "summary_of_the_review": "The paper looks theoretically sound some questions remain regarding the experimental efficacy of the proposed regularizer claimed by the authors.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an interesting hypothesis about deep nets' generalization behavior inside RL methods: it suggests that the nets' implicit regularization favors a particular form of degeneracy, in which there is excessive aliasing of state-action pairs that tend to co-occur. It proposes a new regularizer to mitigate this problem. It evaluates the hypothesis and the regularizer empirically, and it provides suggestive derivations to motivate both.

The reviewers praised the comprehensive empirical analysis, the insights into learning, and the combination of empirical and theoretical evidence. The authors participated responsively and helpfully in the discussion period, and addressed any concerns raised by the reviewers.

This is a strong paper: it derives and motivates a novel hypothesis about an important problem, and analyzes this hypothesis both mathematically and experimentally.
Decision: Accept (Spotlight)