Efficient Deep Reinforcement Learning Requires Regulating Overfitting
Authors: Qiyang Li
Aviral Kumar
Ilya Kostrikov
Sergey Levine
Venue: ICLR 2023 poster (2023)
Keywords: Reinforcement Learning
Sample Efficient RL
Statistical Overfitting

Abstract: Deep reinforcement learning algorithms that learn policies by trial-and-error must learn from limited amounts of data collected by actively interacting with the environment. While many prior works have shown that proper regularization techniques are crucial for enabling data-efficient RL, a general understanding of the bottlenecks in data-efficient RL has remained unclear. Consequently, it has been difficult to devise a universal technique that works well across all domains. In this paper, we attempt to understand the primary bottleneck in sample-efficient deep RL by examining several potential hypotheses such as non-stationarity, excessive action distribution shift, and overfitting. We perform thorough empirical analysis on state-based DeepMind control suite (DMC) tasks in a controlled and systematic way to show that high temporal-difference (TD) error on the validation set of transitions is the main culprit that severely affects the performance of deep RL algorithms, and prior methods that lead to good performance do in fact, control the validation TD error to be low. This observation gives us a robust principle for making deep RL efficient: we can hill-climb on the validation TD error by utilizing any form of regularization techniques from supervised learning. We show that a simple online model selection method that targets the validation TD error is effective across state-based DMC and Gym tasks.

Published as a conference paper at ICLR 2023
EFFICIENT DEEP REINFORCEMENT LEARNING REQUIRES REGULATING OVERFITTING
Qiyang Li, Aviral Kumar, llya Kostrikov, Sergey Levine UC Berkeley
{gcli, aviralk, kostrikov, syleeiine@berreeeeyyco
ABSTRACT
Deep reinforcement learning algorithms that learn policies by trial-and-error must learn from limited amounts of data collected by actively interacting with the en- vironment. While many prior works have shown that proper regularization tech- niques are crucial for enabling data-efficient RL, general understanding of the bottlenecks in data-efficient RL has remained unclear. Consequently, has been difficult to devise a universal technique that works well across all domains. In this paper, we attempt understand the primary bottleneck in sample-effcciet deep RL by examining several potential hypotheses such as non-stationarity. excessive action distribution shift, and overfitting. We perform thorough empirical analysis state-based DeepMind control suite (DMC) tasks in controlled and system- atic way to show that high semporal-differeet of transitions is the main culprit that severely affects the performance of deep RL algorithms, and prior methods that lead to good performance do in fact, control the validation TD error to be low. This observation gives robust principle for making deep RL efficient: we can hill-climb on the validation TD error by uti- lizing any form of regularization techniques from supervised learning. We show that a simple online model selection method that targets the validation TD error is effective across state-based DMC and Gym tasks.
(TD) error on the validation set
INTRODUCTION
Reinforcement learning (RL) methods, when combined with high-capacity deep neural net function approximators, have shown promise in domains such as robot manipulation (Andrychowiez al., 2020), chip placement (Mirhoseini et al., 2020), games (Silver et al., 2016), and data-center cool- ing (Lazic et al., 2018). Since every unit of active online data collection comes at an expense (e.g., running real robots, chip evaluation using simulation), it is important to develop sample-efficient deep RL algorithms, that can learn efficiently even with limited amount of experience. Devising such efficient RI algorithm has been important thread of research in recent years (Janner al., 2019; Chen et al., 2021; Hiraoka et al., 2021).
In principle, off- off-cy RL methods (e.g., SAC (Haarnoja et al., 2018), TD3 (Fujimoto al., 2018), Rainbow (Hessel et al., 2018)) should provide good sample efficiency, because they make it possible improve the policy and value functions for many gradient steps per step of data collection. How- ever, this benefit does not appear to be realizable in practice, as taking too many training steps per each collected transition actually harms performance in many environments. Several hypotheses, such as overestimation (Thrun & Schwartz, 1993; Fujimoto et al., 2018), non-stationarities (Lyle al., 2022), or overfitting (Nikishin al., 2022) have been proposed as the underlying causes. Building on these hypotheses, several mitigation strategies, such as model-based data augmenta- tion (Janner et al., 2019), the use of ensembles (Chen et al., 2021), network regularizations (Hi- raoka et al., 2021), and periodically reseting the RL agent from scratch while keeping the replay buffer (Nikishin et al., 2022), have been proposed as methods for enabling off-policy RL with more gradient steps. While each of these approaches significantly improve sample efficiency, the efficacy of these fixes can be highly task-dependem (as we will show), and understanding the underlying issue and the behavior of these methods is still unanswered.

---

Published as a conference paper at ICLR 2023
In this paper, we attempt understand why taking more gradient steps can lead worse perfor- mance with deep RL algorithms, why heuristic strategies can help in some cases, and how this challenge can be mitigated in more principled and direct way. Through empirical analysis with the recently proposed tandem learning paradigm (Ostrovski et al., 2021), we show that in the early stages of training, TD-learning algorithms tend to quickly obtain high validation temporal-difference (TD) error (i.e., the error between the Q-network and the bootstrapping targets on a held-out val- idation set), and give rise to worse final solution. We further show that many existing methods devised for the data-efficient RL setting are effective insofar as they control the validation TD error to be low. This insight gives a robust principle for making deep RL, efficient: in order to improve data-efficiency, we can simply select the most suitable regularization for any given problem by hill- climbing on the validation TD error.
We realize this principle in the form of a simple online model selection method, that attempts to automatically discover the best regularization strategy for a given task during the course of online RL training, that we call Automatic model selection using Validation TD error (AVTD). AVTD trains several off-policy RL agents on a shared replay buffer where each agent applies a different regularizer. Then, AVTD dynamically selects the agent with the smallest validation TD error for acting in the environment. We find that this simple strategy alone often performs similarly or out- performs individual regularization schemes across wide array of Gym and DeepMind control suite (DMC) tasks. Critically, note that unlike prior regularization methods, whose performance can vary drastically across domains, our approach behaves robustly across all domains. To summarize, our first contribution is an empirical analysis of the bottlenecks in sample-efficient deep RL. We rig- orously evaluate several potential explanations behind these challenges, and observe that obtaining high validation TD-error in the early stages of training is one of the biggest culprits that inhibits performance of data-efficient deep RL. Our second contribution a simple active model selection method (AVTD) that attempts to automatically select regularization schemes by hill-climbing on validation TD error. Our method often matches or outperforms the best individual regularization scheme across wide range of Gym and DMC tasks.
2 PRELIMINARIES AND PROBLEM STATEMENT
The objective in RL is to maximize the long-term discounted return in a Markov decision process (MDP), (8, A, P,r,7), consisting of a state space S, an action space A, a transition dynamics model P(s'|s, (s a), a reward function r(s, a), and a discount factor 7 E [0, 1). The Q-function Q" (s. a) for a policy =(als) is the expected discounted reward obtained by executing action a at state s and following (a|s) thereafter, Q a) := E. 1E 2'r(s,, a.)]. The optimal Q-function is achieved when it satisfies the Bellman equation: Q" a) = Eg [r(s, a) + y maxa Q* a a")]- Practical off- aff-policy methods (e.g., Mnih et al., 2015; Hessel et al., 2018: Haarnoja et al., 2018) train a Q-network, Qo (parameterized by 0), to minimize the temporal difference (TD) error:
L(0)- = E,,.,.5 [(r(s, a) + 7Q(s' (s))
(1)
where D is the replay buffer consisting of the transitions (s, a, s') collected so far, Q is the target Q- network that is often updated to follow the Q-network Qa with delay or smoothing (Fujimoto et al., 2018) so that the target does not move too quickly, and a' is usually drawn from a policy и (als)) that can maximize or approximately maximize Qo a). In theory, these off-policy algorithms can be made very sample efficient by minimizing the TD error fully over any data batch, which in practice translates to making more update steps to the Q-network per environment step, or higher "update- to-data" ratio (UTD) (Chen et al., 2021). However, when done naïvely, this can lead to worse performance (e.g.. DMC (Nikishin et al., 2022) and on MuJoCo gym (Janner et al., 2019)).
There have been many prior methods proposed for dealing with high UTD issues (e.g.. DroQ (Hi- raoka et al., 2021), REDQ (Chen et al., 2021), and resets (Nikishin et al., 2022)). However, we find that none of these prior methods and other simple baseline regularization schemes such as weight decay, dropout and spectral normalization (Miyato al., 2018) work well across all the tasks (see Appendix A, Figure 4). What is the primary culprit that can explain the high UTD challenge? Can address it in a more direct and principled way?
2

---

Published as a conference paper at ICLR 2023
3 THE PRIMARY CULPRIT BEHIND FAILURE OF HIGH UTD DEEP RL
In this section, we attempt to understand the underlying causes behind the failure of off-policy RL algorithms in the high UTD deep RL and whether prior sample-efficient RL algorithms have addressed these problems appropriately. We examine several plausible hypotheses that prior works posit: Q-value overestination due to distribution shift (Fujimoto et al., 2019; Kumar et al., 2020), non-stationarity due changing data distributions (Lyle et al., 2022), well as early overfitting to the replay buffer (Nikishin et al., 2022). We first describe the setup for our empirical analysis. Then, in the next section, we demonstrate through controlled study that the aforementioned hypothesized reasons are not sufficient to explain the challenges with high UTD. Then, by demonstrating that high UTD deep RL usually results in high generalization gap in the TD error, we argue that the main culprit behind the failure mode of high UTD learning is the high validation TD error. We validate this hypothesis by evaluating some recently proposed regularizers, and show that these regularizers are effective insofar as they control the validation TD error to be low.
Experimental setup. We first describe the setup for our analysis. Many of the experiments in our empirical study utilize passive sources of data obtained from previous online RI. runs. We replay this data in different ways to control for and examine various hypotheses. For generating this logged data, we utilize one run of resetting SAC agent from Nikishin et al. (2022), trained with a UTD value of 9. We amalyze a standard SAC agent in the high UTD regime. Since we operate in the offline regime, to stabilize TD learning, we additionally normalize the features of the last layer (following prior works on TD stability (Bjorck et al., 2021a; Kumar et al., 2021a)). We refer this as feature normalization (FN). We added FN in the last layer of the Q-network in our analysis except DroQ, as it already utilizes LayerNorm. For fair comparisons, we use feature normalization in all the experiments including the online setting in this section. While we keep most of the hyperparameters the same, there are still some small differences between the online and offline settings. In the online setting, we use the standard SAC which uses entropy backup in the bellman update. In the offline setting, we remove the entropy term in the bellman backup and use deterministic backup (the mean of the action from the Gaussian actor is used). Our analysis focuses on the fish-swim environment from DMC suite since high UTD training results in the largest gap in this domain (see the online column in Appendix C, Figure 8). We obtain similar trends for many other experiments; complete set of our analysis results are in Appendix C. The confidence interval in our performance curves refers to the standard eπror computed over 8 random seeds. See implementation details about different regularizers in Appendix B.
3.1
CAN POOR DATA COLLECTION, DISTRIBUTION SHIFT OR NON-STATIONARITTY EXPLAIN THE FAILURE OF HIGH UTD LEARNING?
First observe that, expected, the performance of standard SAC agent degrades the UTD value increases (Figure 1 1-left). We will now attempt to understand if this performance degradation can be attributed to (a) poor data collection, (b) excessive action distribution shift and overestimation in the Q-function or (c) non-stationarity of the replay buffer.
(a) Quality of data used for training. One might speculate that SAC behaves poorly in the high UTD regime due to its inability to effectively collect exploratory data. To understand this might be the primary source of issues in high UTD learning, we analyze the behaviors of RL methods with high UTD in an offline setting, when training on identical data. If indeed the negative effects of higher UTD ratios are entirely due to exploration and data collection, we would expect this change to greatly mitigate the bottlenecks with higher UTD ratios. In this study, we trained SAC with different UTD values on the aforementioned logged dataset, but replayed the data sequentially (following the tandem learning protocol (Ostrovski et al., 2021)). This approach mimics how a typical online RL agent would gradually observe data as it explores, but here, the dataset itself is independent of the agent being trained. We refer this setting the offline streaming setting. As shown in Figure I-middle left, the performance of SAC still degrades as UTD increases, even though the data comes from a high performing agent. This suggests that the poor data collection alone does not explain the failure of high UTD learning. Since, this setting still preserves the challenges of learning from data, including the effect of the training distribution and data quantity, we investigate these next.
(b) Non-stationarity in replay buffer data distributions. Another potential explanation for the issues in learning with high UTD is non-stationarity: with higher UTD, the algorithm makes more
3

---

Published as a conference paper at ICLR 2023
UTD=20
UTD=9
UTD=3
UTD=1
Offline Shuffled Streaming
Q-value gap (Shuffled Streaming)
Online
Offline Streaming
600
0.2
Return
400
200
0
0.0
0.0
0.5
1.0 0.0
0.5 1.0 0.0
0.5
1.0
0.0
0.5
1.0
Environment Steps (x10*)
Figure The effects of varying UTD ratios on the performance of SAC agents augmented with feature normalization on fish-swim task under online (left), offfine streaming (middle left), and offline shuffled streaming (middle right) settings. The plot on the right shows the AQ in the shuffled streaming setting. Typical offline RI. issue of being over-optimistic Results in other tasks in Figure 14, where find identical small gaps for UTD=1, and 9.
OOD actions does not appear in the regime we study.
FN +
UTD=20
UTD=3
+ Reset
UTD=9
UTD=1
FN WD
DroQ
600
Return
400
Train. TD
1.00
0.10
0.01
Valid. TD
1.00
0.10
0.01
0.0
0.5
1.0
0.0
0.5
1.0
Environment Steps (x10%)
Environment Steps (x10%)
(a) (UTD). Higher UTD leads to lower performance (top), lower training TD error (middle, except for UTD=20), and higher validation TD error (bottom).
(b) (Regularizers). Different regularization methods reduce the validation TD error. Vali- dation TD error correlates with the return better than training TD error.
Figure 2: Diagnostic analysis on fish-swim under the offline shuffled streaming. All agents use feature normalization in the last layer to stabilize TD learning. The evaluation of the TD error is done on the growing trining/heldout replay butfer (collected by the online SAC agent that resets periodically during training).
gradient updates on the learned policy, allowing the distribution of the learned policy to change more drastically between iterations of learning. Sudden changes in the data distribution and non- stationary target values of this sort have been regarded as challenges in online RL (Igl et al., 2020). As before, to understand if high UTD challenges arise from non-stationarity. we construct an exper- iment that removes this factor. We rerun SAC with different UTDs in the offline streaming setting
4

---

Published as a conference paper at ICLR 2023
Method
AQ
MC Bias
Train. TD
Valid. TD
Avg. Rank on fish-swim
2.278 + 0.017
2.224 1 0.033
2.436 + 0.045
1.360 + 0.062 1.605 = 0.052
Avg. Rank on DMC tasks
1.819 = 0.044
1.697 + 0.044
1.701 = 0.050
Table I: Average performance rank selected based on different metrics across 7 DMC tasks. The model selection done among four different UTDs (1. 3. 9, and 20). The rank value for each method represents the rank of the method selected by the corresponding metric in terms of its evaluation return. The rank value ranges from to with being the best and being the worst. The Avg. Rank is first averaged over environment steps (in an interval of 5000 steps), and then the resulting values are used to compute the mean and the standard error in the table (across both seeds and the environments).
from above, but now also reshuffle the buffer before training. That is, while the previous streaming setting replayed data in order it was collected by the online RL agent, this new setting presents data sequentially, but not in the same order that it was collected. This ensures that the data distribution of samples used for training stationary and does not change over the course of training. We refer this the offline shuffled streaming setting. Note however that the underlying RL algorithm still observes new data points as it trains for longer. We still observe a similar performance trend in this shuffled streaming setting, as shown in Figure 1-middle right, indicating that -tn-stationarity of the data distribution alone also does not explain the failure of high UTD learning.
(c) Distribution shift and dittr--diittibution (OOD) actions. Our analysis far suggests that the challenges in learning with high UTD are related to effective learning from passive data: even when the data quality and non-stationarity are accounted for, the performance with high UTDs is worse. One might speculate that an obvious challenge for learning from data is action distribution shift or OOD actions (Fujimoto et al., 2019; Levine et al., 2020: Kumar et al., 2019): higher UTDs require more off-policy Bellman backups, resulting in backups from OOD actions, and Q-value overestimation (Thrun& & Schwartz, 1993; Van Hasselt et al., 2016; Fujimoto et al., 2018). Note that overestimation in the Q-function in general stems from multiple sources such as imperfect mini- mization of Bellman error, optimism due to OOD actions, and constraints imposed by the function class. In this study, our goal is to investigate if OOD actions are issue in the high UTD regime. Hence, we plot the gap in Q-values at actions chosen by the policy and the actions in the dataset: AQ = Eg-D
(Q a²) B - ( a²) (which measures how overestimated are the Q-values due to action distribution shift following (Kumar et al., 2020)). By inspecting the Q gap (see Figure 1, rightmost for fish-sw swiswwmmmmmm and Figure 14 for other environments), we find that for UTD=113,9, the gaps are generally very similar to each other and for UTD=200, the gaps are higher but still controlled. Despite the similarity in the Q gaps for UTD=1,3.9, the performance is widely different (see Figure 10, the leftmost column). Therefore, the performance degradation from ramping up the UTD cannot be explained due to action distribution alone. Note that we are not claiming that action distribution shift is not a problem in general, but that our evidence shows that in the high UTD online RL settings that we study, distributional shift does not explain the performance difference. This is perhaps expected as our dataset contains all the experience from an online RI. run, and is hence of high coverage.
3.2 CAN OVERFITTING EXPLAIN THE FAILURE OF HIGH UTD LEARNING?
Even after using data of high quality obtained from the run of resetting SAC, and after correcting for non-stationarity and distribution shift, we find that the challenges with high UTD RL still remain. This hints the possibility that the actual underlying issue is some form of overfitting. This moti- vates us to measure the TD errors on held-out validation dataset and observe its relationship with the high UTD failture cases.
As shown in Figure 2a, the validation TD error tends to be correlated with the failure cases with high UTD. Note that the validation TD error also correlates well with the increase of UTD ratio. While this observation hints that the overfitting might be the root cause of the high UTD failure, it is hard to precisely quantify the amount of overfitting in TD learning due to the dynamic nature of the training objective. It is possible that this seemingly overfitting issue co-exists with/or is caused by another cause that we have not uncovered. Nevertheless, compared to existing metrics such as the Q-gap and the Q-function estimation bias relative the true Q-function (Fujimoto al., 2018;
5

---

Published as a conference paper at ICLR 2023
Chen al., 2021; Wang al., 2021). validation TD error is relatively more robust indicator for the high UTD failure. See Table 1 for a summary based on the performance rank. The validation TD error is especially effective empirical evidence that the validation TD error is a good indicator for the high UTD failure on Fish-swime m, but how about other environments? It turns that most other DMC tasks that suffer from the high UTD learning issue also exhibit the same trend (see Figure 10 in Appendix C). This suggests that the biggest challenge that needs be handled in such data-efficient deep RL settings is controlling validation TD error.
fish-swim (more details in Appendix C.I). We have shown
3.3 CAN MITIGATING HIGH VALIDATION TD ERROR EXPLAIN THE GOOD PERFORMANCE OF PRIOR REGULARIZERS?
We provided empirical evidence in the previous section that obtaining high validation TD error often correlates well with the failure of naive RL methods with high UTD, compared to a number of other previously hypothesized explanations//etriis. In this section, we attempt to understand the performance improvements from a variety of previously proposed regularizers, can be attributed to their effectiveness in controlling the validation TD error We note that none of the methods we study enable high UTD learning across all tasks (as we have previously discussed), instead focuses on understanding whether methods that work well in each setting also achieve lower validation TD errors. These regularizers include dropout (Gal & Ghahramani, 2016) (DO, used by Hiraoka et al. (2021)), weight decay (Loshchilov & Hutter, 2017) (WD, used by Lillicrap et al. (2015)), spectral normalization (Miyato et al., 2018) (SN, used by Gogianu et al. (2021); Bjorek et al. (2021b)), periodic resets (Nikishin et al., 2022), and a combination of LayerNorm (Ba et al., 2016) and dropout (DroQ (Hiraoka et al., 2021)). All of these regularizers operate differently: dropout injects stochasticity into the Q-network, weight decay controls the parameter norm; spectral normalization controls the maximum singular value of the weight matrix. We observe that SAC exhibits lower validation TD errors when trained with these regularizers in the high UTD regime (as shown in Figure 2b for fish-swin swiim and Appendix C, Figure 13 for other environments). This further highlights that a reduction in the valiclation TD error does correspond to better performance. We would also highlight that the regularizers that achieve the lowest validation TD) error offline are usually one of the top performing methods online (Figure 5).
our study
4 AUTOMATIC MODEL SELECTION BASED ON VALIDATION TD (AVTD)
The performance of various regularization methods above indicates that no single regularizer per- forms well on all the tasks. More so, it is also unreasonable to expect that a single regularizer would perform well every deep RL problem. However, if allows us to automatically identify a good regularization approach from among a set of candidate approaches, we would expect such a principle to perform well given a broad set of regularization methods. Previously, we observed that the validation TD error of different regularization approaches correlates well with performance in the offline setting. Can we somehow use this correlation to our advantage and select the best regularization approach automatically?
devise general principle that
A naive approach that directly follows from our analysis would train multiple independent agents with different regularizers in parallel, for a small number of initial steps, then select the one with the smallest validation TD error and use for the rest of training. While intuitive, this approach may not necessarily work: TD error depends on the scale of the reward function, and typically as an online RL agent makes progress towards maximizing reward and observes higher reward value, TD error increases. This means that this naive approach will select the agent that has made the least progress towards maximizing reward as it is likely to be the undesirable one that attains the smallest TD error. To address this shortcoming, we consider a simple modification of this idea: instead train multiple agents with different regularizers on a shared replay buffer, such that the data collection does not confound the evaluation of TD error.

---

Published as a conference paper at ICLR 2023
Algorithm 1 AVTD
Input: A collection of off-policc RL agents {(Qg. mg), ... (Q/ m ) greedy exploration coefficient = 0.1
2: for each environment step do
3: With probability E, +- arg max; L(0;; Dheldout). Otherwise, j - Unif([K])
4: Sample action from Ty and use it to act in the environment
5: Add the new transition in the replay buffer: D +- DU (8,,,,
6: for i = 1... K do
7: Update Q3 and Ti using the replay buffer D
8: end for
9: After every 10 episodes, collect heldout trajectory and add to Dheldout with the same action
selection strategy above for 'D
10: end for
At each environment step, AVTD picks the agent with the lowest validation TD error to take actions in the environment. Essentially, all the agents are utilizing the same buffer, similar the offline streaming setting, except that the active agent (the agent that is taking action currently) collects data that goes into the replay buffer. As we will show, this does not reduce the correlation between the validation TD error and the performance, and this selection strategy can reliably select the best performing algorithm without incurring the additional sample complexity that might result from running multiple learners in sequence to pick the best one. An overview is shown in Algorithm 1.
EXPERIMENTAL EVALUATION OF AVTD
The goal of experiments is validate the principle that hill-climbing validation TD error mitigate statistical overfitting can improve performance in data-effcient deep RI. To this end, we evaluate our active model selection method, AVTD along with strategies for comparisons. Through experiments, we will establish that automatically selecting the regularization strategy (or strength) via AVTD is able to match or outperform the best individual strategy. Concretely, we will answer the following questions: (1) Is AVTD able to select the best (2) How important is utilizing a valida- tion set in AVTD?, and (3) Does AVTD match or improve the performance over the best performing regularization approach it must select from? We first present answers to questions (1) and (2) and then present our final results in (3). Implementation details are in Appendix B.
regularization
regularization coefficient online out of set of candidates?,
(1) Is AVTD able to select the best regularization strength online? To answer this question, we use five DroQ agents with different dropout rates (0.003, 0.01, 0.03, 0. 0.0) and evaluate if AVTD able to select the best dropout rate for each task independently. We show in Figure 16a that AVTD can reliably match the performance of the best regularizer on the four Gym tasks train on, without apriori knowing which coefficient would perform well. One might argue that simply training ensemble of these coefficients could achieve similar effect. We show that this is not the case in Figure 16b by comparing to the uniform selection strategy where randomly selected agent used act in the environment for any given rollout.
(2) Is there any benefit to specifically using validation TD error in AVTD? To answer this ques- tion, we performed a study where we automatically adjust the regularization strategy by hill climbing on the training TD error instead of the validation TD error. On fish-swin, we observe that uti- lizing validation TD error critical, and training TD error leads to worse performance (Figure 17). Qualitatively, we observe that the regularization strategy selected by the hill climbing on training TD error is the one that does not add any regularization, resulting in worse performance. This demon- strates that the principle of hill climbing on the validation TD error is more robust than hill climbing on the training TD error, further corroborating the insights from our empirical analysis.
(3) Can AVTD match or exceed the performance of individual regularizers that it dynami- cally selects from, across a wide range of tasks? To evaluate overall performance of AVTD in comparison with each individual regularizer, we evaluate AVTD, individual regularizers, and other prior methods on DMC tasks and MuJoCo Gym tasks. The comparative evaluation of AVTD and other methods is shown in Figure 3. For AVTD,
combination of five regularization

---

Published as a conference paper at ICLR 2023
Oracle
Ours
SAC (UTD=1)
Reset
WD=0.01
DroQ=0.03
Aggregated
Ant-v4
Walker2d-v4
Hopper-v4
Humanoid-v44
Normalized Score
6000
0.8
2000
0.0
0.2
0.0
0.2
0.0 hopper-hop
0.2
0.0
0.6
cheetah-run
acrobot-swing swingup
hopper-stand
0.4
Return
8.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5 fish-swim
1.0
0.0
0.5
1.0
0.0
0.0
0.2
1000 humanoid-stand
humanoid-ru
quadruped-run
finger-turn hard
500
0.0
1.0
0.5
0.5 1.0
1.0 0.0
1.0
Environment Steps (x106)
Figure 3: AVTD comparing to the Performance of various sample-efficient RI. methods based SAC on Gym (top row) and DMC tasks (bottom rows). Oracle: the best method (selected in hindsight by the average return over the first 300K environment steps). account for the additional samples utilized by any model selection method for constructing the validation set; SAC: the standard SAC agent trained with UTD ratio of I; Reset: the standard SAC agent trained with high UTD ratios (9 on DMC and 20 on Gym) and resets periodically (after every 200K/100K steps DMC/Gym tasks); WD=0.01: the standard SAC agent trained with the high ratios and regularized with weight decay in the Q-network; DroQ (Hiraoka al., 2021). AVTD performs more reliably across the board, often matching the top performing method on each environment. We use representative baselines in this plot; See Appendix C.4, Figure 18 for the full results with all baselines and the aggregated normalized score computation protocol.
scale x-axis for this oracle approach by to
strategies: LayerNorm, LayerNorm + WD with weight of 0.01. WD with weight of 0.01 alone, and LayerNorm + Dropout with fractions 0.03 or 0.01. This set of regularizers includes least one regularizer that performs well on each of the tasks (e.g., DroQ regularizers work well on Gym tasks; weight decay works well on DMC tasks). However, no single strategy performs well across all the tasks. The goal of AVTD is to identify the best performing method, and preferably improve each case. To interpret the performance of AVTD, we also plot an oracle upper bound, that identifies the best regularization method for every task with unlimited on the fly access to the run of each regularizer. AVTD frequently matches the best performing method (with the exception of the DMC acrobot winu task) and, in the case of the hardest task (humanoid) rurun, outper- forms prior methods, indicating that selecting the regularizer based on validation error is consistently effective. Observe that AVTTD closes the gap between the best individual regularizer (in this case, WD=0.01) and this oracle approach by 30%, indicating that it is effective.
Method
LN
IN
WD+0.01
DroQ=0.03
DroQ-0.01
AVTD
Aug. Ronk
3.346 1 0.104 3.317
0.085
3.490
0.118
4.125 1. 0.131
3.798
0.154
2.923 1. 0.110
Table 2:
Average performance rank over the first 300k steps. The rank value for each method ranges from to with being the best and being the worst. The Avg. Rank averaged over both environments (4 Gym DMC tasks) and environment steps (first 3 10 steps). Standard error is computed over random seeds.
To further understand the efficacy of AVTD in selecting the best regu- larizer, we rank each of the individual regularization techniques and AVTD in terms the average return obtained in the first 300K steps for all the tasks and compute the average rank attained by
Average algorithm rank:
8

---

Published as a conference paper at ICLR 2023
ery approach. A more effective regularization approach across the board should attain a smaller rank (ideally, close to 1.0). As shown in Table 2, all prior regularizers exhibit statistically overlapping values of average rank, indicating that none of the methods is the best across the board. On the other hand, AVTD attains an average rank of 2.923, improving over all the other methods significantly. This indicates the efficacy of AVTD in attaining good performance across the board.
RELATED WORK
In image-based RL domains, prior works have identified overfitting issues (Song et al., 2019) and found data augmentation to help (Kostrikov et al., 2020; Yarats et al., 2021; Raileanu et al., 2021). Cetin al. (2022) identified "seelf-overfitingg" issue that caused by TD learning with a convolu- tional encoder and low magnitude rewards. Our work is distinct as we mainly focus on state-based tasks with mostly dense rewards. Overfitting is also studied in offline RL (Kumar et al., 202 202lb; Arnob et al., 2021; Lee et al., 2022), and while we do run some analysis in offline RL, we follow the tandem learning protocol (Ostrovski et al., 2021), where the offline dataset is generated via an active RL agent and is not arbitrary distribution. In the sample-efficient deep RL setting that we study in this paper, Nikishin et al. (2022) observed that forcing the TD learning to fit on limited initial data with many gradient steps can hinder the learning progress later on in the training. This prior work speculates that this observation could be due to some "overfittingllkke" phenomenon. This is different from our work it does not utilize a held-out validation set, and our validation TD error metric cannot be used to directly conclude the existence of overfitting, but rather as an indicator for the high UTD failure. Finally, our analysis also examines the feasibility of various other hypotheses (e.g., non-stationarities of the replay buffer (Lyle et al., 2022; Igl et al., 2020), action distribution shift (Fujimoto et al., 2019), value under/over-estiman Wang et al., 2021)) towards explaining issues in data-efficient deep RL. We find that the validation TD error correlates with the performance better than other metrics that we examine.
(Fujimoto et al., 2018; Chen et al., 2021;
Regularization in deep RL. Regularization schemes, such as dropout (Gal & Ghahramani, 2016), layernorm (Ba et al., 2016), or batchnorm (Cheng et al., 2016) have been effective in improving the sample efficiency of deep RL algorithms. For example, Hiraoka et al. (2021) 1) uses dropout and lay- ermorm on top of SAC to attain near state-of-the-arr performance on MuJoCo gym Liu et al. (2019) actors can improve both on-policy and off-policy RL algo-
found that L2 weight regularization rithms. Nikishin et al. (2022) propose periodic resets of critic weights, and can also be interpreted as form of regularization. Despite the empirical successes of these regularization approaches, the understanding of the principle behind these regularization approaches is lacking. Our analysis sheds light on the connection of these regularization approaches to statistical overfitting. We also observe that the efficacy of these regularizers is quite domain dependent: not all regularizers work in all domains (see Figure 4). On the contrary, we do not propose yet another regularizer, but a method to select from among regularizers. Our method, AVTD, is also related to theoretical algorithms that attempt to do online model selection (Foster et al., 2019; Lee et al., 2021; Cutkosky et al., 2021), that study this problem from theoretical perspective. Khadka et al. (2019) also utilizes multiple learners and actively selects from them based on an estimate of return, distinct from validation TD-error.
7 DISCUSSION
In this work, we attempted to understand the primary bottlenecks in data-efficient deep RL. Through rigorous empirical analysis, we showed that poor performance in high UTD deep RL is often correlated with high validation TD error, and the effectiveness of many existing regularizers can be explained by their ability to control the validation TD error. We use this experimental design to devise a principle for obtaining sample-efficient deep RL: by targeting this metric by an active model selection strategy, that automatically adjusts regularization based on validation -D-error, we can often match the best and outperform existing regularizers on each task, achieving better overall performance. While AVTD can work well across a number of domains, several important questions remain. For instance, it is not clear why and when certain regularization strategies work better than others. If we can answer this question, we can optimize for validation TD error in a more straightforward fashion without requiring multiple parallel agents. This likely would require understanding the learning dynamics of TD-lcarning which is an interesting topic for future work. Reducing the computational cost of our method is also an interesting avenue for future work.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We would like thank the anonymous reviewers and Zhixuan Lin for feedback and discussion on OpenReview. We are thankful of Philip Ball, Chuer Pan, and the members of the RAIL lab for feedback and suggestions on the early drafts of the paper.
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural informa- tion processing systems, 34:29304-2995 293200, 2021.
OpenAl: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
Samin Yeasar Arnob, Riashat Islam, and Doina Precup. Importance of empirical sample complexity analysis for offline reinforcement learning. arXiv preprint 12. 2021.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint arXiv: 1607.06450, 2016.
Johan Bjorck, Carla P Gomes, and Kilian Q Weinberger. Is high variance unavoidable in ri? case study in continuous control. arXiv preprint 10.11:2110.11222, 2021a.
Johan Bjorek, Carla P Gomes, and Kilian Q Weinberger. Towards deeper deep reinforcement learn- ing. arXiv preprint arXiv:2106.0/ 151, 2021b.
Edoardo Cetin, Philip Ball, Steve Roberts, and Oya Celiktutan. Stabilizing off-policy deep rein- forcement learning from pixels. arXiv preprint arXiv:2207. 00986, 2022.
Xinyue Chen, Che Wang, Zijian Zhou, and Keith W. Ross. Randomized ensembled double q- learning: Learning fast without model. In International Conference Learning Representa- lions, 2021. URL https::////. wwwnreeiiieeeeeeee net/ d Zm0
Cheng, LY Lo, NLS Tang, et Crossnorm: novel normalization strategy for microarray data in cancers. sci rep 6: 18898, 2016.
Ashok Cutkosky, Christoph Dann, Abhimanyu Das, Claudio Gentile, Aldo Pacchiano, and Manish Purohit. Dynamic balancing for model selection in bandits and rl. In International Conference on Machine Learning, pp. 2276-2285. PMLR, 2021.
Dylan J Foster, Akshay Krishnamurthy, and Haipeng Luo. Model selection for contextual bandits. Advances in Neural Information Processing Systems, 32, 2019.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning (ICML), pp. 1587-1596, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference Machine Learning, pp. 2052-2062, 2019.
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059, 2016.
Florin Gogianu. Tudor Berariu, Mihacla C Rosca, Claudia Clopath, Lucian Busoniu, and Razvan Pascanu. Spectral normalisation for deep reinforcement learning: an optimisation perspective. In International Conference Machine Learning, pp. 3734-3744 PMLR, 2021.
Tuomas Haarojja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- on machine learning, pp. 1861-1870. PMLR, 2018.
10

---

Published as a conference paper at ICLR 2023
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-second AAAI conference artificial intelligence, 2018.
Takuya Hiraoka, Takahisa Imagawa, Taisei Hashimoto, Takashi Onishi, and Yoshimasa Tsu- ruoka. Dropout q-functions for doubly efficient reinforcement learning. arXiv preprint 10.00221 2021.
Maximilian Igl, Gregory Farquhar, Jelena Luketina, Wendelin Boehmer, and Shimon Whiteson. The impact of non- -on-stationarity on generalisation in deep reinforcement learning. arXiv preprint NXiv:2006.008266 2020.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. In Advances in Neural Information Processing Systems, pp. 12498- 12509, 2019.
Shauharda Khadka, Somdeb Majumdar, Tarek Nassar, Zach Dwiel, Evren Tumer, Santiago Miret, Yinyin Liu, and Kagan Tumer. Collaborative evolutionary reinforcement learning. In Interna- tional conference on machine learning, pp. 3341-3350. PMLR, 2019.
lyya Kostrikov. JAXRL: Implementations of Reinforcement Learning algorithms in JAX, 10 2021.
URL https: /g thub. cwm/ http:/ikostrrioooooo jaxrl.
Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004.13649 2020.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Sys- tems, pp. 11761-11771, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.04777 2020.
Aviral Kumar, Rishabh Agarwal, Tengyu Ma, Aaron Courville, George Tucker, and Sergey Levine. DR3: Based Deep Reinforcement Learning Requires Explicit Regularization. arXiv preprint arXiv:2112.04716 2021a.
Aviral Kumar, Anikait Singh, Stephen Tian, Chelsea Finn, and Sergey Levine A workflow for offline model-free robotic reinforcement learning. 5th Annual Conference on Robot Learning, 2021b. URL httph:tpp///// httpp:::://ppeeeeeeeeete
weteeeewweeeeeeeeeeeeeeeeeeeeeee forrmm? id= f 4y4Byxxxb.......
Nevena Lazic, Craig Boutilier, Tyler Lu, Echern Wong, Binz Roy, MK Ryu, and Greg Imwalle. Data center cooling using model modei-predietive control. Advances in Neural Information Processing Systems, 31, 2018.
Jonathan Lee, Aldo Pacchiano, Vidya Muthukumar, Weihao Kong, and Emma Brunskill. Online model selection for reinforcement learning with function approximation. In International Con- ference on Artificial Intelligence and Statistics, pp. 3340-3348. PMLR, 2021.
Jonathan Lee, George Tucker, Ofir Nachum, and Bo Dai. Model selection in batch policy optimiza- tion. In International Conference on Machine Learning, pp. 12542 -25229 PMLR, 2022.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. Interna- tional Conference on Learning Representations (ICLR), 2015.
Zhuang Liu, Xuanlin Li, Bingyi Kang, and Trevor Darrell. Regularization matters in policy optimization-an an empirical study on continuous control. arXiv preprint arXiv:1910009191,, 2019.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1rXiv: 2017.
11

---

Published as a conference paper at ICLR 2023
Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in rein- forcement learning. arXiv preprint arXiv:2204.09560, 2022.
Azalia Mirhoseini, Anna Goldie, Mustafa Yazgan, Joe Jiang, Ebrahim Songhori, Shen Wang, Young-Joon Lee, Eric Johnson, Omkar Pathak, Sungmin Bac, et al. Chip placement with deep reinforcement learning. arXiv preprint arXiv:2004. 10746, 2020.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization 2018.
for generative adversarial networks. arXiv preprint
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 51 518(7540):529-533. feb 2015. ISSN 0028-0836.
Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International Conference on Machine Learning, pp. 16828-16847 16847. PMLR, 2022.
Georg Ostrovski, Pablo Samuel Castro, and Will Dabney. The difficulty of passive learning in deep reinforcement learning. Advances in Neural Information Processing Systems, 34:23283-23295, 2021.
Roberta Raileanu, Maxwell Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus. Automatic data augmentation for generalization in reinforcement learning, Advances in Neural Information Processing Systems, 34:5402-5415
2021. 2021.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwieser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game go with deep neural networks and tree search. nature, 529(7587):484-888 2016.
Laura Smith, llya Kostrikov, and Sergey Levine. A walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning. arXiv preprint arXiv: 078600 2022.
Xingyou Song, Yiding Jiang, Stephen Tu, Yilun Du, and Behnam Neyshabur. Observational over-
fitting in reinforcement learning, arXiv preprint arXiv:1912.0220 2019.
Sebastian Thrun and Anton Schwartz. Issues in using function approximation for reinforcement learning. Proceedings of the 1993 Connectionist Models Summer School Hillsdale, NJ. Lawrence Erlbaum, volume 6, pp. 1-9, 1993.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
Hang Wang, Sen Lin, and Junshan Zhang. Adaptive ensemble q-learning: Minimizing estimation bias via error feedback. Advances in Neural Information Processing Systems, 34:24778-24790, 2021.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous con- trol: Improved data-augmented reinforcement learning. arXiv preprint arXiv:1107009655, 2021.
12

---

Published as a conference paper at ICLR 2023
Appendices
A
FAILURE CASE OF EXISTING REGULARIZERS
(UTD=1)
(High UTD)
WD-0.01 (High UTD)
DroQ=0.01 (High UTD)
Humanoid-vd
Ant-v4
qua druped--uu
fish-swim
6000
600
5000
5000
800
500
400
3000
3000
300
2000
200
1000
1000
200
300
0.0
0.1
0.2
0.3
6.0
0.1
0.2
0.3
0.00
0.25
0.50
0.75
1.00
6.00
0.25
0.50
0.75
1.00
Enwtronment
Environment Stops 1K10
Enviromment Steps Ix10")
Environment Stops
Figure 4: Failure cases for commmon sample-efficient RL methods across DMC and MuJoCo gym benchmark. DroQ is the state-of-the-art method on gym tasks, Reset one the top performing algorithm on DMC from Nikishin et al. (2022) that utilizes resets, and WD=0.01 a simple regularization baseline that we study in our work that utilizes weight decay on the Q-network. While DroQ and Resets attain good performance the Gym tasks, they perform poorly the other set of tasks from the DMC suite. In contrast, weight decay performs well the DMC tasks attains poor performance Gym.
B IMPLEMENTATION DETAILS
AVTD. In all our experiments, we use 0.1 unless specified otherwise. This means that at cach environment step, there is 10% chance that random agent is picked and the action is sampled from that random agent. One held-out episode of transitions is collected after every 10 episodes where the actions picked with the same strategy above. These environment steps also counted towards the number of steps taken by the AVTD for fair comparisons.
Weight decay (WD). For all our experiments with weight decay, we apply AdamW (Loshchilov & Hutter, 2017) the weight matrices of the Q-network (not bias) except the last layer of the network (that maps the last layer feature to scalar). When LayerNorm and weight decay are used together, weight decay is applied the bias and the scale learned in the LayerNorm. Unless specified otherwise, we use a weight decay coefficient of 0.01.
Spectral normalization (SN). For spectral normalization, we follow the implementation of Gogianu et al. (2021) where we use I power iteration to keep running estimate of the singular vector (that corresponds to the largest singular value) and backpropagate through the norm. We fol- low the best-performing setting Gogianu al. (2021) where SN is only applied the penultimate layer of the Q-network (the layer that has the 256 x 256 weight).
LayerNorm (LN), For all our experiments with LayerNorm (Ba et al., 2016), we use it right before each ReLU activation the Q-network and learns additional per- feature-element scales and biases.
Feature Normalization (FN). For all our experiments with feature normalization, we use it right before the last layer of the Q-network where is parameterized
Qo a) = w T (s)
[s, a)]l2
where is the last layer weight of the Q-network and fo a) gives the post-activation feature right before the last layer. This trick has been applied in many prior works to improve the stability of TD learning (e.g., (Bjorck et al., 2021a Kumar et al., 2021a)).
13

---

Published as a conference paper at ICLR 2023
Initial Temperature
1.0
Target Update Rate
update rate target networks
0.005
Learning Rate
learning rate for the Adam optimizer
0.0003
Discount Factor
0.99
Batch Size
256
Network Size
(256, 256)
Warmup Period
of initial random exploration steps
10000 for DMC, 5000 for gym MuJoCo
Table 3: Hyperparameters used for the SAC algorithm (Haarnoja et al., 2018)
Dropout (DO). For all our experiments with dropout, we apply it in the Q-network before the ReLU activation (before LayerNorm when combined together, e.g., DroQ (Hiraoka et al., 2021)). Unless specified otherwise, we use a dropout rate of 0.03 for DO and 0.01 for DroQ. Our DO implementation follows the open-source implementation of DroQ in Smith et al. (2022) (https //gi thub. com/ foom/kkott kostril kov wa walkkin_the____eeee rk). On DMC tasks, we turn off the policy delay the DroQ baseline keep it consistent with other baselines (e.g.. Reset, weight decay) as they all do not use policy delay.
Reset. For our experiments with Reset (Nikishin et al., 2022), we use the same strategy as the original paper where re-initialize the agent from scratch periodically while keeping the replay buffer. For DMC tasks, we use reset frequency of 200K steps (same as the original paper). For gym tasks, we use reset frequency of 100K steps. To our surprise, resetting every 100K could already match the performance of DroQ (Hiraoka et al., 2021).
Gym experimental setup. For all the experiments on Gym tasks, we follow DroQ (Hiraoka et al., 2021) where we update the actor once per every 20 critic update steps and 3 x 10yh environment steps. We use a warm-up period of 5000 steps where random actions are taken before updating the agents. We use UTD ratio of 20 (also used in DroQ (Hiraoka al., 2021) and RedQ (Chen al., 2021)).
DMC experimental setup. For all the experiments on DMC tasks, we use a UTD ratio of 9, warm-up period of 10000 steps where random actions are taken before updating the agents, and run 10" environment steps. Unless specified otherwise, no policy delay is used for experiments on DMC (actor update frequency is the same as the critic update frequency).
SAC. For the SAC implementation used in this paper, build our code top of the jaxrl codebase: https:://// //tt// com/ http:/ikotttkkoo/eeeeeeeee kov/jaxr] (Kostrikov, 2021). The actor action distribution parameterized by Tanh-transformed diagonal Gaussian with learnable and state- dependent mean and log standard deviation. Both actor and critic (Q-function) networks are ini- tialized with orthogonal initialization with a multiplier of /2, following the implementation in https: / /iihhb.. con/ http//eeennniiiiiiieeeee wwtiiiiiiiiiiiirri siith_reeets (the official repository of Reset (Nikishin et al., 2022)). The hyperparameter used for SAC is attached as follows (see Table 3):
Evaluation return computation. Unless specified otherwise, the return for individual seed throughout the paper is estimated by running the policy with the deterministic mode (e.g., taking the mean of the Gaussian distribution for each action) for 10 independent trials.
14

---

Published as a conference paper at ICLR 2023
C DMC ANALYSIS
C.I COMPARISON OF THREE METRICS: Q-GAP, ESTIMATION BIAS USING MC RETURNS AND TD VALIDATION ERROR.
In this section, we plot the evaluation return against three metrics. The first metric AQ := Bg=D.a"><tals ()) [Qa a"I a= Qo(s, a²)]. The second metric is the estimation bias of the estimated Q-function compare to the actual return of the policy on the state-action distribution of the policy: Ex (s. a) a)).. In experiment, the second term is estimated by the Monte-Carlo discounted return (by computing the discounted reward-to-go for each transition in the Ei= 7" -"r((srr,a ar) (with the trajectory being
[(s, a az)l 1). We use 10 trajectories. The third metric is the TD error (Equation 1) on an independently collected held- out dataset. Figure 6 and 7 shows the metric values and the evaluation returns different DMC environments with different UTDs and different regularizers respectively under the offline shuffled streaming setting. These plots generated using 7 separate trials (different from the trials used in the plots in the rest the paper). This is because the MC Bias information was not being logged during the original 8 trials. The results for the different UTDs are summarized in Table 1.
trajectory): Q" (s;,
C.2 THE EFFECT OF REGULARIZERS ON THE ONLINE PERFORMANCE.
To further study whether the validation TD error is indicative of the performance beyond the offline setting, we plot the online evaluation performance against the validation TD error in the offline shuffled streaming setting. The method that achieves the lowest TD error in the shuffled streaming setting usually performs well online (Figure 5).
FN
FN + DO
FN WD
FN + SN
FN + Reset
DroQ
quadruped-run (Online)
hopper-hop (Online)
humanoid-run (Online)
fish-swim (Online)
Return
200
200
500
500
0
quadruped-run
hopper-hop (Shuffled)
humanoid-run (Shuffled)
fish-swim (Shuffled)
Valid. TD, Lvalid(e)
(Shuffled)
10.0
1.0
0.1
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Environment Steps (x10%)
Figure 5: The effect of regularization approaches on the online performance and its correlation with the TD validation error in the offline shuffled streaming setting. On quadruped-ree run, hopper-hop. humanoid- irruun
fish-swe swim the performance improvements correlate well with the validation TD emor in the offline setting. The method with the lowest validation TD enor ranks first in performance on hopper-hoo (Reset), huamncide rnn (WD), and fish-swim (Reset). For guadruped-run Reset achieves the lowest validation TD error and it one of the top performing method. We include a more complete set of results in Appendix C, Figure 11 (Online), Figure (Offline Shuffled).
15

---

Published as a conference paper at ICLR 2023
UT0=1
Ulb
UTD-9
UTD-20
hopper-hop
hopper-hop
hopper-hop
hopperhop
300
Voild TD. Coell
1of
0.4
Retumn
200
MC Boz
0.2
10-4
0.0
0.0
0.2
0.4
0.8
10
0.0
0.2
0.4
0.6
0.8
10
0.0
0.2
0.4
0.5
0.8
1.0
1. 0.0
0.2
0.4
0.5
0.8
Environment
Environment
Environment (x30f)
Enxtronment (X30M)
hopper-stand
hopper-stand
hopper-stand
hopper-sland
0.6
100
Valls. TD. Loadl))
101
Retum
0.4
MC slas
500
50
0.6
1.0
Environmant Staps (K10%)
Environment Staps (K10f)
Envionment Stops (H309)
Envdronment Stups (H10f)
humanoid-stand
humanoid-stand
humanoid-stane
humanoid-stand
Valle TD. L.a.0))
perum
400
600
WC 9005
100
10z
50
10a
0.4
0.8
0.4
0.4
0.0
1.0
Enviroonment Steps (x10%)
Enviiromeent Steps (X10%)
Envvi Steps (x300%
Environment Steps (x109)
humaroid-run
humanoid-run
humancid-run
humanoid-run
Valle TD, Load8)
100
100
Return
MC Bios
50
10
10
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.8
0.8
1.0
0.0
1.0 0.2
0.4
0.8
0.8
Errvitonement Stwps (K10%)
Envirormmett Stwps
(X10P)
Encicoement Steps (x309)
Endironment Steps (x300))
fish-swim
fish-swim
fish-swim
fish-swim
Vold. TD. Loa08)
600
Rehum
400
200
30-1
0.0
0.4
0.8
0.4
0.8
1.0
0.4
0.8
1.0
0.4
1.0
Steps
Steps
quadruped-run
quadruped-run
quadruped-run
quadruped-run
Vold. TD, (.m00)
600
B35
0
1.0
8
200
-200
0.4
0.8
0.4
Enwironment Steps (x10
Enviromment Steps (x100) 0%)
Envrrnment (x30*)
Envtronment Steps (x10')
finger-tum hard
fingerturm hard
finger-tur hard
finger-tum hard
1000
Veid TD. Awald)
0.4
40
MC Ba5
500
3
20
0.
0.0
0.0
0.2
0.4
0.6
0.8
100
0.0
0.2
0.4
0.6
0.8
100
0.0
0.2
0.4
0.6
0.5
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment Steps (x10")
Environment Steps (x10")
Envronment (epps(xx30)
Emdroomment Steps (x20")
Figure 6: Comparison of three metrics with varying UTDs under the offline shuffled streaming setting: the Q-gap (AQ). the estimation bias (MC Bias), and the Validation TD Error (Lvatia))) (0)). Plots generated using separate trials (compared to Figure 10).
16

---

Published as a conference paper at ICLR 2023
WD
PN SM
FN 4 Rasei
Drog
hopper-hop
hopper-hop
hopper-hop
hopper-hop
300
30
Valld TD. Loald
0.2
Retumm
200
0.1
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.8
0.0
0.2
0.4
0.5
0.8
1.0
0.0
0.6 Envronment
02
0.4
0.0
1.
Environment (x10f)
Environment (x10f)
Enxtronment
hopper-stand
hopperstand
hopper-stand
hopper-stand
0.4
75
Retum
500
8 0.2
1.0
Environment Staps (x10f)
Environment Staps (K10f)
Envfronment Stops (H30f)
Enviromment Staps (4109)
humancid-stand
humanoid-stard
humanoid-stand
humanoid-stand
750
20
Retum
500
8
250
100
0.8 Erivirormentt Steps (X10%)
0.4
0.8
0.0
0.4
0.0
0.6
Envéronment Staps (K10f]
Envirenment Steps (x309)
Environment Staps (x110%
humancid-run
humanaid-run
humanaid-run
humancid-run
Return
1.0
40
200
0.0
0.0
0.2
0.4
3.6
6.8
1.0
0.0
6.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.8
0.8
1.0
0.5
022
0.4
0.6
0.8
1.0
Enviroorment Slwps (K10F)
Envirormmet Stwps (X10F)
Endironment Steps
(x304)
Envirormmnt Steps (x319)
fish-swim
fish-swim
fish-swim
fish-swim
Volld TD, Luai00)
600
Rehum
400
6.1
INC 835
10
100
200
0.0
20
0.8
0.0
0.4
0.8
0.0
0.4
0.6
1.0
Steps
quadruped-run
quadruped-run
quadruped-run
quadruped-run
1.5
750
92tum
5.0
1.0
100
250
0.5
100
0.8
0.6
Environment Steps (x10f)
Environment Steps (x10f)
Enstronment Steps Ex30
Environment Steps (x30
fingerttrn hard
finger-uum hard
fingentum hard
finger-turn hard
1000
(a200)
9ztum
500
406
0.
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment Steps (x10")
Environment Steps 1x10")
Enstronment Steps (X30")
Envronmennt (t20s"
Figure 7: Comparison of three metrics with varying regularizers under the offline shuffled streaming setting: the Q-gap (AQ), the estimation bias (MC Bias), and the Validation TD Error (Luunia (0)). Plots generated using separate trials (compared Figure 13)
C.3
SUPPLEMENTARY RESULTS.
This section contains additional plots for the 7 DMC environments with different UTDs and different regularizers on evaluation return, training/validation TD error.
17

---

Published as a conference paper at ICLR 2023
UTD-20
UTD=9
UTD-3
UTD=1
hopper-hop (Online)
hopper-hop (Streaming)
hopper-hop (Shuffled)
300
300
300
Retumn
200
Return
200
Return
200
100
0
0.4
Environment Steps 1x10
Environment Steps 1x10%)
Environment Steps (X109)
hopper stand (Online)
hopper stand (Streaming))
hopper stand (Shuffled)
Return
Retur
Betur
500
500
1.0
0.4
0.4
1.0
Environment Steps (x10%)
Environment Steps (K10%)
Environment Steps (x10%)
humanoid (Online)
humanaid (Streaming)
humanaid (Shuffled)
600
Returm
500
Return
400
Retur 200
400
200
0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
022
0.4
0.6
0.8
1.0
Environment Steps (H10*)
Environment Steps Ix10")
Environment Steps (x10*)
humanold-run (Online)
humanoid-mu (Streaming)
humanoid-run (Shuffied)
200
200
200
Reeurn
Ruturn
Rdurn
100
100
100
0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment Stops (x107)
Environment 1x10%)
Environment (x109)
fish swim (Online)
fish swim (Streaming)
fish swim (Shuffled)
600
Return
400
Return
400
Return
400
200
2000
200
0
0
0.4
0.6
0.4
0.4
1.0
Environment Steps (M10")
Environment Stops IX10")
Environment Steps (xI0')
quadruped-run (Online)
quadruped-nu dun (Streaming)
quadruped (Shuffled)
750
750
750-
Retum
500
Retur
500
Return
500
250
250
250
Environment Steps (x10*)
Environment Steps Ix10f)
Environment Skeps (x10f)
finger-tum_ hard (Online)
finger-tum, hard (Streaming)
tinger-turn hard (Shutfied)
Retum
Retum
Ratum
500
500
500
0-
0.0
0.2
0.4
0.6
08
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment Stops (K10f)
Environment Stops IX10f)
Environment Stops (x109)
Figure 8: The effects of UTD ratios on DMC tasks. All agents use feature normalization in the last layer feature stabilize TD learning. Among all the tasks considered, almost all tasks exhibit the failure mode of high UTD in the online setting except on hopper-hop where UTD=20 performs the best. In the offline setting, the performance degrade trend cleaner where the agents trained with UTD=1 performs the best across the board (except on hopper-hop on the streaming setting, humanoid-zun on the shuffled streaming setting. and finger-turn.hare on both offline settings.) and the agents trained with UTD-20 performs the worst across the board (except on guadrupedo on shuffled streaming setting and finger-turn hard on both offline settings).
18

---

Published as a conference paper at ICLR 2023
UTD=20
UTD=9
UTD-3
UTD=1
Valisattin Error, Lo Vallostion Emrr, hate()) Valleation TD Emor, (arel9 Validazion on TD Emor. boual)) Validmtin Laae(8) Validation baae()) Validation Erorr (aaef)
hopper-hop
hopper-hop
hopper-hop
1000
10'
10º
750
500
10a
10a
250
10-2
0.0
0.2
0.4
0.2
0.4
0.6
Environment Steps (x109)
Environment Steps (x10%)
Environment Steps (x109)
hopper-tand
Training TD Eroor Lia)
hopper-stand
hopper-stand
1000
10
10a
500
10a
10
10
10m2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment Steps (x109)
Environment Steps (x109)
Environment Steps (x109)
humanoid-starn
humandid-stand
comannid-sstand
10a
750
500
100
10
250
Training
10-4
10-2
0.0
0.2
044
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment Steps (x109)
Environment Steps (x109)
Environment (x109)
humanoid-run
Training Lia)
humanoid-run
humanoid-run
1000
10a
10a
750
10a
250
0.0
0.2
Environment Steps (x10f)
Environment Steps (x10f)
Environment Steps (x109)
fish-swim
fish-swim
fish-swim
1000
102
10a
Training Error.
Ratum
750
500
10a
10a
250
10-2
10-2
0.0
0.6 Envrronment (x119)
0.4
Environment Steps (x109)
Environment Steps (x10*)
quadruped-run
Training Error, Ue)
quadruped-run
quadruped-run
1000
10a
10a
Retum
500
10a
10a
250
10-²
10-2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment Steps (x10%)
Environment Steps (x109)
Environment Steps (x10')
finger-tum_hann hard
finger-turn_han
finger-turn_hard hard
1000
750
500
10a
10a
Training
10-1
10-2
0.0
0.8 Environment (x10%)
0.2
0.4
0.6
1.0
0.0
0.2
0.4
0.8 Steps (x10*)
0.6
1.0
0.0
0.2
04
0.6
0.8 1.0
Environment (x109)
Figure 9: The effects of UTD ratio on DMC tasks in the offfine streaming setting. All agents use feature normalization in the last layer stabilize TD learning. For all tasks except quadruped- run (where the agent trained with UTD=I is doing bit worse than other agents with UTD=3 and UTD=9 but achieving lower validation TD error) and finger-ture hard (where performance does not seem to matter among different UTD ratios), the TD validation error corre- lated well with performance.
19

---

Published as a conference paper at ICLR 2023
UTD=20
UTD=9
UTD-3
UTD=1
Validatton Loae(8) Vallcation Emer, Late()) Valleation TD Emor, (arel9) Validazion on TD Emor. boual)) Validmtio Laae(8)) Validation Laae()) Validation Eror. Luaef)
hopper-hop
hopper-hop
hopper-hop
1000
10'
10¹
750
500
10a
10a
250
10-1
10-2
0.0
0.2
0.4
0.0
0.2
0.4
0.6
Environment Steps (x109)
Environment Steps (x10%)
Environment Steps (x109)
hopper-ttnnd
Training Eroor Lia)
hopper-stand
hopper-stand
1000
10
10a
500
109
10
10-h
10m2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment Steps (x109)
Environment Steps (x109)
Environment Steps (x109)
rrmanoid-stang
humanoid-stand
cormanoid-stand
10a
750
500
100
10
250
Training
10-4
10-2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment Steps (x109)
Environment Steps (x109)
Environment (x109)
humanoid-run
humanoid-run
humanoid-run
1000
10a
102
750
10a
250
Training
0.0
0.2
Environment Steps (x10f)
Environment Steps (x10f)
Environment Steps (x119)
fish-swim
fish-swim
fish-swim
1000
102
102
Training Error.
Ratum
750
500
10a
10a
250
10-2
10-2
0.0
0.0
0.4
0.6
Environment Steps (x109)
Environment Steps (x10*)
Ster (x119)
quadruped-run
Training TD Error, Ue)
quadruped-run
quadruped-run
1000
10a
10a
Retum
500
10"
10a
250
10-²
10-2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment Steps (x10%)
Environment Steps (x109)
Environment Steps (x10')
finger-tum hard
finger-turn_aan
finger-turn_han hard
1000
750
500
10a
10a
Training
10-1
10-2
0.0
0.8 Environment (x10%)
0.2
04
0.6
1.0
0.0
0.8 Environment Steps (x10*)
0.2
0.4
0.6
1.0
0.0
0.2
0.4
0.6
0.8 1.0
Environment (x109)
Figure 10: The effects of UTD ratio on 7 DMC tasks in the offline shuffled streaming setting. All agents feature normalization in the last layer to stabilize TD learning. For all tasks except Finger-tuzn_hard (where the performance of the agents trained with UTD=1,3,9 are indistin- guishable while the validation TD errors are) and hepper-hop (where the performance of the agents train with UTD=1,3 are indistinguishable while the validation TD errors are), the TD valida- tion error correlated well with performance.
20

---

Published as a conference paper at ICLR 2023
FN
Reset
Droq
hopper-hop (Online)
hopper-hop (Streaming)
hopper-hop (Shuffled)
300
300
300
Retum
200
Return
200
Return
200
100
Environment Steps 1x10
Environment Steps 1x10%)
Environment Steps (X109)
hopper stand (Online)
hopper stand (Streaming)
hopper-stand (Shuffled)
Return
Retur
Retum
500
500
0
0.4
Environment Steps (x10*)
Environment Steps 1K10%)
Environment Steps (x10%)
(Online)
humanaid (Streaming)
(Shuffled)
400
Returm
Retum
Return
500
200
200
0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment Steps (+10*)
Environment Steps Ix10")
Environment Steps (x10*)
humanoid-run (Online)
humanoid-mn (Streaming)
humanoid-run (Shuffied)
200
200
200
Reeurn
Ruturn
Rdurn
1.00
100
100
0
0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment
1x10%)
Environment 1x10%)
Environment (x10f)
fish swim (Online)
fish swim (Streaming)
fish swim (Shuffled)
Return
400
Return
4:00
Return
400
200
2.00
200
0
0.4
0.4
Environment Steps (M10")
Environment Steps IX10")
Environment Steps (x10')
quadruped-run (Online)
quadruped-ru run (Streaming)
quadruped-rum (Shuffled)
750
750
750
Retum
500
Retur
500
Retur
500
250
250
250
Environment Steps (x10*)
Environment Steps Ix10f)
Environment Steps (x10f)
finger-tum_ hhard (Online)
finger-tum, hard (Streaming)
fingerstur, hard (Shuffied)
Retum
Retum
Ratum
500
500
500
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment Steps (K10f)
Environment Steps IX10f)
Environment Steps (x10f)
Figure 11: The effects of various regularization approaches on 7 DMC tasks. Among all the tasks considered, fish-swim humanoid-run and hopper-hop have the most similar relative performance ordering. On humanoid-stand, all the other methods except the base SAC + FN correlates have similar performance ordering across online/offine settings. On qua druped- run. we observe that the performance gap is much bigger in the offline settings, but the relative ordering of DO. WD. Reset and DroQ roughly preserved (FN and FN SN are much worse in the shuffled streaming setting). The ordering is the most ambiguous in hopper- -eeerttt sapper-stande stand.
21

---

Published as a conference paper at ICLR 2023
Reset
Droq
TD Emor. Loual)) Validatinn Laae(8) Validation baae()) Validation Erorr. Laa
hopper-hop
hopper-hop
hopper-hop
10'
10'
Error,
Return
10a
10a
10-1
10-2
0.0
0.2
0.0
0.2
0.4
0.6
Environment Steps (x109)
Environment Steps (x10%)
Environment Steps (x109)
hopper-tand
TD Eroor LiB)
hopper-stand
hopper-stand
10
10a
Return
10a
10
10
10m2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
100
Environment Steps (x109)
Environment Steps (x109)
Environment Steps (x109)
humanoid-starn
humanoid-stand
humanoid
10a
400
Return
100
10a
200
Training
10-a
10-2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment (x109)
Environment Steps (x10f)
Environment (x109)
humanoid-run
humanoid-run
humanoid-run
10a
10a
Return
100
10a
Training
0.0
0.2
Environment (x10f)
Environment Steps (x10f)
Environment Steps (x109)
Valisation Loae(8) Vallcation Emror, hate()) Valleation Emor, (arel9
fish-swim
fish-swim
fish-swim
102
10a
600
Return
400
10a
10a
200
Training
10-2
10-2
0.0
0.0
0.6 Environment (x109)
0.4
Ensironment (x109)
Environment Steps (x10f)
quadruped-run
Training Error, Ue)
quadruped-run
quadruped-run
104
10a
750
Retum
500
10a
10a
250
10-²
10-2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment Steps (x10%)
Environment Steps (x109)
Environment Steps (x10')
finger-tum_hann
finger-turn_han
finger-turn_har hard
1000
500
10a
10a
Training
10-1
10-2
0.0
0.2
0.4
0.6
0.8 (x10*)
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment Steps (x10*)
(x109)
Figure 12: The effects of different regularizations on 7 DMC tasks in the offline streaming set- ting. All agents use feature normalization in the last layer to stabilize TD learning. On sish-wwim humanoi i d- -run and humano dd-tandd the evaluation returns of different regularization ap- proaches generally correlates well with their TD errors. On hopper- -op, finger- turn hard and hopper-st tanda no obvious correlation can be seen as all of approaches perform quite simi- larly. Specifically, on fish-swim. the top performing method correlates better with the validation TD error compared to the training TD error.
22

---

Published as a conference paper at ICLR 2023
Reset
DroQ
Valisattin Emrr, Loae(8) Vallcation Emrr, hate()) Valleation TD Emor, Larel9 Validazion on TD Emor. boual)) Validation Laae(8) Validation baae()) Validation Eroor. Laa
hopper-hop
hopper-hop
hopper-hop
10'
10¹
Training Error
Return
200
10a
10a
100
10-1
10-2
0.0
0.2
0.0
0.2
0.4
0.6
Environment Steps (x109)
Environment Steps (x10%)
Environment Steps (x109)
hopper-tand
TD Eroor Lia)
hopper-stand
hopper-stand
10
10a
Return
109
10
10
10m2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment Steps (x109)
Environment Steps (x109)
Environment Steps (x109)
humanoid-stern stand
humandid-stand
humanoid
10a
Return
100
10
200
Training
10-a
10-2
0.0
0.2
044
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment (x109)
Environment Steps (x109)
Environment (x109)
humanoid-run
Training Lia)
humanoid-run
humanoid-run
10a
10a
Return
100
10a
0.0
0.2
Environment (x10f)
Environment Steps (x10f)
Environment Steps (x109)
fish-swim
fish-swim
fish-swim
102
102
600
Training Error.
Return
400
10a
10a
200
10-2
10-2
0.0
0.0
0.0
0.4
Environment Steps (x109)
Environment Steps (x10*)
Environment (x109)
quadruped-run
Training Ue)
quadruped-run
quadruped-run
10a
10a
750
Retm
500
10"
10a
250
10-²
10-2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment Steps (x10%)
Environment Steps (x109)
Environment Steps (x10')
finger-turn_han hard
finger-turn_han
finger-turn_hen hard
1000
500
10a
10a
Training
10-1
10-2
0.0
0.2
04
0.6
0.8 (x10f)
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8 1.0
Environment Steps (x10*)
(x109)
Figure 13: The effects of different regularizations on 7 DMC tasks in the offline shuffled streaming setting. All agents feature normalization in the last layer to stabilize TD learning. On humano i d stand, nuuanoiid- i-soiddduun.. quadruped run and fish- 1 swim, the top performing methods tend have lower TD error. Specifically, on fish-sae bwwwim,, the top performing method correlates better with the validation TD error compared to the training TD error.
23

---

Published as a conference paper at ICLR 2023
UTD=20
UTD=9
UTD=3
UTD=1
hopper-hop
humanoid-run
3
3
2
1
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment Steps (x10%)
Environment Steps (x10%)
hopper-stand
quadruped-run
3
3
2
DO
2
1
0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment Steps (x10%)
Environment Steps (x10f)
humanoid-stand
finger-turn_hard
3
0
ON
2
1
0
0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment Steps (x106)
Environment Steps (x106)
The effects of UTD ratio on the gap in Q-values 6 DMC tasks in the offline shuffled streaming setting. All agents use feature normalization the last layer to stabilize TD learning.
Figure 14:
24

---

Published as a conference paper at ICLR 2023
fish-swim
hopper-hop
quadruped run
chectah-run
500
800
250
Ratumm
200
Retum
600
Batumm
200
400
200
100
200
200
Erviorment Steps (04)
Strstroreeet Steps (XL0P)
Environmeet Steps (x109)
Envirrmmnt Slaps (x104))
humanold-ruu
humanold--co
fingor-tum_noe hard
hopper-stand
177
8.00
900
125
0.0
Neturn
600
6.00
400
200
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.4
0.6
1.0
0.0
0.2
0.6
0.8 (x307))
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environm
Environment
Figure 15: The data collecting policy for the offline analysis. The online training RL agent is standard SAC with UTD ratio of 9 and gets periodically reset after every 200K steps.
25

---

Published as a conference paper at ICLR 2023
C.4 FULL RESULTS FOR AVTD
How effective AVTD online model selection? Figure 16 shows two experiments that involve an ensemble of DroQ agents. Figure 17 studies how important it is to use held-out validation set compared the training AVTD.
Ours
DroQ=0.003
DroQ=0.03
(DroQ=0.01
LN
DroQ=0.01
DroQ=0.1
Random (DraQ=0.00
Humanoid-v4
Hopper-v4
(DroQ=0.000
(DroQ=0.1)
5000
Humanoid-v4
Hopper-v4
5000
Return
Walker2d-v4
Ant-v4
Return
Walker2d-v4
Ant-v4
5000
5000
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
Environment Steps (x10%)
Environment Steps (x10%)
(a) Can AVTD select the best regularization (b) Can training ensemble of agents match the strength in the ensemble? plot AVTD against performance AVTD: Random corresponds train- the five agents trained independently on return ing an ensemble with completely random agent selec- metric. AVTD consistently matches the top perform- tion (s = 1.0) where the evaluation return of cach agent ing regularizer, suggesting that it can select the best in the ensemble is shown separately. AVTD consis- tently outperforms all agents in the ensemble.
regularization strength.
Figure 16: How effective is AVTD at online model selection with DroQ agents? In both experiments, we use five DroQ agents with different dropout rate: 0.1, 0.03, 0.01, 0.003, and (denoted as LN).
Ours
Ours (train)
DroQ=0.01
WD=0.01
Train
DroQ=0.03
Valid
LN+
600
1.00
Return
400
TD Error
0.10
200
0.0
0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment Steps (x10%)
Environment Steps (x10%)
Figure 17: AVTD ys. model selection via training TD error on fish-swim Left: return; Right: val- idation TD error of each agent in AVTD (right) and training TD errr of each agent in AVTD (train) (mid). AVTD consistently picks the regularizer with the best performance (WD=0.01) whereas AVTD (train) consis- tently picks the agent that overfits the most. Thus, AVTD (train) achieves lower return compared AVTD.
Aggregated performance computation. To compute the aggregated performance for each method, we use the following protocol - For each environment, we normalize the return by the best average return achieved each task (taking the maximum over all agent and all environment
26

---

Published as a conference paper at ICLR 2023
steps and the average over all eight seeds). After obtaining the normalized return for each environ- ment, method and seed, we aggregate them over nine DMC tasks and four Gym tasks to obtain the sample efficiency curve for the aggregated normalized score following Agarwal et al. (2021). Since DMC experiments were run for a larger number of steps (10%) than Gym experiments (3 x 10%), only take the performance of the DMC for the first 3 x 10 steps compute the aggregated normalized score.
Oracle
SAC (UTD=1)
WD=0.01
LN+WD=0.01
LN
Ours
Reset
DroQ=0.03
DroQ=0.01
Aggregated Normalized Score
Ant-v4
Walker2d-v4
Hopper-v4
Humanoid-v4
6000
0.8
2000
0.6
0.0
0.2
0.0
0.2
0.0
0.2
0.0
0.2
cheetah-run
acrobot
hopper-hap
hopper-stand
1000
0.4
Return
500
0.2
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5 fish-swim
1.0
0.0
0.5
1.0
0.0
0.2
humanoid-stand
humanaid-run
quadruped-run
finger-turn hard
1000
500
8.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Environment Steps (x106)
Figure 18: AVTD compared to different regularizers and the standard SAC baseline with UTD=1. For Gym tasks, UTD=20 is used and the actor is updated once per 20 critic updates. For DMC tasks, UTD=9 is used and the actor updated with every critic update.
27


Reviews:
[{"review_id": "GxlGCnC4R8", "reviewer": "Reviewer_rmvQ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work systematically tests a number of possible reasons for off-policy DRL methods not performing as well as they potentially could.\nFrom this, the authors conclude that statistical overfitting is a key contributor to the poor performance.\nBased on this insight, they propose a method for keeping statistical overfitting low. This method is shown to successfully reduce overfitting and improve RL performance.", "summary_of_the_review": "This is a high-quality paper. It begins with a clear starting question that is thoroughly addressed.\nBased on the investigation, a concrete conclusion is drawn and a novel method is produced.", "strength_and_weaknesses": "This is a high-quality paper.\nIt begins with a clear starting question that is thoroughly addressed.\nThe experiments performed for this are clear and compelling.\nBased on the investigation, a concrete conclusion is drawn and a novel method is produced.\nThis method is novel and well-motivated.\n\nThe generated insights have the potential to be the foundation for further methods.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "FRPGBtAPPyN", "reviewer": "Reviewer_yYZT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "[\"Long\" summary]\n\nThe paper deals with sample efficiency in deep reinforcement learning (DRL). In particular, there was a known problem in DRL with high update-to-data (UTD) ratio, addressed by several recent studies with different regularizers. The authors aim to understand why DRL usually fail with high UTD and how we can overcome this problem in general. Addressing this problem is a vital step toward data-efficient DRL. The paper first examined possible reasons that cause the failure, including data quality, non-stationarity and s distribution shift. The empirical results show that any reason alone cannot explain the failure. Therefore, the authors conjectured that the main cause is statistical overfitting.\n\nThen, to solve the statistical overfitting the paper proposed. An algorithm was proposed, in which multiple DRL models with various regularizers (each proposed by a previous study) are simultaneously trained while sharing one replay buffer. At each step, the model with lowest validation TD error is selected to compute the policy. Experiments were conducted to show the effectiveness of the algorithm. Furthermore, some additional experiments results are shown in Sec.5 to support the design choices.\n\n[In short]\n\nThe first contribution of the paper is the empirical results for better understanding the failure with high UTD, which are helpful and necessary for data-efficient DRL.\n\nThe second is the introduction of an algorithm to adaptively select the agent with proper regularizer to handle high UTD.", "summary_of_the_review": "The paper touches an interesting and essential problem in DRL: the failure with high UTD ratio using off-policy algorithms. I appreciate the paper because it provides insightful empirical results toward understanding the failure. Meanwhile, the proposed algorithm, AVTD, which aims to overcome the failure, does not demonstrate satisfying performance though it costs more computation. I think at least the first contribution of the paper is significant to DRL community. Hopefully the authors can figure out a better way to overcome the difficulty in the near future. Overall, I lean toward an acceptance.", "strength_and_weaknesses": "[Pros]\n- The paper provides important insight about why DRL fail when trying to exploit the data by high UTD ratios. This is helpful and necessary toward data-efficient DRL.\n- The experimental evaluations are relatively comprehensive and well-defined.\n\n\n[Cons]\n- If I understand correctly, AVTD (Algorithm 1) needs to train all the off-policy agents, which means the computation cost is almost the sum of the individuals (K is the number of agents with different regularizers).\n- The overall performance of AVTD (Figure 9, aggregated normalized score) is almost the same as WD=0.01 and only marginally outperforms Reset. In particular, AVTD performed the worst in Acrobot swing-up. Practically saying, why should we use AVTD since it is less computationally efficient while not significantly better than baseline methods.\n\n\n[Questions]\n- What does \"AVTD\" stand for?\n- Page 13, SN, we follow the implementation of WHAT?\n- Is there any possibility to theoretical investigate the high UTD failure?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "CHf3l9_4yTU", "reviewer": "Reviewer_zRMn", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In the paper under review, it is shown that the TD for validation data (validation TD) correlates with the performance of the RL + regularization methods in high UTD (replay ratio) settings. \nBased on this finding, a method that selects the regularization method with the smallest validation TD from multiple regularization methods is proposed. \n", "summary_of_the_review": "I am leaning toward acceptance.\nThe main strength of this paper is proposing a framework for unifying multiple RL methods (more precisely, regularization methods) and performing an empirical analysis of the framework.\nCurrently, the success or failure of RL methods (+ hyperparameter settings) is highly task-dependent. Thus, the framework to unify individual methods is very important. \n\nThe presentation of the paper seems to be not very much polished. \nMore careful modification of the presentation would further improve the paper's quality.\n", "strength_and_weaknesses": "# Strengths:  \n[Novelity] [Quality]:  \n* The comparison of several regularization methods in addition to methods such as Reset, DroQ, and Spectrum normalization, was performed on several benchmark tasks. This comprehensive comparison result would be a good experiment stack. \n\n* Through many experiments in several settings (eg., approx. offline learning based on the tandem learning framework), it is shown that validation TD correlates with actual performance. A unifying method (AVTD) in which the regularization method is selected in accordance with its validation TD score is proposed. It is shown that the performance of this unifying method is better than that of each regularization method. This kind of unification of RL methods is very important in practice, because, currently, the success of RL methods (+ hyperparameter values) is highly task-dependent.\n\n\n# Weaknesses: \n[Clarity] [Quality]:  \nThe paper is basically well-written, but there are typos and unclear points. \n\n> RL algorithm has been a important \n\n-> RL algorithm has been an important\n\n\n> Concretely, our method, AVTD, trains several off-policy RL agents on a shared replay buffer where each agent applies a different overfitting regularize.\n\nSince AVTD appears for the first time here, the name in its non-abbreviated form should be provided as well: our method, <non-abbreviated form> (AVTD)\n\n> In theory, these off-policy algorithms can be made very sample efficient by making sure the Q-network fits the current replay buffer well,\n\nneeds reference. \n\n> Our analysis analyzes a standard SAC agent in the high UTD regime. \n\n-> We analyze a standard SAC agent in the high UTD regime. \n\n> we plot the training and validation TD errors as well as the standardized TD gap (STD gap),\n\nWhy introduce a Standardized TD gap? (Why not just use validation TD?) Validation TD also correlates with performance, and the AVTD algorithm uses validation TD. So I don't see any motivation to introduce Standardized TD gap. \n\n> Figure 2\n\nWhy not merge it with Figure 1 and arrange the figures horizontally? In the current version, the legend and some captions are duplicated and consume space. \n\n\n> We would also highlight that the regularizers that achieve the lowest validation TD error offline are usually one of the top performing methods online (Figure 5)\n\nIt would help to understand the correlations more easily if there were figures summarizing the results shown in Figures 5 and 12. \ne.g., how many times did the method with the lowest TD error rank in 1st place in actual performance? and how many times did it rank in 2nd or lower places?\n\n\n> The solution of (Nikishin et al., 2022) which\n\n-> The solution of Nikishin et al. (2022) which\n\n> (Khadka et al., 2019) also\n\n-> Khadka et al. (2019) also\n\n\n> Algorithm 1. 10: After every nepisode episodes, collect a heldout trajectory and add to Dheldout with the same action selection strategy above for D.\n\nWhat is the size of the Heldout dataset \\mathcal{D}_{\\text{heldout}}? If the size is large, this impairs overall sample efficiency since we need to consume non-trivial numbers of samples for validation.\n\n\n\n> Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. CoRR, abs/1801.01290, 2018a. URL http://arxiv.org/abs/1801.01290.\n\n> Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp. 1861–1870. PMLR, 2018b.\n\n> Maximilian Igl, Gregory Farquhar, Jelena Luketina, Wendelin Boehmer, and Shimon Whiteson. The impact of non-stationarity on generalisation in deep reinforcement learning. arXiv preprint arXiv:2006.05826, 2020a.\n\n> Maximilian Igl, Gregory Farquhar, Jelena Luketina, Wendelin Boehmer, and Shimon Whiteson. Transient non-stationarity and generalisation in deep reinforcement learning. arXiv preprint arXiv:2006.05826, 2020b.\n\n> Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Proceedings of the 36th International Conference on Machine Learning, 2019a.\n\n> Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp. 2052–2062, 2019b.\n\n> Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Modelbased policy optimization. Advances in Neural Information Processing Systems, 32, 2019a.\n\n> Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Modelbased policy optimization. In Advances in Neural Information Processing Systems, pp. 12498–12509, 2019b.\n\nDuplicate citations.\n\n> Spectral normalization (SN). For spectral normalization, we follow the implementation of\n\n-> Spectral normalization (SN). For spectral normalization, we follow the implementation of Gogianu et al. (2021)? Bjorck et al. (2021)?? or Miyato et al. (2018)???\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster