Dynamic Update-to-Data Ratio: Minimizing World Model Overfitting
Authors: Nicolai Dorka
Tim Welschehold
Wolfram Burgard
Venue: ICLR 2023 poster (2023)

Abstract: Early stopping based on the validation set performance is a popular approach to find the right balance between under- and overfitting in the context of supervised learning. However, in reinforcement learning, even for supervised sub-problems such as world model learning, early stopping is not applicable as the dataset is continually evolving. As a solution, we propose a new general method that dynamically adjusts the update to data (UTD) ratio during training based on under- and overfitting detection on a small subset of the continuously collected experience not used for training. We apply our method to DreamerV2, a state-of-the-art model-based reinforcement learning algorithm, and evaluate it on the DeepMind Control Suite and the Atari 100k benchmark. The results demonstrate that one can better balance under- and overestimation by adjusting the UTD ratio with our approach compared to the default setting in DreamerV2 and that it is competitive with an extensive hyperparameter search which is not feasible for many applications. Our method eliminates the need to set the UTD hyperparameter by hand and even leads to a higher robustness with regard to other learning-related hyperparameters further reducing the amount of necessary tuning.

Published as a conference paper at ICLR 2023
DYNAMIC UPDATE-TO-DATA RATIO: WORLD MODEL OVERFITTING
MINIMIZING
Nicolai Dorka!
Tim Welschchold
Wolfram Burgard? Universityy of Technology Nuremberg
'University of Freiburg dorkašcs, uni ceai-freiburggcom
ABSTRACT
Early stopping based on the validation set performance is a popular approach to find the right balance between under- and overfitting in the context of supervised learning. However, in reinforcement learning, even for supervised sub-problems such world model learning, early stopping is not applicable as the dataset is continually evolving. As solution, we propose a new general method that dy- namically adjusts the update to data (UTD) ratio during training based on under- and overfitting detection on a small subset of the continuously collected experi- ence not used for training. We apply our method DreamerV2. a state-of-the-art model-based reinforcement learning algorithm, and evaluate it on the DeepMind Control Suite and the Atari 100k benchmark. The results demonstrate that one can better balance under- and overestimation by adjusting the UTD ratio with our approach compared the default setting in DreamerV2 and that it is competitive with an extensive hyperparameter search which is not feasible for many applica- tions. Our method eliminates the need to set the UTD hyperparameter by hand and even leads to higher robustness with regard to other learning related hyper- parameters further reducing the amount of necessary tuning.
INTRODUCTION
In model-based reinforcement learning (RL) the agent leams a predictive world model to derive the policy for the given task through interaction with its environment. Previous work has shown that model-based approaches can achieve equal or even better results than their model-free coun- terparts [Silver et al. (2:18); [Schrittwieser et al.) (2020); Chua et al.] (2018); Hafner et al. (202T) An additional advantage of using a world model is, that once it has been learned for one task, can directly or after some finetuning be used for different tasks in the same environment potentially making the training of multiple skills for the agent considerably cheaper. Learning a world model in principle supervised learning problem. However, in contrast to the standard supervised learn- ing setting, in model-based RL the dataset is not fixed and given at the beginning of training but is gathered over time through the interaction with the environment which raises additional challenges.
typical problem in supervised learning is overfitting on a limited amount of data. This is well studied and besides several kinds of regularizations common solution is use a validation set that is not used for training but for continual evaluation of the trained model during training. By considering the learning curve on the validation set it is easy to detect if the model is under- or overfitting the training data. For neural networks a typical behavior is that too few updates lead to underfitting while too many updates lead to overfitting. In this context, the validation loss is great tool to balance those two and to achieve small error on unseen data.
For learning a world model on dynamic dataset there unfortunately no established method to determine if the model is under- or overfitting the training data available at the given point in time. Additionally, in model-based RL a poorly fit model can have a dramatic effect onto the learning result as from it the agent derives the policy, which influences the future collected experience which again influences the learning of the world model. So far, in mrodel-based RL this commonly ad- dressed with some form of regularization and by setting an update-to data (UTD) ratio that specifies how many update steps the model does per newly collected experience, similar to selecting the total number of parameter updates in supervised learning. Analogously to supervised learning, higher

---

Published as a conference paper at ICLR 2023
UTD ratio is more prone to overfit the data and lower to underfit it. State-of -fec-----hh-rrttt methods set the UTD ratio at the beginning of the training and do not base the selection on a dynamic perfor- mance metric. Unfortunately, tuning this parameter is very costly as the complete training process has to be traversed several times. Furthermore, a fixed UTD ratio often sub-optimal because different values for this parameter might be preferable at different stages of the training process.
In this paper, we propose a general method - called Dynamic Update-to-Data ratio (DUTD) that adjusts the UTD ratio during training. DUTD inspired by us- ing early stopping to balance under- and overfitting. stores a small portion of the collected experience in a separate validation buffer not used for training but in-
Environment
Palicy
stead used to track the development of the world models aceuracy in order to detect under- and overfitting. Based on this, we then dynamically adjust the UTD ratio.
Train every
Training
Data
We evaluate DUTD applied to DreamerV2 Hafner al. (2021) on the DeepMind Control Suite and the Atari1000 benchmark. The results show that DUTD in- creases the overall performance relative to the default DreamerV2 configuration. Most importantly, DUTD makes searching for the best UTD rate obsolete and is competitive with the best value found through ex- tensive hyperparameter tuning of DreamerV2. Further, our experiments show that with DUTD the world model becomes considerably more robust with respect to the choice of the learning rate.
UTD ratio
World Model
update
Validation
Evaluate
Data
Figure Overview of DUTD, A small subset of the experience collected from the environment is stored in a valida- tion set not used for training. The world model is trained for one update after
In summary, this paper makes the following contribu- tions: i) we introduce method to detect under- and overfitting of the world model online by evaluating it on hold-out data; ii) We use this information to dynamically adjust the UTD ratio to optimize world model perfor- mance; iii) Our method makes tuming the UTD hyperpa- rameter obsolete; iv) We exemplarily apply our method state-of-the-art model-based RI. method and show that it leads to an improved overall performance and higher robustness compared its default setting and reaches a competitive performance to an extensive hy- perparameter search.
every UTD ratlo many environment steps. From time to time, e.g., after an episode ended, the UTD ratio is adjusted depend- ing on the detection fitting of the world model on the valida- tion data. The policy is obtained from the world model either by planning or learn- ing and collects new data in the environ- ment.
under- or over-
2 RELATED WORK
In reinforcement learning there fitting describes overfitting to a specific environment such that performance on slightly different environments drops significantly. This appears in the context of sim-to-real, where the simulation is different from the target environment on which a well performing policy is desired, or when the environment changes slightly, for example, because of different visual appearance Zhang et al. (2018b): Packer et al. (2018); Zhang et al. (2018a); Raileanu et al. (2020); Song et al. (2020). In contrast, intra-tass task overfitting appears in the context of learning from limited data fixed envi- ronment when the model fits the data too perfectly and generalizes poorly to new data. We consider intra-task opposed to inter-task generalization.
forms of generalization and overfitting. Inter-task over-
In model-based reinforcement learning, there is also the problem of policy overfitting rate dynamics model |Armuuam et al. (2018); Jiang et al. (2015). As a result, the policy optimizes over the inaccuracies of the model and finds exploits that do not work on the actual environment. One approach to use uncertainty estimates coming from an ensemble of dynamics models to be more conservative when the estimated uncertainty is high Chua et al. 2018). Another approach to prevent the policy from exploiting the model is to use different kinds of regularization on the plans the policy considers Arumugam et al. (2018). In contrast to these previous works, we directly
inaccu-
2

---

Published as a conference paper at ICLR 2023
tackle the source of the problem by learning a better dynamics model. Consequently, our method is orthogonal to and can easily be combined with the just mentioned line of work.
Directly targeting the overfitting of the dynamics model can be done through the usage of a Bayesian dynamics model and the uncertaintics that come with such a model. Gaussian processes have been used successfully in this context Deisenroth & Rasmussen (2011) although it is difficult scale this to high-dimensional problems. Another way to reduce overfitting of the dynamics model is to use techniques from supervised learning. This includes for example regularization of the weights, dropout Srrvastava et al. (2014), or data augmentation Laskin et al. 2020); Schwwazzrr et al. (202T). All of these are also orthogonal our method and can be combined with to learn an even better dynamics model. Another popular approach is early stopping Strand (1974); Anderssen & Prenter [981]; ; Morgan & Bourlard (1989), where the training is stopped before the training loss converges. Our method can be regarded as the analogy of early stopping in dynamic dataset scenario.
Reducing the number of model parameters can prevent overfitting but can decrease performance compared to the right amount of training steps with more parameters. Our method overcomes this problem by automatically choosing the right amount of training steps for given network.
Hyperparameter optimization for RL algorithms is also related to our work. For example, AlphaStar Silver et al. (2018) has been improved by using Bayesian optimization Chen et al. (2018). /Zhang al. [2021) demonstrated that model-based RL algorithms can be greatly improved through auto- matic hyperparameter optimization. A recent overview on automated RL given by Parker-Holder et al. (2022). However, most of these approaches improve hyperparameters by training the RL. agent on the environment in an inner loop while keeping the hyperparameters fixed during each run. Our work deviates from that by adapting a hyperparameter online during training of a single run. The approach of Schhau et al. (019)) also falls into this category and dynamically adapts behavior- related parameters such as stochasticity and optimism. Similarly, the algorithm Agent57 Badia al. (2020) adaptively chooses from a set of policies with different exploration strategies and achievs human level performance on all 57 Atari games [Bellemare et al. (2013). Another approach adapts hyperparameter that controls under- and overestimation of the value function online resulting in model-free RL algorithm with strong performance on continuous control tasks Dorka et al. (202T).
In contrast to these approaches, our method directly learns a better world model by detecting under- and overfitting online on a validation set and dynamically adjusts the number of update steps accord- ingly. This renders the need to tune the UTD ratio hyperparameter unnecessary and further allows to automatically have its value being adapted to the needs of the different training stages.
THE DUTD ALGORITHM
In this section, we will first introduce the general setup, explain early stopping in the context of finding the right data fit and propose a new method that transfers this technique to the online learning setting. Lastly, we explain how the method can be applied to DreamerV2.
3.1 MODEL-BASED REINFORCEMENT LEARNING
We use the classical RL framework Sutton & Barto (0118) assuming Markov decision process (S, A, P, R). In this framework, the agent sequentially observes the current state E S in which executes action ae E A. receives a scalar reward r, according the reward function R, and transitions to next state $t+1 generated by the unknown transition dynamics P. The goal is to learn policy that selects actions in each state such that the total expected return Σ F; is maximized.
Model-based RL approaches leamn world model P(s St, az) also called dynamics model- and a reward model At(r; 8x) that attempt to reflect their real but unknown counterparts. These models can then be used to learn a good policy by differentiating through the world model or by generating imaginary rollouts on which an RL algorithm can be trained. Alternatively, the learned model can be used in planning algorithm to select an action in the environment.
3.2 UNDER- AND OVERFITTING
well-known problem in supervised learning is that of overfitting, which typically corresponds to low the training data and a high error on test data not seen during training. Usually, this
3

---

Published as a conference paper at ICLR 2023
happens if the model fits the training data too perfectly. In contrast to this, underfitting corresponds to the situation in which the model even poorly fits the training data and is characterized by both a high training and test error. To measure the performance of the model on unseen data, the available data is often split into a training and a validation set. Generally, only the training set is used to train the model while the validation set is used to evaluate its performance
new data.
For iterative training methods- - like gradient descent based methods - overfitting is often detected by observing the learning curves for training and validation error against the number of training steps. A typical behavior is that in the beginning of the training both training and validation loss are decreasing. This is the region where the model is still underfitting. At some point, when the model starts overfitting the training data, only the training loss decreases further while the validation loss starts to increase. The aforementioned early stopping method balances under- and overfitting by stopping the training once the validation loss starts to increase.
While supervised learning one can easily select well fit model by using the validation loss, in reinforcement learning one cannot apply this technique as the dataset is not fixed but dynamic and is constantly growing or changing. Furthermore, the quality of the current policy influences the quality of the data collected in the future. Even though learning a world model is in principle a supervised task, this problem also occurs in the model- -ased RL framework.
3.3 DYNAMIC UPDATE-TO-DATE RATIO
A typical hyperparameter in many RL algorithms is the update-to-data (UTD) ratio which specifies the number of update steps performed per environment step (i.e., per new data point). This ratio can in principle be used to balance under- and overfitting as one can control it in way that not too few or too many updates steps are done on the currently available data. However, several problems arise while optimizing this parameter. First, it is very costly to tune this parameter as requires run the complete RL. training several times making it infeasible for many potential applications. Second, the assumption that one fixed value is the optimal choice during the entire training duration does not necessarily hold. For example, if data from a newly explored region of the environment is added to the replay buffer might be beneficial to increase the number of update steps.
To address these problems, we propose - -rrpoos--UTD--.. a new method that dynamically adjusts the UTD ratio during training. It is inspired by the early stopping criterion and targets at automatically bal- ancing under- and overfitting online by adjusting the number of update steps. As part of the method, we store some of the experience in a separate validation buffer not used for training. Precisely, every d environment steps we collect consecutive transitions from few separate episodes dedicated to validation and every i environment steps the world model is evaluated on the validation buffer, where k should be much smaller than d. As the world model learning task is supervised this is easily done by recording the loss of the world model on the given validation sequences. The current validation loss then compared to the validation loss of the previous evaluation. If the loss has decreased, we assume the model is still in the underfitting regime and increase the UTD rate by specified amount. If the loss has increased, we assume the model to be in an overfitting regime and hence reduce the UTD rate. To allow for finer resolution at the high-update side of the allowed interval we adjust the UTD rate in log-space, meaning it is increased or decreased by multiplying with a value of or l/c respectively, where e is slightly larger than 1. The update formula at time step & then becomes
ratio, = utd ratio b; b b= C, if validation < validation_loss:
(1)
L. if validation 2 validation loss;
DUTD is a general method that can be applied to any model-based RL. algorithm that learns world model a supervised way. The implementation can be either terms of the UTD ratio or the data- to-update ratio which is its inverse and which we call IUTD (i.c., the number of environment steps per update step). It is more convenient to use the UTD ratio if several updates are performed per environment step and the IUTD if an update step is only performed after some environment steps. Methodologically, the two settings are the same as the two ratios describe the same quantity and are just the inverse of each other.
A high-level overview of DUTD is shown in Figure D and the pseudocode is described in Algo- rithm both explained in terms of the IUTD ratio as will apply DUTD the DreamerV2
4

---

Published as a conference paper at ICLR 2023
algorithm Hafner et al. (2021) for which several update steps per environment step become compu- tationally very costly. However, in both framework both scenarios can be addressed by letting the ratio be fractional.
3.4 APPLYING DUTD TO DREAMERV2
We apply DUTD to Dreamer V2 Hafner
Algorithm 1 DUTD (in terms of inverted UTD ratio)
et al. (2021, which is model-based RL algorithm that builds on Dreamer Hafner
Input: Initial inverted UTD ratio iutd_ number of steps after which additional validation data is col- lected d, number of validation transitions collected 8, steps after which the iutd_ratio is updated k, iutd up- date increment c
al. (2020) which again builds PlaNet
Hafner et al. (019). DreamerV2 learns a world model through latent imagination. The policy is learned purely in the la- tent space of this world model through an actor-critic framework. It is trained on imaginary rollouts generated by the world model. The critic is regressed onto A-targets Schulman et al. (2015); SSttonn & Barto (2018) and the actor is trained by a combination of Reinforce Williams (1992) and dynamics backpropagation loss. The world model learns an image encoder that maps the input a cate- gorical latent state on which a Recurrent Space- Model [Hatner et al.] (2019) learns the dynamics. Thrre predictors for image, reward, and discount factor are learned on the latent state. The to- tal loss for the world model a combina- tion of losses for all three predictors and a
for = 1 to total num of env steps do
Act according to policy Taa 8) and observe next state
if mod d = 0 then
Collect transitions and store experience in separate validation buffer; increment / = (++ end if
if mod iutd ratio == 0 then
Perform one training step of the transition model end if
if' mod == 0 then
Compute model loss L on validation dataset # Overfitting
2 Lirrecious then
iutd_ratio iutd_ratio
else
# Underfitting
iuld ralio iuld ralio/c end if pyrecious end if end for
loss between the la- tents predicted by the dynamics and the latents from the encoder.
To apply DUTD evaluate the image reconstruction loss on the validation set. Other choices also possible but we speculate that the image prediction is the most difficult and important part of the world model. One could also use combination of different losses but then one would potentially need a scaling factor for the different losses. As we want to keep our method simple and prevent the need of hyperparameter tuning for our method, we employ the single image loss. The source code of our implementation is publicly available
EXPERIMENTS
We evaluate DUTD applied to DreamerV2 on the Atari 100k benchmark Kaiser et al. (2019) and the DeepMind Control Suite Tassa et J.I][l (2018). For each of the two benchmarks we use the respective hyperparameters provided by the authors in their original code base. Accordingly, the baseline IUTD ratio is set to value of 5 for the control suite and 16 for Atari which we also use as initial value for our method. This means an update step is performed every and 16 environment steps respectively. For both benchmarks we set the increment value of DUTD to = 1.3 and the IUTD ratio is updated every 500 steps which corresponds to the length of one episode the control suite (with a frame- skip of 2). Every 100, 000 steps DUTD collects 3, 000 transitions of additional validation data. We cap the IUTD ratio in the interval [1. 15] for the control suite and in [1, 32] for Atari. This is in principle not necessary and we find that most of the time the boundaries, especially the upper one, is not reached. A boundary below would be possible by using fractions and doing several updates per environment step, but this would be computationally very expensive for DreamerV2. All other hyperparameters are reported in the Appendix. They were not extensively tuned and we observed that the performance our method is robust with respect to the specific choices. The environment steps in all reported plots also include the data collected for the validation set.
https://gii thub. com col www.coolnnooouio dut.
5

---

Published as a conference paper at ICLR 2023
Median
Mean
Optimality Gap
DUTD-DreamerV2 DreamerV2-
0.2
0.16
0.32
0.60 0.65
Human Normalized Score
Figure 2: Aggregated metrics over 5 random seeds on the 26 games of Atari 100k with 95% con- fidence intervals according to the method presented in [Agawall al. (202T). The intervals are estimated by the percentile bootstrap with statified sampling. Higher mean, median, interquantile mean (IQM) and lower optimality gap are better.
The Atari 100k benchmark Kaiser et al. (201 includes 26 games from the Arcade Learning Envi- ronment Bellemare et al. (201)) and the agent is only allowed 100, 000 steps of environment inter- action per game, which 400, 000 frames with frame-skip of 4 and corresponds roughly two hours of real-time gameplay. The final performance per run is obtained by averaging the scores of 100 rollouts with the final policy after training has ended. We compute the human normalized score of each continuous control. Agents receive pixel inputs and operate with frame-skip of 2 in the original DreamerV2. V2. We trained for million frames on most environments and to save computation cost for million frames if standard DreamerV2 already achieves its asymptotic performance well before that mark. The policy is evaluated every 10, 000 frames for 10 episodes. For both benchmarks, each algorithm is trained with 5 different seeds on every environment.
human scure rundom scure" The DeepMind Control Suite provides several environments for
score rrandom score
Our experiments are designed to demonstrate the following:
The UTD ratio can be automatically adjusted using our DUTD approach
DUTD generally increases performance (up to 300% on Atari 100k) by learning an improved world model compared to the default version of DreamerV2
DUTD increases the robustness of the RL agent with regard to learning-related hyperparameters
DUTD competitive with the best UTD hyperparameter found by an extensive grid search
4.1 PERFORMANCE OF DUTD COMPARED TO STANDARD DREAMERV2
For Atari100k, Figure shows results aggregated over the 26 games with the method of Agarwal al. (02T), where the interquantile mean (IQM) ignores the bottom and top 25% of the runs across all games and computes the mean over the remaining. The optimality gap describes the amount by which a minimal value of human level performance not reached. In Figure II present the learning curves for each environment. The results show that DUTD achieves a drastically stronger performance on all considered metrics compared to DreamerV2 with the fixed default IUTD ratio of 16. It increases the interquantile mean (IQM) score by roughly 300% and outperforms the human baseline in terms of mean score without any data augmentation.
800
800
400
DreamerV2-IUTD. 5
DUTD-
1.5 Environment Steps (in millions)
Sample efficiency curves aggregated from the results for environments of the DeepMind Con- trol Suite for DreamerV2 with the default UTD ratio and when it is ad- justed with DUTD. The IQM score at different training steps is plotted against the number of environment steps. Shaded regions denote point- wise 95% stratified bootstrap con- fidence intervals according to the method by Agarwal et al. (021).
Figure 3:
Figure shows the aggregated results for two million frames over ten environments of the Control Suite, which we list the Appendix. The curves per environment are presented in Figure of the Appendix further including results for ten more environments which the algorithms run until one million frames. Compared to the manually set default UTD every
ratio, DUTD matches improves the performance environment. Overall, DUTD improves the performance sig- nificantly although its average IUTD rate over all games and checkpoints is 5.84 similar to the default rate of showing that DUTD better exploits the performed updates.

---

Published as a conference paper at ICLR 2023
Acrbbot SwirbotSerguu
Carbbre Swingup
Chooton Run
HopporHop
Quachiped Walk
DJTD-DreameVV Ve- DreasiV2_in iited
080
300
200
100
Stapss
ETRFOrmTMTT Stapps
1377 EESTeer Stace
Ennionmmett S3age
Figure 4: Learning curves for five environments the Control Suite for DUTD-DreamerV? and standard DreamerV2 when non-default learning rates used. The first row shows the results for lower than default learning rate of 0.0001 and the second row for a higher one 0.001. The default learning rate is 0.0003 and its results shown in Figure 12 The solid line represents the mean and the shaded region pointwise standard deviation in each direction computed over 5 runs.
Median
Mean
Optimality
DUTD DreamerV2.
DreamerV2_ J
DreamerV2 IUTD10-
Dreamerv2 JUTD7-
DreamerV2 JUTD4-
DreamerV2 IUTD2-
DreamerV2_JUTD1
0.1
0.2
0.3
0.16 0.24 0.32
0.75
1.00
1.25
0.60
0.66
0.72
Human Normalizdd Score
Figure 5: Aggregated metries 5 random seeds on the 26 games of Atari 100k, ef. Figure 2 the methodology. DUTD is compared to Dreamer with different choices for the IUTD rate.
4.2 INCREASED ROBUSTNESS WITH DUTD
As DUTD dynamically adjusts the UTD ratio which allows to modify the training process online, we formed the hypothesis that with DUTD the underlying RL algorithm is more robust to suboptimal learning hyperparameters. Similar to supervised learning on fixed dataset the optimal number of updates to tradeoff between under- and overfitting will be highly dependent on hyperparameters like the learning rate. To investigate this, evaluated DreamerV2 with and without different learning rates of the dynamics model. The standard learning rate on the control suite is 0.0003. Hence, trained with both higher learning rate of 0.001 and lower of 0.0001 on a subset of the environments. The resulting learning curves are displayed in Figure Compared to the default learning rate the performance of DreamerV2 with the standard fixed IUTD ratio of 5 is overall lower and decreases substantially for some of the environments for both non-default learning rates. However, using DUTD the algorithm achieves considerably stronger results. This shows that using DUTD the algorithm is more robust to the learning rate, which is an important property when the algorithm is applied in real world settings such as robotic manipulation tasks, since multiple byperparameter sweeps are often infeasible in such scenarios. The need for more robustness as offered by DUTD is demonstrated by the performance drop of DreamerV2 with learning rate differing by a factor of 3 and the fact that on Atari different learning rate is used.
method for
4.3 COMPARING DUTD WITH EXTENSIVE HYPERPARAMETER TUNING
In the previous sections, we showed that DUTD improves the performance of DreamerV2 with its default IUTD ratio significantly. Now we want to investigate how well DUTD compares to the best hyperparameter value for IUTD that can be found through an extensive grid search on each benchmark. While for many applications such search is not feasible we are interested in what can be expected of DUTD relative to what can be regarded as the highest achievable performance.

---

Published as a conference paper at ICLR 2023
Acnobot Swingup
Cortool Swingup
Chestoh Run
Hopper Hop
Guadruped Walk
Invern UTD
DUTO DreamorV2
Dreamee/2
Endhrrmmet
Invirommnt Steps
Enserommnt Since
Enwinneeent Sleps
Envitommet Sinpe
Figure 7: IUTD ratio against environment steps for DUTD and the standard DreamerV2 on five environments. For each environment the mean over 5 runs is plotted as the solid line and the shaded region represents one pointwise standard deviation in each direction.
On the Atari 100k benchmark we evaluate DreamerV2 with IUTD rates of 1,2.4, 7, 10 and 16 (the default value) and denote the algorithms with DreamerV2-IUTDD aggregated results over all games and seeds in Figure 5 show an increase in performance when the number of updates increases up to IUTD rate of Increasing it further leads to declining results. Thus, there is sweet spot and one can not simply set the IUTD rate very low and expect good results. Averaged over all runs and checkpoints the IUTD rate of DUTD at 3.91 which is in the region of the best performing hyperparameters of 2 and 4. This is also reflected by the fact that DUTD achieves similar performance to these two optimal choices.
DreamerV2-IUTD 2, etc. The
We further evaluate Dreamer V2 with IUTD ratios of 2, (the default one), 10, and 15 on ten environments of the control suite. An IUTD value below 2 is not possible as a single run would take roughly two weeks to run on our hardware. The aggregated sample efficiency curves in Figure 6 further support the hypothesis that DUTD is
6:00
Score
400
DUTD-DreamerVz
competitive with the results of an extensive grid search. Only an IUTD choice of 2 gives slightly better sample ef- ficiency but reaches lower final performance. To further investigate the behaviour of DUTD we report the adjusted inverted UTD ratio over time for five environments in Figure 7 and for all environments in Figure [3 in the Appendix. Interestingly, the behavior is similar for all the environments. At the start of the training, the ratio very low and then it quickly oscillates around value of roughly 5 for most environments and an even higher value for a few others. On cheetah_run and hopper_hop the IUTD oscillates around the default value of 5 most of the time and still, DUTD reaches a higher performance than Dreamer as can be seen in the single environment plot in Figure 12 of the Appendix. This result supports the hypothesis that a static IUTD) rate be suboptimal for some environments and that DUTD successfully balances over- and underfitting during the training process.
Dreame/V2--UD_ 22
Dreeme-V2-UTTD.5
DreamerV2-UUDD 10
DreamervZ.UUD
0.0
0.5
1.0
1.5
2,0
Environment Sleps (inmillonn)
Figure 6: Sample efficiency curves showing the IQM score aggregated from the results for ten environments of the DeepMind Control Suite for DreamerV2 with different choices for the IUTD ratio. Shaded regions denote pointwise 95% stratified bootstrap con- fidence intervals.
4.4 EVALUATION FOR A HIGH NUMBER OF SAMPLES
Next we investigate the behaviour of DUTD if training is continued for many environment steps. We randomly selected games from the Atari benchmark and trained the algorithms for 40 million frames. The resulting learning curves displayed in FigureBJshow that DUTD maintains its advantage also this setting. The significantly improved performance achieved with an IUTD ratio of 13.51 averaged over all games and checkpoints. In Figure of the Appendix we show the development of the IUTD ratio over time for each environment. We can see that with DUTD after an initial phase with lower IUTD ratio it oscillates around a value not too far from the highly tuned default ratio of 16. This means DUTD significantly improves performance over plain DreamerV2 without requiring substantially more updates. The experiment further highlights the benefits of DUTD. Evaluating different choices for a fixed IUTD ratio in this setting is highly expensive and for low values of the TUTD ratio almost impossible single run with the default value takes already several days
8

---

Published as a conference paper at ICLR 2023
Al
en
Anidar
Pacman
Qoet
Seaquest
2000
5000
2000
X00
2000
89
Enderommnt Frames
Envirommet Frames
An
Envirommnt Frames
Enstrnmett
Envirommnt Finmes
Figure 8: Learning curves for DreamerV2 with and without DUTD on 5 randomly selected environ- ments of the Atari benchmark. For each environment the mean over 3 runs plotted as the solid line and the shaded region represents one pointwise standard deviation in each direction.
to train. DUTD improves upon the highly tuned default choice and removes the need to tune this hyperparameter in an inner loop.
4.5 GENERALITY OF DUTD
To demonstrate the generality of DUTD we applied it to PlaNet Hafner al. (2019) which is an- other model-based RL algorithm. We evaluated the resulting method on three environments of the DeepMind Control Suite using the same hyperparameters for DUTD as for Dreamer. The results in Figure of the appendix show that DUTD also improves the performance of PlaNet validating that DUTD is general method and indicating its usefulness for different base algorithms.
DISCUSSION
We presented novel and general method denoted DUTD that is designed detect under- and overfitting on evolving datasets and is able to dynamically adjust the typically hand-set UTD ratio an automated fashion. As in early stopping, the underlying rationale is that too many updates can lead to overfitting while too few updates can lead to underfitting. DUTD quickly identifies such trends by tracking the development of the world model performance on validation set. It then accordingly increases or decreases the UTD ratio in the case of underfitting or overfitting.
In our experiments, we demonstrated how successfully apply DUTD model-based RL al- gorithm like DreamerV2. The experiments show that DUTD can automatically balance between the under- and overfitting of the world model by adjusting the UTD ratio. As a result, DUTD re- moves the burden of manually setting the UTD ratio, which otherwise needs to be tuned for new environments making it prohibitively expensive to apply such algorithms in many domains. At the same time, DUTD increases the performance of DreamerV2 significantly compared to its default UTD rate and is competitive with the best hyperparameter found for each domain through an exten- sive hyperparameter search. Moreover, notable property of changes in the learning rate. This is important, as the learning rate often has to be tuned for new environments. For example, in DreamerV2 the default learning rate differs between Atari and the DeepMind Control Suite. In the context of real world problems such tuning is undesirable and often too costly. At the same time, the hyperparameters of DUTD can easily be set and do not have big influence on the final performance. We recommend updating the UTD rate after fixed time interval that is similar to the average episode length. The data used for validation should not exceed 10% of all data.
is its robustness to
An interesting avenue for future work would be explore non nun-super objectives for model-free RL algorithms that can be used for evaluation on the validation set. This would allow the usage of DUTD to adjust the UTD ratio of such algorithms. Another potential way to further boost the performance of our method is use k-fold cross validatio with ensemble of world models such that every transition can be used for training.
We are convinced that DUTD is further step in the direction of autonomy and the easy applicability of RL algorithms to new real world problems without the need to tune any hyperparameters in an inner loop. More generally, work shows that it might be fruitful to use knowledge about the underlying learning dynamics to design algorithms that dynamically adjust parts of the learning algorithm.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work was supported by the European Union's Horizon 2020 Research and Innovation Program under Grant 87 17144--OpenDR.
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Deep reinforcement learning the edge of the statistical precipice. Advances in Neural Informa- lion Processing Systems, 34, 2021.
RS Anderssen and PM Prenter. A formal comparison of methods proposed for the numerical solution of first kind integral equations. The ANZIAM Journal, 22(4):488-55) -500, 1981.
Dilip Arumugam, David Abel, Kavosh Asadi, Nakul Gopalan, Christopher Grimm, Jun Ki Lee, Lu- cas Lehnert, and Michael Littman. Mitigating planner overfitting in model-based reinforcement learning. arXiv preprint 1rxiv:181..01129, 2018.
Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Zhaohan Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. In International Conference on Machine Learning, pp. 507-517. PMLR, 2020.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Yutian Chen, Aja Huang, Ziyu Wang, loannis Antonoglou, Julian Schrittwieser, David Silver, and Nando de Freitas. Bayesian optimization in alphago. arXiv preprint arXiv:1812.00885, 2018.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn- ing in a handful of trials using probabilistic dynamics models. Advances in Neural Information Processing Systems, 31, 2018.
Marc Deisenroth and Carl E Rasmussen. Pilco: A model-based and data-efficient approach policy search. In Proceedings of the 28th International Conference on machine learning (ICML-I1), pp- 465-472. Citeseer, 2011.
Nicolai Dorka, Joschka Boedecker, and Wolfram Burgard. Adaptively calibrated critic estimates for deep reinforcement learning. In Deep RL Workshop NeurIPS 2021, 2021.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha. Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, pp. 2555-2565. PMLR, 2019.
Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2020.
URL https::/ http::///ppeevvveeeeee ewwereeeieww
1d=S110TC4tDS
Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In International Conference Learning Representations, 2021. URL
Ittps://openzevier net/ forum? id 1d00baayybbbb
Nan Jiang, Alex Kulesza, Satinder Singh, and Richard Lewis. The dependence of effective planning horizon model accuracy. In Proceedings the 2015 International Conference on Autonomous Agents and Multiagent Systems, pp. 1181-1189. Citeseer, 2015.
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Mitos, Blažej Osiński, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model based reinforcement learning for atari. In International Conference on Learning Representations, 2019.
Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Rein- forcement learning with augmented data. Advances in Neural Information Processing Systems, 33, 2020.
10

---

Published as a conference paper at ICLR 2023
Nelson Morgan and Hervé Bourlard. Generalization and parameter estimation in feedforward nets: Some experiments. Advances in neural information processing systems, 2:630-637, 1989.
Charles Packer, Katelyn Gao, Jemej Kos, Philipp Krähenbühl, Vladlen Koltun, and Dawn Song. Assessing generalization in deep reinforcement learning. arXiv preprint arXiv: 12282, 2018.
Jack Parker-Holder, Raghu Rajan, Xingyou Song, André Biedenkapp. Yingjie Miao, Theresa Eimer, Baohe Zhang, Vu Nguyen, Roberto Calandra, Aleksandra Faust, et al. Automated reinforcement learning (autorl): A survey and open problems. arXiv preprini arXiv:2201 1.0396,, 2022.
Luis Pineda, Brandon Amos, Amy Zhang, Nathan O. Lambert, and Roberto Calandra. Mbrl-lib: A modular library for model-based reinforcement learning. Arxiv, 2021. URL https://arxiv.
brg/aba/22 2104. 10159
Roberta Raileanu, Maxwell Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus. Automatic data augmentation for generalization in reinforcement learning. 2020.
Tom Schaul, Diana Borsa, David Ding, David Szepesvari, Georg Ostrovski, Will Dabney, and Simon Osindero. Adapting behaviour for learning progress. arXiv preprint 1912vv:9/2.000990 2019.
Julian Schrittwieser Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839) 2020.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.024433 024388 2015.
Max Schwarzer, Ankesh Anand, Rishab Goel, Devon Hjelm, Aaron Courville, and Philip Bach- man. Data-efficient reinforcement learning with pred-prre representations. In International Conference on Learning Representations, 2021. UR//ttpp::/ooenneeiie... foormmr
id=uCOoPZZRaUUL
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419): 140- 1144, 2018.
Xingyou Song, Yiding Jiang, Stephen Tu, Yilun Du, and Behnam Neyshabur. Observational overfit- ting in reinforcement learning. In International Conference on Learning Representations, 2020.
URL httttst:/pppereeetttee wttp://wwww...eet...eeeeeee..ee.
/ forum? Pinammii--HJa Ho 12h111
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 1929-11929-1958, 2014. URL /
srivastaval4a. 1htmm111....111111... html
Otto Neall Strand. Theory and methods related the singular-function expansion and landweber's iteration for integral equations of the first kind. SIAM Journal on Numerical Analysis, 11(4): 798-825, 1974.
Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. MIT Press, 2018. ISBN 78-0262039246.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, al. Deepmind control suite. arXiv preprint / 000600, 2018.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229-22 2566, 1992.
Amy Zhang, Nicolas Ballas, and Joelle Pineau. A dissection of overfitting and generalization in continuous reinforcement learning. arXiv preprint arXiv:1806.0799 077337 2018a.

---

Published as a conference paper at ICLR 2023
Baohe Zhang, Raghu Rajan, Luis Pineda, Nathan Lambert, André Biedenkapp. Kurtland Chua, Frank Hutter, and Roberto Calandra. On the importance of hyperparameter optimization for model-based reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pp. 4015 PMLR, 2021.
Chiyuan Zhang, Oriol Vinyals, Remi Munos, and Samy Bengio. A study on overfitting in deep reinforcement learning. arXiv preprint arXiv:1804.088 06893, 2018b.
12

---

Published as a conference paper at ICLR 2023
A FURTHER RESULTS
A. APPLYING DUTD TO PLANET
To demonstrate the generality of DUTD we additionally applied it to PlaNet Hafner al. (2019)
with the same hyperparameters for DUTD as we also used for DreamerV2. As base source code on which we implemented DUTD we Pinedaa et al. (202T). We evaluated the resulting algorithm on three environments of the DeepMind Control Suite that were also used in the original publication of PlaNet. We used seeds and evaluated the algorithms every 25000 environment frames. The results in Figure show that DUTD also improves the performance of PlaNet. This is further evidence for the generality of DUTD.
Carppole Swingup
Cheelah
Average Evaluation Reaur
800
700
400
400
600
100
850
100
300
DUTD-PlsNot
100
PlaNet
100
50200 10000 380000 250060
Environment 200000
153000 100000
Figure 9: Learning curves for PlaNet with and without DUTD on three environments of the Deep- Mind Cotrol Suite. The solid line the mean over seeds and the shaded area represents one pointwise standard deviation. We used uniform filter of size 3.
A.2 DETAILED RESULTS FOR APPLYING DUTD TO DREAMERV2
The ten environments of the DeepMind Control Suite used to generate the aggregated curves in the Figures and are: acrobot swingup, cheetah run, finger turn easy, finger turn hard, hopper hop, quadruped run, quadruped walk, reacher hard, walker walk, and walker run.
We evaluated on all 20 environments used in the original Dreamer paper Saperr Hafner al. (2020) but to save computation stopped training for ten enviromments at 1 million steps because standard Dreamer already reaches its asymptotic performance well before that mark. The aggregated curves are gen- erated from the other 10 environments for which training ran until 2 million steps. Figure 12 shows the single learning curves for all environments. Please note, that on the million steps environments with DUTD the asymptotic performance is reached much faster often twice fast.
In the Figures To each single environment.
12
and 15 we present the more detailed results of our experiments for
Acrobot Swi rguo
Chectan Run
Firgor T.m Easy
Fngee Tum Hard
Hopper Hop
Avarage Saatindoo wn Metumm
3ramee22
200
Stops
Stops
Figure 10: Learning curves for different choices of the IUTD ratio for each of the environments. The solid line is the mean over 5 seeds and the shaded area represents one pointwise standard deviation.
13

---

Published as a conference paper at ICLR 2023
096
Nestogo Silledar Rut.m
Afrecic
Freetury
Frosthlee
Gopber
Hero
Avenje Eral. of Petum
2000
Manae M
154
Jamresonnd
Kar Kangaroo
Ktul
Kung Moster
Ms Pooman
Netropa Notrop--eaaa unrur Remm
of
pusn
vot
Phonag
Prinade Eye
Oberrt
Rcad Purrne
Sexquesst
Reargg breuetonn Takum
606
6000
180
Up N
40100
Dasrage Destasion
2000
1008
Frene
Figure 11: Learning curves for DreamerV2 with and without DUTD on the 26 environments of the Atari 100k benchmark. The solid line is the mean over seeds and the shaded area represents one pointwise standard deviation.
14

---

Published as a conference paper at ICLR 2023
Bolanne
Carppolc
Ressnge Eusuuttin
600
DUTD-Dmemnnn Dmoomrrm Disamery?
200
200
34
Cap
Firger Spin
Farger Easy
Finger
Anarage Eveluaton
400
210
Hoppor Hop
HopperSrand
Pendurm Swioup
Quadruzed Rur
Quadruped Walk
Keeegg Rreee
88
Acaraga bosd.
86
62
Figure 12: Learning curves for DreamerV2 with and without DUTD for 20 environments of the DeepMind Control Suite. The solid line the mean 5 seeds and the shaded area represents one pointwise standard deviation.
15

---

Published as a conference paper at ICLR 2023
DJTD-Dmmamer22
Cheetsh Pan
Cup Calch
Finger Soin
Ernger Tten Eaay
Firger Turn Hard
Inveted UTD Rato
60
00
00
Hopper Hop
Hoppor
Swinggup
Guadruppd
Inveted UTD Salt
Reachee
Reacher
Wasker Run
Stand
Wasker Walk
UTD Rato
Figure 13: IUTD ratio against environment steps for DUTD and the standard DreamerV2 on all environments. For each environment the mean over 5 runs is plotted as the solid line and the shaded region shows represents one pointwise standard deviation in each direction.
Amidar
1/70
Elwkonnent Frames
Enshormont
Enviornent Farres
Envirommnn Franiss
Enrronoon Framie
Figure 14: IUTD ratio against environment steps for DUTD and the standard DreamerV2 on environments of Atari for which the algorithms were trained until 40 million frames. For each environment the mean over 3 runs is plotted the solid line and the shaded region shows represents one pointwise standard deviation in each direction.
16

---

Published as a conference paper at ICLR 2023
496
Nestogo Sieledar RoL.m
1.30
Demon Afrscc
Freetury
Frosthlee
Gogber
1150
Jameebond
Ka Kangaroo
Ktul
Kung Moster
Ms Pooman
Netroge E4l LoPion Ronm
Not
10000
vo0
Phon
Privdde Eye
Obert
Rcadd Plurner
Sexquesst
Aesrgga Ere Ereluaton Takum
20080
1000
1.00
Up
09000
Passagg
1900
Frames
Figure 15: Learning curves for different choices of the IUTD ratio for each of the 26 environments of the Atari 100k benchmark. The solid line is the mean over 5 seeds and the shaded area represents one pointwise standard deviation.
17

---

Published as a conference paper at ICLR 2023
B HYPERPARAMETERS
In Table 1 we give an overview of all hyperparameters related to DUTD. All other hyperparameters the standard DreamerV2 hyperparameters as given the open source codebase On the DM Control Suite we reduced the number of steps after which to collect new data for the validation set by a half during the first 400k steps as for some environments strong policy is learned very quickly and hence validation set with more recent transitions that better represent the kind of transitions the agent encounter makes more sense. We have because we started our first experiments with this but from some limited additional experiments it seems not to have a big impact on performance.
Table I: Hyperparameters values for DUTD applied to Dreamer V2 and the corresponding hyperpa- rameter in the original DreamerV2.
HYPERPARAMETER
ATARI
DM CONTROL
INITIAL IUTD RATIO
LOWER BOUNDARY FOR THE IUTD RATIO
UPPER BOUNDARY FOR THE IUTD RATIO
32
15
IUTD UPDATE INCREMENT C
1.3
1.3
NUMBER OF STEPS
WHICH TO UPDATE THE IUTD RATIO
500
500
VALIDATION SET MAXIMUM SIZE -
12,000
10,000
NUMBER OF STEPS AFTER WHICH TO COLLECT NEW DATA FOR
THE VALIDATION SET - d
100,000
100,000
NUMBER OF ADDITIONAL TRANSITIONS
VALIDATION SET EACH TIME NEW VALIDATION DATA IS COLLECTED
3,000
3,000
STANDARD DREAMERV2
IUTD RATIO
16
5
C HYPERPARAMETER SENSITIVITY OF DUTD
Most hyperparameters of our method are straightforward to set and do not need any tuning. Updating the UTD ratio after the maximum episode length of 500 in DM Control Suite (DMC) is value that we directly transferred to the Atari benchmark without further tuning. The initial value for the UTD ratio has effect, it gets quickly adjusted. The lower and upper limits for the UTD ratio are not reached often and hence do not affect performance given they are chosen lavish enough. We did not tune those. We tried a few choices for the number of additional transitions each time new validation data is collected and the number of steps after which we do so but did not find it to affect performance and fixed one choice for both benchmarks.
The multiplicative factor e is the most important hyperparameter of our method and we hence con- ducted an additional experiment evaluating its sensitivity on the Atari 100k benchmark over 5 random seeds. We show the aggregated metrics for different multiplicative factors in Figure [6
The results show that seen over all metrics and relative to the baseline results the performance not very sensitive with respect to the choice of the multiplicative factor. For the mean our default factor of 1.3 even gives slightly worse results than all other factors. Further, we argue the fact that the same setting of hyperparameters of DUTD works for very different benchmarks, Atari and DMC, shows that DUTD is not very sensitive to its hyperparameters and that the default values given by will most likely work for wide range of tasks. While extensive hyperparameter search for the optimal UTD ratio might give slightly better results than DUTD with some fixed multiplicative factor, DUTD is still favorable for many real world applications where such tuning is too costly.
18

---

Published as a conference paper at ICLR 2023
Median
IQM
Mean
Optimality Gap
DUTD-1.5
DUTD-1.4
DUTD-1.3
DUTD-1.1
DreamerV2_iUTDD
0.2
0.16
0.32
0.6
0.9
1.5
0.60 0.65 0.70 0.75
Human Normalized Score
Figure 16: Aggregated metrics over 5 random seeds the 26 games of Atari 100k, cf. Figure for the methodology. We investigate the sensitivity of DUTD to its own most important hyperparameter for values of 1.1, 1.2, 1.3 (default used in the main experiments), 1.4, and 1.5.
19


Reviews:
[{"review_id": "jc6MJz-lzL", "reviewer": "Reviewer_U8TF", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to improve the training of Model Based Reinforcement Learning methods by introducing an automatic method of setting/dynamically tuning the update-to-data (UTD) ratio, a key hyper parameter which controls how much the world model over- or under-fits the training data.  They propose using a held out set of validation samples, updated throughout training, to estimate whether the world model is over- or under-fitting and to update the UTD ratio accordingly.  This method both enables the parameter to be set automatically, avoiding the need for hyper parameter tuning, and also allows the parameter to be dynamically updated as the training stage evolves.  They present results using their approach with DreamerV2 on both the DeepMind Control Suite and the Atari100k benchmark, demonstrating comparable performance on Atari100k and a slight improvement on DeepMindControl Suite when compared to training with the best fixed value of UTD when it is carefully tuned.  Additionally they show that the final performance on both environment is highly dependent on carefully tuning UTD when using a fixed value, but the performance of their method is robust to a wide range of values for its primary hyper-parameter.", "summary_of_the_review": "This paper (1) provides strong evidence that tuning the UTD parameter substantially impacts the outcomes of DreamerV2 training (2) provides a method for automatically tuning the UTD parameter that consistently matches or improves the performance of the best fixed value of UTD when chosen with careful tuning.  \n\nThis method is likely broadly applicable to MBRL practitioners and can be used as a standard tool for effectively training world models, saving time/resources in hyper parameter tuning and making results more directly comparable by removing the confounding factor of how well tuned the UTD parameter is.  However, it is not clear that this is the case since the method has only been evaluated on one such method, DreamerV2 (my main concern when making my acceptance recommendation).  Additionally, there isn’t clear evidence for the claim that *adaptively* setting UTD throughout training is more beneficial than choosing the optimal fixed UTD (my main concern when evaluating correctness at 3 out of 4).", "strength_and_weaknesses": "Strengths:\n- The method, though intuitive, is not standard practice and they make strong case for why DUTD could benefit all MBRL practitioners by improving their training performance while reducing the amount of manual tuning necessary.  Such improvements also enable fairer comparisons between future methods by removing one of many possible confounding factors in their evaluation.\n- Thorough experimentation and presentation of findings, including their many plots in the Appendix supports both drawing additional conclusions and reproducibility. Reporting results using methods described in Agarwal et al 2021 supports fair comparisons and a stronger ability to draw meaningful conclusions.\n- Not mentioned in the work, but the method also allows for the UTD parameter to be adaptively set for each individual task automatically, with different tasks using substantially different average values (as seen in Figure 12).\n\nWeaknesses:\n- These results are only evaluated on DreamerV2, therefore it is unclear if they extend to other MBRL methods or are Dreamer specific.\n- The claim that DUTD is more robust to hyper parameters than a fixed value of UTD is supported by comparing DUTD to the default fixed value of UTD which was already demonstrated to be sub-optimal.  Since it was already demonstrated that DUTD outperforms the default UTD with the default hpms then it is unsurprising that this continues to be the case when changing the hpms.  To isolate whether the adaptive choice of UTD as done by DUTD is more robust to varying hpms (the goal of this ablation), it should be compared to the best fixed UTD value as chosen by careful tuning.\n- Similarly, it seems a possible takeaway from the longer runs is the value of adaptive UTD setting as we see in Figure 13 that the value chosen by DUTD varies throughout training.  However, because the comparison in Figure 8 is to the default value instead of the best tuned value of UTD, it is unclear whether there is benefit from adaptively setting UTD throughout training or just from choosing a good value of UTD.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "b2hUlubfPY", "reviewer": "Reviewer_V4T4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper takles the problem of overfitting in the RL setting, where early stopping is not directly applicable due to the evolution of the dataset. The proposed method adjusts the update to data (UTD) ratio during training which trades off between underfitting and overfitting. The experiments are conducted using a model-based RL algorithm (DreamerV2) on the DeepMind Control Suite and Atari 100k benchmark, which showed competitiave resutls compared with extensive hyperparameter search.  \n\n", "summary_of_the_review": "The paper is easy to follow and the proposed method leads to easy applicability of RL algorithms in real life, which will bring benefits especially to practitioners. However, the novelty of the paper is limited and it does not provide much insights into a principled way of trading off under- versus overfitting in RL problems. \n\n", "strength_and_weaknesses": "Strength: \n* The paper is well written and easy to follow. \n* The proposed method is easy to use in practice. \n* The experiments showed promising results with the DreamerV2 algorithm. \n\nWeakness: \n\nthe novelty of the paper is limited. The proposed method is a heuristic method which does not add much theoretical contribution. The form of the UITD ratio update is intuitive but it is not clear why this particular multiplicative form is chosen among other possibilities.  \n\nThe choice of c is not well explained or examined, I am also curious to know whether having an adaptive form of c that changes over time helps. \n\nAll the results are based on one model-based RL algorithm (DreamerV2).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FOPIwCUs3wi", "reviewer": "Reviewer_uze4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper looks at overfitting in world model learning. It proposes a mechanism to dynamically update the update to data ratio during training. This can be seen as an analogue of early stopping in supervised learning. The paper introduces a small validation replay buffer used for adjusting the UTD ratio and adapts it based on the target validation loss. The paper evaluates the approach building on the dreamerv2 method. Empirically this approach is competitive with dreamerv2 with a tuned UTD ratio.", "summary_of_the_review": "Nice paper that thoroughly evaluates a simple idea to dynamically update the UTD in world model based algorithms and shows strong empirical results. ", "strength_and_weaknesses": "Strengths:\n* simple idea that is relatively easy to implement and could potentially be extended to other settings as well.\n* clear, well-written paper with extensive empirical evaluation.\n* promising empirical results. The fact that the approach is competitive with a UTD ratio tuned by grid search is particularly promising because tuning the UTD is often very costly and time-consuming in practice.\n\nWeaknesses:\n* in appendix B there seems to be some special casing for the control suite. This detracts somewhat from the simplicity of the methods. Can you show empirically what happens when this is removed?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster