MAD-TD: Model-Augmented Data stabilizes High Update Ratio RL
Authors: Claas A Voelcker
Marcel Hussing
Eric Eaton
Amir-massoud Farahmand
Igor Gilitschenski
Venue: ICLR 2025 Spotlight (2025)
Keywords: reinforcement learning
model based reinforcement learning
data augmentation
high update ratios
Primary area: reinforcement learning
TL;DR: Using model generated data can lead to stable learning with high update ratios in off-policy RL

Abstract: Building deep reinforcement learning (RL) agents that find a good policy with few samples has proven notoriously challenging. To achieve sample efficiency, recent work has explored updating neural networks with large numbers of gradient steps for every new sample. While such high update-to-data (UTD) ratios have shown strong empirical performance, they also introduce instability to the training process.  Previous approaches need to rely on periodic neural network parameter resets to address this instability, but restarting the training process is infeasible in many real-world applications and requires tuning the resetting interval. In this paper, we focus on one of the core difficulties of stable training with limited samples: the inability of learned value functions to generalize to unobserved on-policy actions. We mitigate this issue directly by augmenting the off-policy RL training process with a small amount of data generated from a learned world model. Our method, Model-Augmented Data for TD Learning (MAD-TD) uses small amounts of generated data to stabilize high UTD training and achieve competitive performance on the most challenging tasks in the DeepMind control suite. Our experiments further highlight the importance of employing a good model to generate data, MAD-TD's ability to combat value overestimation, and its practical stability gains for continued learning.

Published as a conference paper at ICLR 2025
MAD-TD: MODEL-AUGMENTED DATA STABILIZES HIGH UPDATE RATIO RL
Claas A Voeleeer* University of Toronto Vector Institute
Marcel Hussing*
Eric Eaton
University of Pennsylvania
University of Pennsylvania
Amir-massoud Farahmand
Igor Gilitschenski
Polytechnique Montréal
University of Toronto
Mila Quebec AI Institute University of Toronto
Vector Institute
ABSTRACT
Building deep reinforcement learning (RI. agents that find good policy with few samples has proven notoriously challenging. To achieve sample efficiency, recent work has explored updating neural networks with large numbers of gradi- ent steps for every new sample. While such high update-to-data (UTD) ratios have shown strong empirical performance, they also introduce instability to the training process. Previous approaches need to rely periodic neural network parameter resets to address this instability, but restarting the training process is infeasible in many real-world applications and requires tuning the resetting interval. In this paper, we focus on onc of the core difficulties of stable training with limited sam- ples: the inability learned value functions to generalize to unobserved on-policy actions. We mitigate this issue directly by augmenting the off-policy RL training process with a small amount of data generated from a learned world model. Our method, Model-Augmente uses small amounts of generated data to stabilize high UTD training and achieve competitive performance the most challenging tasks in the DeepMind con- trol suite. Our experiments further highlight the importance of employing a good model to generate data, MAD-TD's ability to combat value overestimation, and its practical stability gains for continued learning.
Data for Temporal Difference learning (MAD-TD)
INTRODUCTION
Instead of solely relying data gathered by target policy, off-policy reinforcement learning (RL) aims to leverage experience gathered by past policies (Sutton & Barto, 2018) to fit a value function for the target policy. Ideally, training over many iterations should help extract important information from past data. However, recent work has shown that simply increasing the number of gradient update steps, the replay ratio or update-to-data (UTD) ratio, can lead to highly unstable learn- ing (Nikishin et al., 2022; D' D'Oro et al., 2023; Hussing et al., 2024; Nauman et al., 2024b).
Prior work has stabilized the learning by using double Q minimization to reduce overestimation (Fujimoto et al., 2018), training ensemble methods to improve value estimation (Chen et al., 2020; Hiraoka et al., 2022), introducing architectural regularization (Hussing et al., 2024; Nauman al., 2024b), or resetting networks periodically throughout the learning process (D'Oro et al., 2023; Schwarzer et al., 2023; Nauman et al., 2024b). However, while useful, pessimistic underestima- tion and architectural regularization are insufficient by themselves to combat the problem (Hussing et al., 2024), and so most methods resort to either network resets or ensembles. Critic ensembles can be expensive to train, and resetting has several important limitations: in real systems, re-executing random policy can be expensive or unsafe, the resetting interval needs to be carefully tuned (Hussing et al., 2024), and when storing a full reset buffer is infeasible, resetting loses important information.
We narrow in on a key issue contributing to unstable training: wrong value function estimation on unobserved on-policy actions (Thrun & Schwartz, 1993; Tsitsiklis & Van Roy, 1996). Off-policy

---

Published as a conference paper at ICLR 2025
RL uses the values of states sampled under old policies with actions from the target policy to update the value function. However, these state-action pairs themselves are not in the replay buffer and hence their value estimate is not directly improved by training. Consequently, learned function which achieves low error on seen data is not guaranteed to generalize well to actions that differ from past actions. This problem is related to overfitting (Li et al., 2023) and contributes overestimation (Thrun & Schwartz, 1993; Hasselt, 2010; Fujimoto et al., 2018). However, overfitting assumes that train and test set are drawn from the same distribution, while we focus on the distribution shift between data collection and target policy. Previous work has investigated the difficulty of off- learning due to this shift (Maei et al., 2009; Sutton al., 2016; Hasselt, 2010; Fujimoto al., 2018), yet there are no tractable mitigation strategies that work well in the high UTD regime with deep RL.
To corroborate our hypothesis that generalization to unobserved actions a major obstacle for train- ing at high UTDss we examine the behavior of value functions on on-policy transitions. Our ex- periments reveal that value functions generalize significantly worse to unobserved on on-policy action transitions than to validation data from the same distribution the training set. Building on this, we propose to improve on-policy value estimation by using model-generated on-policy data.
Previous investigations into model-based deep RL have focused learning values fully in model roll-outs (Buckman et al., 2018; Janner et al., 2019; Hafner et al., 2020; Ghugare et al., 2023) and the associated challenges (Zhao et al., 2023; Hansen et al., 2024). In contrast, show that mixing a small amount of model-generated on-policy data with real off-policy replay data is sufficient to achieve stable learning in the high UTD regime. Our method, Model-Augmenter Data for Temporal mitigates the generalization issues of the value function in the hard-
Difference Marning(MAD-TDD est tasks of the Didd control (DMC) benchmark (Tunyasuvunakool al., 2020b) and achieves strong and stable high UTD learning without resetting or redundant ensemble learning.
The main contributions of this work are twofold: First, we empirically show the existence of mis- generalization from off-policy value estimation to on-policy predictions. We connect this issue to the challenge of stable learning with high update ratios and highlight how increasing the update ratio increases Q function overestimation. Second, we provide a new method called MAD-TD that improves the value function accuracy on unobserved on-policy actions with model- generated data and stabilizes training at high update ratios. This method proves to have equivalent performance to or outperform previous strong baselines.
MATHEMATICAL BACKGROUND
7) with state space X, action space A, transition kernel P X'A M(X),, a reward function r X'A - R, starting state distribution E M(X') and a discount factor E [0, 1) (Puterman, 1994; Sutton & Barto, 2). For space Y we use M(Y) to denote the set of probability measures over the space. Our goal is to learn a policy π X + M(A) that maximizes the discounted sum of future rewards
We consider a standard RL setting, the discounted infinite-horizon MDP (X,A,P,r,,) P.
arg max
Σ
Epr (1)((1,,)))20 pl
TEll
where actions are sampled according to the policy and new states according to the transition kernel.
2.1 OFF-POLICY VALUE FUNCTION LEARNING
As an intermediate objective, many algorithms attempt to simplify the direct policy optimization problem by first learning a policy value function Q*, which is defined via a recursive equation
Q" r(z, a) "EEe 1Q (a', ,d))
The policy can then be incrementally improved by picking k (x) arg max QTL a) at every time step k. In practice, Q" and π are often parameterized as neural networks and learned from data. To increase the sample efficiency of the algorithm, it is common to store all collected interaction data independent of the collection policy in a replay buffer D = { ae Tt, 244 ) As the Q-value only depends on the policy via the policy evaluation at the next state, it is possible
2

---

Published as a conference paper at ICLR 2025
Figure 1: A visualization of the core issue investigate. Even if replay buffer contains good coverage for two policies (Told and Tnew) starting from P To. this does not guarantee that it contains transition for executing an action under the new policy on a state visited under the old. However, this state-action pair's value estimate is used to update the value of state zo via Equation 3, without being grounded in an observed transition.
m -aee
Space covered under
old 9
21
oli
41
Gg
Space covered under
to estimate Q-values from past interaction data by minimizing the fitted Q-learning objective
C (
Σ 1 1 D -
( ar)
[T+ 7Q (21+1,
with a' (T-1x1411) (3)
Here Hag denotes the stop gradient operation introduced to avoid the double sampling bias and all data contained in the replay buffer is colored blue. However, the value the next state 2++1 is evaluated with an action that is not guaranteed to be in the replay memory, as the target policy can be different from the policy used gather the sample. This means that we require the value to generalize to potentially unseen actions. We provide a visualization of this issue in Figure 1.
3 INVESTIGATING THE ROOT CAUSE OF UNSTABLE Q LEARNING
Minimizing Equation 3 finds the policy Q function over replay buffer with sufficient coverage of all states and actions that this policy visits. However, in most continuous control RL algorithms (Lil- licrap al., 2016; Haarnoja et al., 2018; Fujimoto et al., 2018). this update is interleaved with policy update steps. The data in D then necessarily becomes off policolicy as training progresses.
This means that the number of actor and critic optimization steps needs be balanced with gath- ering new data. Obtaining new on- policcy data is vital to continually improve policy performance (Ostrovski et al., 2021), but performing more update steps before gathering new data ensures that the existing data has been used effectively to improve the policy. The replay ratio (Fedus et al., 2020) or update-to-dae
( (UTD) ratio (Nikishin et al., 2022), which governs the number of gradient steps per environment step, is therefore vital hyperparameter.
Naively training with high UTD ratios can lead to collapse in off-policy deep RL (Nikishin et al., 2022). We conjecture that one of the major causes of the instability of high UTD off off-policy learning are wrong Q values on unobserved actions. This is a well-known problem for off off-policy TD learn- ing (Baird, 1995; Tsitsiklis & Van Roy, 1996; Sutton et al., 2016; Ghosh & Bellemare, 2020). To differentiate the problem from overfitting to the training distribution, we use the term misgeneral- ization highlight the importance of the distribution shift in causing the issue. Our experiments in Subsection 3.2 show that generalization to on-policy actions is more difficult than generalization to a validation dataset that follows the training distribution, and that higher UTDs exacerbate the issue.
3.1 ACTION DISTRIBUTION SHIFT CAN CAUSE OFF-POLICY Q VALUE DIVERGENCE
To highlight the role that on-policy actions play in stabilizing Q value learning, we show an analysis the stability of Q learning with linear features. The core ideas follow Sutton et al. (2016) and are also explored by Tsitsiklis Van Roy (1996); Sutton (1988). We assume that the Q function is parameterized with fixed features and weights Q(I a) Φ(z, a) 0. Let X and A be the sizes of the state and action space respectively. Let P E R A be the matrix of transition probabilities from state- state-action pairs to states. A policy
then be expressed as mapping II R X from states to the likelihood of taking each action. R X-A is the vector rewards. D E AxX-X is matrix where the main diagonal contains the discounted state-action occupancies of pa starting
3

---

Published as a conference paper at ICLR 2025
dog.walk
dog walk
Q value
dog. walk
Roward
dog.walk
Q Reward
330
1000)
100
UTD
On-soliy
UTD 8
2.5
Vil
235
250
UTD
valure loss
6.0
2.5
200
25
010
0.0
0..
1.0
2,0
0.0
0.5
1.0
20
0.0
0..5
1.0
Environment steps
Environment
XMF
Environment steps
XIIf
Environment steps
X30
Figure 2: Left: the train, validation, and on-policy validation error of the Q function at UTD 1. Right: the values and return curves of TD3 agents across different UTD 1, 8, and 16.
from p. If we assume access a mixed replay buffer D UKD ,..,, D=n) gathered with different policies, the Q learning loss for a target policy II can be written as
L(
a (TT -R 7PII 01sz)
(4)
The stability of learning with this loss can be analyzed using the gradient flow
=
24 2Φ Σ D -PT) T 0 2Φ Σ R
(5)
This gradient flow is guaranteed to be stable around fixed point O* if the key matrix E D (PI) is positive definite (Suttonn, 1988). Details and a proof the following state- ment are provided in Appendix C. We can decompose the key matrix and see that the positive definiteness depends on the difference in policy between the replay buffer and the target policy
D (T- PII)
n D (T- PIT,) Σ no guarantees
D(TP II)
positive definite
In general, we can provide guarantees for the second term outside of the on-policy case (II; II) where it becomes 0. The stability depends on the difference between the target policy and the data- collection policies.
the target policy takes actions which are not well covered under the training policies, the remainder can be non positive definite. This also matches the intuition that learning fails if we simply do not have sufficient evidence for the Q function of unobserved actions.
When using features, the eigenvalue conditions on the key matrix are only sufficient, not necessary, as the features can allow for sufficient generalization between observed and unobserved state-action pairs. In deep RL, the features e are updated alongside with the weights, making it hard provide definitive mathematical statements on stability. With good function approximation, we could hope that the learned value function generalizes correctly to unseen actions. In the next section we inves- tigate this for a non-trivial task from the DMC suite and highlight that, while the value function does not diverge irrecoverably, good generalization
guaranteed either.
3.2 EMPIRICAL Q VALUE ESTIMATION WITH OFF-POLICY DATA
In environments with large state-action space, ensuring coverage is difficult. To investigate whether learning is stable nonetheless, we train a model-free TD3 agent on the dog walk environment (Tun- yasuvunakool al., 2020a). The architecture is presented in Subsection 4.1, and is regularized to prevent catastrophic divergence (Hussing et al., 2024; Nauman et al., 2024a) and uses clipped dou- ble learning (Fujimoto et al., 2018). This means uses the most common techniques which designed to prevent misgeneralization and overestimation.
While training TD3 agent (Fujimoto al., 2018), a 5% probability. At regular intervals we compute the critic loss on this validation set. In addition, reset our simulator each validation state and sample action from the target policy. We then
transitions in validation buffer with
4

---

Published as a conference paper at ICLR 2025
simulate the ground truth on-policy transition and compute the loss over these. This allows to test how well our value function generalizes to target policy state-action pairs (as depicted in Figure 1).
The results are presented in Figure 2 and show a gap both between the train and validation sets, as well as the validation and the on-policy sets. While we use the on on-policy state-actions to update the Q value, these estimates are not actually consistent with the environment. Furthermore, the Q value overestimation grows with increasing UTDs. This phenomenon was previously discussed in the context of over-training on limited data (Hussing et al., 2024).
The experiments show that the problem outlined in Subsection 3.1 is not merely a mathematical curiosity, but that Q value generalization out-of-replay-distibbuss and becomes more difficult with increasing update ratios. Even though full divergence is not ob- served as new data is continually added to the replay buffer, it takes long time for the effects of severe early overestimation to dissipate.
actions is difficult in practice,
3.3 PREVIOUS ATTEMPTS TO COMBAT MISGENERALIZATION AND OVERESTIMATION
Prior strategies that deal with misgeneralization can be grouped into three major directions: archi- tectural regularization to prevent divergence of the value function, pessimism or ensemble learning to combat overestimation, and networks resets to restart learning. While all of these interventions help to some degree, they each either do not solve the problem in full or cause additional issues. We outline highly related work here and provide an additional related work section in Appendix B.
Architectural regularization Architecture changes (Hussing et al., 2024; Nauman al., 2024a;b; Lyle et al., 2024) and auxiliary feature learning losses (Schwarzer et al., 2021; Zhao et al., 2023; Ni et al., 2024; Voelcker et al., 2024) are largely reliable interventions, and have shown pro- vide improvements without much drawbacks in prior work. However, as Hussing et al. (2024) and our experiment presented in Subsection 3.2 highlight, by themselves they can mitigate catastrophic overestimation and divergence, but do not guarantee proper generalization.
Pessimism and ensembles To combat overestimation directly, the most prominent approach in continuous action spaces is Clipped Double Q Learning (Fujimoto et al., 2018). Here, Q value estimate is obtained from two independent estimates Q1 and Q2. If the error of the two critic estimators is assumed to be independent noise on the true critic estimate then using the minimum over both estimates is guaranteed to underestimate the true critic value in expectation. However, in complex settings this assumption on the the error of the critic estimates may not hold.
Ensembles (Lan et al., 2020; Chen al., 2020; Hiraoka et al., 2022: Farebrother et al., 2023) or online tuning of the rate pessimism (Moskovitz et al., 202 1) have been proposed to obtain tighter lower bounds the Q value. However, these strategies can be expensive as redundant models or hyperparameter tuning are needed. As a simpler strategy, recent works have also employed clipping to obtain an upper bound of the Q function prevent divergence (Fujimoto al., 2024).
Resetting Finally, network resets been shown to mitigate training problems (Nikishin et al., 2022; D'Oro et al., 2023; Schwarzer al., 2023; Nauman al., 2024b) in high UTD regimes. However, in cases where the agent fails to explore any useful parts of the state space within the reset interval, restarting the learning process will not improve performance (Hussing et al., 2024). This makes tuning the resetting interval both important and potentially difficult and no tuning recipes have been presented. Resetting also potentially hazardous strategy in wallworldd applications, where re- executing random policy might be costly or infeasible due to safety constraints. Finally, it heavily relies on the assumption that all past interaction data can be kept the replay buffer.
Data generation Lu et al. (2024) attempts to combat failures of high UTD learning by supple- menting replay buffer with data generated from a trained diffusion model. This idea is inspired by the hypothesis that failure to learn in high-UTD settings is caused by lack of data (Nikishin al., 2022). The method, SynthER, improves learning accuracy on simple tasks in the DMC benchmark. However, demonstrate that simply adding more data is insufficient to combat misgeneralization by comparing SynthER to MAD-TD in Appendix B and Subsection E.4.
All of these strategies are somewhat able to alleviate the problem of out-of--istribuu mation, yet none of them directly address the issue at the root. In the next chapter, we present an alternate approach that aims to directly regularize the action value estimates under the target policy.
value esti-
5

---

Published as a conference paper at ICLR 2025
4 MITIGATION VIA MODEL-GENERATED SYNTHETIC DATA
As value functions misgeneralize due to lack of sufficient on-policy data, we propose to obtain synthetic data from learned model instead. However, model-based RL can also cause problems such as compounding world model errors and optimistic exploitation of errors in the learned model. By using both real and model-ggnneated data, we can trade-off these issues: on-policy data improves the value function and limits the impact of off- distribution shifts, while using only limited number of godee--generate samples prevents model errors from deteriorating the value estimates.
Our approach builds on the TD3 algorithm (Fujimoto et al., 2018) and uses an update ratio of 8 by default. Our critic is updated with both model-based and real data following the DYNA framework (Sutton, 1990). More precisely, we replace small fraction of samples {z, a, T, in each batch with samples from a learned model f starting from the same state {x. =(x), F. 2'} with f, f ~ P(-12, (x)). In our experiments, a is set to merely 5%. We found that this small amount provides competitive performance across a wide range of values (compare Subsection E.3). We term this approach Model-
Data for Temporal Difference learning (MAD-TD)).
Model vs Q function generalization We expect that a learned models will yield better general- ization than the Q function for two reasons. First, the policy is updated each step find an action that maximizes the value function. This means we are effectively conducting an adversarial search for overestimated values. The model's reward and state estimation error on the other hand are inde- pendent of this process. We test the adversarial robustness of our in Subsection 5.3. Second, our experiment shows that value functions primarily diverge at the be- ginning of training. In these cases, coverage is low and on-policy state-action pairs are often not available. Obtaining a slightly wrong, yet converging value estimate can then be more useful than a diverging one. Even as more data is gathered, new policies might not revisit old states with a high likelihood. Therefore even as training continues we expect the model data to provide some benefit.
value functions
4.1 DESIGN CHOICES AND TRAINING SETUP
Our model is based on the successful TD-MPC2 model (Hansen et al., 2024) combined with the deterministic actor-critic algorithm TD3 (Fujimoto al., 2018). We aim to reduce the complexity of TD-MPC2 to the minimal necessary components to achieve strong learning in the DM Control suite, and thus forgo added exploration noise, SAC, ensembled critics, and longer model rollout for training or policy search. We outline several design choices here and refer to Section D for more detail. We additionally ablate our version of the model against TD-MPC2 in Subsection E.S.
Like TD-MPC2, we parameterize the state with a learned encoder & : X - 2 with a SimNorm nonlinearity (Lavoie al., 2023). This transformation groups latent vector into groups of k entries and applies softmax transformation over each group. This bounds the norm of the features, which has been shown aid with stable training (Hussing et al., 2024; Nauman et al., 2024a).
Encoder:
We use the HL-Gauss transformation to represent the Q function (Farebrother al., 2024). The critic loss is the cross-entropy between the estimated Q function's categorical representation and the bootstrapped TD estimate. To stabilize learning, we initialize the critic network towards predicting 0 for all states.
Critic representation and loss:
Model loss: The world model predicts the next state latent representation and the observed reward from a given encoded state ((x) and action a. The loss has three terms: the cross-entropy loss over the SimNorm representation of the encoded next state, the MSE between the reward predictions, and the cross-entropy between the next state critic estimate and the predicted state's is critic estimate. This final term replaces the MuZero loss in TD-MPC2 with a simplified variant based on the IterVAML loss (Farahmand, 2018). We provide the exact mathematical equations for the loss in Appendix D.
Training: We train the architecture by interleaving one environment step with one round of updates with a varying number of gradient steps governed by the UTD parameter. For each update step, a new mini- is sampled independently from replay buffer of previously collected experience. We found that varying the number of update steps only for the critic and actor while keeping the update ratio for the model and encoder updates at leads to significantly more stable learning.
Run-time policy improvement with MPC: Following the approach outlined by Hansen et al. (2022), the learned model can also be used at planning time obtain better policy. Using the model

---

Published as a conference paper at ICLR 2025
dog
1030
dog walk
dog stand
dog trot
Epizode reward
77M
750
730
754
200
300
560
2401
250
250
230
250
0.0
1.0
1.5
0.0
0.5
1.0
1.5
2.0
0.5
1.3
1.5
0.0
0..5
1.0
Environment steps
X30*
Environment steps
XMP
Environment steps
XII
Environment steps X31
UTD 1)
MAD-TD: UTD Mree--rre (UTD 8)
MAD-TD (UTD 16)
Modolfres (UTD 3)
Modal free (UTD LB)
Figure 3: Return curves for the dog tasks with differing UTD values. The return increases or remains stable when training with MAD-TD. Without model data, the performance decreases under high UTD. MPC is turned off in these runs to cleanly evaluate the impact of model data critic learning.
for MPC at planning time exploits the same benefit of models as the critic learning improvement: obtain a model-corrected estimate of the value function and choose our policy accordingly. As we only train our model for one step, we also conduct the MPC rollout for one step into the future.
5 EXPERIMENTAL EVALUATION
We conduct all of our experiments on the DeepMind Control suite (Tunyasuvunakool et al., 2020b). Following Nauman et al. (2024b)'s recommendations we focus our main comparisons and ablations on the two hardest settings, the humanoid and dog environments (which we will refer to as the hard suite). Subsection E.8 we furthermore show results for the metaworld benchmark (Yu al., 2019). Implementation details can be found in Appendix D. Unless stated otherwise evaluate MAD-TD with a UTD of and use the same hyperparameters across all tasks.
Note that even though we refer to training MAD-TD without using model data for the critic as "model-free", the algorithm still benefits from the model through feature learning which has proven to be a strong regularization technique in high UTD settings (Schwarzer et al., 2023). All main result curves are aggregated across 10 seeds per task. We plot mean and bootstrapped confidence intervals the library provided by
for the mean the 95% certainty interval. For aggregated plots, Agarwal et al. (2021). Additional comparisons on more environments are presented in Appendix E.
5.1 IMPACT OF USING MODEL-GENERATIR DATA
We first repeat the experiment presented in Subsec- tion 3.2 and show the results in Figure 4. Using model-based data closes the gap between on-policy and validation loss. We also observe that the ini- tial Q overestimation disappears, which is consistent across all hard environments (see Subsection E. This provides evidence that we are indeed able to overcome the unseen action challenge.
dog walk model-free
dog walk modellbased
160
20.0
15
7.
Q value loas
0.0
0.5
1.0
2.0
0.5
1.2
3.5
Enviromment staps
x10
Environment stapp
Performance with and without model data at varying UTD ratios: In Figure 3 we present the impact of using model-based data across different UTD ratios. Humanoid results are found in Subsection E.2. As directly evident, across the dog tasks, we observe stagnating or deteriorating performance when increasing the update ratio, consis- tent with reports in prior work. However, when using a small fixed amount of model generated data, this trend is reversed across all tested environments, with performance improving at least remain- ing consistent. We find that with model-based data, training is stable across a range of UTDs, even beyond those tested in recent high UTD work (Nauman al., 2024b). We also note that observe only limited benefits from increasing the UTD ratio when properly mitigating misgeneralization, except for the highly challenging dog task.
Figure 4: Mean loss values with and without generated data (see Figure 2) for UTD 1.
7

---

Published as a conference paper at ICLR 2025
Comparison with baselines: As our method combines model-free and model- based updates, compare our method against both TD-MPC2 (Hansen et al., 2024), a strong model-based baseline, and BroNet (Nauman et al., 2024b), recent algorithm proposed for high UTD learning. Since Nauman et al. (2024b) and Hansen et al. (2024) trained with differing numbers of action repeats, and we found that the performance does not cleanly translate between these regimes, present our method both with an action repeat value of and 2. Some hyperparameters are adapted to the AR=1 setting (compare Table 2). The results are presented aggregate in Figure with per environment curves show in Subsection E.6 for hard tasks and Subsection E.7 for wider range of DMC tasks.. We find that our method performs on par or above previous methods, and strikingly it is able to achieve higher returns faster than both TD-MPC2 and BRO.
5.2 PERFORMANCE AND STABILITY IMPACT OF RESETTING
Resetting comparison: To investigate if our technique enables more stable training. we set up an experiment test the effects of resetting on our method. Figure 6 presents aggregate results comparing our approach and BRO, both with and without resetting. Across all tasks we find that resetting barely improves MAD-TDs performance with the tested hy- perparameters. Benefits can only be observed on some seeds and can most likely be attributed to restarting the exploration process (Hussing et al., 2024). However, the BRO algorithm is not able to achieve reliable performance without resets. These results highlight that mitigating the problems related to incorrect generalization lize training, and that these problems likely a major cause of the fail- ure of high UTD learning in the DMC tasks. Conjectured problems like the primacy bias effect (Nikishin al., 2022) need to be carefully investi- gated as we do not find evidence that a primacy bias impacts MAD-TD's performance in the DMC environments. Our work of course does not preclude the existence of phenomena such as loss of stability in different
Per Timestop Average Regre:
3m5
2011
125
BRO
1100
0.25
0.50
0.76
1.00
Environment stops x:10f
the value function stabi-
Figure 7: Mean average regret (L) on the hard suite. Lower regret cor- responds to faster, more stable training. MAD- TD outperforms BRO. environments, architectures, training setups. More discussion on this can found in Appendix B.
Continued training: To highlight the pitfalls of resets, we employ a common RL theory metric the per timestep average regret
1 Reg (T) - Σ (R" -R)
where R, denotes the approximate cumulative return in episode t and R the optimal return. We use the maximum return any of the algorithms achieved ". lower bound the optimal return R Regret quantifies how much better the algorithm could have performed throughout training. In other words, in situations where continued learning is crucial, such as many safety critical applications, regret might be a better measure of performance. It captures not only how good the final policy is, but also how well the algorithm adapts over time, and minimizes mistakes. We present a comparison
Median
IQM
Mean
MAD-TD (AR=2, 2 mil)
BRO (AR=2, 2 mil)
TD-MPC2 (AR 2 mil)
MAD-TD (AR=2, 1 mil)
BRO (AR=2, 1 mil)
TD-MPC2 (AR-2 2, mil)
MAD-TD (AR=1, 1 mil)
BRO (AR-L. mil)
600
750
900
450
600
750
800
500
600
700
800
Scores
Figure 5: Performance comparison on the hard tasks for MAD-TD, BRO, and TD-MPC, with vary- ing number of steps and action repeat settings. MAD-TD is par with all basclines, has higher mean and IQM when trained for 2 million time steps and action repeat 2, and strongly outperforms TD-MPC2 and BRO at million time steps with action repeat 2.
8

---

Published as a conference paper at ICLR 2025
Median
IQM
Mean
MAD-TD (AR-2, reset)
MAD-TD (AR=2, reset)
BRO (AR-2, reset)
BRO (AR-2, reset)
MAD-TD (AR J reset)
BRO (AR=1, reset)
BRO (AR=1, no reset)
250
500
750
400
800
800
750
Scores
Figure 6: Resetting evaluation of MAD-TD and BRO. Lighter color denotes performance with reset, and darker without. While MAD-TD's performance only increases slightly when adding resetting, BRO is unable to achieve strong performance in and setting without resetting.
dog.run
1000
dog walk
dog.stand
1000
deg. trat
Epicede reword
700
750
7:30
700
500
500
990
500
200
100
0.0
1.0
3.5
0.3
1.0
2.0
5.0
1,3
1.5
0.0
0.5
1.00
Environment stops
x..0
Environment steps xBI
Environment stops X
Environment stops x310
MAD- ID (mods bosed)
Modsifree
MAD: ID (mrdde bosed, random actions)
Figure 8: Return curves for the dog tasks when using on-policy, random and no model- data. When generating model-based data with random actions, performance of MAD-TD drops close to the model-free baseline, highlighting the importance of on-policy actions.
of MAD-TD and the resetting- based BRO in Figure 7 using action repeat of 1. The results show, even though both algorithms are close in their final return, their training behavior differs vastly. MAD-TD has lower regret showcasing its strength in continued deployment.
5.3 FURTHER EXPERIMENTS AND ABLATIONS
To further test our approach, we present two additional experiments on the hard suite: changing the action selection for the model data generation, and reducing the model performance. In addition, we investigate the impact of using model based data on the smoothness of the learned value function.
Off-policy action selection in the model: To verify that the improvement in performance is due to the off off-policy correction provided by the model, we repeat the hard suite experiments with a UTD of 8 and 5% model data, but chose actions randomly from uniform distribution across the action space. The results are presented Figure They highlight that random state-action pairs do not provide the necessary correction and the performance deteriorates to that of the model-free baseline.
Smaller model networks: To study the effect of the modeling capacity on our method, we ablate the size of the latent model by reducing the network size across the hard suite. The results presented in Figure 9. We see that reducing the network size has an immediate and monotonic impact the performance of our approach, suggesting that the model learning accuracy and prediction capacity is indeed vital for our approach to function well. However, even with small models of 64 hidden units, we still see some benefits from training with the model predicted data.
Perturbation robustness of the model-corrected values: To motivate our method, we conjectured that one problem with training actor-critic methods is that the actor conducts quasi adversarial search for overestimated values on the learned critic.' To substantiate this claim, we used the iterated projected gradient method Madry al. (2018) estimate the smoothness of the learned value
QQaas because the actor is constrained find action close the replay huffer sample.

---

Published as a conference paper at ICLR 2025
Median
IQM
Mcan
MAD-TD [512,512]
MAD-TD [256,256]
MAD-TB TD [64,64]
MAD-TD TD [16.16]
Model-free [512,512]
700
800
900
6:90
720
800
880
600
660
720
750
Scores
Figure 9: Performance evaluation when reducing the model size of the latent model MAD-TD. The performance predictably drops with decreasing hidden layer size, however only strongly de- creasing the model size below 64 reduces the performance below that of the model-free ablation.
functions on the humanoid environments at a UTD of with and without model data. The results in Figure 10 show that not using any model data leads to value functions with higher oscillations, either across the whole training run (humanoid_run), or in the middle of training (stand and walk).
CONCLUSION
Our experiments allow us to conclude that wrong generalization of the value functions to unseen, on-policy actions is indeed major challenge that prevents stable off-policy RL, both in theory and in practice. Augmentdd Data for Temporal Difference learning (MAD-TD) is able to leverage the learning abilities of latent self-prediction models to provide small, yet crucial amounts of on-policy transitions which help stabilize learning across the hardest DeepMind Control suite tasks. With relatively simple model architecture and learning algorithm, this method proves to be on par with, or even outperform other strong approaches, and does not rely on mechanisms such as value function ensembles resetting which were previously conjectured to be necessary for stable learning in high UTD regimes. However, we highlight limitations of the approach in Appendix A.
Our work opens up exciting avenues for future work. The issue of poor generalization off-policy learning can likely be tackled with other approaches such as diffusion models (Lu et al., 2024) or better pre-trained foundation models, and our presented experiments provide an important baseline for such work. Furthermore, while we have purposefully kept our approach as simple as possible validate our hypothesis, many ideas from the model-base RL community such as uncertainty quantification (Chua et al., 2018; Talvitie et al., 2024), multi-step corrections (Buckman al., 2018; Hafner et al., 2020), or policy gradient estimation (Amos et al., 2021) can be combined with our approach. Our insight that surprisingly little data is necessary to achieve strong correction can likely be leveraged in these other approaches as well to trade-off model errors and value function errors more carefully. Finally, while chose the data roll out in models random, insights can likely be combined with ideas from the area of DYNA search control (Pan et al., 2019; 2020) to select datapoints on which the correction has the most impact.
humanoid run
humanoid walk
humanoid stand
10.0
154.0
300
7.5
112.5
75
MSE
0.0
100
Environment stops
Environment stops
Environment stops x.10
MAD-TD (mostebbased)
Mode4-fre
Figure 10: Magnitude of the difference between Q(x, (x)) and Q(x, ā), where ā an adversarial perturbation of (x). We larger perturbation for the runs without model correction data.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We thank the members of the TISL and AdAge labs the University of Toronto for enlightening discussions. We acknowledge the great help of Evgenii Opryshko, Taylor Killian, Amin Raksha, Maria Attarian, and Heiko Carrasco for providing detail feedback on our writing, and Evgenii for help with the experiments. For the adversarial experiments, we received helpful advice from Avery Ma, Jonas Guan, and Anvith Thudi.
We thank the anonymous reviewers for their helpful feedback and in-depth discussion.
EE and MH's research was partially supported by the Army Research Office under MURI award W91 1-NN20--100888 the DARPA Triage Challenge under award HR001 1 12420305, and by the Uni- versity of Pennsylvania ASSET center. Any opinions, findings, and conclusion or recommendations expressed in this material are those of the authors and do not necessarily reflect the view of DARPA, the Army, or the US government.
AMF acknowledges the funding from the Natural Sciences and Engineering Research Council of Canada (NSERC) through the Discovery Grant program (2021-03701) CV acknowledges the fund- ing from the Ontario Graduate Scholarship. Resources used in preparing this research were pro- vided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and compa- nies sponsoring the Vector Institute.
REFERENCES
Zaheer Abbas, Rosie Zhao, Joseph Modayil, Adam White, and Marlos C. Machado. Loss of plas- ticity in continual deep reinforcement learning. 2023.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Mare G Bellemare. Deep reinforcement learning at the edge of the statistical precipice. In Advances in Neural Infor- mation Processing Systems, 2021.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Mare G Bellemare. Reincarnating reinforcement learning: Reusing prior computation to accelerate progress. In Ad- vances Neural Information Processing Systems, 2022.
Brandon Amos, Samuel Stanton, Denis Yarats, and Andrew Gordon Wilson. On the model-based stochastic value gradient for continuous reinforcement learning. In Learning for Dynamics and Control. PMLR, 2021.
Oron Anschel, Nir Baram, and Nahum Shimkin. Averaged-DQN. Variance reduction and stabiliza- tion for deep reinforcement learning. International Conference on Machine Learning, 2017.
Leemon Baird. Residual algorithms: Reinforcement learning with function approximation. In Machine learning proceedings 1995, pp. 30-37. Elsevier, 1995.
Philip J. Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learn- ing with offline data. In International Conference on Machine Learning, 2023.
Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lee. Sample- efficient reinforcement learning with stochastic ensemble value expansion. Advances in neural information processing systems, 2018.
Johan Samir Obando Ceron, João Guilherme Madeira Araújo, Aaron Courville, and Pablo Samuel Castro. On the consistency of hyper-paramets parameter selection in value-based deep reinforcement learn- ing. Reinforcement Learning Journal, 2024.
Xinyue Chen, Che Wang, Zijian Zhou, and Keith W Ross. Randomized ensembled double q- learning: Learning fast without a model. In International Conference on Learning Represen- fations, 2020.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn ing in handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, 2018.
11

---

Published as a conference paper at ICLR 2025
Pierluca D'Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Luc Bacon, Mare G Bellemare, and Aaron Courville. Sample-efficien reinforeement learning by breaking the replay ratio barrier. In International Conference on Learning Representations, 2023.
Eric Eaton, Marcel Hussing, Michael Kearns, and Jessica Sorrell. Replicable reinforcement learning. In Conference Neural Information Processing Systems, 2023.
Mohamed Elsayed, Qingfeng Lan, Clare Lyle, and A Rupam Mahmood. Weight clipping for deep continual and reinforcement learning. In Reinforcement Learning Conference, 2024.
Amir-massoud Farahmand. Iterative value-aware model learning. In Advances in Neural Informa- fion Processing Systems, 2018.
Amir- masio- Farahmand, André Barreto, and Daniel Nikovski. Value-Aware Loss Function for Model-based Reinforcement Learning. In International Conference on Artificial Intelligence and Statistics, 2017.
Jesse Farebrother, Joshua Greaves, Rishabh Agarwal, Charline Le Lan, Ross Goroshin, Pablo Samuel Castro, and Mare G Bellemare. Proto-value networks: Scaling representation learn- ing with auxiliary tasks. In International Conference on Learning Representationss 2023.
Jesse Farebrother, Jordi Orbay, Quan Vuong, Adrien Ali Taiga, Yevgen Chebotar, Ted Xiao, Alex Ir- pan, Sergey Levine, Pablo Samuel Castro, Aleksandra Faust, Aviral Kumar, and Rishabh Agarwal. Stop regressing: Training value functions via classification for scalable deep RL. In International Conference on Machine Learning. 2024.
William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney. Revisiting fundamentals of experience replay. In International Con- ference on Machine Learning, 2020.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp. 2052-20022000 2062, 2019.
Scott Fujimoto, Wei-Di Chang, Edward Smith, Shixiang Shane Gu, Doina Precup. and David Meger. For sale: State-action representation learning for deep reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Dibya Ghosh and Mare G Bellemare. Representations for stable off-policy reinforcement learning. In International Conference on Machine Learning. 2020.
Raj Ghugare, Homanga Bharadhwaj, Benjamin Eysenbach, Sergey Levine, and Russ Salakhutdinov. Simplifying model-based RL: Learning representations, latent-space models, and policies with one objective. In International Conference on Learning Representations, 2023.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernado Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. In Advances in neural information processing systems, 2020.
Christopher Grimm, André Barreto, Satinder Singh, and David Silver. The value equivalence prin- ciple for model-basee reinforcement learning. In Advances in Neural Information Processing Systems, 2020.
Christopher Grimm, André Barreto, Gregory Farquhar, David Silver, and Satinder Singh. Proper value equivalence. In Advances in Neural Information Processing Systems, 2021.
Tuomas Haarojja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Confer- on Machine Learning, 2018.
12

---

Published as a conference paper at ICLR 2025
Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2020.
Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In International Conference on Learning Representations, 2021.
Nicklas Hansen, Hao Su, and Xiaolong Wang- TD-MPC2: Scalable, robust world models for tinuous control. In The Twelfth International Conference on Learning Representations, 2024.
Nicklas A Hansen, Hao Su, and Xiaolong Wang. Temporal difference learning for model predictive control. In International Conference on Machine Learning, 2022.
Hado van Hasselt. Double q-learning. In Advances in Neural Information Processing Systems, 2010.
Takuya Hiraoka, Takahisa Imagawa, Taisei Hashimoto, Takashi Onishi, and Yoshimasa Tsuruoka. Dropout q-functions for doubly efficient reinforcement learning. International Conference on Learning Representations, 2022.
Marcel Hussing, Claas A Voelcker, Igor Gilitschenski, Amir-massoud Farahmand, and Eric Eaton. Dissecting deep with high update ratios: Combatting value divergence. In Reinforcement Learn- ing Conference, 2024.
Maximilian Igl, Gregory Farquhar, Jelena Luketina, Wendelin Boehmer, and Shimon Whiteson. Transient non-stationarity and generalisation in deep reinforcement learning. In International Conference Learning Representations, 2021.
Garud N. Iyengar. Robust dynamic programming. Mathematics of Operations Research, 2005.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. In Advances in Neural Information Processing Systems, 2019.
Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline rI? In International Conference Machine Learning, 2021.
Tyler Kastner, Murat Erdogdu, and Amir-massoud Farahmand. Distributional model equivalence for risk-sensitive reinforcement learning. Advances in Neural Information Processing Systems, 2023.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), San Diega, CA, USA, 2015.
Qi Kuang, Zhoufan Zhu, Liwen Zhang, and Fan Zhou. Variance control for distributional rein- forcement learning. In Proceedings of the 40th International Conference on Machine Learning, 2023.
Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit underrparameetrrizaion inhibits data-efficient deep reinforcement learning. In International Conference on Learning Rep- resentations, 2021.
Qingfeng Lan, Yangchen Pan, Alona Fyshe, and Martha White. Maxmin I-learning: estimation bias of q-learning. In International Conference on Learning Representations, 2020.
Controlling the
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforce- ment learning: State-of-the-arl, pp. 45-73. Springer, 2012.
Samuel Lavoie, Christos Tsirigotis, Max Schwarzer, Ankit Vani, Michael Noukhovitch, Kenji Kawaguchi, and Aaron Courville. Simplicial embeddings in self-supervised learning and down- stream classification. In International Conference on Learning Representations, 2023.
Hojoon Lee, Hanseul Cho, HYUNSEUNG KIM, DAEHOON GWAK, Joonkee Kim, Jaegul Choo, Se- Young Yun. and Chulhee Yun. Plastic: Improving input and label plasticity for sample efficient reinforcement learning. In Advances in Neural Information Processing Systems, 2023.
13

---

Published as a conference paper at ICLR 2025
Hojoon Lee, Hyeonseo Cho, Hyunseung Kim, Donghu Kim, Dugki Min, Jaegul Choo, and Clare Lyle. Slow and steady wins the race: Maintaining plasticity with hare and tortoise networks. In International Conference Machine Learning, 2024.
Qiyang Li, Aviral Kumar, Ilya Kostrikov, and Sergey Levine. Efficient deep reinforcement learning requires regulating overfitting. In International Conference on Learning Representations, 2023.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. Interna- tional Conference on Learning Representations, 2016.
Angelo Lovatto. Thiago P. Bueno, Denis D. Mauá, and Leliane N. de Barros. Decision-awee aware model learning for actor-critic methods: When theory does not meet practice. In "I Can't I Believe It's Not Better!" at NeurIPS Workshops, 2020.
Cong Lu, Philip Ball, Yee Whye Teh, and Jack Parker- Holder. Synthetic experience replay. Ad- vances in Neural Information Processing Systems, 2024.
Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in rein- forcement learning. In International Conference on Learning Representations, 2021.
Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney. Understanding plasticity in neural networks. In International Conference on Machine Learning, 2023.
Clare Lyle, Zeyu Zheng, Khimya Khetarpal, Hado Hasselt, Razvan Pascanu, James Martens, and Will Dabney. Disentangling the causes of plasticity loss in neural networks, 2024.
Alcksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference Learning Representations, 2018.
Hamid Maci, Csaba Szepesvari. Shalabh Bhatnagar, Doina Precup. David Silver, and Richard Sutton. Convergent temporal-ditee -ifferreee learning with arbitrary smooth function approximation. Advances in neural information processing systems, 22, 2009.
Diganta Misra. Mish: self regularized non-monotonic activation function. British Machine Vision Conference, 2020.
Thomas M. Moerland, Joost Broekens, Aske Plaat, and Catholijn M. Jonker. Model-based rein- forcement learning: A survey. Foundations and Trends Machine Learning, 16(1), 2023.
Ted Moskovitz, Jack Parker-Holdee Aldo Pacchiano, Michael Arbel, and Michael Jordan. Tacti- cal optimism and pessimism for deep reinforcement learning. Advances in Neural Information Processing Systems, 2021.
Michal Nauman, Michal Bortkiewicz, Piotr Miloš, Tomasz Trzcinski. Mateusz Ostaszewski, and Marek Cygan. Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of rein- forcement learning. In Forty-first International Conference on Machine Learning, 2024a.
Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski. Piotr Mitoš, and Marek Cygan. Big- ger, regularized, optimistic: scaling for compute and sample-efficient continuous control. Ad- vances Neural Information Processing Systems, 2024b.
Tianwei Ni, Benjamin Eysenbach, Erfan Seyedsalehi, Michel Ma, Clement Gehring, Aditya Ma- hajan, and Pierre-Luc Bacon. Bridging state and history representations: Understanding self- predictive rl. To appear in International Conference on Learning Representations, 2024.
Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Lue Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International Conference on Machine Learning, 2022.
Evgenii Nikishin, Junhyuk Oh, Georg Ostrovski. Clare Lyle, Razvan Pascanu, Will Dabney, and André Barreto. Deep reinforcement learning with plasticity injection. Advances in Neural Infor- mation Processing Systems, 36, 2024.
14

---

Published as a conference paper at ICLR 2025
Arnab Nilim and Laurent El Ghaoui Robust control of markov decision processes with uncertain transition matrices. Operations Research, 2005.
Georg Ostrovski, Pablo Samuel Castro, and Will Dabney. The difficulty of passive learning in deep reinforcement learning. Advances Neural Information Processing Systems, 2021.
Yangchen Pan, Hengshuai Yao, Amir-massoud Farahmand, and Martha White. Hill climbing value estimates for search-control in dyna. In International Joint Conference on Artificial Intelli- gence, 2019.
Yangchen Pan, Jincheng Mei, and Amir-massoud Farahmand. Frequency-based search-control in dyna. In International Conference Learning Representations, 2020.
Andrew Patterson, Samuel Neumann, Raksha Kumaraswamy, Martha White, and Adam White. Cross-enviromment hyperparameter tuning for reinforcement learning. Reinforcement Learning Journal, 2024.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce- ment learning. In Proceedings of the 34th International Conference on Machine Learning, 2017.
Martin I.. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming John Wiley & Sons, Inc., USA, 1st edition, 1994. ISBN 0471619779.
Julian Schrittwieser, loannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839), 2020.
Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bach- man. Data-efficient reinforcement learning with self-predictive representations. In International Conference Learning Representations, 2021.
Max Schwarzer, Johan Samir Obando Ceron, Aaron Courville, Mare G Bellemare, Rishabh Agar- wal, and Pablo Samuel Castro. Bigger, better, faster: Human-level Atari with human-level effi- ciency. In International Conference on Machine Learning, 2023.
David Silver, Hado Hasselt, Matteo Hessel, Tom Schaul, Arthur Guez, Tim Harley, Gabriel Dulac-Arnold, David Reichert, Neil Rabinowitz, Andre Barreto, et al. The predictron: end-to-end learning and planning. In International Conference on Machine Learning, 2017.
Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evci. The dormant neuron phe- nomenon in deep reinforcement learning. In International Conference Machine Learning, 2023.
Richard Sutton. Learning to predict by the methods of temporal differences. Machine learning, 1988.
Richard S Sutton. Integrated architectures for learning, planning, and reacting based on approximat- ing dynamic programming. In Machine learning Proceedings. 1990.
Richard Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018
Richard Sutton, Rupam Mahmood, and Martha White. An emphatic approach to the problem of off-policy temporal-difference learning. Journal of Machine Learning Research, 17(73):1-29, 2016.
Erin Talvitie. Zilei Shao, Huiying Li, Jinghan Hu, Jacob Boerma, Rory Zhao, and Xintong Wang. Bounding- box inference for error-aware model-based reinforcement learning. Reinforcement Learning Journal, 2024.
Sebastian Thrun and Anton Schwartz. Issues in using function approximation for reinforcement learning. In Proceedings of the 1993 Connectionisl Models Summer School, 1993.
15

---

Published as a conference paper at ICLR 2025
Dhruva Tirumala, Thomas Lampe, Jose Enrique Chen, Tuomas Haarnoja, Sandy Huang, Guy Lever, Ben Moran, Tim Hertweck, Leonard Hasenclever, Martin Riedmiller, Nicolas Heess, and Markus Wulfmeier. Replay across experiments: A natural extension of off-policy RL. In International Conference on Learning Representations, 2024.
John Tsitsiklis and Benjamin Van Roy. Analysis of temporal-difffernnce learning with function approximation. Advances in Neural Information Processing Systems, 1996.
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm_control: Software and tasks for continuous control. Software Impacts, 6, 2020a.
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm_control: Software and tasks for continuous control. Software Impacts, 6:100022, 2020b. ISSN 2665-9638, doi: https://oii.og/
s0.10.//j.simpa.2000 2020.10002...
Giuseppe Vietri, Borja Balle, Akshay Krishnamurthy, and Steven Wu. Private reinforcement learn- ing with PAC and regret guarantees. In Proceedings of the 37th International Conference on Machine Learning, 2020.
Claas A Voelcker, Victor Liao, Animesh Garg, and Amir-massoud Farahmand. Value gradient weighted model-based reinforcement learning. International Conference Learning Represen- tations, 2022.
Claas A Voelcker, Tyler Kastner, Igor Gilitschenski, and Amir-massoud Farahmand. When does self-prediction help? understanding auxiliary tasks in reinforcement learning. In Reinforcement Learning Conference, 2024.
Ran Wei, Nathan Lambert, Anthony D McDonald, Alfredo Garcia, and Roberto Calandra. A unified view on solving objective mismatch in model-based reinforcement learning. Transactions on Machine Learning Research, 2024.
Wolfram Wiesemann, Daniel Kuhn, and Breç Rustem. Robust markov decision processes. Mathe- matics of Operations Research, 38(1):153-183, 2013.
Guowei Xu, Ruijie Zheng, Yongyuan Liang, Xiyao Wang, Zhecheng Yuan, Tianying Ji, Yu Luo, Xiaoyu Liu, Jiaxin Yuan, Pu Hua, Shuzhen Li, Yanjie Ze, Hal Daumé III, Furong Huang, and Huazhe Xu. Drm: Mastering visual reinforcement learning through dormant ratio minimization. In International Conference on Learning Representations, 2024.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning, 2019.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 2020.
Yi Zhao, Wenshuai Zhao, Rinu Boney, Juho Kannala, and Joni Pajarinen. Simplified temporal consistency reinforcement learning. In International Conference on Machine Learning, 2023.
16

---

Published as a conference paper at ICLR 2025
A LIMITATIONS
The core limitation of our methodology relies in the assumption that a sufficiently strong environ- ment model can indeed be learned online. While proof of feasibility exists for many interesting RL benchmarks in the forms of the Dreamer (Hafner et al., 2021) and TD-MPC2 (Hansen et al., 2024) lines of work among many others, for a completely novel environment practitioner will still have to test if current model learning schemes are sufficient to achieve strong control policies.
Furthermore, we can only generate data from the states visited under a past policy. There is still difference between the state distribution of the replay buffer, and the target policy stationary distri- bution. While this difference does not seem to lead catastrophic failures in the DMC benchmarks, the distribution shift might be more problematic in other environments.
Finally, we observe an interesting failure cases of our idea: in some simple environments we sur- prisingly observe worse performance with our network architectures compared to the BRO baseline. This issue likely due the fact that the TD-MPC2 architecture is tuned for learning in complex high-dimensional problems, which leaves it potentially over weer-parameterized on simple tasks.
While our work shows that reduced learning capacity due to plasticity does not seem be the major contributor to learning problems in benchmarks like DMC, that does not exclude the possibility that related issues appear nonetheless after accounting for the off off-policy value estimation problem. We did not test increasing the reset ratio even further other prior work has done, as we already observed no benefits from increasing the replay ratio from 8 to 16 in most of our experiments and performed on par beyond previous baselines. Issues in reinforcement learning are often entangled in a complex way, c.g. a failure in exploration can lead to stagnant data in the replay buffer which prevents critic from further improving its estimates, leading worse exploration and on.
B EXTENDED RELATED WORK
Beyond mitigating value function overestimation and unstable learning (see Subsection 3.3), other works have approached the difficulty of off off-policy learning and high update ratios from other per- spectives. Here, we survey further related papers which do not provide direct background for this work, but are nonethless relevant as either alternative approaches or possible enhancements.
Other ways of incorporating off-po data Having access to more diverse data has been shown to be beneficial for reinforcement learning, when this data carefully used mitigate the problems resulting from off-policy training. Ball et al. (2023) show that large offline replay buffer can be used to improve training by sampling online training batches both from online data and offline data, and labelling the offline transitions with a reward of O. Agarwal et al. (2022) and Tirumala et al. (2024) also highlight that previously collected replay buffers can be used to improve training performance on agents. In this work, focus the online setting where do not have access to a replay buffer of previously collected transitions. These ideas however can easily be combined by e.g. training model from available larger offline data buffer.
SynthER Another related approach to obtain additional data is the diffusion- method pro- posed by (Lu et al., 2024). In this work, the replay buffer data is augmented with additional samples obtained from a diffusion model that is trained on the replay buffer. The underlying hypothesis of SynthER is that the failure of high-UTD learning stems mostly from lack of diverse data in the replay buffer. They demonstrate on the easier DM Control tasks that simply adding data from a generative model can be beneficial learning. This is opposed that Uigh-UTD learning is difficult specifically due to the lack of off-policy action corrections. As SynthER does not provide results on the hard DMC tasks, we reran the original code to compare our claims The results and discussion can be found in Subsection E.4.
our hypothesis. which claims
In the online off-policy regime, Fujimoto et al. (2024) recently proposed TD7, which incorporates similar architectural choices to MAD-TD. They use a self-predictive encoder to learn good state representations, but concatenate them with the state and action representation provided by the envi- romment to limit loss of information. This design choice proved to be beneficial but would require learning a observation-space next-state prediction, which is difficult in practice, especially in high dimensional environments. To address the policy distribution shift, TD7 does not update the actor
17

---

Published as a conference paper at ICLR 2025
at every timestep but instead collects several full trajectories with a fixed policy and then conducts update steps afterwards. However, this interval still needs to be balanced as a hyperparameter. TD7 was not evaluated on DMC, which is why we do not present comparison.
Model-based reinforcement learning As surveying model-based reinforcement learning is a rather sizable tasks, we refer readers to the survey by (Moerland et al., 2023) for reference. Decision- aware latent models such as the one Hansen et al. (2024) and we use have been studied specifically in several different variants. Silver et al. (2017) proposes a latent model that is trained with TD learning. which provides the basis for the Schrittwieser et al. (2020) algorithm. The addition of a latent self-prediction loss was first proposed by Li et al. (2023) to stabilize learning problems with the TD learning loss. This interplay was further studied by Ni et al. (2024) and Voelcker et al. (2024) in recent works.
From a theoretical angle, decision-aware losses similar to those used in MuZero where first studied by Farahmand et al. (2017) and Farahmand (2018). Grimm al. (2020) and Grimm al. (2021) further study the loss landscape and minimizers of such losses, while Kastner et al. (2023) studied the extension of the loss distributional settings.
While previous works have called the stability of the VAML loss into question (Lovatto et al., 2020; Voelcker et al., 2022), we find that it is stable and performant when combined with the HL-Gauss representation Farebrother et al. (2024) and an auxiliary B BYL style loss Grill et al. (2020); Li et al. (2023). Compared to MuZero is also significantly easier implement.
A more thorough overview on the topic of decision-aware learning can be found by Wei et al. (2024).
Offline reinforcement learning In the context of batch reinforcement learning or offline RI. (Lange et al., 2012: Fujimoto et al., 2019), the action distribution shift is a known phenomenon. The main counter to the problem however does not rely closing the generalization gap, but on ex- plicit pessimistic regularization Jin et al. (2021). Such pessimistic regularization has been shown to be highly detrimental in online RL, as it removes the capability for the agent to explore its environ- ment efficiently (D'Oro et al., 2023; Hussing et al., 2024). In offline RL, authors have explored the capability of models to provide some improvements to generalization (Yu et al., 2020). However, in online RL. the community has mostly relied on the hope that additional optimistic exploration based on the value function will close the generalization gap without explicit interventions. We show that this is not the case.
Loss of plasticity A phenomenon that was originally reported in continual learning is that ten- dency for neural network based agents to lose their ability to learn over time. This phenomenon has also been investigated in the realms of RL (Igl et al., 2021), as RL can effectively be thought of as type of continual learning problem. Sometimes the phenomenon is referred to as plasticity loss (Lyle al., 2021; Abbas et al., 2023). As highlighted before, we do not find strong evidence for the primacy bias or loss of plasticity during our experiments on the DMC suite.
However, that does not imply that the phenomenon does not exist. In fact, we believe that resolving stability issues such as those presented in our paper will help us to better isolate other nuanced issues such as plasticity loss more clearly. Previous studies have identified and combated plasticity loss using feature rank maximization (Kumar et al., 2021), regularization (Lyle et al., 2023), additional neural network copies (Nikishin et al., 2024), minimizing dormant neurons (Sokar et al., 2023; Xu et al., 2024), various neural network architecture changes (Lee et al., 2023), slow and fast network updates (Lee et al., 2024) or weight clipping (Elsayed et al., 2024).
It is unclear how many improvements obtained by these changes can be explained by divergence effects (Hussing et al., 2024) or stability issues such as those established our work as there seems to be non-zero overlap in techniques that combat either. Nauman et al. (2024a) have argued that many RL training problems can be difficult to disentangle from the plasticity loss phenomenon. An interesting direction of future work is to test for plasticity loss with well regularized offlcy value function learning, for instance by combining our method with separate solutions established for plasticity loss such those from Lyle et al. (2024).
is also not unlikely that the training dynamics of the state-based dense-rewaad tasks on the DMC suite more benign than those found in Atari games. Many works plasticity loss have stud-
18

---

Published as a conference paper at ICLR 2025
ied sparse image-based control tasks with pure Q learning approaches, such DQN the Atari benchmark (Sokar et al., 2023; Lee et al., 2024). The problem may be more prevalent when replay buffers cannot be maintained in full and the RL setting becomes true continual learning problem.
Other stability perspectives Our work studies the stability of losses during training. We highlight that forgoing resetting decreases regret as the executed policies more stable in the sense that they are not reset at regular intervals. We also highlight that -generree data can somewhat improve the stability of policies against adversarial attacks. However, there are other notions of stability that should be considered relevant and that are orthogonal to our work. Here we will give non extensive overview into the different directions that exist as a starting point for the reader. For instance from theoretical perspective, stability can be formulated as differential privacy (Vietri et al., 2020) or algorithmic replicability to obtain identical policies (Eaton et al., 2023). From a theoretical as well as practical perspective, issues such robustness to adversarial attacks (Nilim & Ghaoui, 2005; lyengar, 2005; Wiesemann et al., 2013; Pinto et al., 2017). Finally, from an empirical perspective robustness to hyperparameters (Ceron et al., 2024; Patterson et al., 2024) and attempts variance reduction to get more reliable solutions (Anschel et al., 2017; Kuang et al., 2023) can be considered notions of stability.
C MATHEMATICAL DERIVATIONS
While the proof by Sutton (1988) which use a basis discusses the stationary distribution of the Markov chain P", we define our loss in terms of a discounted state-action occupancy. We therefore briefly prove an auxiliary result extend the analysis the case of discounted state occupancy probabilities. Note that when we talk about positive-deffnieness, we use a definition which applies potentially non-symmetric matrices, and merely requires that u Xu for all vectors u.
Proposition 1. Let P be stochastic matrix. Define the discounted state occupancy distribution of P.oo for some starting state distribution and some discount factor & E [0, 1) as
1 = =1
Σ P T=0
Let D be a diagonal matrix whose entries correspond the discounted state occupancy distribution. Then the matrix D(I -5--4 P) is positive definite.
Proof. First, note that
(1-7)p" + +7 P P n²
by the definition of μ and the properties of the infinite sum. Therefore,
1 7 (4- (1 7)p)
p T P
Sutton (1988) asserts that matrix A is positive definite iff A A the diagonal entries of a symmetric matrix are positive and its off-diagonal entries are negative, then it suffices to show that the and column sums of matrix are positive.
T is positive definite. Furthermore,
For
D(T 7P) (T- -P-TP
the off off-diagonal terms are clearly non positive as D is diagonal. On the main diagonal, we have 2(14 pp) which positive as p(ulpp:) < 1. It now suffices to show that the row and column sums of D(T P) are positive. For the row sum, we can make use of the fact that P is stochastic matrix, so
D - D(1->P)1- D(1 71) 2
For the column sum, we make use of the fact that 10 = μ. Then
M(I - -P) - - (1- r)) = -1-77pp 1.
As p is a probability vector the final inequality holds for all [0. 1).
19

---

Published as a conference paper at ICLR 2025
All conditions presented by Sutton (1988) hold, and therefore we have D(I-PP) is positive definite.
To derive the gradient flow stability conditions in Subsection 3.1, we first restate the loss function
L(O)
D [ - + 7P
7 T Olsk))
The stability of learning with this loss can be analyzed using the gradient flow (Sutton et al., 2016). To derive the gradient flow, compute the gradient of the loss function with regard to the parame- ters 0. As the loss has relatively simple quadratic form and the derivative is a linear transformation, it decomposes nicely
VoL(0) = 2Φ
D
0 R 7 PII TA) 8) TT
(8)
i=1
24
D (I-~PII) T R)
(9)
2Φ
D (1 - ( pd Φ 0 - 24 Σ D R.
(10)
Using the equation for the gradient flow θ = V with learning rate f. we obtain
72
n D
i = w D (1- 1 0 - n
(II)
This gradient flow is guaranteed to be stables (meaning it will not diverge around the stationary point 0*) if the key matrix Σ D (I -PIII is positive definite (Sutton, 1988).
We can decompose our key matrix into the on-policy key matrix and a remainder easily
DT (P1)
(12)
DT (I - 7 + 7PIII - -PII.)
(13)
D (I PIT,) + D P(II,
(14)
The first group of summands are all positive definite, following Proposition 1. As the sum of positive definite matrices is positive definite, the claim stands.
However, the second group has no such guarantees. This highlights the role that the target policy action selection plays in the stability of Q learning.
D IMPLEMENTATION
Our experiments are implemented the jax library to allow for easy parallelization of multiple ex- periments across seeds. All networks follow the standard architecture from Hansen et al. (2024) with two changes: instead of using an ensemble of critics, we opt for single double critic pair. We also do not use a stochastic policy, instead simply using a deterministic network with tanh activation as used in Lillicrap et al. (2016); Fujimoto et al. (2018). Full hyperparameters are presented in Table and the architecture can be found in Table 1. We use mish activation functions (Misra, 2020) and the Adam optimizer to train our models (Kingma & Ba, 2015). For reference, our code is available at https: / / tub. com/ /atpptaaptttt ve agent alb mwdd/mddddd..
20

---

Published as a conference paper at ICLR 2025
Encoder /
Dense Layer
Mish
in sissszze [1) out size=512
Dense Layer
Simnorm(8
out out.size=512
Dense Layer
Mish
in size |A,, out_size=512
Latent Model F
Dense Layer
Mish
out.size=512
Dense Layer
Simnorm(8)
out_size=511
Dense Layer
Mish
misizz=512-- + [A], out
head
Dense Layer
Mish
out.size=512
Dense Layer
out_size=1
Dense Layer
Mish
sizez=251 out size=512
Actor it
Dense Layer
Mish
out size=512
Dense Layer
tanh
out size=|AA
Table I: Network architecture for MAD-TD.
Parameter
Parameter
Initial steps (Random policy)
5000
HL-Gauss vmax
1
Batch size
512
HL-Gauss num bins
151
RL learning rate
0.0003
Model data proportion
0.95
Model learning rate
0.0003
Reset interval (where applicable)
200000
Encoder learning rate
0.0001
Model encoder update ratio
Soft update
0.995
Actor & critic update ratio
varying
Discount factor 7
0.99
MPC number of samples
512
Model forward prediction steps
(4 for AR=1)
MPC iterations
Gradient clipping
10.0
MPC top
HL-Gauss vmin
150 AR
MPC temperature
0.5
Table 2: Hyperparameters. We adapted three parameters to the action repeat = 1 setting, as the magnitude of the reward changes.
Loss functions: As use the HL-Gauss representation (Farebrother et al., 2024) for the critic, the loss is the cross-entropy between the estimated Q function's is categorical representation Qrep and the bootstrapped TD estimate,
LQ = Σ TD(Qrop), log Qrep
where the indices denote the positions of the categorical vector representation used by HL-Gauss. This is the same loss that is used for the two-hot encoding in Hansen et al. (2024), the only difference the target encoding function. For more details, see Farebrother et al. (2024).
We latent encoder n k-dimensional simplices (Lavoie et al., 2023). Writing P for the learned world model and l p((2,, a) for reward and next latent-state samples, the loss for model and encoder is
X A Z that maps into the SimNorm space, the space of
Lmodel a, r,) = f.ew (x, a, + Storward(r, a, a²x + Lo(x, a, r,,,
(15)
Lrew (x, a, (r-A)- R)3
(16)
nikk
Liorward = Σ (((x)) log
(17)
where the index is again element-wise across the simplex representation used for the latent state. Note that propagate the critic learning gradients into the encoder only for the real data and not the model generated one to prevent instability.
Baseline results We took available results from Nauman et al. (2024b) and Hansen al. (2024) for all plots where possible, and used the official implementation of BRO to rerun the experiments without resetting and with differing action repeats. Other hyperparameters were left as-is.
21

---

Published as a conference paper at ICLR 2025
E FURTHER RESULTS
E.1 Q VALUE OVERESTIMATION
1000
dog run
Return
200
dog. run
Q value
1000
dog walk Return
500
dog walk value
Eptodd reward
120
730
200
500
100
250
50
200
10
0.00
0.5
1.03
0.0
0.5
1.0
1.5
9.0
0.0
08.5
1.0
1.5
0.0
0.5
1.00
X10*
XII
X32
dog.stamm
dog. valun
dog. Retur
dog..ror
1000
1000
Eptodd revard
720
730
900
250
0.5
1.0
1.5
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
0.0
0..
1.00
X20
X01
X30
humanoid
humanoid
humanoid
humanoid volue
0.0
0.5
1.3
2.5
0.5
1.0
1.5
2.0
3.0
0.5
1.0
1.5
0.0
0.5
110
2.0
xB0
x10
humancid stand Return
humancid stand value
Epioode reward
2.0 xM/
Environment steps X200
Environment steps
MAD-TD (UTD MAD-TD (UTD (UTD 8)
MAD-TO (UTD (UTD 38)
(UTD 1)
Figure 11: Return curves and Q values with differing UTD values.
We plot the return curves and corresponding Q estimates for different UTD values and with and without model-generatee generated data on the hard suite. The results are presented in Figure 11. As we see, across all tasks the model free variant strongly overestimates the values, especially in the beginning.
22

---

Published as a conference paper at ICLR 2025
E.2
HUMANOID RESULTS
humanoid yun
humanoid .walk
humanoid stand
Episode
0.3
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
20
0.0
0.5
1.0
1.5
2.0
Environment
steps
Environment steps
Environment steps
MAD-TD (model-based)
MAD-TD (model-free)
(model-baze random actions)
Figure 12: Return curves for the humanoid tasks when using on-policy (blue), random (green) and no model-generated data (orange). The observed performance impacts are comparable the dog case.
For several experiments, only showed the dog results from the main suite avoid cluttering the main body of the paper. The corresponding humanoid results are presented in Figure 11 and Figure 12, corresponding to Figure and Figure respectively. As the plots highlight, the main insights transfer across the hard tasks.
23

---

Published as a conference paper at ICLR 2025
E.3 DIFFERENT QUANTITIES OF MODEL DATA
We evaluate using more model data to update our value functions and provide the results in Fig- 13. Aggregated scores presented in Figure 14. We observe that the majority of gain is obtained when using limited amount of model data, and larger amounts only provided limited gains in some humanoid runs. When using high amounts of model generated data, we observe deteriorat- ing performance, which implies that the agent learns to exploit the model instead of solving the real task. This observation is consistent with similar observation about model exploration prior work Zhao et al. (2023).
dog,
dog walk
dog. stand
dog. trot
1060
1000
1000
Eptocde reserd
200
300
500
200)
250
250
130
200
humanoid_run
humancid, l
humannidl stand
1000
Epocce reierd
200
300
500
200
250
230
1.5
stops
Environment steps
Environment stops
MAD-TD (0%) MAD-TD
MAG-TD (5%)
MAD-TD (15%)
MAG-TD
MAD-TD (100%)
Figure 13: Return curves on the hard suite. We see that using substantially more data than 5% does not improve performance in statistically significant way.
Mean
0%
5%
25%
50%
75%
95%
100%
250
500
750
Scores
Figure 14: Aggregate statisties for differing values of a (amount of model data used) at UTD 8.
24

---

Published as a conference paper at ICLR 2025
E.4 SYNTHER COMPARISON
We present a comparison of our method and SynthER on the hard DMC tasks. Results can be found in Figure 15.
As is evident from the lack of strong performance of SynthER, merely increasing the amount of generated data is insufficient to combat the failure of learning at high UTD. We find that the values of the SynthER agents quickly diverge on all tasks in which is unable to learn. This strengthens our hypothesis that for hard tasks, off-policy action correction is vital achieve strong results.
1000
dog Jun
1000
dog.
1000
cog stand
1000
dog
Epioode revord
720
750
700
700
000
040
3.50
0.25
11.00
01.00
1.00
0.00
0.35
0.30
0.75
1,00
0.10
01:35
0.50
0.25
100
0.00
0.20
0.70
11.75
XII
X.I
X
humanoid
Prrnanoid
humanoid
1000
1000
1000
Epioade revard
700
750
7.0
500
040
200
3.00
0.25
0,50
0.70
100
8,00
0.35
0.30
0.75
1,00
0.10
0125
0,20
0.25
1,00
Environment steps
xIIf
Environment stops x.Iff
Environment steps x100
Systhe
Figure 15: Performance curves for MAD-TD and SynthER the hard DMC tasks. SynthER fails to achieve nontrivial results on most tasks, only outperforming a random policy on the humanoid walk and stand tasks.
25

---

Published as a conference paper at ICLR 2025
E.5 TD-MPC2 ABLATION
As described in the main paper, we simplify the base model of TD-MPC2 to improve the compu- tational efficiency of the algorithm. This is necessary conduct high UTD experiments. Here, present a direct comparison the original TD-MPC2 model, and our adapted version (Figure 16). We compare MAD-TD without any model generated data, at UTD which corresponds the stan- dard setting of TD-MPC2. As pointed out in the main paper. all of our changes to the base model boil down to setting different hyperparameters, such the rollout length, achieve faster learning-
We find that this does not significantly change the overall results achieved by the base model, and we are therefore confident to attribute performance gains our presented method.
dog, run
dog walk
dog stand
dog. trot
1000
1000
1000
1000
Eptsods revard
120
750
730
910
0.1
1.0
0.55
9.0
3.0
1.5
1.10
X00
XII
X30
humanoid_rum
humanoidd
humanoid
hopper.hop
Egizoe reward
750
500
250
150
130
250
acrobot swingup
cheetah
walker run
quadruped run
1000
1000
1000
Eptocde revard
750
7.30
750
200
500
500
2001
2001
750
100
550
0.0
1.0
2.0
0.0
0.5
1.0
2.0
3.0
0.5
1.3
1.5
0.0
0.5
1.0
2.0
x1F
x11
finger, turn. hard
pendulum swingup
quadruped walk
reacher hard
1000
1000
1000
1100
Eptooe revard
720
730
500
900
1.14
3.5
0.5
1.0
2.0
006
0.5
1,0
1.5
0.5
1.0
X:20
XII
K10
hopper.stand
Egicode reward
750
500
steps
Environment
MD (MAD-
10-MPC2
Figure 16: Performance variation of the base MAD-TD model compared to TD-MPC2. Our changes only very few times lead to lower performance which acceptable given the large reduction in computational cost.
26

---

Published as a conference paper at ICLR 2025
E.6 MAD-TD. BRO, TD-MPC2 PER ENV ON THE HARD SUITE
We present the return curves for MAD-TD and the baselines per environment on the hard suite. Figure 17 shows the results with action repeat and Figure 18 with action repeat Perhaps sur- prisingly, the results of the algorithms are not fully consistent across this regime. Partially, this can be explained by the fact that our method and TD-MPC2 were first developed in the regime of action repeat 2. while BRO was only evaluated in the action repeat setting. This suggests that the performance of each method depends in a non-trivial fashion on hyperparameter tuning. Yet, across both action repeat setting MAD-TD outperforms BRO without resetting consistently and only under-performs any previous algorithm on the dog trot task in the action repeat setting.
Lotodd
500
33
120
000
x30
x37
110
2000
hamancd.oo
180
namanc veal
2000
humancid stand
130
703
10
Enderrentt
Prsinoment
Enatcrment
WA0-TD
TD-MPC2
BR0
BRO (ss nait
Figure 17: Return curves with action repeat set
deg.noh
Epoode
203
5.5
1.0
1.20
100
110
X18
x17
X00
x17
2000
180
2000
humondi_nnadd
130
703
110
Lpiode
20
1.30
MAD-TD 2B0
BRO (ro Hust)
Figure 18: Return curves with action repeat set
We conjecture that the remaining gap in performance seems to be most likely attributable to explo- ration and optimism. While we focus on learning accurate value functions, Bro contains several components which are specifically designed to improve exploration. Investigating the tension be- tween exploration and accurate value function fitting is an important direction for future work.
Bro and MD-MPC2 are explicitly evaluated without their exploration bonuses in separate evaluation rollouts. We however do not conduct such as separate evaluation as we do not add any additional ex- ploration noise to our training. When plotting training performance, the gap between MAD-TD and Bro further closes, suggesting an important trade-off between test time and training performance.
27

---

Published as a conference paper at ICLR 2025
E.7 RESULTS ACROSS FURTHER DMC ENVIRONMENTS
We conducted experiments all DMC environments which shown benefit from the interventions in prior work (D' et al., 2023; Nauman et al., 2024b).
dog. run
dog walk
dog stand
1000
1020
Ephode reward
750
230
250
200
0.0
0.5
0.0
0.0
humancid
humanoid wwalk
Episode reward
000
700
300
300
200
250
250
0.00
0.3
1.01
0.5
1.0
1.5
9.0
0.5
200
X
x10
humanoid stand
hopper hop
acrobot swingup
1000
1000
1000
Episode reward
730
750
700
230
0.0
0..5
1.0
1.5
2.0
0.0
0.3
1.0
1.5
2,0
0.0
0.5
1.0
1.5
20
cheetah.nuu run
walker run
quadruped run
1020
Episode reward
800
200
230
250
200
finger.tur_hhre
perddulumsswingur
quaddruped.oo
Episode reward
750
750
770
000
500
300
266
250
11.0
01.5
1.0
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.5
2.0
X11
reacher hard
swim
hopper stand
1000
1000
1000
Episade reward
200
0.0
0.3
Environment
Environment steps
Enviroement steps x10
(3%, MPC)
MAD-TD 19%)
(0%, MPC)
(0%)
Figure 19; Return curves evaluating the impact of model-based data for critic learning and MPC. Overall, MPC and model-based critic learning both stabilize the learning process, conjectured.
28

---

Published as a conference paper at ICLR 2025
dog_walk
dog_stand
Egisode reward
230
730
710
000
530
300
290
950
750
100
01.3
1.0
2.0
0.2
1.0
1.3
2,0
0.0
0.3
1.5
2.3
XXI
X12
KID
dog trot
humancid run
humanoid walk
1000
1000
1000
Epiaode reward
730
000
200
0.0
0.5
1.40
2.0
0.0
0.5
1.0
1.5
9.0
0.0
0.5
1.5
7.0
X
x11
huumancid_staree sttand
hopper_hop
acrobot acrobottsswiggup
1000
Episode reward
500
290
950
750
XXI
x100
x10
1000
cheetah
1000
run
10000
quadruped
Egiande reward
730
750
770
000
500
200
230
01.5
1.0
2.0
0.5
1.0
1.5
2.0
0.0
0.5
1.5
2.0
X11
finger_ture Barrr_uun_haarr hard
pendulum_swinguup swingup
quadruped avalk
1000
1020
1C00
Eploode reward
Vi60
250
250
0.0
0.5
1.0
1.5
0.0
7.0
X10f
1000
reacher hard
fish swim
hopper stand
Egisode reward
000
200
230
250
200
Environment stops
Environment stops Sc10
Environment steps x1I
MAD-TD
MAD-TD (reset)
MAD-TD (ro MPC)
MAD TD (so MPC, reset)
Figure 20: Return curves for the impact of resetting MAD-TD with and without MPC. Without MPC, resetting can still improve the performance, but with MPC, we see no significant benefits from resetting across environments except pendulum. The hopper results highlight the importance (and danger) of the reset interval, as seemingly the reset algorithm is not able to recover "in time" improve performance.
29

---

Published as a conference paper at ICLR 2025
dog_walk
dog_stand
Egisode reward
230
730
734
000
530
300
290
950
750
100
01.3
1.0
2.0
0.3
1.0
1.3
2,0
0.0
0.3
1.3
23
XXP
x12
KII
dog trot
humancid run
humanoid walk
1000
1000
1000
Epiaode reward
V30
730
000
0.0
0.5
1.0
2.0
0.5
1.0
1.5
0.0
0.0
0.5
7.0
X
humancid stand
hopper_hop
acrobot acrchot_swinguup
1000
Episode reward
500
990
950
150
X.I
x100
X10
1000
cheetah
1000
run
10.00
quadruped
Egisnde reward
750
750
770
000
500
200
230
0.55
1.0
2.0
0.5
1.0
1.5
2.0
0.0
0.5
1.5
2.0
X
X3I
finger_ture Jurn, Juand
pondulum_swwiguup
quadruped
1000
1020
1C00
Eploode reward
250
250
0.0
0.5
1.0
1.5
0.0
7.
X10f
1000
reacher hard
fish swim
hopper stand
Egisode reward
000
200
230
250
200
Environment stops
XMI
Environment stops X110
Environment steps xlI
MAD-TD
TD.MPC2 ((AR-1!)
Figure 21: Comparison of MAD-TD and TD-MPC2 across more environments of the DMC suite. We observe gains compared to TD-MPC2 in the hard tasks, especially terms of early learning per- formance, while TD-MPC2 has advantages the pendulum-swingur and acrobot_swingup tasks. These seem to be exploration and stability issues for which the longer model rollouts TD-MPC2 seem to help.
30

---

Published as a conference paper at ICLR 2025
E.8 METAWORLD
To broaden the basis of comparison, we compare our method to BRO and TD-MPC2 on 9 selected environments from the metaworld suite. Results be found in Figure 22.
Overall, we observe that our method performs strongly on tasks in which the agent has access to dense reward, such as lever-pull and button press. MAD-TD demonstrates the ability quickly and stably bootstrap reward when available. When exploration is challenge, learning can take longer with MAD-TD. Strong exploration for high-UTD algorithms not the focus of MAD-TD and remains an open problem (Hussing et al., 2024). This is consistent with our core hypothesis: high UTD learning benefits in cases where fitting correct value function is challenging. such as pick-place-wall the core challenge is exploration, as the agent receives no reward signal for the majority of early training. We therefore cannot expect high UTD learning to improve the performance in these tasks.
tasks
As pointed out, BRO and to a lesser extent TD-MPC2 have the benefit of exploring with optimism bonuses and ensembled value functions. We removed these from our method to cleanly study the impact model generated data. However, improvements to exploration are mostly orthogonal to proposed method and can be freely combined in future work.
Finally, as also shown by Nauman et al. (2024b), there is a curious failure case TD3 compared to SAC in the case of environments with sparse rewards. In the absence of the entropy penalty form the SAC loss function, the tanh policy of TD3 tends to saturate, which can stymie exploration completely. This is, the best of knowledge, not diseussed in the literature, and should be investigated in future work.
assembly
coffee-pul
coffee push
1.00
1000
1.00
Epicode reward
0.25
0.75
0.30
0.50
0.25
0.20
0.20
0.00
0.00
0.00
disacsomble
lever-pull
button-press
1.00
1.00
Egioode reward
0.75
0.75
0.75
0.30
0.23
0.25
0.25
0.00
0.70
x10
XII
pick-plase wak-place-wall
1.00
Episode reward
0.75
0.75
0.75
0.25
0.22
0.20
0.0
0.00
0.25 Environment steps
0.25
0.50
0.75
Environment steps
x00
11
Environment steps xII
Figure 22: Performance comparison on Metaworld between MAD-TD, BRO, and TD-MPC2. MAD-TD performs strongly on tasks which provide sufficient reward information to bootstrap the value function quickly, while learning more slowly on sparse reward tasks. This is consistent with the core goal of our algorithm, to stabilize and improve value function learning.
31


Reviews:
[{"review_id": "nkanSXbqkx", "reviewer": "Reviewer_zdTG", "summary": "This paper addresses the challenge of unstable training in off-policy reinforcement learning (RL) methods when the update-to-data ratio is high. The authors identify the root cause of this instability as the difficulty in learning accurate value functions from limited data. To mitigate this issue, they propose a novel approach called Model-Augmented Data for Temporal Difference learning (MAD-TD). MAD-TD leverages model-generated data to improve the accuracy of value functions on unobserved on-policy actions, thereby stabilizing training even at high update ratios. They empirically show that MAD-TD achieves competitive performance on tasks from the DeepMind control suite.", "questions": "1. Have you explored different values for the percentage of model generated data in MAD-TD?\n2. How does the performance of MAD-TD vary with different levels of world model accuracy? Have you conducted any ablation studies on less accurate models to understand how sensitive MAD-TD is to the world model quality?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper tackles an interesting and critical problem setting in sample-efficient reinforcement learning (RL), which has significant implications for many real-world applications where data collection is costly or time-consuming.\n2. The authors provide both empirical evidence and theoretical analysis to demonstrate the importance of addressing incorrect Q-value learning in off-policy RL with limited samples, making a compelling case for their proposed solution.\n3. The proposed method, MAD-TD, demonstrates competitive performance compared to existing baselines on challenging DeepMind Control (DMC) tasks, showcasing its potential as a viable solution for improving sample efficiency in RL.", "weaknesses": "1. Limited Baseline Comparison: The paper only compares the proposed method, MAD-TD, against two baselines (BRO and TD-MPC) and their variants, which may not be sufficient to demonstrate its performance comprehensively. A more extensive comparison with other state-of-the-art methods would strengthen the paper's claims.\n2. Lack of Ablation Study: Although the authors outline critical design choices in Section 4.1, they do not conduct an ablation study to investigate the importance of these choices. This omission makes it difficult to understand the individual contributions of each design element to the overall performance of MAD-TD.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zN3Usvw4DS", "reviewer": "Reviewer_6ahP", "summary": "In this paper the authors consider the reasons for RL to become unstable in UTD situations and they suggest a fix for this which they call MAD-TD. Basically, the data is augmented by that generated from a model to enable stabilisation. The authors then illustrate the success of their results using experiments.", "questions": "1. In the un-numbered equation on Page 3 (Please number equations!) a certain matrix is partitioned into a positive definite and a non-positive definite term. I was not quite sure what to make of the discussion below this. It of course, follows that if \\gamma is shown sufficiently small, or that the first term strongly dominates the second. I would appreciate more discussion of this.\n\n2. It wasn't clear to me what sort of guarantees we expect with the authors approach - it seemed to be more of a \"it seemed to work most of the time\"....except when it didn't. I didn't quite understand the insight about why in certain circumstances the approach didn't work. Is that related to the assumption that a sufficiently high fidelity model cannot always be learned? Some more insight would be useful.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I found this an interesting paper and the authors approach seems to generate useful results. The paper is well written and the presentation allows the reader to understand the rationale behind most of the work.\n\nThe paper combines some mathematical and intuitive insight into the stability problem. It then uses this insight as motivation for their new approach MAD-TD which seems to show some success based on the experimental results. \n\nI thought the authors gave quite a balanced presentation of the strengths, but also possible weaknesses of their work, which is commendable.", "weaknesses": "The main limitation of the paper is precisely that which the authors point out themeselves i.e. that the assumption that a sufficiently high fidelity of the model can be learned online is valid. This is necessary as the \"augmented data\" is generated from this model. However, the authors have been quite up-front with this and, despite this short-coming, their results seem to show success with their MAD-TD approach. \n\nAnother shortfall of the techniques is that of course, the main \"proof\" of the results is via experiments, although some mathematical insight is given. In other words, the authors do not prove that their MAD-TD approach is able to prevent instability, they observe, through their experiments that it seems to do well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ctmt3mB07o", "reviewer": "Reviewer_jfC9", "summary": "The authors address the instability problem in off-policy reinforcement learning with the Replay Ratio. Building on previous results showing that neural networks trained on a specific set of experiences with the UTD lead to various instabilities in training, they show that the current solutions mitigate some of the problems to a greater or lesser extent, e.g., feature normalization mitigates the overestimation problem. Still, it does not allow the model to generalize to unseen actions. Therefore, they propose using a world model to augment a small part of the experiences from the replay buffer with its help.", "questions": "1. Referring to Weakness 1.: What would the results be on the other environments or benchmarks, like Meta-World (especially environments like Stick-pull, coffee-push/pull, assembly, and others) or MyoSuit?\n2. Is there any advantage to using AR=2 rather than AR=1? Could you extend Figure 5 with AR=1 and 2 million time steps?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Authors propose a method that works effectively with high UTD without resets.\n2. The topic is relevant.\n3. The paper is well-written.", "weaknesses": "1. The results look pretty good, but then there is a question about robustness. Authors sometimes refer to [1], but the same paper points out that different effects can be obtained in RL in different environments. I agree with the theses in the manuscript, but I think it is valuable to validate these strong results on other benchmarks so that scientists in the future will know to what extent this is a general solution. Please see question 1 for more details.\n2. Minor: Equations are without numeration.\n3. Lack of an analysis of the impact of the percentage of augmented samples on performance. Could you provide plots showing performance across a range of alpha values (e.g., 1%, 5%, 10%, 25%, 50%) on a subset of representative tasks? This would give a clearer picture of the method's sensitivity to this hyperparameter.\n\n\n[1] Nauman, M., Bortkiewicz, M., Miłoś, P., Trzcinski, T., Ostaszewski, M., & Cygan, M. Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning. In Forty-first International Conference on Machine Learning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yZIxRZZZKW", "reviewer": "Reviewer_UXUR", "summary": "This paper mainly investigates the issue of overestimation of unseen on-policy actions and instability caused by a high update-to-data (UTD) ratio in off-policy reinforcement learning via theoretical analysis and experiments. To address this issue, the authors introduce a new method named Model-Augmented Data for Temporal Difference Learning (MAD-TD), which combines model-generated synthetic data with real data to enhance and stabilize the off-policy RL training process. The experiments conducted on the DeepMind Control benchmark demonstrate that MAD-TD outperforms other baselines and leads stable learning even in high UTD settings.", "questions": "How will the model data proportion affect the result? I suggest the author provide some preliminary results or insights regarding the impact of model data proportion and conduct ablation studies, if necessary, to show the trade-off on this key parameter in your method. Besides, is it possible to develop a self-adaptive mechanism to control the model data proportion during the training to enhance the final performance? To my understanding, the model error could be large at the beginning of the training process. Therefore, using fewer model-generated data at first and gradually adding the proportion as the world model converges might be helpful.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The authors provide thorough theoretical analysis and experimental results, demonstrating the effectiveness of MAD-TD in stabilizing the training process and achieving strong performance.\n2. The paper is well-organized and the key idea is clearly delivered. Open-source code is also provided for reproducibility.", "weaknesses": "1. Using an incorrect format (ICLR 2024) for submission.\n2. Typos (Also, please add indexes to each equation to make it easy to be referred to):\n- In the first equation about the definition of $L(\\theta)$, chapter 3.1, there is an extra \")\".\n- In the last equation of chapter 3.1, $n$ is missing on the top of the $\\Sigma$ colored in red.\n3. There's not much originality in the key idea of simply combining the model-generated data with the real data to augment the training process, which has already been used in previous works [1] and other model-based reinforcement learning algorithms [2-3]. As the distribution shift problem has been widely researched in offline RL, it's not surprising that a similar problem will occur in off-policy RL. Although the authors provide theoretical analysis to show the instability brought by target policy action selection, more profound results are expected, such as how will the model error affect the performance and when to trust model-generated data in off-policy RL.\n4. The algorithm is mainly based on TD-MPC2, while several components are changed. While the proposed MAD-TD is compared with the original TD-MPC2, the influence of these changes on performance has not been clearly explained or demonstrated.\n\n[1] Lu, Cong, Philip Ball, Yee Whye Teh, and Jack Parker-Holder. \"Synthetic experience replay.\" Advances in Neural Information Processing Systems 36 (2024).\n[2] Sun, Yihao, Jiaji Zhang, Chengxing Jia, Haoxin Lin, Junyin Ye, and Yang Yu. \"Model-Bellman inconsistency for model-based offline reinforcement learning.\" In International Conference on Machine Learning, pp. 33177-33194. PMLR, 2023.\n[3] Rigter, Marc, Bruno Lacerda, and Nick Hawes. \"Rambo-rl: Robust adversarial model-based offline reinforcement learning.\" Advances in neural information processing systems 35 (2022): 16082-16097.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)