Towards General-Purpose Model-Free Reinforcement Learning
Authors: Scott Fujimoto
Pierluca D'Oro
Amy Zhang
Yuandong Tian
Michael Rabbat
Venue: ICLR 2025 Spotlight (2025)
Keywords: Deep reinforcement learning
model-free
general-purpose
Primary area: reinforcement learning
TL;DR: We introduce a model-free RL algorithm with a strong performance on a variety of popular benchmarks.

Abstract: Reinforcement learning (RL) promises a framework for near-universal problem-solving. In practice however, RL algorithms are often tailored to specific benchmarks, relying on carefully tuned hyperparameters and algorithmic choices. Recently, powerful model-based RL methods have shown impressive general results across benchmarks but come at the cost of increased complexity and slow run times, limiting their broader applicability. In this paper, we attempt to find a unifying model-free deep RL algorithm that can address a diverse class of domains and problem settings. To achieve this, we leverage model-based representations that approximately linearize the value function, taking advantage of the denser task objectives used by model-based RL while avoiding the costs associated with planning or simulated trajectories. We evaluate our algorithm, MR.Q, on a variety of common RL benchmarks with a single set of hyperparameters and show a competitive performance against domain-specific and general baselines, providing a concrete step towards building general-purpose model-free deep RL algorithms.

Published as a conference paper at ICLR 2025
TOWARDS GENERAL-PURPOOS MODEL-FREE REINFORCEMENT LEARNING
Scott Fujimoto, Pierluca D'Oro, Amy Zhang, Yuandong Tian, Michael Rabbat Meta FAIR
ABSTRACT
Reinforcement learning (RL) promises framework for near-universal problem- solving. In practice however, RL algorithms are often tailored to specific bench- marks, relying carefully tuned hyperparameters and algorithmic choices. Re- cently, powerful model-based RI. methods have shown impressive general results across benchmarks but come at the cost of increased complexity and slow run times, limiting their broader applicability. fying model-free deep RL algorithm that address diverse class of domains and problem settings. To achieve this, we leverage model-based representations that approximately linearize the value function, taking advantage of the denser task objectives used by model-based RL while avoiding the costs associated with planning or simulated trajectories. We evaluate our algorithm, MR.Q. Q. on variety of common RL benchmarks with single set of hyperparameters and show a com- petitive performance against domain domain-specific and general basclines, providing a concrete step towards building general-purpose model- fode--free deep RL algorithms.
this paper, we attempt to find a uni-
Gym Locemotion actions, vector
Proprioceptive
actions, vector obs.
Cont. actions, pixel
actions, pixel
ID3-Normalized
088
0.6
Homan- Homan-Nomalinn Nomalizzd
1.2
0.9
ID-MPC2
Dreame
Reward (1k)
0.4
0.6
TD MPC2
Dreamer V3-
Reward
0.4
TD-MP22 H
Dreamer
2.0
3.
Dreamer
+
0.3
0.6
TD7
MR. Q
0.2
MR Q
TD7
MR Q
DrQ-v2
PPO
1.0
MR Q
Raunbow
DQN
PPO
0.0
0.0
Parameter Count
Training FPS
Evalustion FPS
Parameter Count
Gym (HallCheetah-v4)
Gym (HallCheetab-v4)
Gym (HalrChovtah--v4)
Atari (Any)
MR.Q IM)
MR.Q (49)
MR.Q (1.9k)
MR. Q (4.4M)
DreamerV3 (9.7M)
DreamerV3 (18)
DreamerV3
DreamerV3 (187.3M)
TD-MPC2 5.5M
MPC2 (14)
TD-MPC2 (27)
Rainbow (6.5M)
10
10
30
40
50
0.0
0.5
1.0
1.5
2.0
50 100
150
200
Parameters (1M)
Frames per second
Frames per second (Ik)
Parameters (IM)
Figure Summary of results. Aggregate mean performance across four common RL benchmarks and 118 environments featuring diverse characteristics (c.g., observation and action spaces, task types), Error bars cap- ture 95% stratified bootstrap confidence interval. Our algorithm, MR.Q, achieves a competitive performance against both state-of-the-ar Nomain-specifff and general baselines, while using single set of hyperparameters. Notably, MR.Q accomplishes this with fewer network parameters and substantially faster training and evalua- tion speeds than general purpose basel methods.
INTRODUCTION
The conceptual premise of RL is inherently general- sareall-ppooeee-ea an RL agent can learn optimal behav- ior with only two basic elements: well-defined objective and data describing its interactions with
Correspondence: sfuj noto@meta com. Code: http:///gthh..com/ffcccooteeeene

---

Published as a conference paper at ICLR 2025
the environment. In reality, however, most RL algorithms anything but general-purpose. Instead, RL algorithms are highly specialized and typically characterized by specific problem classes, such discrete versus continuous actions or vector versus pixel observations, with each category of algorithmic choices
Table I: Hyperparameter differences between Rain- bow (Hessel al., 2018) and TD3 (Fujimoto al., 2018). TD3 uses an expected moving average (EMA) update with an effective frequency of 1-0.996
requiring its and hyperparameters. For example, Rainbow and TD3 (Hessel et al., 2018; Fujimoto et al., 2018), common methods for Atari and MuJoCo respectively (Bellemare et al., 2013; Todorov et al., 2012), have more differences than sim- ilarities in their shared hyperparameters (Ta- ble witttho accounting for further algorith- mic differences.
200.
Hyperparameter
Rainbow
TD3
Discount factor
0.99
0.99
Optimizer
Adam
Adam
Learning Rate
6.25-
10
10-3
Adam
1.5- 10
10-s
Replay buffer size
1M
1M
To some extent, general-purpose algorithms do exist-policc gradient methods (Williams, 1992: Schulman et al., 2015; 2017) and many evolutionary approaches (Rechenberg, 1978; Back, 1996; Rubinstein, 1997; Salimans et al., 2017) require few assumptions on the underlying problem. Unfortunately, these methods often offer
Minibatch size
32
100
Target network update Effective target update freq.
Iterative
EMA
8k
200
Initial random steps
20k
1k
poor sample efficiency and asymptotic performance compared more domain-specific approaches, and in some instances, can require extensive re-tuning over numerous implementation--eve de- tails (Engstrom et al., 2020; Huang al., 2022).
Recently, DreamerV3 (Hafner et al., 2023) and TD-MPC2 (Hansen et al., 2024), have showcased the potential of general- purppse model-based approaches, achieving impressive single-tast task performance on diverse set of benchmarks without re- re-tuning hyperparameters. However, despite their success, model-based methods also introduce substantial algorithmic and computational complexity, making them less practical than lightweight domain-speciff model-free algorithms.
This paper presents a general model-free RL algorithm that leverages model-based representations to achieve the sample efficiency and performance of model-based methods, without the computa- tional overhead. A recent surge of high-performing model-free RL algorithms with dynamics-based representations (Guo et al., 2020; 2022; Schwarzer et al., 2020: 2023; Zhao al., 2023; Fujimoto et al., 2024; Zheng et al., 2024; Scannell et al., 2024) has showcased the potential of this family of algorithms when tailored for a single benchmark. Recognizing the similarity between these model- based and model-free approaches, our hypothesis is that the true benefit model-based objectives is in the implicitly learned representation, rather than the model itself, and thus prompting the question:
Can model-based representations alone enable sample-efficient general -urppss learning?
Our proposed approach is based on learning features that approximately capture linear relationship between state-action pairs and value. To do so, we draw heavily from modern dynamics-based representation learning methods (see Related Work) as well the work of Parr et al. (2008), who show that both model-basee and model-free objectives converge to the same solution in linear space. eliminate any environment-
By mapping states and actions into single, unified embedding. specific characteristics of the input space and allow for standardized set of hyperparameters.
We evaluate method, MR.Q. on four widely used RI. benchmarks and 118 environments, and achieve competitive performance against state-of-the-aar domain-specific and general baselines without algorithmic or hyperparameter changes between environments benchmarks.
2 RELATED WORK
General-purpose RL. Although many traditional RL methods are general-purpose in principle, practical constraints often force assumptions about the task domain. For example, algorithms like Q-lerrning and SARSA (Watkins, 1989; Rummery & Niranjan, 1994) can be conceptually extended to continuous spaces, but are typically implemented using discrete lookup tables. In practice, early examples of general decision-making approaches can be found in on-policy methods with function approximation. For instance, both evolutionary algorithms (Rechenberg, 1978; Back, 1996; Rubin- stein, 1997; Salimans et al., 2017) and policy gradient methods (Williams, 1992; Sutton al., 1999;
2

---

Published as a conference paper at ICLR 2025
Schulman al., 2015; 2017) offer update rules with convergence guarantees and independence to the imput space. However, despite their generality, these methods are also hindered by poor sample efficiency and are prone to local minima, limiting their suitability for many practical applications.
In contrast, the design of deep RL algorithms tends to favor more specialized approaches that align closely with single benchmark- e.g.. DQN+>Atar (Bellemare et al., 2013; Mnih et al., 2015), DDPG+~MuJoCo (Todorov et al., 2012; Lillicrap et al., 2015), or AlphaGo++>Go (Silver et al., 2016). Generalizing beyond these initial benchmarks can often require significant engineering, tuning, or algorithmic discovery (Luong et al., 2019; Schrittwieser et al., 2020; Haydari & Yilmaz, 2020; Ibarz al., 2021). In imitation learning, GATO achieved generalist behavior, but relied on large expert datasets (Reed et al., 2022). Recently, DreamerV3 (Hafner et al., 2023) demonstrated strong capability over many benchmarks without re-tuning, but used costly large models and simulated rollouts. Our objective is to discover lightweight model-free approach to general-purposs learning.
Dynamics-based representation learning. Building representations from system dynamics is a long-standing approach for adaptation, partial observability, and feature selection (Dayan, 1993; Littman & Sutton, 2001; Parr et al., 2008). Numerous model-free methods have been developed to learn representations by predicting future latent states (Munk et al., 2016; Van Hoof et al., 2016; Zhang et al., 2018; Gelada et al., 2019; Lee et al., 2020; Guo et al., 2020; 2022; Schwarzer et al., 2020; 2023; Zintgraf et al., 2021; Yu et al., 2021; 2022; Fujimoto et al., 2021; 2024; McInroe al., 2021; Seo et al., 2022; Kim et al., 2022; Tang et al., 2023; Zhao et al., 2023; Zheng et al., 2024; Ni et al., 2024; Scannell et al., 2024). Unsurprisingly. these model-free approaches closely relate to model-based counterparts which learn a latent dynamics model for planning or value estima- tion (Watter et al., 2015; Finn et al., 2016; Karl al., 2017; Ha & Schmidhuber, 2018; Schrittwieser et al., 2020; 2021; Ye et al., 2021; Hansen et al., 2022; 2024; Hafner et al., 2019; 2023; Wang et al., 2024). Our approach, MR.Q. Q, is most closely related to the state-action representation learning in TD7 (Fujimoto et al., 2024). At a high level, MR.Q differs from TD7 by discarding the original input and including losses over the reward and termination. MR.Q also differs significantly in im- plementation, drawing inspiration from prior work to determine a set of design choices that performs well across benchmarks, including multi-step returns, unrolled dynamics, and categorical losses.
Our motivation also relates to linear MDPs (Jin et al., 2020; Agarwal et al., 2020) and linear spectral representation (Ren et al., 2022; 2023; Zhang et al., 2022; Shribak et al., 2024). The latter aims to learn low-rank decomposition of the transition dynamics of the MDP and recover a linear relation- ship between an embedding and the value function. Similarly, our work connects to two-stage linear RI., where non-linear embedding is learned for linear RL (Levine al., 2017; Chung al., 2019).
State abstraction. Our work is closely related to bisimulation metrics (Ferns et al., 2004; 2011; Castro, 2020) and MDP homomorphisms (Ravindran, 2004; der Pol et al., 2020a;b; Rezaei- Shoshtari et al., 2022) which rely on measures of similarity in reward and dynamics for state or action abstraction. These concepts have inspired practical approximations bisimulation metrics as a means of shaping representations in deep RL agents, particularly those using image-based observations (Zhang et al., 2020; Castro et al., 2021; Zang et al., 2022).
BACKGROUND
Reinforeement learning (RL) problems are described by a Markov Decision Process (MDP) (Bell- man, 1957), which we define by a tuple (S, A, R, 7) of state space S, action space A, dynam- ies function p. reward function R and discount factor 7. Value-based RI. methods learn a value function Q" (s.a) = F. [E==0 7'rel0 = 8, ag a) that models the expected discounted sum of rewards Tr R( st, az) by following a policy æ which maps states 5 to actions a.
The true value function Q" is estimated by approximate value function Qs. We use subscripts to indicate the network parameters 0. Target networks, which are used to introduce stationarity in prediction targets, have parameters denoted by an apostrophe, e.g., Q8. These parameters are periodically synchronized with the current network parameters (f' + 0).
4 MODEL-BASED REPRESENTATIONS FOR Q-LEARNING
This section presents the MR.Q algorithm (Model-based Representations for Q-lcarning), a model- free RI. algorithm that learns an approximately linear representation of the value function through
3

---

Published as a conference paper at ICLR 2025
model-basee objectives. Value-based RL algorithms learn value function Q that maps state-action acte-action pairs (s, a) to values in R and a policy т that maps states s to actions a. Like many representation learning methods for RL, MR.Q adds an initial step that transforms states and state-action pairs into embeddings 2, and ssa, which serves as inputs to the downstream policy and value function.
Jw 8 V.g
Iw : ($,a) a) - Zsu
Te 22 a,
Qa Zna IR.
While neither the value function nor policy require explicit representation learning, using interme- diate embeddings has two main benefits:
Introducing an explicit representation learning stage can enable richer alternative learning signals that are grounded in the dynamics and rewards of the MDP, as opposed to relying exclusively on non-stationary value targets used in both value and policy learning.
2. Representation learning transform the input into unified, abstract space that is decou- pled from the original input characteristics, e.g., images or action spaces. This abstraction allows us to filter irrelevant or spurious details and use unified downstream architectures, improving robustness to environment variations.
To learn these embeddings, draw inspiration from lincar feature selection, revisiting the work of Parr et al. (2008), as well as MDP homomorphisms (Ravindran & Barto, 2002). In Section 4.1 we highlight how model-based objectives can be used to learn features that share an approximately lin- ear relationship with the true value function. Then in Section 4.2, we relax our theoretical motivation for practical algorithm based on recent advances in dynamics-based representation learning.
4.1 THEORETICAL MOTIVATION
Consider a linear decomposition of the value function, where the value function Qssa)) is repre- sented by features zsa and linear weights w:
Q(s.))= z.a W.
(3)
Our primary objective to learn features Zeo that share an approximately linear relationship with true value function Q" However, since this relationship is only approximate, we use these features input to non-linear function (Zza). rather than relying solely on linear function approximation.
We start by exploring how to find features that can linearly represent the true value function. Given dataset D of tuples (s, s', a'), we consider two possible approaches for learning value func- tion Q: A model-free update based on semi- momi-gradient TD (Sutton, 1988; Sutton & Barto, 1998):
- w aED Vw (z w |r +
(4)
A model- based approach to learn Wmb, based on rolling out estimates of the dynamics and reward:
Wmb := Σ Ww,,
(5)
E=
Wr argmin ED (zlaw-r))
Wp argmin W
(z W )=]
(6)
Closely following Parr et al. (2008) and Song al. (2016), converge to the same solution (proofs for this section can be found in Appendix A).
show that these approaches
Theorem 1.
The fixed point of the model-free approach (Equation 4) and the solution the the model-
based approach (Equation 5) are the same.
From the insight of Theorem 1, can connect the value error VE, the difference between approximate value function Q and the true value function Q",
VE(s, a) =
to the accuracy of reward and dynamics components of the estimated model (Theorem 2).
Theorem 2. The value error of the solution described by Theorem is bounded by the accuracy of the estimated dynamics and reward:
max [VE(s,)) 1 7 (s,a)e5xA
[Kta w, E s. max /w,| Σ J2 W, (a (8)
4

---

Published as a conference paper at ICLR 2025
Parr et al. (2008) and Song et al. (2016) use a related insight regarding the Bellman error to infer an approach for feature selection. However, with the advent of deep learning, we can instead directly learn the features zsa by jointly optimizing them alongside the linear weights w, and W p- This is accomplished by treating the features and linear weights as unified end-to-end model and balancing the losses in Equation 6 with hyperparameter A:
L(zsa We Wp) ED (2 Wr Wr-i -r)
+) AED [ Wp Zs'a') 2
learming
Dynamics learning
However, the resulting Equation has some notable drawbacks.
Dependency on The dynamics target z depends action a' determined by the policy m. In policy optimization problems, this introduces non-stationarity, where the target embedding must be continually updated to reflect changes in the policy. This creates undesirable interdependenee between the policy and encoder.
Undesirable local minima. Jointly optimizing both the features Zea and the dynamics target can lead to undesirable local minima, similar to the issues encountered with Bellman residual minimiza-
tion (Baird, 1995; Fujimoto al., 2022). This can result in collapsed or trivial solutions when the dataset does not fully cover the state and action space or when the reward is sparse.
To address these issues, suggest relaxations our proposed, theoretically grounded approach:
[(Zsa wr, W;) = En (z.n w, - -) AB an W
Z
(10)
Adjustment
We propose two key modifications to alleviate the aforementioned issues. Firstly, dependent embedding z as the dynamics target, rather than the state-action embedding Zy'a This eliminates any dependency the current policy while still capturing the environment's 's dynamics.
state-
Secondly, to mitigate the issue local minima, we use a target network (s') to generate the dynamics target Z where the parameters w' are periodically updated track the current network parameters w. Empirical evidence from prior work suggests that this approach can yield significant performance gains (Grill et al. (2020); Assran et al. (2023), see Related Work), although it no longer guarantees convergence to a fixed point.
Due these two changes, even if the modified objective defined by Equation 10 is minimized, we can no longer assume there is a linear relationship between the embedding Zan. and the value function. However, can instead allow for non-linear relationship, replacing linear weights w with a non-linear function (zza). We can show that this relationship exists as long as the features are sufficiently rich (i.e., such that a MDP homomorphism is satisfied (Ravindran & Barto, 2002)).
Theorem 3. Given functions f()) 2, and g(z,; a) Zsas then if there exists functions and R such that for all (s,a) E 5xxx A:
[R( En [R(s,))],
((2)(as)) = Σ p())s,a);
(11)
812,=2,'
then for any policy where there exists a corresponding policy 7 (alz,) = (als)), there exists function Q equal to the true value function Q" over all possible state-action pairs (s, a) S x A:
(Z...) = (s,
(12)
Furthermore, Equation 11 guarantees the existence of an optimal policy # (a(az))) =" (als).
Consequently, even the features Zsa do not linearly represent the true value function, i.e., the loss in Equation 9 cannot be not exactly minimized, Z still be used in a non-linear relationship to represent the value function. Furthermore, Theorem 3 outlines a similar objective as the original linear objective defined in Equation 9, in learning the reward and dynamics of the MDP.
These results motivates the practical algorithm discussed in the following section. Using the adjusted loss defined in Equation 10, will aim to learn features with approximately linear relationship to the true value function, but use non- non-linear value function with those features to account for the error induced by approximations.
5

---

Published as a conference paper at ICLR 2025
4.2 ALGORITHM
We now present the details of MR.Q (Model-based Representations for Q-learning). Building on the insights from the previous section, our key idea is learn state-action embedding 2sa that approximately linear with the true value function Q To account for approximation emors, these features are used with non-linear function approximation to determine the value.
The state embedding vector Z, is obtained as intermediate component by training end-to-end with the state-action encoder. MR. Q handles different input modalities by swapping the architecture of the state encoder. Since 2 is vector, the remaining networks independent of the observation space and use feedforward networks.
Given the transition (s, (s,,rr,d,s) from the replay buffer:
Output MR.Q
Update MR.Q
Trained end-to-end
if % Turger 0 then
State Encoder
2, (s)
Target networks: 8 10 w/ 0,0,00000 p,
State-Action Encoder
Esa = gN (z
Reward scaling: + F, F meanpr.
MDP predictor
2 f, d 80
for Target time steps do Encoder update: Equation 14.
Decoupled RI
Value
Q (zya)
Value update: Equation 19.
Policy
ax TT (z,)
Policy update:
Equation 20.
The encoder loss is composed of three terms based on the reward, dynamics and terminal signal that unrolled over short horizon. The value function and policy trained independently, using standard losses (Silver et al., 2014; Fujimoto et al., 2018). We use LAP (Fujimoto et al., 2020) sample transitions with priority according to their TD errors (Schaul et al., 2016), the absolute difference between the predicted value and the target value in Equation 19.
The target network, reward scaling (defined in Equation 19), and the encoder are updated period- ically every Turget time steps. This synchronized update schedule keeps the input and target out- put fixed for the downstream value function and policy within each iteration, thus reducing non- stationarity in the optimization (Fujimoto et al., 2024).
4.2.1 ENCODER
The encoder loss is based on unrolling the dynamics of the learned model over short horizon. Given a subsequence of an episode (50, do, F1, di, ,1, 81, by encoding the initial state 8o, then by repeatedly applying the state-action encoder gu and linear MDP predictor m:
) the model is unrolled
2',, d = ( "
Im m,
where d := fow (so).
(13)
The final loss is summed over the unrolled model and balanced by corresponding hyperparameters:
Hise
LEncoder Af,coderrf.gg (f,g,
ARoward (F') ADynamics (2')) + ATeemina (d') (d)). (14)
ATeminal is set to 0 until the first terminal transition (i.e., d= 0) is viewed. This approach is commonly used in model-based RL (Oh et al., 2015; Hafner al., 2023; Hansen al., 2024), well as dynamics-based representation learning (Schwarzer et al., 2020; 2023; Scannell et al., 2024).
Reward loss. While our theoretical analysis suggests using the mean-squared error to train the predicted reward, we find that a categorical representation of the reward is more effective in practice for predicting sparse rewards and is robust reward magnitude. This empirical benefit consistent with prior work (Schrittwieser et al., 2020; Hafner et al., 2023; Hansen et al., 2024; Wang et al., 2024). Our reward loss function uses the cross entropy CE between the predicted reward and two-hot encoding of the reward r:
PReward (P) CE (F, Two-Hot(r)).
(15)

---

Published as a conference paper at ICLR 2025
To handle wide range of reward magnitudes without prior knowledge, the locations of the two-hot encoding are spaced at increasing non-uniform intervals, according to symexp(#) (ign(=)(exp (x) 1) (Hafner et al., 2023).
Dynamics loss. The dynamics loss minimizes the mean-squared error between the predicted next state embedding 2 and the next state embedding 2 from the target encoder fur:
Lyynamies (2x) = (Z2- -Zz)²
(16)
As discussed in the previous section, using the next state embedding 2 eliminates the dependency on the policy that would occur when using a state-action embedding target.
Terminal loss. The predicted scalar terminal signal d trained simply using a MSE loss with the binary terminal signal d:
Lerminnll (d) := (d 2)
(17)
4.2.2 VALUE FUNCTION
Value learning is primarily based on TD3 (Fujimoto et al., 2018). Specifically, we train two value functions and take the minimum output between their respective target networks determine the value target. Similar to TD3, the target action is determined by the target policy Tws, perturbed by small amount of clipped Gaussian noise:
aargmxxa a' 0x
for discrete A, clip(a',-1,1) for continuous A, where a' = TT clip((,-eec)) e e~((000²).
(18)
Discrete actions are represented by one-hot encoding, where the Gaussian noise is added to each dimension. Action noise and the clipping scaled according the range of the action space.
We modify the TD3 loss in few ways. Firstly, following numerous prior work across bench- marks (Hessel et al., 2018; Barth-Maron et al., 2018; Yarats et al., 2022: Schwarzer et al., 2023), the Huber loss instead of mean-
predict multi-step returns over horizon Hq. Secondly, squared error to eliminate bias from prioritized sampling (Fujimoto et al., 2020). Finally, the target value is normalized according to the average absolute reward in the replay buffer:
1
Ha-1 Σ
Ho (adi)) r 2
Liaaue (Q:) Huber @
== F' min Qø (MsH)a (19)
j=1.2
The value F' captures the target average absolute reward, which is the scaling factor used to the most recently copied value functions Qoy. This value is updated simultaneously with the target networks
p' -. F. Maintaining a consistent reward scale keeps the loss magnitude constant across different benchmarks, thus improving the robustness single set of hyperparameters.
4.2.3 POLICY
For both continuous and discrete action spaces, the policy is updated using the deterministic policy gradient (Silver al., 2014):
Lpolicy (az) -0.5
Σ (i(Zaa,) + Apre activ 2 where ax activ(z,)) (=112
(20)
To make the loss universal between action spaces, use Gumbel-Softmar (Jang al., 2017; Lowe et al., 2017; Cianflone et al., 2019) for discrete actions, and Tanh for continuous actions. A small regularization penalty is added to the square of the pre-activations z, before the policy's 's final acti- vation to help avoid local minima when the reward, and value, is sparse (Bjorck et al., 2021).
For exploration, Gaussian noise is added to each dimension of the action (or one-hot encoding of the action). Similar to Equation 18, the resulting action vector is clipped to the range of the action space for continuous actions. For discrete actions, the final action is determined by the argmax operation.

---

Published as a conference paper at ICLR 2025
Gym Locemotion
DMC Proprioceptive
DMC- Visual
Atari 57 tasks
tasks
28 tasks
28 taske
TD2 N---Nommllzdd
1.2
1.5
Toral Reward (1K)
0.8
0.6
Toral Reward (1k)
0.4
0.6
0.5
Homun- Noemalized
3.0
0.9
0.4
0.3
2.0
0.6
1.0
0.3
0.2
0.0
0.0
0.0
0.0
0.6
1.0
0.3
0.5
0.3
0.5
2.0
Time Steps (1M)
Time Steps (1M)
Time Steps (1M)
Time Steps (IM)
MR.Q
DreamerV3
A TD-MPC2
PPO
TD7
DrQ-v2
Rainhow
DQN
Figure 2: Aggregate learning curves. Average performance each benchmark. Results
10 seeds.
The shaded area captures a 95% stratified bootstrap confidence interval. Due to action repeat. 500k time steps DMC correspond to 1M frames in the original environment and 2.5M time steps in Atari corresponds to 10M frames in the original environment.
EXPERIMENTS
We evaluate MR.Q on four popular RL benchmarks and 118 environments, and compare its performance against strong domain-specific baselines, general model-based approaches, Dream- erV3 (Hafner et al., 2023) and TD-MPC2 (Hansen et al., 2024), and general model-free algorithm, PPO (Schulman et al., 2017). Rather than establish MR.Q as the state-of-the-art approach in any par- ticular benchmark, our objective is to demonstrate its broad applicability and effectiveness across diverse set of tasks with single set of hyperparameters. The baselines use author- suggested default hyperparameters and are fixed across environments. Additional details can be found in Appendix B.
5.1
MAIN RESULTS
Aggregate learning curves are displayed in Figure 2, with full results displayed in Appendix C.
Gym - Locomotion. This subset of the Gym benchmark (Brockman et al., 2016; Towers et al., 2024) considers locomotion tasks in the MuloCo simulator (Todorov et al., 2012) with continuous actions and low level states. Agents are trained for 1M time steps without any environment prepro- cessing. We evaluate against three baselines: TD7 (Fujimoto et al., 2024), a state-of-the-ar (or near) approach for this benchmark, well as TD-MPC2, DreamerV3, and PPO. To aggregate results, normalize using the performance of TD3 (Fujimoto et al., 2018).
DMC- Proprioceptive. The DeepMind Control suite (DMC) (Tassa et al., 2018) is collection of continuous control robotics tasks built on the MuJoCo simulator. These tasks use the proprioceptive states as the observation space, meaning that the input is a vector, and limit the total reward for each episode at 1000, making it easy to aggregate results. We report results on all 28 default tasks that were used by either TD-MPC2 or DreamerV3. Agents are trained for 500k time steps, equivalent to 1M frames in the original environment due to action repeat. For comparison, we evaluate against the same three algorithms as in the Gym benchmark, with TD-MPC2 considered state-of-the the- (or near) for this benchmark. We also include TD7 due to its strong performance in the Gym benchmark.
DMC- Visual. The visual DMC benchmark includes the same 28 tasks as the proprioceptive bench- mark, but uses image- based observations instead. Agents are trained for 500k time steps. For base- lines, we include DrQ-v2 (Yarats al., 2022), given its state-of-the-ar (or near) performance in model-free RI, alongside TD-MPC2, DreamerV3, and PPO.
The Atari benchmark built on the Arcade Learning Environment (Bellemare et al., 2013). This benchmark uses pixel observations and discrete actions and includes the 57 games used by D3eamerV3. We follow standard preprocessing steps, including sticky actions (Machado et al., 2018) (full details in Appendix B.3). Agents are trained for 2.5M time steps (equivalent to 10M frames), a setting which has been considered by prior work (Sokar et al., 2023). For comparison, we evaluate against three baselines: the model-based approach DreamerV3, as well as model-free ap- proaches, DQN (Mnih et al., 2015), Rainbow (Hessel et al., 2018), 8), and PPO. Results are aggregated by normalizing scores against human performance.
Atari.
8

---

Published as a conference paper at ICLR 2025
Discussion. Throughout experiments, we find the presence of "no free lunch", where the top- performing baseline in one benchmark fails to replicate its success in another. Regardless, MR. Q achieves the highest performance in both DMC benchmarks, showcasing its ability to handle dif- ferent observation spaces. Although it falls slightly behind TD7 in the Gym benchmark, MR.Q is the strongest method overall across all continuous control benchmarks. In Atari, while DreamerV3 outperforms MR.Q. it relies on model with 40 times more parameters and struggles comparatively in the remaining benchmarks. When compared to the model-free baselines, MR. Q surpasses PPO, DQN, and Rainbow, demonstrating its effectiveness with discrete action spaces.
5.2 DESIGN STUDY
To better understand the impact of certain design choices and hyperparameters, we attempt variations of MR.Q, and report the aggregate results in Table 2.
Table 2: Design study. Average difference in normalized performance from varying design choices across each henchmark over seeds. Negative changes are highlighted lightly [-0.01, Damaging changes are highlighted moderately [-0.2, -0.5). Catastrophic changes are highlighted boldly (s -0.5). Positive changes
are similarly highlighted (> 0.01).
Gym Locomotion DMC Proprioceptive Reward (Ik)
DMC Visual
Atari 1M Hluman-Normalizer
Design
TD3-Normalized
Reward (Ik)
Relaxations
Linear value function
-1.17
-1.19,11.15
-0.58
[-0.99, -0.56)
-0.41
[00442, -0.39]
-1.35
-1.41,11.22
Dynamies target
-0.10 [-0177 -0.04]
-0.15
[-0.15,
-0.05
[-0.05, -0.04]
-0.38
[-0.81, 0.05]
No target encoder
-0.53
[-060, 4.44]
-0.35
|-0.35,
-0.15
|-0.15, -0.15]
-0.86
[-0.89, -0.83]
Revert
-1.47
[-1.54,
-0.72
[-0.73, -0.72]
-0.52
[0.52, -0.51]
-1.69
11.70.-1.67
Non-linear model
-0.01
[-0.07, 0.03]
-0.00
[-0.02, 0.01]
-0.01 [-0.02, -0.00]
-0.07
[-0.32, 0.18]
Loss functions
MSE reward loss
0.10 [-0.02, 0.19]
-0.06 [-0.08, -0.05]
-0.05
[-0.07, -0.04]
-0.79
[-0.66, -0.73]
No reward scaling
-0.04 [-0.09, 0.02]
-0.01 [-0.02, 0.00]
-0.00 [-0.01, 0.01]
0.18
[-0.25, 0.56]
No min
-0.09 [-0.16, -0.01]
-0.01 [-0.02, 0.01]
0.00 [-0.01, 0.01]
0.13
[-0.10, 0.58]
No LAP
-0.10 [-0.24, -0.00]
0.00 [-0.00, 0.01]
-0.01 [-0.02, -0.01]
-0.13
[-0.38, 0.14]
No MR
-0.56 -0.69, -0.43]
-0.19 100119
-0.07 [-0.09, -0.03]
-0.78
(-0.88, -0.69]
Horizons
1-step return
-0.33 [-0.46, -0.21]
-0.04 [-0.05, -0.02]
-0.03 [-0.03, -0,02]
-0.70
(0.811 -0.99]
No unroll
0.07 [0.01, 0.14]
-0.01 [-0.01, -0.00]
-0.04 [-0.06, -0.01]
-0.33 [-0.41, -0.28]
Relaxations. In Section 4.1, we outlined loss (Equation 9) that, if globally minimized, would provide features that are linear with the true value function. MR.Q in practice relaxes this theoret- ical result by modifying the loss and using non- non-linear value function. In Linear value function, we replace the non-linear value function with a linear function. In Dynamics target, we replace the state embedding dynamics target with state-action embedding Z determined from the target state-action encoder gw- In No target encoder. we use the current encoder to generate the dynamics target z and jointly optimize it within the encoder loss. In Revert, we consider all of the afore- mentioned changes simultancously, using linear value functions and setting the dynamics target as a Mtte-action embedding determined by the current encoder. Non-linear model, we replace the linear MDP predictor with individual networks that predict each component separately from Zza.
Loss functions. MR.Q's loss functions use several unconventional choices. MSE reward loss, we replace the categorical loss function on the predicted reward in Equation 15 with the mean- squared error (MSE). In No reward scaling, we remove the reward scaling in Equation 19, setting F 7 = 1. In No min, take the mean over the target value functions instead of the minimum in Equation 19. In No LAP, we remove prioritized sampling (Fujimoto et al., 2020) and use the MSE instead of the Huber loss in the value update. Lastly, in No MR, remove model-based representation learning and train the encoder end-to-end with the value function.
Horizons. Finally, we consider the role of extended predictions. In 1-step return, remove multi-step value predictions and use TD learning. In No unroll, we remove the dynamics unrolling in Equation 14, by setting the encoder horizon H = 1.

---

Published as a conference paper at ICLR 2025
Discussion. The results of our design study show the benefit of balancing theory with practical relaxations. The experiments further validate our design choices and hyperparameters. We highlight two results in particular: (1) increasing the model capacity in the "non-linear model" experiment, does not improve performance. This outcome suggests that maintaining an approximately linear relationship with the value function can be more impactful than increased capacity. (2) Our study also reveals a key distinction between the Gym and Atari benchmarks-white the "MSE reward loss" and "No unroll" variants offer moderate performance gains in Gym, they significantly degrade performance in Atari. This discrepancy highlights how hyperparameters can overfit to individual benchmarks, emphasizing the importance of evaluating algorithms across multiple benchmarks.
DISCUSSION AND CONCLUSION
This paper introduces MR.Q, a general model-free deep RL algorithm that achieves strong per- formance across diverse benchmarks and environments. Drawing inspiration from the theory of model-based representation learning, MR. demonstrates that model-free deep RL is promising avenue for building general-purpose algorithms that achieve high performance across environments, while being simpler and less expensive than model-based alternatives.
Our work also reveals insights which design choices matter when building general-purpose model-free deep RL algorithms and how common benchmarks respond to these design choices.
Model-based and model-free RL. MR.Q integrates model-based objectives with model-free backbone during training, effectively blurring the boundary between traditional model-based and model-free RL. While MR.Q could be extended to the model-based setting by incorporating plan- ning or simulated trajectories with the state-action encoder, these components can add significant execution time and increase the overall complexity and tuning required by a method. Moreover, the performance of MR.Q UR.Q in these common RL benchmarks demonstrates that these model-based that the representation itself could be the most
components may be simply unnecssary--uugeeiinn valuable aspect of model-based learning, even in methods that do use planning. This argument is echoed by D3eamerV3 and TD-MPC2, which rely on short planning horizons and trajectory gen- eration, while including both value functions and traditional model-free policy updates. As such, may be necessary to examine more complex settings, to reliably see benefit from model-based search or planning, e.g., (Silver et al., 2016).
Universality of RL benchmarks. Our results demonstrate that there is a striking lack of positive transfer between benchmarks. For example, despite the similarities in tasks and the same underlying MuJoCo simulator, the top performers Gym and DMC fail to replicate their success on the oppos- ing benchmark. Similarly, although DreamerV3 excels at Atari, these performance benefits do not translate to continuous control environments, underperforming TD3 in Gym and outright failing to learn the Dog and Humanoid tasks in DMC (see Appendix C). These findings show the limitations of single-benchmark evaluations, indicating that success on one benchmark may not translate easily to others, and highlights the need for more comprehensive benchmarks.
Linitations. MR.Q is only the first step towards a new generation of general- purppose model-free deep RL algorithms. Many challenges remains for fully general algorithm. In particular, MR.Q Q is not equipped to handle settings such as hard exploration tasks or non-Markovian environments. Another limitation is our evaluation only considers standard RL benchmarks. Although this allows direct comparison with other methods, established algorithms such as PPO have demonstrated their effectiveness in highly unique settings, such as team video games (Berner et al., 2019), drone rac- ing (Kaufmann et al., 2023), and large language models (Achiam et al., 2023; Touvron et al., 2023). To demonstrate similar versatility, new algorithms must undergo the same rigorous testing across a range of tasks that is beyond the scope of any single study.
As the community continues to push the boundaries of what is possible with deep RL, we believe that building simpler general-purpose algorithms has the potential to make this technology more accessible to a wider audience, ultimately enabling users to train agents with case. Perhaps one day with just the click of a button.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We would like thank Brandon Amos, Mikhael Henaff, Luis Pineda, Paria Rashidinejad, and Qinqing Zheng for insightful discussions and comments.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Igge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Alckh Agarwal. Sham Kakade, Akshay Krishnamurthy, and Wen Sun. Flambe: Structural com- plexity and representation learning of low rank mdps. Advances neural information processing systems, 33:20095-20105 2020.
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with joint-embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15619-15629 2023.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: 1607.06670,5 2016.
Thomas Back. Evolutionary algorithms in theory and practice: evolution strategies, evolutionary programming, genetic algorithms. Oxford university press, 1996.
Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Zhaohan Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. In International Conference on Machine Learning, pp. 507-517. PMLR, 2020.
Leemon Baird. Residual algorithms: Reinforcement learning with function approximation. In Machine Learning Proceedings 1995, pp. 30-37. Elsevier, 1995.
Gabriel Barth-Maron, Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva TB, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributional policy gradients. Interna- tional Conference on Learning Representations, 2018.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Richard Bellman. markovian decision process. Journal of mathematics and mechanics, pp. 679- 684, 1957.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, al. Dota 2 with large scale deep reinforcement learning. arXiv preprint 1912v:1922.066 2019.
Johan Bjorck, Carla P Gomes, and Kilian Q Weinberger. Is high variance unavoidable in rl? study in continuous control. In International Conference on Learning Representations, 2021.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Pablo Samuel Castro. Scalable methods for computing state similarity in deterministic markov decision processes. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 10069-10076. 2020.
Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Mare G Belle- mare. Dopamine: A research framework for deep reinforcement learning. arXiv preprint
1rXiv:1812.0 10, 2018.
11

---

Published as a conference paper at ICLR 2025
Pablo Samuel Castro, Tyler Kastner, Prakash Panangaden, and Mark Rowland. Mico: Improved representations via sampling-base based state similarity for markov decision processes. Advances in Neural Information Processing Systems, 34:30113-30122 2021.
Wesley Chung, Somjit Nath, Ajin Joseph, and Martha White. Two-timescale networks for nonlinear value function approximation. In International Conference on Learning Representations, 2019.
Andre Cianflone, Zafarali Ahmed, Riashat Islam, Avishek Joey Bose, and William L Hamilton. Discrete off-policy policy gradient using continuous relaxations, 2019.
Djork-Amé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network
learning by exponential linear units (elus). arXiv preprint arXiv: 11 2015.
Peter Dayan. Improving generalization for temporal difference learning: The successor representa- tion. Neural Compulation, 5(4):613-624, 1993.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras. Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep rl: A case study ppo and trpo. In International Conference on Learning Representations, 2020.
Norm Ferns, Prakash Panangaden, and Doina Precup. Metrics for finite markov decision processes. In UAI, volume 4, pp. 162-169, 2004.
Norm Ferns, Prakash Panangaden, and Doina Precup. Bisimulation metrics for continuous markov decision processes. SIAM Journal on Computing, 1666):1662-17144 2011.
Chelsea Finn, Xin Yu Tan, Yan Duan, Trevor Darrell, Sergey Levine, and Pieter Abbeel. Deep spatial autoencoders for visuomotor learning. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 512-519. IEEE, 2016.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. International Conference on Machine Learning, volume 80, pp. 1587- 1596. PMLR, 2018.
Scott Fujimoto. David Meger, and Doina Precup. An equivalence between loss functions and non- uniform sampling in experience replay. Advances in Neural Information Processing Systems, 33, 2020.
Scott Fujimoto, David Meger, and Doina Precup. A deep reinforcement learning approach to marginalized importance sampling with the successor representation. Proceedings of the 38th International Conference Machine Learning, volume 139, pp.
PMLR, 2021.
Scott Fujimoto, David Meger, Doina Precup, Ofir Nachum, and Shixiang Shane Gu. Why should i trust you, bellman? The Bellman error is poor replacement for value error. In International Conference on Machine Learning, volume 162, pp. 6918-6943. PMLR, 2022.
Scott Fujimoto, Wei-Di Chang, Edward J. Smith, Shixiang Shane Gu, Doina Precup, and David Meger. For SALE: State-action representation learning for deep reinforcement learning. Thirty- seventh Conference on Neural Information Processing Systems, 2024.
Carles Gelada, Saurabh Kumar, Jacob Buckman, Olir Nachum, and Mare Bellemare. Deepmdp: Learning continuous latent space models for representation learning. In International Conference on Machine Learning, pp. 2170-2179. PMLR, 2019.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
dean-Bastin Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Beraaro Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21284 2020.
12

---

Published as a conference paper at ICLR 2025
Zhaohan Daniel Guo, Bernardo Avila Pires, Bilal Piot, Munos, and Mohammad Gheshlaghi Azar. Bootstrap latent mhent-predictive representations for multi- task reinforcement learning. In International Conference on Machine Learning, pp. 3875-3886. PMLR, 2020.
Grill, Florent Altché, Rémi
Zhaohan Daniel Guo, Shantanu Thakoor, Miruna Pislar, Bernardo Avila Pires, Florent Altché, Corentin Tallec, Alaa Saade, Daniele Calandriello, Jean-Bastien Grill, Yunhao Tang, al. Byol- explore: Exploration by bootstrapped prediction. Advances in neural information processing systems, 35:3 18:5-8188887 2022.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv: 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Confer- ence on Machine Learning, volume 80, pp. 1861-1870. PMLR, 2018.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International conference machine learning. pp. 2565-2565. PMLR, 2019.
Danijar Hafner, Jurgis Pasukonis. Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint wwXiv:2301.04110 2023.
Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpe2: Scalable, robust world models for contin- uous control. In The Twelfth International Conference on Learning Representations, 2024.
Nicklas A Hansen, Hao Su, and Xiaolong Wang. Temporal difference learning for model predictive control. In International Conference on Machine Learning, pp. 8387-8406. PMLR, 2022.
Charles Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array program- ming with NumPy. Nature, 585(7825):357-362 September 2020.
Ammar Haydari and Yasin Yılmaz. Deep reinforcement learning for intelligent transportation sys- tems: A survey. IEEE Transactions on Intelligent Transportation Systems, 23(1): 2020.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-second AAAI conference on artificial intelligence, 2018.
Shengyi Huang, Rousslan Fernand Julien Dossa, Antonin Raffin, Anssi Kanervisto, and Weixun Wang. The 37 implementation details of proximal policy optimization. In ICLR Blog Track, 2022.
Julian Ibarz, Jie Tan, Chelsea Finn, Mrinal Kalakrishnan, Peter Pastor, and Sergey Levine. How to train your robot with deep reinforcement learning: lessons we have learned. The International Journal of Robotics Research, 40(4-5):698-7211 2021.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In International Conference on Learning Representations, 2017.
Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on learning theory, pp. 2137-2143. PMLR, 2020.
Maximilian Karl, Maximilian Soelch, Justin Bayer, and Patrick van der Smagt. Deep variational bayes filters: Unsupervised learning of state space models from raw data. stat, 1050:3, 2017.
Elia Kaufmann, Leonard Bauersfeld, Antonio Loquercio, Matthias Müller, Vladlen Koltun, and Davide Scaramuzza. Champion-level drone racing using deep reinforcement learning. Nature, 620(7976):982-988 987, 2023.
13

---

Published as a conference paper at ICLR 2025
Kyungsoo Kim, Jeongsoo Ha, and Yusung Kim. Self seff-predictive dynamics for generalization of vision-based reinforcement learning. In IJCAI, pp. 3150-3156, 2022.
Arsenii Kuznetsov, Pavel Shvechikov, Alexander Grishin, and Dmitry Vetrov. Controlling overesti- mation bias with truncated mixture of continuous distributional quantile critics. In International Conference on Machine Learning, pp. 5556-5566. PMLR, 2020.
Alex X Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. Advances Neural Information Pro- cessing Systems, 33:741-752, 2020.
Nir Levine, Tom Zahavy, Daniel J Mankowitz, Aviv Tamar, and Shie Mannor. Shallow updates for deep reinforcement learning. Advances Neural Information Processing Systems, 30, 2017.
Timothy Lillicrap. Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: http::00900099990 02971, 2015.
Michael Littman and Richard Sutton. Predictive representations of state. Advances neural information processing systems, 14, 2001.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, In International Confer- ence on Learning Representations, 2019.
Ryan Lowe, Yi I Wu, Aviy Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi- agent actor-critic for mixed cooperative-compettiiy environments. Advances in neural informa- fion processing systems, 30, 2017.
Nguyen Cong Luong, Dinh Thai Hoang, Shimin Gong, Dusit Niyato, Ping Wang, Ying-Chang Liang, and Dong In Kim. Applications of deep reinforcement learning in communications and networking: A survey. IEEE communications surveys & tutorials, 21(4):313333174, 2019.
Marlos C Machado, Mare G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-562, 2018.
Trevor McInroe, Lukas Schäfer, and Stefano V Albrecht. Learning temporally-consiitent represen- tations for data-efficient reinforcement learning. arXiv preprint urXiv:2110.049333 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 51 518(7540):529 533, 2015.
Jelle Munk, Jens Kober, and Robert Babuška. Learning state representation for deep actor-critic control. In 2016 IEEE 55th Conference Decision and Control (CDC), pp. 4667-4673. IEEE, 2016.
Tianwei Ni, Benjamin Eysenbach, Erfan Seyedsalehi, Michel Ma, Clement Gehring, Aditya Ma- hajan, and Pierre-Luc Bacon. Bridging state and history representations: Understanding self- predictive rl. In The Twelfth International Conference on Learning Representations, 2024.
Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L Lewis, and Satinder Singh. aation-conditional video prediction using deep networks in atari games. Advances in neural information processing systems, 28, 2015.
Ronald Parr, Lihong Li, Gavin Taylor, Christopher Painter-Waid Wakefield, and Michael L Littman. An analysis of linear models, linear value-function approximation, and feature selection for rein- forcement learning. In Proceedings of the 25th international conference on Machine learning, pp. 752-759, 2008.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems, pp- 8024-8035, 2019.
14

---

Published as a conference paper at ICLR 2025
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dor- mann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268): 2021. URL http://jnmll.or///pppss///// httprappprra22222003133h0..aaa 1364.html.
Balaraman Ravindran. An algebraic approach to abstraction in reinforcement learning. University of Massachusetts Amherst, 2004.
Balaraman Ravindran and Andrew G Barto. Model minimization in hierarchical reinforcement learning. In Abstraction, Reformulation, and Approximation: 5th International Symposium, SARA 2002 Kananaskis, Alberta, Canada August 2-4, 2002 Proceedings 5, pp. 196-211. Springer, 2002.
Ingo Rechenberg. Exolutionsstratteiien. In Simulationsmetthdden in der Medizin und Biologie: Workshop, Hannover, 29. Sept.-l. Oki. 1977, pp. 83-114. Springer, 1978.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gómez Colmenarejo, Alexander Novikov, Gabriel Barth-maron, Mai Giménez, Yury Sulsky, Jackie Kay, Jost Tobias Springenherg, Tom Ec- cles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. Transactions on Machine Learning Research, 2022. ISSN 2835-8856.
Tongzheng Ren, Tianjun Zhang, Csaba Szepesvári, and Bo Dai. A free lunch from the noise: Prov- able and practical exploration for representation learning. In Uncertainty in Artificial Intelligence, pp. 1686-1696. PMLR, 2022.
Tongzheng Ren, Chenjun Xiao, Tianjun Zhang, Na Li, Zhaoran Wang, Dale Schuurmans, Bo Dai, et Latent variable representation for reinforcement learning. In The Eleventh International Conference on Learning Representutions, 2023.
Sahand Rezaei-Shoshtari Rosie Zhao, Prakash Panangaden, David Meger, and Doina Precup. Con- tinuous mdp homomorphisms and homomorphic policy gradient. In Advances in Neural Infor- mation Processing Systems, 2022.
Reuven Y Rubinstein. Optimization of computer simulation models with rare events. European Journal of Operational Research, 99(1)89-1122 1997.
Gavin A Rummery and Mahesan Niranjan. On-line Q-learning using connectionist systems, vol- ume 37, University of Cambridge, Department of Engineering Cambridge, UK, 1994.
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies scalable alternative to reinforcement learning. arXiv preprint arXiv: 2017.
Aidan Scannell, Kalle Kujanpää, Yi Zhao, Mohammadreza Nakhaei, Amo Solin, and Joni Pajari- nen. iqrl-implicitly quantized representations for sample-efficient reinforcement learning. arXiv preprint arXiv:2406.0266 2024.
Tom Schaul, John Quan, loannis Antonoglou, and David Silver. Prioritized experience replay. In International Conference Learning Representations, Puerto Rico, 2016.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604-699 2020.
Julian Schrittwieser, Thomas Hubert, Amol Mandhane, Mohammadamin Barckatain, Ioannis Antonoglou, and David Silver. Online and offline reinforcement learning by planning with a learned model. Advances in Neural Information Processing Systems, 34:27580-27591 2021.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, pp. 1889-1897, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017.
15

---

Published as a conference paper at ICLR 2025
Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bach- man. Data-efficient reinforcement learning with self- representations. In International Conference on Learning Representations, 2020.
Max Schwarzer. Johan Samir Obando Ceron, Aaron Courville, Mare G Bellemare, Rishabh Agar- wal, and Pablo Samuel Castro. Bigger, better, faster: Human-level atari with human-level effi- ciency. In International Conference on Machine Learning, pp. 30365-30380. PMLR, 2023.
Younggyo Seo, Kimin Lee, Stephen L James, and Pieter Abbeel. Reinforcement learning with action-frre pre-training from videos. In International Conference on Machine Learning, pp- 19561-19579. PMLR, 2022.
Dmitry Shribak, Chen-Xiao Gao, Yitong Li, Chenjun Xiao, and Bo Dai. Diffusion spectral represen- tation for reinforcement learning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International Conference on Machine Learning, pp- 387-395, 2014.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484-88,, 2016.
Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evei. The dormant neuron phe- nomenon in deep reinforcement learning. In International Conference Machine Learning, pp. 32145-32168. PMLR, 2023.
Zhao Song, Ronald E Parr, Xuejun Liao, and Lawrence Carin. Linear feature encoding for rein- forcement learning. Advances in neural information processing systems, 29, 2016.
Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, (1):9-44, 1988.
Richard Sutton and Andrew G Barto. Reinforcement Learning: An Introduction, volume 1. MIT press Cambridge, 1998.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
Yunhao Tang, Zhaohan Daniel Guo, Pierre Harvey Richemond, Bernardo Avila Pires, Yash Chan- dak, Rémi Munos, Mark Rowland, Mohammad Gheshlaghi Azar, Charline Le Lan, Clare Lyle, et al. Understanding self-predictive learning for reinforcement learning. In International Confer- on Machine Learning, pp. 33632-33656. PMLR, 2023.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, al. Deepmind control suite. arXiv preprint 180i 000900 2018.
Emanuel Todoroy, Tom Erez, and Yuval Tassa. Mujaco: A physics engine for model-based control. In IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 5026-5033. IEEE, 2012.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307. 092888 2023.
Mark Towers, Ariel Kwiatkowski, Jordan Temr, John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: standard interface for reinforcement learning environments. arXiv preprint 240.::440.777033 2024.
16

---

Published as a conference paper at ICLR 2025
Elise van der Pol, Thomas Kipf, Frans A Olichoek, and Max Welling. Plannable approximations to mdp homomorphisms: Equivariance under actions. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, pp. 1431-1439, 2020a.
Elise van der Pol, Daniel Worrall, Herke van Hoof, Frans Oliehoek, and Max Welling, Mdp homo- morphic networks: Group symmetries in reinforcement learning. Advances Neural Information Processing Systems, 33:4199 4210, 2020b.
Herke Van Hoof, Nutan Chen, Maximilian Karl, Patrick van der Smagt, and Jan Peters. Stable rein- forcement learning with autoencoders for tactile and visual data. In 2016 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 3928-3934. IEEE, 2016.
Guido Van Rossum and Fred I. Drake Jr. Python tutorial. Centrum voor Wiskunde en Informatica Amsterdam, The Netherlands, 1995.
Shengjie Wang, Shaohuai Liu, Weirui Ye, Jiacheng You, and Yang Gao. Efficientzero v2: Mastering discrete and continuous control with limited data. arXiv preprint arXiv:2403.00564, 2024.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Mare Lanctot, and Nando Freitas. Ducling network architectures for deep reinforcement learning. In International Conference on Machine Learning. pp. 1995-2003 2016.
Christopher John Cornish Hellaby Watkins. Learning from delayed rewards. PhD thesis, King's College, Cambridge, 1989.
Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. Advances in neural information processing systems, 28, 2015.
Ronald J Williams.
Simple statistical frolient- algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-25 256, 1992.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous con- trol: Improved data-augmented reinforcement learning. In International Conference on Learning Representations, 2022.
Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, and Yang Gao. Mastering atari games with limited data. Advances in neural information processing systems, 34:25476-25488, 2021.
Tao Yu, Cuiling Lan, Wenjun Zeng, Mingxiao Feng, Zhizheng Zhang, and Zhibo Chen. Playvirtual: Augmenting cycle-consistent virtual trajectories for reinforcement learning. Advances Neural Information Processing Systems, 34:5276-5288 2021.
Tao Yu, Zhizheng Zhang, Cuiling Lan, Yan Lu, and Zhibo Chen. Mask-based latent reconstruc- tion for reinforcement learning. Advances in Neural Information Processing Systems, 35:25117- 25131, 2022.
Hongyu Zang, Xin Li, and Mingzhong Wang, Simsr: Simple distance- state representations for deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8997-9005, 2022.
Amy Zhang, Harsh Satija, and Joelle Pineau. Decoupling dynamics and reward for transfer learning. arXiv preprint arxiv: 10689, 2018
Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learn- ing invariant representations for reinforcement learning without reconstruction. In International Conference on Learning Representations, 2020.
Tianjun Zhang, Tongzheng Ren, Mengjiao Yang, Joseph Gonzalez, Dale Schuurmans, and Bo Dai. Making linear mdps practical via contrastive representation learning. In International Conference on Machine Learning, pp. 26447-26466 PMLR, 2022.
Yi Zhao, Wenshuai Zhao, Rinu Boney, Juho Kannala. and Joni Pajarinen. Simplified temporal con- sistency reinforcement learning. In International Conference on Machine Learning, pp. 42227- 42246. PMLR, 2023.
17

---

Published as a conference paper at ICLR 2025
Ruijie Zheng, Xiyao Wang, Yanchao Sun, Shuang Ma, Jieyu Zhao, Huazhe Xu, Hal Daumé III, and Furong Huang. Taco: Temporal latent action-driven contrastive loss for visual reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Luisa Zintgraf, Sebastian Schulze, Cong Lu, Leo Feng, Maximilian Igl, Kyriacos Shiarlis, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: Variational anyes-adaptive deep rl via meta-learning. Journal of Machine Learning Research, 22(289):1-39, 2021.
18

---

Published as a conference paper at ICLR 2025
Appendix
Proofs
19
B Experimental Details
23
B.1 Hyperparamelers
23
B.2 Network Architecture
24
B.3 Environments
26
B.4 Basclines
28
B.5 Software Versions
28
C Complete Main Results
29
C.I Gym
29
C.2 DMC Proprioceptive
30
C33 DMC Visual
32
C.4 Atari
34
Complete Ablation Results
36
D.I Gym
36
D.2 DMC Proprioceptive
37
D.3 DMC- Visual
39
D.4 Atari
41
A PROOFS
Theorem 1. The fixed point of the model-free approach (Equation 4) and the solution of the model- based approach (Equation 5) are the same.
Proof. Let be a matrix containing state-action embeddings 2na for each state-action pair (s, a) Sx A. Let z' be the corresponding matrix of next state-action embeddings z Let R be the vector of the corresponding rewards r(s, a.).
The linear semi-gradient TD update:
W Wt aZ (Zw: (R+ wi))
(21)
- Zw, aZ R an Z
(22)
= (I-a(ZZ' Z -2 Z)) w, + aZ R
(23)
(1 (T-aA)we aB,
(24)
where := - and B = ZTR.
The fixed point of the system:
Wmf (I - (a-a))mmf Wmf + aB
(25)
W mf (I aA) w mf aB
(26)
a'Amm aB
(27)
Wmf A B.
(28)
19

---

Published as a conference paper at ICLR 2025
The least squares solution to Wp and w,
w= (2 z
(29)
W, = (2= 2
(30)
By rolling out W and wr, we arrive at model-based solution:
00 Q == W = z Σ
7 wr.
(31)
t=
Simplify Wmb:
Wbb- Σ WW wr
(32)
Wmb (T -1, Wr
(33)
Wmb = (1- zzZ) (z²Z)" 2J
(34)
(T ) Wmb R
(35)
(2 (z'2-70'2')
Wmhh R
(36)
Wmb A B
(37)
Wmb Wmf-
(38)
Theorem 2. The value error of the solution described by Theorem I is bounded by the accuracy of the estimated dynamics and reward:
(RZu., w,, Eej--E [rjss[[[]]] max W: Σ 1 Wp - (l (39)
max [VE(s, S 1 1-7 (s.a)ESKA
Proof. Let w be the solution described Theorem 1, i.c. w = Wmb = Wmf- Let pT a) be the discounted state-action visitation distribution according the policy ı starting from the state-action pair (s,a).
Firstly from Theorem 1, can show that
w = (1-7Ww) (T WW>>W)) WW) W w, w - WWW, w- W,.
(40)
(41)
(42)
Simplify VE(s.a):
vB(s, Q(s, ("
(43)
a(x,a) (s,
(44)
= a) Ey,o,, [r + (s'(''
(45)
= aQ(x,) En,y,a
[r + 7 (Q(s', a') - VE(s', a'))]
(46)
= a(x,,) Engy,
r + 7(s
TEE a (VE(s',a')
(47)
a(x,a) En
- 2 W, I wr 2 a')] aa [VE(s',a')
(48)
=Q(,, Ey
[ z Wr z W. 7 22//a/ w zJa W,w w + z Wyw)
(49)
+TE a [VB(s', "''""a") .a)]
= II1 w Ey,s' 2 w, + Jn w, + (2),2r w 2/ Wp + 2 Wyw))
(50)
7E of (VE(s'.'')
20

---

Published as a conference paper at ICLR 2025
x w E, r- Ena W, 2: 2., -Tlbss,a (2.'a W 2 Wpw + 2 Wn
(51)
+ +nllEss [VE(s', .a))
x 20 w 20 Wyw -.w-eyp-
Er [r- 2:0 wr. -rilBsr,a [Z W 2 W [w]
(52)
t (E((',a')
zaa (w W.w. wr) -Er 2nn Wr wr]
y'lBss far (3 2050 Wow)
(53)
aa [VB(s', ,d)]
= =- Er 2ea wr] -yBer,a (3 w - zz Wn Npw/ Ba [VE(s',a')]
(54)
(z w. - Er [r]] 7 (zia 1e [2 1) w y [VB(s', ,')].
(55)
Then given the recursive relationship, akin to the Bellman equation (Sutton & Barto, 1998), the value error VE recursively expands the discounted state-action visitation distribution p². For (s, e Sxxxx A:
VE(s, à)
1-7 ((aa))"" (8,a) [zzaa,, w, Egja,a [r]) + (z!a Wo Ev,a (2)(er [) (56)
Taking the absolute value:
[VB(,,a) =
E (saa)pp (5,a) [Z wr - Brisoo [r]] + (z W Es aa.a (2 w
(57)
1
[VB(S,a)
E (ss.a))p (8,0) [ w, E,jo,a [r]] 7 ( W, Bg [2 w|
(58)
1
=
max
(1 w, -FBytaa [r] + 7 ( Wp - -Eyr,ala [2]rar] w/)
(59)
1 - y (s,0)eSxA
1
125ow,, Erls [r] max w: El W,, EE aa [Zoo']] (60)
max
1 (a.0)SxA
Theorem 3. Given functions f(s) us and g(z.: = Zsa then if there exists functions P and R such that for all (s E A:
E Egli(zza) = ER [R(s,a)),
p(zx\Zoo)= = Σ p(5)s.a):
(61)
then for any policy π where there exists a corresponding policy ((clz.) = (als). there exists a function Q equal to the true value function Q over all possible state-action pairs (s, a) E S x A:
(Zzza)== (s.
(62)
Furthermore, Equation 61 guarantees the existence of optimal policy 4" (alzz)) x² (als).
Proof. Let
h Σ n [R(s.. 8,00 = a]
(T,
(63)
t=0
(zsa)) Σ
7 EEz [R( 0, =.00==a a)
(64)
1-0
Then
(ī ER(R((,a))
(65)
= =E R(( (z
(66)
Qu(aa))
(67)
21

---

Published as a conference paper at ICLR 2025
Assuming Q7-1(s,a) = Qn-a then noting that p(z|zzo) if z that is not in the image of f(s)= = Zz.
(x(5,0) =x
(68)
00 (s) (Zza.)
(69)
EE((((,,a]]+ ΣΣ 1(
(70)
ΣΣ plz 2sa (n-) 2
(71)
(Zaa)
(72)
Thus ((zxa) = lim, 0a Qu (2sa) exists, as Qn can be defined as function of p. R. and it for all n.
Similarly, let ı be an optimal policy. Repeating the same arguments we see that
(x) [E([((s a)] yR aa (Q
(73)
=RR[R(s.a)) a)] 7 Σ ((')ss,))) a) max Q (5',a') =EA(((s, + Ei(er max Qn- (Z
(74)
(75)
Qu
(76)
Thus there exists function (u((z,))) Q"" (8, a). consequently, there exists an optimal policy 4* (alz,) argmax, as)
22

---

Published as a conference paper at ICLR 2025
B EXPERIMENTAL DETAILS
B.1 HYPERPARAMETERS
Table 3: MR.Q Hyperparameters. Hyperparameters values kept fixed across all benchmarks.
Hyperparameter
Value
Dynamics loss weight ADynamies
Reward loss weight AReward
0.1
Terminal loss weight ATeminal
0.1
Pre-activation loss weight Agee-activ
le
Encoder horizon Hrse
Multi-step returns horizon HQ
TD3
Target policy noise a
(.200.22)
(Fujimoto et al., 2018)
Target policy noise clipping
(-0.3,0.3)
LAP
Probability smoothing a
0.4
(Fujimoto et al., 2020)
Minimum priority
Initial random exploration time steps
10k
Exploration
Exploration noise
0.2.)
Discount factor y
0.99
Replay buffer capacity
1M
Common
Mini-batch size
256
Target update frequency Turget
250
Replay ratio
Optimizer
AdamW (Loshchilov & Hutter, 2019)
Learning rate
1e-4 4
Weight decay
1e-4
2, dim
512
Z.n dim
512
Za dim (only used within architecture)
256
Encoder Network
Hidden dim
512
Activation function
ELU (Clevert al., 2015)
Weight initialization
Xavier uniform (Glorot & Bengio, 2010)
Bias initialization
0
Reward hins
65
Reward range
[-10, 10] (effective: [-22k,220 22k])
Optimizer
AdamW
Learning
3e-4
Hidden dim
512
Value Network
Activation function
ELU
Weight initialization
Xavier uniform
Bias initialization
Gradient clip norm
Optimizer
AdamW
Learning rate
3e
Hidden dim
512
Policy Network
Activation function
ReLU
Weight initialization
Xavier uniform
Bias initialization
Gumbel-Softmad 7 (Jang et al., 2017)
23

---

Published as a conference paper at ICLR 2025
B.2 NETWORK ARCHITECTURE
This section describes the networks used in our method using PyTTorch code blocks (Paszke et al., 2019). The state encoder and state-action encoder described separate networks for clarity but are trained end-to-end as single network. The value and policy networks are trained independently from the encoders.
Preamble
1 import torch
2 import torch.nn as
3 import torch. nn. furctional as
4 from functools import partial
5
dim 512
za_dim 256
8 ism__dmm 512
9
10
def in_activ(selE, x) F. norm (x, (x.shape(-11,)
11
12
return self.activ(x)
State Encoder / Network
For image inputs, four convolutional layers used, each with 32 output channels, kernel size of 3, strides of 2, 1), and ELU activations (Clevert et al., 2015). The convolutional layers are followed by linear layer taking in the flattened output followed by LayerNorm (Ba al., 2016) and final ELU activation.
For vector inputs, three layer multilayer perceptron (MLP) is used, with hidden dimension 512 and LayerNorm followed by ELU activations after each layer.
The resulting state embedding z, is trained end-to-end with the state-action encoder, also used downstream by the policy network (without propagating gradients).
1 if Lmage_observation__gg
sclnnzs_enni nn.Conv2d (state
(stattcchhccnccnnnee,,,e channels, 32, 3, stride=2) self.zs_cnnz nn.Conv2d (32, 32, 3, stride=2)
3
4 self.za_cnn3 nn..onv2d (32, 32, 3, stride-2)
5 self.zs_cnnd nn.Conv2d (32, 32, 3, stride-1) 84 input
Assumes 84
self.zs_lin Lineir (1568, di_i
8
else:
9
self.za_mpp Ln.Linear (state_di) __dm_,__m_____ 512)
10
self.zs_mlp2 Ln. (512, 512)
self.za_mlpp Lineir (512, dimd
12
13 self. F.elu
14
15 def con_forward(self, state):
state state/255, 0.5
17
25 salf.actti/see1f.esccmmm (state))
18
23 self.acttivseeff.aacomnt (28))
19
self.activv/eel..raccmm3/// (28))
20
self.accivveeef..2__cmml (28))
21
zs as.restape 1568)
22
return in an lin
23
24
def mip_forwarddself, state)
25
28 self. a (statee) (state))
26
25 selI:._n_cciiveee..........
27
return alf.iir_attivee (self. m2p_22__mmp33pmmm22p2__ (23))
aelf.
24

---

Published as a conference paper at ICLR 2025
State-Action Encoder g Network
Action input is processed by a linear layer followed by am ELU activation. Afterwards, the processed action is concatenated with the state embedding and processed by three layer MLP with hidden dimension 512, and LayerNorm followed by ELU activations after the first two layers.
The resulting atate-action embedding 2 is used by linear layer to make predictions about reward, the next state embedding, and the terminal signal. It is also used downstream by the value network (without propagating gradients).
1 self. nn Linear (action dim, za dim)
2 self.28 zaal on.Linear (2a_dim + za_dim, 512)
3 self.zsa2 = nn.Lirear (512, 512)
4 self.zsa3 nn. (512, zsa_dim)
self.model nn. (zsa_ _im, output_dim)
self.activ F.elu
7
8 def seeard(ssel, 2s, action)
9
aclv (self. (action))
zsa torch.cat (1zs, zal, 1)
11
zaa
fsel (zsa)) (2sa))
12
28a self. (ssa)
13
14
return self.model (zsa),
Value Q Networks
The value network is four layer MLP with hidden dimension 512, and LayerNorm followed by ELU activations after the first three layers.
Two value networks are used with the same network and forward pass.
self. 1.1 nn. Linear (zsa_dim, 512)
2 sel f.12 nn. Lineir (512, 512)
3 self.13 on. (512, 512)
4 self.14 on. Linear (512, 1)
self.activ F.elu
7 def (self,, 2sa)
8
= htt..=iicctiis self..n_activiself...
9
10
self. self..n__ctivvseef.. (q))
return self.14
Policy Network
The policy network is three layer MIP with hidden dimension 512, and LayerNorm followed by ReLU activations after the first two layers.
For discrete actions, the final activation is the Gumbel Sofumax with 10. For continous actions, the final activation is tanh function.
1 self.l1 nn Linear dim, 512)
Linear (hdim, 512) 3 self.13 nn. (iinear(512 action_dim)
2 self.12
4 self.activ F.relu
5
6 if self.tinal partial (E (aaiiaall..gguaaa... gumbel_ tau=10)
8 else:
9
self.finalaactivv - torch.tanh
10
11
def fozward zs):
12
self. activiself a (zs))
13
self.in Sollf.in_ctiv(seeli wwt.vcself..11eeccc..
14
return self. self. (3
25

---

Published as a conference paper at ICLR 2025
B.3 ENVIRONMENTS
All main experiments were run for 10 seeds (the design study is based on 5 seeds). Evaluations are based on the average performance over 10 episodes, measured every 5k time steps for Gym and DM control and every 100k time steps for Atari.
Gym Locomotion. For the gym locomotion tasks (Todorov et al., 2012; Brockman et al., 2016; Towers et al., 2024), we choose the five most common environments that appear in prior work (Fu- jimoto et al., 2018; 2024; Haarnoja al., 2018; Kuznetsov al., 2020). We use the -v4 version. No preprocessing is applied. When aggregating scores, we use normalize with the TD3 scores obtained from TD7 (Fujimoto et al., 2024):
T random score
TD3-Normalize(()) := TD3 score random score
(77)
Random
TD3
Ant-v4
-70.288
3942
HalfCheetah-v4
-289.415
10574
Hopper-v4
18.791
3226
Humanoid-vá
120.423
5165
Walker2d-v4
2.791
3946
DM Control Suite. For the DM control suite (Tassa et al., 2018), we choose the 28 default en- vironments that appear either in the evaluation of TD-MPC2 or DreamerV3. We omit any custom environments included by the TD-MPC2 authors. The same subset of tasks are used in the evalua- tion of proprioceptive and visual control. Like prior work, for both observation spaces, we use an action repeat of 2 (Hansen et al., 2024). For visual control, the state (network input) is composed of the previous? 3 observations which are resized to 84 x 84 pixels in RGB format (Tassa et al., 2018).
Atari. For the Atari games (Bellemare al., 2013; Brockman et al., 2016; Towers et al., 2024), we use the games in the Atari-57 benchmark that appears in prior work (Hessel et al., 2018:
Schrittwieser et al., 2020; Badia al., 2020; Hafner et al., 2023). For DQN and Rainbow, two games (Defender and Surround) are missing from the Dopamine framework (Castro et al., 2018)
and are omitted. We use the -v5 version. For MR.Q, we the common preprocessing steps (Mnih et al., 2015; Machado et al., 2018; 8; Castro et al., 2018), where an action repeat of 4 is used and the observations grayscaled, resized 84 84 pixels and set to the max between the 3rd and 4th frame. The state (network input) is composed of the previous 4 observations.
Consider the 16 frame sequence used by single state, where fi is the ith grayscaled and resized frame and Oj is the jth observation set to the max of two frames
d1
a5
for fI,
f2. Sa. fo. (f.
fo.
fs.
f1o. fn f12. fis. fus.fs
(78)
o1=max(Jofh
og=max( f10.f11)
og-max(f14.fi)
then the state is defined follows:
Og max f2.
max [8, S7)
(79)
02 max (J10, Si))
Log max(f1s, f1s)_
When aggregating scores, we normalize with Human scores obtained from (Wang et al., 2016):
random score
Humman-Normalizd((a) Human score random score
(80)
26

---

Published as a conference paper at ICLR 2025
Random
Human
Alien
227.8
7127.7
Amidar
5.8
1719.5
Assault
222.4
742.0
Asterix
210.0
8503.3
Asteroids
719.1
47388.7
Atlantis
12850.0
29028.
BankHeist
14.2
753.1
BattleZone
2360.0
37187.5
BeamRider
363.9
16926.5
Berzerk
123.7
2630.4
Bowling
23.1
160.7
Boxing
0.1
12.1
Breakout
1.7
30.5
Centipede
2090.9
12017.0
ChopperCommand
811.0
7387.8
ClizbCreber
10780.5
35829.4
Defender (not used)
2874.5
18688.9
DemonAttack
152.1
1971.0
DoubleDunk
18.6
-16.4
Enduro
0.0
860.5
FishingDerby
99.77
-38.7
Freeway
0.0
29.6
Frostbite
65.2
4334.7
Gopher
257.6
2412.5
Gravitar
173.0
3351.4
Hero
1027.0
30826.4
IceHockey
-11.2
0.9
Jamesbond
29.0
302.8
Kangaroo
52.0
3035.0
Krull
1598.0
2665.5
KungFuMaster
258.5
22736.3
MontezumaRevenge
0.0
4753.3
MsPacman
307.3
6951.6
Name NameThisGame ThisGame
2292.3
8049.0
Phoenix
761.4
7242.6
Pitfall
-229.4
6463.7
Pong
-20.7
14.6
PrivateEye
24.9
69571.3
Qbert
163.9
13455.0
Riverraid
1338.5
17118.0
RoadRunner
7845.0
Robotank
2.2
11.9
Seaquest
68.4
42054.7
Skiing
-17098.1 1
-4336.9
Solaris
1236.3
12326.7
SpaceInvaders
148.0
1668.7
StarGunner
664.0
10250.0
Surround (not used)
-10.0
6.5
Tennis
-23.8
-8.3
TimePilot
3568.0
$229.2
Tutankham
11.4
167.6
UpNDown
533.4
11693.2
Venture
0.0
1187.5
VideoPinball
16256.9
17667.9
WizardOrWor
563.5
4756.5
YarsRevenge
3092.9
54576.9
Zaxxon
32.5
9173.3
27

---

Published as a conference paper at ICLR 2025
B.4 BASELINES
D3 (Hafner et al., 2023). Results for Gym and DMC were obtained by re-running the au- thors' code (https:/github.com//fte htt.bb/oo/oaaajaarrreeesss
Commit 251910d04cs 10110000A00dd9993377775770111111 efa0e57a95) over 10 seeds, using the author-suggested hyperparameterss from the DMC benchmark. Code was modified slightly match our evaluation protocol. Atari results are based on the authors' reported results.
D2Q (Yarats al., 2022). We use the authors' reported results whenever possible. For miss- Commit
ing any results, we re-ran the authors' code (e6650076c665dd222765522eee
440fe9488ef) for 10 seeds.
DQN. (Mnih et al., 2015). Results were obtained from the Dopamine framework (Castro et al., 2018).
PPO. (Schulman et al., 2017). Results were gathered using Stable Baselines 3 (Raffin et al., 2021) and default hyperparameters. The default MLP policy was used for Gym and PMCoproprioeeeptiv and the default CNN policy was used for DMC-visual and Atari.
Rainbow. (Hessel et al., 2018). Results were obtained from the Dopamine framework (Castro al., 2018).
TD-MPC2. (Hansen et al., 2024). Results for DMC were obtained by re-running Commit
the authors' code their main branch http://ggttuub.coommccllssssssst (fofadeoffec733044bb3388777777 (1053888177114488884444)6) As the Gym environments include a termina-
tion signal, results for Gym were obtained by rumning their episodic branch (https://githubb.cmm wwwllaaaannnnmmmmmet//////ssoooo
Commit 3789fcd5b872079add6 205.99000000020999443333427a77a0
All experiments run for 10 seeds and the default author-suggestel hyperparameters for all tasks.
TD7. (Fujimoto al., 2024). Results for Gym were obtained from the authors. Results for Commit clc280de15 33444488806 bbcc33944277ddd84bd using our setup for DMC. All experiments 10 seeds and use the default author-suggested hyperparameters from the Gym benchmark.
DMC were obtained by re-running the authors' code (htps://github.com// httphhuu...oo/uuii///17777
B.5 SOFTWARE VERSIONS
Gymnasium 0.29.1 (Towers et al., 2024)
MuJoCo 3.2.2 (Todorov et al., 2012)
NumPy 2.1.1 (Harris et al., 2020)
Python 3.8 (Van Rossum & Drake Jr, 1995)
PyTorch 2.4.1 (Paszke et al., 2019)
28

---

Published as a conference paper at ICLR 2025
C COMPLETE MAIN RESULTS
C.I
GYM
Table 4: Gym Lecomotion final results. Final average performance at 1M time steps 10 seeds. The [bracketed values] represent a 95% bootstrap confidence interval. The aggregate mean, median and interquartile mean (IQM) are computed over the TD3- normmlaal score (sce Appendix B.3).
Task
TD7
PPO
TD-MPC2
DreamerV3
MR.Q
Ant
8509 [8164, 8852)
1584 [1355, 1802]
4751 [3012, 6261]
1947 [1121, 2751]
6901 74821
HallCheetah
17433 (17284, 17550]
1744 [1525, 2120]
15078 (14050, 16012]
5502 [3887, 71171
12939 [11663, 137621
Hopper
3511 [3245, 3746]
3022 (2587, 3356)
2081 [1233, 2916]
2666 (2071. 3201]
2692 [2131. 3509]
Humanoid
7428 (7300, 7555]
477 (22)
6071 [5767, 6327]
4217 (2811
10223 [9929, 10498]
Walker2d
(5535, 6521)
(1875, 3067]
(1659, 4220]
(3746, 5190)
[5644, 6386]
Mean
1.57 11.54, 1.60]
0.45 [0:41, 0.481
1.04 10,90, 1.16]
0.76 [0.67, 0.851
1.46 [1.41. 1.52]
Median
1.55 [1,45, 1,63]
0.41 (0.36, 0.47)
1.18 (0.80, 1.23]
0.81 [0.56, 0.90]
1.53 [1.43, 161]
IQM
1.54 [1.49, 1.58]
0.41 (0.35, 0.46]
1.05 [0.87, 1.19]
0.72 [0.62, 0.85]
1.50 [1.44, 1,55]
HallCheetah
Hopper
Humanokd
10k
17.5k
3.5k
10k
Toal Reward
6k
12.5k
2.5k
6k
7.5k
2k
2.5k
500
-24
0.0
0.2
0.4
10.
0.8
1.0
-2.5%
0.0
0.2
0.4
0.6
0.8
1.0
-500
0.0
0.2
0.4
0.6
0.8
1.0
-24
0.0
0.2
0.4
0.6
0.8
1.0
Timesteps (IM)
Timestops
Timestops (1M)
Timesteps (IM)
Walker2d
Total Reward
3k
1k
-1k 8.0
Timestops (IM)
MR.Q
D3eamerV3
! TD-MPC2
PPO
TD7
Figure 3: Gym- Locomotion learning curves. Results are over 10 seeds. The shaded area captures 95% boostrap confidence interval.
29

---

Published as a conference paper at ICLR 2025
C.2 DMC PROPRIOCEPTIVE
Table DMC- Proprioceptive final results. Final average performance 500k time steps (IM time steps in the original environment due to action repeat) over 10 seeds. The [bracketed values] represent a 95% bootstrap confidence interval. The aggregate mean, median and interquartile mean (IQM) are computed over the default reward.
TD-MPC2
DreamerV33
MR.Q
swingu
58 [38, 75]
[33, 45]
584 [55L, 615]
230 [193, 266]
567 [523, 616]
hall_in_cun hall_in_cup-cutch cutch
983 [981, 985]
769 [689, 841]
984 [982. 986]
968 [965, 973]
981 [979. 984]
cartpole-balance
999 [998, 1000]
999 [1000, 1000]
996 [995, 998]
998 1997. 1000]
909 [999. 1000]
cartoll--alaanee. sparse
1000 [1000, 1000]
[1000, 1000)
1000 [1000, 1000]
999
1000]
1000
1000]
cartpole-swingup
869 [866, 873]
776 [661, 853]
875 [870. 880]
736 [591, 838]
866 [866, 866]
carpoll-swinguppssarr
573 [333, 806]
391 [159, 625]
845 (839, 849)
702 [560, 792]
798 [780, 818)
cheetab-run
821 [642, 913]
269 [247. 295]
917 1915, 920
699 [655, 744
914 [911. 9171
dog run
69 [36, 1D1]
(26, 28]
265 [166, 342]
[4, 5]
569 [547. 595]
dog-stand
582 [432, 741]
129 [122,
506 (266, 715]
22 27]
967 [9600] 975]
dog-trot
21 [13, 30]
31 [30, 34]
407 [265, 530)
10 [6, 17]
877 [845, 898)
dog-walk
116]
[37,
(240,
211
(908, 924]
Jinger spin
335 (99, 596]
459 (420, 497]
986 (986,, 988
666 [577, 763
937 [917. 956]
dinger-tum.casy
912 1774, 983]
182 1153, 2111
979 (975, 983]
906 1883, 9271
953 [931, 974]
linger hard
470 1199. 727
[35, 791 103 (84, 128]
9477 (916, 9771
864 [812, 9001
950 [910, 9741
fish-swim
86 [64, 120]
659 [615, 706]
813 [808, 819]
792 [773, 810)
hopper-hop
87 [25, 160]
425 [368,, 500)
165]
251 [195, 301]
bopper-stand
670 [466, 829]
128 (56, 216]
952 (9444, 958]
747 [669. 806]
951 [948, 955]
1121,
11.
(170, 236
humanoid stand
317 [117, 516]
$ 65
658 [506, 745]
[5,
868 [822, 903]
humancid-wwe walk
176 [42, 320]
111.21
754 1725, 7911
21
662 [610, 7241
pendulum swingup
500 [251. 743]
115 170, 164]
846 [830, 862]
774 [740, 802]
748 (597. 829]
quadruped-run
645 [567, 713]
144 [122, 170]
942 [933,, 947]
130 [92, 169]
947 [940, 954]
quadruped-wakk
[939, 957]
122 [103,
963 (959,
193 [137, 245)
[959, 967]
reacher-easy
970 [951, 982]
367 1188, 558]
983 1980, 986
966 1964, 970]
983 1983, 9851
reacher hard
[861, 936]
125 (40, 234]
960 (936,
1864, 955]
1975, 980]
walker run
804 [783, 825]
97 [91, 104]
854 [851, 859]
$10 [4300 588)
793 [765, 815)
walker stand
983 1974, 9891
431 1363, 4951
991 1990, 994]
941 [934, 948
988 1987. 990]
walker walk
977 [975, 980]
283 (253, 312]
981 [979,
898 [875,
(978, 980]
Mean
[544, 590] 613 [548, 718]
254
(769,
[520, 5391
[829, 8421
Median
127 [112, 145]
896 [893, 899)
700 [644, 741]
927 [914, 934]
JQM
612 [569, 657]
154 (135, 167]
868 [860, 880]
577 [557, 594]
[903, 914]
30

---

Published as a conference paper at ICLR 2025
scrobok-swingup
ball cuccuhhctchh
cortpole-bulance
cartpole-bllance www.-baaaaneeeeeeeeeeeeeeeeeee -pareseeeeeeeeeeeeeeeeeeeeeeeee
Total Reward
600
600
600
300
400
400
400
100
100
swinpule
swiipule wpprrrg.ppppaseeeeeee..... sparse
cheetah-rut run
dog run
1k
Total Reward
500
600
600
300
-00
dog
dog-trot
dog-walk
finger spin
1k
1k
1k
11
Tocal Reward
800
800
600
600
600
400
200
200
200
200
0
200
-200
Tiner-tum_eeag
finger Jurn_hard
fisb-swim
hopper-hop
800
Total Reward
800
800
600
000
300
400
400
1100
hopper-stand
bumanoid-run
humamoid-staad
humanoid-waad
250
1k
Total Reward
600
400
200
-200
pandulum-swingup
quadruped-run
quadruped-walkk
reacher-easy
1k
1k
1k
1k
Total Rewaad
800
800
600
600
200
200
200
200
-200
200
reacher-hard
walkerr-on
walker-stand
walker-walk
1k
Total Reward
800
800
800
600
600
600
200
200
0
-200
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4 0.5
Timestops (IM)
Timesteps
(IM)
Timesteps (IM)
Timesteps (IM)
MR.Q
D3eamerV3
TD-MPC2
PPO
Figure 4: DMC Proprioceptive learning curves. Time steps consider the number of environment interac- tions, where 500k time steps equals IM frames in the original environment. Results are over 10 seeds. The shaded area captures 95% boostrap confidence interval.
31

---

Published as a conference paper at ICLR 2025
C.3 DMC VISUAL
Table 6: DMC Visual final results. Final average performance at 500k time steps (IM time steps original environment due to action repeat) over 10 seeds. The [bracketed values] represent a 95% bootstrap confidence interval. The aggregate mean, median and interquartile mean (IQM) are computed over the default reward.
DrQ-x2
DreamerV3
MR.Q
swingu
219]
[1, 41
197 [179, 217]
121 [106, 145]
287 [254, 316]
hall_jn hall_in_cup-cutch cutch
[821. 973]
105 282]
932 [899, 961]
[969, 973]
[975, 980]
cartpole-balance
993 1990, 996]
353 [231, 485]
972 [948, 991]
998 (997. 1000]
909 [999, 999]
cartoll-balaace sparse
[887, 1000]
487 [233,
[1000, 1000)
1000]
1000 1000]
cartpole-swingup
864 [854, 873]
596 [437, 723]
690 [521, 813]
725 [003, 807]
868 [860, 875]
carpolloswwnggupsspss
774[741) [741, 805]
636 (404, 804]
[351. 726
[777, 816]
cheetab-run
728 [701, 7531
155 1110, 210]
431 5567.556
618 1576, 661
775 [752, 8071
dog run
10 19, 12]
19, 14]
[10, 18]
9 [6, 14]
60 (44, 80]
dog-stand
43 [37_49]
56]
117 [72, 148]
61 [30,
[201, 232]
dog-trot
14 [11, 18]
13 [12, 15]
20 [14, 25]
14 [13, 16]
68 [55, 79]
dog-walk
22 [18, 291
[17, 281
Jinger spin
860 [787., 922]
241 [107, 3771
786 (492, 9841
656 (544, 765]
963 [938, 982]
dinger-tum.casy
503 [399, 615]
189 [144, 233]
562 1317, 7791
491 [447, 5421
953 (927, 974]
linger hard
223 [121, 340]
60 1201
903 1870, 940]
494 (401, 571]
1905, 957]
fish-swim
84 [65, 107]
77 [64, 92]
[21, 64]
90 (84, 96]
79 [68, 93]
hopper-hop
224 [170, 278]
187 1119,
205 [125, 287]
(230, 315]
hopper-stand
917 (903, 931]
[0, 21
582 [321. 7941
888 (875, 900]
852 [703, 930]
humanoid-re
11,11
humanoid stand
[7.
6 [6, 7]
$ [5, 7]
5 [5.
87.8)
humancid-wwad walk
21
111.
1 21
2 12,
pendulum swingup
838 [8133 861]
[0,
748 [574, 850]
761 [709, 807]
(816, 842]
quadruped-run
459 (4122 507]
118 [98, 139]
262 [184, 330]
328 [255, 397]
498 [476. 522]
quadrupedwwakk
(699, 796]
149 [113, 184]
246 [179,
316 [260, 379]
(797, 867]
reacher-easy
938 (903, 9731
113 [55, 1921
956 (932, 978
735 (678, 796
979 1978, 9821
hard
1580, 831]
301
461]
(945, 9771
walker-run run
546 (475, 612]
39 [35, 44]
665 [566, 719]
669 [615, 708]
61.5 [571, 655)
walker stand
980 (977, 984]
253 [210, 310]
937 (907. 962]
969 (966, 973]
980 1977, 9851
walker walk
766 [489, 957]
47 56]
958 [952,
942 (936, 949]
(968, 973]
Mean
[497. 523
492 5121
[595,
Median
626 [528, 665]
49 [32, 53]
572 [419, 654]
493 (420, 532]
813 [779. 822]
JQM
[519, 564]
[46, 67]
[458, 537]
(430, 473]
[678, 703]
32

---

Published as a conference paper at ICLR 2025
sciobo--winngup
caripele-balane -aalancee
cartpele-balinnn www.a-caeapaaaaeeeeeeeeeeeee--ee
350
Total Reward
500
800
800
600
000
600
150
400
400
400
50
-50
cartpole-swee swingup
caripole swingup.sparse
cheetah run
dog run
80
700
700
Total Reward
60
600
40
300
20
-100
-100
dog-stand
dog-trot
dog-walk
finger sigger--pin
80
80
11
200
Total Reward
60
800
600
40
100
20
200
0
finger-tum tum_easy
fnger-turn.hard
fisb-swim
hopper-hop
Total Reward
800
600
80
600
150
400
200
40
50
200
-50
hopper-stand
humancid-run
humanoid-stand
Bumanoid-walk
1.6
Total Reward
2.5
12
1.5
pendulum-swingup
quadruped-run
quadrupedwalkk
reacher-easy
1k
600
1k
Total Reward
800
600
400
600
200
200
200
reacher-hard
walker-run
walker-stand
walker-walk
700
1k
Total Reward
800
800
800
500
600
600
600
300
200
100
200
200
0
-100
0
0.00
0.1
0.2
0,3
04
0.5
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
Timesteps (IM)
Timesteps (IM)
Timesteps (IM)
Timesteps (IM)
MR.Q
D3eamerV3
TD-MPC2
PPO
DrQ-v2
Figure 5: DMC Visual learning curves. Time steps consider the number of environment interactions, where 500k time steps equals IM frames in the original environment. Results are over 10 seeds. The shaded area captures 95% boostrap confidence interval.
33

---

Published as a conference paper at ICLR 2025
C.4
ATARI
Table 7: Atari final results. Final average performance at 2.5M time steps (10M time steps in the original environment due to action repeat) over 10 seeds. The [bracketed values] represent a 95% bootstrap confidence interval. The aggregate mean, median and interquartile mean (IQM) are computed over the human-now nommalized score.
Tack
DQV
Rainhow
Dreamer
Q
Alien
925 [879, 968]
1220 [1191, 1268)
320 [251, 3833
4638 [3863, 5813]
2834 (2261, 3388)
Amidar
178 [109, 186]
[250, 330]
126 (90, 167]
470 (419, 5241
[525, 667]
Assault
988 1957, 1011]
1430 1199, 1475]
423 (221, 581]
2518 (2909, 4179]
1296 [1254, 1343)
Asterix
238. [2313, 2669]
2699 [2598, 2783]
296 [216. 4031
7319 [6251, 8354]
3358 (3004_3797)
Asteroids
423 [408, 436]
816)
206 11000
14821
[638, 796]
Atlantis
(6893,
80837 [51139, 126780)
2000 (2000, 2000]
664529
973362]
556845 [4694155 660042)
Banklleist
474 [448, 193]
895 [589, 901]
187 [41, 4211
801 [691. 1002)
809 [639, 960)
HattiZZone
3598 [3225, 3875]
202099 117157, 223751
2200 (14600 31001
22599 (21055, 246699
19880 [13450, 26060)
BeamRider
369 [728, 1065]
5982 [5664] 6268]
479 [3448 5811
5635 [3161. 79621
2299 [1921, 2813)
Berzark
458 [406, 508]
(413, 484]
384 [3100 4001
758 (681, X23]
(456, 588)
Howling
29 127,2, 321
44 [500 521
138, 501
101 109, 1951
39 [45, 721
Boning
[31] 44]
68 [66, 213
-3 115,00 01
97 197, 991
96 (95, 971
Becakout
111
137 1621
(28,
Comtppde
2832 12418, 32151
4992 (47844 51381
20067 (17410, 227581
17935 [1616], 198177
Chopper
997 [971, 1022]
2265 (21600 2357]
688 [501, 878)
15172 (12900, 17219)
5748 (4822. 065
CrambClimber
64611 [40203, 787091
103539 (192149) 106850
996 [174, 17271
192811 (1284406 1359500
116954 [11157] 122033)
Defender
119254 (113371 122052)
119954
1220821
1333
34187 (29814, 392611
40457 (36892, 43638)
DemonAmuck
1503 [1382, 1600]
3477 [2369, 2678)
139 [116, 165]
[3443, 62311
5974 (4491,7289)
Doubieeaak
-18 1:200 -28)
-IM (-19.-19)
[-5.0] O]
21 (20, 221
-10 [-15,
Endaro
589 [567] 6171
1001 [1555, 1635
13 19, 171
476 [175, 7821
1845 [1758, 1938)
FishingDathy
-172
[-91, -871
Freeway
10, 191
32 182,
181
16, 321
32 [32,
Prosthite
269 [238, 294]
2510 12040, 2823
245 1251, 2591
5183 (2151. 82911
4561 (3299) 5740)
Goyler
1470 [1316, 15000
4279 (4139) 4425)
136 150) 174]
38711 [260066 48187)
19174 (14952, 23587)
Gravitr
167 1153,
202 1184, 2181
831 (768,
4901
Hero
3679 [3404, 2945]
E3t0 17914, 109633
1741 [1062, 2303]
20582 (19845, 21 31583
13450 [11955 14781)
keeHockey
-29 [1100, -2]
-5 [-66-
-8 --00 -83
14 [13. 16]
0 [11.22
Jamesbond
47 [42, 521
5541500) 5201
88 162, 1061
836 1568, 13191
624 1588, 0021
Kangarno
539 [525,
5501 7151]
(780,
13418)
2007 17851, 115911
(3042, 4400]
5972 15003, 6047]
121 (136, 735]
23092 (16679, 281721
9309 18646, 5953)
Muster
15997 113182, 188131
18074 [10041, 20864]
52 118, 951
702005 1501144 945781
29369 126954, 31595)
MovizzunnRevenge
0 [0,00] 0]
0 0]
D 10,000 00
1310 (598, 2 2180]
50 140)
MsPacman
[2121,
2347 [2292, 2403
5511]
(4191,
SameThisGame
4000 (3814, 4187]
9604 [8352, 8931]
1084 (663, 1501]
16742 (14542, 17103]
8693 (807), 9199)
Phoenis
4948 (4236, 5627]
4830 (4707. 4068]
101 181, 120
19822 114003, 16429]
5178 (5025, 5322)
Pitfall
.00 1-89, 335]
-14 [-29, -6]
-1 1:58 -2]
D 100 0)
-20 [-60, DJ
Pong
PrivateFye
118
[T8. 166]
-17 (-592) 762)
3046 1975, 5118)
100 [100, 100)
Qhert
1058 [1246, 2139
5355 [4363, 6783]
484 1993, 5701
168077 [16075, 17564]
3938 (3210, 4327)
Riveraid
3198 [3167. 3223]
4373 (40K0, 4440]
1045 (833, 1241]
9160 (8177, 10077)
10791 (9307. 12511]
ReadRuener
27980 [27269, 28002]
33412 [33459, 34435]
66455 (40606, 104163)
40579 (47425, 51426)
Robotanic
414.51
[18,
551
15 (12,
Sesquest
200 [277, 318]
1641 [1621, 1661]
250 [214, 2803
3416 (2665, 4436]
3522 (2401, 4850)
Skiing
-19568 1-19793, -1993]]
2240T0 (-25305 -22677
27901 [50000, 223704)
-30043 (-509944 -297644
33000 (-00000 -3000]
Solaris
1804]
[1143,
2709]
1799, 1430)
Spucehnesers
663 [651, 675]
744 (731, 764]
294 (235, 354]
1433 [1039, 1943]
(626, 768)
StarGunnner
692 (662. 219]
1488 (14700 1505]
415 1916, 4091
2090 [1678, 2649]
3488 (1052, 8241]
Sumound
3458 [1002, 8243]
3488 [1082, H241]
- 100 100
5 (4, 71
[-4
Tonnis
1194
(-2.00
(-11. 0J
OJ
TimeP
1539 [1479, 1613]
2008 (2627, 2787]
548 (550 680)
7729 (3128, 13016)
4982 (4208, 4528)
Tutankhum
112 197, 123)
179 [165. 191]
29 [17 45]
255 [240, 269]
164 [145, 185]
UpNDoww
7669 [7116, 8147]
12392 [11489, 13312)
595 (428. 737]
284807 (178615, 391 588)
73095 [4083%, 108810)
[14,
[0,
304]
ViduoPinball
5129 [4611, 5649]
26345 (23075, 20067]
1005 (0. 2485]
22345 [20669, 23055]
53826 [40600, 67972]
WizrddWWer
481 [506, 542]
2213 [1827, 2617]
225 (185, 264]
7086 (6818, 7730)
2599 [2259, 2042]
Yarseevenge
9426 (9177, 9656]
10708 [10405, 11071]
1891 (925, 29641]
62209 (57783, 67113)
34861 (29734, 40020)
112 230]
3661 41931,
10,00
17347 [15300, 19385]
8850 [8045, 9740)
Mean
0.25 (0:24, 0.26]
1.08 (1002, 1.14)
0.09 (-0.10, -0007
3.74 (3.29,
2.54 (2.34, 2.75)
0.32 (0. 30, 0.12]
(0.40, 0.47]
0:01 (0.00, 0LO1]
1.25 [1.11, 1.47]
0.96 (0.78, 0.981
JUM
0.37 10.16, 0.171
0.61 (0,00, 0.621
0.02 (0.01, 0002)
1.46 [1.34, 1.511
0.90 (0.88, 0.94)
34

---

Published as a conference paper at ICLR 2025
Alien
Amidar
Assault
Astorix
Asteroids
Atlantis
Total Reward
IM
600K
2k
400
200k
Henk
BeamRider
Bowling
Boxing
Totul Reward
25k
900
160
600
15k
700
120
80
3k
500
40
200
300
-200
-1k
100
0
-40
Breakout
Centipede
ChopperCommand
CrazyClimber
Defender
DemonAttack
Total Reward
140
20k
15%
14k
100k
140k
35k
7k
6k
60k
15k
20
Sk
20k
1k
-20
22
-20c
5k
-1k
DoubleDunk
Enduro
DishhngDerhy
Freeway
Froshitte
Gopher
Total Reward
188
50k
0
750
40
15
10K
-120
-10k
Gravitar
Hero
Hockeykey
Jamesbond
Kangaroo
Krull
Total Reward
1k
25k
20
14k
35k
600
15k
10
0
000
10k
6k
25k
15k
200
5k
10
2k
Sk
-200
-5k
20
200
-k
.kk
FuMattrr
MonterumaaRevenge
MsPacman
Name ThiseThhiaaas
Phoenix
Pitfall
Total Reward
100k
1.5k
14k
60k
100
8k
-200
4k
-300
-250
-1k
2k
-400
Pong
PrivateRye
Qhert
Rivernid
RoadRunner
Robotank
Total Reward
7k
17.9k
12k
175k
5k
12.5k
125k
5k
7.5k
1k
2.5k
-25
11
2..kk
-2kk
Seaquest
Skiing
Solaris
Spacelmaderss
StarGunner
Surround
Total Reward
16k
3.5k
2k
10k
-00
2.5k
1.5k
6k
2.5
-24k
1.5k
1k
-2
-25k
500
500
2k
-7
-1k
-32k
-500
-2k
-12
Tennis
TimePilot
Tutankham
UpNDown
Venture
VideoPinhall
Tocal Reward
0
350k
250
-10
250k
150
150k
50
10
10
5
10
Frames (IM)
Frames (IM)
Frames (IM)
Frames (IM)
Frames (IM)
Frames (1M)
WizardOTWer 70k
YeesRevenge
Zaxzon
Total Reward
50k
12.5k
7.3k
10k
-1k
.10K
22.5 5k
10
Frames (1M)
Frames (1M)
Frames (1M)
MR.Q
DreamerV3
+ PPO
Rainbow
DQN
Figure 6: Atari learning curves. Time steps consider number environment interactions, where 2.5M time steps equals 10M frames in the original environment, Results are over 10 seeds. The shaded area captures a 95% boostrap confidence interval.
35

---

Published as a conference paper at ICLR 2025
D COMPLETE ABLATION RESULTS
In this section, we show a per-environment breakdown of each variation in the design study in Section 5.2. Each table reports the raw score for each environment. The [bracketed values] represent a 95% bootstrap confidence interval. The aggregate mean, median and interquartile mean (IQM) are computed over the the difference in the normalized score. We use TD3 to normalize for Gym, raw scores divided by 1000 for DMC and human scores to normalize for Atari (see Appendix B.3). Highlighting is used designate the scale of the difference in normalized score:
< 0.5)
-0.2,0..5 -0.5)
-0.01,-0.2)
0.01,
[0.2, 0.5)
(2 0.5)
D.1
GYM
Task
MR.Q
Linear value Function
Dynamics largel
No Largel encoder
Ant
6901 [6221] 7482]
1844 [1663, 2018]
5867 [5543, 6289]
3970 (2468, $509]
HalfCheetah
12939 [11663, 13762]
3383 [3054, 3732]
14019 [13746, [4285]
12838 [12459, [3266]
Hopper
[ 3309]
1720,
2890
2007 [2164,
Hurnanoid
10223 (9929, 10498]
461 [395, 532]
8370 (7651, 8988]
305 2272, 356]
Walker2d
6039 [5644, 6386]
1999,
5844 (5146, 6477]
(5570, 6323]
Mean
..17 19,19 -1.15]
-0.10 (-017,,7
-0.53 (0.60. 40:46
Median
1.25 [11 1,288 11211]
.0.05 [-0.23, 0.09]
0.02 (.0.66 0.02]
JQM
-.13 [-115,
15.15 -1.11]
-0.08 (-0019,0.01]
-0.25 [-0.37, -0.16]
Task
Q
Non-linear
MSF loss
Ant
6901 [6261, 7482]
422 1770, 846]
7215 [6971, 7466]
7153 [5991, 7815]
HalfCheetah
12939 [11663, 13762]
-658 -604)
(12649, 14053]
(1409,, 14710]
Hopper
2692 33091
103 [39. 1891
2492 3424]
2869 [2090, 3689]
Humanoid
(9929, 10498]
[104, 277]
[9612, 10688]
[10017, 10983]
Walker2d
6039
[5644, 6386]
260 [-5, 638)
5548 (4980, 17]
6626 [5256, 7984]
Mean
1.47 1,544 1.39]
.0.01 [-0.07, 0.03]
0.10 [-0.02, 0.19]
Median
1.47
[-1 53, -1.37]
0.01 [-0.03, 0.08]
0.07 [0.00, 0.]7]
JQM
[-1.88 -1.39)
-0.01 [-0.05,
0.09 [-0.01,
Task
MR. Q
No reward scaling
No min
No LAP
Ant
6901 [6261] 7482]
6866 [6227, 7547]
6936 [6582, 7329]
6817 [6616, 7039]
HallCheetah
12939 [11663, 13762]
13502 [13333, 13673]
14143 [13819, 14515]
13185 [13085, 13299]
Hopper
2692 [2131, 3309]
[2090, 3064]
(1728, 2626]
[1883, 3465]
Humanoid
10223 (9929. 10498]
9515 [8520, 10245]
10528 [10202, 10837]
8441 [6206, 9738]
Walker2d
6039
[5644, 6386)
5743 (5362, 6102)
4293 (3547, 5107)
5463 (4134, 6376]
Mean
-0.04 [-0.09, 0.02]
-0.09 [-0.16, -0.01]
-0.10 [-0.24, -0.00)
Median
[-0.11, 0.04]
[-0.08, 0.07]
-0.02 (-0.12, 0.02]
IQ
-0.04 [-0.10, 0.031
-0.04 [-0.10, 0.02]
-0.06 (-0.18. 0.00)
Task
MR. Q
No MR
1-stop return
No unroll
Ant
6901 [6261, 7482)
4195 [2573, 5819]
7757 [7729, 7799]
7528 7832)
HalfCheetah
[11663, 13762]
(9238, 12495]
13123 [10691, 14653]
[13817, 15002]
Hopper
2692 3309]
1877 [1524, 2153]
2737 (2131, 3343)
2578 [1857, 3414]
Humanoid
10223 (9929, 10498]
3942 (3262, 4624
2328 [4911, 3337]
10617 110504, 10731]
Walker2d
6039 [5644, 6386]
4155 [3251, 4897]
4747 [3197, 6229)
6077 [5752, 6355]
Mcan
-0.43
-0.33 -0.21]
[0.01. 0.10
Median
00.88 [-0.71, .0277
0.01 [-0.21, 0.16]
0.08 (0.06. 0.16]
IQM
[.0666, .0.28]
.0.10 [.032,0..2,0 0.1.3
(0.04, 0.15]
36

---

Published as a conference paper at ICLR 2025
D.2 DMC PROPRIOCEPTIVE
MR.Q
Linear value
Dynanics lurget
No target encuder
acrobot-swingup
567 [517. 621]
30 [15, 461
626 [578, 684]
16 [9,
ball iniiin.uu cutch
981 (979, 983]
820 [658, 922]
980 (978, 983]
569 7196
purpole-halance
999 [999, 1000]
449 [380, 520]
999 [999, 1000]
992 [986, 997]
carpolle-bllanee.e sparse
1000 [1000, 1000]
183 [142, 224]
1000 [1000, 1000]
1000 [1000, 1000]
cartpoleswwnnuup
866 1866, 866]
267 [225, 310]
869 [866, 876]
852 [840, 861]
cartpule-swingue sparse
798 [779,
[809,
cheetab. run
914 [911, 917]
194 [376, 411]
919 (919. 92.]
904 [899,910) 910)
dog-run
569 [546, 595
181
254 (202, 305
11 [8, 16]
dog stand
967 [959, 975]
[17, 29]
672 [520, 830)
36 [27, 51]
dog-troi
877 [845, 898]
15 [11, 20]
319 [279. 365]
12 [8. [9]
dog-walk
916 [908, 924]
111,
312 (247. 396
10 [7, 15]
Jinger-spin
937 [917. 958
736 (670, 8251
942 1916, 9711
869 [698s, 9633
finger easy
[928, 975]
238
947 [900, 979]
[509. 756)
Anger-tum_hhee hard
950 [908, 974]
23 [1. 63]
923 [878, 9666
431 [301, 563)
lish sish-wiim
792 [772. 811]
165, 102)
410 [307, 493)
97 [65. 129]
hopper-hop
251 [201, 295]
10 17]
199 [131, 265]
[0.
hopper-stand
951 [948, 955]
66 [30, [02]
639 [413, 866)
4
bumanoid-run
200
111,1
humanoid stand
868 |823, 907]
6 15, 71
716, 8]
716,99
humanoid
[609, 721]
[1,2]
[2.
[2,
pendulum-sue swiiggu
748 [594, 830]
357 [114, 617]
826 (812, 840]
784 [706, 834]
quadruped-run
947 [940, 954]
172 [93, 252]
942 (930, 951]
829 [757, 895)
quadruped
963 [959, 968]
[52, 141]
939 [935, 943]
952 [946, 960]
reacher-casy
983 [983, 985]
802 [722, 877]
984 [983, 986]
983 [981, 986]
reacher- hard
977
853
(965,
(972. 979]
walker run
793 [766, 816]
238 (207. 274]
730 [585, 814]
776 792]
walker
[987, 990]
[780, 921]
[985, 991]
[986, 991]
walker-walk
978 (978. 980]
504 [397, 613]
975 (975, 977]
974 (973, 976
Mean
0.58 [0.59, .0566]
.0.15 [-0.15 -0.15]
.0.35 [-0.55, -0.34]
Median
0.58 |..64, 00.77]
-0.01 |-0.02, -0.00)
...2 [-0.33, -0.21
IQM
[0.64, .0.60)
[0.066 -0.03]
[.0.9, .0.25]
Task
MR.Q
Revert
Non linear model
MSE reward loss
acrobot-swingup
567
553 [478, 629]
[547, 612)
hall in catch
981 [979, 983]
301 [233, 365]
982 (982, 984]
983 [982, 985)
carrole-halance
999 [999, 1000]
272 [206. 332]
999 [999, 1000]
[998, 1000]
cartpole-alanne.co sparse
1000 [1000, 1000]
197 [177, 214]
1000 [1000, 1000]
1000 [1000, 1000]
carpoole-winnuup
866 1866, 866]
191 199, 2631
866 (866, 867]
865 [865, 866]
798 (779, 818)
824 (809, 839]
812 (786, 834)
cheetab-run
914 9911
74 [31, 123
900 (906, 913]
910 (907, 915]
dog-run run
[546, 595|
[516,
[513, 545)
dog stand
967 [9.59, 975]
22 [18, 29]
962 [938, 982]
964 [958, 971]
dog-trot
877 [845, 898]
[3.
868 [816, 914]
[831, 886]
dog-walk
916 (908, 924]
4 [4. 6]
920 (915, 925]
724 [473, 882)
linger-spin
937 [917. 958
11
868 [767, 9511
907 1875, 9471
finger tum_easy
953 [928, 975]
159 (60, 280]
972 [968, 978]
935 [894, 977]
liner-turn.hard
950 1908, 974]
60 120, 1001
931 [887, 9751
947 1910, 9691
sish-swim
792 [772,
[53, 90]
(754, 824]
[766. 821]
hopper-hop
251 [201, 295]
0 [0. I]
288 [222, 332]
174 [119, 230]
sopper-stand
951 [948, 955]
[33
848 [664, 945]
[707, 936)
bumanoid-ruu
200 [169, 236]
111.1
205 [191, 2211
91 [45, 121]
humancid stand
868 [823, 907]
717. 81
811 [712, 878]
214 [14, 566]
humanoid. walk
662 [609, 721]
[2.2]
668 [590, 734]
77 [3, 224]
pendulum swingup
748 [594] 830]
61 |20, 1031
819 [802, 836
827 [811, 843)
quadruped run
947 [940, 954]
[40,
[934, 954]
(941, 956]
quadruped-we walk
963 [9559, 968]
81 [36, 130]
963 [961, 967]
966 [961, 971]
reacher-casy
983 [983, 985]
789 [727, 856]
983 (982, 984]
964 [927, 984)
hard
977 1975. 9791
$26 1361,, 6951
953 (914,
976 1974, 978]
walker run
793 [766, 816]
[22, 31]
795 [784, 811]
778 (709, 821]
walker-stand
988 1987, 990]
221 [179, 264]
983 [971, 991]
988 [987, 990]
walker-walk
978 (978, 980]
33 [23, 46]
974 (972, 978]
967 (948, 979]
Mean
-072 [R0.3, 40.72]
-0.00 [-0.02, 0.011
-0.06 1-0.08, -0.05]
Median
0.78 0.800, .0.76]
.0.00 [-0.00.00 0.00]
.0.00 [-0.01, 0.00]
JQM
.0.78 [.0.78, -0.76]
-0.00 [-0.01, 0.00)
-0.01 [-0.02, -0.00]
37

---

Published as a conference paper at ICLR 2025
Task
MR.Q
No reward scaling
No min
No LAP
nembot-swingup
567 [517, 621]
593 [532, 672]
623 [572, 678]
566 [520, 612]
ball wii..cuuupcarel
981 [979. 983]
983 [982, 984]
982 [980, 984]
981 [980, 984]
carpole-balance
999 1999, 1000)
998 (999,
999 [1000, 1000]
999 (998, 1000]
cartpole-alannee sparse
1000 [1000, 1000]
1000 [1000, 1000]
1000 [1000, 1000]
992 [982, 1000]
curpoole-winnup
866 1866, 866]
865 1864, 866]
868 (866, 874]
865 [865, 866]
cartpule-swing wwwpll/swiigguppppss....
798 [779, 818]
647 1318 8221
799 [780, 812]
796 [779, 810]
cheetah run
914 [911. 917]
911 [909, 914]
910 [893, 92.1]
908 [905, 913]
dog-run
569 [546, 595]
586 [546,
577 [540, 610]
536 (499,573]
dog-stand
967 [959,, 975]
959 (9400 979]
946 (917. 969]
971 [966, 976)
dog-trot
877 [845,
817 1713,
[767, 906
[764, 8977
dog-walk
916 [908. 924]
901 [890, 917]
747 [4477, 908]
899 [886, 914]
linger-spin
937 [917. 958]
873 1768, 9471
926 (907, 950]
915 (892, 9488]
linger tuw
953 [928, 975]
977 1973,
976 (972, 980]
975 [967. 983]
finger-tum_ham
950 [908, 974]
946 [905, 969]
894 [835, 95.]
949 [909, 972]
fish-swim
792 [772, 811]
745 [663,
785 [763, 810]
788 [754, 826]
hopper-bop
251 [201. 295]
343 [263, 477
336 [322, 3521
347 [265. 4311
hopper stand
[948, 955]
934 2912,
(926. 9471
[935, 9448]
bumanoid-run
200 [169, 236]
184 [149, 214]
198 [175, 225]
202 [191. [12]
humanoid stand
868 |823, 907]
810 1655, 8991
833 [793, 8711
880 [856, 900)
humanoodd walk
662 [609, 721]
665 1589, 765]
597 [292, 808]
697 [561, 828]
pendulum-swingww
748 [594, 830]
816 (790, 838]
825 [81 839]
815 [792, 836]
quadrupod-run
947 [940, 954]
951 [944,
946 (941, 951]
[927, 947]
quadruped-we walk
963 [959, 9688
966 (961. 971
959 (942, 9721
955 [9412, 96.7
reacher easy
983 [983, 985]
964 (926.
983 [981, 986]
[982, 986]
reacher-hard
977 [975. 979]
971 1968, 975]
978 (974, 9821
974 [969, 981]
walker run
793 [766, 816]
804 1783, 820]
806 (779, 821]
812 [803, 822)
walker
988 [987, 990]
989 [988, 990]
989 [986, 992]
986 [985, 987]
walker-walk
978 (978, 980]
979 (978, 980]
978 (976, 980]
977 [974, 980]
Mean
00.01 [-0.02, 0.00]
-0.01 [-0.02, 0.01]
0.00 [-0.00, 0.01]
Median JQM
-0.00 [-0.00, 0.00)
-0.00 [-0.00, 0.00]
-0.00 [-0.00, 0.00]
-0.00 [-0.00, 0.00]
-0.00 (-0.00, 0.00]
-0.00 [-0.00. 0.00]
Task
MR.Q
No MR
1-step return
No unroll
acrobot-swingup
567 [S17, 621]
576 [483, 665]
440 [360, 528)
515 [455, 598]
ball .ww. catch
981 [979, 983]
981 [980, 984]
984 (983, 985]
982 [981, 984]
cartpole-halanes
999 [999, 1000]
994 1991, 999]
999 [1000, 1000]
999 [999. 1000]
cartpole-balancesspe
1000 [1000, 1000]
1000 [1000, 1000]
961 [886, 1000]
1000 [1000, 1000]
cartpoloswwingup
866 1866,
870 1864,
881 (879, 882]
[861, 867]
cartpole-swinguee
wwinsuppssarree
798 (779, 818]
684 [528, 814]
845 [845, 8471
818 [811, 831]
cheetah-run
914 [911, 917]
871 [823, 907]
922 [921, 92.4
[908., 911]
dog-run
569 [546, 595]
68 [63, 75]
299 [196, 360]
514 [473, 554]
dog-stand
967 [959, 975]
494 1452, 530]
606 [344, 865)
955 (944, 971]
dog-trot
877 [845, 898]
[49, 80]
725 [679, 756]
857 [833, 883]
dog-walk
916 [908, 924]
102 181, 122)
788 [739, 832]
920 [905, 934)
finger-spin
937 [917. 958]
1731,
983 1977, 9881
[781, 9410]
finger tum,easy
953 [928, 975]
947 [913, 974]
980 (979, 982]
950 [917. 976)
finger-tum,hand
950 [908, 974]
846 [756, 926]
968 [958, 976]
947 [907,] 971]
fish-swim
792 [772. 811]
706 [683, 727]
498 [323, 651]
709 [618, 783]
hopper-bop
251 |
133, 142)
364 [336, 394
297 [169, 442)
hopper-stand
951 [948, 955]
365 [233, 491]
952 (947,, 959]
949 [944, 955]
humancid-run
200 [36)
1 [1. 2]
190 [124, 241]
192 [172, 214]
humamoid
[823, 907]
201 [9, 517]
753 [665, 838)
[806, 913]
humanoid walk
662 [609. 721]
84 [3, 247]
761 [689, 827]
675 (72]
penduhum-swingw
748 (594, 830)
827 [812, 842]
823 [807, 841]
819 [793, 843]
quadruped-run
947 [940, 954]
871 [793, 933]
945 (940, 950]
950 [944, 955]
guadruped-we walk
963 [959, 968
951 1943, 962]
962 (958, 968]
962 1955, 969]
reacher-casy
983 [983, 985]
980 (979,
984 [984, 986]
981 [976, 986]
reacher-hard
977 [975, 9791
949 (909, 974]
980 (979, 9831
954 [913, 978]
walker
793 [766, 816]
780 [769, 790]
[827, 843]
[702, 825]
walker-ster stand
988 [987. 990]
983 [980, 988]
990 [990, 992]
988 [988, 990]
walker-walk
978 (978, 980]
976 (975. 977]
979 [977. 982]
975 [969. 981]
Mean
19 [-0. -0.18)
.0.04 [-0.05, -0.02]
-0.01 [.0.01,-0.00]
Median
-0.05 [-0.08, -0.01]
0.00 [0.00, 0.00]
-0.00 [-0.00, 0.00]
IQM
-0.06 |-0.08, -0.05|
0.00 [-0.00, 0.01]
-0.00 [-0.01, -0.00)
38

---

Published as a conference paper at ICLR 2025
D.3 DMC VISUAL
MR.Q
Linear value
Dynamies lurget
No target encuder
acrobot-swingup
287 [253, 3171
[5, 22]
296 (281. 323]
16 [0, 38]
ball cup cutch
977 1975, 980]
644 [328, 904]
972 (965, 976]
605 [496, 726)
curpole-halance
999 [999, 999]
306 [254, 149]
998 [998, 999]
978 [947, 997]
carpolee-bllanee.e sparse
1000 [1000, 1000]
243 [183, 327]
1000 [1000, 1000]
1000 (1000, 1000]
cartpoleswwingup
868 [861, 875]
229 [181, 294]
861 (859, 865)
689 [487, 808)
cartpule-wwingle sparse
797
14]
267
cheetab. run
775 [752, 805]
230 [159, 294]
831 [761, 875]
745 (723, 784]
dog-run
60 [44, 801
[17, 22]
36 [34, 391
10 [6, 201
dog stand
216 (201, 253]
76 [70. 82]
191 [155, 247]
60 [44, 89]
dog-troi
65 [54, 79]
19 [16, 24]
46 [42, 53]
[9. 10]
dog-walk
77 [70,
[26, 33]
62 [57, 72]
16 21]
Jinger-spin
965 [938, 982
789 1598, 9231
786 (672, 9311
929 1893, 9811
finger tumm easy
[925,
198,
[691, 969]
[855, 969]
Inger-tum,haet hard
932 [905, 957]
66 [0, 1001
859 [777. 963]
492 [385, 577]
lisb-swim
[67, 931
[45, 109)
71 [38, 106]
65 [49, 84]
hopper-hop
270 (229. 317]
[0
184 (165, 204]
[00
hopper-stand
852 [705, 932]
713. 11]
911 [900, 922]
[3, 9]
bumancid-run
111.
humanoid stand
7 17.
5 14, 81
6 [S, 8]
16, 81
humanoid
[1.2]
[1.
[1.2]
pendulum-sue swwingu
829 [8115, 842]
97 [0, 192]
749 [632, 840]
191 [93,, 287]
quadruped-run
498 [474, 523]
131 [77, 187]
488 (468, 5171
[566, 594]
quadruped-we
833 [796, 868]
105 [57, 155]
717 [445, 895]
817 [790, 868]
reacher-casy
979 [978. 982]
605 [398, 868)
977 (973, 98.]
979 [970, 986]
reacher- hard
[945, 9771
288 [195,
1971,
(963, 975]
walker run
615 [571, 655]
158 [133, 193]
531 [463, 577]
611 [590, 631]
walkerr-ttane
(977,
707 [601,
[981, 984]
[979. 988]
walker-walk
970 [968. 973]
350 [228. 529]
904 [850, 9511
968 (957, 972]
Mean
0041 [0.42, .0:39]
.0.05 [.0.55 -0.04]
.0.15 [-0.5, -0.15)
Median
.0.37 |-0.44. -0.37]
.0.01 [-0.01, -0.00]
.0.03 [-0.05, -0.03]
IQM
[0043, -0.38]
[.002 .0.01]
.0.05 [.0.66. .0.04]
Task
MR.Q
Revert
Non linear model
MSE reward loss
acrobot-swingup
287 1253. 3171
[13, 231
[235, 314]
[242, 294]
hall in cup catch
977 [975, 980]
195 [91, 297]
973 [970, 977]
974 [969, 978]
carpole-halance
999 [999, 999]
190 [163, 212]
999 [999, 999]
[998, 1000]
cartpole-alanne..o sparse
1000 [1000, 1000]
346 [193, 639]
1000 [1000, 1000]
1000 [1000, 1000]
carpoole-winnuup
868 1861, 875]
115 182, 175]
849 (819. 873]
876 (875, 878]
797 [777, 818]
768 [684, 824]
33 [3] 60]
cheetab-run
775 [752, 805]
69 [36, 129)
763 (757, 770]
732 (712, 757]
dog-run run
[44, 80]
13,44
401
36
dog stand
216 (201, 233]
17 [16, 19]
200 [193, 207]
195 (187, 208]
dog-trot
[54, 79]
6]
[47, 56]
47 [44, 50]
dog-walk
77 [70,
16. 81
69 [62. 78]
63 [58, 71]
linger-spin
965 1938, 9821
10. 21
907 [841, 9711
924 1884, 980]
finger tum_easy
953 [925, 974]
133 (99, 200)
932 (889, 977]
844 (786, 881]
finger-tun.hall
932 1905, 9571
66 (0, 1001
938 (903, 9671
900 1867, 964]
sish-swim
[67, 93]
[47, 60]
79]
75 105]
hopper-hop
270 [229, 317]
0 [0, 2]
308 [257. 368]
102 [14,155 151]
sopper-stand
852 [705, 952]
133 8]
935 [929, 942]
[914, 925]
bumanoid-rtu
1 [1.2]
1 111.1
[1.
111.
humancid stand
7 [7.
15. 71
[6,
717.
humanoid. walk
2
[1.2]
[2.
[1,
pendulum swingup
829 [815, 842
66 [0. 100)
820 [795, 844
581 [99], 842]
quadruped run
(474,
(69,
578)
[478, 544)
quadruped-we walk
833 [796, 868]
76 [34, 100)
762 [727. 788]
835 (751, 880]
reacher-casy
979 (978, 982]
583 [395, 684)
939 (900, 979]
979 (976, 984)
haad
965 1945, 9771
1001
868 [753, 9561
941 [879, 976]
walker run
615 (571, 655]
[29, 39]
612 [593, 635]
596 [505, 643)
walker-stand
980 (977, 984]
280 (269. 294]
982 (977, 987]
983 [984, 984]
walker-walk
970 (968, 973]
22 [19, 25]
951 (917, 972]
969 (961, 976)
Mean
-0.52 |R0.22 -0:51]
-0.01 [-0.02, -0.00)
-0.05 [-0.07, -0.04]
Median
0.68 0.772, .0.62]
.0.01 [.0.01.-.000 -0.00]
.0.01 [-0.01, 0.00]
JQM
.0.57 [-0.58, .0.56]
.0.01 [-0.01,-0.00]
.0.01 0..00)
39

---

Published as a conference paper at ICLR 2025
Task
MR.Q
No reward scaling
No min
No LAP
nembot-swingup
287 [253, 317]
323 [275, 368]
332 [301, 391]
280 (235, 354]
ball cupcuppcatchh
977 [975, 980]
973 [971, 977]
975 [974, 976
971 [966, 978)
cartpolebblaacee
999
1999,
1998,
(997.
cartpole-balanes sparse
1000 [1000, 1000]
1000 [1000, 1000]
1000 [1000, 1000]
1000 [1000, 1000]
carppole-wwingup
868 1861, 875]
860 1829, 877]
879 [879. 880)
798 [785, 821]
cartpole-swingg
797 1777. 818]
813 1805. 8231
805 1763, 829]
764 [736, 799)
cheetah run
775 [752, 805]
720 (678. 758]
751 [734, 762]
706 [670, 741)
dog-run
[44,
[49, 73]
42 [37, 51]
62 [45, 91]
dog-stand
216 (201, 233]
317 [239, 387]
228 [224. 232]
279 [229. (15)
dog-trot
154,
155, 791
(48, 531
611
dog-walk
77 [70,
89 [83, 96]
86 [69, 106]
91 [85, 101]
linger-spin
965 1938, 9821
903 1776. 9751
870 (709, 982]
940 1864, 9791
linger tum.eaay
953 [925, 974]
873 [775, 954]
963 [952, 975]
844 (785, 879)
finger-tum.ham
932 [905, 957]
923 [885, 962]
933 [874, 976]
932 [858, 974]
fish-swim
[67,
159, 87]
[49, 61]
63 89]
hopper-hop
270 [229, 317]
244 [204, 298
255 [244, 275]
186 [152, 204]
hopper stand
(902,
(877, 896]
bumanoid-run
1 [1
[[
[I.
humancid stand
7 17.
16. 81
615.
10 16, 181
humanokd walk
2 [2,
2 [2.
[2,
[2,
pendulum-swingwe
829 [8155, 842]
823 [798, 846]
831 [809, 843]
829 [808, 841]
quadrupod-run
498 523
505 [471
539 [500,
463 [428, 485]
quadruped-ww walk
833 1796, 868
823 1781, 8671
849 1745, 9091
799 [713, 905
reacher easy
[978,
962 [924,
(897, 981]
[885, 980)
reacher-hard
965 1945, 9771
972 1970. 975
936 (872, 9751
973 1973, 974]
walker run
615 [571, 655]
600 [544, 632
666 [643, 682]
662 [629, 7255
walker-stanm
980 1977, 984]
986 [984,
982 [975, 988]
984 [984, 985]
walker-walk
970 [968, 973]
970 (968, 974]
970 [968, 972]
972 [964, 979]
Mean
00.00 [-0.01, 0.01]
0.00 [-0.01, 0.01]
-0.01 [-0.02,
Median
[-0.00, 0.00]
0.00 [-0.00, 0.00]
-0.00 [-0.01, 0.00)
JQM
-0.00 [-0.00, 0.00]
0.00 [-0.00, 0.00)
-0.01 [-0.02, 0.00]
Task
MR.Q
No MR
1-step return
No unroll
acrobot-swingup
287 [253, 317]
362 [305, 421]
91 [76, 112]
126 (77, 159]
ball .uu. catch
977 (975, 980]
898 (746, 977]
980 (979, 982]
976 [973, 980]
cartpole-halanes
999 [999, 999]
998 [998,
998 (998, 1000]
998 [999. 999]
carrpoleballaneespee
1000 [1000, 1000]
1000 [1000, 1000]
1000 [1000, 1000]
1000 [1000, 1000]
carpooloswinnup
1864,
[835, 875]
[863, 879)
cartpole-swingee
wwinsuppssurree
797 [777, 818]
459 [139, 7801
712 [685, 7591
529 816]
cheetah-rum
[752.
782 [765, 805]
[674, 677]
[679, 845)
dog-run
60 [44, 80]
22 [21, 24]
30 (21,
45 [36, 56]
dog-stand
216 (201, 233]
137 [127, 148]
160 [143, 191]
200 1195, (17)
dog-trot
[54, 79]
[29. 36]
29 [25, 32]
47 49]
dog-walk
77 [70, 83]
42 [36, 50]
55 [33, 671
67 [63, 76)
linger-spin
887
1978, 9891
[588, 9600
finger tum,easy
953 [925, 974]
694 [539, 805]
942 (874, 979]
869 [766, 962]
finger-tum,hant
932 [905, 957]
622
908 (872, 974]
(780, 973]
fish-swim
79 [67, 93]
72 [60, 93]
64 [58. 72]
67 [55, 90)
hopper-bop
270 |229, 3171
192 1166, 2161
248 (231, 280]
242 [219, 270]
hopper-stand
852 (705, 932]
918 (897,
877 (820,
(907, 940] 1 [1.
humancid-run
1 [11.2
21
11..1
humanoid
7
717. 81
7 [5,
humanoid walk
2 [2.
2 [2. 3]
2 [2, 3]
2 [2,
penduhum-swingue
829 [815,, 842]
819 [787, 844]
828 (81 839]
[382, 811]
quadruped-run
498 [4744, 523]
478 [432, 515]
456 (424, 476]
398 [326, 465]
muddruped-wald
833 (796, 868
701 1663, 731
666 [627, 7201
730 [663, 769)
reacher-casy
979 (978,
978 (976,
972 (948,
1977, 980)
reacher-hard
965 9945.9771
545 [214, 858
978 (972, 9821
893 (776, 9671
[571, 655
[639,
[619, 696]
walker-sthe stand
980 (977, 984]
974 [962. 986]
986 (982, 991]
983 [981, 987]
walkerrwalk
970 [968, 973]
955 1948, 964]
971 [967, 978]
971 [971, 972]
Mean
.0.07 [-0.09. .0.03]
.0.03 [-0.03, -0.02]
.0.04 [-0.06, -0.01]
Median
-0.02 [-0.03, -0.01]
-0.01 (-0.01.-0.00
-0.01 --0001
JQM
-0.03 |-0.03, -0.01]
-0.01 [-0.02, -0.01]
-0.02 [-0.02, -0.01]
40

---

Published as a conference paper at ICLR 2025
D.4 ATARI
MR.Q
Dynamics target
larget encudde
Alien
2471 [1848, 31551
596 [561, 6311
1176 [1138, 1215]
2040| [1585, 2495]
Amidar
443 (376, 499]
48 138, 61]
214 [182, 247]
249 [232, 268]
Assmmit
1125 (1094, 1160]
366 [3599 374]
911 [906, 917]
1057 [880, 1234]
Asterix
2216 [208 2346]
[585, [035]
1940 [1865, 2015]
2407 [1860, 2955]
Asteroids
602 [493, 689)
609 [595, 623]
776 [716, 837]
763 9391
Allamiis
445022 [282338, 630730]
[13080,
529745
913740)
77770]
BankHeist
542 [348, 749]
93 [76, 111]
646 (326, 966]
40 (38,43)
BattleZone
16520 110560. 22760]
11300 111100, 11500]
8250 (2100, 14400]
4600) (2500, 6700
BeamRider
2007 [1855, 2194]
584 [528, 642]
1201 [1015, 1387]
1468 [1298, 1639)
Berzerk
430 [383, 472]
315 (216, 415]
381 [359, 403]
359 [275, 444]
Bowling
50
31 33)
81
40 48]
Boxing
95 (94, 971
39 437
90 188, 931
92 [89,96]
Breakout
[21.
4]
[1.2]
Centipede
14954 [13541, 16508]
6709 [6207, 7213]
7853 [7680, 8026]
5167 [3661] 6674]
ChopperCommand
4348 [3756, 5002]
890 [780, 1000]
3055 [2870, 3240)
1385 [1060, 1710]
CrazyClinber
104766 (99290, 109629]
23016 (216100 25010]
92455
[84000, 100910]
40240 (29150, 51330]
Defender
25062 [234066, 29182]
7825 [5640, 10010]
17592 (92900, 25895]
11627 [5745, 175 10]
DemenAttack
4660 (4072, 5241]
1608 1852]
3221
(204, 352]
DoubleDunk
[-11, -91
-18 |-24, -131
-15 |-22, 110
-20 1-21, -191
Enduro
1480 [1378. 1592]
[319, 368]
[621, 623]
[667, 7133
DerbyngDerby
-34 [-43, -27]
-90 [-96, -86]
-64 [-66, -62]
-81 [-87, -75]
Freeway
31 [31, 32]
20 [19, 23)
32 [33, 33]
11 [0, 22]
Frostbite
4003 [2871, 5163]
198 (198, 198]
268 [267, 269]
824 (258, 1390]
Gopher
4936 (3923, 5730]
598 [524, 672]
853 [832, 874]
4371 [3794, 4948)
Gravitar
1252, 322]
1130, 250]
1250, 455
Hero
8391 (6845, 10060]
615 [112, 1118]
7560 [7560, 7560]
2200 [1377, 3024]
IceHockey
[-3,-1]
115]
[=10, -4]
[000 .6]
Jamesbond
551 [534, 573]
46 [40, 50)
412 [310,
102 [80, 125]
Kangaroo
4833 12716, 7064]
555 [520, 590]
6830 (600, 13060)
685 (590, 780)
Krull
8660 (8 198, 9147]
6078 (5777, 6379]
7460 [696], 7959]
9088 (8624, 9552]
KungFuMaster
26150 (21973, 30490]
10400 (9320, 114800
17020 [7520, 26520]
12130 (11280, 12980]
MontezunuRevenge
[|0,00
[0,0]
MsPacman
4395 [3799, 5002]
826 [757, 896]
2950 [2490, 3410]
3171 [1873, 4469]
ThisGGe
7511 [7085, 7911]
2339 (2258, 2436]
6660 [6568, 6752]
7015 (6396, 7634]
Phoenix
4843 [4635, 5033]
570 (286, 854)
3996 (1843,4150)
4260 (3774
Pitfall
1-19, 11
|-50,
0
-66 1-122, -121
Pong
15
[-21, -20]
14
PrivatcEye
100 [100, 100]
45 [0, 90]
100 1100, 100]
90 1800 100]
Qbert
(2554, 4366]
[228,
[488,
[615, 880]
Riverraid
7362 [7062, 7630]
1997 829, 2165]
6860 [6391, 7330]
6342 (5450, 7234]
RoadRunner
27152 [19731, 34480]
3245 [2410, 4080]
21835 [20960, 22710)
35120 (33050, 37190]
Robotank
10 [9. 13]
713,11
7 [4. 10]
(3.9)
Seaquest
2660 (2055, 35791
305 [214, 400]
895 [834, 956]
1227 [378, 2076]
Skiing
-30000 (-30000, -30000]
-30000 [-30000, -30000]
-30000 [-30000, -30000)
-0000 (-30000, -30000]
Solaris
1262 1863, 1686
480 [0, 960]
710 [378, 1042|
1219 11198, 1240]
Spacelivaders
(429, 524]
(230, 255]
[263, 344]
[269, 323]
StarGunner
1146 (996, 1437]
1060 [880, 1240]
960 [920, 1000]
970 [960, 980]
Surround
[-7.-5]
-9 [-9, -0]
[-8.-7]
-7 [-9, -6]
Tennis
D 01
-24 [-24, 224
[-10,
[0,0]
TimePilos
3101 (2772, 3482]
2525 [1430, 3620]
2525 (2040, 3010]
1710 [1020, 2400)
Tutankham
1124,
90
78 12, 155]
UpNDown
26477 [1 (11956, 43260]
3180 (2321, aD40]
3045 [2667, 3424]
4523 (4422, 4625]
Venture
[0, 0]
65 [0, 130)
0]
[0
VideoPinball
18826 [15048, 23233]
12994 [10570, (S419)
9170 (7880, 10460)
19734 [14479, 24989)
WizardOrWor
1918 [1706, 21541
635 [480, 790]
1260 [670, 1850]
1440 (930, 1950)
YarsRevenge
27299 (23434, 30493]
6404 [6391. 6417]
23613 116984, 30244]
21163 [21047, 21280]
Zaxxon
3820 (2577, 4854)
[0,
690 (0. 1380]
[0,
Mean
-1.35 [-1.41, -1.29]
40.38 [-0.81, 0.05|
.0.6 0.89, 4083)
Median
-0.42 [-0.55, -0.42]
-0.16 (-0.16, -0.11]
-0.18 [-0.19, -0.12)
JQM
-0.56 [-0.60, -0.55
-0.20 [-0.22, -0.15]
-0.26 [-0.27, -0.25]
41

---

Published as a conference paper at ICLR 2025
Task
MR.Q
Revert
Non lineur model
MSE reward loss
Alien
2471 [1848, 3155]
66 [32. 100]
2167 [1426, 3169]
734 [617. 856]
Amidaar
1376, 4991
601
(364, 570]
177]
Assault
1125 [1094, 1160]
166 [359. 374]
1033 [998, 1068]
923 [873, 987]
Asterix
2216 [2081, 2346]
492 [425, 560)
1987 [1560, 2414]
2503 [2050, 3040]
Asteroids
602 [493, 689]
249 [239, 259]
563 (424, 740]
765 [624, 952]
Allantis
445022 (282338, 630730]
7310 [3140, 11480]
444370 (241216. 647524]
87410 [31610, 153510
BankHeist
[348, 749]
14 28]
1006 [961. [042]
[195, 309]
ZattleZone
16520 110560, 22760]
3550 [3300, 38001
23820 (203000, 26200]
4566 [3900, $2001
BeamRider
2007 [1855, 2194]
1510, 582]
1904 [1777, 2046]
1489 [1446, 1543]
Berzerk
430 [383,
315 [275, 355]
527 [498, 579]
354 [295, 398]
Bowling
50 [37, 65]
[27. 55]
69 [58, 82]
30 [27, 35]
Boxing
95 [94, 971
29 [8]]
95 191. 98]
93 [89, 98]
Breakout
25 [21, 32]
[1,
17 [17. 18)
11 [7. 18]
Contipede
14954 [13541, 16508]
2877 [2821, 29331
10053 13594]
11624 [8061,
ChopperCommand
4348 [3756, 5002]
530 [420, 6401
2918 (2006. 3830
2806 [1610, 3570]
Crazy Climbere
104766 (99290, 109629]
1700 (0, 3400]
(98066, 102282]
107220 [104990, 109150)
Defender
25962 [234066, 29182]
1917 [1780, 2055]
24283 (22936, 25425]
15231 [7875, 21485]
DemonArtack
4660 [4072, 5241]
149 [147, 151]
2467 [1370, 3548]
[288. 331]
DoubleDunk
[-11, -91
-23 1-24, -231
-10 I-14, -81
-10 |-14, -91
Enduro
1480 (1378, 1592]
[0,
1117 (1059, 1173]
800 (694, 899]
Derb
[-43,
1-91,
1-36,-30
1-75,
Freeway
31 [31, 32]
10,
31 [31, 32)
30 [30, 31]
Frostbite
[2871, 5163]
[151, 190]
2693 (834, 4491]
3954 [266, 7285]
Gopher
4936 [3923, 5730]
385 [280, 490]
7216 (3645, 11049]
2484 [886, 3514]
Gravitar
275 1232, 322]
82 180, 851
300 (156, 4191
140 [50, 310]
Hero
8391 (6845, 10060]
[0,
7633 (7577. 7693]
933 [0. 2799)
IceHockey
-2 [-3,-1]
144 [-1 II]
[-2,-1]
[-9,-7]
Jamesbond
573]
(462,
(280, 4551
Kangaroo
4833 (2716, 7064]
80 [0. 160]
5732 (3148, 7954]
2793 [540, 7060]
Krull
8660 [8 1989 9147]
10 [0, 20]
(8178, 8593]
8886 [7771, 9458]
KunsSeMasser
26150 (21973, 30490]
1815 1130, 3500]
24644 [19158, 30310)
19536 [12710, 31840]
MontezunnRevenge
12 [0. 341
|0,
240 [80, 4001
[0,
MsPacian
4395 [3799, 5002]
283 [182, 385]
3721 (3169, 4499)
1457 [1382, 1549]
Name GhimGaee
7511 [7085, 7911]
2675 [2609, 2742]
6162 [5941, 6450)
6091 [5656, 6475]
Phoenix
4843 [4635, 5033]
[517. 939]
(4300, 4800]
3638 [3317, 3959]
Pitfall
-8 [-]
-1012 [-2000, -24]
-3 [-9, 0]
-22 [-68, 0]
Pong
15 [13,
[-2 =21]
16 [15,
13 [8.
PrivatcEye
100 [100, 100]
50 [20, 80)
100 [100, 100]
33 [0. 100)
Qbert
3600 (2554, 4366]
137 [125, 150]
4295 (4006, 4586]
861 1785, 968]
Riverraid
7362 [7062, 7630]
1660 [1131, 2190]
6679 (5053, 7770]
3418 [482, 6556]
RoadRunner
27152 [19731, 34480]
[0, 0]
26678 [19418, 32016]
24583 [15870, 35950]
Robotank
13]
[3
10 [8.
[2,
Seaquest
2660 [2055, 3579]
134 [20, 248]
2344 [1541, 3450]
1676 [368, 2336]
Skiing
-30000 [-30000, -30000]
-3000 [-30000, -30000)
-30000 (-30000, -30000]
30000 (-30000, -30000)
Solaris
1262 1863, 1686]
878 18. 1748]
1280 [498, 2062]
870 [370, 1736]
Spacelwvaders
478 [429, 524]
207 [198, 216]
536 (405, 667]
253 [235, 278]
StarGunner
1146 (996, 1437]
930 17 1150]
1014 (994, 1048]
976 [9600, 1000]
Surround
-6 (-7,-5)
-9 [-9, -9]
-5 (-7
-8 [-9,
[]
[:24,
TimePilot
3101 [2772, 3482]
2195 [2040, 2350]
3822 (3282, 4418]
2376 [2070, 2880]
Tutankham
130 [124. 139]
14 [0. 30)
138 [127, 151]
37 [0. 112]
UpNDown
26477 111956, 43260]
1692 (1526, 1859]
34574 (9812, 71080]
4568 (4174, 4972]
Venture
D [0, 0]
[0, 0]
74 [0, 210)
0 [0.
VideoPinball
18826 [15048, 23233]
6961 [6299, 7623]
14689 [10497, [7911]
11244 19717, 12780)
WizardOrWor
1918 [1706, 2154]
$75 [550, 600)
1852 [1650, 2062]
1190 [1000, 1430]
YavsRevenge
(23434, 30493]
[0,
(25242, 32299]
Zaxxon
3820 (2577, 4854]
[0,
3144 [1128. 11]
0 [0.
Mean
-1.69 7-1,00
-0.07 [-0.32, 0.18]
-0.79 [-0.86, -0.73]
Median
-0.63 [-0.64, -0.63]
-0.01 [-0.02, 0.00]
-0.24 (-0.24, -0.17]
[-0.69, (0.65
[-0.05,
-0.23 [-0.24, -0.201
42

---

Published as a conference paper at ICLR 2025
Task
MR.Q
No reward scaling
No min
No LAP
Alien
2471 [1848, 3155]
2074 [1402, 2821]
2375 [1921, 3052]
2265 [1784, 2768]
Amidaar
1376, 4991
454]
[465, 686]
428]
Assault
1125 [1094, 1160]
1235 [1140, 1330]
1154 [967. 1254]
1095 [1027. 1170]
Asterix
2216 [2081) 2346]
2476 [205 2901]
2881 [2600, 3045]
2503 (2250, 2800]
Asteroids
602 [493, 689]
614 [520, 711]
769 [713, 878]
509 (447,568)
Allantis
445022 (282338, 630730]
608658 (262742, 9466668
3862133 (77480, 961750)
329941 (196522, S06576]
BankHeist
542 [348, 749]
806]
304 [205, 436]
235 388]
ZattleZone
16520 110560, 22760]
13660 [8040, 18780]
21300 (20100, 22200)
13650 (9270, 17290]
BeamRider
2007 [1855, 2194]
1989 [1947, 2031]
1989 [1948, 2016]
[1194, 1840)
Berzerk
430 [323
427 [361, 524]
601 [398, 708]
531 [464, 593]
Bowling
50 [37, 65]
78 [66, 88]
38 [28, 50]
[58, 78]
Boxing
95 [94, 971
91 96]
95 [92, 971
95 [94,
Breakout
25 [21, 32]
22 [20, 26]
27 [24, 30)
21 [18, 25
Contipede
14954 [13541, 16508]
(12806,
11288 (8365,
12236 (9602,, 14654]
ChopperCommand
4348 [3756, 5002]
2796 [2228, 3378]
3283 42730,4770)
3490 [2724, 4341]
Crazy Climbeeeer
104766 (99290, 109629]
(94550, 112412]
110046 (107390, 113610]
(77424, 100499)
Defender
25962 [234066 29182]
30912
[278711, 33695]
32336 (27280, 38695]
32385 [29809, 34752]
DemonArtack
4660 [4072, 5241]
4893
(4222,, 5345]
2468 [712, 3513]
4280 (2797, 6098)
DoubleDunk
[-11, -91
-11 [-14,
J-13, -71
-11 1-14,
Enduro
1480 (1378, 1592]
1450 [1311, 1593]
967 [0. [461]
1117 (883, 1287]
Derbyng
[-43,
-22 F:25,
-17 1-28,
-35 [-46,
Freeway
31 [31, 32]
25 [13, 32]
32 [31, 34]
32 [32, 32]
Frostbite
[2871, 5163]
3247 [1532, 4771]
2401 [267, 4457]
[144,
Gopher
4936 [3923, 5730]
5802 [1467, 13322)
11774 [7552, 18634)
11483 (4836, 23229]
Gravitar
275 [232, 322]
256 [215, 305]
393 1185,
329 [286, 382]
Hero
8391 (6845, 10060]
8775 (7575, 11125]
7594 [7525, 7670)
7519 (7379, 764D)
IceHockey
-2 [-3,-1]
[-5, -2]
[-3,-1]
[-7. 4]
Jamesbond
573]
[460,
[406, $28]
Kangaroo
4833 (2716, 7064]
6148 [3412, 8600]
8033 (5540, 9190]
4616 [23992 6998]
Krull
8660 [8 1989 9147]
8878 (7898, 9491]
8430 (6464, 9987]
8535 [8189, 8924]
KunsSeMasser
26150 (21973. 30490]
24292 [18954, 29568]
25553 [23960, 27040]
23422 121084, 25671]
MontezunnRevenge
12 [0. 341
[0,00
190 1100, 3701
20 [0, 501
MsPacman
4395 [3799, 5002]
4086 [3847, 4413]
3860 [3394, 4470]
3602 [3051, 4116]
Name GhimGaee
7511 [7085, 1911]
8323 [7308, 9338]
7992 (7194, 8458]
7681 [6795, 8351]
Phoenix
4843 [4635, 5033]
4940 (2541
4780 [4605, 5008]
4717 [4388, 5016]
Pitfall
-8 [-19.-1]
0 [0
0 [0,0]
-I [-3, 0]
Pong
15 [13,
[14,
[19, 19]
[10, 17]
PrivatcEye
100 [100, 100)
40 [0, 80]
125 [100, 177]
50 [20,
Qbert
3600 (2554, 4366]
2848 [1410, 4287]
6567 [4638, 7722
3365 [2590, 4045]
Riverraid
7362 [7062, 7630]
6669 [5 104, 7779]
7464 [7213, 7688]
6191 [5522, 6889]
RoadRunner
27152 [19731, 34480]
37306 (32906, 40582]
38703 (35530, 45050)
33950 (27807, 39550)
Robotank
13]
Seaquest
2660 [2055, 3579]
1998 [1141, 2626]
2005 [1596, 2348]
1421 [986, 1782]
Skiing
-30000 [-30000, -30000]
-3000 (-30000, -30000)
-0000 (-30000, -30000)
-30000 [-30000, -30000)
Solaris
1262 1863, 1686]
1175 1660, 1770
772 [216, 1606]
886 [637, 1106]
Spacelwvaders
478 [429, 524]
458 [431, 493]
$15 (488, 532]
494 [459, 5300
StarGunner
1146 (996, 1437]
1074 (964, 1266)
1993 [1240, 22010]
[976, 1005]
Surround
-6 (-7,-5)
-5 (-7,-5)
-5 [-6, -4]
-7 [-9.-5]
[]
0]
20]
TimePilot
3101 [2772, 3482]
3118 [2620, 3618]
4433 (4190, 4830]
3616 [2880, 4259)
Tutankham
130 [124. 139]
144 [87, 184]
156 [154, 160)
125 [99, 150]
UpNDown
26477 111956, 43260]
13859 (6682, 21316]
37177 (17986, 54020)
17527 (8653, 28247]
Venture
D [0, 0]
0 [0,
0 [0, DJ
0 [0,
VideoPinball
18826 [15048, 23233]
18345 115176, 21920]
22721 [18151, 285
20490 [14091 2779
WizardOrWor
1918 [1706, 2154]
1906 [1070, 2984]
1790 [1560, 2010)
1549 [1375, 1 1708)
YarsRevenge
(23434, 30493]
31727]
121118, 33567]
Zaxxon
3820 (2577, 4854]
1336 [0, 3300]
6746 [4940, 7760]
1488 [369, 2733]
Mean
0.18 [-0:25, 0.56]
0.13 [-0.10, 0.58]
-0.13 [-0.38, 0,
Median
-0.00 [-0.01. 0.00]
0.01 [0.00, 0.04]
-0.03 [-0.07, -0.00)
IQM
[-0.02, 0.03]
10.03, 0.06
-0.04 1-0.08, -0.01]
43

---

Published as a conference paper at ICLR 2025
Task
MR.Q
No MR
1-step return
No unroll
Allen
2471 [1848, 3155]
2886 [2458, 3315]
1342 [1226, 1454]
3056 (2300, 3614]
Amidaar
1376, 4991
[204, 4211
240 [213, 287]
[335, 471]
Assault
1125 [1094, 1160]
1105 [1075, 1142]
1045 [904, 1134]
1079 [1003, 1140]
Asterix
2216 [2081, 2346]
2298 [1920, 2731]
2708 [21 3710]
2388 [2190, 2740]
Asteroids
602 [493, 689)
485 584)
7233 (689. 783]
581 (406, 766]
Allantis
445022 (282338, 630730]
12834 150058, 17186]
60986 (25200, 874601
57426 (40850, 88730]
BankHeist
542 [348, 749]
726]
123 160]
783 [231, 1080]
ZattleZone
16520 [10560, 22760]
22000 (12100, 29080]
4700 [3600, 58001
23713 [19600, 26100]
BeamRider
2007 [1855, 2194]
1849 [1726, 2004]
1438 [1213, 1604]
1389 [1274, 1535)
Berzerk
430 4723
4.77 [333, 537]
361 [290, 400]
443 [399, 489]
Bowling
50 [37, 65]
84 (73,
44 [30, 67]
72 [60, 88]
Boxing
95 [94, 971
92 1899 95]
93 192. 951
95 [94, 96]
Breakout
25 [21, 32]
15 114, 18)
14 [7. 23]
27 [17, 44]
Contipede
14954 [13541, 16508]
10517 [9157. 11949]
8927 17098, 10732]
11984 [11005, 13691]
ChopperCommand
4348 [3756, 5002]
3394 (764)
2520 11980, 29001
2866 [1670, 3990]
Crazy Climber
104766 (99290, 109629]
83734 (71466, 93070)
(65140, 70290]
104076 (82650, 114820]
Defender
25962 [234066, 29182]
14469 [9247, 18763]
9658 [3245, 17870]
34718 (29120, 44180]
DemonArtack
4660 [4072, 5241]
746 [527. 994]
1861 11675, 1990]
2840 (2064, 3618)
DoubleDunk
[-11, -91
-14 [-18, -131
-7 1-10,
-11 I-14,
Enduro
1480 (1378, 1592]
897 [784, 1075]
1064 [1055, [069]
1075 [1018, 1131]
Derbyng
[-43, -271
1:30, -8]
[955
1-63,
Freeway
31 [31, 32]
26 [13, 33)
32 [32, 35]
32 [32, 33]
Frostbite
4003 [2871, 5163]
3098 [1645, 4358]
[254, 3182]
4182 (3473, 5451]
Gopher
4936 [3923, 5730]
1833 (1552, 21
5597 [4242, 6812)
4746 (2378, 6822]
Gravitar
275 [232, 322]
89 [0, 267]
166 (70]
270 1145, 415]
Hero
8391 16845, 10060]
7584 [7548, 7644]
7594 (7544, 7676]
6879 [5384, 7694]
IceHockey
-2 [-3.-1]
-5 [-8,
-6 [-8, :5]
-2 [-4,-]]
Jamesbond
573]
(406, 4621
[495,
[460, 625]
Kangaroo
4833 (2716, 7064]
7508 [5608, 9500]
6940 (1520, 10460]
9166 (8520, 9500]
Krull
8660 1989 9147]
8403 [7433, 9087]
7386 [6611, 788
7785 (7176, 8271]
KunsSuMasser
26150 (21973. 30490]
19066 [15450, 22028)
15300 [14760, 16350]
29020 150,0 30700)
MontezunnRevenge
12 [0. 341
010,00 DJ
[0, 0]
0 [0, 0J
MsPacman
4395 [3799, 5002]
3297 [2679. 3908]
2282 [1956, 2592]
2839 [2612, 2954]
Name ThimGaee
7511 [7085, 7911]
3638 [32m, 3999]
5590 [5026, 5941]
6529 [6069, 6983]
Phoenix
4843 [4635, 5033]
4101 [3752, 4503]
[3435, 4306]
4941 (4613, $193]
Pitfall
-8 [-19,-1]
-27 [-67, -5]
0 [0, 0]
0 [0, 0]
Pong
15 [13, 17]
12 [8, 16]
14 [9,
12 (8, 16]
PrivatcEye
100 [100, 100)
3068 [24, 9080]
100 [100, 100]
100 [100, 100]
Qbert
3600 (2554, 4366]
4491 [3362, 5870]
2517 [2032, 3105]
4220 [3918, 4508]
Riverraid
7362 [7062, 7630]
7479 [6818, 8239]
6733 [5928, 7759]
5856 [3607, 7855]
RoadRunner
27152 [19731, 34480]
29182 [23250, 35114]
36145 [33590, 38700]
37636 [33610, 40160)
Robotank
19. 13]
11 [6,
[6.
12
Seaquest
2660 [2055, 3579]
1556 [1248, 1866]
2166 [2154, 2178]
2194 121 2284]
Skiing
-30000 [-30000, -30000)
30000 (-30000, -30000)
33000 (-30000, -30000)
50000 (-30000, -30000]
Solaris
1262 1863, 1686]
552 [319, 881]
1107 [662, 1552]
1088 [638, 1386]
Spacelwvaders
478 [429, 524]
550 (476, 632]
389 [382, 396]
551 [502, 596]
StarGunner
1146 (996, 1437]
1272 [1020, 1730]
1290 [1000, 1580]
1423 [990, 2220]
Surround
-6 (-7,-5)
-8 [-9, -7]
-5 [-6, -5]
-5 [-10,, I]
[|
[-1,0]
TimePilor
3101 [2772, 3482]
2440 [1540,
2535 [2030, 3040]
2236 [1170, 3810)
Tutankham
130 [124. 139]
112 [65, 148]
151 [120, 182]
148 [127, 181]
UpNDown
26477 111956, 43260]
25451 (17297, 34036]
4477 (2993, 5962]
31342 (31770, 84771]
Venture
D [0, 0]
[0, 0)
[0. 0]
[0, 0)
VideoPinball
18826 [15048, 23233]
8524 [3899, 12519]
13506 17955. 19058]
27525 (22242, 35815]
WizardOrWor
1918 [1706, 2154]
2058 [1632, 2640]
1545 [1490, 1600]
1573 [1430, 1700)
YarsRevenge
(234344 30493]
30666 (28659, 33225]
18513 116940 200881
24565]
Zaxxon
3820 (2577, 4854)
5758 [4712, 6962]
330 [0, 660)
[0, 0]
Mean
-0.78 E..88, -0.69]
-0.70 [00811 -0:59]
-0.33 [-0.41 0.28
Median
-0.06 [-0.10, -0.01]
-0.12 [-0.12, -0.07]
-0.00 [-0.02, 0.00]
IQM
1-0.14,-0.05
(60.16, -0.131
1-0.04, -0.00)
44


Reviews:
[{"review_id": "lHyAtq3pvF", "reviewer": "Reviewer_qhCX", "summary": "The authors propose a generalist, model-free deep reinforcement learning algorithm named MR.Q, which integrates several advanced algorithmic techniques to achieve performance improvements and reduce training costs across multiple benchmark datasets.", "questions": "1. What exactly is the objective of the model-based elements mentioned by the authors? It seems that all techniques described are already present in existing model-free reinforcement learning frameworks.\n2. The authors only conducted experiments on the Gym, DMC, and Atari benchmarks, which are classic but relatively homogeneous environments. Claiming generality might be overstated given these limited evaluation. For example, the widely accepted generalist algorithm PPO has demonstrated its effectiveness across a broad range of tasks. To establish the generality of this method, additional testing on tasks with some heterogeneity from these benchmarks, such as fine-tuning on large models, would strengthen the claim.\n3. Equation (7) seems to be derivable directly from Equation (3). Could you clarify the phrase \"from this insight\"?\n4. The authors describe the linear decomposition of the value function as the motivation, yet the final algorithm design and ablation studies prefer using common nonlinear function approximations. The authors claim that Equations (5-6) are inspired by model-based representations, but related ideas have already appeared in existing model-free algorithms. While this work is innovative and effective from an algorithmic design perspective, the theoretical sections seem somewhat retrofitted to fit the empirical results.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. MR.Q combines various state-of-the-art techniques and demonstrates promising performance gains with a single set of hyperparameters across multiple benchmarks, showing both efficacy and resource efficiency.\n2. The paper provides detailed ablation studies, offering insights that may benefit future research in this field.", "weaknesses": "The theoretical motivation presented by the authors is somewhat confusing and challenging to interpret. Please refer to Question 3 for details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pzMFekgZQh", "reviewer": "Reviewer_b3JB", "summary": "This paper presents a model-free deep RL algorithm, designed to achieve consistently high performance across RL settings. The method (MR.Q) is based on the widely accepted principle that self-supervision (world models) can provide an effective state encoding for policy and value learning. Additional components are added to ensure hyperparameter robustness and the method is evaluated with a single set of hyperparameters in each setting. A theoretical analysis is provided, suggesting an approximately linear relationship between model-based and value-based representations. MR.Q is evaluated on MuJoCo, DMC, and Atari benchmarks, with comparisons to a handful of algorithms and an extensive number of ablations.\n\nAs detailed below, I was unable to find some key details regarding the author's experiments, so my current score is provisional. I am highly open to raising or lowering my score following discussion with the authors.", "questions": "Were the baselines tuned, and how was this done if so?\n\nDo you have evidence that A) MR.Q is less hyperparameter sensitive than baselines, or B) the best single set of hyperparameters for MR.Q achieves higher performance than the equivalent for a baseline?\n\nAs far as I can tell, the proposed symexp loss will have higher resolution at higher reward values. This is counterintuitive, as one would expect precision to increase for values near 0. Can the authors explain this?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The paper is very well presented and has a clear structure.\n* The core approach - using world model representations to support policy learning - is sound and well-motivated in prior literature.\n* Strong efforts are made to make the model hyperparameter-insensitive, which is particularly valuable in RL where hyperparameter tuning is often ignored or exploited to give the appearance of improved performance.\n* The evaluation is extensive in the number of benchmarks and ablations. A consistent set of hyperparameters is used by the method, strengthening the claim of robustness.", "weaknesses": "## Hyperparameters\nOne of the primary aims of the paper is hyperparameter robustness, with multiple components being designed to improve robustness and a single set of hyperparameters being used to evaluate MR.Q. However, aiming for robustness does not allow hyperparameters to be ignored in evaluation. In fact, *more* attention should be paid to hyperparameters, to demonstrate that MR.Q is less sensitive hyperparameters than baseline methods, or performs better when all algorithms are constrained to a single hyperparameter setting. \n\nUnfortunately, the paper does the opposite and provides no detail (that I could find) regarding the hyperparameter settings of the baseline algorithms, nor how or if they were tuned. This is a critical weakness of the paper, as the possibility of untuned baselines undermines the claimed performance improvements.\n\nOne example of this is that DreamerV3 outperforms MR.Q on only 1/4 benchmarks (Atari). Notably, Atari is also the only benchmark for which the results come from the reference work. For the remaining 3 benchmarks, the authors run DreamerV3 themselves. This does not imply that the authors failed to tune DreamerV3 or that the comparison was unfair, however, given the lack of detail regarding their tuning procedure or the hyperparameter sensitivity of the methods, the result is seriously undermined.\n\nAnother example is that the paper seems to maintain model sizes from the original implementations of each algorithm. However, the size of the model is not defined by the algorithm, so it would be much more convincing to normalize the parameter count between algorithms and show that MR.Q achieves superior performance at the *same* size, or has a better scaling curve.\n\n## Baselines\nSome key algorithms are missing, most notably PPO. It is not expected that the authors compare to every popular RL algorithm, but PPO is undoubtedly the closest thing to a \"general-purpose\" RL algorithm in current literature.* A number of recent methods have also claimed to serve as \"general-purpose\" RL algorithms (e.g. PQN, Gallici et al., 2025). These would make for valuable comparisons in future work but are not required for acceptance.\n\n*As of 31st Oct, PPO has 5460 citations in 2024, compared to 269 (Dreamer v3), 32 (TD7), 39 (TD-MPC2), 1531 (TD3). Not a perfect proxy for the methods' generality, but certainly indicative.\n\n## Ablation study\nThe ablation study fails to provide the statistical significance of the results and lacks analysis. Furthermore, the \"reverting to theory\" ablations are highly unsurprising so provide little contribution, and many of the remaining ablations show minimal performance gains. Some of these components are designed to handle edge cases (e.g. reward scaling handling environments without normalized reward), meaning the benefit is unclear from mean performance across a suite, but may be apparent by examining the performance distribution. For these in particular, further analysis of the results would strengthen the contribution of each component.\n\n## Theory\nThe theory is based on highly constrained linear assumptions and provides little justification for the proposed method. The idea of using world model representations for policy learning is intuitive and can be explained much more clearly without sacrificing correctness. I understand there is a pressure on authors to provide \"theoretical rigour\", but in this case it adds little to the paper and would be better replaced with an extended analysis of the empirical results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fTumdA6wMS", "reviewer": "Reviewer_dWbu", "summary": "The paper proposes to leverage model-based representations for model-free reinforcement learning. Model-based approaches are only used for learning the representation, but no planning or simulation is employed. The authors integrate model-based representations in the proposed method MR.Q. They evaluate MR.Q on 3 benchmarks (Gym, DMC, Atari) and find that it leads to faster learning speed and improved performance compared to competitors, albeit not consistently across domains.", "questions": "- Could you clarify your use of “generalist”? \n- There seems to be a clear trend of Dreamerv3 underperforming on continuous control environments. Any intuitions why? \n- How do the Training FPS of MRQ. in Figure 1 compare to other non-planning baselines on DMC-Visual (TD7) and Atari (Rainbow, DQN)?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors propose an algorithm named, MR.Q that performs well across domains. They conduct well-selected ablations on MR.Q to highlight what effect their design decisions have on the algorithm. The empirical results are broad and seem sound. Generally, a useful contribution to the field.", "weaknesses": "The authors argue that the benefits of model-based approaches may stem from their learned representations, rather than from their planning abilities. While this may be true to some degree, we assume that this is highly specific to the domains they conduct experiments on. In particular, Gym and DMC environments and most Atari games do not require much planning to be solved. However, we know that in other domains planning plays a critical role. Therefore, we believe it is important for the authors to clarify that the benefits of model-based representations alone depend on the domain at hand, to avoid any misconceptions.\n\nThe authors refer to their method as “generalist”, which is a term commonly used for multi-task agents in the literature. However, all conducted experiments are in single task settings. Instead, “generalist” in the context of this paper refers to being robust to different environments (reward scales, dynamics, with the same hyperparameters). We therefore strongly encourage the authors to rethink this terminology to avoid confusions.\n\nThe results for Dreamerv3 are surprisingly bad on continuous control tasks. This suggests that this baseline is purely tuned for these environments. \n\nThe proposed approach comes with a large list of hyperparameters, which may limits its practical applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JPcv0NRMVp", "reviewer": "Reviewer_fYrs", "summary": "This paper introduces MR.Q, a model-free RL algorithm that uses model-based representations to generalize across a variety of domains with the same set of hyper-parameters. MR.Q leverages the fact that the fixed-point between linear reconstructions of dynamics and value are the same. MR.Q relaxes the theory to show that non-linear value functions using this learned representation improve performance. The algorithm trains a policy by using a deterministic policy gradient similar to DDPG. To prevent trivial solutions, the dynamics reconstruction loss uses the next state as a target, as opposed to the next state-action encoding. A number of other tricks are used, including a two-hot representation of reward, target policy noise from TD3, pre-activation regularization, Huber loss, and scaling by the average reward. These components are tested in a series of ablations, and evaluations are conducted over Gym, DMC, and Atari, comparing to DreamerV3, TD-MPC2, TD7, Rainbow, and DQN.", "questions": "Some questions:\n1) Is there a reason that TD-MPC2 and TD7 were chosen over Truncated Quantile Critics (TQC) and Randomized Ensembled Double Q-Learning (REDQ)? Are there any papers showing your baselines to be superior?\n2) Are there any experiments evaluating whether a non-linear reward and transition reconstruction might be superior?\n3) Have you evaluated you algorithm with sequence models than can handle POMDPs?\n4) Have you compared to other self-supervised model-based algorithms (esp. those using contrastive learning or those designed for POMDPs?)\n5) You mention that the number of frames is not equal to the number of time-steps due to sticky actions. If you plot the number of frames on the x-axis, are the results still the same? Does your algorithm require more frames than others?\n6) It looks like the curves for Atari are not converged. Could you run longer?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper is well-written and address the import question of robustness of RL algorithms across various domains. The paper is theoretically motivated, and performs many ablations. The proposed algorithm is clear and if the evaluations hold, could prove to be of great interest to the community.", "weaknesses": "The primary weaknesses are evaluation and novelty. A few points are listed here, and some clarifying questions in the questions section:\n1) While it is not possible to evaluate every benchmark, the Minecraft benchmark from Dreamer, which I believe tests long-term credit assignment, is notably missing. It could be great to see how this method holds up when reward is sparse, exploration is difficult, and policy gradient variance may be high. Procgen is also missing, which may be one of the harder domains in the Dreamer paper.\n2) The model-based representations are not particularly novel. For example, consider VariBAD (Zintgraf et al., 2020), which reconstructs reward and dynamics functions. \n3) One piece of novelty is that the reconstruction of reward and transitions use a linear function of the same latent encoding. However, experiments show that a non-linear value reconstruction is useful, and it may be the same for the model-based objectives as well. Are there experiments testing this?\n4) It would be great to see comparisons to other baselines here that use contrastive learning as well, e.g., Neural Predictive Belief Representations (Guo et al., 2018).\n5) I am also concerned that the method cannot handle general POMDPs. Is it true that only the state and action are passed the policy and not history? If you are only using a latent that reconstructs the reward and dynamics, this statistic must condition on history, and is not general enough for any POMDP, but only a distribution of MDPs, as in meta-RL (Zintgraf et al., 2020).\n6) It would be great to see a comparison to your method, but with additional synthetic trajectories simulated by your learned model. Without this experiment, it is hard to know if the compromise between model-based RL and model-free RL is the right trade-off.\n\nMinor - Clarity:\nA) Could you further clarify and motivate the two-hot classification and the symexp so the reader does not have to look externally?\nB) Could you give more motivation and explanation of the two equalities in Equation (5)? Also, how is Wp defined for when given a specific time-step, t?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)