Learning Transformer-based World Models with Contrastive Predictive Coding
Authors: Maxime Burchi
Radu Timofte
Venue: ICLR 2025 Spotlight (2025)
Keywords: model-based reinforcement learning
transformer network
contrastive predictive coding
Primary area: reinforcement learning
TL;DR: We introduce TWISTER, a Transformer model-based reinforcement learning algorithm using action-conditioned Contrastive Predictive Coding to learn high-level feature representations and improve the agent performance.

Abstract: The DreamerV3 algorithm recently obtained remarkable performance across diverse environment domains by learning an accurate world model based on Recurrent Neural Networks (RNNs). Following the success of model-based reinforcement learning algorithms and the rapid adoption of the Transformer architecture for its superior training efficiency and favorable scaling properties, recent works such as STORM have proposed replacing RNN-based world models with Transformer-based world models using masked self-attention. However, despite the improved training efficiency of these methods, their impact on performance remains limited compared to the Dreamer algorithm, struggling to learn competitive Transformer-based world models. In this work, we show that the next state prediction objective adopted in previous approaches is insufficient to fully exploit the representation capabilities of Transformers. We propose to extend world model predictions to longer time horizons by introducing TWISTER (Transformer-based World model wIth contraSTivE Representations), a world model using action-conditioned Contrastive Predictive Coding to learn high-level temporal feature representations and improve the agent performance. TWISTER achieves a human-normalized mean score of 162% on the Atari 100k benchmark, setting a new record among state-of-the-art methods that do not employ look-ahead search. We release our code at https://github.com/burchim/TWISTER.

Published as a conference paper at ICLR 2025
LEARNING TRANSFORMER-BASER WORLD MODELS WITH CONTRASTIVE PREDICTIVE CODING
Maxime Burchi, Radu Timofte
Computer Vision Lab, CAIDAS & IFI, University of Würzburg, Germany
maxime wwb.uuiiihe. (uniiiinii-- -er-riiuuuuuuuu-urrrr wwwrzburg.cc..
ABSTRACT
The DreamerV3 algorithm recently obtained remarkable performance across di- verse environment domains by learning an accurate world model based on Re- current Neural Networks (RNNs). Following the success of model-based rein- forcement learning algorithms and the rapid adoption of the Transformer archi- tecture for its superior training efficiency and favorable scaling properties, recent works such as STORM have proposed replacing RNN-based world models with Transformer-bite -aased world models using masked self-attention. However, despite the improved training efficiency of these methods, their impact on performance re- mains limited compared to the Dreamer algorithm, struggling learn competitive Transformer- based world models. In this work, we show that the next state pre- diction objective adopted in previous approaches is insufficient fully exploit the representation capabilities of Transformers. We propose to extend world model predictions longer time horizons by introducing TWISTER (Transformer bessoommeer-b-eeddddd World model wIth contraSTivE Representations), world model using action- conditioned Contrastive Predictive Coding learn high-level temporal feature representations and improve the agent performance. TWISTER achieves human- normalized mean score of 162% on the Atari 100k benchmark, setting new record among state-of-the-art methods that do not employ look-ahead search. We release our code at https:/ /gi thub. www.ccoo//bbeeeeee.. bww.hii//TIIITE..... TWISTER.
INTRODUCTION
Deep Reinforcement Learning (RL) algorithms have achieved notable breakthroughs in recent years. The growing computational capabilities of hardware systems have allowed researchers to make significant progress, training powerful agents from high-dimensional observa- tions like images (Mnih al.) 2013) or videos (Hafner ct al. 2020) using deep neural networks (LeCun et al. 2015 function approximations. Following the rapid adoption of Convolutional Neural Networks (CNNs) (Le- Cun al. 1989) in the field of Computer Vision for their efficient pattern recognition ability, neural networks were applied visual reinforcement learning prob- lems and achieved human to superhuman performance in challenging and visually complex domains like Atari games (Mnih et al., 20155] Hessel et al., 2018). the game of Go (Silver et al. 2018, Schrittwieser et al.] (2020), StarCraft II (Vinyals et al. 2019) and more recently, Minecraft [Baker et al. 2022 Hafner et al. (003)).
M ean Human Normelized Score (Atari 100k)
Parsheman boses word mode
1NN-aased
105%
112%
100%
50%
Median Nermalized (Atari 100k)
80%
77%
58%
53%
42%
25%
25%
0%
433 4030
Figure I: Human--ormaalizd mean and median scores of recently published model-based methods on the Atari 100k benchmark. TWISTER outperforms other model-based approaches. TWM, IRIS, STORM and A-IRIS employ Transformer-based world model while DreamerV3 uses RNN-based model.
Following the success of neural networks in solving re- inforcement learning problems, madel-based approaches learning world models using gradient backpropagation were proposed to reduce the amount of necessary in- teraction with the environment to achieve strong re-

---

Published as a conference paper at ICLR 2025
sults (Kaiser et al., 2020; Hafner et al., 2020; 2021: 2023] Schrittwieser et al.) 2020) World mod- els (Suttoon 1991; Ha Schmidhuber 2018) summarize an agent's experience into a predictive model that can be used in place of the real environment to learn complex behaviors. Having cess to a model of the environment enables the agent to simulate multiple plausible trajectories in parallel, improving generalization, sample efficiency and decision-making via planning.
Design choices for the world model have tended to- ward Recurrent Neural Networks (RNNs) (Hafner et al. 2019) for their ability to model temporal relationships effectively. Following the success of the Dreamer algo- rithm (Hafner et al., (020) and the rapid adoption of the Transformer architecture (Vaswani et al. 2017) for its su- perior training efficiency and favorable scaling proper- ties compared to RNNs, research works have proposed replacing the one-layer recurrent-based world model of Dreamer with a Transformer-baser world model using masked self-attention (Chen et al., 2022 Micheli et al. 2023 Robine et al. 2023). However, despite the im- proved training efficiency of these methods, their impact on performance remains limited compared to the Dreamer algorithm, struggling learn competitive Transformer- based world models. Zhang et al. (2024) suggested that these findings may be attributed to the subtle differences between consecutive video frames. The task of predicting the next video frame in latent space may not require a complex model in contrast to other fields like Neural Language Modeling (Kaplan et al. 2020) where a deep understanding of the past context is essential to accurately predict the next tokens. As shown in Figure 2 the cosine similarity between adjacent latent states of the world model is very high, making it relatively straightforward for the world model to predict the next state compared to more distant states. These findings motivate our work to complexify the world model objective by extending predictions to longer time horizons in order to learn higher quality feature representations and improve the agent performance.
0.6
0.5
5im
0.4
0.3
10
Num steps future (K)
Figure 2: Cosine Similarities between TWISTER latent state 2t and future states 24+ aggregated over all 26 games of the Atari 100k benchmark. We show average similarities over 5 seeds.
In this work, we show that the next latent state prediction objective adopted in previous approaches is insufficient to fully exploit the representation capabilities of Transformers. We introduce TWISTER, Transformer model-based reinforcement learning algorithm using action-conditioned Contrastive Predictive Coding (AC-CPC) to learn high-level temporal feature representations and improve the agent performance. CPC (Oord al. 2018) was initially applied to speech, image, and text domains as pretraining pretext task. It also showed promising results on DeepMind Lab tasks (Beattie et al.) 2016) being used as an auxiliary loss for the A3C agent (Mnih et al. 2016). Motivated by these findings, we apply the CPC objective to model-based reinforcement learning by conditioning CPC predictions on the sequence future actions. This approach enables the world model to accurately predict the feature representations of future time steps using contrastive learning. As shown in Figure TWISTER sets a new record on the commonly used Atari 100k benchmark (Kaiser et al.) 2020) among titte-of-theaate methods that do not employ look-ahead search, achieving human- normalized mean and median score of 162% and 77%, respectively.
RELATED WORKS
2.1 MODEL-BASED REINFORCEMENT LEARNING
Model-based reinforcement learning approaches use a model of the environment to simulate agent trajectories, improving generalization, sample efficiency, and decision-making via planning.. Follow- ing the success of deep neural networks for learning function approximations, researchers proposed to learn world models using gradient backpropagation. While initial works concentrated on simple environments like proprioceptive tasks (Silver al.) 2017 Henaff et al. 2017 Wang et al.) 2019
Wang & Ba 2020) using lww-dimensional observations, more recent works focus on learning world
models from high-dl nigh-dimensional observations like images (Kaiser et al.) 2020 Hafner et al.) 2019).
One of the earliest model-based algorithms applied to image data is SimPLe (Kaiser et al. 2020).
which proposed to learn world model for Atari games in pixel space using a convolutional autoen-
2

---

Published as a conference paper at ICLR 2025
coder. The world model learns to predict the next frame and environment reward given previous observation frames and selected action. It is then used to train a Proximal Policy Optimization (PPO) agent (Schulman et al.] 2017) from reconstructed images and predicted rewards. Concur- rently, PlaNet (Hafner et al. 2019) introduced a Recurrent State-Space Model (RSSM) using a Gated Recurrent Unit (GRU) CCho et al. 2014) learn world model in latent space, planning ing model predictive control. PlaNet learns a convolutional variational autoencoder (VAE) (Kingma) & Welling 2013) with a pixel reconstruction loss to encode observation into stochastic state rep- resentations. The RSSM learns to predict the next stochastic states and environment rewards given previous stochastic and deterministic recurrent states. Following the success of PlaNet on DeepMind Visual Control tasks (Tassa et al., 2018). Dreamer (Hafner et al., 2020) improved the algorithm by learning an actor and a value network from the world model representations. DreamerV2 (Hafner et al. 2021) applied the algorithm to Atari games, utilizing categorical latent states with straight- through gradients (Bengio et al. 2013) in the world model to improve performance, instead of Gaussian latents with reparameterized gradients (Kingma & Welling (013). DreamerV3 (Hafner
et al. 2023) mastered diverse domains using the same hyper-parameters with a set of architectural changes to stabilize learning across tasks. The agent uses symlog predictions for the reward and value function to address the scale variance across domains. The networks also employ layer nor- malization (Ba et al. 2016) to improve robustness and performance while scaling larger model sizes. It stabilizes policy learning by normalizing the returns and value function using an Expo- nential Moving Average (EMA) of the returns percentiles. With these modifications, DreamerV3 outperformed specialized model-free and model-based algorithms in a wide range of benchmarks.
In parallel the Dreamer line of work, Schritschit et al. (2020) proposed MuZero, model-based algorithm combining Monte-Carlo Tree Search (MCTS) (Coulom 2006) with a powerful world model to achieve superhuman performance in precision planning tasks such as Chess, Shogi and Go. The model is learned by being unrolled recurrently for K steps and predicting environment quantities relevant to planning. The MCTS algorithm uses the learned model simulate environ- ment trajectories and output action visit distribution over the root node. This potentially better policy compared to the neural network one is used to train the policy network. More recently, Ye et al. (2021) proposed EfficientZero, sample efficient version of the MuZero algorithm ing self-supervised learning to learn a temporally consistent environment model and achieve strong performance on Atari games.
2.2 BRANSFORMER-BASED WORLD MODELS
Recent works have proposed replacing RNN-based world models by Transformer- based architec- tures using self-attention to process past context. TransDreamer (Chen et al.) (022) replaced Dream- er'3's RSSM by a Transformer State-Space Model (TSSM) using masked self-attention to imagine future trajectories. The agent was evaluated on Hidden Order Discovery tasks requiring long-term memory and reasoning. They also experimented on a few Visual DeepMind Control (Tassa et al.) 2018) and Atari (Bellemare et al.. 2013) tasks, showing comparable performance to DreamerV2. World Model) proposed a similar approach, encod- ing states, actions and rewards as distinct successive input tokens for the autoregressive Transformer. The decoder also reconstructed input images without the world model hidden states, discarding past context temporal information for image reconstruction. More recently, STORM (Zhang et al.) 2024) (Stochastic Transformer-bat based wORld Model) achieved results comparable to DreamerV3 with bet- ter training efficiency on the Atari 100k benchmark. STORM proposed to fuse state and action into
TWM (Robine et al.) 2023) Srrrasfoommr-batte
Table 1: Comparison between TWISTER and other recent model- based approaches learning a world model in latent space. Tokens refers to tokens used by the autoregressive world model. Latent (24) is image representation while hidden (h₂) is world model hidden state carrying historical information.
Attributes
TWM
IRIS
DreamerV3
STORM
A-IRI
TWISTER (ours)
World Model
Transformer
Transforner
GRU
Transformer
Transformer
formfrrmer
Prediction Horizon
Next state
Next
Next
Next state
Next
K 10 steps
Tokens
Latent, action, reward
Latent (1 x 4)
Latent
Lateat
Latent (2 x 2)
Lateut
Latent Representation
Categorical- VAl
VQ-VAE
Categorical- VAE
Categorical-
VQ- VAE
Categorical VAE
Decoder Inputs
Latent
Latent
I Latent, hidden
Latent
Latent, action, image
Latent
Agent State (st)
Latent
Image
Latent, hidden
Latent, hidden
Image
Latent, hidden
3

---

Published as a conference paper at ICLR 2025
single token for the transformer network compared to TWM which uses distinct tokens. This led to better training efficiency with state-of-the-an performance.
Another line of work focused designing Transformer-based world model to train agents from reconstructed trajectories in pixel space. Analogously to SimPLe, the agent's policy and value functions are trained from image reconstruction instead of world model hidden state representations. This requires learning auxiliary encoder networks for the policy and value functions. Contrary to
Dreamer-inspired works that learn agents from world model representations, these approaches also require accurate image reconstruction to train agents effectively. IRIS (Micheli et al. (023) first proposed a world model composed of a VQ- VAE (Van Den Oord et al.J 2017) convert input images into discrete tokens and an autoregressive transformer to predict future tokens. IRIS was
evaluated on the Atari 100k benchmark (Kaiser et al. 2020) showing promising performance in a low data regime. More recently, Micheli et al. (2024) proposed A-IRIS, encoding stochastic deltas between time steps using previous action and image as conditions for the encoder and decoder. This increased VQEVAE compression ratio and image reconstruction capabilities, achieving state-of-the- art performance on the Crafter (Hafner 202T benchmark and better results on Atari 100k.
Table compares the architectural details of recent model-based approaches learning world model in latent space with our proposed method. Following preceding Transformer-basee approaches, we reconstruct image observation from the encoder stochastic state 2 instead of St, which prevents the world model from using temporal information to facilitate reconstruction. The Transformer network uses relative positional encodings (Dai et al., 2019), which simplifies the use of the world model during imagination and evaluation. Absolute positional encodings require the Transformer network to reprocess past latent states with adjusted positional encodings when the current position gets larger than the ones seen during training. We also use the agent state 8; as input for predictor networks during the world model training phase to make actor-critic learning more straightforward.
2.3 CONTRASTIVE PREDICTIVE CODING
Contrastive Predictive Coding (CPC) was introduced by Oord al. (2018) as representation learn ing method based on contrastive learning for autoregressive models. CPC encodes a temporal signal into hidden representations and trains an autoregressive model to maximize the mutual informa- tion between the autoregressive model output features and future encoded representations using an InfoNCE loss based on Noise-Contrastive Estimation (Gutmann & Hyvärinen 2010). CPC was able to learn useful representations achieving strong performance phoneme classification, image classification, text classification tasks, and reinforcement learning with DeepMind Lab 3D environments (Beattie al., (016). While CPC was applied to speech, image, and text domains as pretraining pretext task, it showed promising results on DeepMind Lab tasks being used an auxiliary loss for the A3C (Mnih al. 2016) agent. In this work, we propose to apply CPC to model-based reinforcement learning. We introduce action-conditioned CPC (AC- CPC) that conditions CPC predictions on the sequence of future actions help the world model to make more accurate predictions and Iearn higher quality representations. We describe our use of conditiend CPC more detail in section 3.1
four distinct domains: speech
3 METHOD
We introduce TWISTER, a Transformer model-based reinforcement learning algorithm using action-conditioned Contrastive Predictive Coding to learn high-level feature representations and im- prove the agent performance. TWISTER comprises three main neural networks: a world model, an actor network and a critic network. The world model learns transform image observations into dis- crete stochastic states and simulate the environment to generate imaginary trajectories. The actor and critic networks are trained in latent space with imaginary trajectories generated from the world model to select actions maximizing the expected sum of future rewards. The three networks are trained concurrently using a replay buffer sampling sequences of past experiences collected during training. This section describes the architecture and optimization process of our proposed Transformer- world model with contrastive representations. Analogously to previous approaches, we also detail the learning process of the critic and actor networks taking place in latent space. Figure B shows an overview of our Transformer-basee world model trained with AC-CPC. It also illustrates the imagination process undertaken during the agent behavior learning phase.
4

---

Published as a conference paper at ICLR 2025
ACOIN Profittons
Care
Mached Trmmhome heven
(a) World Model Learning
(b) Agent Behavior Learning
Figure 3: Transformer-basee world model with contrastive representations. The world model learns temporal feature representations by maximizing the mutual information between model states and future stochastic states <'++ obtained from augmented views of image observations. The encoder augmentee
network converts image observations into stochastic states 2,, from which decoder network learns to reconstruct images while the masked attention Transformer network predicts next episode contin- uations, rewards and stochastic states conditioned selected actions.
3.1 WORLD MODEL LEARNING
Consistent with prior works (Hafner et al. 2023 Robine et al. 2023 Zhang et al. 2024), we learn world model in latent space by encoding input image observations o, into hidden representations using a convolutional VAE with categorical latents. The hidden representations are linearly projected categorical distribution logits comprising 32 categories, each with 32 classes, from which discrete stochastic states 2g are sampled. The world model is implemented as a Transformer State-Space Model (TSSM) (Chen al., (2022) using masked self-attention to predict next stochastic states & given previous states 21:1 and actions 01:t. The Transformer network outputs hidden states h2 that are concatenated with stochastic states 24 to form the model states 8, (h 31 The world model predicts environment reward fz, episode continuation é and AC-CPC features è using simple Multi Layer Perceptron (MIP) networks. The trainable world model components are the following:
Encoder Network:
g0(2) 0z)
TSSM
Transformer Network:
h. fo a1a-1)
Dynamics Predictor:
& ~ ( h4)
Decoder Network:
o, ~ ~ (or (x)
(1)
Reward Predictor:
P. (fe si)
Continue Predictor:
è, (oe 8t)
Representation Network:
ef g5((((+))
AC-CPC
AC-CPC Predictor:
al (se,, a1:++))
Transformer State-Space Model We train an autoregressive Transformer network using masked self-attention with relative positional encodings (Dai et al. 2019). During both training, exploration and evaluation, the hidden state sequence computed for the previous segment or state is cached to be reused as an extended context when the model processes mechanism allows the world model to imagine future trajectories from any state, eliminating the need to reprocess latent states with adjusted positional encodings.
next state. This encoding and caching
World model losses Given an input batch containing B sequences of T image observations 01:7. actions 01.7, rewards F1.T and episode continuation flags CI.T, the world model parameters (Ø) are optimized to minimize the following loss function:
B
T
L(の) = BT
ΣΣ
Lrew (0) + Leon (Ø) + Lrec(Ø) + Ldgn( + Lo)
(2)
5

---

Published as a conference paper at ICLR 2025
Irage layel Nost der fer denples
Lont der Hampley
Drakod
Figure 4: AC-CPC predictions made by the world model. We show the target positive sample without augmentation and the predicted most/least similar samples among the batch of augmented image views. We observe that TWISTER learns to identify most/least similar samples to the future target state using observation details such as the ball position, game score or agent movements. AC-CPC necessitates the agent to focus on observation details to accurately predict future samples, thereby preventing common failure cases where small objects are ignored by the reconstruction loss.
Lrew and Leon train the world model predict environment rewards and episode continuation flags, which are used to compute the returns of imagined trajectories during the behavior learning phase. We adopt the symlog cross-entropy loss from D3eamerV3 (Hatner al., 2023), which scales and transforms rewards into twohot encoded targets to ensure robust learning across games with different reward magnitudes. The reconstruction loss Lree trains the categorical VAE learn stochastic representations ž₁ for the world model by reconstructing input visual observations
Lyew (6) SymlogCrossEn((:: rz))
(3a)
Loon (Ø) = CemnarCrossEEnt((C
(3b)
Loee(Ø) = 16: lr
(3c)
The world model dynamics loss Ldyn trains the dynamics predictor network to predict the next stochastic states representations from transformer hidden states by minimizing the Kullback-Leibeer
(KL. divergence between the predictor output distribution P hạ) and the next encoder repre- sentation 4s &t oz). We also add a regularization term to avoid spikes in the KL. loss and stabilize learning by training the encoder representations to become more predictable. Both loss terms use the stop gradient operator sg(-) to prevent the gradients of targets from being backpropagated and are scaled with loss weights 9dym = 0.5 and Breg = 0.1, respectively:
Ldgn Böyn max [L sg( (g(as(a (2: oz)) II Po(ee he) 1)
(4)
+ Breg max [L ( ot) || sg(po(2s hx))))
The Transformer network learns feature representations using action-conditioned Contrastive Pre- learned by maximizing the mutual information between
dictive Coding. The representations model states s and future stochastic states "t:+* obtained from augmented views of image ob- servations. We adopt simple strategy to generate negative samples: Given the sequence batch of augmented stochastic states Z' containing one positive sample, we treat the other B x T samples negatives. The world model learns to distinguish positive samples from negatives using InfoNCE:
K
exp(sim(z'+4 st))
Lepe(
log
K
Enjez exp(sim(z) 8)))
K=
The world model Iearns to predict K = 10 future stochastic states among the batch of augmented samples. We compute similarities as dot products: sim(2/. st) = ( (s 0c:4+)), learning two MLP networks o and P for each step k. Contrary to the original CPC paper, which experiments with continuous feature states, we use discrete latent states for the world model. This requires learning representation network d8 to project discretized stochastic states z/J to contrastive feature representations ef. The AC-CPC predictor p uses the concatenated sequence of future actions a1:+4 condition to reduce uncertainty and learn quality representations.
3.2 AGENT BEHAVTOR LEARNING
The agent critic and actor networks are trained with imaginary trajectories generated from the world model. In order to compare TWISTER with previous approaches that train agents using world

---

Published as a conference paper at ICLR 2025
model representations, we adopt the agent behavior learning settings from DreamerV3 (Hafner et al, 2023). Learning takes place entirely in latent space, which allows the agent to process large batch sizes and increase generalization. We flatten the model states of the sampled sequences along the batch and time dimensions to generate Bimg- = B x T sample trajectories using the world model. The self-attention keys and values features computed during the world model training phase cached to be reused during the agent behavior learning phase and preserve past context. As shown in Figure 3bb the world model imagines H 15 steps the future using the Transformer network and the dynamics network head, selecting actions by sampling from the actor network categorical distribution. Analogously world model predictor networks, the actor and critic networks are designed as simple MLPs with parameter vectors (0) and (1²), respectively.
Actor Network: az To (o((a)(8) 2 (v)(s)
(6)
Critic Network:
Critic Learning Following DreamerV3, the critic network learns to minimize the symlog cross- entropy loss with discretized X-returns obtained from imagined trajectories with rewards and episode continuation flags predicted by the world model:
R f.41 + y2++ 1 (1 (5 ARA +
R Vi ($#+1)
The critic does not use a target network but relies on its own predictions for estimating rewards beyond the prediction horizon. This requires stabilizing the critic by adding a regularizing term toward the outputs of its own EMA network Ver. Equation 8 defines the critic network loss:
B
H
Loritia (z))
Σ
SymllgCrossEnt (D,, R,) + SymlogCrossEnttv,, Vir (sx))
(8)
BH
b=1
discrete returns regression
critie EMA regularizer
Actor Learning The actor network learns select actions that maximize the predicted returns ing Reinforce (Williams] 1992) while maximizing the policy entropy to ensure sufficient exploration during both data collection and imagination. The actor network loss is defined as follows:
B
H
Lrator (0)
sg() log x (at 8c) 1yH n) (ar (x))
BH
entropy regularizer
Where A (RA (st) max(1, S) defines advantages computed using normalized returns.
The returns are scaled using exponentially moving average statistics of their 5th and 95th batch percentiles ensure stable learning across all Atari games:
S EMA(Per(PRe) 95) Per( 5), momentun = 0.99)
(10)
EXPERIMENTS
In this section, we describe our experiments on the commonly used Atari 100k benchmark. We compare TWISTER with SimPLe, DreamerV3 and recent Transformer model-based approaches in Table 2
We also perform several ablation studies on the principal components of TWISTER.
4.1
ATARI 100K BENCHMARK
The Atari 100k benchmark was proposed in Kaiser et al. (2020) to evaluate reinforcement learning agents on Atari games in low data regime. The benchmark includes 26 Atari games with budget of 400k environment frames, amounting to 100k interactions between the agent and the environment using the default action repeat setting. This amount of environment steps corresponds hours (1.85 hours) of real time play, representing a similar amount of time that a human player would need achieve reasonably good performance. The current state-of-the-art is held by EfficientZero V2 Wang et al. 2024), which uses Monte-Carlo Tree Search to select the best action at every time step. Another recent notable work BBF (Schwarzer et al. 2023), a model-free agent using learn- ing techniques that are orthogonal to our work such as periodic network resets and hyper-parametern annealing to improve performance. In this work, to ensure fair comparison and demonstrate the effectiveness of AC-CPC for learning world models, compare method with model-basee ap- proaches that do not utilize look-ahead search techniques. Combining these additional components with TWISTER would nevertheless be interesting research direction for future works.
about two

---

Published as a conference paper at ICLR 2025
4.2 RESULTS
Table 2: Agent scores and human-normalized metrics on the 26 games of the Atari 100k benchmark.
We show average scores over 5 seeds. Bold numbers indicate best performing method for each game.
Game
Random
Human
SimPLc
TWM
IRIS
DreamerV3
STORM
A-IRIS
TWISTER (ours)
Alien
228
420
959
984
391
Amidar
1720
74
122
143
139
205
64
184
Assault
222
742
527
683
1524
706
801
1123
721
Asterix
210
8503
1128
1116
854
932
1028
2492
1306
Bank Hoist
14
753
34
467
53
649
641
1148
942
Baitle Zone
37188
13074
12250
13540
Baxing
12
78
70
78
80
70
88
Breakout
16
20
Chopper Command
811
7388
979
1697
1565
420
1888
1183
910
Crazy Climber
10780
35829
62584
71820
59324
97190
66776
57854
81830
Demon Attack
152
350
2034
303
165
289
Freeway
0
30
17
24
31
U
34
31
32
Frostbiie
Gopher
258
2412
597
1675
2236
3730
8240
6445
22234
1027
30826
2657
7254
7037
11161
11044
7049
8773
James Bond
29
303
100
362
463
445
509
309
573
Kangaroo
52
3035
51
1240
838
4098
4208
2269
6016
Krull
2205
6349
8839
Kung Fu Master
258
22736
14862
24555
21760
21420
26182
21534
23442
1588
1067
Pong
-21
15
13
19
15
18
11
20
20
Private Eye
69571
35
87
100
882
7781
103
1608
Obert
164
13455
1289
3331
746
3405
4522
1444
3197
Road Runner
12
7845
5641
9107
9615
15565
17564
10414
17832
Seaquest
42055
661
525
Up N Down
533
11693
3350
15982
3546
7600
7985
4072
7068
# Superhuman
N/A
10
10
11
12
Nonned Meun (%)
100
33
96
105
112
127
139
162
Normed Median (%)
100
13
51
29
53
77
Table 2 compares TWISTER with SimPLe (Kaiser et al. 2020), DreamerV3 (Hafner et al. 2023)
and recent Transformer model-based approaches (Robine et al. 2023 Micheli et al. 2023 Zhang al.) 2024; Micheli al. 2024) on the Atari 100k benchmark. Following preceding works, we use human- normaine metrics and compare the mean and median returns across all 26 games. The human-normalized scores are computed for each game using the scores achieved by human player
agent score-random score. We and the scores obtained by a random policy: normed score haman score score-randon score" also show stratified bootstrap confidence intervals of the human- normannormalized mean and median in Figure Performance curves corresponding to individual games can be found in the appendix g
TWISTER achieves a human-nowe normallze mean score of 162% and median of 77% on the Atari 100k benchmark, setting a new record among state-of-the-art model-based methods that do not employ look-ahead search techniques. Anal- ogously to STORM, find that TWISTER demonstrates superior performance in games where key objects related to rewards merous, such as Amidar, Bank Heist, Gopher and Ms Paeman. Furthermore, we observe that TWISTER benefits from increased performance in games with small moving objects like Break- out, Pong and Asterix. We suppose that the AC- CPC objective requires the agent focus on the ball's position in these games to accurately pre- dict future samples, thereby preventing failure cases where small objects are ignored by the reconstruction loss. Alternatively, IRIS and A-IRIS solve this issue by learning agents from high-quality reconstructed images. They encode image ob- servations into spatial latent spaces through a VQ-V structure, which allows these approaches to better capture details and achieve lower reconstruction errors with good results for these games. We show CPC predictions made by the world model for diverse Atari games in the appendix A.4
Mean
Median
TWISTER
STOTM
1VM
SesPaa
0.4
0.5
3.5
0:25
0.50
0.75
Human Normal zed Score
Figure 5: Mean and median scores, com- puted with stratified bootstrap confidence inter- vals (Agarwal et al., 2021). TWISTER achieves normalized mean of 1.62 and median of 0.77.
8

---

Published as a conference paper at ICLR 2025
4.3 ABLATION STUDIES
In order to study the impact of AC-CPC on TWISTER performance, we perform ablation studies on all 26 games of the Atari 100k benchmark, applying one modification at time. We experiment with the number of CPC steps predicted by the world model. We show that data augmentation helps to complexify the AC-CPC objective and improve its effectiveness. We find that conditioning CPC predictions on the sequence of future actions leads to more accurate predictions and improves the quality of representations. We also study the effect of world model design on AC-CPC effectiveness. Table 3 shows the aggregated scores obtained for the main ablations after 400k environment steps.
Table 3: Ablations of the AC-CPC loss, contrastive samples augmentation, conditioning on future actions and using DreamerV3's RSSM. We perform one modification time and evaluate on the 26 Atari games. The detailed results obtained for individual games can be found in the appendix A.6
Metries
TWISTER
No AC-CPC
DreamerV3 World Model
No Action Conditioning
Data Augmentation
Normed Mean (%)
162
112
121
III
120
Normed Median (%)
77
42
Number of Contrastive Steps We experiment with several numbers of CPC steps, comparing human-normalized metrics over all 26 games of the Atari100k benchmark. Figure 6a shows that TWISTER achieves the best ture, corresponding to 0.67 seconds of game time. We find that AC-CPC has a significant effect on TWISTER performance up certain amount of steps. We observe increase in human- normalized mean and median scores with the number of predicted CPC steps. However, a degrada- tion of the results is noticed when predicting 15 steps into the future. The difference in median score indicates decrease in performance for middle-scoring games.
mean score when predicting 10 steps into the fu-
World Model Architecture We study the impact of world model design on AC-CPC effective- ness to learn feature representations. Figure 6b shows the effect of AC-CPC on performance when replacing the TSSM of TWISTER with DreamerV3's RSSM (Hafner et al. (023). While the two approaches achieve similar results without the AC-CPC objective, we find that AC-CPC has signif- icant effect on TWISTER, improving performance on most games. These findings can be attributed to the fact that Transformers are generally more effective than RNNs at learning feature representa-
tions due to several key architectural differences. The capacity self-attention to model temporal relationships without recurrence makes the Transformer architecture highly effective at capturing context and learning hierarchical features. On the other hand, the recurrent nature of RNNs can lead to vanishing gradients and slower convergence, particularly with long sequences.
Mean
Median
Mean
Median
1.5
0.75
1.0
0.50
0.25
0.5
0.25
0.0
0.00
0.0
0.00
200K
400k
200K
400K
200K
430k
200K
400K
sttee staps
stees
Dreameev3
10 stops (TWISTER)
AC-PCCCPC
AC-OPC
(a) Number of Contrastive steps
(b) World Model Architecture
Mean
Median
Median
1.5
0.75
1.5
0.75
0.50
0.50
0.25
0.5
0.25
0.0
0.00
0.0
0.00
400k
200K
400K
200K
400k
200K
400K
TWISTER
Action Conditioiing
Data Augmentttion
Random Snifts
(TWISTER)
(c) Action Conditioning
(d) Data Augmentation
Figure 6: Ablations made on the Atari 100k benchmark. The results averaged over 5 seeds. We study the effect of data augmentation, action conditioning and the number predicted CPC steps on TWISTER performance. We also study the effect of world model design AC-CPC effectiveness.

---

Published as a conference paper at ICLR 2025
Actions Conditioning We find that condition- ing the CPC prediction head on the sequence of future actions leads to more accurate predic- tions and higher quality representations. Figure 7 shows the aggregated CPC loss and prediction ac- curacy for training and validation sequences over all Atari games. We report the average num- ber of times the similarity for the positive sam- ple is higher than for the negative samples in the contrastive loss. Without knowing the sequence of future actions, the world model cannot pre- dict future environment states accurately, which makes the task almost insolvable and counterpro- ductive beyond certain amount of CPC steps. We observe a decrease in accuracy compared to TWISTER when predicting multiple steps with- out knowing the sequence of future actions. Fig- ure 66c shows the aggregated numan-normaliize scores over the 26 games when removing the con- dition of future actions for CPC predictions. We find that the CPC objective does not bring notable performance improvements when removing future actions conditioning.
Taniing
Weldation
1st step
10th stop
1sl stpp
TPF step
600
4.0
2.0
00
de Card
2005
4030
2004
400
2004
2004
189c
Figure 7: Aggregated CPC loss and prediction accuracy over all 26 games. We use a validation replay buffer of 100k samples to compare CPC loss on unseen trajectories. The trajectories are obtained from collection of DreamerV3 and TWISTER agents pretrained with 5 seeds.
Effect of Data Augmentation The effect of data augmentation on CPC performance was studied by Kharitonov et al. (202T). In their work, they propose to introduce data augmentation for CPC to learn higher quality speech representa- tions, yielding better performances. In this work, we apply image augmentation to contrastive sam- ples in order to complexify the AC-CPC objective and make the representation learning task more challenging. We apply the commonly used ran- dom crop and resize augmentation during training for its effecttivnness in the area of image-based contrastive learning (Chen et al., (2020). The use of random crops requires the world model identify several key elements in the observations in order to accurately predict positives samples. We also experiment with random shifts (Yarats et al. 2021), shifting the image up to 4 pixels in height and width but found it to have lesser impact on the learning objective. Figure 6dd shows the aggregated
1st step
5th step
10th step
6.0
racoon 50t5
4.0
0.0
1.0
CRC
400K
200K
400K
200K
400K
Figure 8: Effect of data augmentation on AC- CPC objective complexity. We aggregate CPC loss and prediction accuracy over all Atari games for different time horizons.
scores for studied augmentation techniques. We find that random crop and resize helps the best to improve final performance. Not using image augmentations for negative and positive samples reduces the impact of AC-CPC on TWISTER performance, achieving lower mean and median scores. We show the impact of data augmentation on the AC-CPC objective complexity for different time horizons in Figure g
5 CONCLUSION
We propose TWISTER, a Transformer model-based reinforcement learning agent learning high- level temporal feature representations with action-conditioned Contrastive Predictive Coding. TWISTER achieves new state-of-the-arr results on the Atari 100k benchmark among model-based approaches that do not employ look-ahead search with a human-normalized mean and median score of 162% and 77%, respectively. We study the impact of learning contrastive representations on Transformer-baseee world models and find that the AC-CPC objective significantly helps to improve the agent performance. We also show that data augmentation and future actions conditioning play an important role in the learning of representations to complexify the AC-CPC objective and help the model to make accurate future predictions. Following early findings, we hope that this work will inspire researchers to further study the benefits of sulf-supe learning techniques for model-based reinforcement learning.
10

---

Published as a conference paper at ICLR 2025
6 ACKNOWLEDGMENTS
This work was partly supported by The Alexander von Humboldt Foundation (AvH).
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural informa- fion processing systems, N4:29304-29320 2021.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
Bowen Baker, 1lge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 2022.
Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, et al. Deepmind lab. arXiv preprint arXiv:1612.03801, 2016.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint 13oxiv:330..344 2013.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912-9924, 2020.
Chang Chen, Yi-Fu Wu, Jaesik Yoon, and Sungjin Ahn. Transdreamer: Reinforcement learning with transformer world models. arXiv preprint arXiv:2202.09481, 2022.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint
2014.
Rémi Coulom. Efficient selectivity and backup operators monte-carlo tree search. In International conference on computers and games, pp. 72-83. Springer, 2006.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdi- nov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv: 02860, 2019.
Fei Deng, Ingook Jang, and Sungjin Ahn. Dreamerpro: Reconstruction-free model-based reinforce- ment learning with prototypical representations. In International conference on machine learning. pp. 4956-4975. PMLR, 2022.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 297-304. JMLR Workshop and Conference Proceedings, 2010.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. Advances in neural information processing systems, 31, 2018.

---

Published as a conference paper at ICLR 2025
Danijar Hafner. Benchmarking the spectrum of agent capabilities. arXiv preprint arX1::2009.06780, 2021.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. International conference on machine learning, pp. 2555-2566 PMLR, 2019.
Danijar Hafner, T Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2020.
Danijar Hafner, Timothy Lillicrap. Mohammad Norouzi, and Jimmy Ba. Mastering atari with dis-
crete world models. In International Conference on Learning Representations, 2021.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301 000.0000.0.,..... 2023.
Nicklas Hansen, Xiaolong Wang, and Hao Su. Temporal difference learning for model predictive control. arXiv preprint arXiv:2203.0443 04955, 2022.
Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpe2: Scalable, robust world models for contin- uous control. arXiv preprint arXiv:2310. 16828, 2023.
Mikael Henaff, William F Whitney, and Yann LeCun. Model-based planning with discrete and continuous actions. arXiv preprint arXiv:1705 0715007777.77 2017.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018.
Lukasz Kaiser, Mohammad Babacizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model-based reinforcement learning for atari. In International Conference on Learning Representations, 2020.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXi::2001.0836/ 2020.
Eugene Kharitonov, Morgane Rivière, Gabriel Synnaeve, Lior Wolf, Pierre-Emmanuel Mazaré, Matthijs Douze, and Emmanuel Dupoux. Data augmenting contrastive learning of speech rep- resentations in the time domain. In 2021 IEEE Spoken Language Technology Workshop (SLT), pp. 212-222. IEEE, 2021.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization, arXiv preprint 1rXiv:1412.6988 2014.
Diederik Kingma and Max Welling. arXiv:1312.61 14, 2013.
Auto-encoding variational bayes.
arXiv preprint
Yann LeCun, Bernhard Boser, John Denker, Donnie Henderson, Richard Howard, Wayne Hubbard, and Lawrence Jackel. Handwritten digit recognition with a back-propagation network. Advances in neural information processing systems, 2, 1989.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):46-444, 2015.
Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers sample-efficient world mod- cls. In International Conference on Learning Representations, 2023.
Vincent Micheli, Eloi Alonso, and François Fleuret. Efficient world models with context-aware tokenization. In Forty-first International Conference on Machine Learning, 2024, URL https: //openreviewt wwtp:/wwwwetttotmm//// ERRWEE
12

---

Published as a conference paper at ICLR 2025
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-333
2015. 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. International conference on machine learning, pp. 1928-1937. PMLR, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint
2018.
Jan Robine, Mare Höftmann, Tobias Uelwer, and Stefan Harmeling. Transformer-based world mod- els are happy with 100k interactions. International Conference on Learning Representations, 2023.
Julian Schritwieserr, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588(783))6604-6) 2020.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: I 170.006347 2017.
Max Schwarzer, Johan Samir Obando Ceron, Aaron Courville, Mare G Bellemare, Rishabh Agar- wal, and Pablo Samuel Castro. Bigger, better, faster: Human-level atari with human-level effi- ciency. In International Conference on Machine Learning, pp. 30365-30380. PMLR, 2023.
David Silver, Hado Hasselt, Matteo Hessel, Tom Schaul, Arthur Guez, Tim Harley, Gabriel Dulac- Arnold, David Reichert, Neil Rabinowitz, Andre Barreto, et al. The predictron: End-to-end learning and planning. In International Conference on Machine Learning, pp. 3191-3199, PMLR, 2017.
David Silver, Thomas Hubert, Julian Schritwieserr, loannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play, Science, 362(6419): 1144, 2018
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-15) 1958, 2014.
Richard S Sutton. Dyna, integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 164) 1991.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, al. Deepmind control suite. arXiv preprint arXiv:1801.00690 2018
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathicu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(77722:350--354 2019.
Shengjie Wang, Shaohuai Liu, Weirui Ye, Jiacheng You, and Yang Gao. Efficientzero v2: Mastering discrete and continuous control with limited data. arXiv preprint arXiv:2403.00566, 2024.
13

---

Published as a conference paper at ICLR 2025
Tingwu Wang and Jimmy Ba. Exploring model-based planning with policy networks. In Interna- tional Conference on Learning Representations, 2020.
Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking model-based reinforce- ment learning. arXiv preprini arXi:: 1907.0207, 2019.
Ronald J Williams. Simple statistical gradient-oolowing algorithms for connectionist reinforcement learning. Machine learning, 8:229-2566 1992.
Denis Yarats, Rob Fergus, and Ilya Kostrikov. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In 9th International Conference Learning Represen- tations, ICLR 2021, 2021.
Weirui Ye, Shaobuai Liu, Thanard Kurutach, Pieter Abbeel, and Yang Gao. Mastering atari games with limited data. Advances in Neural Information Processing Systems, 34:25476-25485 2021.
Weipu Zhang, Gang Wang, Jian Sun, Yetian Yuan, and Gao Huang. Storm: Efficient stochastic transformer based world models for reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1
ATARI 100K EVALUATION CURVES
Alien
Amidaar
Assault
Asterix
200
800
1500
150
1000
500
100
50
400
500
0
400k
400k
200k
Heist
Baille
Boxing
Breakout
100
1000
10000
500
50-
200k
400k
200k
400%
200k
400k
200k
400k
Chopper Command
Crazy Climber
Demon Allack
Freeway
100000
800
1000
75000
600
50000
800
25000
200
10
200k
400k
0
200k
400k
200k
400k
200k
400k
Fre stiit
Gopher
Hero
James Bond
400
300
30000
750
500
200
10000
5000
250
100
200k
400k
200k
200k
400k
200k
400k
KangarDo
Krull
Kung Fu Master
Ms Pacman
10000
10000
30000
3000
20000
2000
6000
4000
10000
1000
2000
Pong
Eye
Runner
20
4000
2000
2000
10000
110
-20
200k
400k
200k
400k
200k
400k
0
200k
400k
Seaquest
Up Down
Normed Mean
N Normed Median
600
10000
7500
0.5
400
5000
0.4
2500
0.2
0.0
200k
400k
200k
400k
200k
400K
0
200k
400K
Figure 9: Evaluation curves of TWISTER on the Atari benchmark for individual games (400K environment steps). The solid lines represent the average scores over 5 seeds, and the filled areas indicate the standard deviation across these seeds.
15

---

Published as a conference paper at ICLR 2025
A.2 MODEL ARCHITECTURE
Table Architecture of the encoder network. The size of submodules is omitted and can be derived from output shapes. Each convolution layer (Conv) followed by layer normalization (LN) and a SiLU activation layer. The encoder downsamples images with strided convolutions layers using kernel size of stride of 2 and a padding of I. We flatten output features and project them categorical distribution logits using a Linear layer. Stochastic states 2 are sampled from Softmax probabilities and encoded one hot vectors.
Submodule
Oulput shape
Input image (oy)
Cony LN + SILU
32
32 x 32
Conv
Cony + LN SILU
128
8
Conv + LN SILU
256 x
Flatten
4096
Lincar
1021
Reshupe Sollmux
Sample + One Hot (outputs 2,)
32 x 32
Table Architecture of the decoder network. Images are reconstructed from stochastic states. Each transposed convolution layer (ConvTrans) uses kernel size of 4, stride of and padding of 1.
Submodule
Output shape
Input stochastic state (5t)
32 32
Flatien
1024
Linear
4096
Reshape
256 4
KranTraas I.N SiL.U
64 x 16 16
ConvTrans + + SiLU
32 x
Cony ToanT (outputs
3 x 61
64
Table 6: Transformer block. Dropout (Srivastava et al. 2014) is used in each Transformer submodule reduce overfitting. We also apply Dropout to attention weights in the MHSA module.
Submedule
Module alias
Output shape
features (lahel .r))
Multi-head self-attention Dropout
Residual (add m1)
LN (label as 22)
Linear ReLU
T 1024
Dropout Residual (add mn))
Food Forward
512
T' x 512
Table 7: Transformer network. The stochastic states 20:T-1 and one-hot encoded actions d0:7-1 R xx x are combined using an action mixer network (Zhang et al. 2024). The features are processed by the Transformer network compute hidden states hyrr-
Suhmodule
Module alins
Outpat shape
Inputs stochastic states (Rg:m-1) Flatten
x 32
T" 1024
Concar actions 0a:77-1
Action Mixer
(1024
Linear LN SiLU Lincar
T. x 512
612
Transformer black x Outputs hidden states (b1/r)
Transformer Network
T. x 512
16

---

Published as a conference paper at ICLR 2025
Table 8: Networks with Multi Layer Perceptron (MLP) structure. Inputs are first flattened and concatenated along the feature dimension. Each MLP layer is followed by layer normalization and SiLU activation except for the last layer which outputs distribution logits.
Nerwork
MI P layers
Inputs
Hiddlnn dimension
Output dimension
Output Distribution
predicter
8t
Symlog Bernulli
Continue predictor
512
Representation nehwork
512
512
NIA N/A
AC-CPC predictor
des But+k
512
Critie network
512
255
Symlog Discrete
Actor network
512
One bot Categorical
A.3 HYPER- HYPEERPPRRMETTES
Table 9: TWISTER hyper-parameters, We apply the same hyper-parameters to all Atari games.
Parameter
Symbol
Sctting
General
Batch Size
16
Sequence Length
64
Optimizer
Adam Kingma Ba 2014
Image Resolution
64 04 (RGB)
Training Step Policy Step
Instances
Transformer Network
Transformer Blocks
Number Attention Heads
8
Dropout Probahility
0.1
Attention Context Length
8
World Model
Stochastic State Features
32
per
Dynamics Loss Scale
9ayn
0.5
Representation Loss Scale
Drag
0.1
AC-CPC
10
Random Crop Resize Scale
(0.25, 1.0)
Random Crop Resize Ratio
(0.75, 1.33)
Learning Rate
10 -4
Adam Betas
8r. 9
0.9, 0.999
Adam Epsilon
10-8
Clipping
1000
Actor Critic
Imagination
Retur Discount
0.997
Return Lambda
0.95
Critic EMA Decay
0.98
Normalization Momentum
Actor Entropy
1
Learning Rate
3 10
Adam Betns
31. 92
0.9,
Adamn Epsilon
10-3
Gradient Clipping
17

---

Published as a conference paper at ICLR 2025
A.4 AC-CPC PREDICTIONS
Image Nost Samplex
Simter Sample::
Kong Mierr
Brokou
Soinn
Fostb
Batk Ztre
Food Puree
Jonne Bord
My Popore
Frewnyy
Figure 10: AC-CPC predictions made by the world model for diverse Atari games. We show the tar- get positive sample without augmentation and predicted most/least similar samples among the batch of augmented image views. We observe that TWISTER successfully learns to identify most/least similar samples the future target state using observation details such as the ball position in Pong, the game score in Kung Fu Master or the agent movements in Boxing. AC-CPC necessitates the agent focus on observation details to accurately predict future samples, thereby preventing com- mon failure cases where small objects are ignored by the reconstruction loss.
18

---

Published as a conference paper at ICLR 2025
A.5
WORLD MODEL PREDICTIONS
1=
1:
King Po Manter
Pong
Bodin
Beskoot
Cbon
Kangino
Fronhie
Battl Zone
Roodd R Humner
Jamen Bond
Adhent
Ui N Down
Freeuity
Figure 11: World Model Predictions. We show the decoder reconstruction of trajectories imagined by the world model over 64 time steps. We use context frames and generate trajectories of 59 steps into the future using the Transformer network and dynamics predictor head. Actions are predicted by the actor network by sampling from the categorical distribution.
19

---

Published as a conference paper at ICLR 2025
A.6 ABLATIONS RESULTS
Table 10: Ablations of the AC-CPC loss, contrastive samples augmentation, conditioning on future actions and using DreamerV3's RSSM. We show agent scores and human-normalizze metrics the 26 games the Atari 100k benchmark. The results are averaged over 5 seeds and bold numbers indicate best performing agent for each game.
Game
TWISTER No CPC
DreamerV3 World Model
No Action Conditioning
No Data Augmentation
Alien
970
1147
1040
1101
1154
Amidar
184
173
174
191
154
Asterix
1306
1165
1401
1082
1173
Heist
942
Battle Zone
9920
5800
15540
12860
8980
Boxing
82
77
84
Breakout
14
34
59
Chopper Command
910
984
1242
620
646
Crazy Climber
87272
77454
Demon Artack
289
21.5
456
339
356
Froeway
32
Frostbite
305
714
571
884
953
Gopher
22234
1387
3318
2972
5851
Hero
9944
11079
James Bond
573
493
432
335
316
Kangaroo
Krall
8839
8096
7469
8054
9065
Kung Fu Master
23442
22232
25518
19412
17566
Ms Pacman
2206
2025
1681
1927
2294
Pong
20
13
18
20
20
Eye
Qhert
3197
2579
3542
4443
4231
Runner
17832
Senquest
532
474
569
491
467
Up Dowa
7068
5816
30135
5378
7213
# Superhuman
11
Normed Mcan
162
Normed Median (%)
77
44
69
42
68
20

---

Published as a conference paper at ICLR 2025
A.7 DEEPMIND CONTROL SUITE RESULTS
We assess TWISTER's performance on continuous action control tasks by evaluating on the Deep- Mind Control Suite (Tassa et al., 2018). The suite was designed to serve as a reliable performance benchmark for reinforcement learning agents in continuous action space, including diverse control tasks with various complexities. Similarly to DreamerV3 (Hafner al., (023). we evaluate on 20 tasks using only high-dimennsonal image observations as inputs and budget of 1M environment steps for training.
approaches applied to continuous control. DreamerPro (Deng et al.] (0022) proposed reconstruction-fre variant of the Dreamer algorithm. Similarly to Sw Caron et al. (2200) the agent learns hidden representations by encouraging consistent cluster assignments for different augmentations of the same images Caron et al. 22020). More recently. TD-MPC2 Hansen et al. (2023) extended the TD-MPC Hansen et al. (022 agent to multitask learning and demonstrated state-of-the-an performance on diverse continu- control tasks. TD-MPC unrolls its world model over the batch of sampled trajectories to predict the sequence of future latent states and environment quantities. The agent also learns a Q-value function to estimate long- tergterm returns using Temporal Difference (TD) learning. It uses Model Pre- dictive Control (MPC) for planning, selecting actions that maximize expected returns using world model value predictions.
We compare TWISTER with DreamerV3 and two other recent
Table TT shows the results obtained on the 20 tasks after 1M environment steps. We obtain Dream- results
erPro and TD-MPC4 results using official implementations. TWISTER obtains state-of-the performance with a mean score of 801.8. We also experiment with removing the AC-CPC objective and find that it has a positive impact on most of the tasks. AC-CPC particularly improves perfor- mance on complex tasks such as Acrobot Swingup, Quadruped Run / Walk and Walker Run.
Table 11: Agent scores on the DeepMind Control Suite under visual inputs. We show average scores over seeds (1M environment steps). Bold numbers indicate best performing method for each task. We also underline TWISTER numbers to indicate tasks where AC-CPC improves performance.
Task
DreamerPro
DreamerV3
TD-MPC2
TWISTER (No CPC)
TWISTER (ours)
Acrobot Swingup
438.3
210.0
216.0
81.6
239.4 4
In Cup Catch
962.2
957.1
717.2
964.8
966.8
Cutpole Balance
998.5
996.4
431.1
998.5
997.9
Cartpole Balance Sparse
Carppole Swingup
871.0
819.1
808.1
759.1
819.2
Cartpole Swingup Sparse
811.2
792.9
739.0
524.4
735.0
Cheetah Run
898.3
728.7
550.3
695.
694.0
Finger Spin
600.6
818.5
986.0
845.8
976.3
Finger Turn Easy
956.4
Finger Turn Hard
719.6
810.8
871.8
960.3
910.3
Hopper Hop
211.1
313.9
Hopper Stand
927.5
900.6
803.0
936.5
932.0
Peadulom Swingup
845,2
806.3
743.2
$29.2
832,1
Quadruped Run
616.2
352.3
361.9
503.8
652.1
Quadruped Walk
676.9
352.6
252.6
741.6
904.9
Reacber Easy
886.1
Reacher Hard
294.0
499.2
876.9
380.0
565.8
Walker
750.0
728.1
711.2
Walker Stand
974.8
976.7
915.8
969,5
976.9
Walker Walk
956.6
955.8
945.1
961.2
951.3
Mcan
770.9
739.6
720.9
728.7
801.8
Median
808.5
795.9
21

---

Published as a conference paper at ICLR 2025
Acrobat Swingup
Ball Catch
Cartpole Balance
Cartpole Balance Sparse
1000
1000
250
200
150
100
400
200
200
200K
400K
200K
400K
200K
400K
200K
400K
Cartpole Swingup
Cartpole Swingup Sparse
Cheetah
Finger
800
800
600
800
600
400
400
400
600
200
200
200
0-
200K
400K
200K
400K
200K
400K
200K
400K
Finger Easy
finger
Hopper Hop
Hopper
1000
300
800
800
800
600
200
400
400
200
200K
400K
200K
400K
200K
400K
200K
400K
Pendulum Swingup
Quadruped Run
Quadruped Walk
Reacher Easy
1000
800
1000
1000
800
600
600
400
400
400
400
200
400K
400K
200K
400K
200K
400K
Reacher Hard
Walker Run
Walker Stand
Walker Walk
1000
2000
600
600
800
600
600
200
400
200
400
200
Mcan
Median
800
800
400
400
200
200
Figure 12: Evaluation curves of TWISTER on the DeepMind Control Suite for individual tasks (IM environment steps). The solid lines represent the average scores over 5 seeds, and the filled areas indicate the standard deviation across these 5 seeds.
22


Reviews:
[{"review_id": "jWZM3yuZSj", "reviewer": "Reviewer_1PDj", "summary": "The paper proposes to combine Transformer-based world models with contrastive learning, and outperforms DreamerV3 and recent Transformer-based MBRL agents on the Atari 100k benchmark.", "questions": "Some clarifying questions on implementation details\n- Did you use EMA encoder or any stop gradient operation when obtaining future latents $z'_{t+k}$?\n- Is the augmentation consistent across time (e.g., cropping the same region across time rather than choosing a random cropping region at each time step)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper provides a detailed overview of DreamerV3 and recent Transformer-based MBRL agents. Table 1 provides a good summary of implementation differences.\n- There is adequate ablation study to justify the design of action-conditioned CPC loss and the CPC steps used. The finding that random crop and resize work better than random shift is quite intriguing.\n- Sufficient implementation details are provided.\n- The paper achieves strong results on the Atari 100k benchmark.", "weaknesses": "- The paper did not discuss similar works that combine contrastive learning with Dreamer, notably [DreamingV2](https://arxiv.org/abs/2203.00494) and [DreamerPro](https://proceedings.mlr.press/v162/deng22a.html). While targeting different tasks, DreamingV2 has a quite similar contrastive loss design.\n- From Table 2, it is quite intriguing that the proposed method TWISTER achieves significantly higher score on Gopher than previous methods. Further from Table 10, without the contrastive loss, the score on Gopher drops much more significantly than on other games. I am a bit concerned that the overall improvement brought by contrastive learning might be dominated by this single game.\n- Based on the first two points, the paper could be strengthened by showing the general applicability of the method. For example, the paper can investigate more tasks (e.g., Crafter, DMC), or more backbones (e.g., S4/S5, see [S4WM](https://openreview.net/forum?id=GDYuzX0rwj&noteId=O5ZiA3xL4E) and [R2I](https://openreview.net/forum?id=1vDArHJ68h))", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XTH1uB6br2", "reviewer": "Reviewer_2k8g", "summary": "The paper explores the use of contrastive predictive coding (modified objective function) for learning efficient world models. \n\nThe paper is very well written.\n\nMotivation: The work argues that the task of predicting the next frame may not require a complex model (as the successive video frames are somewhat similar or change in distribution is very sparse). The table 1 is very helpful on distinguishing how different methods effect each other. \n\nProposed Method: The work builds upon transformer based world models and CPC loss to learn high level representations for improving the sample efficiency of the learning agent.\n\nAblations: The work conducts thorough ablations to understand the role of different components (number of steps in CPC loss, role of data augmentation, role of world model architecture i.e. RNNs v/s Transformer) and the effect of action conditioning.", "questions": "- \"As shown in Figure 2, the cosine similarity between adjacent latent states of the world model is very high, making it relatively straightforward for the world model to predict the next state compared to more distant states.\"\n\nIn the introduction paper talks about cosine similarity for the proposed method, it will also be interesting to plot the cosine similarity for all the ablations the paper propose (number of contrastive steps, future action conditioning, effect of data augmentation) (as well as the dreamer baseline). \n\n- It will be helpful if authors can include another benchmark to make sure the proposed method shows gains on other environments too (like Crafter as mentioned in section 2.2 for related work).", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper is very well written.\n- The paper does a good job of ablating various different components used in proposed method.\n- The paper does a good job of citing relevant literature.", "weaknesses": "- There's no weakness as such. The paper validate all the claims made in the introduction via careful experimentation and proper ablations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zqadIjSs5j", "reviewer": "Reviewer_z6Gc", "summary": "This paper introduces a novel world model for deep reinforcement learning that integrates DreamerV3 with transformer-based architectures and contrastive predictive coding. Instead of predicting only the next state, this world model is designed to predict multiple time steps into the future. The method demonstrates state-of-the-art performance on the Atari 100k benchmark.", "questions": "- [Q1] Do the authors have any insight into why the performance drops when predicting 15 steps into the future?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- [S1] The paper presents a novel combination of established approaches, specifically transformer-based world models and CPC, which leads to strong performance results on the Atari 100k benchmark.\n- [S2] Ablation studies are conducted thoroughly, providing insight into the components of the model. (Though see Weakness [W2] for potential areas of improvement.)\n- [S3] The related work section on world models is thorough, with Table 1 providing a clear and valuable summary.", "weaknesses": "- [W1] The explanation of AC-CPC is somewhat vague, which is problematic given the importance of CPC in the proposed method. Some specific concerns include:\n  - (i) The authors do not clarify the necessity of the representation network, which is only briefly mentioned in Eq. (1).\n  - (ii) Notation introduced in Eq. (1), specifically $e_t^k$ and $\\hat{e}_t^k$, is not referenced further in the text.\n  - (iii) Visualizations of the representation networks and AC-CPC predictors in Figure 3(a) could enhance clarity.\n- [W2] The ablation studies section could benefit from improved consistency. Specifically, the order of the studies differs across Table 3, the main text, and Figure 6, which may lead to confusion. I recommend aligning the order across these references. Additionally, inconsistent naming conventions add to the lack of clarity. Suggested improvements:\n  - In Figure 6(a), \"Number of CPC steps\" could be updated to \"Number of contrastive steps\" to match the main text.\n  - The paragraph titled \"Future Actions Conditioning\" could be revised to \"Action Conditioning,\" aligning with Figure 6(d).\n- [W3] Many figures suffer from small text, making them challenging to read without zooming. Figure 6, in particular, has unnecessary whitespace around plots, which could be adjusted to improve readability.\n- [W4] Including evaluations on other benchmarks, such as the DeepMind Control Suite, would strengthen the soundness of the findings. However, I acknowledge that this may be difficult to implement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9pyFhTT2w5", "reviewer": "Reviewer_TrKK", "summary": "In this paper, the authors investigated the Model-Based Reinforcement Learning (MBRL) agent with contrastive predictive coding. There have been many tries to apply Transformer as the world model backbone for MBRL. However, the authors argued the impacts from the tries are limited due to the limitations of the predictive loss (one-step next observation prediction), and proposed a new method equipped with the contrastive predictive coding loss. Through this, they reported the state-of-the-art performance on Atari100k benchmark among the MBRL agents without look-ahead search.", "questions": "- In Figure 2, you showed well why one-step predictive loss can be insufficient to learn the world model, but what means the shaded lines in the graph?\n- Did you test the agent performance on Atari100k benchmark when doesn’t apply data augmentation? Only giving temporally near samples as the positive sample (without argumentation) is better I think (if the performance from that is comparable with the version equipped by the data argumentation).\n- There is another study to apply contrastive loss to MBRL agent [1], considering it as a reference can be helpful to make this paper better.\n- For AC-CPC Predictor, how did you input the actions? Did you concatenate them?\n- Can you test the performance when not using reconstruction loss for the world model training? It can be more native version of MBRL with contrastive loss.\n- In lines 426-432, your analyses why AC-CPC objective is helpful and what the previous works missed, make sense. As following your analyses, it makes sense that the reconstruction-based policy learning methods (IRIS and \\delta-IRIS) show better performance than TWM or STORM in the Mean Human Normalized Score measurement. Then, why they (IRIS and \\delta-IRIS) showed worse performance in Median Human Normalized Score measurement? Interestingly, your method outperforms for the two measurements. Can you explain which factors make this difference?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The breakthrough performance on Atari100k among the MBRL agents without look-ahead search. They showed clearly better performance than previous works, while the works have shown comparable performances for each other.\n- Good motivation and validation of that. Their hypothesis, the predictive loss for next frame is not good enough to learn meaningful information, is reasonable and they validated it well through their proposed agent and experiments.\n- Well written paper. We can enjoy to read this paper, because of well summarized related works, well discussed their motivation and diverse empirical ablation studies.\n- The analyses for the experimental results. They properly discussed their experimental results, for instance, why their model shows better performance on Atari100k (because of capturing small but important objects which can be missed only through reconstruction loss). Ablations of their architectures are studied substantially.\n- The detailed hyper-parameters are shared in the appendix.- We didn’t find a critical weakness from this paper.", "weaknesses": "- We didn’t find a critical weakness from this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)