Become a Proficient Player with Limited Data through Watching Pure Videos
Authors: Weirui Ye
Yunsheng Zhang
Pieter Abbeel
Yang Gao
Venue: ICLR 2023 poster (2023)
Keywords: Pre-training
Fine-tune
MCTS
Reinforcement learning
Vector Quantization

Abstract: Recently, RL has shown its strong ability for visually complex tasks. However, it suffers from the low sample efficiency and poor generalization ability, which prevent RL from being useful in real-world scenarios. Inspired by the huge success of unsupervised pre-training methods on language and vision domains, we propose to improve the sample efficiency via a novel pre-training method for model-based RL. 
Instead of using pre-recorded agent trajectories that come with their own actions, we consider the setting where the pre-training data are action-free videos, which are more common and available in the real world. We introduce a two-phase training pipeline as follows: for the pre-training phase, we implicitly extract the hidden action embedding from videos and pre-train the visual representation and the environment dynamics network through a novel \Changes{forward-inverse} cycle consistency \Changes{(FICC)} objective based on vector quantization; for down-stream tasks, we finetune with small amount of task data based on the learned models. Our framework can significantly improve the sample efficiency on Atari Games with data of only one hour of game playing. We achieve 118.4\% mean human performance and 36.0\% median performance with only 50k environment steps, which is 85.6\% and 65.1\% better than the scratch EfficientZero model. We believe such pre-training approach can provide an option for solving real-world RL problems. The code is available at \url{https://github.com/YeWR/FICC.git}.

Published as a conference paper at ICLR 2023
ВECOME A PROFICIENT PLAYER WITH LIMITED DATA THROUGH WATCHING PURE VIDEOS
Weirui Ye Yunsheng Zhang Pieter Abbeel Yang G20
Tsinghua " Artificial Intelligence Laboratory? Shanghai Qi Zhi Institute UC Berkeley*
ywr20emails ts wew.stsinghhte nnghua edu. en, ys-zhangg1@tsiinghuaa org. pabbeel@berke/l edu, geyyangiisstsiinngaacccm cn
ABSTRACT
Recently, RL has shown its strong ability for visually complex tasks. However, it suffers from the low sample efficiency and poor generalization ability, which pre- vent RL from being useful in real-world scenarios. Inspired by the huge success of unsupervised pre-training methods on language and vision domains, we propose improve the sample efficiency via a novel pre-training method for model-based RL. Instead of using pre-recorded agent trajectories that come with their own ac- tions, we consider the setting where the pre-training data are action-free videos, which are more common and available in the real world. We introduce two-phase training pipeline as follows: for the pre-training phase, we implicitly extract the hidden action embedding from videos and pre-train the visual representation and the environment dynamics network through a novel forward-inverse cycle consis- tency (FICC) objective based vector quantization; for down-stream tasks, we finetune with small amount task data based on the learned models. Our frame- work significantly improve the sample efficiency Atari Games with data of only one hour of game playing. We achieve 118.4% 88.4% mean human performance and 36.0% median performance with only 50k environment steps, which is 85.6% and 65.1% % better than the scratch EfficientZero model. We believe such pre- vreirrining approach can provide an option for solving real-worla RL problems. The code available at ht https: //g/t////tt gightuuubbb.... com/ Wwton//eW///FCC....
4 G(ts, 4512)
4411 D(ss, Fq)
Inverse
Forward
Latent action
Cycle consistency
Figure The newarddinverss cycle consistency builds an unsupervised training objective for model-based reinforcement learning from pure videos.
INTRODUCTION
Recently, deep reinforcement learning algorithms have achieved great success various tasks, including simulated games, robotics manipulations, protein structure analysis and even controlling nuclear fusion (Schrittwieser et al., 2020 Jumper et al., 2021 Degrave et al.) 2022). However, the great success of these RL algorithms is based on huge amounts data. For example, it requires over 20 million games Go for AlphaZero (Silver al. 2017). and Liu al. (202T) spend millions of data for playing simulated humanoid football. But in real applications or complex tasks, it is impossible acquire such amounts of data through interactions with environments.
To keep strong performance while requiring much less data, some researchers propose to use model- based reinforcement learning (MBRL) algorithms. They build environmental world models in assis-
"Equal contribution
Corresponning author

---

Published as a conference paper at ICLR 2023
tance of planning to increase the sample efficiency. And experiments have proved the high sample efficiency of MBRL (Kasser et al. 2019 Hafner et al. 2019). The high sample efficiency of MBRL shows the great potential for handling sequential decision-making problems in complex simulated environments and real-world (Hafner et al., 2020 Ye et al. 2021).
Although MBRL has improved sample efficiency lot, it still requires non-trivial amount of in- teractions to finish each task (Moerland et al.] 2020; SSchrittwieser et al., 2020). And each time, learns from scratch, which makes difficult to deploy quickly on different downstream tasks. Con- sequently, a good approach to improving this is to pre-train the world model with some data first. However, the datasets equipped with actions are hard to obtain on a large scale. This is because to collect a large-scale, high-quality dataset with actions, we need a good policy in the first place, and this becomes a chicken-and-ege problem. Instead, pure videos without action labels are more accessible and affordable in the real world. There are huge amount of video datasets without action labels on the Internet.Thar in this paper, we study how to pre-train the world models with action-free videos for MBRL.
We propose to pre-train a latent dynamics model based on inverse latent action prediction from pure videos without any action labels. We propose novel cycle consistency loss by chaining the forward dynamics and the inverse dynamics, as shown in Figure I This loss can pre-train the forward models as well as the inverse models in the visual MBRL algorithms. Afterward, downstream tasks based on the pre-trained models. Experiments show that our method can build sound representation and dynamies pretrained models for the downstream task. We achieve 1 118.4% mean human performance and 36.0% median performance with only 50k environment steps, which 85, 55.6% and 65.1 % better than the scratch EfficientZero model. Our contributions are the following:
fine-tune the
We systematically study the problem of pre-training from action-fred frree videos for model- based RI., which could be the foundation for future sample efficient, robust, and few-shot generalizable robots in the physical world.
We propose a forward dynamics- inverse dynamics cycle consistency pre- pre-training method that can jointly infer the latent actions from the video and train the representation function as well as the dynamics function. We also propose a practical fine-tuning scheme that achieves high performance on many downstream tasks.
Our framework achieves the SoTA on the 60-minute Atari games and significantly outper- forms others. Experiments show that the model pre-trained on distinct environmental data together can be fine-tuned well to the corresponding environments without re-pre-training.
2 RELATED WORKS
Unsupervised Pre-training in NLP and CV In recent work, researchers have found that the lan- guage model pre-trained with unsupervised learning can quickly and well generalized to down-steam language tasks (Devlin et al.) 2018 Yang al. 2019). Some researches show that the pre- pre-trained model can be a good few-shot learner (Brown et al.
2020) or multi-task learner (Radford et al.)
2019). More importantly, the two-stage procedure of training has become more popular for large models in NI.,, such as Transformers (Vaswani et al. computer vision, similar unsupervised pre-training methods can build sound representation models for various visual tasks based on transformer (Li et al. learning and reconstruction are two common techniques to achieve these goals (He et al. 2020 Grill
2017] Brown et al. (020). People find that in
2019 Dosovitskiy et al. 2020). Contrastive
et al. 2020 He et al. 2022). Generally, all these methods aim to model a universal representation function, which can be fine-tuned well to some specific vision tasks, e.g. classification or detection.
Unsupervised for Representation Learning in RL Inspired by the great success of unsupervised pre-training in NLP and CV, researchers attempt to learn representations for visual RL in an unsu- pervised manner. People find that contrastive learning on online visual RL helps to extract good latent states for robotics control tasks (Laskin et al. 2020 Schwarzer et al. 2020). Ye et al. (2021) propose to improve the sample efficiency of basel RL through temporal contrastive learn- ing. Furthermore, Stooke et al. (202I) introduce a new unsupervised learning task to decouple the representation learning from policy learning. Xiao et al. (2022) propose to do better motor control from the masked visual pre- training method from world images. Besides, some researchers attempt do pre-training and fine-tuning for RL down-stream tasks. Parisi et al. (202)) find that
2

---

Published as a conference paper at ICLR 2023
ImageNet pre-trained visual representations be competitive ground ground-truth state representations for control policies. Schwarzer et al. (2021) propose to pre-train from offline datasets based on sev- eral unsupervised learning objectives and then fine-tune for down-stream tasks. For model-based algorithms, Deng et al. (2022) propose to pre-train the world model with unsupervised losses. All of the work above either study the model-free pre-training, or assume there actions available during model-based pre-training. Our work studies the action-free model-based pre-training problem.
Unsupervised learning from action-free videos Pure videos action-free videos are much more accessible and affordable in the real-world. Unsupervised learning from action-free videos has been studied in computer vision and imitation learning before. Dwibedi et al. (2019) and Waag et al. (2019) propose to utilize the temporal cycle consistency as the unsupervised learning objective for multiple vision tasks. Menapace al. (202T) propose to predict action labels for playable video gen- eration from unlabeled videos with image reconstruction objectives. In imitation learning, [Edwards] al. (2019) studies the imitation from observation alone problem and proposes a similar unsuper- vised objective to ours. However, they do not work the reinforcement learning problem. The closest work to ours is Seo et al. (2022), where they pre-train the model with evidence lower bound based Dreamer-v2 (Yarats et al. 2021). However, unlike our approach, they do not explicitly infer latent actions.
3 PROBLEM FORMULATION
Before introducing our algorithm, we give some basic notations for the model-based reinforce- ment learning (MBRL. algorithms following conventions in Moerland et al.] (2020). Firstly, the formal definition of Markov Decision Process (MDP) (Puterman, (014) defined as the tuple (S,A, P. r. 7). Here, S is the state space, A is the action space, p(s++1 ar) is the probability function of the transition dynamics, Ti = r(s, az) is the reward function corresponding specific tasks, and y E [0, 1] is the discount factor.
In model-based reinforcement learning algorithms, dynamics model learning is learned ap- proximate the transition dynamics of the environments. People aim to train a dynamics model D : A - S through supervised learning. For the visual-based tasks, it more computation- efficient to build representation network R O - S to extract the latent states S from observations O. Afterward, exploration techniques or planning algorithms can be applied based on the models R, D. One can utilize those models with model-free algorithms, such as predicting rewards.
In this work, we study the pre-training of R and D model-based RL. Here R and D are task agnostic, and they are thus widely applicable different downstream tasks. The R model can be learned from data, no matter with actions or without actions. However, D usually requires explicit actions as input, which is expensive to obtain in the real world. In this work, we study the task of pre-training R and D end-to-end from pure, i.e., action-free, videos.
4 METHOD
Compared to the data with action labels, pure videos more common and accessible in the real world. However, pre-training the world models with such action-free data is challenging because lack the ground truth action train the action-based dynamics model. To address this dilemma, we propose a novel forward-inverse cycle consistency (FICC), an unsupervised training objective for pre-training the models model-based RL. Our FICC framework is shown in Figure 2 first employs an inverse dynamics model that infers latent action z from the current state s; and next state 8141 from pure videos. With the inferred latent action and the current state st, forward dynamics model maps them to the predicted next state $1+1. Our proposed norward-invers cycle consistency loss is pulling the $141 close to 8t+1. Here the forward and inverse in FICC refer to forward dynamics and inverse dynamics, respectively. The cycle here refers to the 8t+1. where it is first mapped to the latent action 2 and then mapped back to itself again. We will discuss the FICC framework in detail in Section 4. To avoid shortcuts, model. called Latent Action Generator (LAG), which is discussed in Section 4.2 Finally, we will introduce how to use such pre-trained model for the downstream tasks in Section
propose a special design for the inverse
4.3
3

---

Published as a conference paper at ICLR 2023
4.1 FORWARD-INVERSS CYCLE CONSISTENCY IN PRETRAINING
Figure 2 shows our overall Forward-inverse cycle consistency framework. First, we use convolu- tional neural network R to extract states from the observations. This allows us to have a compact and semantic meaningful latent state to operate on. The representation model also alleviates the computation cost for the inverse and forward dynamics models. Then an inverse dynamics model takes in and 8t+1 and output the inferred latent action z. Note that we do not require the inferred latent action z to have a one-to-one correspondence to the ground truth action. The mapping from the actual action to latent actions will be later learned when fine-tuning on the downstream task. Finally, the forward model takes in the inferred latent action z and the current state 8, to predict the next state 1- We use a cosine similarity loss to pull the 8 close to the true state 51+1-
Besides the cosine similarity cycle consis- tency loss, find that adding extra re- construction loss further helps to stabilize the training. We add two reconstruction losses. First, we reconstruct o, from 8t, which will ensure that all information has been encoded in the latent state. Second, also reconstruct O1+1 - o, from 8, and 2. This helps the inverse dynamics model to focus on the changes in the environ- ment. We also unrolled those losses for five steps, since the dynamics models usu- ally unroll for multiple steps in the down- stream tasks. summary, the unsuper- vised pre-training loss is:
Torppral Cumsiacur
Forwardd-laverse Cycle Consistency (FICC)
Figure 2: The temporal bowward-invers cycle consis-
tency (FICC) for pre-training environmental models.
eycle consistency
C (3 1(41)
In p(o+4++ - Oz (5t, Zgg)
In p(olls))
(I)
similarity
reconstruction
reconstruction
where St R(or), 81+1
R(o++1), 2g = inverse(st. $1+1), 8.41
D(s 2g). The hyper-
parameters and more implementation details such as unrolling are in Appendix
4.2 LATENT ACTION GENERATOR BASED ON VECTOR QUANTIZATION
There is one caveat in the FICC framework we introduced in Section that is, there might be a shortcut in the cycle consistency. Since the latent action z is a continuous vector, it can encode unlimited information theoretically if the scalar precision of Z is infinite. Consequently, there is one shortcut that the latent action 2 can encode most information of $t+1 rather than the difference of st and $1+1- And the inverse model can directly copy all the information in st+1- Correspondingly, the forward model can learn an identity function. This shortcut will achieve zero loss the cycle consistency defined, but won't be useful for the downstream task all. The reason that the above shortcut happens is because the latent action contains too much information. We know that in practice, the action we take usually has small amount of information. For example, in video games, the actions are some key strokes. In the robotics domain, the action can be represented by few numbers describing the joint location and speed, etc.
In order to avoid this shortcut, we propose an inverse- inverse-dynamic, dynamics model called Latent Action Gen- erator (LAG) that regularizes the latent action have limited amount of information. People have proposed several techniques to enforce information bottlenecks in the neural networks, such as using low dimensional features, variational information bottlenecks (Alemi al. (0166), or vector quantizations (Van Den Oord et al.) 2017). In theory,
any of those techniques here to avoid the shortcut. In this paper, we explore using vector quantizations for this purpose in favor of their scalability and ease of training (Van Den Oord et al. 2017).
Figure 3 shows our proposed LAG model. First, a CNN takes in st and S141, outputting a latent action embedding Ze E RD. We also define a dictionary of vector quantization embeddings {er, E2, ................................ , }. The dimensionality of each vector in the dictionary is the same as Z.. The output of LAA is defined as the nearest dictionary embedding ze, denoted 2q. The zn is the latent action
4

---

Published as a conference paper at ICLR 2023
61
Ra
Qunnticatio
X
CNN
(artion-left
(ietlon-rghh)
X
Ncarol Neghter
Latent Artions Generatue (LAGG)
Generate Lafent Action
Figure Left: Acquire the latent action embeddings through vector quantization based on the adjacent frames. Right: Illustration of the latent action representations. Different actions will be mapped into different embeddings.
we discussed in Section 4.1 In the standard VQ-VAE, VAE, there are multiple quantized embeddings for single image. However, we find that using only one embedding is enough in our case.
The training of LAG is similar to VQVAE Van Den Oord et al. 2017), where the codebook loss brings the code closer the encoder output, and the commitment loss prevents the encoder from switching between different codes. The training objective of the LAG is:
Loq = [ S14 1]] - ell2 + (8+,
8t+1) sg(e]ll2
(2)
where =n. denotes the CNN that outputs sg denotes the stop-gradient, is the commitment trade- off, and it is set to We conduct the ablation study to show the significance of the information
bottleneck (Section 5.4 More implementation details are attached in Appendix D
Finally, the unsupervised training objective in the pre-training phase is L Lee + aLvg. a =
4.3 FINE-TUNING WITH ACTION ADAPTER
During fine-tuning dtwn-stream tasks, the agent interacts with the en- vironment with real actions. How-
Algorithm Building Action Adapter
1: Given pre- pre-trained LAG, embedding codebook e
2: Cla, ki] <- 0,a E A, E K; M <- Dict(}
ever, the dynamics model is pre- trained via the latent action embed-
3: for L E N do
4: Obtain triple (st, 01,,0,,541)
ding. The only gap between them the format of actions. Fine-tuning can be quite straightforward if each real action a, mapped to the corre- sponding latent action embedding e* after the representation model and dynamics model D are pre-trained. For convenience, we name such map M A - R. as action adapter. Then the dynamics model can pre- dict the next states with the action adapter: $ = D(s M(a)))
5: e₁ LAG S(+1)
6: C(a, k| + kla,k]+!
7: end for
8: Sort Ta, = (C(a, k]. k), E A, K by Cla, k]
9: Such that a < b, Ta,k To, and k < l,Ta To,
10: for Cla, k), a, E T do
11:
if M.keys( V ek E +.valuus() then Continue
12:
13: end if
14: M[aa] <-
15: end for
16: Return
We statistically build such adapter during fine-tuning as Algo. shows.
During fine-tuning, the embedding codebook frozen for a stable inference of the dynamics model. When a caate-action-stazl state transition triple (st, at, St41) is collected in self-play, the pre- trained LAG will infer the corresponding latent action e, and store the relationship tuple (a,, et). We map each real action to the most common latent action. We also require different real actions to map to different latent actions. Please see Appendix [D.3 for more implementation details.
With the pre-trained R. D and the action adapter M, we fine-tune them on the downstream tasks. In this work, we use EfficientZero (Ye et al. 2021) as the model-based RL algorithm because of its high performance with limited data. EfficientZero is composed of the representation network R, the dynamics network D, the policy network π, the value network Vs)) and the reward predictor r(s,a), a). Given the pretrained R and D, fine-tune with the two learning rates: the small fine-tune learning rate ly for the pretrained components, and the large learning rate I, for the components that not initialized (T,, V (s), a)).
5

---

Published as a conference paper at ICLR 2023
5 EXPERIMENTS
As mentioned above, we propose three key components for handling the pre-training and fine-tuning with pure videos: 1) forward-inverse cycle consistency; 2) latent action generator; 3) statistical action adapter for mapping the real actions latent actions. In this section, aim to answer the following two questions: 1) Whether the pre-training can help downstream RL tasks; 2) What is the importance of each component. We conduct systematic experiments to answer those questions.
5.1 EXPERIMENTAL SETUP
Environments Recently, sample efficiency has become an important topic in RL. Kaiser al. (2019) introduce the Atari 100k benchmark, including 26 games. This benchmark consists of 100k steps of interactions with environments, which is equivalent two hours of human game playing. This benchmark is a popular benchmark for evaluating the sample efficiency of RL algorithms due to the complex visual observations and diverse tasks (Kaiser et al. 2019 Kielak 2020 Srinivas al.
2020 Kostrikov et al., [2020] Ye et al. (222]) Recent developments in sample efficient RL (Ye et al,, 2021 have achieved super-human performance on the Atari 100k benchmark. In order further challenge the RL algorithm, we propose to use the Atari 50k benchmark, which only consists of 50k steps or one-hour game-play of interactions. We conduct the experiments on top of EfficientZero, which is the current SoTA model-based algorithm on the Atari 100k benchmark.
papeer-prraetters The hyper-parameters for pre-training are listed in Appendix B For fine-tuning, we train for 50k steps and follow other hyper-parameter settings in EfficientZero, which are listed We update the action adapter every 1000 transitions.
in the Appendix C
Evaluation Mnih et al. (2015) provide baseline of human sample-efficiena and we follow the common evaluation metric for the 26 Atari games, namely comman-normalieed score (HNS). It is defined as (scOreagent score / /scoOreummmn scOrerandom). In each game, average the scores over 100 evaluations across 3 runs with different seeds. Following the previous works, we report the median (Mdn) and mean (Mn) HNS over the 26 games. And use the statistical tools proposed by Agarwal et al. (2021) to quantify uncertainty.
Pretraining Dataset We the EfficientZero replay buffer the pre-training dataset. We train the EfficientZero for IM transitions from scratch and save the replay buffer as the pre-training dataset.
Baselines We compare the following baselines. EfficientZero (Ye et al. 2021 trained from scratch: since we use EfficientZero as our base model-based RL method, we train from scratch for the 50k environment steps as the baseline. To the best of our knowledge, there are few works studying pre- training with action-free videos on Atari games. SGI (Schwarzer et al.) 202I) digs into the similar learning paradigm and achieves high sample-efficiency on Atari games. We also compare to SGI, which proposes pre-training with action-labeled data on Atari games. All the comparisons are under the same setting of 50k environment steps. We note that SGI an "oracle" method since it uses ground truth actions during pre-trrin
5.2 PERFORMANCE COMPARISON
HNS Scores We compare our method with the above baselines on 26 Atari games for 50k en- vironment steps. As illustrated in Table our method outperforms the previous SoTA method Efficien(Zero(EZZ) on large scale in such limited data settings. 20 of total 26 games get superior performance with our pre-trained models. Specifically, we achieve 0.360 median HNS as well as 1.184 mean HNS and outperform the human level in 6 games. Compared to SGI, method with access to ground truth actions during pre- traiinnni we also outperform by a large margin. AAgaraal al. (2021) propose to use the inter-quartile mean (IQM) normalized score and quantify the uncer-
Table 1: Comparisons of HNS on 26 Atari 50k for fine-tuning with 3 different runs. Method
Median
Mean
IQM
require action
EZ. scratch
0.218
0.638
0.208
w.o. pre-training
SGI
0.132
0.739
0.290
EZ + FICC (ours)
0.360
1.184
0.353
X

---

Published as a conference paper at ICLR 2023
Humen Normellzed Score
Human Normellonn leed Score
0.74
Human
0.93
Environment Steps
Environment Steps Krull
Environment Steps
DemonAttack
Pong
Btoeizee Itrer
EPk exiters
Human Normalzzd Score
20
Humen Normal lced Score
Human
2.0
2.8
Environment Steps
1+4
Environment Steps
Environment Steps
Figure 4: Learning curves on 6 out of 26 Atari games. We compare the curves of vanilla EZ and EZ with our pre-trained model from action- free videos. indicates that our pre-trained model is able to improve the sample efficiency during the fine-tuning process.
tainty via percentile bootstrap confidence intervals. And our method is much superior on Atari 50k, including all these metrics.
Sample Efficiency Figure shows the learning curves of our method and the baseline EZ method on Atari games. Here the orange curve is the vanilla EZ, while the blue curve is EZ with our pre-training method. We find that our pre-training method consistently improves the HNS scores in all environments. Further, our method outperforms the baseline under almost any amount data.
5.3 SHARING PRETRAINING AMONG MULTIPLE ENVIRONMENTS
The pre-training model experimented with in Section 5.2 trained per environment. That is, each pre-trained model only sees the data from the downstream task. However, in natural language processing and the image recognition domain, large-scale pre-trained models are only trained once and later applied to many different downstream tasks. A natural question is that, can our framework supports training one model and fine-tune on multiple tasks?
In this part, we aim to answer this question. We select set of 6 environments (as shown in Table 2), and pre-train a single model with our method on data from them. accommodate the larger seale of pre-training data size, we use 3 residual blocks instead of in the representation and dynamics model. All other settings are the same as before. We name this model as EZ-L, where denotes "large". Firstly, pre-train the EZ-L model across all the 6 environmental datasets and obtain the action embedding codebook e. Then we do fine-tuning on the cnvironments respectively based on the pre-trained EZ-L model. For each task, build the corresponding action adapter M during fine-tuning. In this way, we pre-train one model and fine-tune on multiple tasks. Here we follow
Table 2: Pretrain games with different styles and rules in a single model, and then fine-tune each game with the same pre- preitrained model (EZ-L). Experiments show that EZ-L achieves superior performance compared EZ scratch on most games.
Games
Boxing
Breakout
CrazyClimber
MsPacman Pong
Qbert
EZ scratch
0.597
8.211
1.183
0.040
0.589
0.199
EZ with FICC
1.949
12.477
1.533
0.051
0.888
0.305
EZ-L with FCC
1.230
9.361
1.133
0.051
1.099
0.279
7

---

Published as a conference paper at ICLR 2023
exactly the same fine-tuning hyper-panne ppeer-paramtters. The results are listed in Table 2 EZ-L improves upon EZ scratch baseline in 5 out of 6 environments. In Pong, it even outperforms the environment--pecifie pre-training by a large margin. In summary, show that it is possible to have only single pretrained model and get the benefits of the pre-training in most environments.
We found that
5.4 ABLATION STUDY
In this section, we will conduct some ablation studies investigate the two significant components of our method, the forward-inverse cycle consistency in Sec. 4. and the vector quantization for latent action generation in Sec. 4.2 We choose four different games for this ablation. We run each experiment for three different seeds with 100 evaluations. Other byper-parameten are all the same. We make more ablation studies about the datasets and dimensions of latent action 2 in App. A
FICC and Reconstruction Objectives Ablation Here we do ablation studies to further investigate the effect of the FICC and the reconstruction objectives mentioned in Eq. | As for the FICC ab- lation, we remove the image difference reconstruction objective In p(r+0 0r|st, =q). To show the individual performance delta of each reconstruction loss, we remove the image reconstruction objective In p(oc The results are shown in Table 3 We find that the removal of the image differ- ence cycle consistency loss has deteriorated the performance by large margin. But it is still better than the EZ from scratch baseline. This supports our claim that this loss encourages the inverse dynamics to focus on the changes in the environment. Moreover, the model pre-trained without OR has comparable performance to the Pre-train with FICC. Therefore, the cycle-consistent difference reconstruction is more significant and it can work well without OR. Compared with FICC and FICC without OR, the former has more advantages in most environments, except that the latter has obvious performance improvement in the Boxing environment. We believe that some properties in the Box- ing environment make the model learn harmful representations from observation reconstruction.
VQ Avoids Shortcut Learning Ablation In Sec. 4.2 we discuss the need for the information bottleneck. Here we ablate this component to investigate the significance of it. The latent action generator takes in the adjacent states and outputs one embedding to represent latent actions. We build a LAG to generate latent action without the quantization step. The output of the neural network Re 28t,, 81+1) ) is directly fed into the dynamics model instead of utilizing the vector quantization to replace ze with Zq. For convenience, we name such LAG with direct outputs as LAG-D and the LAG with VQ technique as LAG-VQ. Here "D" is short for direct. Following the previous ablation environments and parameters, the results are listed in Table For some games, the LAG-D keeps comparable performance to LAG-VQ (Breakout). However, for some games, such Krull underlined the table, LAG-D performs even worse than the scratch model. It indicates that the model pre-trained without the information bottleneck harms the fine-tuning on some down-stream tasks, which can be caused by short-cut learning. We notice that in some game, such as Pong, the LAG-D is slightly better than the LAG-VQ. However, from the perspective of most games, LAG-VQ is much better than LAG-D. Table 3:
Ablation of the cycle consistency objectives for 50k environments steps fune-tuning, The cycle consistency improves the performance of all four evaluation games.
Games
Boxing
Breakout
Krull
Pong
EZ scratch
0.597
8.211
1.164
0.589
Pre-train with FICC
1.949
12.477
6.455
0.888
FICC without Observation Reconstruction (OR)
6.002
12.128
3.692
0.816
FICC without Difference Reconstruction (DR)
1.160
11.156
3.367
0.648
Table 4: Ablation of the types of LAG for 50k environment steps fune-tuning. LAG-VQ is the original implementation. while LAG-D outputs the latent action directly from the neural networks. For games like Krull, LAG-D behaves even worse than the scratch model. LAG-D fails to build an information bottleneck, which might cause some short-cut learning during pre-training.
Games
Boxing
Breakout
Krull
Pong
scratch
0.597
8.211
1.164
0.589
LAG-VQ
1.949
12.477
6.455
0.888
LAG-D
1.174
12,435
0.712
1.095
8

---

Published as a conference paper at ICLR 2023
Tole
M
F
Figure 5: Dynamics prediction visualization of the pre-trained model. Top left: Breakout. Top right: Krull, Bottom left: Boxing. Bottom right: Pong. The first line of each environment is the ground truth images, and the second line the images predicted and generated by the model.
Instance 2
65.3
Figure 6: Visualization of the latent action effects. Each row in the figure shows a different observa- tion. The yellow dashed line the anchor of the board position at time step The left two images of each line are the current observation on and the ground truth next step observation 0s+1- The six figures on the right are the reconstructed images with the index 1 to index latent actions as the input to the dynamic function, which results in distinct predictions 0,+1.
5.5 VISUALIZATION ANALYSIS
In this section, do some visualizations verify the effectiveness of the latent action generator (LAG) as well as the dynamics model based on the latent actions.
Dynamics Prediction In Figure we visualize the dynamic function output of the pre- treitrained model with 5 unroll steps. each step t, we use the LAG to get latent action embedding 2, from the current state 8, and next state 1, 1. and consistent with the pre-training algorithm. From the results, we observe that the pre-trained model can make high-quality reconstruction for the images of the next several steps. The prediction ability of the model includes the motion of objects in the scene (the ball in Breakout and Pong), the motion of the agent itself (Breakout's guard, Pong's s right guard, Boxing's left character), and the motion of other characters (Boxing's right character). This shows that our LAG can effectively encode the differences between the two states.
decoder to reconstruct the output. The whole process
Latent Action Embeddings The effects of latent actions are visualized in Figure Here we select the game Breakout and visualize 6 of the total 20 latent actions. Results show that latent actions have the same meanings for different rows. For example, latent action with index 6 indicates that the guard moves to the left, while actions with index indicate that the guard moves to the right.
DISCUSSION
In this paper, we propose to pre-train the environment models via a novel forward-inverse cycle consistency from the action-free videos. To achieve this, vector quantization based latent action generator introduced to avoid the shortcut learning. We evaluate the pre-trained model via fine- tuning for Atari games with limited 50k environment steps. Experiments prove the efficiency of our method. As for the limitations, current architecture is difficult to handle the continuous action space. We leave this future work.

---

Published as a conference paper at ICLR 2023
7 REPRODUCIBILITY STATEMENT
The main implementations of our proposed method are in Section 4.1 4.2 and 4.3 In addition, the settings the experiments and hyper-paramette we choose are Appendix B And the implemen- tation details are in Appendix D
ACKNOWLEDGMENTS AND DISCLOSURE OF FUNDING
This work is supported by the Ministry of Science and Technology of the People's Republic of China, the 2030 Innovation Megaprojects "Program on New Generation Artificial Intelligence" (Grant No. 2021AAA0100000) This work is also supported by grant from the Guoqiang In- stitute, Tsinghua University.
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare. Deep reinforcement learning at the edge the statistical precipice. Advances in Neural Informa- fion Processing Systems, 2021.
Alexander A Alemi, Ian Fischer, Joshua Dillon, and Kevin Murphy, Deep variational information 2016.
bottleneck. arXiv preprint
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. Mag- netic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897):414- 419, 2022.
Fei Deng, Ingook Jang, and Sungjin Ahn. Dreamerpro: Reconstruction-fret model-based rein- forcement learning with prototypical representations. International Conference on Machine Learning, pp. 4956-4975. PMLR, 2022.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 1929, 2020.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. Tem- poral sycle-consistenss learning. In Proceedings the IEEE/CVF conference computer vision and pattern recognition, pp. 1801-1810, 2019.
the
Ashley Edwards, Himanshu Sahni, Yannick Schroecker, and Charles Isbell. Imitating latent policies from observation. In International conference on machine learning, pp. 7755-1763. PMLR, 2019.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-2128. 2020.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603. 2019.
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with dis- crete world models. arXiv preprint arXiv=2010.021993 2020.
10

---

Published as a conference paper at ICLR 2023
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9733 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger,
Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583- 589, 2021.
Lukasz Kaiser, Mohammad Bahaeizadeh, Piotr Milos, Blazej Osinski, Roy H Camphell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model-based reinforcement learning for atari. arXiv preprint 1xXxv:1903.000 2019.
Kacper Kielak. Do recent advancements in model- deep reinforcement learning really improve data efficiency? arXiv preprint arXiv:2003. 10181, 2020.
Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint TrXiv:2004.1364 2020.
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl:
Contrastive unsupervised representa- tions for reinforcement learning. In International Conference on Machine Learning, pp. 5639- 5650. PMLR, 2020.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-We Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:/908..00 035577 2019.
Siqi Liu, Guy Lever, Zhe Wang, Josh Merel, SM Eslami, Daniel Hennes, Wojciech M Czarnecki, Yuval Tassa, Shayegan Omidshafiei, Abbas Abdolmaleki, et al. From motor control to team play in simulated humanoid football. arXiv preprint arXiv:2105. 2021.
Willi Menapace, Stéphane Lathuilière, Sergey Tulyakov, Aliaksandr Siarohin, and Elisa Ricci. Playable video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10061-10070, 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. nature, 51 (188750))529-533. 2015.
Thomas M Moerland, Joost Broekens, and Catholijn M Jonker. Model-based reinforcement learn- ing: A survey. arXiv preprint arXiv:2006.16712 2020.
Simone Parisi, Aravind Rajeswaran, Senthil Purushwalkam, and Abhinav Gupta. The unsurprising effectiveness of pre-trained vision models for control. arXiv preprint arXiv:2203.03355 2022.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley Sons, 2014.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):004-69) 2020.
Max Schwarzer, Ankesh Anand, Rishab Goel, Devon Hjelm, Aaron Courville, and Philip Bach- man. Data-efficient reinforcement learning with self-predictive representations. arXiv preprint arXiv:2007.05929, 2020.
Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch, Ankesh Anand, Laurent Charlin, R De- von Hjelm, Philip Bachman, and Aaron C Courville. Pretraining representations for data-efficient reinforcement learning. Advances in Neural Information Processing Systems, 34:1 1e111266666999 2021.
11

---

Published as a conference paper at ICLR 2023
Younggyo Seo, Kimin Lee, Stephen L James, and Pieter Abbeel. Reinforcement learning with action-free pre-training from videos. In International Conference on Machine Learning, pp- 19561-19579. PMLR. 2022.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self- self-play with general reinforcement learning algorithm. arXiv preprint arXiv:1712.0183 2017.
Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised representa- tions for reinforcement learning. arXiv preprint arXiv:2004.000 14,,, 2020.
Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In International Conference Machine Learning, pp. 9870-9877 PMLR, 2021.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Xiaolong Wang, Allan Jabri, and Alexei A Efros. Learning correspondence from the cycle- consistency of time. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2566-2576, 2019.
Tete Xiao, Ilija Radosavovic, Trevor Darrell, and Jitendra Malik. Masked visual pre-training for motor control. arXiv preprint arXiv:2203. 06173, 2022.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous con- trol: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.096-5. 2021.
Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, and Yang Gao. Mastering atari games with limited data. Advances in Neural Information Processing Systems, 34:25476-25485 2021.
12

---

Published as a conference paper at ICLR 2023
A MORE ABLATION STUDIES, COMPARISONS, AND VISUALIZATIONS
A.1 ABLATION: DIFFERENT DATASETS
Currently, the dataset used in this work is from the EfficientZero replay buffer. It is important to investigate the impact of pre-training data with different collecting methods. To measure this, we make ablations for different pre-training datasets under the same settings. We compare the perfor- mance of the models pre- rreitra from EZ replay buffer, random trajectories, DQN weak dataset, and expert trajectories. As for DQN weak dataset, which is similar to SGI, we choose the first IM steps from the DQN Replay dataset. As for the expert trajectories, we rollout an expert agent repetitively with 25% random actions to increase dataset diversity. Here, each dataset contains 1M transitions. We keep the same experimental setting and choose the same 4 environments with three different seeds for fine-tuning on Atari 50k. The results are as 15
Table 5: Ablation of different collected datasets for 50k environment steps fine-tuning. There are four kinds of datasets: EZ Replay buffer, Random trajectories, DQN Weak dataset (first 1M transi- tions), and Expert trajectories. The model pre-trained from Random trajectories can perform well. And our method can fit with datasets collected by distinct agents.
Games
Boxing
Breakout
Krull
Pong
EZ scratch (No pre-training)
0.597
8.211
1.164
0.589
EZ. Replay buffer
1.949
12.477
6.455
0.888
Random trajectories
2.216
11.604
4.136
0.736
DQN Weak dataset
3.279
10.422
5.566
0.942
Expert trajectories
2.785
11.655
5.159
0.768
Notably, SGI cannot work under the random dataset the DQN weak dataset. But method can work. Specifically, we have three conclusions: (1) Our pre-training method is not sensitive to the quality of datasets. Even if pre-trained under random trajectories, the model has improved significantly among the 4 environments.
(2) Our pre- pre-training method is not sensitive to the data- collection method. In terms of the DQN weak dataset, our method can still work well and even outperform the model pre-trained under EZ Replaybuffer in 2 environments. (3) Expert trajectories not the best choice of pre-training dataset in method.
A.2 ABLATION: DIMENSIONS OF LATENT ACTION ze OF LAG-D
In Sec. 5.4 we compare the performance between LAG and LAG-D which generates the latent actions without vector quantization. And results show that the model pre-trained with LAG-D can cause shortcut learning and harm the fine-tuning in some games. Therefore, another question is: can the shortcut learning be avoided for LAG-D through simply reducing the dimension of the latent action ze. To answer this, we reduce the dimension of ze from 5 to 2 and use the LAG-D for pre- training. We keep the same experimental setting and choose the same 4 environments with three random seeds. The results are listed in Table After reducing the dimension of ze, the LAG-D still fails in the game Krull. It indicates that simply reducing the dimension of že cannot avoid shortcuts.
Table 6: Ablation of different dimension of ze in LAG-D for 50k environment steps fine-tuning. Here we reduce the dimension of s, to and the LAG-D still fails in the game Krull.
Games
Boxing
Breakout
Krull
Pong
EZ scratch
0.597
8.211
1.164
0.589
LAG- (dim=5)
1.949
12.477
6.455
0.888
LAG-D (dim=5)
1.174
12.435
0.712
1.095
LAG-D (dim=2)
1.465
10.925
0.759
0.741
A.3 COMPARISONS OF MORE ALGORITHMS
Apart from the comparison to SGI, we also compare our method with APV Seo APV et al, (2022), which proposes stack action-free dynamics model before the original dynamics model based on
13

---

Published as a conference paper at ICLR 2023
Dreamer V2 Hafner et al. (2020). We try to their method on the Atari benchmark, however, we find that it completely fails under the 50k sample limit. In order to achieve non-zero perfor- mance,
10x data during fine-tuning, namely Atari 500k. As for pre- pre-training data for APV, we choose the best choice suggested by [Schwarzer et al. (202T): random sample 3M samples from checkpoints throughout the DQN training- All the other training details follow the original paper. We run on the same 4 ablation environments used in our paper and aggregate the results with three seeds. The results are shown in Table 7 In general, our method has superior performance than APV because we are able to infer the latent actions during pre-training. But APV pre-trains the model without action conditions.
Table 7: Comparison between our method and APV. Our method outperforms APV in general.
Games
Boxing
Breakout
Krull
Pong
EZ scratch; 50k
0.597
8.211
1.164
0.589
EZ Pre-train with FICC; 50k
1.949
12.477
6.455
0.888
APV; 50k
0.057
-0.013
0.623
0.020
ALV; 500k
0.186
0.080
35.028
0.161
B HYPER-PARAMETER FOR PRE-TRAINING
The hyper-parameters of pre-training are listed in Table The million pre-training data is col-
lected from EfficientZero training from scratch.
Parameter
Setting
Observation down-sampling
96 96
Frames stacked
4
Frames skip
Minibatch size
256
Optimizer
SGD
Optimizer: learning rate
0.02
Optimizer: momentum
0.9
Optimizer: weight decay Learning rate schedule
10-4
0os 0.02 -> 0.0002
Max gradient norm Training steps Pre-training data
50K
1M
Unroll steps for training dynamics
Shape of the state
64 6 x 6 20
VQ: Number of latent action embeddings
VQ: Dimension of latent action embeddings
Table Hyper- parameterss for FICC pre-training.
C EYPER-PARAMETE FOR FINE-TUNING
During fine-tuning, we follow the same training hyper-parameters of EfficientZero except for reduc- ing the learning rate of the pre-trained models, namely the representation model and the dynamics model. For the pre-trained model, choose 0.05 the initial learning rate while choose 0.2 for those un un- pretraiidd models. We use SGD optimzier as well as constant learning rate scheduler, which is also the same EfficientZero.
14

---

Published as a conference paper at ICLR 2023
D IMPLEMENTATION DETAILS
D.1 NETWORK DESIGN DETAILS
The design of the representation and dynamics is consistent with EfficientZero since we use as the model-based algorithm the downstream tasks. The implementation details be founded at (Ye) et al. 2021) Appendix A.I. The representation network takes 3 x 96 x 96 tensor as input, which down-sampled from Atari game image, and outputs the state tensor with shape 64 x 6 6. The dynamics network takes the current state and current action as input and output the next state 8141 with shape 64 6 x Regarding dynamics network, the only difference between us and EZ that the shape of the input action tensor is 5 x 6 x 6 instead of 1 x 6 x 6 because we map the ground truth action to the latent action embedding.
The design of the LAG is listed here:
Concatenate the current state 8, and the state 81+1 into 128 planes.
convolution with 64 output planes, with BN and ReLU
residual block with planes.
convolution with output planes, with BN and ReLU
Flatten the planes with shape 5 x x 6 into 180 dimensions vector.
I fully connected layers with 5 output dimensions, with is the dimension of latent action
embeddings.
VQ model to quantized the vector to an action embedding.
Expand dimension at the end the action embedding and repeat 6 times to 6 x 6.
Note that the pre-trained LAG is used and only used for makeup counting table C' during finetuning.
The design of the image decoder listed here:
3 transposed convolution with 64 planes, stride 2, with BN and ReLU
convolution with 3 output planes.
Sigmoid activation to output RGB image.
D.2 UNROLLING DETAILS IN TRAINING
The unroll and loss calculation in our pre-training process is as follows: (1) Randomly select trajectory from the video pre- pre-training data and generate image sequences with the length T + 1, namely Op. oI, Or chronologically.
(2) Obtain the state tensor using the representation network: 8, = R((o)) for / from to T. (3) Generate all latent action vectors using the latent action generator (LAG)): 2 T.AG(se, (8t, St+1) for from 0 to T 1, while building the training objective f of the LAG using (2). For convenience, we set the first predicted state špto be so i.c. ån R(oo) 8p- (4) Predict next 7 time-step state tensor using dynamics network and latent action: 5r+1 D(,,, 21) for from 0 to 1. Note that compared with the MuZero MuZero-style method, our dynamics network uses latent action 2 as input rather than ground truth action a, because of our action-free pre-train setting. (5) Use reconstruction head and difference reconstruction head to predict o, and d, = o++1 o, from & for from T 1. The two reconstruction heads both consist four transposed 2d- convolution layers. (6) Finally, calculate Leett for & from 0 to T using Eq.(1) and sum up the losses: Lee + aLeg ET-1 (Lee,t + aLvg,t) a for backpropagation.
D.3 ADAPTER DETAILS
In our implementation, we maintain the counting table C in the self-play workers. At the end of each MCTS search, after selecting the action o and getting the next state $141, follow line 5 to line 7 in [lgorithm[]] put s, and $ into the pre-trained LAG to get the index & of the latent action embedding and update the counting table C.
15

---

Published as a conference paper at ICLR 2023
Table 9: Scores achieved on the Atari 50k benchmark.
Game
Random
Human
EfficientZero
SGI
Ours
Alien
227.8
7127.7
456.0
839.1
625.7
Amidar
5.8
1719.5
48.1
150.9
88.6
Assault
222.4
742.0
586.2
723.5
814.4
Asterix
210.0
8503.3
2185.2
410.3
2603.7
Bank Heist
14.2
753.1
84.3
779.5
75.4
BattleZone
2360.0
37187.5
9442.2
4370.0
6220.0
Boxing
0.1
12.1
7.3
30.3
23.5
Breakout
1.7
30.5
238.2
43.3
361.0
ChopperCmd
811.0
7387.8
727.3
826.7
651.7
Crazy Climber
10780.5
35829.4
40416.8
53437.7
49182.7
Demon Attack
152.1
1971.0
1975.3
1533.1
3906.3
Freeway
29.6
7.1
14.3
Frostbite
65.2
4334.7
235.3
311.8
245.6
Gopher
257.6
2412.5
1226.6
457.9
1172.1
Hero
1027.0
30826.4
4676.3
3767.8
11073.0
Jamesbond
29.0
302.8
82.4
295.7
191.3
Kangaroo
52.0
3035.0
224.0
320.3
344.0
Krull
1598.0
2665.5
2841.0
5900.7
8489.0
Kung Fu Master
258.5
22736.3
13363.3
18804.7
12985.7
Ms Paeman
307.3
6951.6
575.0
1443.2
644.1
Pong
-20.7
14.6
0.1
-1.4
Private Eye
24.9
69571.3
96.7
100,0
66.7
Qbert
163.9
13455.0
2813.3
1093.8
4219.7
Road Runner
11.5
7845.0
3143.3
8824.6
3317.0
Scaquest
68.4
42054.7
301.5
654.1
880.8
Up Down
533.4
11693.2
3128.9
27720.0
4808.9
Normed Mean
0.000
1.000
0,638
0.739
1.184
Normed Median
0.000
1.000
0.218
0.132
0.360
The self-play workers will regularly communicate with the train worker transfer the counting table C they maintained. The train worker will add up all the counting table, execute line 9 to line 17 in Algorithm D then distribute the adapter M each self self-play worker and reanalysis worker.
D.4 FULL RESULTS
The full results of the Atari 26 games with 50k environment steps listed follows. Here the EfficientZero trains from scratch with 50k interactions. As for SGI, we choose the SGI-M, which consumes 3M mixed of 50M DQN replay buffer and then finetune the SGI for 50k steps. SGI requires actions as input. In contrast, our model consumes M replay buffer of EfficientZero training from scratch pretrain the environment models. Then we finetune for 50k steps.
16


Reviews:
[{"review_id": "svhHord_ir-", "reviewer": "Reviewer_Krh3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper targets the low sample efficiency of current reinforcement learning approaches. It focuses on a central question: can we leverage pre-trained data without actions to achieve a good initialisation? The experiments on Atari show consistent improvements using the proposed pretraining strategy that is based on cycle consistency.", "summary_of_the_review": "The paper addresses an important problem and proposes a simple method that achieves excellent results. There are some weaknesses in the evaluation, which are however overall outweighed by the strengths, as the evaluation still demonstrates the applicability of the method sufficiently. ", "strength_and_weaknesses": "## Strengths\n\n* The paper tackles an important problem - sample efficiency of RL agent training. The approach is based on learning a representation through unsupervised pre-training that can then easily be adapted when training the agent. The topic of the paper should be of interest to a large audience.\n\n* The experimental evaluation is thorough and shows excellent results, improving over SGI that uses action labels during pre-training, which are not as easy to obtain as simple replays.\n\n* The method is simple and general and should be easily adaptable to other tasks and methods.\n\n\n## Weaknesses\n\n### W1 Pretraining Dataset\nThe main motivation of the paper is that for pre-training it is easier to collect unlabelled videos than state-action pairs. The evaluation does not fully show the full extend of this, as the data comes from training an agent on the environment. Pragmatically, at that point training a second agent is not useful, as the data collection already provides a trained agent. It could be helpful to analyze this dependency by varying the data-collection method. One could use human data, youtube videos or a random agent and measure the effect on the performance. \n\n### W2 Generalization\nFrom the paper it is not clear if the model is pre-trained for each environment individually, or if the pre-training happens across all environments and only the agent training is done separately. The second scenario is of course a much stronger showcase of the benefits of the approach. \n\n### Minor:\nEq 2: when $\\beta=1$, the stop-gradient operator should be superfluous as the loss then has the same gradient as simply $\\Vert z_e - e \\Vert^2_2$. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "jlhHDUHEClQ", "reviewer": "Reviewer_1ysS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Paper proposes a framework which can use action-free videos (instead of action-tagged videos) to train RL models by a 2-phase pipeline. In phase 1, the approach infers the hidden action embedding from the videos, pre-train the visual representation and the predicted next frames. A vector-quantization step was also introduced to prevent the model from learning the shortcut mapping (identity function). For the down-stream tasks, the model uses an approach termed Action Adapter to map the action embedding to the actual actions.\n\nEmpirical experiments outperformed the SOTA methods such as EfficientZero (without pre-training) and SGI (pre-training with actions tagged videos) for the custom Atari 50k dataset. The proposed method was pretrained with 1M transition of replay frames from EfficientZero model.\n\nFurther experiment was done to pre-train a single model for 6 different games to demonstrate that the model can quickly adapt to different rules with a single pre-trained model. In this setting, the \"unified\" model outperformed EfficientZero (without pre-training) for 5 out of 6 games.\n\nAblation studies were also done to demonstrate the importance of the various sub-components (differences reconstruction, vector quantization).", "summary_of_the_review": "1. Good paper with significant scientific and empirical contributions.\n2. However, some experimental setups appear to be incomplete.\n3. Overall, the work is of interests to other researchers in the RL and should be accepted for ICLR.\n\n", "strength_and_weaknesses": "Strengths\n1. Novelty, contributions and experimental evidence are well-linked.\n2. Paper is clear and concise.\n\nWeaknesses\n1. Novelty is quite limited in the sense that pre-training is well-established concept for other learning paradigm. However, to my knowledge, it was not applied to RL as the paper proposed.\n2. It is not clear why paper only experimented on Atari 50k when Atari 100k is the standard benchmark. Given the efficiency of the proposed method, it should not be difficult to also extend the experiments to 100k. This will also allow direct comparisons for other SOTA methods.\n3. Seo et al was cited as a comparable method. But no experimental was done to compare. Why?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "dYQBgBQCdV", "reviewer": "Reviewer_wEqK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an approach to pretrain a network to learn features for RL from videos of the task without actions. The network uses a reconstruction loss and a cycle-consistency loss to learn this. The approach is evaluated on several atari game tasks.", "summary_of_the_review": "The paper proposes a method to pretraing RL networks. The approach is straightforward, though not especially novel. The experiments show the benefit over random init, but are lacking comparisons to other works.", "strength_and_weaknesses": "Efficient RL learning is an important task, and this approach shows benefit over random initialized training. The ablations show that some of the components, e.g., reconstruction loss, are useful. However, there are no comparisons to any previous works, which makes it hard to know how well or efficient this approach is compared to others. The approach is only evaluated on relatively simple videos of atari games, how would this approach work for more complex or real videos? How much of the approach relies on the pretraining data? For example, if it was purely random trajectories vs. top human player trajectories, how would that influence the pretraining? In Table 2, it seems that EZ-L is usually worse, which seems to contradict the writing.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MthNKrP6KYD", "reviewer": "Reviewer_HyxY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a pre-training mechanism that improves sample efficiency for model-based reinforcement learning. They propose to use raw videos by extracting action embeddings to pretrain their representations and dynamics. The pretrained networks are then finetuned with limited task data.\n\nIn summary, the authors study how pre-training from raw videos can improve model-based RL, propose a method based on cycle-consistency to empower pre-training, and achieve SoTA performance on benchmarks related to Atari games. Emphasis is made on sample efficiency and large improvements over EfficientZero.", "summary_of_the_review": "The paper proposes a novel idea that shows performance improvements and efficiency gains. In general, the ideas are clearly transmitted and the concept seems simple but powerful. The paper does lack more in-depth comparisons to state of the art methods, and could be clearer in certain sections, but the general idea is sound and interesting. As it stands, the paper is above the acceptance threshold, and could be improved if certain sections are re-written and figures replaced. ", "strength_and_weaknesses": "## Strengths\n- The paper is clear and understandable\n- The propose approach is based on a simple concept, yet appears to generate large benefits when compare to previous techniques.\n- The reconstruction losses used in conjunction with FICC seem pertinent.\n- The performance increase in the experiments shown is laudable.\n\n## Weaknesses\n- Some claims are unclear or not well defined. For example:\n     a. unrolling the losses, end of Sec. 3.1 -> how was this implemented?\n     b. latent action z has too much information, first paragraph, Sec 3.2 -> can you give information on how you determined this?\n- Loss combinations: $L_{cc}$ and $L_{vq}$ appear quite different and could have different scales that can impact training. There is, however, no controlling factors in the final loss. How was this analyzed?\n- Writing: some sections could be improved in terms of clarity and grammar, e.g. 4.2, 4.3\n- Comparisons: the technique is mainly compared to EfficientZero, but more comparisons to other state of the art techniques would be pertinent.\n- Section 4.3 is unclear: are we still building an action adapter M? If not, how do you adapt to the different amount of actions in each environment?\n- Ablation studies: there is no ablation showing the individual performance delta of each reconstruction loss (difference reconstruction and full reconstruction) - we only see an ablation related to difference reconstruction\n- shortcut learning ablations: there is no analysis showing how reducing the dimensionality of $z_e$ impacts performance. It would be nice to motivate the use of vector quantization instead of just simple dimensionality reduction with quantitative experiments. \n- Figure 6 is compelling, but could be much clearer. Labeling each column and highlighting the displacement in the breakout bar would improve its clarity. \n\n\n## Questions\n- Is the Vector quantization enough to stop FICC from converging to trivial solutions? What is the dimensionality of $z_e$ and $z_q$? There could be a world where these vectors, if large enough, still encode enough information to trivially reconstruct $s_{t+1}$, particularly when finetuned on downstream tasks\n- Are the action embeddings $e_k$ trained during finetuning? It's clear that R and D are finetuned with learning rate $l_f$, but what happens with $e_k$ is unclear", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster