PlayVirtual: Augmenting Cycle-Consistent Virtual Trajectories for Reinforcement Learning
Authors: Tao Yu
Cuiling Lan
Wenjun Zeng
Mingxiao Feng
Zhizheng Zhang
Zhibo Chen
Venue: NeurIPS 2021 Poster (2021)
Keywords: Reinforcement learning
data efficiency
representation learning
augmentation
virtual trajectory
cycle consistency

Abstract: Learning good feature representations is important for deep reinforcement learning (RL). However, with limited experience, RL often suffers from data inefficiency for training. For un-experienced or less-experienced trajectories (i.e., state-action sequences), the lack of data limits the use of them for better feature learning. In this work, we propose a novel method, dubbed PlayVirtual, which augments cycle-consistent virtual trajectories to enhance the data efficiency for RL feature representation learning. Specifically, PlayVirtual predicts future states in a latent space based on the current state and action by a dynamics model and then predicts the previous states by a backward dynamics model, which forms a trajectory cycle. Based on this, we augment the actions to generate a large amount of virtual state-action trajectories. Being free of groudtruth state supervision, we enforce a trajectory to meet the cycle consistency constraint, which can significantly enhance the data efficiency. We validate the effectiveness of our designs on the Atari and DeepMind Control Suite benchmarks. Our method achieves the state-of-the-art performance on both benchmarks. Our code is available at https://github.com/microsoft/Playvirtual.

PlayVirtual Augmenting Cycle-Consistent Virtual Trajectories for Reinforcement Learning
Tao Yu Cuiling Lan
Wenjun Zeng² Mingxiao Feng' Zhizheng Zhang² Zhibo Chen
*University of Science and Technology of China Microsoof Research Asia
yutao6660mail. ustc. wsw..utttt... edu, en, {culan, wezeng) vezenn)"microsoff com
fmxustc@mail ustc. edu, cn, srrzhhang@microoooft com, chenzhibo@ustc edu.
Abstract
Learning good feature representations is important for deep reinforcement learning (RL). However, with limited experience, RL often suffers from data inefficiency for training. For un-experienced or less-experienced trajectories (i.e., state-action sequences), the lack of data limits the use of them for better feature learning. In this work, we propose novel method, dubbed Play Virtual, which augments cycle-consistent virtual trajectories to enhance the data efficiency for RL feature representation learning. Specifically, Play Virtual predicts future states in a la- tent space based on the current state and action by a dynamics model and then predicts the previous states by backward dynamics model, which forms a tra- jectory cycle. Based on this, we augment the actions to generate a large amount of virtual state-action trajectories. Being free of groudtruth state supervision, we enforce trajectory to meet the cycle consistency constraint, which can signifi- cantly enhance the data efficiency. We validate the effectiveness of our designs on Atari and Deep Control Suite benchmarks. Our method achieves the soaite--ffthheeaee performance both benchmarks. Our code is available at httpss//ghauul.ooo///mcssstttttp
Introduction
Deep reinforcement learning (RL) combines the powerful representation capacity of deep neural networks and the notable advantages of RL for solving sequential decision-making problems. It has made great progress many complex control tasks such as video games [35] 46 3, and robotic
control [2 54 36). Despite the success of deep RL, it faces the challenge of data/sample inefficiency
when learning from high-dimensional observations such as image pixels from limited experience 27 61 29). Fitting high-capability feature encoder using only scarce reward signals is data inefficient and prone to suboptimal convergence (49] Humans can learn to play Atari games in several minutes, while RL agents need millions of interactions [44]. However, collecting experience in the real world is often expensive and time-consuming. One may need several months collect interaction data for robotic arms training [22] or be troubled by collecting sufficient patient data to train a healthcare agent [52]. Therefore, from another perspective, making efficient use of limited experience for improving data efficiency becomes vital for RL.
Many methods improve data efficiency by introducing auxiliary tasks with useful self-supervision to learn compact and informative feature representations, which better serves policy learning. Previous works have demonstrated that good auxiliary supervision can significantly improve agent learning,
maximizing Predictive Information [38 1 42 31 or promoting discrimination through contrastive learning
like leveraging image reconstruction [49], the prediction of future states 41
13
30 40
This work done when Tao Yu was intern at Microsoft Research Asia. Corresponding Author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

2242
Superviston
Encoder
Poliey Learning
RL Loss
Q
DM
DM
2p+1
Zz+2
2
Ausittary Task
Auaiitaar Loss
1
Ac+1
Bc+K-1
(a)
(b)
Z
9 c+1
2112
2
0
DM
DM
DM
1
Consistency
-
ăc: -
ä(+K-1 ~A
BDM
8DM
RDM
Z
2+x-1
Figure 1: Illustration of the main pipeline of our method. (a) A glance the overall framework which consists of an encoder for learning the latent state representation 2t, a policy learning head, and our auxiliary task module. The auxiliary task module consumes real trajectory as shown in (b) and an augmented virtual trajectory as shown in (c), respectively. In (b), we train the dynamics model (DM) be predictive of the future state based on the input state and action, with the supervision from the future state. To enhance data efficiency, as shown in (c), we augment the actions to generate virtual trajectory formed by forward and backward trajectory. Particularly, the forward state-action trajectory is obtained based on the current state Z₁, the DM and a sequence of gegmmetted/gneerateee actions (e.g., c. for steps). Similarly, based on the predicted future state 2 backward dynamies model (BDM), and that sequence of augmented actions, we obtain the backward state-action trajectory. For the virtual trajectory. we add the consistency constraint on the current state z, and the predicted current state z' for oplimizing the feature representations.
Although the above methods have been proposed to improve the data efficiency of RL, the limited experience still hinders the achievement of high performance. For instance, the current state-of-the-an method SPR [40] only achieves about 40% of human level on Atari [2] when using data from 100k interactions with the environment. Some methods improve data efficiency by applying modest image augmentation (i.e., transformations of the input images like random shifts and intensity) [28 [5]. Such perturbation images improves the diversity of appearances of the input images. However, it cannot enrich the experienced trajectories (state-action sequences) in training and thus the deep networks are still deficient in experiencing/ingesting flexible/diverse trajectories.
29 59 32 25
In this work, to address the above problem, we propose a new method dubbed Play Virtual, which augments cycle-consiee ceceecconsistent virtual trajectories to improve data efficiency. Particularly, we predict future states in the latent space by dynamics model in the forward prediction (trained using real trajectories to predict the future state based on the current state and action) and then predict the previous states by a backward dynamics model, forming loop. In this way, we can augment the actions to generate a large amount of virtual/fitiitouus state-action transitions for feature representa- tion training, with the self-supervised cycle consistency constraint (which is a necossary/desiraalc condition for good feature encoder). Note that such design is free of groundtruth supervision. Our augmentation to generate abundant virtual trajectories can significantly enhance the data efficiency. As illustrated in Figure I top of a baseline RL framework, we introduce our Play Virtual for augmenting state-action virtual trajectories with cycle consistency constraint. The dynamics model infers the future states recurrently based on the current state and set of randomly sampled actions, and the backward dynamics model predicts the previous states according to the predicted future state and those sampled/augmented actions. We enforce the backwardly predicted state of the current time step to be similar to the original current state to meet the cycle consistency constraint.
We summarize the contributions of this work follows:

---

We pinpoint that augmenting the experience of RL in termss of trajectories is important for feature representation learning RL (which is sequential decision-making problem) to enhance data efficiency. To our best knowledge, we are the first to generate virtual trajectories (experience) for boosting feature learning.
We propose practical method Play Virtual for augmenting virtual trajectories under self-supervised cycle consistency constraint, which significantly improves the data efficiency.
We demonstrate the effectiveness of our Play Virtual on discrete control benchmark Atari [2] and continuous control benchmark DMControl Suite [43, where our PiatVirtual achieves the best performance on both benchmarks.
2 Related Work
2.1 Data/Sample-effciint RL and Representation Learning for RL
Learning from visual observations is a fundamental yet challenging problem in RL. In practice, collecting experience is expensive and time-consuming. Thus, in general only limited experience is available for training RL agent, which results in the difficulty in fitting a high-capability feature encoder, i.e., learning powerful feature representations. Therefore, data-efficient RL has attracted a lot of attention and many methods are designed to make efficient use of limited experience for improving data efficiency. These methods can be grouped into three categories. (i) Auxiliary task
based methods introduce auxiliary task to help representation learning of the states (21 49 41 [3
30 40 38
(ii) Data augmentation based methods increase the diversity of image appearance through data augmentation [50] 28 40). But they do not augment virtual actions. (iii) World model based methods explicitly model the environment in order to enable the planning or
29 59
promote the policy learning [6 17 611. We only focus the first two categories since the third one
is not specifically designed to enhance the efficiency of feature representation learning.
In recent years, unsupervised representation learning has made significant progress in natural language processing 6 33 and computer vision [38 18 [2]. aims learn generic and discriminative feature representations without groudtruth supervision, i.e., by introducing some unsupervised auxiliary tasks. In RL, good state representation removes the redundancy and noise elements from the original high-dimensional state, and reshapes the original state space into compact feature space. Recently, many works explore representation learning in RL and have shown promising performance improvement. UNREAL [2I] introduces a number of auxiliary tasks such as reward prediction. Yarats el al. [49 introduce an auxiliary reconstruction loss to aid feature representation learning. Considering the ability to model what will happen next is necessary for success on many RL tasks [4 31,, some works train agents to be predictive of the future states. [41, PBL. [R] SLAC 33 and SPR [40] explicitly predict the future states by modeling dynamics. Similarly, CPC (38) ST-DIM DI. DRIML [34]. ATC [42] and PI-SAC [31] maximize the mutual information between the current state and the future state by using InfoNCE [38 42), Deep InfoMax (20 II 34), or Conditional Entropy Bottleneck 9 31. Some works exploit contrastive learning to learn
discriminative representations 29 59
32 25
CURL [29] extracts high-level features from raw
pixels using contrastive learning, by encouraging the similarity between the augmentations of the
same image and the dissimilarity between different images.
Inspired by the success of data augmentation in computer vision, DrQ (50] and RAD 28 explore the effectiveness of data augmentation in RI. and show that increasing the diversity of the training images by simple image augmentation (such as random cropping) can improve the data-efficiency. In SPR [40], besides the prediction of its own latent state representations multiple steps into the future, they improve performance by adding data augmentation on the future input image as the future state supervision. However, all these methods train the encoder using the real interaction transitions, There is a lack of an efficient mechanism to generate reliable state-action pair transition trajectories for better training the feature encoder.
In this paper, we propose a method, dubbed Play which enables the augmentation of trajec- tories with unsupervised cycle consistency constraint for training the feature representation. This effectively enhances the data efficiency and our trajectory augmentation is conceptually orthogo- nal/complementaryy to the previous augmentation auxiliary task based methods.
3

---

2.2 Cycle Consistency
Many works have explored the high-level idea cycle consistency to address different challenges in various tasks such as image- -imeeagge translation [60] [1 24 image matching [56] 58 57]. feature representation learning [47] 7 26]. For image-to-image translation, CycleGAN [60] introduces the eycle consistency constraint to Generative Adversarial Networks [TO] to remove the requirement of groudtruth paired images for training, by enforcing the back-translated image to be the same as the original one. Zhang al. [55] learn the correspondences that align dynamic robot behavior across two domains using cycle consistency: given observations in time t, the future prediction in time + 1 should be consistent across two domains under the consistent action taken. The purpose is to enable a direct transfer of the policy trained in one domain to the other without any additional fine-tuning, To learn visual correspondence from unlabeled video, Wang el al. 447 propose track target backward and then forward to meet temporal cycle consistency on the feature representations, using the inconsistency between the start and end points as the loss function. Kong et al. (26) propose cycle- contrastive learning for learning video representations, which is designed to find correspondences across frames and videos considering the contrastive representation in their domains respectively. where the two domain representations form a frideevdaae cycle.
Different from the above works, in order improve data efficiency in RL, we propose to augment virtual state-action trajectories to enrich the "experience" of the feature encoder for representation learning. To ensure the reasonblenees/ccoreetness of the generated transitioon//raaeccooisss and make use of them, we take the necessary condition of good trajectory, i.e., satisfying the cycle consistency of the trajectory, constraint to optimize the network.
3 Cycle-Consistent Virtual Trajectories for Representation Learning in RL
3.1 Background
We consider reinforcement learning (RL) in the standard Markov Decision Process (MDP) setting where an agent interacts with the environment in episodes. We denote the state, the action taken by the agent and the reward received timestep in an episode as 8t, a,, and rt, respectively. We aim to train an RL agent whose expected cumulative reward in each episode is maximized.
With the observation being high-dimensional short video clip each timestep, the powerful represen- tation capability of deep neural networks for encoding state and a strong RL algorithm contribute to the success of RL agent. Similar to [29]. we use the widely adopted RL algorithm Rainbow
[19] for discrete control benchmarks (e.g., Atari (2) and Soft Actor Critic (SAC) [15] for continuous
control benchmarks (e.g., DMControl Suite (43). Following SPR [40], we introduce a dynamics
model (DM) to predict the future latent states multiple steps, which enables forward state-action
trajectory. We take SPR 40 our baseline scheme.
3.2 Overall Framework
Considering the data efficiency in RL with limited experience, we propose a method named Play Vir- tual to efficiently improve the feature representation learning of RL. Our key idea is to augment the actions to generate virtual state-action trajectories for boosting the representation learning of the encoder. Particularly, we eliminate the need of groudtruth trajectory supervision for the augmented sequences by using a cycle consistency constraint, which thus enhances data efficiency in training.
Figure M Illustrtess the main pipeline of our framework (with some details not presented for clarity). As shown in (a), it consists of an encoder which encodes the input observation s; into low-dimensional latent state representation 2,, an RL policy learning head (Rainbow [19] or SAC (15),, and our auxiliary task module. Particularly, as shown in (c), our auxiliary task module consists of a dynamics model (DM) which predicts future latent state based on the current state and the action, and a backward dynamics model (BDM) for backward state prediction. Following SPR 40), the DM is trained with the real state-action trajectory under the supervision of the future state (see (b)) to assure its capability of generating "correct" state transition. However, under limited experience, the encoder has few opportunities to be trained by those un-experienced or less-experienced state-action trajectories, which should be important to enhance data efficiency. To address this problem, as illustrated in (c), we add BDM which predicts the previous state based on the current state and

---

the previous action. Together with the DM, the forward predictions and backward predictions form a cycle/loop, where the current state and the backwardly predicted current state are expected be the same. Particularly, we augment the actions to generate virtual trajectories in order to train the network to "see" more flexible experiences with cycle consistency constraint. Our method contains three main components which we describe below.
Dynamies Model for Prediction of Future States. A one-step Markov transition (Sr, a,s St++1) contains a current state S, E S, an action a, E A, and the next state Sz-1 E S. The transition model determines the next state S given the current stateaaction pair (s,, ar).
Considering the ability to model what will happen next is important for RL tasks, many works train agents to be predictive of the future states to learn good feature representations [40]
[3 30).
Following SPR [40), we introduce a dynamics model (DM) l((:,)) :) to predict the transition dynamics
(Zt, a.) 7 in the latent feature space, where 2, f(s)) is encoded by the feature encoder f()) of the current input video clip St. As illustrated in Figure [T]. based on the current input state 24 and
sequence of actions a K-1 11 obtain sequence of K predictions Z:+1:t4K K of the future state representations using the action-conditioned transition model (i.e., DM) h(s-)) .) by computing the next state iteratively as
[h(z+ as+x) h((24kk as+k)
if k: = if k: = 1.1.
2
(1)
We train the DM with the supervision of the future state representations obtained from the recorded real trajectory (iee. from the recorded future video clip). Following SPR [40], we compute the
prediction loss by summing over difference (error) between the predicted representations Z and
observed representations ZL+k at timesteps t + for 1 S k I K measured in a "projection" metric space as:
Ppred Σ d(zt+k k; 2(tk),
where d denotes the distance metric a "projection" space 40 (see Appendix A for more details).
This module has two roles in our framework. (i) The future prediction helps to learn good feature state representation, which enables the scheme SPR [40] that
our strong baseline. (ii) It paves the way for our introduction of cycle-consistent virtual trajectories for improving data efficiency.
Backward Dynamics Model for Prediction of Previous States: Backward transition model intends to determine the previous state s, given the next state S,+1 and the causal action a,. We introduce a backward dynamics model (BDM) b(.,-) to predict the backward transition dynamics (Z++1, a.) - Zt in the latent feature space.
37. backward induction has been exploited to predict the preceding states that terminate at a given high-reward state, where these traces of (state, action) pairs are used to improve policy learning. Their purpose is to emphasize the training on high-reward states and the probable trajectories leading to them to alleviate the problem lack of high reward states for policy learning. In contrast, we introduce BDM which predicts previous states (to have a backward trajectory) in order to build a cycle/loop with the forward trajectory to enforce the consistency constraint for boosting feature representation learning.
In previous works
As illustrated in Figure [(c), based on the hidden state ži+k and a sequence of actions á 11:1 we obtain a sequence of k predictions 2 of the previous state representations using the BDM b(.s-) by computing the previous state iteratively as
[b(2)+ āc+k-1)
if K if = - 1, 2.... 1.
2
b(2/+2) 11
Action Augmentation and Cycle Consistency Constraint. Given the DM, BDM, current state, and sequence of actions, we can easily generate forward trajectory and a corresponding backward trajectory which forms a loop/cycle/forward baakkardd trajectory. As we know, for an encoder which is capable of encoding observations to suitable feature representations, the feature representations trajectory should in general be
of the start state 2, and the end state z/ of a similat/consistent, given reasonable DM and BDM.
5

---

Therefore, as illustrated in Figure [(c) we enforce a consistency constraint between the start state z; and the end state z to regularize the feature representation learning. this way. by augmenting actions (generating/sampligg virtual actions), we can obtain abundant virtual cycle-consistent tra- trajectories.
jectories for training. Note that in the training, we do not need supervision of the states from real
Here, we mathematically define the cycle-consistent feature representation in forward-backward trajectory as below.
Definition 1. Given a (forward) dynamics model and a backward dynamics model b, cycle- consistent feature representation 2, in horward-backwarr trajectory I is representation of the current state that meets the following condition when experiencing any sequence of K actions {a,, a(+1....,, a++K-1] 2 sampled from action space A:
E.-ldd((z z:) =0,
where dm distance metric on space M and z is the prediction of 2₁ after experiencing sequence of actions in forward prediction and backward prediction as
forward: 2, 21, 2c+h+1 h(2r+k; a1t8); for 00,, KK-1
backward: z'4K 2LtK, Z4+ 0(144411,8-1 for k = K - 1, K 2.......
Given the state z, encoded from the current input St of time t, we randomly sample M sets actions in the action space A. We calculate the cycle consistency loss
M Σ m=1
Leye
dmizz,, 21).
M
We describe the alternative distance metrics du on space M and study the influence on performance in Section
we model the backward dynamics using a BDM. This is basically feasible for many real-world applications, e.g., robotic control, and many games. Consider a robotic arm: given a current state (e.g., spatial position and rotation) and the previous action (e-g., quaternions
Discussion: In our scheme, similar to 11 g
can deduce the previous state without much effort. This holds in most games such as chess or Atari. For some cases where there are many-to-one transition (i.e., different states with the same action may be transited into the same state), BDM may learn the most probable state or a mean state to minimize the prediction errors through the powerful fitting capacity of neural networks. More discussion can be found in Appendix C.
six-degree-of-freee ofreeeeffffffommmm- parameters),
Overall Training Objective. The overall training objective of our method is as below:
Listal Lrt + Xprea&pred + Aeye
(5)
where Lrt, Cpred, and Laye denote the RL loss (please refer to Rainbow 19 for discrete control games, SAC [15 for continuous control games), prediction loss (see Eq. (2). and cycle consistency
loss (see Eq. 44), respectively. Apred and Aaye are the hyperparameters for balancing the contributions
of different losses.
Experiments
We introduce the experimental setup including environments, evaluation, and implementation details. We conduct extensive ablation studies to demonstrate and analyze the effectiveness our designs.
4.1 Setup
Environments and Evaluation. We evaluate our method on the commonly used discrete control benchmark Atari (2), and the continuous control benchmark of DMControl (3 Following 29 50
we measure the performance of different methods at 100k interaction steps (400k environment steps with action repeat of 4) on Atari (also refer to as Atari-100k), and at 100k and 500k environment steps
on DMControl (refer to as DMControl-100k. DMC-100k, DMControl-5000 or DMC-500k).

---

respectively. In general, using Atari- 100k on 26 selected games 61 45 23 29
and DMControl-
100k
29 50 has been a common practice for investigating data efficiency.
For Atari-100k, we measure the performance by score, and the median human-normalized score (i.e. median HNS) of the 26 games. Human-nowe noraannzed score on a game is calculated by 8g Sg-Sp
SĄ=Sa, where S
the agent score, SR the score of random play, and SH the expert human score. For DMControl,
the maximum possible score for every environment is 1000 [43]. Following (29) 49 [6] 50 [2],
we evaluate models on the six commonly used DMControl environments. Additionally, we use the median score on them to reflect the overall performance.
our strong baseline (dubbed Baseline) and build our method on top of SPR by augmenting cycle-consistent virtual trajectories for better representation learning (see Figure [[e).. For the backward dynamics model, we use the same architecture as that of the dynamics model. We set the number of prediction steps K to 9 by default. We simply set the number of action sets, i.e., the number virtual trajectories M to 2) 2I, which is proportional to the size of action space |AA in that Atari game. To generate an action sequence, we randomly sample an action from the discrete action space at each step. We report the results of SPR [40] by re-running their source code in all Tables except for Table (results in which are copied from their paper. being higher than our reproduced results).
Implementation Details. For the discrete control benchmark of Atari, we use SPR
For the continuous control benchmark of DMControl, considering the SPR is originally designed only for discrete control, we build a SPR-like scheme SPR as our baseline (dubbed Baseline) for continuous control games. Particularly, we use the encoder and policy networks of CURL (29] as the basic networks. Following SPR [40), we remove the contrastive loss CURL and introduce BYOL [2] heads to build SPR-like baseline scheme. We use the network architecture similar to
dynamics model DBC [53 to build the dynamics model SPR and the backward dynamics
model in our Play Virtual. We follow the training settings in CURL except the batch size (reduced from 512 to 128 to save memory cost) and learning rate. We set to 6, and set M to a fixed number 10 with actions randomly sampled from the uniform distribution of the continuous action space.
We set Xpred and Aege 1. For dy, we use the distance metric in SPR 40 More imple- mentation details can be found in Appendix A. All our models are implemented via PyTorch [39].
4.2 Performance Comparison with State-of-the-Arts
Table 1: Scores achieved by different methods Atari-100k. We also report median HNS. We run our Play Virtual with 15 random seeds given that this benchmark is susceptible to high variance across multiple runs. Note that here we report the results of SPR [40] copied from their paper (i.e., 41.5%), which is much higher than our reproduced results using their released source code (i.e., 37.1% 97.1%)
Game
Human
Random
SimPL.e661]
DER
OTR 23
CURL (2
DrQ [50]
SPR (40)
Play Virtual
Alien
?127.?
227.8
616.9
739.9
824.7
558.2
771.2
801.5
947.8
Amidar
1719.5
5.8
88.0
188.6
82.8
142.1
102.8
176.3
Assault
742.0
222.4
$27.2
431.2
351.9
600.6
452.4
$71.0
702.3
Asterix
8503.3
210.0
1128.3
470.8
628.5
734.5
603.5
977.8
933.3
Bank Heist
753.1
14.2
34.2
$1.0
182,1
131.6
168.9
380.9
245.9
Battle Zone
37187.5
23603.0
5184.4
10124.6
40660.
14870.0
12954.0
16651.0
13260.0
Boxing
12.1
0.1
9.1
0.2
2.5
6.0
35.8
38.3
Breakout
30.5
1.7
16.4
1.9
9.8
4.9
16.1
17.1
20.6
Chopper Command
7387.8
811.0
1246,9
861.8
1033.3
1058.5
780.3
974.8
922.4 4
Crazy Climber
35829.4
10780.5
62583,6
16185,3
21327.8
12146.5
20516.5
42923.6
23176.7
Demon Attack
197L.0
152.1
208.1
508.0
711.8
817.6
1113.4
$45.2
1131.7
Freeway
29.6
0.0
20.3
27.9
25.0
26.7
9.8
24.4
16.1
Frostbite
4334.7
65.2
254.7
866.8
231.6
1181.3
331.1
1821.5
1984.7
Gopher
2412.5
257.6
771.0
349.5
778.0
669.3
636.3
715.2
684.3
Jamesbond
302.8
29.0
125.3
301.6
112.3
471.0
236.0
365.4
394.7
Kangaron
779.3
2384.7
Krull
2665.5
1598.0
4539.9
2851.5
3277.9
4229.6
4018.1 1
3688.9
3880,7
Kung Masser
17257.2
14259.0
Ms Pacman
6951.6
307.3
1450.0
1204.1
941.9
1465.5
960.5
1313.2
1335.4
Pong
-3.0
Private Eye
69571.3
24.9
$8.3
97.8
100.0
218.4
-13.6
124.0
93.9
Qbert
13455.0
Road Runner
7845.0
11.5
5640.6
9600.0
2696.7
$661.0
8895. 1
14220.5
13534,0
Seaquest
42054.7
68.4
683.3
354.1
286.9
384.5
301.2
583.1
$27.7
Up Down
11693,2
533.4
3350.3
2877.4
2847.6
2955.2
3180.8
28138.5
10225.2
Median HNS (%)
100
14.4
16.1
20.4
26.8
47.2

---

Table 2: Scores (mean and standard deviation) achieved by different methods the DMControl- -00k and DMControl-500k We run our Play Virtual with random seeds. Note that SPR 440 is originally designed only for discrete control. For the continuous-contro) environments, extend SPR to new version named SPR
100k Step Scores
PlaNet [6
Dreamer 17
SAC+AE 49
SLAC 30
CURL,29
DrQ
SPR 40
Play Virtual
Finger, spin
136 216
56
915
Curipole, swingup
297 39
326 2:27
311
+: 11
582 1: 146
759 92
799 = 42
816 1: 36
Reacher, casy Cheetah, run
314
155
274
538
+ 233
213
638
785 142
138 88
235 + 137
267 +
319 + 56
299
344 =
467 = 36
474 + 50
Walker, walk
224 48
24
612 164
173
Ball in cup, catch
0 +
246 + 174
391 + 82
512 + 110
769 43
913 53
861 233
926 + 31
Median Score
137.0
295.5
351.0
436.5
560.0
685.5
719.0
800.5
500k Step Scores
Finger, spin
561 + 284
79% + 183
884 + 128
673 + 92
926 + 45
938 + 103
924 132
963 +:40
Caripole. swingup
475 71
762 +
735
841 :: 45
868 10
870
865 =
Reacher, casy
210 + 390
795 + 164
627 + 58
929 44
942
925
942 - 66
Cheetah,
+ 131
570 + 253
96
716
Walker, walk
351 + 58
897 + 49
847 + 48
842 51
902 43
921 45
916 =
928 + 30
cup, catch
380
794
852
963
Median Score
405.5
794.5
764.5
757.5
914.0
929.5
920.0
935.0
Comparison Atari. On Atari-100K, Table [ shows the comparisons with the state-of-the-art methods. We also report the results of random play (Random) and expert human play (Human) (copied from [48).. Play Virtual achieves a median HNS of 47.2%, significantly outperforming all previous methods. Play Virtual surpasses the baseline SPR [40] (with median HNS of 41.5% reported in their paper) by 5.7%. We have re-run the released source code of SPR with 15 random seeds and obtain a median HNS 37.1%, which suggests that our improvement over SPR actually 10.1 %.
Comparison on DMControl. For each environment in DMControl, we run our PlayVirtual with 10 random seeds to report the results. Table 2 shows the comparisons with the fatte-of-the-eee methods. Our method performs the best for the majority (5 out of 6) of the environments on both DMControl- -00k and DMControl-S0OL. (i) On DMControl- 1 100k which is low data regime, our method achieves the highest median score of 800.5, which is about 11.3 1.3% % higher than SPR 16.7% higher than DrQ (50) and 42.9% higher than CURL [29]. (ii) On DMControl-500
our method achieves median score of 935.0, which approaches the perfect score of 1000 and outperforms all other methods. Therefore, our method achieves superior performance in both data-efficiency and asymptotic performance.
4.3 Ablation Studies
We use the median HNS and median score measure the overall performance on Atari and DMCon- trol, respectively. We run each game in Atari with 15 random seeds. To save computational resource, we run each environment in DMControl with 5 random seeds (instead of 10 as in Table 2
Effectiveness of Play Virtual. As described in baseline
Table 3: Effectiveness of PlayVirtual
top of Baseline, which is SPR (40) for discrete control on Atari, and SPR for continual control on DMCon- trol. "w/o Pred" denotes disabling future prediction Baseline. Baseline+ BDM denotes the scheme that BDM is incorporated into Baseline.
Section 4.1] we take SPR [40] (i.e., Baseline) on discrete control benchmark Atari, and SPR continuous control bench- mark DMControl. On top of Baseline, we val- idate the effectiveness of our Play Virtual which augments cycle-consistent virtual trajectories for improving data efficiency. Table 3 shows the comparisons. We can see that Play Vir- tual achieves median HNS of 47. 47.2%, which outperforms Baseline by 10.1 % on Atoki- On DMControl- 100, Play Virtual improves Base- line from 728.0 to 797.0 in terms of median score (i.e., a relative gain of 9.5%). As a com- parison, Baseline outperforms Baseline w/o Pred by 3.7% on Atari-100k, where "Pred" denotes the prediction of future state in SPR / SPRT(i.. the contribution of SPR (40). The large gains of our Play Virtual over Baseline demonstrate the effectiveness of our Play Virtual in boosting feature
Model
Atari-100k
DMControl-100k
Baseline w/o Pred
33.4
680.0
Baseline
37.1
728.0
Baseline+BDM
38.4
741.0
Play Virtual
47.2
797.0

---

representation learning. In addition, to further benchmark Play Virtual's data efficiency, we compare the testing performance in every 5k environment steps at the first 100k on DMControl, where the result curves in Appendix B show that our Play Virtual consistently outperforms Baseline.
One may wonder whether the major performance gain of our Play Virtual is attributed to the intro- duction of backward dynamics model (BDM) or by our augmentation of virtual trajectories. When we disable the augmentation of virtual trajectories. our scheme degrades to Baseline+ BDMM, where a BDM is incorporated into the baseline SPR (or SPR !) and only the real trajectories go through the BDM. In Table 3 we can see that introducing BDM does not improve the performance obviously and our augmentation of cycle-consistent virtual trajectories for regularizing feature representation learning is the key for the success.
Influence of Prediction Steps K. We study the influence of K for both our Play Virtual and the baseline scheme SPR/SPRT. Table shows the performance. When K = O, both schemes degrade to Baseline w/o Pred (where future pre- diction is disabled in SPR/SPR\) ). We have the following observatiios//concluiioss (i) Given the same number of prediction steps K (beside
Table 4: Influence of prediction steps K for our Play Virtual and the baseline scheme SPR/SPR
Benchmark
Model
K=0
K=3
K=6
K=9
K=12
Atari-1008 100k
Varyual
33.4
33.9
34.8
35.2
39.2
47.2
43.1
DMC-100K
SPR
664.0
725.0
723.0
728.0
721.5
Play Virtual
664.0
775.5
797.0
795.0
794.5
O), our Play Virtual consistently outperforms the baseline scheme SPR/SPR on both benchmarks Atari- 1ww.////Oooooooo---/ 100k. (ii) Our Play Virtual achieves the best performance at K 9 on Atari and K = 6 on DMControl, which outperforms the baseline at the same K by 10.1 1 and 9.5% (relative gain) on Atari and DMControl, respectively. Note that the performance of SPR [40] obtained using their source code at K=5 (note K=5 is used in SPR paper) is 36. 1 % (which is 41.5% in their paper) on Atari-10Ok. (iii In SPR [S0]/ a too small number of prediction steps cannot make the feature representation sufficiently predictive of the future while a too large number of prediction steps may make the RL loss contributes less to the feature representation learning (where a more elaborately designed weight Apred is needed). Our Play Virtual follows similar trends.
What does Augmenting Sycle-Consistent Vir- fual Trajectories Help? We propose the aug- mentation of cycle-consistent virtual trajectories in order to boost the feature representation learn- ing of RL for improving data efficiency. In the training, the cycle consistency loss Leye over the virtual trajectories would optimize the param- eters of the encoder, DM and BDM. One may wonder what the gain is mainly coming from. Is it because the DM is more powerful/accurae that enables better prediction of future states? Or is it because the encoder becomes more powerful to provide better feature representation? We validate this by letting the cycle consistency loss Laye not update DM, where DM is only optimized by prediction loss Inpred as in SPR. We denote this scheme as PlayVirtul--ND. Shhows that obtain a gain of 6.9% in Play VtayVirtal--N from the regularization of Leye on the encoder and a gain of 10.1% % in PlayVirtual from the regularization on both the encoder and DM on Atari. Similar trend is observed on DMControl. This implies that the augmentation of cycle-consistent virtual trajectories is helpful DM training but the main gain is brought by its regularization on the feature representation learning of the encoder.
Table 5: Impact of the augmentation of cycle- consistent virtual trajectories on feature represen- tation learning. Play Virtual-ND denotes that we do not use the cycle consistency loss over virtual trajectories to update the dynamic model.
Model
Atari-100k
DMControl-100k
Baseline
37.1
723.0
Play VirtVlrtuN Play Virtual
44.0
777.5
47.2
797.0
Influence of Distance Metric dı on Space M. For the distance metric d.m in space M, compare cosine distance on the latent feature space Miatent, i.e., dy( 2,) = 2 2 [|c [|:]]] and on the "projection" space Mprojj as in SPR 40 (see Appendix A for more details). We compare the influence of feature space for calculating cycle consistency loss and show the results in Table 6 On the Atari benchmark, our Play Virtual with distance metric on space Miatent and with distance metric on space Maroj significantly outperforms Baseline by 7.7% and 10.1% respectively. This demonstrates the effectiveness of our key idea of exploiting virtual trajectories for effective representation learning. Maroj performs 2.4% better than Miatent- That maybe because for Play Virtual and Baseline for Atari, latent feature z, (which corresponds to 64 x 7 7 feature map) preserves more spatial information than projected feature, where the former is less robust to
9

---

be matched across two augmented observations due to spatial misalignment. On the DMControl benchmark, our Play Virtual with distance metric on space Mintent and with distance metric on space Mproj significantly outperforms Baseline by 70.5 and 69.0 in terms of median score, respectively. The performance of Miatent and Mproj are similar. Note that the latent feature 24 of Play Virtual or Baseline (built based on CURL) corresponds to feature vector which is obtained after fully connected layer the backbone network, which does not face the spatial misalignment problem caused by the augmentation. We use Mproj the default metric space in this work.
Table 6: Influence of distance metric space M. Mintent and Mproj denote the of the latent feature space and the "projection" space, respectively.
Model
Atari-100k
DMControl-100k
Baseline
37.1
728.0
PlayVirtual(
44.8
798.5
Play Virtual Mproi))
47.2
797.0
Influence of the Number of Virtual Trajectories M. Table 7 shows the influence of the number of virtual trajectories M. We can observe that small M (less generated virtual trajectories) is inferior to suitable M. That may be because too small M cannot cover diverse experiences for feature representation learning. When M is too large, it brings less additional benefit. That may be because a suitable number of trajectories is enough for regularizing the network training. We observe that the performance drops when M is too large. That may be because a very large M would increase the optimization difficulty in practice.
Table 7: Influence of the number of virtual trajectories M.
Atari-100k
M
|A] 39.5
2[.4] 47.2
3|,4|
Median HNS(%)
37.1
42.5
DMControl-10W
M
0
1
10
20 806.0
30
Median Score
723.0
770.5
797.0
792
Conclusion
With limited experience, deep RL suffers from data inefficiency. In this work, propose method, dubbed Play Virtual, which augments cycle-consistent virtual state-action trajectories to enhance the data efficiency for RL. feature representation learning. Play Virtual predicts future states based on the current state and a sequence of sampled actions and then predicts the previous states, which forms trajectory cycle/loop. We enforce the trajectory to meet the cycle consistency constraint to regularize the feature representation learning. Experimental results on both the discrete control benchmark Atari and continuous control benchmark DMControl demonstrate the effectiveness of our method, where we achieve the state-of-thee performance on both benchmarks.
Acknowledgments and Disclosure of Funding
This work was supported in part by the National Key Research and Development Program of China 201 8AAA01010000 and NSFC under Grant U1908209, 61632001 and 62021001.
10

---

References
[1] Anand, A., Racah, E., Ozair, S., Bengio, Y., Côté, M.-A., and Hjelm, R. D. Unsupervised state representation learning in atari. In Advances in Neural Information Processing Systems, 2019.
[2] Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 2013.
[3] Berner, C., Brockman, G., Chan, B., Cheung, V., Debiak, P., Dennison, C., Farhi, D., Fischer, Q., Hashme, S., Hesse, C., et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint 1912.0660,, 2019.
[4] Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., and Joulin, A. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 2020.
[5] Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual representations. In International Conference Machine Learning, pp- 1597-1607. PMIR, 2020.
[6] Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805, 2018
[7] Dwibedi, D., Aytar, Y., Tompson, J., Sermanet, P., and Zisserman. Temporal cycle- consistency learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1801-1810, 2019.
[8] Edwards, A. D., Downs, L., and Davidson, J. C. arXiv preprint 1803. 10227, 2018
reinforcement learning.
[9] Fischer, I. The conditional entropy bottleneck. Entropy, 22(9):999, 2020.
[10] Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A. C., and Bengio, Y. Generative adversarial nets. In Advances in Neural Information Processing Systems, 2014.
[11] Goyal, A., Brakel, P., Fedus, W., Singhal, S., Lillicrap. Levine, S., Larochelle, H., and Bengio, Y. Recall traces: Backtracking models for efficient reinforcement learning. In International Conference on Learning Representations, 2019.
[12] Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., Piot, B., kavukeuoglu, k., Munos, R., and Valko, M. Bootstrap your own latent new approach to self-supervised learning. In Advances in Neural Information Processing Systems, 2020.
[13] Guo, Z. D., Pires, B. A., Piot, B., Grill, J.-B., Altché, F., Munos, R., and Azar, M. G. Boot- strap latent- predictied representations for multitask reinforcement learning. In International Conference Machine Learning, pp. 3875-3888 PMLR, 2020.
[14] Ha, D. and Schmidhuber, J. World models. arXiv preprint 180. 2018.
[15] Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al. Soft actor-critic algorithms and applications. arXiv preprint
1rXiv:1812.05900. 2018.
[16] Hafner, D., Lillicrap. T., Fischer, I., Villegas, R., Ha, D., Lee, H., and Davidson, J. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning. pp. 2555-250 2565. PMLR, 2019.
[17] Hafner, D., Lillicrap, T., Ba, J., and Norouzi, M. Dream to control: imagination. In International Conference on Learning Representations, 2020.
Learning behaviors by latent
11

---

[18] He, K., Fan, H., Wu, Y., Xie, S., and Girshick. R. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 9729-979999 -9338, 2020.
[19] Hessel, M., Modayil, J., Van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., and Silver, D. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[20] Hjelm, R. D., Fedorov, A., Lavoic-Marchildonn S., Grewal, K., Bachman, P., Trischler, A., and Bengio, Y. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations, 2018.
[21] Jaderberg, M., Mnih, V., Czarnecki, W. M., Schaul, TT,, Leibo, J. Silver, D., and Kavukcuoglu, K. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv:1611.00 1...99977 2016.
[22] Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., Quillen, D., Holly, E., Kalakrishnan, M., Vanhoucke, V., et al. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv preprint arXiv: /806, 10293, 2018.
[23] Kielak, K. Do recent advancements in model- deep reinforcement learning really improve data efficiency? arXiv preprint arXiv: 200::2003.11110 10181, 2020.
[24] Kim, T., Cha, M., Kim, H., Lee. J. K., and Kim, J. Learning to discover cross-domain relations with generative adversarial networks. In International Conference on Machine Learning, pp. 1857-1865. PMLR, 2017.
[25] Kipf, T., der Pol, E., and Welling, M. Contrastive learning of structured world models. In International Conference on Learning Representations, 2020.
[26] Kong, Q., Wei, W., Deng, Z., Yoshinaga, T., and Murakami, T. Cycle-contrast for self-supervised video representation learning. In Advances in Neural Information Processing Systems, 2020.
[27] Lake, B. M., Ullman, T. D., Tenenbaum, J. B., and Gershman, S. J. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
[28] Laskin, M., Lee, K., Stooke, A., Pinto, L., Abbeel, P., and Srinivas, Reinforcement learning with augmented data. In Advances in Neural Information Processing Systems, 2020.
[29] Laskin, M., Srinivas, A., and Abbeel, P. Curl: Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning, pp. 5639-5650. PMLR, 2020.
[30] Lee, A. X., Nagabandi, A., Abbeel, P., and Levine, S. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. In Advances in Neural Information Processing Systems, 2020.
[31] Lee, K.-H., Fischer, I., Liu, A., Guo, Y., Lee, H., Canny, J., and Guadarrama, S. Predictive information accelerates learning in RI. arXiv preprint arXiv:2007. 12401, 2020.
[32] Liu, G., Zhang, C., Zhao, L., Qin, T., Zhu, J., Jian, L., Yu, N., and Liu, T.-Y. Return-base based contrastive representation learning for reinforcement learning. In International Conference on Learning Representattinss 2021.
[33] Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. Roberta: robustly optimized bert pretraining approach. arXiv preprint 1XXv:1907.11162. 2019.
[34] Mazoure, B., Tachet des Combes, R., DOAN, T. L., Bachman, P., and Hjelm, R. D. Deep reinforcement and infomax learning. Advances in Neural Information Processing Systems, 2020.
[35] Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533,, 2015.
12

---

[36] Nagabandi, A., Clavera, L. Liu, S., Fearing, R. S., Abbeel, P., Levine, S., and Finn, C. Learning adapt in dynamic, real-world environments through rett-reiiforcemmett learning. arXiv preprint arXiv: 1803. / 1347, 2018.
[37] Nair, S., Babaeizadeh, M., Finn, C., Levine, S., and Kumar, V. Trass: Time reversal as self-supervissin. In IEEE International Conference on Robotics and Automation (ICRA), pP. 115-121. IEEE, 2020.
[38] Oord, A. d., Li, Y., and Vinyals, O. Representation learning with contrastive predictive coding. arXiv preprini arXiv:1807.033: 1807.03748, 2018.
[39] Paszke, A., Gross, S., Massa, F. Lerer, A., Bradbury. J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: imperative style, high-performance deep learning 2019.
library. arXiv preprint arXiv:1912.0//000
[40] Schwarzer, M., Anand, A., Goel, R., Hjelm, R. D., Courville, A., and Bachman, P. Data-efficient reinforcement learning with self-predictive representations. International Conference on Learning Representations, 2021.
[41] Shelhamer, E.,, Mahmoudieh, P., Argus, M., and Darrell, T. Loss is its own reward: Self- supervision for reinforcement learning. ArXiv, abs/1612.07307, 2017.
[42] Stooke, A., Lee, K., Abbeel, P., and Laskin, M. Decoupling representation learning from reinforcement learning. arXiv preprint arXiv:2009.08319 2020.
[43] Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, d. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. Deepmind control suite. arXiv preprint arXiv: 1801.006100 2018.
[44] Tsividis, P., Pouncy, T., Xu, J. L., Tenenbaum, J., and Gershman, S. Human learning in atari. In AAAI Spring Symposia, 2017.
[45] van Hasselt, H. P., Hessel, M., and Aslanides, J. When to use parametric models in reinforcement learning? In Advances in Neural Information Processing Systems, 2019.
[46] Vinyals. O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, H.,
Powell, R., Ewalds, T., Georgiev. P., et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):3003354 2019.
[47] Wang, X., Jabri, A., and Efros, A. A. Learning correspondence from the cycle-consistency of time. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 2566-2576, 2019.
[48] Wang, Z., Schaul, T., Hessel, M., Hasselt, H., Lanctot, M., and Freitas, N. Dueling network architectures for deep reinforcement learning. In International Conference Machine Learning, pp. 1995-2003. PMLR, 2016.
[49] Yarats, D., Zhang, A., Kostrikov, I., Amos, B., Pineau, J., and Fergus, R. Improving sample efficiency in model-free reinforcement learning from images. arXiv preprint arXiv:1910.01741, 2019.
[50] Yarats, D., Kostrikov, I., and Fergus, R. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. International Conference on Learning Representations, 2021.
[51] Yi, Z., Zhang, H., Tan, P., and Gong, M. Dualgan: Unsupervised dual learning for image-to- image translation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2849-28577 2017.
[52] Yu. C., Liu, and Nemati, Reinforcement learning healthcare: A survey, arXiv preprint arxiv:1908.0877 2019.
[53] Zhang, A., McAllister, R. T., Calandra, R., Gal, Y., and Levine, S. Learning invariant repre- sentations for reinforcement learning without reconstruction. In International Conference on Learning Representations, 2021.
13

---

[54] Zhang, M., Vikram, S., Smith, L., Abbeel, P., Johnson, M., and Levine, S. Solar: Deep structured representations for model-based reinforcement learning. In International Conference on Machine Learning, pp. 7444-7453. PMLR, 2019.
[55] Zhang, Q Xiao, T., Efros, A. A., Pinto, L., and Wang, X. Learning cross-domain corre- spondence for control with dynamics cycle-consisteney. arXiv preprint arXiv:2012.09811 2020.
[56] Zhou, T., Jae Lee, Y., Yu, S. X., and Efros, A. A. Flowweb: Joint image set alignment by weaving consistent, pixel-wise correspondences. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 1191-1200, 2015.
[57] Zhou, Krahenbuhl, P., Aubry, M., Huang, Q. and Efros, A. A. Learning dense correspon- dence via 3d-guides cycle consistency. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 117-126, 2016.
[58] Zhou, X., Zhu, M., and Daniilidis, K. Multi-image matching via fast alternating minimization. In Proceedings of the IEEE International Conference Computer Vision, pp. 4032-4040, 2015.
[59] Zhu, J., Xia, Y., Wu, L., Deng, J., Zhou, W., Qin, T., and Li, H. Masked contrastive representtio learning for reinforcement learning. arXiv preprint arXiv:2010.074700 2020.
[60] Zhu, J.-Y., Park, T., Isola, P., and Efros, A. A. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision. pp. 2223-2232, 2017.
[61] Lukasz Kaiser, Babaeizadeh, M., Mitos, P., Osiński, B., Campbell, R. H., Czechowski, K., Erhan, D., Finn, C., Kozakowski, P., Levine, S., Mohiuddin, A., Sepassi, R., Tucker, G., and Michalewski, H. Model based reinforcement learning for atari, In International Conference on Learning Representaionss, 2020.
14


Reviews:
[{"review_id": "gLEd4Dt-0QC", "reviewer": "Reviewer_j7qw", "summary": "The paper proposes to use cycle consistency for self-supervision and shows that this approach effectively increases RL data efficiency on discrete and continuous control benchmarks. The ablation study provides the useful intuition that cycle consistency helps mainly with regularizing the visual encoder and less so with training the forward dynamic model. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrength\n- The paper is clearly written. The experimental results are relatively strong. \n- Cycle consistency is intuitively a simple auxiliary tasks that could strengthen learning signals and it is nice that the paper shows empirical supports for this argument. \n- Appendix B.2 is very helpful for understanding how the runtime overhead. \n\nWeakness\n\n- The author pinpoints the problem that the encoder is less likely to be trained on transition pairs that are under-explored by the current policy. However why training an extra backward model, with the same amount of real-world data, would address the problem? In order to make this claim, one needs to see to what extent do these virtual trajectories lie in the latent space of meaningful states. Is it possible that the forward and backward models are exploiting meaningless latent space or the current policy, such that the virtual trajectories become degenerated solutions to the auxiliary task? \n\nMinor\n- Line 345, RDM (I assume it's reversed dynamics model) is inconsistent with BDM used in the rest of the paper. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors addressed the limitations and potential societal impact. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "WBNqHATsW4l", "reviewer": "Reviewer_kMqF", "summary": "The paper aims to improve data efficiency and state representation of RL by introducing cycle consistency objective and augmenting training with virtual trajectories generated by jointly learned dynamics models.\n\nIn addition to the (forward) dynamics model to predict future states like SPR, this work also learns a backward dynamics model to predict the previous states given current state and previous action. A cycle consistency loss is introduced to enforce consistencies between forward and backward state transitions on a given sequence of actions. All models are trained jointly to optimize the model-free RL loss (Rainbow and SAC), prediction loss over the observed next / previous states, and cycle consistency loss between forward and backward trajectories. The learned dynamics models are auxility tasks that improve state representation of the encoder, but are not directly used in policy learning (like model-based RL) otherwise.\n\nThe method outperforms SPR and other baselines on Atari and DeepMind Control Suite benchmark.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed method looks good and well motivated overall. Introduction of backward dynamics model, cycle consistency loss, and virtual trajectory augmentation are interesting contributions over existing work like SPR, that is shown to be effective in improving state representation and data inefficiencies.\n\nThe method is focused on using learned dynamics models and data augmentation to learn better state representation, but not on improving the policy learning (RL loss). The cycle consistency loss provides learning signals that enhances the encoder representation without ground truth supervision, which is a nice way to improve data efficiency. One question I have is\nwhether the cycle consistency loss is trained on both virtual trajectories and real trajectories?\nIIUC, real trajectories will only update the encoder but not DM and BDM, while virtual trajectories can update all of those. It would be nice to have a summary of what components are being updated by each loss for different examples. Related to that, can you clarify what Baseline + BDM in the ablation study consists of?\n\nThe authors evaluated their method on both discrete and continuous control tasks, with ablation studies for backward dynamics model, prediction steps, and the impact of cycle consistency loss on encoder and dynamics model. Empirical results on both Atari and DeepMind Control Suite benchmark are impressive, outperforming the baselines and existing SOTA. The evaluation is mainly on the quality of the final policy, but not on the state representation or dynamics models, which are the main contributions of this work. Some form of comparison of the dynamics model with SPR will be interesting, but not required.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is no clear limitations or negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "LI3YAfZtb0x", "reviewer": "Reviewer_wH3w", "summary": "In order to increase sample efficiency of image-based Reinforcement Learning (RL), the authors propose PlayVirtual. They leverage the auxiliary task of learning a backwards and forwards dynamics model as well as as cycle-consistency loss between the two models. They use the model to generate virtual trajectories which they use to train the RL algorithm. The authors evaluate the method on DMC and Atari tasks as well as providing interesting ablations about the usefulness of each aspect of the method. \n\n\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: This paper, to my knowledge, presents a novel method in image-based RL. Cycle consistency is known to be useful in computer vision, and leveraging it and recent work in self-supervised representation learning is very interesting. The method tackles a problem of significance and the authors have explained different aspects of PlayVirtual well. The ablations and experiments are quite informative. The figures and the paper are clear and it is overall well written. \n\n\n\nWeaknesses: \n\nWhile the experiments are detailed and comparisons are thorough, I have a few concerns about the  exact reported results. It would be good to see exactly how much the virtual data helped the method. For example other methods both learn an RL policy and a model, and use synthetic data as well, such as [1], Dreamer, SLAC etc. An ablation over the number of synthetic trajectories would be useful. Additionally, do the authors have intuitions as to why the Atari experiments don't seem to show too much of a benefit over the other methods? \n\nIn its current form, it is unclear how the method exactly differs from SPR, and the authors should clarify that. Why does SPR underperform PlayVirtual? Additionally, it would be good if the authors provided more intuition as to why a cycle consistency loss is good. \n\n\n\n[1] When to Trust Your Model: Model-Based Policy Optimization. Janner et al., 2019. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have disucssed limitations but not broader soceital impact. Image-based RL, especially in robotics settings has concerns such as safety etc which can definitely be discussed. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a new state representation learning algorithm for RL by introducing cycle consistency objective and augmenting training with virtual trajectories generated by jointly learned dynamics models. The idea seems novel and sound. Most reviewers find the paper clearly written with a novel idea, and the experimental results are strong enough on Atari and Deepmind suite to support the hypothesis of this work. 

Learning state representation and world models for RL has been a popular area, this work manages to also leverage ideas from SPR, and also add a  cycle consistency loss to enforce consistencies between forward and backward state transitions on a given sequence of actions, which is a novel idea.

One question that I have is the possibility to use the learned model not only for regularizing representation learning but also for model-based control, since right now the RL part is still based on model-free methods. But on the overall, this is still an interesting paper for the NeurIPS community.
Decision: Accept (Poster)