PlayVirtual: Augmenting Cycle-Consistent Virtual Trajectories for Reinforcement Learning
Authors: Tao Yu
Cuiling Lan
Wenjun Zeng
Mingxiao Feng
Zhizheng Zhang
Zhibo Chen
Venue: NeurIPS 2021 Poster (2021)
Keywords: Reinforcement learning
data efficiency
representation learning
augmentation
virtual trajectory
cycle consistency

Abstract: Learning good feature representations is important for deep reinforcement learning (RL). However, with limited experience, RL often suffers from data inefficiency for training. For un-experienced or less-experienced trajectories (i.e., state-action sequences), the lack of data limits the use of them for better feature learning. In this work, we propose a novel method, dubbed PlayVirtual, which augments cycle-consistent virtual trajectories to enhance the data efficiency for RL feature representation learning. Specifically, PlayVirtual predicts future states in a latent space based on the current state and action by a dynamics model and then predicts the previous states by a backward dynamics model, which forms a trajectory cycle. Based on this, we augment the actions to generate a large amount of virtual state-action trajectories. Being free of groudtruth state supervision, we enforce a trajectory to meet the cycle consistency constraint, which can significantly enhance the data efficiency. We validate the effectiveness of our designs on the Atari and DeepMind Control Suite benchmarks. Our method achieves the state-of-the-art performance on both benchmarks. Our code is available at https://github.com/microsoft/Playvirtual.

PlayVirtual Augmenting Cycle-Consistent Virtual Trajectories for Reinforcement Learning
Tao Yu Cuiling Lan
Wenjun Zeng² Mingxiao Feng' Zhizheng Zhang² Zhibo Chen
*University of Science and Technology of China Microsoff Research Asia
yutao6660mail. ustc. wsw..utttt... edu, en, {culan, wezeng) vezeng)omicrosoff com
fmxustc@mail ustc. edu, cn, srrzhhang@microoooft com, chenzhibo@ustc edu.
Abstract
Learning good feature representations is important for deep reinforcement learning (RL). However, with limited experience, RL often suffers from data inefficiency for training. For un-experienced or less-experienced trajectories (i.e., state-action sequences), the lack of data limits the use of them for better feature learning. In this work, we propose novel method, dubbed Play Virtual, which augments cycle-consistent virtual trajectories to enhance the data efficiency for RL feature representation learning. Specifically, Play Virtual predicts future states in a la- tent space based on the current state and action by a dynamics model and then predicts the previous states by backward dynamics model, which forms a tra- jectory cycle. Based on this, we augment the actions to generate a large amount of virtual state-action trajectories. Being free of groudtruth state supervision, we enforce trajectory to meet the cycle consistency constraint, which can signifi- cantly enhance the data efficiency. We validate the effectiveness of our designs on Atari and Deep Control Suite benchmarks. Our method achieves the soaite--ffthheeaee performance both benchmarks. Our code is available at httpss//ghauul..oo///mcssstttttp
Introduction
Deep reinforcement learning (RL) combines the powerful representation capacity of deep neural networks and the notable advantages of RL for solving sequential decision-making problems. It has made great progress many complex control tasks such as video games [35] 46 3, and robotic
control [2 54 36). Despite the success of deep RL, it faces the challenge of data/sample inefficiency
when learning from high-dimensional observations such as image pixels from limited experience 27 61 29). Fitting high-capability feature encoder using only scarce reward signals is data inefficient and prone to suboptimal convergence (49] Humans can learn to play Atari games in several minutes, while RL agents need millions of interactions [44]. However, collecting experience in the real world is often expensive and time-consuming. One may need several months collect interaction data for robotic arms training [22] or be troubled by collecting sufficient patient data to train a healthcare agent [52]. Therefore, from another perspective, making efficient use of limited experience for improving data efficiency becomes vital for RL.
Many methods improve data efficiency by introducing auxiliary tasks with useful self-supervision to learn compact and informative feature representations, which better serves policy learning. Previous works have demonstrated that good auxiliary supervision can significantly improve agent learning,
maximizing Predictive Information [38 1 42 31 or promoting discrimination through contrastive learning
like leveraging image reconstruction [49], the prediction of future states 41
13
30 40
This work done when Tao Yu was intern at Microsoft Research Asia. Corresponding Author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

2242
Superviston
Encoder
Poliey Learning
RL Loss
Q
DM
DM
2p+1
Zz+2
2
Ausittary Task
Auaiitaar Loss
1
Ac+1
Bc+K-1
(a)
(b)
Z
9 c+1
2112
2
0
DM
DM
DM
1
Consistency
-
ăct -
ä(+K-1 ~A
BDM
8DM
RDM
Z
Z
2+x-1
Figure 1: Illustration of the main pipeline of our method. (a) A glance the overall framework which consists of an encoder for learning the latent state representation 2t, a policy learning head, and our auxiliary task module. The auxiliary task module consumes real trajectory as shown in (b) and an augmented virtual trajectory as shown in (c), respectively. In (b), we train the dynamics model (DM) be predictive of the future state based on the input state and action, with the supervision from the future state. To enhance data efficiency, as shown in (c), we augment the actions to generate virtual trajectory formed by forward and backward trajectory. Particularly, the forward state-action trajectory is obtained based on the current state Z₁, the DM and a sequence of gegmmetted/gneerateee actions (e.g., c. for steps). Similarly, based on the predicted future state 2 backward dynamies model (BDM), and that sequence of augmented actions, we obtain the backward state-action trajectory. For the virtual trajectory. we add the consistency constraint on the current state z, and the predicted current state z' for oplimizing the feature representations.
Although the above methods have been proposed to improve the data efficiency of RL, the limited experience still hinders the achievement of high performance. For instance, the current state-of-the-an method SPR [40] only achieves about 40% of human level on Atari [2] when using data from 100k interactions with the environment. Some methods improve data efficiency by applying modest image augmentation (i.e., transformations of the input images like random shifts and intensity) [28 [5]. Such perturbation images improves the diversity of appearances of the input images. However, it cannot enrich the experienced trajectories (state-action sequences) in training and thus the deep networks are still deficient in experiencing/ingesting flexible/diverse trajectories.
29 59 32 25
In this work, to address the above problem, we propose a new method dubbed Play Virtual, which augments cycle-consiee ceceecconsistent virtual trajectories to improve data efficiency. Particularly, we predict future states in the latent space by dynamics model in the forward prediction (trained using real trajectories to predict the future state based on the current state and action) and then predict the previous states by a backward dynamics model, forming loop. In this way, we can augment the actions to generate a large amount of virtual/fitiitouus state-action transitions for feature representa- tion training, with the self-supervised cycle consistency constraint (which is a necossary/desiraalc condition for good feature encoder). Note that such design is free of groundtruth supervision. Our augmentation to generate abundant virtual trajectories can significantly enhance the data efficiency. As illustrated in Figure I top of a baseline RL framework, we introduce our Play Virtual for augmenting state-action virtual trajectories with cycle consistency constraint. The dynamics model infers the future states recurrently based on the current state and set of randomly sampled actions, and the backward dynamics model predicts the previous states according to the predicted future state and those sampled/augmented actions. We enforce the backwardly predicted state of the current time step to be similar to the original current state to meet the cycle consistency constraint.
We summarize the contributions of this work follows:

---

We pinpoint that augmenting the experience of RL in termss of trajectories is important for feature representation learning RL (which is sequential decision-making problem) to enhance data efficiency. To our best knowledge, we are the first to generate virtual trajectories (experience) for boosting feature learning.
We propose practical method Play Virtual for augmenting virtual trajectories under self-supervised cycle consistency constraint, which significantly improves the data efficiency.
We demonstrate the effectiveness of our Play Virtual on discrete control benchmark Atari [2] and continuous control benchmark DMControl Suite [43, where our PiatVirtual achieves the best performance on both benchmarks.
2 Related Work
2.1 Data/Sample-effciint RL and Representation Learning for RL
Learning from visual observations is a fundamental yet challenging problem in RL. In practice, collecting experience is expensive and time-consuming. Thus, in general only limited experience is available for training RL agent, which results in the difficulty in fitting a high-capability feature encoder, i.e., learning powerful feature representations. Therefore, data-efficient RL has attracted a lot of attention and many methods are designed to make efficient use of limited experience for improving data efficiency. These methods can be grouped into three categories. (i) Auxiliary task
based methods introduce auxiliary task to help representation learning of the states (21 49 41 [3
30 40 38
(ii) Data augmentation based methods increase the diversity of image appearance through data augmentation [50] 28 40). But they do not augment virtual actions. (iii) World model based methods explicitly model the environment in order to enable the planning or
29 59
promote the policy learning [6 17 611. We only focus the first two categories since the third one
is not specifically designed to enhance the efficiency of feature representation learning.
In recent years, unsupervised representation learning has made significant progress in natural language processing 6 33 and computer vision [38 18 [2]. aims learn generic and discriminative feature representations without groudtruth supervision, i.e., by introducing some unsupervised auxiliary tasks. In RL, good state representation removes the redundancy and noise elements from the original high-dimensional state, and reshapes the original state space into compact feature space. Recently, many works explore representation learning in RL and have shown promising performance improvement. UNREAL [2I] introduces a number of auxiliary tasks such as reward prediction. Yarats el al. [49 introduce an auxiliary reconstruction loss to aid feature representation learning. Considering the ability to model what will happen next is necessary for success on many RL tasks [4 31,, some works train agents to be predictive of the future states. [41, PBL. [R] SLAC 33 and SPR [40] explicitly predict the future states by modeling dynamics. Similarly, CPC (38) ST-DIM DI. DRIML [34]. ATC [42] and PI-SAC [31] maximize the mutual information between the current state and the future state by using InfoNCE [38 42), Deep InfoMax (20 II 34), or Conditional Entropy Bottleneck 9 31. Some works exploit contrastive learning to learn
discriminative representations 29 59
32 25
CURL [29] extracts high-level features from raw
pixels using contrastive learning, by encouraging the similarity between the augmentations of the
same image and the dissimilarity between different images.
Inspired by the success of data augmentation in computer vision, DrQ (50] and RAD 28 explore the effectiveness of data augmentation in RI. and show that increasing the diversity of the training images by simple image augmentation (such as random cropping) can improve the data-efficiency. In SPR [40], besides the prediction of its own latent state representations multiple steps into the future, they improve performance by adding data augmentation on the future input image as the future state supervision. However, all these methods train the encoder using the real interaction transitions, There is a lack of an efficient mechanism to generate reliable state-action pair transition trajectories for better training the feature encoder.
In this paper, we propose a method, dubbed Play which enables the augmentation of trajec- tories with unsupervised cycle consistency constraint for training the feature representation. This effectively enhances the data efficiency and our trajectory augmentation is conceptually orthogo- nal/complementaryy to the previous augmentation auxiliary task based methods.
3

---

2.2 Cycle Consistency
Many works have explored the high-level idea cycle consistency to address different challenges in various tasks such as image- -imeeagge translation [60] [1 24 image matching [56] 58 57]. feature representation learning [47] 7 26]. For image-to-image translation, CycleGAN [60] introduces the eycle consistency constraint to Generative Adversarial Networks [TO] to remove the requirement of groudtruth paired images for training, by enforcing the back-translated image to be the same as the original one. Zhang al. [55] learn the correspondences that align dynamic robot behavior across two domains using cycle consistency: given observations in time t, the future prediction in time + 1 should be consistent across two domains under the consistent action taken. The purpose is to enable a direct transfer of the policy trained in one domain to the other without any additional fine-tuning, To learn visual correspondence from unlabeled video, Wang el al. 447 propose track target backward and then forward to meet temporal cycle consistency on the feature representations, using the inconsistency between the start and end points as the loss function. Kong et al. (26) propose cycle- contrastive learning for learning video representations, which is designed to find correspondences across frames and videos considering the contrastive representation in their domains respectively. where the two domain representations form a frideevdaae cycle.
Different from the above works, in order improve data efficiency in RL, we propose to augment virtual state-action trajectories to enrich the "experience" of the feature encoder for representation learning. To ensure the reasonblenees/ccoreetness of the generated transitioon//raaeccooiss and make use of them, we take the necessary condition of good trajectory, i.e., satisfying the cycle consistency of the trajectory, constraint to optimize the network.
3 Cycle-Consistent Virtual Trajectories for Representation Learning in RL
3.1 Background
We consider reinforcement learning (RL) in the standard Markov Decision Process (MDP) setting where an agent interacts with the environment in episodes. We denote the state, the action taken by the agent and the reward received timestep in an episode as 8t, a,, and rt, respectively. We aim to train an RL agent whose expected cumulative reward in each episode is maximized.
With the observation being high-dimensional short video clip each timestep, the powerful represen- tation capability of deep neural networks for encoding state and a strong RL algorithm contribute to the success of RL agent. Similar to [29]. we use the widely adopted RL algorithm Rainbow
[19] for discrete control benchmarks (e.g., Atari (2) and Soft Actor Critic (SAC) [15] for continuous
control benchmarks (e.g., DMControl Suite (43). Following SPR [40], we introduce a dynamics
model (DM) to predict the future latent states multiple steps, which enables forward state-action
trajectory. We take SPR 40 our baseline scheme.
3.2 Overall Framework
Considering the data efficiency in RL with limited experience, we propose a method named Play Vir- tual to efficiently improve the feature representation learning of RL. Our key idea is to augment the actions to generate virtual state-action trajectories for boosting the representation learning of the encoder. Particularly, we eliminate the need of groudtruth trajectory supervision for the augmented sequences by using a cycle consistency constraint, which thus enhances data efficiency in training.
Figure M Illustrtess the main pipeline of our framework (with some details not presented for clarity). As shown in (a), it consists of an encoder which encodes the input observation s; into low-dimensional latent state representation 2,, an RL policy learning head (Rainbow [19] or SAC (15),, and our auxiliary task module. Particularly, as shown in (c), our auxiliary task module consists of a dynamics model (DM) which predicts future latent state based on the current state and the action, and a backward dynamics model (BDM) for backward state prediction. Following SPR 40), the DM is trained with the real state-action trajectory under the supervision of the future state (see (b)) to assure its capability of generating "correct" state transition. However, under limited experience, the encoder has few opportunities to be trained by those un-experienced or less-experienced state-action trajectories, which should be important to enhance data efficiency. To address this problem, as illustrated in (c), we add BDM which predicts the previous state based on the current state and

---

the previous action. Together with the DM, the forward predictions and backward predictions form a cycle/loop, where the current state and the backwardly predicted current state are expected be the same. Particularly, we augment the actions to generate virtual trajectories in order to train the network to "see" more flexible experiences with cycle consistency constraint. Our method contains three main components which we describe below.
Dynamies Model for Prediction of Future States. A one-step Markov transition (Sr, a,s St++1) contains a current state S, E S, an action a, E A, and the next state Sz-1 E S. The transition model determines the next state S given the current stateaaction pair (s,, ar).
Considering the ability to model what will happen next is important for RL tasks, many works train agents to be predictive of the future states to learn good feature representations [40]
[3 30).
Following SPR [40), we introduce a dynamics model (DM) l((:,)) :) to predict the transition dynamics
(Zt, a.) 7 in the latent feature space, where 2, f(s)) is encoded by the feature encoder f()) of the current input video clip St. As illustrated in Figure [T]. based on the current input state 24 and
sequence of actions a K-1 11 obtain sequence of K predictions Z:+1:t4K K of the future state representations using the action-conditioned transition model (i.e., DM) h(s-)) .) by computing the next state iteratively as
[h(z+ as+x) h((24kk as+k)
if k: = if k: = 1.1.
2
(1)
We train the DM with the supervision of the future state representations obtained from the recorded real trajectory (iee. from the recorded future video clip). Following SPR [40], we compute the
prediction loss by summing over difference (error) between the predicted representations Z and
observed representations ZL+k at timesteps t + for 1 S k I K measured in a "projection" metric space as:
Ppred Σ d(zs+k k; 2(tk),
where d denotes the distance metric a "projection" space 40 (see Appendix A for more details).
This module has two roles in our framework. (i) The future prediction helps to learn good feature state representation, which enables the scheme SPR [40] that
our strong baseline. (ii) It paves the way for our introduction of cycle-consistent virtual trajectories for improving data efficiency.
Backward Dynamics Model for Prediction of Previous States: Backward transition model intends to determine the previous state s, given the next state S,+1 and the causal action a,. We introduce a backward dynamics model (BDM) b(.,-) to predict the backward transition dynamics (Z++1, a.) - Zt in the latent feature space.
37. backward induction has been exploited to predict the preceding states that terminate at a given high-reward state, where these traces of (state, action) pairs are used to improve policy learning. Their purpose is to emphasize the training on high-reward states and the probable trajectories leading to them to alleviate the problem lack of high reward states for policy learning. In contrast, we introduce BDM which predicts previous states (to have a backward trajectory) in order to build a cycle/loop with the forward trajectory to enforce the consistency constraint for boosting feature representation learning.
In previous works
As illustrated in Figure [(c), based on the hidden state ži+k and a sequence of actions á 11:1 we obtain a sequence of k predictions 2 of the previous state representations using the BDM b(.s-) by computing the previous state iteratively as
[b(2)+ āc+k-1)
if K if = - 1, 2.... 1.
2
b(2/+4) 11
Action Augmentation and Cycle Consistency Constraint. Given the DM, BDM, current state, and sequence of actions, we can easily generate forward trajectory and a corresponding backward trajectory which forms a loop/cycle/forward baakkardd trajectory. As we know, for an encoder which is capable of encoding observations to suitable feature representations, the feature representations trajectory should in general be
of the start state 2, and the end state z/ of a similat/consistent, given reasonable DM and BDM.
5

---

Therefore, as illustrated in Figure [(c) we enforce a consistency constraint between the start state z; and the end state z to regularize the feature representation learning. this way. by augmenting actions (generating/sampligg virtual actions), we can obtain abundant virtual cycle-consistent tra- trajectories.
jectories for training. Note that in the training, we do not need supervision of the states from real
Here, we mathematically define the cycle-consistent feature representation in forward-backward trajectory as below.
Definition 1. Given a (forward) dynamics model and a backward dynamics model b, cycle- consistent feature representation 2, in horward-backwarr trajectory I is representation of the current state that meets the following condition when experiencing any sequence of K actions {a,, a(+1....,, a++K-1] 2 sampled from action space A:
E.-ldd z:) =0,
where dm distance metric on space M and z is the prediction of 2₁ after experiencing sequence of actions in forward prediction and backward prediction as
forward: 2, 21, 2c+h+1 h(2r+k; a1t8); for 00,, KK-1
backward: z'4K 2LtK, Z4+ 0(144411,8-1 for k = K - 1, K 2.......
Given the state z, encoded from the current input St of time t, we randomly sample M sets actions in the action space A. We calculate the cycle consistency loss as:
M Σ m=1
Leye
dmizz,, 21).
M
We describe the alternative distance metrics du on space M and study the influence on performance in Section
we model the backward dynamics using a BDM. This is basically feasible for many real-world applications, e.g., robotic control, and many games. Consider a robotic arm: given a current state (e.g., spatial position and rotation) and the previous action (e-g., quaternions
Discussion: In our scheme, similar to 11 g
can deduce the previous state without much effort. This holds in most games such as chess or Atari. For some cases where there are many-to-one transition (i.e., different states with the same action may be transited into the same state), BDM may learn the most probable state or a mean state to minimize the prediction errors through the powerful fitting capacity of neural networks. More discussion can be found in Appendix C.
six-degree-of-freee ofreeeeffffffommmm- parameters),
Overall Training Objective. The overall training objective of our method is as below:
Listal Lrt + Xprea&pred + Aeye
(5)
where Lrt, Cpred, and Laye denote the RL loss (please refer to Rainbow 19 for discrete control games, SAC [15 for continuous control games), prediction loss (see Eq. (2). and cycle consistency
loss (see Eq. (4), respectively. Apred and Aaye are the hyperparameters for balancing the contributions
of different losses.
Experiments
We introduce the experimental setup including environments, evaluation, and implementation details. We conduct extensive ablation studies to demonstrate and analyze the effectiveness our designs.
4.1 Setup
Environments and Evaluation. We evaluate our method on the commonly used discrete control benchmark Atari (2), and the continuous control benchmark of DMControl (3 Following 29 50
we measure the performance of different methods at 100k interaction steps (400k environment steps with action repeat of 4) on Atari (also refer to as Atari-100k), and at 100k and 500k environment steps
on DMControl (refer to as DMControl-100k. DMC-100k, DMControl-5000 or DMC-500k).

---

respectively. In general, using Atari- 100k on 26 selected games 61 45 23 29
and DMControl-
100k
29 50 has been a common practice for investigating data efficiency.
For Atari-100k, we measure the performance by score, and the median human-normalized score (i.e. median HNS) of the 26 games. Human-nowe noraannzed score on a game is calculated by 8g Sg Sg-Sp
SĄ=Sa, where S
the agent score, SR the score of random play, and SH the expert human score. For DMControl,
the maximum possible score for every environment is 1000 [43]. Following (29) 49 [6] 50 [2],
we evaluate models on the six commonly used DMControl environments. Additionally, we use the median score on them to reflect the overall performance.
our strong baseline (dubbed Baseline) and build our method on top of SPR by augmenting cycle-consistent virtual trajectories for better representation learning (see Figure [[e).. For the backward dynamics model, we use the same architecture as that of the dynamics model. We set the number of prediction steps K to 9 by default. We simply set the number of action sets, i.e., the number virtual trajectories M to 2) 2I, which is proportional to the size of action space |AA in that Atari game. To generate an action sequence, we randomly sample an action from the discrete action space at each step. We report the results of SPR [40] by re-running their source code in all Tables except for Table (results in which are copied from their paper. being higher than our reproduced results).
Implementation Details. For the discrete control benchmark of Atari, we use SPR
For the continuous control benchmark of DMControl, considering the SPR is originally designed only for discrete control, we build a SPR-like scheme SPR as our baseline (dubbed Baseline) for continuous control games. Particularly, we use the encoder and policy networks of CURL (29] as the basic networks. Following SPR [40), we remove the contrastive loss CURL and introduce BYOL [2] heads to build SPR-like baseline scheme. We use the network architecture similar to
dynamics model DBC [53 to build the dynamics model SPR and the backward dynamics
model in our Play Virtual. We follow the training settings in CURL except the batch size (reduced from 512 to 128 to save memory cost) and learning rate. We set to 6, and set M to a fixed number 10 with actions randomly sampled from the uniform distribution of the continuous action space.
We set Xpred and Aege 1. For dy, we use the distance metric in SPR 40 More imple- mentation details can be found in Appendix A. All our models are implemented via PyTorch [39].
4.2 Performance Comparison with State-of-the-Arts
Table 1: Scores achieved by different methods Atari-100k. We also report median HNS. We run our Play Virtual with 15 random seeds given that this benchmark is susceptible to high variance across multiple runs. Note that here we report the results of SPR [40] copied from their paper (i.e., 41.5%), which is much higher than our reproduced results using their released source code (i.e., 37.1% 97.1%)
Game
Human
Random
SimPL.e66]]
DER
OTR 23
CURL (2
DrQ [50]
SPR (40)
Play Virtual
Alien
?127.?
227.8
616.9
739.9
824.7
558.2
771.2
801.5
947.8
Amidar
1719.5
5.8
88.0
188.6
82.8
142.1
102.8
176.3
Assault
742.0
222.4
$27.2
431.2
351.9
600.6
452.4
$71.0
702.3
Asterix
8503.3
210.0
1128.3
470.8
628.5
734.5
603.5
977.8
933.3
Bank Heist
753.1
14.2
34.2
$1.0
182,1
131.6
168.9
380.9
245.9
Battle Zone
37187.5
23603.0
5184,4
10124.6
40660.
14870.0
12954.0
16651.0
13260.0
Boxing
12.1
0.1
9.1
0.2
2.5
6.0
35.8
38.3
Breakout
30.5
1.7
16.4
1.9
9.8
4.9
16.1
17.1
20.6
Chopper Command
7387.8
811.0
1246,9
861.8
1033.3
1058.5
780.3
974.8
922.4 4
Crazy Climber
35829.4
10780.5
62583,6
16185,3
21327.8
12146.5
20516.5
42923.6
23176.7
Demon Attack
197L.0
152.1
208.1
508.0
711.8
817.6
1113.4
$45.2
1131.7
Freeway
29.6
0.0
20.3
27.9
25.0
26.7
9.8
24.4
16.1
Frostbite
4334.7
65.2
254.7
866.8
231.6
1181.3
331.1
1821.5
1984.7
Gopher
2412.5
257.6
771.0
349.5
778.0
669.3
636.3
715.2
684.3
Jamesbond
302.8
29.0
125.3
301.6
112.3
471.0
236.0
365.4
394.7
Kangaron
779.3
2384.7
Krull
2665.5
1598.0
4539.9
2851.5
3277.9
4229.6
4018.1 1
3688.9
3880,7
Kung Masser
17257.2
14259.0
Ms Pacman
6951.6
307.3
1450.0
1204.1
941.9
1465.5
960.5
1313.2
1335.4
Pong
-3.0
Private Eye
69571.3
24.9
$8.3
97.8
100.0
218.4
-13.6
124.0
93.9
Qbert
13455.0
Road Runner
7845.0
11.5
5640.6
9600.0
2696.7
$661.0
8895. 1
14220.5
13534,0
Seaquest
42054.7
68.4
683.3
354.1
286.9
384.5
301.2
583.1
$27.7
Up Down
11693,2
533.4
3350.3
2877.4
2847.6
2955.2
3180.8
28138.5
10225.2
Median HNS (%)
100
14.4
16.1
20.4
26.8
47.2

---

Table 2: Scores (mean and standard deviation) achieved by different methods the DMControl- -00k and DMControl-500k We run our Play Virtual with random seeds. Note that SPR 440 is originally designed only for discrete control. For the continuous-contro) environments, extend SPR to new version named SPR
100k Step Scores
PlaNet [6
Dreamer 17
SAC+AE 49
SLAC 30
CURL,
DrQ
SPR 40
Play Virtual
Finger, spin
136 216
56
915
Curipole, swingup
297 39
326 2:27
311
+: 11
582 1: 146
759 92
799 = 42
816 1: 36
Reacher, casy Cheetah, run
314
155
274
538
+ 233
213
638
785 142
138 88
235 + 137
267 +
319 + 56
299
344 =
467 = 36
474 + 50
Walker, walk
224 48
24
612 164
173
Ball in cup, catch
0 +
246 + 174
391 + 82
512 + 110
769 43
913 53
861 233
926 + 31
Median Score
137.0
295.5
351.0
436.5
560.0
685.5
719.0
800.5
500k Step Scores
Finger, spin
561 + 284
79% + 183
884 + 128
673 + 92
926 + 45
938 + 103
924 132
963 +:40
Caripole. swingup
475 71
762 +
735
841 :: 45
868 10
870
865 =
Reacher, casy
210 + 390
795 + 164
627 + 58
929 44
942
925
942 - 66
Cheetah,
+ 131
570 + 253
96
716
Walker, walk
351 + 58
897 + 49
847 + 48
842 51
902 43
921 45
916 =
928 + 30
cup, catch
380
794
852
963
Median Score
405.5
794.5
764.5
757.5
914.0
929.5
920.0
935.0
Comparison Atari. On Atari-100K, Table [ shows the comparisons with the state-of-the-art methods. We also report the results of random play (Random) and expert human play (Human) (copied from [48).. Play Virtual achieves a median HNS of 47.2%, significantly outperforming all previous methods. Play Virtual surpasses the baseline SPR [40] (with median HNS of 41.5% reported in their paper) by 5.7%. We have re-run the released source code of SPR with 15 random seeds and obtain a median HNS 37.1%, which suggests that our improvement over SPR actually 10.1 %.
Comparison on DMControl. For each environment in DMControl, we run our PlayVirtual with 10 random seeds to report the results. Table 2 shows the comparisons with the fatte-of-the-eee methods. Our method performs the best for the majority (5 out of 6) of the environments on both DMControl- -00k and DMControl-S0OL. (i) On DMControl- 1 100k which is in low data regime, our method achieves the highest median score of 800.5, which is about 11.3 1.3% % higher than SPR 16.7% higher than DrQ (50) and 42.9% higher than CURL [29]. (ii) On DMControl-500
our method achieves median score of 935.0, which approaches the perfect score of 1000 and outperforms all other methods. Therefore, our method achieves superior performance in both data-efficiency and asymptotic performance.
4.3 Ablation Studies
We use the median HNS and median score measure the overall performance on Atari and DMCon- trol, respectively. We run each game in Atari with 15 random seeds. To save computational resource, we run each environment in DMControl with 5 random seeds (instead of 10 as in Table 2
Effectiveness of Play Virtual. As described in baseline
Table 3: Effectiveness of PlayVirtual
top of Baseline, which is SPR (40) for discrete control on Atari, and SPR for continual control on DMCon- trol. "w/o Pred" denotes disabling future prediction Baseline. Baseline+ BDM denotes the scheme that BDM is incorporated into Baseline.
Section 4.1] we take SPR [40] (i.e., Baseline) on discrete control benchmark Atari, and SPR continuous control bench- mark DMControl. On top of Baseline, we val- idate the effectiveness of our Play Virtual which augments cycle-consistent virtual trajectories for improving data efficiency. Table 3 shows the comparisons. We can see that Play Vir- tual achieves median HNS of 47. 47.2%, which outperforms Baseline by 10.1 % on Atoki- On DMControl- 100, Play Virtual improves Base- line from 728.0 to 797.0 in terms of median score (i.e., a relative gain of 9.5%). As a com- parison, Baseline outperforms Baseline w/o Pred by 3.7% on Atari-100k, where "Pred" denotes the prediction of future state in SPR / SPRT(i.. the contribution of SPR (40). The large gains of our Play Virtual over Baseline demonstrate the effectiveness of our Play Virtual in boosting feature
Model
Atari-100k
DMControl-100k
Baseline w/o Pred
33.4
680.0
Baseline
37.1
728.0
Baseline+BDM
38.4
741.0
Play Virtual
47.2
797.0

---

representation learning. In addition, to further benchmark Play Virtual's data efficiency, we compare the testing performance in every 5k environment steps at the first 100k on DMControl, where the result curves in Appendix B show that our Play Virtual consistently outperforms Baseline.
One may wonder whether the major performance gain of our Play Virtual is attributed to the intro- duction of backward dynamics model (BDM) or by our augmentation of virtual trajectories. When we disable the augmentation of virtual trajectories. our scheme degrades to Baseline+ BDMM, where a BDM is incorporated into the baseline SPR (or SPR !) and only the real trajectories go through the BDM. In Table 3 we can see that introducing BDM does not improve the performance obviously and our augmentation of cycle-consistent virtual trajectories for regularizing feature representation learning is the key for the success.
Influence of Prediction Steps K. We study the influence of K for both our Play Virtual and the baseline scheme SPR/SPRT. Table shows the performance. When K = O, both schemes degrade to Baseline w/o Pred (where future pre- diction is disabled in SPR/SPR\) ). We have the following observatiios//concluiioss (i) Given the same number of prediction steps K (beside
Table 4: Influence of prediction steps K for our Play Virtual and the baseline scheme SPR/SPR
Benchmark
Model
K=0
K=3
K=6
K=9
K=12
Atari-1008 100k
VaryVal
33.4
33.9
34.8
35.2
39.2
47.2
43.1
DMC-100K
SPR
664.0
725.0
723.0
728.0
721.5
Play Virtual
664.0
775.5
797.0
795.0
794.5
O), our Play Virtual consistently outperforms the baseline scheme SPR/SPR on both benchmarks Atari- 1ww.////Oooooooo---/ 100k. (ii) Our Play Virtual achieves the best performance at K 9 on Atari and K = 6 on DMControl, which outperforms the baseline at the same K by 10.1 1 and 9.5% (relative gain) on Atari and DMControl, respectively. Note that the performance of SPR [40] obtained using their source code at K=5 (note K=5 is used in SPR paper) is 36. 1 % (which is 41.5% in their paper) on Atari-10Ok. (iii In SPR [S0]/ a too small number of prediction steps cannot make the feature representation sufficiently predictive of the future while a too large number of prediction steps may make the RL loss contributes less to the feature representation learning (where a more elaborately designed weight Apred is needed). Our Play Virtual follows similar trends.
What does Augmenting Sycle-Consistent Vir- fual Trajectories Help? We propose the aug- mentation of cycle-consistent virtual trajectories in order to boost the feature representation learn- ing of RL for improving data efficiency. In the training, the cycle consistency loss Leye over the virtual trajectories would optimize the param- eters of the encoder, DM and BDM. One may wonder what the gain is mainly coming from. Is it because the DM is more powerful/accurae that enables better prediction of future states? Or is it because the encoder becomes more powerful to provide better feature representation? We validate this by letting the cycle consistency loss Laye not update DM, where DM is only optimized by prediction loss Inpred as in SPR. We denote this scheme as PlayVirtul--ND. Shhows that obtain a gain of 6.9% in Play VlayVirtal--N from the regularization of Leye on the encoder and a gain of 10.1% % in PlayVirtual from the regularization on both the encoder and DM on Atari. Similar trend is observed on DMControl. This implies that the augmentation of cycle-consistent virtual trajectories is helpful DM training but the main gain is brought by its regularization on the feature representation learning of the encoder.
Table 5: Impact of the augmentation of cycle- consistent virtual trajectories on feature represen- tation learning. Play Virtual-ND denotes that we do not use the cycle consistency loss over virtual trajectories to update the dynamic model.
Model
Atari-100k
DMControl-100k
Baseline
37.1
723.0
Play VirtVlrtuN Play Virtual
44.0
777.5
47.2
797.0
Influence of Distance Metric dı on Space M. For the distance metric d.m in space M, compare cosine distance on the latent feature space Miatent, i.e., dy( 2,) = 2 2 [|c [|:]]] and on the "projection" space Mprojj as in SPR 40 (see Appendix A for more details). We compare the influence of feature space for calculating cycle consistency loss and show the results in Table 6 On the Atari benchmark, our Play Virtual with distance metric on space Miatent and with distance metric on space Maroj significantly outperforms Baseline by 7.7% and 10.1% respectively. This demonstrates the effectiveness of our key idea of exploiting virtual trajectories for effective representation learning. Maroj performs 2.4% better than Miatent- That maybe because for Play Virtual and Baseline for Atari, latent feature z, (which corresponds to 64 x 7 7 feature map) preserves more spatial information than projected feature, where the former is less robust to
9

---

be matched across two augmented observations due to spatial misalignment. On the DMControl benchmark, our Play Virtual with distance metric on space Mintent and with distance metric on space Mproj significantly outperforms Baseline by 70.5 and 69.0 in terms of median score, respectively. The performance of Miatent and Mproj are similar. Note that the latent feature 24 of Play Virtual or Baseline (built based on CURL) corresponds to feature vector which is obtained after fully connected layer the backbone network, which does not face the spatial misalignment problem caused by the augmentation. We use Mproj the default metric space in this work.
Table 6: Influence of distance metric space M. Mintent and Mproj denote the of the latent feature space and the "projection" space, respectively.
Model
Atari-100k
DMControl-100k
Baseline
37.1
728.0
PlayVirtual(
44.8
798.5
Play Virtual Mproi))
47.2
797.0
Influence of the Number of Virtual Trajectories M. Table 7 shows the influence of the number of virtual trajectories M. We can observe that small M (less generated virtual trajectories) is inferior to suitable M. That may be because too small M cannot cover diverse experiences for feature representation learning. When M is too large, it brings less additional benefit. That may be because a suitable number of trajectories is enough for regularizing the network training. We observe that the performance drops when M is too large. That may be because a very large M would increase the optimization difficulty in practice.
Table 7: Influence of the number of virtual trajectories M.
Atari-100k
M
|A] 39.5
2[.4] 47.2
3|,||
Median HNS(%)
37.1
42.5
DMControl-10W
M
0
1
10
20 806.0
30
Median Score
723.0
770.5
797.0
792
Conclusion
With limited experience, deep RL suffers from data inefficiency. In this work, propose method, dubbed Play Virtual, which augments cycle-consistent virtual state-action trajectories to enhance the data efficiency for RL. feature representation learning. Play Virtual predicts future states based on the current state and a sequence of sampled actions and then predicts the previous states, which forms trajectory cycle/loop. We enforce the trajectory to meet the cycle consistency constraint to regularize the feature representation learning. Experimental results on both the discrete control benchmark Atari and continuous control benchmark DMControl demonstrate the effectiveness of our method, where we achieve the state-of-thee performance on both benchmarks.
Acknowledgments and Disclosure of Funding
This work was supported in part by the National Key Research and Development Program of China 201 8AAA01010000 and NSFC under Grant U1908209, 61632001 and 62021001.
10

---

References
[1] Anand, A., Racah, E., Ozair, S., Bengio, Y., Côté, M.-A., and Hjelm, R. D. Unsupervised state representation learning in atari. In Advances in Neural Information Processing Systems, 2019.
[2] Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 2013.
[3] Berner, C., Brockman, G., Chan, B., Cheung, V., Debiak, P., Dennison, C., Farhi, D., Fischer, Q., Hashme, S., Hesse, C., et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint 1912.0660,, 2019.
[4] Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., and Joulin, A. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 2020.
[5] Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual representations. In International Conference Machine Learning, pp- 1597-1607. PMIR, 2020.
[6] Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805, 2018
[7] Dwibedi, D., Aytar, Y., Tompson, J., Sermanet, P., and Zisserman. Temporal cycle- consistency learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1801-1810, 2019.
[8] Edwards, A. D., Downs, L., and Davidson, J. C. arXiv preprint 1803. 10227, 2018
reinforcement learning.
[9] Fischer, I. The conditional entropy bottleneck. Entropy, 22(9):999, 2020.
[10] Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A. C., and Bengio, Y. Generative adversarial nets. In Advances in Neural Information Processing Systems, 2014.
[11] Goyal, A., Brakel, P., Fedus, W., Singhal, S., Lillicrap. Levine, S., Larochelle, H., and Bengio, Y. Recall traces: Backtracking models for efficient reinforcement learning. In International Conference on Learning Representations, 2019.
[12] Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., Piot, B., kavukeuoglu, k., Munos, R., and Valko, M. Bootstrap your own latent new approach to self-supervised learning. In Advances in Neural Information Processing Systems, 2020.
[13] Guo, Z. D., Pires, B. A., Piot, B., Grill, J.-B., Altché, F., Munos, R., and Azar, M. G. Boot- strap latent- predictied representations for multitask reinforcement learning. In International Conference Machine Learning, pp. 3875-3888 PMLR, 2020.
[14] Ha, D. and Schmidhuber, J. World models. arXiv preprint /80. 2018.
[15] Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al. Soft actor-critic algorithms and applications. arXiv preprint
1rXiv:1812.05900. 2018.
[16] Hafner, D., Lillicrap. T., Fischer, I., Villegas, R., Ha, D., Lee, H., and Davidson, J. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning. pp. 2555-2505 2565. PMLR, 2019.
[17] Hafner, D., Lillicrap, T., Ba, J., and Norouzi, M. Dream to control: imagination. In International Conference on Learning Representations, 2020.
Learning behaviors by latent
11

---

[18] He, K., Fan, H., Wu, Y., Xie, S., and Girshick. R. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 9729-979999 -9338, 2020.
[19] Hessel, M., Modayil, J., Van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., and Silver, D. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[20] Hjelm, R. D., Fedorov, A., Lavoic-Marchildonn S., Grewal, K., Bachman, P., Trischler, A., and Bengio, Y. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations, 2018.
[21] Jaderberg, M., Mnih, V., Czarnecki, W. M., Schaul, TT,, Leibo, J. Silver, D., and Kavukcuoglu, K. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv:1611.00 1.1.99977 2016.
[22] Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., Quillen, D., Holly, E., Kalakrishnan, M., Vanhoucke, V., et al. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv preprint arXiv: /806, 10293, 2018.
[23] Kielak, K. Do recent advancements in model- deep reinforcement learning really improve data efficiency? arXiv preprint arXiv: 200::2003.11110 10181, 2020.
[24] Kim, T., Cha, M., Kim, H., Lee. J. K., and Kim, J. Learning to discover cross-domain relations with generative adversarial networks. In International Conference on Machine Learning, pp. 1857-1865. PMLR, 2017.
[25] Kipf, T., der Pol, E., and Welling, M. Contrastive learning of structured world models. In International Conference on Learning Representations, 2020.
[26] Kong, Q., Wei, W., Deng, Z., Yoshinaga, T., and Murakami, T. Cycle-contrast for self-supervised video representation learning. In Advances in Neural Information Processing Systems, 2020.
[27] Lake, B. M., Ullman, T. D., Tenenbaum, J. B., and Gershman, S. J. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
[28] Laskin, M., Lee, K., Stooke, A., Pinto, L., Abbeel, P., and Srinivas, Reinforcement learning with augmented data. In Advances in Neural Information Processing Systems, 2020.
[29] Laskin, M., Srinivas, A., and Abbeel, P. Curl: Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning, pp. 5639-5650. PMLR, 2020.
[30] Lee, A. X., Nagabandi, A., Abbeel, P., and Levine, S. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. In Advances in Neural Information Processing Systems, 2020.
[31] Lee, K.-H., Fischer, I., Liu, A., Guo, Y., Lee, H., Canny, J., and Guadarrama, S. Predictive information accelerates learning in RI. arXiv preprint arXiv:2007. 12401, 2020.
[32] Liu, G., Zhang, C., Zhao, L., Qin, T., Zhu, J., Jian, L., Yu, N., and Liu, T.-Y. Return-base based contrastive representation learning for reinforcement learning. In International Conference on Learning Representattinss 2021.
[33] Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. Roberta: robustly optimized bert pretraining approach. arXiv preprint 19Xv:1907.11162. 2019.
[34] Mazoure, B., Tachet des Combes, R., DOAN, T. L., Bachman, P., and Hjelm, R. D. Deep reinforcement and infomax learning. In Advances in Neural Information Processing Systems, 2020.
[35] Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533,, 2015.
12

---

[36] Nagabandi, A., Clavera, L. Liu, S., Fearing, R. S., Abbeel, P., Levine, S., and Finn, C. Learning adapt in dynamic, real-world environments through rett-reiiforcemmett learning. arXiv preprint arXiv: 1803. / 1347, 2018.
[37] Nair, S., Babaeizadeh, M., Finn, C., Levine, S., and Kumar, V. Trass: Time reversal as self-supervissin. In IEEE International Conference on Robotics and Automation (ICRA), pP. 115-121. IEEE, 2020.
[38] Oord, A. d., Li, Y., and Vinyals, O. Representation learning with contrastive predictive coding. arXiv preprini arXiv:1807.033: 1807.03748, 2018.
[39] Paszke, A., Gross, S., Massa, F. Lerer, A., Bradbury. J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: imperative style, high-performance deep learning 2019.
library. arXiv preprint arXiv:1912.0//000
[40] Schwarzer, M., Anand, A., Goel, R., Hjelm, R. D., Courville, A., and Bachman, P. Data-efficient reinforcement learning with self-predictive representations. International Conference on Learning Representations, 2021.
[41] Shelhamer, E.,, Mahmoudieh, P., Argus, M., and Darrell, T. Loss is its own reward: Self- supervision for reinforcement learning. ArXiv, abs/1612.07307, 2017.
[42] Stooke, A., Lee, K., Abbeel, P., and Laskin, M. Decoupling representation learning from reinforcement learning. arXiv preprint arXiv:2009.08319 2020.
[43] Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, d. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. Deepmind control suite. arXiv preprint arXiv: 1801.006100 2018.
[44] Tsividis, P., Pouncy, T., Xu, J. L., Tenenbaum, J., and Gershman, S. Human learning in atari. In AAAI Spring Symposia, 2017.
[45] van Hasselt, H. P., Hessel, M., and Aslanides, J. When to use parametric models in reinforcement learning? In Advances in Neural Information Processing Systems, 2019.
[46] Vinyals. O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, H.,
Powell, R., Ewalds, T., Georgiev. P., et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):3003354 2019.
[47] Wang, X., Jabri, A., and Efros, A. A. Learning correspondence from the cycle-consistency of time. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 2566-2576, 2019.
[48] Wang, Z., Schaul, T., Hessel, M., Hasselt, H., Lanctot, M., and Freitas, N. Dueling network architectures for deep reinforcement learning. In International Conference Machine Learning, pp. 1995-2003. PMLR, 2016.
[49] Yarats, D., Zhang, A., Kostrikov, I., Amos, B., Pineau, J., and Fergus, R. Improving sample efficiency in model-free reinforcement learning from images. arXiv preprint arXiv:1910.01741, 2019.
[50] Yarats, D., Kostrikov, I., and Fergus, R. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. International Conference on Learning Representations, 2021.
[51] Yi, Z., Zhang, H., Tan, P., and Gong, M. Dualgan: Unsupervised dual learning for image-to- image translation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2849-28577 2017.
[52] Yu. C., Liu, and Nemati, Reinforcement learning healthcare: A survey, arXiv preprint arxiv:1908.0877 2019.
[53] Zhang, A., McAllister, R. T., Calandra, R., Gal, Y., and Levine, S. Learning invariant repre- sentations for reinforcement learning without reconstruction. In International Conference on Learning Representations, 2021.
13

---

[54] Zhang, M., Vikram, S., Smith, L., Abbeel, P., Johnson, M., and Levine, S. Solar: Deep structured representations for model-based reinforcement learning. In International Conference on Machine Learning, pp. 7444-7453. PMLR, 2019.
[55] Zhang, Q Xiao, T., Efros, A. A., Pinto, L., and Wang, X. Learning cross-domain corre- spondence for control with dynamics cycle-consisteney. arXiv preprint arXiv:2012.09811 2020.
[56] Zhou, T., Jae Lee, Y., Yu, S. X., and Efros, A. A. Flowweb: Joint image set alignment by weaving consistent, pixel-wise correspondences. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 1191-1200, 2015.
[57] Zhou, Krahenbuhl, P., Aubry, M., Huang, Q. and Efros, A. A. Learning dense correspon- dence via 3d-guides cycle consistency. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 117-126, 2016.
[58] Zhou, X., Zhu, M., and Daniilidis, K. Multi-image matching via fast alternating minimization. In Proceedings of the IEEE International Conference Computer Vision, pp. 4032-4040, 2015.
[59] Zhu, J., Xia, Y., Wu, L., Deng, J., Zhou, W., Qin, T., and Li, H. Masked contrastive representtio learning for reinforcement learning. arXiv preprint arXiv:2010.074700 2020.
[60] Zhu, J.-Y., Park, T., Isola, P., and Efros, A. A. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision. pp. 2223-2232, 2017.
[61] Lukasz Kaiser, Babaeizadeh, M., Mitos, P., Osiński, B., Campbell, R. H., Czechowski, K., Erhan, D., Finn, C., Kozakowski, P., Levine, S., Mohiuddin, A., Sepassi, R., Tucker, G., and Michalewski, H. Model based reinforcement learning for atari, In International Conference on Learning Representaionss, 2020.
14


Reviews:
[{"review_id": "oloBJx2C2vI", "reviewer": "Reviewer_Lrfb", "summary": "This paper introduces PlayVirtual, an approach for learning better representations by training an encoder to learn cycle-consistent forward and backward dynamics models. This differs from model-based approaches in that the virtual trajectories are used to train a representation rather than as experience for training a policy. The approach is evaluated within Atari and DM-control tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I found the paper to be clearly written and well-motivated. While forward and backwards dynamics models have been used for generating experiences for training policies, I thought the idea of using these models to aid in creating unseen trajectories for training a representation was interesting and novel. Several works have used forward dynamics models for training representations, but these methods typically require ground-truth state and action pairs. So being able to sample from the action space to generate new experiences is quite useful.\n\nHowever, I had some doubts in terms of the empirical results. While PlayVirtual achieves higher HNS in the Atari experiments, it performs worse than the baseline that it builds upon, SPR, in 12/26 of the environments. Furthermore, there are no results for SPR in the DM-Control tasks showing individual performance for each task, which makes it difficult to determine if the improved performance was from the PlayVirtual model or SPR. The paper would be greatly improved by including these results in Table 2.\n\nComments:\n- I found the description of the model in Figure 1 and throughout the text to be somewhat confusing. In particular, the original actions are not augmented but rather a new trajectory is formed by sampling from the action space. I think it would make the approach more clear to describe in this way.\n- Was the accuracy of the learned dynamics models measured? \n- I’m curious how different action sampling methods could improve performance. For example, sampling based on “surprise” as done in model-based methods rather than uniformly sampling.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is not discussion of the limitations of the approach. However, there is some discussion about the feasibility of the backwards dynamics model. In particular, it could be the case that in a given state $s_{t+1}$, the same action $a$ could have been taken in different states $s_t$. I.e., there might not be a unique state that $b(s_{t+1}, a_t)$ would map to. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4.5"}, "extra_text": {}}, {"review_id": "cDanT-Gdvf5", "reviewer": "Reviewer_ddpW", "summary": "The paper proposes a method to learn better representations for reinforcement learning tasks. They propose augmenting the feature encoder with two dynamics models: 1) forward dynamics model that predicts future latent state given an action and the current latent state 2) backward dynamics model that takes the current latent state and previous action to predict the previous latent state. They use these 2 models to apply 2 losses: 1) future latent state prediction loss 2) cycle-consistency constraint that ensures that if you iteratively predict the future latent state K times into the future using the forward model and K times back into the past using the backward model, the difference between the initial latent state and predicted latent state should be minimal. \n\nThey show empicirally that these additional losses improve performance of an RL algorithm on multiple benchmarks (Atari and DMControl).  They have an ablation study that shows how important each of the auxiliary losses are on both benchmarks. ", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1) Extensive experiments on multiple RL benchmarks using strong baselines shows that cycle-consistency constraints result in better features which leads to better performance.\n2) Nice ablation study showing the importance of each loss.\n3) Proposed idea is quite general (independent of environment, domain, continuous or discrete. image or state) which might be useful in other fields like robotics. \n4) Well organized paper with enough information to reproduce the work.\n\nWeakness\n1. Analysis of Virtual Trajectories\nSince these trajectories are in the latent space, it is not clear what the latent space is actually encoding. The model can always project a latent state into the \"future\" by trivially mapping it to a constant. This makes backward mapping \"in time\" also easy. This is probably not happening as adding the losses is improving performance. But having some statistics to measure that the predictions done by forward and backward model are meaningful will be nice (how much does the latent state z change as it forward and backward projected). Some visualization might also be useful. Can the authors take an initial frame and the forward model, predict K time steps into the future and use that predicted embedding to get the nearest-neighbor frame from the dataset. Comparison with initial frame and this retrieved frame should give some insight into what is being modeled by the dynamics model. Can the authors use different actions to show the forward and backward models are sensitive to the input action or do they ignore the action? \n\n2. How are actions sampled while applying the loss?\nThe authors state \"In this way, by augmenting actions (generating/sampling virtual actions), we can obtain abundant virtual cycle-consistent trajectories for training\". Do they randomly sample many actions while training the forward backward model? How do the authors obtain the latent for unseen actions as it seems you can only apply loss to seen trajectories as the true z^tilde_t+k comes from the observed trajectory?\n\n3. Virtual trajectories not used in RL loss\nThe proposed virtual trajectories are not used directly for training the RL algorithm i.e. the auxiliary losses only use the augmented trajectories. This is concerning because the authors state that \"we enforce a trajectory to meet the cycle consistency constraint, which can significantly enhance the data efficiency\". As the authors show this constraint is applied after the agent has interacted many times with the environment. Without these initial interaction steps it is difficult to learn a good forward/backward model. While the authors generate trajectories in the latent space these are not used by the RL loss which makes it difficult to motivate that the proposed approach will be significantly sample efficient? \nCan the authors use these virtual trajectories in the RL loss? My concern is that while the performance improvement might due to the auxiliary losses but it might not mean that the virtual trajectories will always be more sample-efficient as they somehow \"enrich the experience\". \"PlayVirtual\" title might suggest agent is virtually interacting with environment but that does not seem to be the case. One explanation might be that the additional constraints help in learning better representations by preventing overfitting to one objective.\n\n4. Do the auxiliary losses have to be applied iteratively?\nThis is not a big weakness but more of a curiosity from my side. From Table 4 the authors show the best performance is achieved for predicting K=9 steps into the future. Is it possible to then only apply  a one forward prediction that is K=9 steps in the future. They can concat the actions of next 9 steps and provide that as input to the forward model. If similar performance is achieved then it might save some training time by preventing the iterative process.\n\n5. Training curves showing how the auxiliary losses change as training proceeds.\n\n6. What is RDM in line 345? Is it supposed to BDM?\n\nJustification of Rating\n\nThis paper involved a lot of experiments across many different environments. The authors did a good job of explaining the method and validating various design decisions empirically. The paper's quality will improve if they add the missing details (asked in weakness section) and justify the use of the term VirtualPlay as it seems the policy is not really using these virtual trajectories directly. Even though their proposed approach improves performance, it is important to establish the exact cause why it does so? Is it better representations or are the \"virtual trajectories\" actually adding more simulated experience to the agent? Currently there is only evidence for the former in the paper.\n\nMissing citations\n1) Connection with the loss used in [25] \nDwibedi, D., Aytar, Y., Tompson, J., Sermanet, P., & Zisserman, A. (2019). Temporal cycle-consistency learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 1801-1810). ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes the authors have adequately addressed societal impact and limitations of their work", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "YqXrcv2CoLg", "reviewer": "Reviewer_1uRz", "summary": "This paper introduces PlayVirtual a new, self-supervised, methodology for use in reinforcement learning settings based on the idea of enforcing cycle-consistency when predicting latent environment representations with (learned) forward/backwards dynamics models. As PlayVirtual is primarily positioned as a means by which to improve data efficiency, the experimental results are targetted to the low-data regime and, following related prior work, report their results on Atari and DeepMind control suites of tasks. For these experiments, the proposed PlayVirtual methodology outperforms competing methods (on average across tasks within the two suites) and thereby obtains SOTA performance.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Post-rebuttal\n\nI appreciate the authors thorough response to several of my concerns and to the questions raised by the other reviewers. I am substantially more confident that the proposed PlayVirtual method's impact on learned representations is meaningful. In my view, the main remaining weakness of this paper is the lack of pedagogical insight as to why and how cycle consistency is improving the representations. Some work has been done in this direction in the rebuttal, I particularly appreciate the experiment with the perturbed actions, but I maintain that the takeaway message from the experiments is still primarily \"cycle consistency is good.\" Nevertheless, the authors responses have made me lean more towards acceptance than rejection and I have updated my rating to reflect this.\n____\n\n# Original review\nThis paper has several clear strengths:\n\n* The proposed PlayVirtual method is intuitive, not overly encumbered by hyperparameters, and is sufficiently generic that it is not difficult to imagine it being used for other reinforcement learning tasks.\n* The experiments provide a strong argument that PlayVirtual can result in a large benefit within the low data regime on commonly used benchmarks.\n* The idea of using simulated trajectories for representation learning instead of policy improvement is interesting.\n\nDespite these strengths I'm struggling to commit to acceptance. \n\nOverall:\n\nI would be more than happy to change my mind but my general impression is that this work feels quite incremental and struggles to meet the standard for novelty for NeurIPS. In particular: (a) this paper builds heavily on SPR and thus seems to have few if any modeling contributions, (b) the idea of using cycle consistency for representation learning has been exhaustively exploited in other domains, (c) experiments are comprehensive but not pedagogical, I learn little more than \"cycle-consistency is good\", and (d) there seems to be little consideration of more realistic/challenging tasks. It thus seems that the main novelty of this work is the third strength that I noted above: the use of simulated trajectories for representation learning. If this is the case I would expected to have some of the following questions answered:\n\n* What is the effect of action sampling on the learned representations? Is it better to sample trajectories from the learned policy, randomly, or from some \"teaching\" policy?\n* How are the encodings learned through cycle consistency different from those learned simply using SPR? What are the types of behaviors that your learned agents exhibit that SPR models don't?\n* How would you extend PlayVirtual to more complicated environments, especially PO-MDPs where we'd expect deterministic forward/backward prediction to break down?\n* How does the performance/capacity of the dynamics models impact the learned representations?\n\nI would expect to increase my rating if some portion of the above questions were answered well or if I could be convinced that the novelty was more substantial than I initially understood.\n\nMinor line-by-line comments:\n\nLine 22\n* I believe you mean \"data inefficient\" rather than \"data efficient\"?\n\n\nLine 23-28\n* It's a bit odd to motivate you work referencing complex settings where collecting data is hard and then apply your model in a space where we can easily collect huge amounts of data.\n \n\nLine 37\n* \"using 100k interaction data\" reads a bit weird. I'd rephrase as \"allowed 100k interactions with the environment\" or similar.\n \nLine 50-56\n* A priori, all of the losses you introduce have an easy optimal solution: predict z_t to be a non-zero constant vector. It's worth reiterating why this isn't an issue (as covered in the SPR paper).\n \n\nLine 159\n* \"has few opportunity\" -> \"has few opportunities\"\n \n\nLines 214-220 (Definition 1)\n* I'm not sure what this \"mathematical definition\" is bringing the paper, indeed it seems to raise more questions than it answers:\n  * What is the expectation over (formally)?\n  * Why bother with an expectation here at all? It seems like you're effectively just saying that d(z_t', z_t) should be identically 0 over the support.\n\n\nLines 224-231\n* I find this discussion to be inadequate. This is especially true for the comment regarding many-to-one transitions which I struggle to believe.\n \n\nLine 244\n* In DMControl, are environment and interaction steps equivalent?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "While the \"Potential Societal Impact\" section of this paper does feel like an afterthought (being the last section of the appendix) I do agree with the authors that this work has little potential for negative societal impact. I would like to seem more discussion of the limitations of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5.5"}, "extra_text": {}}, {"review_id": "IuqJY0R1gcC", "reviewer": "Reviewer_aagy", "summary": "RL algorithms are generally data-inefficient and would benefit from representation learning methods as augmented loss for better data efficiency. Building on top of SPR, this paper proposes using timewise cycle-consistency in the latent space to augment the training data. Specifically, the forward and backward latent dynamics model is trained using supervised learning from the ground truth. In addition, a cycle-consistency loss is imposed on states with randomly sampled actions for N steps. \n\nThe paper experimented on the Atari 100k and DMControl 100k benchmarks and showed SOTA performance in terms of Median HNS for Atari and Median score for DMControl. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While cycle consistency is not new, it was used previously in RL mainly as a way for Sim2real ([Rao et al. 2020](https://arxiv.org/abs/2006.09001), [Zhang et al. 2021](https://arxiv.org/abs/2012.09811)). The authors proposed a new method to incorporate cycle consistency as a regularizer in the latent representations of the states for RL. The results on Atari and DMControl benchmarks are promising and are worth further investigation. The proposed method is simple to implement and does not add much additional cost in terms of training time. \n\n\nWhile the strength of this simple yet effective incremental improvement to SPR is clear, I believe to increase the impact of this paper, the readers would benefit from a more detailed discussion of one or more of the following points:\n1. Good feature representations and data efficiency are related, but not entirely the same. The method proposed claims to help with both, but the experimental result only supports better data efficiency. A good representation can help with: a) OOD generalization and extrapolation b) human interpretability c) Faster learning speed. Is it possible to have experiments to support the claim that cycle consistency leads to better feature representation in RL (e.g. using the criteria in [Anand et al. 2019](https://arxiv.org/abs/1906.08226))? Alternatively you can claim that it only helps with data efficiency. \n2. In 4.2, following the format in SPR, can you also report the mean HNS as well as # of games with super human performance for your best agent here? Did your agent beat the baselines in all metrics or just the median? \n3. Have you tried non-random actions for the cycle consistency loss? \n4. Can you elaborate why you chose to stay in the latent space vs in the observation or action space? It’s quite an important design choice and I can see the pros and cons for both.\n\n\nOther suggestions to improve the readability of this paper\n1. In the paper, it is unclear whether the PlayVirtual predicts the raw state that the agents get from the env, or the latent := Encoder(state). From the implementation, it looks like the latter, but please update the rest of the document to distinguish the raw state/observation from the latent state. (e.g. l7-9, l44-47, etc.)\n2. From figure 1, it looks like the encoder takes 4 frames for $s_t$. Is that just the batch, or is that actually the frames $s_{t-3} \\sim s_t$?\n3. For DMControl, [Laskin et al. 2020](https://arxiv.org/abs/2004.14990) reports a performance on-par with PlayVirtual. So the claim in l14-15 is inaccurate.\n\n\n## Updates after rebuttal\n\nThe authors’ rebuttals made me believe more deeply that there is something interesting with the empirical results. It's good that they also added more about the limitations and future work. More follow up analysis and a better understanding of why it worked would make the paper much stronger.\n\nI’m happy to change my recommendation to accept in order to encourage future work and more analysis on this empirically strong finding, especially around whether it works in more environments and whether it is a general technique that works beyond SPR.\n\nI have changed my ratings accordingly.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "\n1. What is the weakness of your method? How can a user choose when to use it and when not to use it? What about potential future directions? \n2. How does environmental randomness come into play for cycle consistency? Does your method work equally well for deterministic and stochastic environments (either inherent to the env like Kung Fu Master or via sticky actions)?", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: After the author rebuttal and discussion, the reviewers have unanimously agreed to accept the paper. All three reviewers expressed gratitude for the improved clarification of the method and follow-up experiments and the overall empirical value of the presented experiments, but still expressed some reservations along the lines of "The paper would clearly be much stronger if it provided more insight as to why and in what way this cycle consistency loss improves the learned representations". For this reason, I recommend accepting this paper as a poster.

Decision: Accept (Poster)