Self-Supervised Structured Representations for Deep Reinforcement Learning
Authors: Hyesong Choi
Hunsang Lee
Wonil Song
Sangryul Jeon
Kwanghoon Sohn
Dongbo Min
Venue: ICLR 2022 Submitted (2022)
Keywords: Reinforcement Learning
Representation Learning
Optical Flow Estimation
Structured representation
Self-supervised Learning

Abstract: Recent reinforcement learning (RL) methods have found extracting high-level features from raw pixels with self-supervised learning to be effective in learning policies. However, these methods focus on learning global representations of images, and disregard local spatial structures present in the consecutively stacked frames. In this paper, we propose a novel approach that learns self-supervised structured representations ($\mathbf{S}^3$R) for effectively encoding such spatial structures in an unsupervised manner. Given the input frames, the structured latent volumes are first generated individually using an encoder, and they are used to capture the change in terms of spatial structures, i.e., flow maps among multiple frames. To be specific, the proposed method establishes flow vectors between two latent volumes via a supervision by the image reconstruction loss. This enables for providing plenty of local samples for training the encoder of deep RL. We further attempt to leverage the structured representations in the self-predictive representations (SPR) method that predicts future representations using the action-conditioned transition model. The proposed method imposes similarity constraints on the three latent volumes; warped query representations by estimated flows, predicted target representations from the transition model, and target representations of future state. Experimental results on complex tasks in Atari Games and DeepMind Control Suite demonstrate that the RL methods are significantly boosted by the proposed self-supervised learning of structured representations.
The code is available at https://sites.google.com/view/iclr2022-s3r.

Under review as a conference paper at ICLR 2022
SELF-SUPERVISED SPATIAL REPRESENTATIONS FOR DEEP REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Recent reinforcement learning (RL) methods have found extracting high-level features from raw pixels with self-supervised learning to be effective in learning policies. However, these methods focus on learning global representations images, and disregard local spatial structures present in the consecutively stacked frames. In this paper, we propose a novel approach that learns self-supervised spatial representations (S*R) for effectively encoding such spatial structures in an unsupervised manner. Given the input frames, the spatial latent volumes are first generated individually using encoder, and they used to capture the change in terms spatial structures, i.e., flow maps among multiple frames. To be specific, the proposed method establishes flow vectors between two latent volumes via a supervision by the image reconstruction loss.This This enables for providing plenty of local samples for training the encoder of deep RL. We further attempt to leverage the spatial representations in the self-predictive representations (SPR) method that predicts future representations using the sccion-conditioned transition model. The proposed method imposes similarity constraints on the three latent volumes; warped query representations by estimated flows, predicted target representations from the transition model, and target representations of future state. Experimental results on complex tasks in Atari Games and DeepMind Control Suite demonstrate that the RL methods are significantly boosted by the proposed self-supervised learning of spatial representations. The code is available at htttps://siiee googgle.com/ie wttpe/////wweeeeeee/222222-21/2
INTRODUCTION
Deep reinforcement learning (RL) has been an appealing tool for training agents to solve various tasks including complex control and video games (François-Lavet al., 2018). While most approaches have focused on training deep RL agent under the assumption that compact state representations are readily available, this assumption does not hold in the cases where raw visual observations (e.g. images) used inputs for training the deep RL agent. Without designing effective algorithm that uses model-based policy and value improvement operators (Schrittwieser et al. 2021)) without attempting
additional image augmentation (Laskin al. [0200a] Kostrikov et al. 2020), learning visual features from raw pixels only using a reward function might fail to learn good features in terms of the performance and sample efficiency.
To address this challenge, a number of deep RL approaches (Sermanet et al. 2018 Dwibedi et al.) 2018:
Anand et al. 2019 Laskin al. 2020b Mazoure al. 2020 Stooke et 2020 Schwarzer
flow
zg
flow based warping
2
Z
global pixel level similarity
transition model
2P+1
Figure 1: Abstract form of the S method: Warped representation 2 and predicted representation ZP are obtained from query encoder representation 2Í at frame by flow based warping and transition model, respectively. Both global-level and pixel-level similarity are then imposed with target encoder representation Z at future frame k + 1.

---

Under review as a conference paper at ICLR 2022
2021) leverage the recent advance of self-supervised learning which effectively extracts high- level features from raw pixels in an unsupervised fashion. In Laskin et al. (2020b); Stooke et al. (2020), they propose train the convolutional encoder for pairs of images using contrastive loss (van den Oord et al. 2018). For training the RI. agent, given a query and a set of keys consisting of positive and negative samples, they minimize the contrastive loss such that the query matches with the positive sample more than any of the negative samples (Laskin et al., 22200b] Stooke et al. (2220). While the parameters of the query encoder are updated through paooppropngttion using the contrastive loss (van den Oord et al. 2018), the parameters of the key encoder are computed with an exponential moving average (EMA) of the query encoder parameters. The output representations of the query encoder are passed to the RL algorithm for training the agent. [Schwarzer et al. (2021) proposes the self- eeffpredictive representations (SPR) method that trains the RL agent by leveraging the query and positíve sample only, following Grill et al. (2020) that achieves state-of-the-arts performance without the negative samples in the self-supervised learning. Especially, it extends a model-free RL agent by adding the transition model that predicts its own latent representation multiple steps into the future (Schwarzer et al. 2021). The predicted future representations and the representations for future states computed using target encoder serve the query and positive sample, respectively. These approaches have shown compelling performance and high sample efficiency on the complex control tasks when compared existing image-based RL approaches (Kaiser al. 2019 van Hasselt
et al.
et al. 2019
Kielak 2020).
While these approaches can effectively encode the global representations of images with the self- supervised representation learning, there has been no attention on the local spatial structures present in the consecutively stacked images. Our key observation that spatial deformation, i.e., the change in terms of the spatial structures across the consecutive frames, can provide plenty of local samples for training the RL agent. two-frame flow estimation (Dosovitskly et al. 2015 1lg et al. 2017] Jonschkowski et al., 2020), which has been widely used for video processing and recognition in computer vision, can be an appropriate tool modeling the spatial deformation. propose novel approach, termed self-super spatial representations RS*.). that learns spatial representations for effectively encoding the spatial structures in self-supervised fashion. Note that use the term 'spatial representations' to indicate set of feature maps extracted over frames for inferring locally-varying flow maps. The spatial representations generated from an encoder are used to predict the flow maps among the input frames by minimizing self-supervised manner. A flow-based warping is then applied to generate future representations. We further extend our framework by leveraging SPR method (Schwarzer et al. 202T). As depicted in Figure [ we impose similarity constraints on the three representations, query representations warped by the estimated flow maps, self celf-predicted target representations from the transition model (Schwarzer et al. 2021), and target representations of future state.
this work, we
image reconstruction loss in
Note that Shang et al. 202T) encodes the temporal information by concatenating latent differences of input frames, but the simple substraction operation has certain limitations in effectively capturing the spatial deformation. Contrary to Amiranashvilii et al. (2018) in which the flow map is directly fed into RL algorithms with stack of images, our method leverages the flow maps to effectively encode the spatial representations of the images and warp spatial representations at the future state.
Our contribution is summarized as follows.
Our method learns the spatial representations using the self-supervised flow model for encoding local spatial structures from the consecutive frames used in RL algorithms.
We propose to impose the similarity constraint on the spatial representations as well as the global representations, providing plenty of supervision for training the encoder of deep RL.
We compute the future representations through the flow-based warping for imposing the similarity constraint with target representations.
2
RELATED WORK
Self-supervised Representation Learning: The self-supervised representation learning aims to learn general features from large-scale unlabeled images videos without expensive data annota- tions. The contrastive methods have achieved ofte-of--hh-ar performance in the sulf-super representation learning. The contrastive learning aims to bring positive samples closer while sep-
2

---

Under review as a conference paper at ICLR 2022
arating negative samples from each other (Hadsell et al., 2006). Wu et al. (2018) formulate the contrastive learning as a non-pann con-aammetric classification problem at the instance level. and propose to learn visual features with the memory bank and noise contrastive estimation (NCE) (Gutmann & Hyvärinen 2010: Mnih & Kavukcuoglu (013). The method in van den Oord et al. (2018) proposes a probabilistic contrastive loss, called InfoNCE, for inducing representations by leveraging positive and negative samples. The InfoNCE loss has widely been adopted in Chen et al. (2020); He et al.
(2020): Hénaff et al. (2020); Tian et al. (2020). Chen et al. (2020) present simple framework for
contrastive self-supervised learning without specialized architecture (Bachman et al. 2019 Hénaff
ct al.) (020) or memory bank (Wu et al., (018), but it requires large batch size for using enough
negative samples when computing the InfoNCE loss (van den Oord et al.) 2018). He et al. (2020) propose to build dynamic dictionary with a queue to avoid the use of large batches when collecting
negative samples, and also uses the moving averaged (momentum) encoder for target data (positive and negative samples of query data). Grill et al. (2020) use the momentum encoder to produce representations of the targets as means of stabilizing the bootstrap step. This enables for learning the representations with only positive samples, which are generated by data augmentation, for a given query without the need to carefully set up negative samples. The method in Chen & He (2020) further extends this idea by using only stop-gradient operation without using the momentum update. While these approaches focuses on learning global representations of a single image, our method proposes to learn spatial representations for effectively encoding the spatial structures (i.e., flow map) in the consecutive images.
Sell-supervised Representation Learning in Deep RL: Representation learning is crucial for RL algorithms to learn policies with high-dimensional visual observations. The future prediction condi- tioned on the past observations and actions serves as auxiliary tasks to improve the sample efficiency of model free RL algorithms. [Gelada et al. (2019) train a transition model to predict representations of future states together with reward prediction loss. Guo et al. (2020) present Predictions of Boot- strapped Latents (PBL) that builds on multi-step predictive representations of future observations for deep RL. The method in SnShwarre et al. (202I) propose Self- Predictrdet Representations (SPR) based transition model that can predict future representations computed using
on an retion-conditionas target (momentum) encoder. Our method attempts predict the future representations through a warping operation using flow maps computed from the spatial representations of the consecutive frames.
Contrastive learning has been used to extract desired latent representations of visual observations used in the RL algorithms. For training robot agents, [Sermanet agents,/Sermanet et al. (2018) present the time-contrastive networks (TCN) that train viewpoint-invarian representations using a metric learning. This work was extended in Dwibedi et al. (2018) by embedding multiple frames each timestep for learning task-agnostic representations such as position and velocity attributes in continuous control tasks. In Anand et al. (2019), the representations for RL algorithms are learned by maximizing mutual information (Hjelm et |l.] 2019)) across spatially and temporally distinct features of an encoder of visual observations, Schwarzer et al. (2021) leverage the self-supervised learning (Grill et al. 2020) for imposing the similarity constraint between self-predictive and target representations, Laskin et al. (2020b) introduce Contrastive Unsupervised representations for Reinforcement Learning (CURL) that learns the representations from visual inputs using the InfoNCE loss (van den Oord et al. 2018).
Stooke et al. 2020) present Augmented Temporal Contrast (ATC) using image augmentations and InfoNCH loss (van den Oord et al.) 2018) for representation learning, and decouples it from policy learning. From a different perspective. Hansen et al. (2020) propose to adapt the policy network through self-supervised representation learning in unseen environments where it is difficult predict changed rewards. Our method imposes the similarity constraint on the spatial representations as well as the global representations, thus providing plenty of supervision for training the encoder of deep RL.
Visual Correspondence Learning: Visual correspondence estimation is a long-standing research in the computer vision community. It aims to establish a pair of corresponding pixels between two (or more) views taken under different locations (stereo matching) or timestep (optical flow). Recent
methods for stereo matching (Zbontar & LeCun 2016; Chang & Chen 2018 Zhang et al., 2019) and optical flow estimation (Dosovitskiy et al. 2015 Ilg et al. 2017 et al. 2018) have been advanced largely thanks to the expressive power of deep networks. Though both approaches share a similar objective of finding corresponding pixels across views, the optical flow is known to be effective for encoding temporal motion trajectories, while the stereo matching tailored to predicting 3D depth
3

---

Under review as a conference paper at ICLR 2022
query encoder Eq
F F(4
Z
flow based warping
RL MLP head
F(2,, 441)
transition model H
F Fl (a
target encoder E,
stop-arad
Z
Z
1 1+4r+1
4641
Sim(Z7i T2i1 1:
Sim( Teas (21))
0p+1
k++1
Me M++1
F(Za:
Sim Zu. Zn)
Zu
query projection P.
query prediction Q
Za
recon loss C, in (2)
Sa
stop-grad
76
correlation volume Va,
Z6 projection P.
cosine similarity C, in (6)
Figure 2: Overall framework of the S method: Multiple representations generated by the query and target encoders are used to infer set of flow maps from F. The warped representation - is produced using an inverse warping with the set of flow maps. The transition model with action dg also predicts the future representation Zh41. similar to Schwarzer et al. (2021). The target encoder and projection heads are updated using the stop-gradient operation
Chen & He (2020). ( indicates a channel-wise concatenation and 1 x convolution. The encoder representations Z are used inputs in the RL algorithm. In work, Rainbow DQN (van Hasselt et al., 2019) (M 3)
and SAC (Haarnoja et al. 2018a) (M = 2) are used as RL algorithms.
map in the scene. The commonly used architecture for two-frame optical flow estimation involves the feature map extraction of two frames, correlation volume computation, series of convolutions for refinement, and flow regression. While satte-of-theaate flow estimation methods require using ground truth flow maps
explicit supervision (Dosovitskiy al. 2015 1lg et al. 2017 Sun et al. 2018), some unsupervised learning approaches have attempted to infer flow maps with an image reconstruction loss for imposing the constraint that corresponding pixels should have similar intensities (Ren et al. 2017 Meister et al. 2018 Wang et al.) 2018) Jonschkowski et al. 2020). In our work, we present the self-supervised flow network that learns spatial representations from the consecutive frames used in the RL algorithms.
METHOD
We consider the Markov Decision Process (MDP) setting where agent interacts with environments
sequence of observations, actions, and rewards. We denote og, 4g, and r as the observation, the action of the agent, and the reward received at timestep k. Since our method is general framework
that leverages the representation learning for training the RL agent, it can be combined with any RL algorithm. Following the state-of-the-arts RL approaches (Laskin et al. 2020b; Stooke al. 2020 Schwarzer et al.] (021) using the self-supervised learning, we adopt the Soft Actor Critic (SAC) method (Haarnoja et al. 20) for continuous control task in DeepMind Control Suite benchmark,
and Rainbow DQN (van Hasselt et al. 2019) for discrete control task in Atari Games. The proposed self-supervised spatial representations (S*R) used auxiliary task for training RL. agents.
4

---

Under review as a conference paper at ICLR 2022
3.1 SELF-SUPERVISED FLOW MODEL FOR SPATIAL REPRESENTATION
We start with how to generate the spatial representations for capturing spatial deformations from the consecutively stacked frames in self-supervised manner. An instance used by the model-free off-policy RL algorithms (Haarnoja et al. 2018a; van Hasselt et al. 2019) is a stack of images, not single image. Given an input raw observation O% [[k= .... ++ M where Ix is an image at timestep k. the spatial encoder features ex { .... 2R++M } are first generated by applying an encoder individually to each of the input observations og- Note that E wxx is a 3-D volume with spatial resolution h x w and a feature dimension d. We apply query encoder and target encoder to og and O respectively. and denote the output of the query encoder e4 as 29, and the output of the target encoder et as z While the existing methods (Sermanet et al. 2018 Dwibedi et al. 2018 Laskin et al.) 2020b Schwarzer et al., 2021) feeds the stacked frames to the encoder at once, which can be viewed as early fusion (Karpathy et al. 2014), our method generates the set of the spatial latent representations individually with the encoder. Later, they are fused using 1 x 1 convolutional layer in manner similar to late fusion (Simonyan & Zisserman 2014) (see Figure 2 A similar
strategy was also adopted in Shang et al. (022)) for encoding the temporal information when training the RL agent.
The set of spatial representations is used to predict the spatial deformations, i.e., flow maps between two consecutive frames via a supervision by the image reconstruction loss (Godard et al. 2017 2019)
Wang et al. 2020). We compute a correlation volume Va.b E R using a dot product between two latent representations za and 2b (Dosovitskiy et al. 2015) as follows:
V Va.(p..)) aEi -E(-T-1x(---)
Σ
< (p o), 20 +0)
(1)
where and 9 represent 2D feature position in 2n and 25, and F indicates the kernal size for computing correlation, 2F 1. Computing the patch similarity in (T) for all combinations of p and q (totally, 1.-ww" times) yields huge amount of computations. Thus, the maximum displacement for computing
the patch similarity is limited for q E (p) where N(p) represents square window of size 2
centered at p. following visual correspondence literature (Dosovitskiy et al. 2015 1lg et al. 2017 Sun al. 2018).
The correlation volume is fed into series of convolutions followed by the refinement module, producing flow map fa-ob E Rhxxxx from Ia to IT. We follow the architecture of 'FlowNetCorr' in Dosovitskiy et al. (2015) with some modifications in the spatial resolution, encoder feature dimension and search range r in order to keep the model complexity low. Note that while 'FlowNetCorr' is trained with the supervision by ground truth flow maps, our method relies on the image reconstruction loss for self-supervised learning such that
C. (Ja-v)) Σ (a(p)) - 1s(p + fa-+b)] + Fregs F
where I (p) indicates an intensity the pixel corresponding to 2D feature position p. For computing the loss Cy., we resize Ia and In to the size of the spatial representations, h x w. We additionally use the Charbonnier regularization loss Grey (Barron 2019) for producing spatially smooth flow maps. In Figure 2 we denote F(za, = Sa-ob b as the self-supervised flow estimation network including the correlation volume computation, the series of convolutions, and the refinement module.
3.2 REPRESENTATION LEARNING WITH WARPING AND SELF-
Figure 2 iillustrates the overall architecture of the proposed S approach. Following the prior work on the self-supervised learning (He et al. 2020; Grill al., 2020 Chen He| 2020), we use the query encoder E, with the parameters 6g and the target encoder E, with the parameters 6, for encoding the query observation Øg and the target observation 0 While the parameters 0, of the query encoder are updated through baok-prop,g the parameters Og of the target encoder are updated with the query encoder parameters O, using a stop-gradient operation (Chen & He 2020) as O₂ + 00g
we first compute set of M + external flow maps (Se74+1- 1 10 = 0, ..., m) with the self-supervised flow network F such that
Flow Learning and Warping: By minimizing 2
sk41+1 1+4 F( 2bte
for 0,.... M.
(3)
5

---

Under review as a conference paper at ICLR 2022
Note that the external flow map is predicted from the target feature 2 to the query feature into the future state via the inverse 244. Then, we warp the query features ex ( ........ -..,,%+4
warping (Jaderberg et al.] 2015) using M + external flow maps. The warped query features {2k+ 1:.,1..--- 1..1::...**..... -.-,,+++-111 1 are then fused using x convolution, producing the warped query representa- tion 2x4 at the timestep k + 1 (see Figure?2.
We can also predict internal flow maps within the query features ei = 2 as fo F(22. =.]). Various combinations of a and b are possible for computing the internal flow maps, and fint
we choose to compute single flow map Jk-K+M We found that this is an appropriate choice in
terms of computational efficiency and performance as the external flow maps are already used to impose the structural similarity constraint between multiple frames, and is effective in dealing with the case where the external flow between two consecutive frames is relatively small. More details are presented in the Appendix. The loss function Cy for computing the internal and external flow maps is given as
M
S = C.(JR-
+
C, (J
(4)
Representation Learning with Global-level and Pixel-level Similarities: To measure the global- level similarity between the warped query representation 2k4+1 and the target representation Z two projection
which is the fusion of target encoder features d, = {2f 1 1. 2 M++ }.
heads and one predictor in a manner similar to Grill et al. (2020); Chen & He (2020). We project the two representations 2 and Zy41 into a smaller latent space by passing them into the query projection head P. with parameters É, and the target projection head P. with parameters &r. and also apply an additional query prediction head Qg to the query projection. The target projection head parameters & are updated using the query projection head parameters &g with the stop-gradd -oodggrddeett operation as the target encoder, i.e., & f E.. The prediction loss is computed using the cosine similarity between the warped query representation vk+1 representation wi+1 Pr((Z which is defined as
Q₂ (Po (7 (++)) 1) and the observed target
1kt v > lI lla lalllt++11
Lo 1.00))
(5)
In Figure 2 denote the module consisting of the query projection and prediction heads and the target projection heads as 'Sim(Z,, Zz) where Za Zu can be the warped representation from query and spatial representations from target network, respectively.
We further extend our method by leveraging a self-prediction module conditioned on an action. The action-condditoned convolutional transition model Schwarzer et | (2021) is applied in our framework with some modifications. We generate the query representation 2% by applying 1 x 1 convolution to the query features {=ā, * 3) and then feed it into the convolutional transition model H. Note that we use only a single next prediction 2
H( Zg. ag) of the transition from the query representation Zg, unlike Schwarzer et al. (2021) | that recursively generates a sequence of L predictions Z 1: with - H(Zh (k++1) for / = 0, .... L and 2 = Z We found that since the spatial representation learning using the flow estimation and warping provide a sufficient amount of supervision, the single self prediccreiioonnn is enough to train the query encoder. The self ceef-prediition 2 is fed into the query projection head P, and the query prediction head Q2 such that v = Q2 (P, (Py (2) The prediction loss is also computed using the cosine similarity loss L. L. 1k+1)
The loss function Leim for measuring the similarity between the three representations yh 1 Dh+ and t is summarized as
Lxim La( v(+1) +
+ AL,
(Zh+1: A Z Z
(6)
where vi+1 (P, D = (P, (+1)))) )), -
= P. P:( and À is hyperparam- eter of loss functions. We also include L1 loss L, (Z 41+ on the original spatial latent space to impose the pixel-level similarity. Note that L, loss between Z and Z is not used, as the external flow loss (feat) implicitly considers the similarity between the two.

---

Under review as a conference paper at ICLR 2022
Algorithm 1: Self-Supervised Spatial (S²R) Representations
E,, E.: Query encoder. Target encoder F, H: Flow model. Transition model
Pa. Q2. Pr: Query projection head, Query prediction, Target projection head Initialize replay buffer and parameters of Eg. E,, pa. Qq. Pt, F. H;
while Training do
Spatial Representation with Flow Learning
O.... Generate + F8-14+1 with Eg. Eg for
Learn external flow and internal flow with
Representation Learning with Warping and Self-Prediction
Warp set of query features 2 with external flow f Jkite1-ak4/ for i 0, ...,, M.
Generate warped query representation _ by fusing a set of warped query features.
Generate query representation z by fusing set of query features 2/+4 for 0,... M.
Generate predicted target representation Z from Zz using transition model H.
Generate target representation Z by fusing of target features 2x+2+1 for i 0, M. Z goes into RL MLP head.
Compute global and local similarity of 6 from Zi+1, Ze+1. and Zi+1 using 'Sim' in Figure Optimize parameters of Eg. Py, Qg. F. by minimizing Liorat = Lio + acCssi + BL II Update parameters of E, and P. with E, and P,;
end
Finally, the query representation Z hxxxx is fed into the deep RL algorithm, e.g., 64 for DQN (van Hasselt et al.) 2019) and = 32 for SAC (Haarnoja et al. 2018a).
Loss functions:
The final loss function is summarized
Liotal = f. flow + OcLssm + BL.. (Z%),
where LRI indicates the loss of the RL algorithms which use 2% an input. and hyper-parameters that balance the loss functions. We summarize the overall method in Algorithm
3.3 IMPLEMENTATION DETAILS
Self-supervised Flow Model: The input image L, of 84 x 84 for Atari Games and DeepMind Control (DMControl) Suites. The query and target encoders generates 2 2441 7x7664
k...., 3) for Atari Games and 2 2|+1 E 322x32x33 (i k. ..., k + 2) for DMControl Suites, respectively. The search window for computing the correlation volume V is 6 x for Atari games and DMControl Suites. The correlation volume goes through decoder is then applied to provide a dense flow map. The decoder includes three un-convolutional layers, consisting un-pooling and convolution, and the coarser flow maps and encoder feature maps are concatenated into each un-comvolutionall layer (Dosovitskiy et al. 2015).
convolution layers 3 times. The
Action Conditioned Transition Model: This basically follows the structure of Schwarzer al. (2021), but in order to compute the structural similarity with the features of each frame, we set the input and output channel size 256, which was originally set to 64 /Schwarzrr et al. (202T). The transition model includes two convolutional layers interweaved with ReL.U and batch normalization (Toffe & Szegedy 2015), with the current representations Z and the action ag of one-hot vector taken to each location being fed to the first convolutional layer.
The query and target projection heads, P, and Pt, are implemented as the multi-layer perceptron (MLP). For the query prediction head Q reuse the first linear layer of the RL head similar to Schwarzer et al. (2021). We used = 0.1 in (6) and a = 5, B = in 7 to balance the weight of the losses. More details presented in the Appendix.
Other Details:
4
EXPERIMENTAL RESULTS
4.1 EVALUATION ON ATART GAMES
To compare the performance of the proposed method with the we chose Atari 2600 Games introduced in Kaiser et al. (001)) van Hasselt et al. (2019) where only 100K environment

---

Under review as a conference paper at ICLR 2022
Table 1: Quantitative evaluation with state-of-the-arts the 26 Atari games (Kaiser et al. after 100K time steps using 10 random seeds: Numbers in bold represent 1st ranking. S achieves the best performance on 13 out of 26 environments, We compared results with SimPLe (Kaiser et al. Rainbow (DER) (van Hasselt et al. 2019), OverTrained Rainbow (OTRainbow) (Kielak 2020). CURL LLskin et 2020b). DrQ (Kostrikov et al. 2020), and SPR (Schwarzer et
2019
2019).
2021
Gome
Human
Random
Rainbow
SimPLe
DER
OTRainbow
CURL
DrQ
S'R
Alien
7127.7
227.8
318.7
616.9
739.9
824.7
558.2
771.2
801.5
1030.1
Amidar
1719.5
5.8
32.5
88.0
188.6
82.8
142.1
102.8
176.3
114.3
Assault
742.0
222.4
231.0
527.2
431.2
351.9
600.6
452.4
571.0
708.3
Asterix
8503.3
210.0
243.6
1128.3
470.8
628.5
734.5
603.5
977.8
959.3
Bank Heist
753.1
14.2
15.55
34.2
51.0
182.1
131.6
168.9
380,9
95.8
BattleZone
37187.5
2360.0
2360.0
$184.4
10124.6
4060.6
14870.0
12954.0
16651.0
16688.0
Boxing
-24.8
1.2
Breakout
30.5
1.7
1.2
16.4
1.9
9.8
4.9
16.1
17.1
17.5
ChopperCommand
7387.8
811.0
120.0
1246.9
861.8
1033.3
1058.5
780.3
974.8
1251.2
Crazy Clibber
35829.4
10780.5
2254.5
62583,6
16185.3
21327.8
12146.5
20516.5
42923.6
42544.0
Demon Attack
1971.0
152.1
163.6
208.
508.0
711.8
817.6
1113.4
$45.2
884.0
Freeway
29.6
0,0
0.0
20.3
27.9
25.0
26.7
9.8
24.4
24.8
Frustbite
4334.7
65.2
60.2
254.7
866.8
231.6
1181.3
331.1
1821.5
776.9
Gopher
257.6
431.2
349.5
778.0
669.3
636.3
920.3
Hero
30826.4
1027.0
487.0
2656.6
6857.0
6458.8
6279.3
3736.3
7019.2
3977,3
Jamesbond
302.8
29.0
47.4
125,3
301.6
112.3
471.0
236.0
365.4
471.4
Kangaroo
3035,0
52.0
0,0
323,1
779.3
605.4
872.5
940.66
3276.4
1580,0
Krull
2665.5
1598.0
146R.0
4539.9
2851.5
3277.9
4229.6
4018.1
3688.9
4958.3
Kung Fa Master
22736.3
258.5
0.0
17257.2
14346.1
5722.2
14307.8
9111.0
13192.7
17759.5
Ms Pacman
6951.6
307.3
67.0
1480.0
1204,1
941.9
1465.5
960.5
1313.2
1597.3
Pong
-20.7
-20.6
-19.3
1.3
-16.5
-8.5
-8.2
Private Eye
69571.3
24.9
0.0
58.3
97.8
100.0
218.4
-13.6
124.0
158.0
Qbert
13455.0
163.9
123.46
1288.8
1152.9
509.3
1042.4
854.4
669.1
1290.3
Road Runner
7845.0
11.5
1588 46
5640.6
9600.0
2696.7
$661.0
8895,1
14220.5
3175.7
Seaquest
42054.7
68.4
131.69
683.3
354.1
286.9
384.5
301.2
583.1
734.9
Up Down
11693.2
533.4
504.6
3350.3
2877.4
2847.6
2955.2
3180.8
28138.5
4263.8
Table 2: Quantitative evaluation of mean and standard deviation with state-of-the-an on the DM- Control suite (Tassa et al.] (018) after 100K time steps and 500K time steps using 10 random seeds. Numbers in bold represent 1s ranking, and S achieves the best performance on 4 out of environments for 500K time steps. We compared results with state-based SAC and pixel-based
SAC (Haarnoja et al. 2018b). SAC+AE (Yarats et al. 2019), Dreamer (Hafner al. 2019a). PlaNet
(Hafner et al., 2019b). CURL (Laskin et al. (0206), RAD (Laskin et al. 2020a)). and
DrQ (Kostrikoy
al. (2020).
100K stop scores
State
PixeISAC
SAC+AE
Dreamer
PlaNet
CURL
RAD
DrQ
Finger, Spin
811
1791166
740-164
340
136-1216
767.1:66
856.173
901
880
Campole, Swingup
835:222
419:1-00
311:111
326:227
297
5824:14
828:127
7594-92
841:147
Reacher, Easy
746::2
145::30
274::14
314=1155
20:150
5384-233
826::-199
601
621=2022
Cheetab, Run
616:::88
197::15
267:1:24
137
138:888
29911-48
447:188
344:1:7
251:::34
Walker, Walk
881
42.-12
277:112
504
612:1164
Bull Cup, Catch
746:-9
3124163
391::82
246:1174
0::00
7691-43
840:::179
913:5:33
922:1-00
500K stop Scores
State SAC
Pixel SAC
SAC+AE
Dreamer
Planct
CURL
RAD
DrQ
Ours
Finger, Spin
923::21
884
796-1
947:11011
Carppole, Swingup
848::15
419-4-40
735-163
762-227
475=71
841
863-99
868-110
8801:34
Reacher, Easy
923+:24
145::30
6274:58
793=164
210::390
9291:44
955:1711
9424771
932-141
Cheetab, Run
795::330
197:115
5501:33
550
305
5181:28
728-171
660-19
501:163
Walker, Walk
948-154
42-12
8471148
897:1499
351
9024:43
918:116
921
930-775
Cup. 974::33
312:163
794:1:88
879.1.7
974:112
96349
steps. corresponding to two hours of gameplay experiences, are available for training data. This sample-efficient setup, which uses much less environment steps than the standard setup of 50,000K environment steps, has been adopted for evaluating the performance of recent sample-efficient deep
RL algorithms (Kaiser et al. 2019 van Hasselt et al. 2019 Kielak 2020: Laskin et al. 2020b
Table 3: To study the impact of several losses, measured the average performance over 10 random seeds according to the combinations of losses on DMControl Suite (Tassa et al. 2018) with 500K time steps. Refer section 4.3 for 'F', 'F+W', 'P'. and '+P''
500K step scores
F
F+W
P
F4P
F+W+P $²R)
Finger, Spin
729=110
757+100
711166
768-112
834
Cartpole, Swingup
819-38
876++19
753+-15
868++27
880-34
Reacher, Easy
857-45
901+299
904+58
922-+31
932--41
Cheetah, Run
366-63
411+:53
482+-87
392--69
448
Walker, Walk
770-87
8791167
8221553
876111
9141330
Ball Cup. Catch
849=42
953:126
848-107
951120
962-114
8

---

Under review as a conference paper at ICLR 2022
Schwarzer al. 2021> We compared results with various RL algorithms including SimPLe (Kaiser et al. 2019) which learns a pixel-level transition model for Atari, Eata-Efficient Rainbow (DER) (van Hasselt et al., 2019) which modifies the Rainbow hyperparameters for improving the sample efficiency, Ra'nnbboww (Kielak) 2020) which is an over-trained version of the Rainbow for the sample efficiency, CURL (Laskin al.] 2020b) which proposes the use of image augmentation with the contrastive loss (van den Oord et al., 2018) for self-supervised representation learning, DrQ (Kostrikov et al. (020) which uses the modest image augmentation to improve the sample efficiency, and SPR (Schwarzer et al. (02I) which trains an agent to predict its own latent state representations into the future. Following the experimental setup on the mentinned approaches, we evaluated on 26 environments of Atari 2600 games by measuring the average return after 100K interaction steps. As shown Table | the proposed method RS*R) achieved the best performance on 13 out of 26 environments. We trained our method with 10 random seeds, similar to other methods.
4.2 EVALUATION ON DMCONTROL SUITE
Various approaches including ours have been benchmarked on the DMControl Suite where the agent operates from pixels to evaluate challenging visual continuous control tasks (Tassa et al. compared our results with State-SAC which supposes that the agent has access to level state
2018). We
based features, Pixel-SAC (Haarnoja et al./ 2018b) which directly operates from pixels, SAC+AE
(aarats et al., 2019) which uses a joint learning of SAC with 3- VAE (Higgins et al. 2017). VAE (Kingma & Welling 2013). and regularized autoencoder (Vincent al. 2008). Dreamer (Hatner ett al., (019a)) and PlaNet (Hafner et al., (019b) which learn latent space world model, CURL (Laskin et al. 2020b) which uses image augmentation with the contrastive loss (van den Oord et al.) 2018), RAD Laskin et al. (020a)) and DrQ (Kostrikov et al. 2020) which demonstrate that data augmentation can greatly improve the performance of model-free RL. algorithms and achieve state- of-the-art performance on DMControl Suite. Table demonstrates that the self-supervised spatial representations of S achieved best performance on out environments for 500K time steps including Cartpole Swingup, Reacher Easy, Walker Walk and Ball in Cup Catch. We trained our method with 10 random seeds, and the results with random seeds are provided in the Appendix.
4.3 ABLATION STUDY
Table 3 measured the average performance over 10 random seeds according to the combinations of several losses on DMControl Suite (Tassa et al.) 2018) with 500K time steps.
*F" using only the flow loss in (4)
'+W'' using flow loss and similarity loss with 'W' araped query and target representations in (6)
'P' using prediction loss with self- eeredicered and target representations in (6)
'F+P' using the flow loss, LI loss, and the prediction loss in (6).
The network trained with only 'F' produces worse performance compared to 'F+W', 'F+P' and "F+W+P', but still produces comparable performance to state-of state-of-the-art thethrtss, implying that even without the warping and transition model, simply guiding the encoder to extract features for the flow prediction helps the RI. agent to perform well enough. The performance of 'F+W' and 'F+P' is similar, but 'F+W' has slightly better performance on step scores with smaller standard deviations. This implies that the warped query representation to the future state using the estimated flow is capable of providing as useful supervision as the self- seef-predicted representation that uses an eetion-conditioned transition model to predict future representations. The performance was further boosted, when using 'F+W+P' altogether (= R2R). To measure only the impact each loss, data augmentation was not performed. Note that 'P' is similar to SPR method (Schwarzer al. 202T) using only one prediction.
5 CONCLUSION
We have presented the self-supervised spatial representations, termed S²R to encode local spatial structures in an unsupervised manner. The flow maps inferred by the proposed method offer plenty of supervision for learning the spatial representations, and also compute warped predictions at future frame. R achieves state-of-the of-te- art performance on Atari benchmark with 100K steps and DMControl Suites with 100K/500K steps. We have shown the importance of learning the spatial representations in improving the performance and sample-efficiency of image-based RL algorithms. We hope this can facilitate future works various aspects for RL. based on self-supervised learning.

---

Under review as a conference paper at ICLR 2022
6 REPRODUCIBILITY STATEMENT
For the reproducibility of the proposed method, we opened our code ptcttps//sitee google com/view htt.s//wwcll222--- s3r as stated in the Abstract. We also provide the full hyperparameters in Section A of Appendix, others can use the code with the same settings we experimented with.
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Mare G Bellemare. Deep reinforcement learning at the edge of the statistical precipice. In Thirty-Fifth Conference Neural Information Processing Systems, 2021.
Artemij Amiranashvili, Alexey Dosovitskiy, Vladlen Koltun, and Thomas Brox. Motion perception in reinforcement learning with dynamic objects. In Annual Conference on Robot Learning (CoRL), pp. 168- 2018.
Ankesh Anand, Evan Racah, Sherjil Ozair, Yoshua Bengio, Marc-Alexandre Côté, and R. Devon Hjelm. Unsupervised state representation learning in atari. In Advances in Neural Information Processing Systems (NeurIPS), pp. 8766-8779, 2019.
Philip Bachman, R. Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In Advances in Neural Information Processing Systems (NeurIPS), pp. 15509-15519, 2019.
Jonathan T. Barron. A general and adaptive robust loss function. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4331-43333 2019.
Jia-Ren Chang and Yong-Sheng Chen. Pyramid stereo matching network. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 5410-5418, 2018.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML), pp. 1597-1607, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. CoRR, abs/2011.10566, 2020.
Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Häusser, Caner Hazirbas, Vladimir Golkov, Patrick van der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. In IEEE International Conference on Computer Vision (ICCV), pp. 2758-2766, 2015.
Debidatta Dwibedi, Jonathan Tompson, Corey Lynch, and Pierre Sermanet. Learning actionable representations from visual observations. In IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 1577-1584, 2018.
Vincent François- Lavet, Peter Henderson, Riashat Islam, Marc G. Bellemare, and Joelle Pineau. An introduction to deep reinforcement learning. Found. Trends Mach. Learn. 11(3-4):219-354, 2018.
Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Mare G. Bellemare. Deepmdp: Learning continuous latent space models for representation learning. In International Conference on Machine Learning (ICML), pp. 2170-2179, 2019.
Clément Godard, Oisin Mac Aodha, and Gabriel Brostow. Unsupervised monocular depth estimation with left-right consistency. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 270-279, 2017.
Clément Godard, Oisin Mae Aodha, Michael Firman, and Gabriel J Brostow. Digging into self- supervised monocular depth estimation. In Proceedings of the IEEE international conference on computer vision, pp. 3828-3838. 2019.
10

---

Under review as a conference paper at ICLR 2022
Jean- Cean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your latent A new approach to self-supervised learning. In Advances in Neural Information Processing Systems (NeurlPS), 2020.
Zhaohan Daniel Guo, Bernardo Ávila Pires, Bilal Piot, Jean-Bastient Grill, Florent Altché, Rémi Munos, and Mohammad Gheshlaghi Azar. Bootstrap catent-predictive representations for multitask reinforcement learning. In International Conference Machine Learning (ICML), pp- 3875-3886, 2020.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 297-304, 2010.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. International Conference on Machine Learning (ICML), pp. 1856-1865, 2018a. 8a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv: 118.../2...0... 2018b.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 1735- 1742, 2006.
Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint 1rXiv:1912.000 2019a.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, pp. 2555-2566 PMLR, 2019b.
Nicklas Hansen, Yu Sun, Pieter Abbeel, Alexei A. Efros, Lerrel Pinto, and Xiaolong Wang. Self- supervised policy adaptation during deployment. CoRR, 0t//2007.04309 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9726-9735, 2020.
Olivier J. Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, S. M. Ali Eslami, and Aäron van den Oord. Data-efficient image recognition with contrastive predictive coding, pp- 4182-4192, 2020.
Irina Higgins, Arka Pal, Andrei Rusu, Loic Matthey, Christopher Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. Darla: Improving zero-shot transfer in reinforcement learning. In International Conference on Machine Learning, pp. 1490-1490. PMLR, 2017.
R. Devon Hjelm, Alex Fedorov, Samuel Lavoie Marchildnn, Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations (ICLR), 2019.
Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox Flownet 2.0: Evolution of optical flow estimation with deep networks. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 1647-1655, 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning (ICML), pp- 448-456, 2015.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. Spatial transformer networks. In Advances in Neural Information Processing Systems (NIPS), pp. 2017-2025, 2015.
11

---

Under review as a conference paper at ICLR 2022
Rico Jonschkowski, Austin Stone, Jonathan T. Barron, Ariel Gordon, Kurt Konolige, and Anelia Angelova. What matters in unsupervised optical flow. In European Conference on Computer Vision (ECCV). pp. 557-572, 2020.
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, al. Model-based reinforcement learning for atari. arXiv preprint arXiv: MeKiv:/9000 /9030999330 2019.
Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Fei-Fei Li. Large-scale video classification with convolutional neural networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1725-1732, 2014.
Kacper Kielak. Do recent advancements in model-based deep reinforcement learning really improve data efficiency? arXiv preprint 2rXiv:2003.0018/ 2020.
Diederik P Kingma and Max Welling. 1312v:1312.611 2013.
Auto-encoding variational bayes. arXiv preprint
Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004.13649) 2020.
Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data, In Advances in Neural Information Processing Systems (NeurIPS), 2020a.
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. CURL: contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning (ICML), pp. 5639- 5650, 2020b.
Andrew L. Maas, Awni Y. Hannun, and Andrew Y. Ng. Rectifier nonlincarities improve neural network acoustic models. In International Conference on Machine Learning (ICML.) 2013.
Bogdan Mazoure, Remi Tachet des Combes, Thang Doan, Philip Bachman, and R. Devon Hjelm. Deep reinforcement and infomax learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Simon Meister, Junhwa Hur, and Stefan Roth. Unflow: Unsupervised learning of optical flow with bidirectional census loss. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, pp. 7251 2018.
Andriy Mnih and Koray Kavukcuoglu. Learning word embeddings efficiently with noise-contrastive estimation. In Advances in Neural Information Processing Systems (NIPS), pp. 2265-2273, 2013.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv:1912.01703. 2019.
Zhe Ren, Junchi Yan, Bingbing Ni, Bin Liu, Xiaokang Yang, and Hongyuan Zha. Unsupervised deep learning for optical flow estimation. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, pp. 1495-1501, 2017.
Julian Schrittwieser, Thomas Hubert, Amol Mandhane, Mohammadamin Barekatain, Ioannis Antonoglou, and David Silver. Online and offline reinforcement learning by planning with a learned model. arXiv preprint arXiv:2104.06200 2021.
Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bach- man. Dfta-efficient reinforcement learning with self- preficredictive representations, In International Conference on Learning Representations (ICLR), 2021.
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, and Sergey Levine. Tone-contrastive networks: Self-supervised learning from video. In IEEE International Conference Robotics and Automation (ICRA), pp. 1134-1141, 141, 2018.
12

---

Under review as a conference paper at ICLR 2022
Wenling Shang, Xiaofei Wang, Aravind Srinivas, Aravind Rajeswaran, Yang Gao, Pieter Abbeel, and Michael Laskin. Reinforcement learning with latent flow. CoRR, abs/2101.0187,, 2021.
Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. In Advances in Neural Information Processing Systems (NIPS), pp. 568-576, 2014.
Adam Stooke and Pieter Abbeel. ripyt: A research code base for deep reinforcement learning in pytorch. arXiv preprint 199.01500 2019.
Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. CoRR, abs/2009.08319, 2020.
Deqing Sun, Xiaodong Yang, Ming- Yu Liu, and Jan Kautz. Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume. In IEEE Conference on Computer Vision and Putlern Recognition (CVPR), pp. 8934-8943, 2018.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv: 10X1:0/000.00600, 2018.
Yonglong Tian, Dilip Krishman, and Phillip Isola. Contrastive multiview coding. In European Conference on Computer Vision (ECCV), pp. 776-794, 2020.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR, 0tlp/1807.03744 2018
Hado van Hasselt, Matteo Hessel, and John Aslanides. When to use parametric models in rein- forcement learning? In Advances in Neural Information Processing Systems (NeurIPS), pp- 14322-14333, 2019.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pitrre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. (096-1103, 2008.
Longguang Wang, Yulan Guo, Yingqian Wang, Zhengfa Liang, Zaiping Lin, Jungang Yang, and Wei An. Parallax attention for unsupervised stereo correspondence learning. CoRR, (2s/2009.08250 2020.
Yang Wang, Yi Yang, Zhenheng Yang. Liang Zhao, Peng Wang, and Wei Xu. Occlusion aware unsupervised learning of optical flow. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4884-4893. 2018.
Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non parametric instance discrimination. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018
Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Im- proving sample efficiency in model-free reinforcement learning from images. arXiv preprint arXiv:1910.01741 2019.
Jure Žbontar and Yann LeCun. Stereo matching by training a convolutional neural network to compare image patches. The journal of machine learning research, 17(1):287-23188 2016.
Feihu Zhang, Victor Adrian Prisacariu, Ruigang Yang. and Philip H. S. Torr. Ga-net: Guided
aggregation net for end-to-end stereo matching. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 185-194, 2019.
13

---

Under review as a conference paper at ICLR 2022
A IMPLEMENTATION DETAILS
This section provides tables summarizing the hyperparameters used for experiments on Atari Games Kaiser et 2019) and DMControl Suite (Tassa al., 2018) and the detailed description of our network architecture.
A.1
HYPERPARAMETERS ON ATARI GAMES AND DMCONTROL. SUITE
Table 4: Hyperparameters used for R experiments on Atari Games
Parameter
Value
Observation Size
(84, 84)
Random shifts (=4 pixels),
Augmentation
Intensity (scale=0.05)
Image Gray-scale
True
Update
Distributional Q
Stacked Frames
4
Action Repeat
4
Reward Clipping
[-1. 1]
Training Steps
100K
Evaluation Trajectories
Minimum Replay Size
2000
(for sampling)
Max Frames
108K
(per episode)
Support Of Q-Distribution
bins
Discount Factor
0.99
Optimizer
Adam
Optimizer: learning rate
0.0001
Optimizer: 31
0.9
Optimizer: $2
0,999
Optimizer:
0.00015
Max Gradient Norm
10
Multi Step Return
Target Network: Update Period
1 32, 64, 64 8 x 8, 4 x 4,3
Q Network: Channels
Network: Filter Size
Network: Stride
(4, 2), (1,
Network: Hidden Units
256
Non-Linearity
ReLU
Replay Period Every
1
Updates Per Step
2
Exploration
Noisy Nets
Noisy Nets Parameter
0.5
Priority Exponent
0.5
Priority Correction
0.4 +
For the reproducibility of our work, we provide full hyperparameters for our experiments on Atari Games (Kaiser et al.) (209)) in Table We follow the common practices used to set up Rainbow DQN van Hasselt et al. (2019) in existing methods Laskin al. (2020b); [Schwarzer al, (202 for experimenting Atari Games. Table full list of hyperparameters for DMControl suites experiments. We utilize the similar hyperparameters and optimizer to CURL Laskin et al. 2020b).
14

---

Under review as a conference paper at ICLR 2022
Table 5: Hyperparameters used for S experiments DMControl
Parameter
Value
Observation Size
(84, 84)
Observation Rendering
(100, 100)
Augmentation
Random crop, translation
Stacked Frames
3
2 (finger-spin, walker-walk),
Action Repeat
8 (cartpole-swinup))
4 (otherwise)
Evaluation Episodes
10
Discount Factor
0.99
Optimizer
Adam
(31, (2)) (fo, TV. Qo) (81, + (a)
(0.9, 0.999)
(0.5, 0.999)
2c (cheetah-run),
Learning Rate (ʃo, u.. 2+)
le (otherwise)
Learning Rate (a)
le
Batch Size Replay Buffer Size
64
100000
Initial Steps
1000
Hidden Units (MLP)
1024
Function EMA
0.01
Critic Target Update Frequency
Convolutional Layers
4
Number Of Filters
32
Non-Linearity
ReLU
Latent Dimension
50
Initial Temperature
0.1
15

---

Under review as a conference paper at ICLR 2022
A.2 NETWORK ARCHITECTURE
The detailed network architecture of our method is provided in Table We describe the architectures of the encoder, FlowNet and transition model. 'N', 'K' and 'S' of convolution operations represent the channel, kernel size, and stride, respectively. 'LReLU" and 'BN' indicate Leaky Maas et al. (2013) and Batch Normalization Iofffe & Szegedy (2015). We also provide input and output tensors of each layer for better understanding.
Table 6: Detailed describtion of the proposed network architecture
Encoder
Layer
Operations
Input
Output
Conv(N32, K8, S4) ReLU DropOut
I~ enl,~enleeM
2
Conv(N64, K4, S2) ReLU DropOut
en2r~-n22t+
3
Conv(N64, K3, S1) ReLU DropOut en2, en2
en
FlowNet
Layer
Operations
Input
Output
Conv(N32, K1, S1) BN LReLU
en3; en3,, en3r en3 corr concO
en3;
5
Compute Correlation Volume
corr
Concatenation
concO
5
Conv(N256, K3, S1) BN LReLU
conv
6
Conv(N512, K3, S1) BN LReLU
conv conv2p
conv2p
Conv(N512, K3, -1)- LRRLLU
conv2
8
Conv(N512, K3, S1) BN LRRLU
conv2
conv3p
9
Conv(N512, K3, S1) BN LRRLU
conv3p
conv3
10
Conv(N2, K3,
conv3 flow3, conv2 conv3 conv2up, conv3d, flow3up conc3 flow2, conv1
flow3
Upsampling
flow3up, conv2up conv3d
12
Deconv(N256, K4, S2) LReLU
13
Concatenation
conc3
14
Conv(N2, K3,
flow2
Upsampling
flow2up, conv lup conc3d conc2
16
Deconv(N128, K4, S2) LRRLU
conc3 convlup, conc3d, flow2up
17
Concatenation
Conv(N2, K3, S1)
conc2
flowl
19
Upsampling
flowl, em2, conc2 en2up: conc2d, flowlup concl
flowlup, en2up: cone2d
20
Deconv(N130, K4, $2) LReLU
21
Concatenation
conel
22
Conv(N2, K3, S1)
flowO
Transition Model
Layer
Operations
Input
Output
Onehot Encoding (Only for Atari)
ag
2
Concatenate
7 as
conc
3
Conv(N256, K3, S1) ReL.U
It
Conv(N64, K3, S1)- ReL.U
conv
conv2t 2
5
Min-Max Normalize
conv2t
A.3 ENVIRONMENTS
S was implemented in Pytorch (Paszke al. 2019) with the use of rlpyt (Stooke Abbeel 2019
and Mujoco Todorov al. 2012 license, and was simulated on Titan RTX GPU.
16

---

Under review as a conference paper at ICLR 2022
A.4 DETAILS OF USING FLOW MAPS
Use of internal flow: When four consecutive frames are used in Rainbow DQN (M = 3), the external flows are specified using query images og (Ix Ik44)) and target images ox+1 (1++1 T as follows:
flow (2 11
flow + 22)2)
flow (#k42- 234))
flow (#k43) +
The internal flow is computed using query images o
flow (2), 4k+3)-
The external flows computed between consecutive two frames are used warp the query represen- tations. The internal flow is also predicted using the same self-supervised flow network yet with two distant frames (zg 3) )). We found that this often effective in dealing with the case where the flow between two consecutive frames is relatively small. Namely, the internal flow between t and (K: + 3)th frames can be complementary when the external flows rather small and thus flow learning becomes less effective.
The internal flow is computed with a stack of images ok, while the external flows computed between OF and 0x+1- This is why we name two flow parts 'internal' flow and 'external' flow, respectively. Note that the internal flow is computed only between two distant frames (htth and (k + 3)th frames in ox), considering that the external flow is computed between two consecutive frames as above.
Image and representation warping: The representation we use contains spatial information. For instance, suppose the convolutional feature map 2 of the size 32 32 x 64 is generated from an input image I of the size 128 x 128 x 3. The feature map can have lower spatial resolution (by the downsampling operator such as max-pooling or strided convolution) and larger channel dimension than the input image. The 64 dimensional vector at a spatial position of the feature map represents the information of corresponding positions of the input image. Therefore, we can employ the flow used for image warping for warping the feature map (query representations). This kind of implementation has been commonly used in many computer vision tasks where an image alignment is required.
Visualization of flow maps: For better understanding of the proposed method, the examples of the flow maps predicted by R R are represented in Figure
(a)
(b)
(c)
(a)
(b)
(c)
Figure 3: Visualization of the flow maps learned by R: (a) The source frame, (b) the target frame, and (c) the flow map of the task 'Walker, Walk and 'Cheetah, Run" of the DM Control Suite.
17

---

Under review as a conference paper at ICLR 2022
B PERFORMANCE CONSISTENCY EVALUATION
We consider that the performance depends on the choice of the seed, so we measure the performance by using 10 random seeds and averaging results. To prove that performance improvement is consistent and not caused by the noise of the estimation, we observed the change in the average performance according to the number of the random seeds. Figure a represent the quantitative evaluation on the Atari games (Kaiser et al. 2019) and DMControl Suite TTass et al., 2018) according the number of the random seed. We show result of 5 random seeds and 10 random seeds compare the performance. In Figure [ we additionally showed the standard deviation of the performance. The bar graph represents the average of the performance, and the line graph represents the standard deviation of the performance.
From Figure 4 and it can be seen that our results do not differ significantly depending on the number of random seeds. It can be seen that our performance is significantly higher than the SOTA performance regardless of random seed, and is consistent.
Seaquest
Obert
Private Eye
Ms Pacman
Master
Krull
Ismeshond
Gophar
Frostbite
Freeway
Demon Attack
Cracy Climber
Coppperoommand
Breakout
Busing
Bank Heist
Accault
Allen
2000
4000
Figure 4: Quantitative evaluation the 26 Atari games Kaiser et al. 2019) according to the number of the random seeds: We show the mean performance on the 26 Atari games (Kaiser et al.) 2019) after 100K time steps for each 5 random seeds and 10 random seeds show the consistency of the proposed method regardless of random seeds.
18

---

Under review as a conference paper at ICLR 2022
3230
Finger, Spin
Cartpole, Swingup
Reacher, Easy Cheetah, Run Walker, Walk
in Cup,
random seed
10 random seed
random seed
10 random seed
Figure Quantitative evaluation on the DMControl Suite (Tassa al. 2018) according to the number of the random seed: We show the mean performance the DMControl Suite (Tassa et al.) 2018)) after 500K time steps for each 5 random seeds and random seeds to show the consistency of the proposed method regardless of random seed.
C ABLATION STUDY ON DATA AUGMENTATION
As described in RAD (Laskin et al. 2020a) and DrQ (Kostrikov et al.
2020). image augmentation
can improve the data data-efficiency and generalization of RL methods. To study the impact of data augmentation when used with the proposed method, we measured the average performance over 10 random seeds according to the data augmentation on DMControl Suite (Tassa et al. 2018)). In Table 7 we evaluated the performance the proposed method when used with crop and translation proposed in Laskin al. (2020a).
Slightly different from the result presented in Laskin et al. (202aa), Cartpole Swingup and Reacher Easy has the best performance when augmentation was used, Finger Spin and Cheetach Run has the best performance for translation, and Walker Walk and Ball in cup Catch has the best performance for crop. Since R learns flow in end-to-end manner with RL algorithm, it is analyzed that the results are different from those of Laskin et al. 2020a)..
Table study the impact of various data augmentation, we measured the average performance over 10 random seeds according the data augmentation on DMControl Suite (Tasaa et al. 2018 with 500K time steps.
500K step scores
S + no
S crop
S translation
Finger, Spin
8344-95
821=128
954+133
Cartpole, Swingup
880.134
837.16
872-511
Reacher, Easy
932.141
8331887
908_79
Cheetah, Run
448166
412.881
501 63
Walker, Walk
914433
930-75
886=51
Ball in cup, Catch
962+14
988:154
946=42
19

---

Under review as a conference paper at ICLR 2022
D ADDITIONAL EVALUATION METRIC
In all experiments, the evaluation on Atari Games (Kaiser et al. 2019) was conducted by measuring the performance with 10 or more random seeds. following the previous studies including SPR (Schwarzer et al., (02T)) and CURL (Laskin et al. 2020b). Recently, [Agarwal Recanaly,Agaaraal et al. (2021) analyzes the problems related to statistical uncertainty in the existing evaluation method of Atari Games (Kaiser et al.) 2019). Accordingly, we add the results of a more complete evaluation using Probabblity
of Improvement proposed in Agarwal et al. (202T) Figure 6
P(SimPLe X)
P(DER X)
P(OTR X)
S3R
SPR
E CURL OTR
DER
SimPLe
0.30 0.45 0,60 P(CURL X)
3,00,4506000 P(DrQ(E)
0.15 0.30 0.45 P(SPR
S3R
SPR
E
CURL
OTR DER
SimPLe
0.15
0.30 0.45
0.450.600.75
0.45 0.60 0.75 0.90
P(S3R X)
S3R
SPR
Drō
I
CURL
OTR
DER
SimPLe
0.60 0.75 0.90
Figure 6: Evaluation result of 'Probability Improvement" proposed in Agarwal al. 2021
This evaluation method estimates how likely an algorithm improves upon another algorithm Agarwal al. (202T). For instance, 'P(SimPLe>X) indicates the probability (written in the horizontal line) that 'SiPLee' is better than another algorithm, called 'X', which is listed in the vertical line (e.g.. S3R. SPR.... DER). Namely, the probability that SimPLe better than SPR, "PSSmmPLe>SPR)" has an average value of about 0.3.
In this context, in the six graphs above, the smaller the value of S the higher the performance. Also, in the graph of R below, 'P(S3R > X)', S ranks the highest as the remaining bars of algorithms are located the right of the bar of R From this analysis, it can be reconfirmed that SRR has the most superior performance compared to the state-of-the-arr methods on Atari Games (Kaiser et al. 2019) as mentioned in the result of Section 4.1.
20

---

Under review as a conference paper at ICLR 2022
E ANALYSIS ON COMPUTATIONAL COST AND PERFORMANCE IMPROVEMENT
The increase in the computational cost for training is unavoidable because S additionally leverage the flow prediction and warping networks used in the vision task, but we found that the additional computational cost for training is not so significant. For training on DMControl Suite Tassa al. 2018) up to 500K on the same GPU environment, the proposed method takes about 16 hours, whereas the srate-of-the-ee art methods CURL (Laskin et al.] (2200b) and SPR (Schwarzer et al. 202T) take about 10 hours and 13 hours, respectively. Note that the original SPR paper did not provide the code implemented for DM Control Suite, so we conducted the experiments by modifying the original SPR code. Additionally, the flow prediction and warping networks are used only during training. and the inference process is implemented in the same manner as other methods. Therefore, the inference time of our method is exactly the same as that of the state-of-the-ar methods (CURL (Laskin et al., 2020b), SPR (Schwarzer al. 2021), DrQ (Kosrrikov et al. 2020 long as the same encoder for query images is used.
Computational cost:
We further analyzed the performance improvement by method based on the two benchmarks used in Section 4.
26 Atari Games: Based on the evaluation method used in the existing Atari Games (Kaiser et al. 2019), CURL (Laskin et al. 2020b) recorded the highest mean in 7 games out of 26, and SPR (Schwarzer et al. 2021 recorded the highest mean in 11 games out of 26. S has the highest mean in 13 games out of 26. It can be interpreted that the performance increase of the S is not small by considering the quantitative aspects of these games. Also, among the 13 games in which S has an edge. in particular, in 8 games (Alien, Assault, Gopher. Jamesbond, Krull, Kung Fu Master, Ms Pacman, and Seaquest), S records remarkably higher performance compared other methods. Even the performance of certain games is high enough to match that of humans. This is because the proposed method of capturing the local spatial structure is able to derive an effective representation from the images of the specific Atari Games with various movements.
However, S may not be effective for some games. In particular, S did not perform well in the task 'Pong' in Atari Games (Kaiser et al. 2019). The biggest reason for this is that there are too few discriminative spatial structures available in the game images. Therefore, we can be sure that our representation learning method, which effectively captures the spatial structure, will work particularly well for data with much more complex structural features. In other words, while most of the simple methods suffer from training with data with complex structural features, R can be good substitute for addressing this.
A method with relatively cow-compexityy may be preferred depending the situation. However, as mentioned above, it was shown from the Atari benchmark that it is much better choice to use S in the tasks that need capture complex structural features.
In the case of DMControl Suite TTass et al. 2018). the performance at 100K steps is usually based on when most methods do not converge. In Table 2 the performance of S3R recorded the highest mean in 2 tasks out of 6 tasks, and RAD (Laskin et al., 2020a) and DrQ (Kostrikov et al.) 2020) also recorded the highest mean in 2 tasks out tasks, respectively. It can be interpreted that RAD (Laskin et al. (2020a). DrQ (Kostrikov et al., (020) and S²R are the three methods with the highest convergence speed.
DMControl Suite:
In general, the performance at 500K steps after most methods converge is widely adopted for the evaluation. Table S shows the highest performance in 4 out of tasks compared to state- of-the-arts. Considering that this performance is the average value obtained by running 10 random seeds, it is a credible assessment. Also, when compared to the performance improvement rate of other methods, the performance increase of S3R is by no means small.
21


Reviews:
[{"review_id": "pXpSxTXg9l3", "reviewer": "Reviewer_iodY", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The motivation is justified because indeed previous literature has shown that when humans play Atari games, representation of objects are important for efficient learning. Learning global representation may lose detailed information of objects. Although the current work does not attempt to explicitly model objects, the flow information likely captures some object information implicitly.\nI find the results generally convincing and the idea appears novel and interesting.\nHowever, I am a bit worried why the internal flow loss in equation 4 will help the network extract useful information. If my understanding is correct, the flow for warping internal representation is the same flow as the one for warping image (equation 2). How can we guarantee this the extracted query feature does not simply become a fixed linear transformation of the local image patch? As long as the representation in equation 1 can be learned to predict external flow that satisfies the image warping cost, it seems to be able to guarantee the cost in internal flow to be low as well, even if query feature is a simple linear feature of local image patch.\nAlthough the paper has demonstrated good performance in the RL task, it may be interesting, if possible, to provide some visualization of what kind of local feature is actually learned, in the appendix.", "summary_of_the_paper": "The paper proposes a new self-supervised representational learning for reinforcement learning tasks, and demonstrated superior performance in the sample-efficient setting against states-of-the-arts models. The major idea is to force neural networks to encode optical flow between consecutive frames and impose regularization that the local embedding predicted by warping and a state transition model (given action) are both similar to the extracted embedding from the next frame.", "summary_of_the_review": "I think the paper provides an interesting new objective for encouraging the representation by self-supervised learning also captures important local information for efficient learning of RL tasks. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "o-ph0QuR5U2", "reviewer": "Reviewer_pMqF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. Quantitative results are promising. The proposed approach clearly shows promise over prior approaches.\n\n2. Evaluation is thorough. The results hold across different environments and even different agent setups (Rainbow and SAC)\n\n3. The paper is well-motivated on a conceptual and mathematical level.\n\nWeaknesses:\n\n1. The technical novelty is limited. The approach is very similar to SPR with the addition of latent variable warping.\n\n2. It's difficult to conceive how this method could apply to different input modalities (i.e. audio, text, joint positions). The flow-based approach is very much tied to image inputs. \n\n3. While there is improvement in quantitative performance over baselines, the margin of improvement is fairly small.", "summary_of_the_paper": "In this paper, the authors focus on the problem of representation learning for deep reinforcement learning. To this end, they propose an approach to learning structured representations via establishing flows between latent volumes. Similar to SPR, they predict future representations with a transition model conditioned on actions. They evaluate their approach in two domains: Atari and DeepMind Control Suite and show improved performance over the state of the art. ", "summary_of_the_review": "The authors propose an approach to self-supervised learning for reinforcement learning which is very similar to SPR. The key contribution is the addition of a flow-based approach to predicting future latent vectors. Because of this, the generality of the approach is limited to image inputs. However, the results are promising and hold across different domains and agent setups. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Io3XF5ktvYd", "reviewer": "Reviewer_Wtop", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is very well-written and referenced, and the figures are quite clear (especially Fig 2 which gives a great overview of the method). I find the idea of leveraging self-supervised flow models quite interesting as an alternative to brute-force next-step prediction.\n\nMy main concern, however, is that the results the authors chose to put forward are not as good as I'd have liked, which makes me doubt the overall significance of the method:\n- in Atari, the proposed S³R performs significantly better than SPR in 12/26 tasks, and underperforms significantly in 7/26 tasks;\n- in Control suite at 100K steps, S³R significantly outperforms DRQ in just one task out of 6, and underperforms significantly in 2, and both are more-or-less tied at 500K steps;\nSince S³R seems significantly more complex in terms of implementation and computational cost, it seems unlikely that a practitioner in the field would bother compared implementing it rather than using a much simpler variant.\n\nSome remarks/questions:\n- I'm not aware of a definition of \"structured representation\", but I'm not sure I'm convinced by the claim that S³R's repr is more structured  because its features translate to flow maps. I also don't think it's necessary for the paper to make this claim.\n- Introduction: \"Learning visual features from raw pixels only using a reward function leads to limited performance and low sample efficiency.\"  I would tame down/remove that claim. I don't think there's any consensus on that, and work such as  https://arxiv.org/pdf/2104.06294.pdf are able to improve performance and sample efficiency without relying on an explicit visual features representation learning scheme\n- I was surprised by the results of Table 3 (that F+W > P, and that F+W+P > F+W), as I would have expected next-step latent prediction to be a somewhat easy task. Do the authors have any further insight on this, eg on how (dis)similar Z^w and Z^p are, maybe in terms of an auxiliary stop-gradiented Sim operator?\n- Following up on that, could using both Z^w and Z^q as input to the RL head bring an improvement?\n- Notations: the paper (eg fig 2 and the paragraph at the top of p5) seems to imply that a_k is computed from I_k to I_{k+M}, should this be I_{k-M} to I_k instead?", "summary_of_the_paper": "This paper focuses on self-supervised visual representation learning for RL applications. As opposed to previous literature that focused on learning global representations for the current observation, the authors propose learning \"structured\" representations based on flow maps that encode local structure. To do so, they combine:\n- a self-supervised flow model (from previous literature), trained using an image reconstruction loss;\n- a classic action-conditioned 1-step transition model using cosine similarity in latent space\n- a flow based warping (also from previous literature) to provide a second next-step latent prediction task.\nThey evaluate their method on various standard RL tasks in two domains (Atari, DM Control Suite) and show improvements over SOTA in ~half of the tasks.", "summary_of_the_review": "The paper combines existing self-supervised architectures (self-supervised flow estimation and InfoNCE-like similarity loss) to learn visual features in a rather novel and intriguing combination, but the overall increment in RL performance does not seem to justify the added complexity compared to much simpler counterparts.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3ctCfyTn1t", "reviewer": "Reviewer_aHc1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "strengths:\n* the method does outperform competitive baselines on challenging atari games\n* the paper is well written and the method seems good.\n* results are shown with a fixed set of loss weights which shows robustness.\n\nweaknesses:\n* the method is very complex and hard to reproduce. Even if the code is made available, building on top of this method will not be trivial.\n* the ablation experiments are done on domains that favour good flow representations and lack baselines (dm control)\n* the crux of the state-representation problem is that it should make credit assignment easier but the hard exploration games are either missing from the experiments or improvements seem marginal.\n\nremarks:\n* i think structured representation is a misnomer here. This is because the image lattice is the weakest form of structure you could have and because structured representations in the literature refer to objects and parts. So I recommend changing the title to reflect that.\n* i can't seem to find the L_rl loss in your text. Is it SAC ? I guess it's just above the loss functions where you hint that both DQN and SAC where used somehow\n* The method seems to struggle with pong ? Is this because the RL loss in this setting and these hyper parameters is washed away by the auxiliary loss gradients which have a hard time converging ? \n", "summary_of_the_paper": "This paper proposes a representation learning method that leverages unsupervised signals like flow and forward models and the constraints between them and apply it to the state representation problem of RL. \n\nThe method is adding a number of auxiliary rewards to the torso which flow and a latent transition model that are constrained to agree with a contrastive loss. The flow has an architecture inspired by FlowNet and trained with warping reconstruction and a spatial smoothness term. A latent transition model is trained contrastively. Then constraints are added to make the latent of the next step agree with both of these predictions. \n\nThe method is run on 13 Atari tasks and compared there with competitive baselines and it largely outperforms them. An ablation without baselines is performed a few dm control suite tasks  ", "summary_of_the_review": "The method seems reasonable but the approach is very complicated and there are some lacking experiments that would make the claims more compelling. If the community is to be convinced to embrace such a complicated approach experiments need to be more compelling i should think.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: After carefully reading the reviews and the rebuttal, unfortunately I feel this work is not yet ready for acceptance. 

I want to acknowledge the effort put in the rebuttal for this work and I think all the changes greatly increased the value of the work. However, I feel that the work could greatly benefit from running on a different domain where the gain is more considerable. My worry is that the complexity of the method compared to the relatively small improvement (at least as perceived from the current results) will reduce considerably the attention the work will receive from the community (unfairly so). 
Or some of the analysis and ablation done (e.g. the flow visualization) which are now in the appendix could be brought in the main manuscript to be able to drive the message home. An understanding of the impact on the accuracy of the flow model on the overall performance (which as pointed out by reviewer aHc1 is a really hard task in a more natural context). Or maybe a 3D visually complex environment is exactly where this method will shine as flows are more complex and hence more informative. 

Overall I think this is solid work, but I feel it does not manage to convince the reader of the significance of the proposed approach. And hence, if published in this form, I feel it will do a disservice to the work, as it will not receive the attention it merits from the community.
Decision: Reject