Temporal Change Sensitive Representation for Reinforcement Learing
Authors: Qi Gao
Wei Xu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Reinforcement Learning
Representation Learning

Abstract: Image-based deep reinforcement learning has made a great improvement recently by combining state-of-the-art reinforcement learning algorithms with self-supervised representation learning algorithms. However, these self-supervised representation learning algorithms are designed to preserve global visual information, which may miss changes in visual information that are important for performing the task. To resolve this problem, self-supervised representation learning specifically designed for better preserving task relevant information is necessary. Following this idea, we introduce Temporal Change Sensitive Representation (TCSR), which is designed for reinforcement learning algorithms that have a latent dynamic model. TCSR enforces the latent state representation of the reinforcement agent to put more emphasis on the part of observation that could potentially change in the future. Our method achieves SoTA performance in Atari100K benchmark.

Under review as a conference paper at ICLR 2023
TEMPORAL CHANGE SESSITIVE REPRESENTATION FOR REINFORCEMENT LEARING
Anonymous authors Paper under double-blind review
ABSTRACT
Image-based deep reinforcement learning has made great improvement re- cently by combining state-of-the-art reinforcement learning algorithms with self- supervised representation learning algorithms. However, these self-supervised representation learning algorithms are designed to preserve global visual infor- mation, which may miss changes in visual information that are important for per- forming the task, like in Figure [ To resolve this problem, self-supervised repre- sentation learning specifically designed for better preserving task relevant infor- mation is necessary. Following this idea, we introduce Temporal Change Sensitive Representation (TCSR), which is designed for reinforcement learning algorithms that have latent dynamic model. TCSR enforces the latent state representation of the reinforcement agent to put more emphasis on the part of observation that could potentially change in the future. Our method achieves SoTA performance in Atari 10K benchmark.
t=
t= 1
t= 2
t= 3
t= 4
t 5
Ground Truth
TCSR
Predicted
EfficientZero
Predicted
The ground truth observation compared with image reconstructed from latent state rep- resentation predicted by TCSR and EfficientZero. TCSR can not only predict the movement of enemies in the short term (Marked the yellow box) but also predict exactly when and where the UFO will release new enemy till the end of the planning horizon (Marked in the red box). How- ever, EfficientZero fails to predict both these changes. This shows that TCSR more sensitive to the changes in the latent state representation. These change includes but not limited to position, appearance and disappearance task related objects as shown in this figure.
Figure 1:
INTRODUCTION
Deep reinforcement learning has achieved much success in solving image based tasks over the last several years. A critical step solving image based tasks is learning good representation of the

---

Under review as a conference paper at ICLR 2023
image imput. One of the biggest challenges for learning good representation for reinforcement learning is that the reward sparse (Shelhamer et al., 2016). which cannot generate enough training signal to train the representation network. To resolve this problem, self-supervised representation learning loss is often added to facilitate training.
There are many different approaches to image based reinforcement learning. Most of them try combine state-of-the-ar model based or model free backbones like SAC (Haarnoja et al. 2018), Rainbow (Hessel et al., 2018) and MuZero (Schrittwieser et al. 2020) with self-supervised repre- sentation learning algorithms to boost the training of representation. Among these methods, SPR (Schwarzer et al. 2020) and EfficientZero eYe al. (02T)) are state-of-the-a model-free and model based methods in the Atari 100K benchmark. They achieved the best score in 21 out of 26 Atari 100K games combined. They train dynamic model to predict the future latent states from an initial latent state calculated by the image encoder. Both the image encoder and the dynamic model are trained using the SimSiam(Chen He 2020) loss between the predicted latent state and the latent state calculated directly from the future observations.
However, most representation learning algorithms used reinforcement learning not emphasize the change of visual information, while creatures, including humans, are innately sensitive to the change of visual information. A very important part of the neural system is the middle temporal visual area (MT) (Von Bonin & Bailey 1947). Visual information is integrated and differentiated in MT' to capture the movement objects contained in visual information (Allman et al., 1985). The ability capture the changes in visual information helps creatures catch prey or escape enemy Maturana et al. (1600); Suzuki et al.] (2019). To help reinforcement learning agents acquire such ability, we propose Temporal Change Sensitive Representation (TCSR), a self-supervised auxiliary loss specifically designed for reinforcement learning methods that have a latent dynamic model. TCSR enforces the difference between two consecutive unrolled latent states be the same as the difference between two target latent states generated from two consecutive observations with the same augmentation.
TCSR uses EfficientZero YEe et al. 2021 the backbone and inherit most of the hyper-pane prrreeerr On the Atari 100k benchmark, TCSR surpasses EfficientZero in 19 out of 26 games (as shown in perfoooomm performance.
Figure 2 and achieves a new state-of-the-art
Improvement in Human Normalized Score
0.5
-0.5
Assault
Breakout
CrazyClimber
Jamesbond
Asterix
UpNDown
DemonAttack
RoadRunner
Krull
BattleZone
Pong
Qbert
KungFuMaster
Hero
Gopher
Kangaroo
Boxing
BankHeist
Amidar
Freeway
Frostbite
PrivateEye
Seaquest
MsPacman
Allen
ChopperCommand
Figure 2: The improvement of human normalized score by adding TCSR extra self-supervised representation learning auxiliary loss on EfficientZero backbone. TCSR surpasses EfficientZero in 19 out of 26 games in the Atari 100k benchmark
2

---

Under review as a conference paper at ICLR 2023
2
RELATED WORK
2.1 REPRESENTATION LEARNING IN REINFORCEMENT LEARNING
Almost every image based reinforcement learning algorithms learn a lower dimensional latent state representation from images. Self-supervised auxiliary tasks are often used to facilitate the training of the representation network. Some early works {Lange & Riedmiller 2010) YYarats et al. 2019) use widely accepted image reconstruction losses as the auxiliary loss. There was trend (rini vas et al., 2020; He et al., 2019a Banino et al. 202T) of using contrastive losses as auxiliary loss led by CPC (Oord et al., 2018). Recently, similarity based losses (Grill et al. 2020 Chen Hej 2020) are more popular than contrastive losses since they do not need a large number of negative pairs. Temporal consistent/predictive loss is another auxiliary loss that is often used to encourage the representations learned by the agent to contain predictive information (Schwarzer et al.) 2020) Guo et al.] 2020 Nguyen et al.) (02T). Latent state representations learned with temporal consis- tent/predictive loss could also be unrolled by the dynamic/transitions network to simulate experience
(Hafner et al.] 2019a; 2020) or perform planning (Hafner et al.) 2019b). As the backbone of our al- gorithm, EfficientZero (Ye al. 2021 uses the unrolled latent state representation for both training
and planning.
2.2 MODEL BASED REINFORCEMENT LEARNING
Model based reinforcement learning algorithms usually have access to or learn world model. Given the current state and action, the model can predict the next state and the next reward. The model can be used to generate simulated experience and/or perform planning (Sutton & Barto 2018). Learning a model to generate simulated experience and perform planning with high dimensional inputs like images (Seritteehr et al. 2020 Ye al., [022]) is usually more challenging than with low dimensional states (Abbeel et al. 2006 Deisenroth & Rasmussen (011]) Some reinforcement
learning algorithms learn world model only for training the representation network (Schwarzer iet al. 2020; Kaiser et al. 2019 Guo et al. 2020). Dreamer (Hafner et al., [2019a 2020) only use the learned model to generate simulated experience for training. PlaNet (Hafner et al. 20199) only use the learned model for planning. Our work aims to help model based reinforcement learning algorithms with high dimensional inputs train a better model and representation network.
2.3 VIDEO PREDICTION
Video prediction has been classic topic in the field of machine learning (Oprea et al. 2020
Action conditioned video prediction of Atari games could be dated back to 2015 (Oh et al. 2015
Chiappa et al., [017]. They are the foundation work of learning a world model of Atari games for reinforcemntt learning. Some video prediction algorithms focus on the temporal changes of the inputs (Michalski et al. (2014] Finn et al. 2016), which is similar to our work. But the temporal changes they focus on are at the pixel level while we focus on the latent state level, and our algorithm is designed for reinforcement learning.
BACKGROUND
3.1 MCTS OF MUZERO
MuZero (Schrittwieser al. 2020) Monte-Carle Tree Search (MCTS) based Reinforcement Learning method. MuZero operates the MCTS with a representation function, a dynamic function and a prediction function. The representation function h encodes an observation o, into latent state representation $1,0 h(o:). The dynamic function predicts next latent state representation and reward st,4+1, Tt,++1 = g( dc+k) given current latent state representation St,k and action Gc+k- Given latent state St,k, the prediction function predicts the policy and value Pt,k; Ua,k f((,k)) The policy Pt, is used to expand and navigate through the tree. The value U4 is used to estimate the values of each node of the tree.
When collecting data, MuZero performs MCTS at each unroll step t. The action a, will be chosen through UCB based the result of MCTS. Then the action is passed to the environment. The
3

---

Under review as a conference paper at ICLR 2023
resulting next observation o++11 and reward u4-1 stored in the replay buffer. The replay buffer also stores the expected return 2, at the root estimated with MCTS and the action distribution ne at the root.
During training, batch of samples is chosen from the replay buffer. A sample consists ini- tial observation Ot, action sequence Ot:--0t+k-- ground truth reward sequence u4+1,--MMe+K bootstrapped value target sequence 2,++, and action distribution sequence Tr:...T+<K: where K is the maximum unroll length. Initial state is generated from the stacked observation 8t,0 h(:t-n, ..)) Unrolled latent state representation and predicted reward are generated recursively with the dynamic network $t,A+1, 11,++1 ( a++x). Predicted policies and values are gener- ated with prediction function Pt,k, Vt, = f(),,)) for each unrolled latent state representation. At last, the MuZero loss calculated follows:
Lsauzero (t)
Ereward(uetk T(,k) + Lvatuo (244 (1,6)) + Epolicy (Te+ks Pt,k) (E......))
E
Where Lreward Lvalues and policc are Cross-Entrope loss. critical to notice that MuZero does not assume the predicted latent states correspond to actual states. The predicted latent states gen- eralize across future states that have similar subsequence values, rewards and action distributions (Schrittwieser al. 2020).
3.2 TEMPORAL CONSISTENT Loss OF EFFICIENTZERO
EfficientZero (Ye et al. 202T) is an efficient sample variant of MuZero that addresses three prob- lems: no supervision dynamic and representation function, state aliasing and off-policy issue. To address these problems, EfficientZero made three modifications to MuZero. The first modification is adding self-supeee seff-supervised consistent loss. The second modification is adding end-to-end prediction of the value prefix by predicting the value prefix with multiple previous unrolled states instead of just the current state. The third modification is model-based off- cofippllcy correction, which is done by performing another MCTS tree search at each leaf node to obtain more accurate state value.
Among these three modifications, Self-Supervised Consistent Loss made the most contribution the final result. Similar to SPR, MuZero's self-supervised loss uses SimSiam self-supervised framework and trains the dynamic function and representation function the same time. During training, sequence of observation Pt+1l...0ta.. following initial observation o, are drawn from the replay buffer in addition target latent state representation $1+100 St.K will be generated with 8t+k.0 h(o(++) for k E
the action distribution, reward and value prepared for MuZero loss. Then
(1....K) K). At last unrolled state representation 8t,k will pulled toward target states representation $t+-.00 by adding consistent loss on top MuZero loss. Then the EfficientZero loss corresponds to time step is:
Crdficientzeno (1) LMazerr +
LsimSiam (8t,ke St++k.))
RE (l......))
By enforcing the consistent loss, EfficientZero assumes the unrolled latent state representation 8t, unrolled from initial latent $t.0 with ground truth actions at, a,,4..++h-1 represents the ground truth state at time k.
SPR as the backbone of our method because SPR only uses the training signal of the prediction to stimulate the training of the representation network, while Er- ficientZero will be able to take advantage of higher quality unrolled states when performing the MCTS.
We choose EfficientZero
3.3 DATA AUGMENTATION
Augmentation has been indispensable part of recent Imaged based Reinforcement Learning search. RAD (Laskin et al. 2020) has shown that data augmentation improves the sample efficiency and generalization of reinforcement learning. However, data augmentation could also harm the
4

---

Under review as a conference paper at ICLR 2023
performance of reinforcement learning. As shown in RAD, the performance of agents with aug- mentations like cutout are even worse than the baseline. This is because some critical information is removed from the image when doing augmentation. Due to this reason, the random shifts proposed by DrQ (Kostrikov et al. 2020 Yarats et al. 2021) have been the most popular data augmentation method since it makes the least change the input observation while providing enough variance to regularize the representation network. Kostrikov et al.) (2020) also pointed out that data augmen- tation can regularize downstream tasks like Q-learning beyond just regularizing the representation network.
We believe that even with the consistent loss, the augmentation is preserved at a certain level through the representation and dynamic network, affecting the prediction network. This is another reason why EfficientZero performs so well. So, when using augmentation, the notation of the latent state representations becomes:
8,+ h(ōr+k)
8 = h(ōr+k)
Sr,k+1 Istato (Bt,ooBrt,11 ac+kk)
Where and each represent augmentation of parameter time step I. &t,k and &t, should be similar but not necessarily the same.
3.4 SIMSIAM LOss
2020 is state-of-the-art representation learning method. It maximize the similarity between two augmentations of one image. Compare to other representation learning meth- ods like BYOL/Gril Grill al. 2020), SimCLR(Chen et al. 2020a). SwAV (Caron al.) 2020) and MoCoOHe et al., 201 Chen et al. 2020b), SimSiam can perform very well even without negative sample pairs, large batch size and momentum encoder. SimSiam consist of a projector P1 and predictor Ps. Given a backbone representation function and two augmentations i and å of same image x, the SimSiam loss is calcuated as follow.
Sineiam(Chen & He
Ú h(2)
ú h(t)
D ū)
LSimStam (.. D(P2(PI(P)) stopgrad +
stopgrad sloggrrdd
Where 'D is the negative cosine similarity. Since EfficientZero want maximize the similarity between predicted latent state representation and ground truth future latent state representation in- stead of the similarity between augmentation of one image, don't need the symmetry. So the modified SimSiam loss of EfficientZero is as follow.
SSimSiam ($1, $144.0) ( stopprrd)sopp
TEMPORAL CHANGE SENSITIVE REPRESENTATION
Most current representation algorithms focus on the general similarities and/or dissimilarities be- tween different inputs. However, the difference between two observations is usually limited for image based reinforcement learning tasks. Especially when training temporal predictive repre- sentation, the difference between two consecutive observations is only a small area in the image. Furthermore, we consider the changes introduced by the augmentation, a few pixel differences could be easily ignored. Under this circumstance. enforcing the temporal consistency may not be enough for the changes to be preserved in the representation. So, we introduce temporal change sen- sitive representation (TCSR). In addition to enforcing the consistency of the representation, TCSR enforces the consistency of the change of the representation. The training pipeline is as shown in Figure
5

---

Under review as a conference paper at ICLR 2023
Replay Duffer
Muleero Lees
Pi+k
Leoward
U(+k
Training Targela
Sewanl
Pt+t
Apeliy
/
Dt4k
21+6
8c,0
LstuZero
8
L
Obeds
EffiiennZerro Consistent Loss
h
Coussilcut
Sc k.0
01+b
TCSR Loes
h
81,
C
Mekk: O1pk-1
$
Reck-1
0 1
Autral
Ca
Diaegee
S
Oc+7
Figure 3: The training pipeline of Temporal Change Sensitive Representation((CSRR) On the left is the overall training picline shared by MuZero, EfficientZero and TCSR. An inital observation o drawn from the replay buffer and augmentated. Then 8r,0 is obtained by encoding o, with the representation network. Unrolled latent state representations Št,k are generated iterately with the dynamic network. On the right shows the detile of how loss is calcuated for each unrolled latent state representation $1,e in TCSR. St,A- is the previous latent state representation that is used to generate &t, though not marked in the figure for conciseness. Both Laomsistent and LA are SimSiam loss and they don't I share parameters.
4.1 TCSR LOSS
The TCSR Loss works alongside the training of EfficientZero as showen in Consider the training pipeline of EfficientZero with augmentation. A time step t is chosen, and corresponding information retrieved from the replay buffer. Initial latent state representation is generated from augmented stacked observation with representation network $t.0 = h())), where ó, is augmented from the original observation oe. Then unrolled latent state representations generated iteratively as &t,6 = Jaate ( Oc+R-1) for & E {}. The target latent state representations are generated from subsequence observations augmented with another parameter š144 h(ō++4) for E {1,... K), where 0z+4 is augmented from the original observation Or+-k- Note that o, and are different augmentations of Og and Ô for &o E {0... K) share same augmentation parameter (i.e. when the augmentation is random shift, they share the same shifting distance in and y axis, etc.) We define the change representation operation A as below.
Amnoil (8,,)) = 81,k - 81,
Aurger (31+0.0) ) = $,44,0 $144-1.0
At last, the similarity loss between the difference of two consecutive unrolled latent state rep- resentations (& and the difference of two consecutive target latent state representation Aurga (5+4,,0) is added on top of EfficientZero loss with weight of XreSR to formulate TCSR loss.
LTCSR (1) LE.iicienZeea (1) AusST
Lsimiiam (8, Aurger (5++k.0))
(E......))
Notice that the SimSiam loss here uses another set of projection and prediction net separated from the used in EfficientZero loss.
The beauty the TCSR loss is that the change of representation A calculated from two repre- sentations generated by observations augmented with the same parameter. So the minor changes between two consecutive steps will not be overshadowed by the difference introduced by the aug- mentation of different parameters. At the same time, the augmentation is still able to regularize and generalize the training since Amnell and A is generated from observation with different augmen- tation parameter.

---

Under review as a conference paper at ICLR 2023
Table 1: Scores of TCSR and other baselines Atari100K benchmark. TCSR is 13.91% % and 9.48% higher than the result of the EfficientZero source code that our method is base on. Note that the scores reported in the EfficientZero paper cannot be achieved by source code released on GitHub by the original author. So we choose to re-run the EfficientZero source code and report the result as our major baseline, which still the SoTA algorithm on the Atari100K benchmark before our work.
EmickattZerr
F.iiciatZee
Game
Random
Human
SimPLe
CURL
DrQ
SPR
(paper
(Soucee code
TCSR
reprosed)
re-tun)
Alica
7127.7
616.9
558.2
171.2
801.5
808.3
563.3
340,4
Amidar
5.8
1719.5
142.1
102.8
176.3
1488.6
139.4
145,6
Assault
222.4
742
527.2
600.6
452.4
571
1263.
1311.0
1946.7
Asterix
210
8503.3
1128.3
734.5
603.5
977.8
25557.8
9305.2
12389.3
BankHeist
14.2
753.1
34,2
131.6
168.9
380.9
351
242.9
264.7
BatileZone
37187.5
14870
12954
16651
13871.2
Boxing
0.1
12.1
9.1
1.2
35.8
52.7
21.4
21.8
Breakout
30.5
16.4
4.9
16.1
17.1
414.1
274.2
300.8
ChopperCnud
811
7387.8
1246.9
1058.5
780.3
974.8
1117.3
2312.5
1707.3
Crazyllimher
10780.5
35829.4
62583.6
12146.5
20$16.5
42923.6
83940.2
72941.7
91619.3
DemonArtack
152.1
208.1
817.6
1113.4
545.2
13003.9
5591.4
60443.5
Freeway
29.6
20.3
26.7
9.8
24.4
21.8
3.6
3.6
Frostbite
65.2
254.7
296.3
253.4
Gopher
257.6
2412.5
771
669.3
636.3
715.2
3260.3
1568.2
1712.1
Hero
1027
30826.4
2656.6
6279.3
3736.3
7019.2
9315.9
8689.8
10817.0
Jamesbond
29
302.8
125.3
471
236
365.4
517
255.2
441.7
Kangaroo
52
3035
323.1
872.5
940.6
3276.4
724.1
1415.6
1541.
Krull
1598
2665.5
4539.9
4229.6
4018.1
3688.9
5663.3
6505.9
6684.6
KunstuMasterr
258.5
22736.3
17257.2
14307.8
9111
13192.7
30944.8
20949.D
22663.9
McPacman
Pong
-20.7
14.6
12.8
-6..5
-8.5
-5,9
20.1
12.7
17.7
PrivateEyye
24.9
69571.3
58.3
218.4
-13.6
124
96.7
100.0
17.6
Qhert
163.9
13455
1288.8
1042.4
854.4
669.1
13781.9
6006.4
7192.1
RoadRunner
11.5
7845
5640.6
5661
8895.1
14220.5
17751.3
14213.5
15862.5
Scaquest
68.4
42054.7
683.3
583.1
1100.2
1017.4
690.4
UpNDown
533.4
11693.2
3350.3
2455.2
3180.8
28138.5
17264.2
$010.5
8081.5
Normed Meam
0.443
0,381
0.357
0.704
1.943
1.243
1.451
Normed Median
0.144
0.175
0,268
0.415
1.09
0.448
0.602
5 EXPERIMENT
5.1
ENVIRONMENT AND BASELINE
We evaluate TCSR Atari100k (Kasser al. 2019), widely used benchmark for sample effi- cient reinforcement learning. The reinforcement learning agent is allowed to interact and collect 100,000 steps with frame skipping of 4. So total of 400,000 frames generated from the sim- ulator. Atari is mostly used to test the sample efficiency of reinforcement learning algorithms Schwarzer al.) 2020 Kostrikov et al. 2020; Ye al. 2021). We follow the same settings of EfficientZero (Ye et al. 2021) to perform the evaluation. For each task, we perform 6 runs with different seeds and each run with 32 evaluation episodes. The mean accumulated rewards of 6 runs are calculated and recorded as the raw score. Then the human normalized score is calculated for each task with the following equation.
SCOTe SCOTe
ScOrenormed
ScOrChuman SCOTerandom
At last, the mean and median normed score of 26 atari games are used to evaluate the overall perfor- mance of this reinforcement learning agent. Our method is built on the latest source code released by EfficientZero author on GitHub. We could not reproduce the result reported in the EfficientZero paper using the code released by the author. Though the result from running the author released code is still better than other baselines such as SPR (Schwarzer et al.) 2020). Since our change is on top of the EfficientZero source code, we believe it is fair to compare the result of our method with the result from running the EfficientZero source code. We have run the latest source code leased by the EfficientZero author on our machine and report the result in Table The re-run result achieves SoTA performance on 12 out of 26 games in the Ataril005 benchmark and achieves mean and median human normalized score of 1 1.243 and 0.448, which still the SoTA method on the Atari 100K benchmark before our work. We have also included other popular
7

---

Under review as a conference paper at ICLR 2023
al. 2020
Kostrikov al. 2020 Kaiser al. 2019 Srinivas et al.
2020) baseline to compare
with TCSR in Table
5.2 RESULTS
The result of TCSR on the Atari100K benchmark is shown in Table 1 Our method achieves the highest score in 12 out of 26 games. Human normalized score wise, TCSR achieves a high score of 1.451 mean and 0.57 median, which are 16.73% and 34.24% higher than the result of the Effi cientZero source code that our method is based on. Specificaly, TCSR outperformance EfficientZero in 19 out of 26 games Atari 100K benchmark shown in
5.3 VISUALIZATION
To understand how TCSR influences the learned latent state representation, we trained decoder network to visualize what information is contained in the latent state. We only train the decoder with latent state representation generated directly from the representation network and ground truth observation as the target. This ensures that the decoder can only reconstruct information contained in the current latent state representation and cannot predict. Mean square error is used to calculate the loss at pixel lever. We stop the gradient at the latent state representation so that the training of image reconstruction will affect the regular training pipeline. When visualizing, we feed EfficientZero and TCSR with the same observation and action sequence to see the difference in unrolled latent states. An example of the Atari game Assault reconstruction result shown in Figure ! In Assault, a UFO will release enemies to attack the fighter controlled by the agent. When no enemy is on the screen, the UFO will release another enemy immediately. However, when enemies are on the screen, the UFO will release new enemy under certain rule. TCSR can capture such information and correctly predict when and where the UFO will release the new enemy. This explains why TCSR outperforms EfficientZero in 17 out of 26 tasks in the Ataril00k benchmark.
6 CONCLUSION
This paper presented Temporal Change Sensitive Representation (TCSR), a self-supervised auxil- iary task designed for reinforcement learning algorithms that train a dynamic model. We enforce the temporal difference between unrolled latent state representations to be consistent with the temporal difference between target latent state representations. Calculating the difference between two con- secutive states is similar to taking derivative, which is common practice in the field of Mathematics and Physics when trying to learn the dynamics. The results show that our method can help agents better capture critical information in the latent state representations and better unroll those latent state representations. With the help of representation learned by TCSR, the EfficientZero backbone is able achieve mate-of-theearr performance in the Atari 100K benchmark. Possible future tension of TCSR includes applying it to different losses other than SimSiam and higher order of temporal differences.
REFERENCES
Pieter Abbeel, Morgan Quigley, and Andrew Y Ng. Using inaccurate models in reinforcement learning. In Proceedings of the 23rd international conference on Machine learning, pp. 1-8, 2006.
John Allman, Francis Miezin, and EveLynn McGuinness. Stimulus specific responses from be- yond the classical receptive field: neurophysiological mechanisms for local-global comparisons in visual neurons. Annual review of neuroscience, 8(1):407-44) 4300 1985.
Andrea Banino, Adrià Puidomenech Badia, Jacob Walker, Tim Scholtes, Jovana Mitrovic, and Charles Blundell. Coberl: Contrastive hert for reinforcement learning, 2021. URL https:
//arxiv.org/aael... 2aabb//2101 221.7 05431
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. 2020. doi: 10.
R2S50/ARRXIV.2006.099
htthttss:: httt::/aaaiivveao http://xwwv...orrab//////// 09882
8

---

Under review as a conference paper at ICLR 2023
Ting Chen, Simon Korblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. 2020a. doi: 1048550/AXXIV.0002057009 URL
httrg//aa////o https://arxiv: org/ abs 2002. 05709
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. 2020. doi: 10. 48550/ARXIV.201..00555 URLttts:
org/ /tt.///l...1.1.................. 10566
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum
contrastive learning. 2020b. doi: 1048550/ARXXI.2000.022977 URL https://arxiv,
arg/abs/2003 04297
Silvia Chiappa, Sébastien Racaniere, Daan Wierstra, and Shakir Mohamed. Recurrent environment simulators. arXiv preprint (oXxvv:704.0225555 2017.
Mare Deisenroth and Carl E Rasmussen. Pilco: A model-based and data-efficient approach policy search. In Proceedings of the 28th International Conference on machine learning (ICML-I1), pp- 465-472 Citeseer, 2011.
Chelsea Finn, lan Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. 2016. doi: 10448500ARXXV.11650071555 URL htttp://arxxv
org abs/ 1605. 07157
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Ghesh- laghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervised learning. 2020. doi: 10.4555WAARXIV220000733.. URL
abtpg///b////a/ 2tt.bs//20660777 07733
Zhaohan Daniel Guo, Bernardo Avila Pires, Bilal Piot, Bean-Basties Grill, Florent Altché, Rémi Munos, and Mohammad Gheshlaghi Azar. Bootstrap preeit-pre representations for multi- task reinforcement learning. pp. 3875-3886, 2020.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. pp. 1861-1870, 2018.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv: 2019a.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. pp. 2555-2565, 2019b.
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with dis- crete world models. arXiv preprint arXiv:2010.02195 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. 2019a. doi: 1048500ARXXIV.1111.5222 URL
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for
unsupervised visual representation learning. 2019b. doi: 10--550/ARXXI.1111..5222 URL
wtwp//////////ooo//ooo///aaa/saa https::/arxii www. abss 1911 05722
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski. Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-second AAAI conference on artificial intelligence, 2018.
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model-based reinforcement learning for atari. arXiv preprint arXiv: 0003.990330773 2019.
Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004 13649, 2020.

---

Under review as a conference paper at ICLR 2023
Sascha Lange and Martin Riedmiller. Deep auto-en neural networks in reinforcement learning. pp. 1-8, 2010. doi: 10.1 10500000NNN22
Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Rein- forcement learning with augmented data. Advances in neural information processing systems, 33: 19884-19895, 2020.
Humberto R Maturana, Jerome Y Lettvin, Warren S McCulloch, and Walter H Pitts. Anatomy and physiology vision in the frog (rana pipiens). The Journal of general physiology, 43(6):129, 1960.
Vincent Michalski, Roland Memisevic, and Kishore Konda. Modeling deep temporal dependen- cies with recurrent grammar cells. In Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014. URL http:tt:/prccceediiiii wtw.:eeeednngg...uupp....e...... wwwripuu..... cc/ paper/2014/E5 102001//file/////// cd89ffff offfffod 4d 90cb80035744411 (22p335777722222--222 Pap.rd.p-ppp...p......
Tung D Nguyen, Rui Shu. Tuan Pham, Hung Bui, and Stefano Ermon. Temporal predictive coding for model-based planning in latent space. pp. 8130-8139 2021.
Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard Lewis, and Satinder Singh, Action-conditional video prediction using deep networks atari games. 2015, doi: 10.48500/ARXIV1550.007755
/ttss:::::/ http://axxvvvvvo org/ /atb///107 08750
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic doi: 10.48850/ARXXI11180..0553
tive coding. 2018.
URL httpps://erxiv.orr///bb
1807.03748
Sergiu Oprea, Pablo Martinez-Gonzalee Alberto Garcia- Gariia, John Alejandro Castro- Vargas, Ser- gio Orts -rts-asco,ano, Jose Garcia-Rodriguee and Antonis Argyros. A review on deep learning tech- niques for video prediction. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2020.
Castro-Vargas,
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839))604 609, 2020.
Max Schwarzer, Ankesh Anand, Rishab Goel, Devon Hjelm, Aaron Courville, and Philip Bach- man. Data-efficient reinforcement learning with self-predictive representations. 2020. doi:
10.48550/ARXXV.2007.7 .05929. URL htt.thttt////. htttt:tt///aaaxt http://////xxvxoo/////// abs /207 05929
Evan Shelhamer, Parsa Mahmoudieh, Max Argus, and Trevor Darrell. Loss is its own reward: doi: 104855W/ARXIV.11120007,, URL Self-supervision for reinforcement learning. 2016.
htttps://arxi./ http://aaaad...oo////11112121 07307
Aravind Srinivas, Michael Laskin, and Pieter Abheel. Curl: Contrastive unsupervised representa-
tions for reinforcement learning. arXiv preprint arXiv:200..04136, 2020.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Daichi G Suzuki, Juan Pérez-Fernández, Tobias Wibble, Andreas Kardamakis, and Sten Grillner. The role of the optic tectum for visually evoked orienting and evasive movements. Proceedings of the National Academy of Sciences, 116(30):15222-158)) 2019.
Gerhardt Von Bonin and Percival Bailey. The neocortex of macaca mulatta.(illinoo) monogr. med. sci., 5, no. 4.). 1947.
Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Im- proving sample efficiency in model-free reinforcement learning from images. 2019. doi: 10.4850/ARXXXV1110.0144 URL
wttp:///ww..o/////////.../ 01741
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous con- trol: Improved data-augmented reinforcement learning. arXiv preprint 17X::,10770006 2021.
Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, and Yang Gao. Mastering atari games with limited data. 2021. doi: 10.48550/ARXTV2211.000 100220. URL
abs/ fos/2111.. 00210
10

---

Under review as a conference paper at ICLR 2023
Table 2: Hyper-parameters for TCSR
Parameter
Setting
Observation down sampling
96
Frames stacked
4
Fromes skip
4
Reward clipping
True
Terminal loss
Max Trames training episode
12k
Max frames per evaluate spisode
108K
Discount factor
0.997
Minihatch size
256
Optimizer
Optimizer: learning rale
0.2
Optimizer: momenium
0.9
Optimizer: weight
0.0001
Max gradieni norm
5
Priority exponent (n)
Priority correction (8)
0.4
Training steps
120K
Evaluation episodess
Min replay size for sampling
2000
Solf-play network updating inerval
100
Target network updating interval
200
steps
5
TD steps
5
Policy loss coctlicient
1
Value loss one fficient
0.25
Selervuee consistency loss coefficient
2
Temporal change consistency loss coefficient
Recurentt gradient scaling factor
1.0
LSTM borizontal length
5
Dirichlet noise ratio
0.3
Number simulations in MCTS
Reanalyzed policy ratio
0.99
A IMPLEMENTATION DETAIL.
TCSR is implemented on the EfficientZero source code. We followed EfficientZero's implementa- tion on the representation network, dynamic network, reward prediction network, value prediction network and policy prediction network. TCSR has one addition set of SimSiam predictor and pro- jector that have the exact same architecture those in EfficientZero. They are used maximize the similarity between the difference of two consecutive unrolled latent state representation and the difference of two consecutive target latent state representation.
The architecture of predictor in SimSiam network is as follow:
fully connected layers and 256 output dimensions. (BN ReLU)
fully connected layers and 256 output dimensions. (BN ReLU)
fully connected layers and 256 output dimensions. (BN)
The architecture of projector of SimSiam network is as follow:
fully connected layers and 64 output dimensions. (BN + ReLU)
fully connected layers and 256 output dimensions.
Some update had been made the Hyper-parameters in the EfficientZero source code since the publication of EfficientZero paper. TCSR also made minor adjustment to the Hyper-parameters. The Hyper-parameters are as shown in table The learning rate schedule is showen below.
(0.0002 step, 0 < step I 1000
learning rate
0.2,
1000 < step 100K
0.02
100K < step < 120K
B DATA AUGMENTATION
We followed EfficientZero on the implementation data augmentation. Some details regarding the implementation of data augmentation isn't mentioned in EfficientZero paper, but believe these
11

---

Under review as a conference paper at ICLR 2023
details are extra important for TCSR. So we will discuss them here. The data augmentation of EfficientZero consist two part, random shift and intensity, which is defined as follow.
Random Shift
Pad the input image of size 96 96 to 108 x 108 by repeating boundary pixels for 4 pixels. Then random crop the image back to 96 x 96.
Intensity
Generate a random scalar s by 8 1-0.05 10.05-cllmmpy. -2, where ~ (0, 1). Multiply the value on each pixel of input image by
In the main part of this paper, mentioned concept called the "parameter" of augmentation. By "parameter" we mean the crop position and the value scalar s. We say o₃ and ō'41 share the same augmentation parameter when they share the same crop position and scalar s. We say ô and og have different augmentation parameter when the crop position and scalar are sampled separately.
The "parameter" of augmentation is extra important for TCSR. Suppose have two consecutive observation On and 0++1- We augment them with different augmentation to get ó, and 0 and encode them with the representation network h to get & and 8,41,0- If we calculate the difference between 9; and 11+1.0. the result is meaningless since the difference introduced by the augmentation could be bigger than the temporal change.
C DECODER
A decoder is used to demonstrate the quality of unrolled latent representation predicted by TCSR. The architecture of the decoder is as follow:
transposed residual block with 64 planes. (BN + ReLU)
Up sample with nn innectionllinterpollan output size 12 x 12, mode: nearest.
transposed residual block with 64 planes. (BN + ReLU)
Up sample with nn. www.funnnna..iiiiiooo.oo inntrionalinterpolaate, output size 12 x 12, mode: nearest.
transposed residual block with 64 planes. (BN ReLU)
transposed upsample residual block with stride 2 and 32 output planes (BN ReLU)
transposed residual block with 32 planes. (BN + ReLU)
transposed upsample residual block with stride and 12 output planes (BN + ReLU)
12


Reviews:
[{"review_id": "kaHUz9HXNM", "reviewer": "Reviewer_1Xom", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose TCSR; they perform experiments to show that it improves upon EfficientZero and achieves SOTA performance on the Atari100K benchmark.\n", "summary_of_the_review": "This is an interesting and well-motivated paper studying an important topic.  However, the contribution is entirely empirical, and the experiments are deeply flawed.", "strength_and_weaknesses": "This is an interesting and well-motivated paper studying an important topic.\n\nHowever, the contribution is entirely empirical (the contribution boils down to the proposal of TCSR, and the claim that TCSR “achieves SoTA performance”), and the experiments are deeply flawed.  The results are based on 3 runs, which is unacceptable in RL, due to the high variance between runs.  This topic has been repeatedly covered in publications, talks, etc. in recent years, so I will not elaborate further.  It is interesting to notice, however, the fact that the “reported EfficientZero” score greatly outperforms the TCSR score.  The large discrepancy between the “reported EfficientZero” score and the “source code re-run EfficientZero” score likely comes from the low number of runs used.  This discrepancy is an excellent illustration of the problem with only using 3 runs.\n\nClarity issues:\n- The loss functions that are in the definitions of the MuZero loss are not defined (nor is it easy to infer what they are).\n- Similarly, the SimSiam loss is not defined.\n- g_{state} is not defined.\n- The difference between s^ and s~ (defined at the end of section 3) is not clear.  They “each represent an augmentation of a parameter”.  So there are two parameters to augment?  (I do not think that’s correct, but I do not know how else to read it.)\n\nMinor edits:\n- “reward is sparse(Shelhamer et al., 2016)” (missing a space)\n- “The training pipeline is as shown in Figure 3” (missing period)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Vvoe5hbT-H", "reviewer": "Reviewer_JdZQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new auxiliary loss for learning representations for reinforcement learning from images. The loss penalizes the difference between the change of two subsequent latent states for one image augmentation and the change of two subsequent states for a different augmentation. By integrating this loss into \"EfficientZero\" (Ye et al. 2021) the performance on Atari benchmarks good be improved.\n", "summary_of_the_review": "The contribution of the paper is incremental, but could be somewhat interesting.\nHowever, the communication of the work is substandard, and the paper would require a major revision by:\n- describing the proposed method in a more self-contained way,\n- providing an in-depth description on the experimental procedure,\n- improving the structure of the paper,\n- and, ideally, investigating the discrepancy between the reported results of EfficientZero, and the results obtained in the current experiment (I recommend contacting the authors).\nFurthermore, providing the source code for reproducing the experiments would be highly appreciated.\n", "strength_and_weaknesses": "Strength:\n- The paper presents a novel and reasonable auxiliary, that achieves good empirical results.\n\nWeaknesses:\n- The contribution is rather small, as the modification of the previous method is only incremental and not theoretically justified.\n- The presentation is sloppy.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "TqKPGtoknSD", "reviewer": "Reviewer_VxAy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a method named TCSR which is based on EfficientZero. TCSR helps a reinforcement learning agent to learn better latent representations in a self supervised manner.", "summary_of_the_review": "The paper has several weaknesses and it is not ready for publication. See weaknesses for justification.", "strength_and_weaknesses": "Weaknesses: \nThe paper has several weakness in writing, results, explanation of the approach. See below: \n\nResults: \n- The paper does not evaluate the proposed method with significant experiments. Only Table 1 shows results on Atart100K which shows improvement in 11 out of 26 games as compared to the author's claim of 17 out of 26. These results are also not consistent with Figure 2. The table results are also not explained. \n- The visualization shown Figure 1 is highly basic and any encoder should be able to encode the representations. Why many method are failing to encode it is not explained anywhere in the paper. \n- Training details are not provided, the results seems not reproducible with any implementation details. \n- The author's claim self-supervision. I didn't find any information which help TCSR to train in a self-supervised manner. \n\nMethod: \n- The approach is not detailed. The reader cannot understand the proposed method as many details are missing or not explained. For example, Figure 3 is never explained and it is the training pipeline. I am not able to infer what is the training part in this figure. \n- The notations are not clear. In s_t^k, s is state at time t but what is k?\n- The authors gave a good emphasis on consistent loss but it is not explained. an equation mentioned it in sec 3.2 but what is the loss? The equations are not numbered. \n- The TCSR loss in sec 4.1 is not clear. Why t+k time is the target for state at time t?\n\nWriting: \n- Sec 3.1, first paragraph third line, encodes a observation\n- Sec 3.2, represnetation\n- Full stop in last line of page 5, many places in sec 4.1. \n- Table:1 can be written as Table 1.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0jES9RkJeT", "reviewer": "Reviewer_yZGc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes Temporal Change Sensitive Representation, a self-supervised representation algorithm that seeks to capture the temporal representation changes in RL. Experiments on different datasets suggest that the proposed algorithm does outperform benchmark SOTAs in the Atari100k environment. ", "summary_of_the_review": "My rating would be a 3, though I'm happy to adjust my score if the authors address my concern. ", "strength_and_weaknesses": "Strength:\n\nThe paper is well-organized. Each section does what it aims to do. The algorithms are described very clearly.\n\nWeaknesses:\n\n1. The extent of novelty is relatively low. On a high level, the paper appears to be a combination of the existing EfficientZero algorithm(a temporal-aware variant of the MuZero algorithm over non supervised contents, with data augmentation and minor adjustments over similarity losses. More elaboration on novelty would be helpful.\n\n2. The motivation should be stated more clearly. While the experiments are adequate, there should be more explanation as to how/why each set of experiments correspond to the context/setting of the problem. Additionally, it would be helpful to provide some visual illustrations(or other qualitative demonstrations) as to why the algorithm works, as opposed to simply listing the performances SOTA and conclude accordingly. \n\n3. Would it be possible to incorporate experiments on other datasets? Empirical results solely on Atari-based datasets seem a bit inadequate, especially when the algorithm seeks to address representational learning for reinforcement learning problems in general.   ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject