Time-Myopic Go-Explore: Learning A State Representation for the Go-Explore Paradigm
Authors: Marc Höftmann
Jan Robine
Stefan Harmeling
Venue: Submitted to ICLR 2023 (2023)
Keywords: Exploration
Self-Supervised Learning
Go-Explore

Abstract: Very large state spaces with a sparse reward signal are difficult to explore. The lack of a sophisticated guidance results in a poor performance for numerous reinforcement learning algorithms. In these cases, the commonly used random exploration is often not helpful. The literature shows that this kind of environments require enormous efforts to systematically explore large chunks of the state space. Learned state representations can help here to improve the search by providing semantic context and build a structure on top of the raw observations. In this work we introduce a novel time-myopic state representation that clusters temporal close states together while providing a time prediction capability between them. By adapting this model to the Go-Explore paradigm (Ecoffet et al., 2021b), we demonstrate the first learned state representation that reliably estimates novelty instead of using the hand-crafted representation heuristic. Our method shows an improved solution for the detachment problem which still remains an issue at the Go-Explore Exploration Phase. We provide evidence that our proposed method covers the entire state space with respect to all possible time trajectories — without causing disadvantageous conflict-overlaps in the cell archive. Analogous to native Go-Explore, our approach is evaluated on the hard exploration environments MontezumaRevenge, Gravitar and Frostbite (Atari) in order to validate its capabilities on difficult tasks. Our experiments show that time-myopic Go-Explore is an effective alternative for the domain-engineered heuristic while also being more general. The source code of the method is available on GitHub.

Under review as a conference paper at ICLR 2023
TIME MYOPIC GO-EXPLORE: LEARNING A STATE REP- RESENTATION FOR THE GO-EXPLORE PARADIGM
Anonymous authors Paper under double-blind review
ABSTRACT
Very large state spaces with sparse reward signal are difficult to explore. The lack of sophisticated guidance results in a poor performance for numerous reinforce- ment learning algorithms. these cases, the commonly used random exploration is often not helpful. The literature shows that this kind of environments require enormous efforts to systematically explore large chunks of the state space. Learned state representations can help here improve the search by providing semantic context and build a structure on top of the raw observations. In this work we intro- duce novel time-myopic state representation that clusters temporally close states together while providing time prediction capability between them. By adapting this model the Go- Go-Explore paradigm (Ecoffet et al., 2021b), we demonstrate the first learned state representation that reliably estimates novelty instead of using the hand-crafted representation heuristic. Our method shows an improved solution for the detachment problem which still remains an issue at the Go-Explore Exploration Phase. We provide evidence that our proposed method covers the entire state space with respect to all possible time trajectories without causing disadvantageous conflict-vveraps in the cell archive. Analogous to native Go-Explore, our approach is evaluated on the hard exploration environments Montezumaaevenge, Gravitar and Frostbite (Atari) in order to validate its capabilities on difficult tasks. Our experiments show that time-myopic Go-Explore is effective alternative for the domain -enniin-e heuristic while also being more general. The source code the method is available on GitHub: made. publiieleee fiolii... after.... acceptance.
Keywords: Exploration, Sepf-Super Learning, Go-Explore
INTRODUCTION
In recent years, the problem of sufficient and reliable exploration remains area of research in the domain of reinforcement learning. In this effort, an agent seeks to maximize its extrinsic discounted sum of rewards without ending up with sub-optimal behavior policy. A good exploration mechanism should encourage the agent to seek novelty and dismiss quick rewards for a healthy amount of time evaluate long-term consequences of the action selection.
Four main issues are involved when performing an exploration in a given environment: (i) catastrophic forgetting (Goodfellow al., 2013) as the data distribution shifts because the policy changes, (ii) neural network's overconfident evaluation of unseen states (Zhang et al., 201 8), (iii) sparse reward Markov Decision Processes and (iv) the exploration-exploittiie trade-off (Sutton and Barto, 2018; Ecoffet et al., 2021a). The latter causes significant problem: the greater the exploitation the less exploration is done. Hereby, making novelty search less important when the agent can easily reach states with large rewards.
To address these difficulties, Ecoffet et al. (2021b) propose a new approach called Go-Explore and achieve state-of-the-art results on hard exploration problems. However, Go-Explore relies on hand-crafted heuristics. our work we replace their discrete state representations with learned representations particularly designed to estimate elapsed time to improve the novelty estimation. The time distance between two states is used to build an abstraction level on top of the raw observations by grouping temporally close states. Equipped with this capability, model can decide about the acceptance or rejection of states for the archive memory and maintain additionally exploration statistics.

---

Under review as a conference paper at ICLR 2023
Preprocess: 3x Gray + Stack
Siamese encoder
(Oo, 00..0-11:0-2 0t-2
Cona RAI
Fattan
Conw
(O O++i-11 2-
Guen Md.V
Time prediction nelwork
Nomalization
Dat
Dorser
Figure Model architecture. For details see Section 3.
Contribution. This work attempts to improve the exploration problem by introducing learnable novelty estimation method that is applicable to arbitrary inputs. In the experiment section, we demonstrate the reliability of method by comparing its results with native Go-Explore and more broadly with several other exploration- and baseline approaches. The main contributions of this paper the following:
1. We introduce myopic prediction network which learns cooblem-specifee representations that are useful for time prediction. The time distances are later used to determine novelty which generate üime-dependent state abstraction.
novelty estimation method consisting of a siamese encoder and time-
2. We implement a new archive for Go-Explore that includes new insertion criterion, a novel strategy count cell visits and new selection mechanism for cell restarts.
2 PROBLEMS OF GO-EXPLORE
Go-Explore maintains archive of saved states that are used milestones to be able to restart from intermediate states, hereby prevent detachment and derailment (as discussed Ecoffet al. (2021a)). It generates its state representation by down-scaling and removing color (see Figure 2 left). The exact representation depends on three hyperparameters (width, height, pixel-depth), which have to encoding they
be tuned for each environment. If two distinct observations generate the considered similar and dissimilar otherwise. Thus, all possible states are grouped into a fixed number of representatives, which leads to overlap-conflicts when distinct observations receive the same encoding. In these cases one of the states has to be abandoned, which might be the reason that for the Atari environment Montezuma's 's Revenge only 57 out of 100 runs reached the second level in the game (as reported in Ecoffet et al. (2021a)). We conjecture that their replacement criterion favors certain state over another, so the exploration will be stopped at the abandoned state. This can result in decoupling an entire state subspace from the agent's is exploration endeavor.
related issue emerges by ignoring the tpatio-temporal semanties between states that are grouped together into single representation. The down-scaling method is just compressing the image information without considering its semantic content. The consequence is a replacement criterion that resolves archive conflicts between states that are neither spatially nor temporally in close relationship. Therefore a conflict solver has to resolve illogical and non-intuitive conflicts, because in these cases it is not obvious which state should be favored. We have no information about which potentially reachable states are more promising. On top of that, the cell representation is only suitable for states represented by small images. If we have images with higher resolutions the representations
2

---

Under review as a conference paper at ICLR 2023
Scaling
(4, dp)
(left panel) Go- Go-Explore down-scaling operation with hyperparameters (h, dp). (right panel): time distances are directed. Jumping off the cliff faster, than hiking up (image from IHC).
Figure 2:
could get either abstract (and hereby limiting the exploration progress) or too narrow resulting in an overflowing archive. Moreover and from general point of view, it is not clear how we could usefully down-scale an input observation that consists of partial or no image information at all.
3 LEARNING TIME-MYOPIC STATE REPRESENTATIONS
The basic idea of Go-Explore is to save intermediate results, so that the environment state can be restored from memory try new action sequences and obtain different outcomes. However, the difficulty is to decide when to store a state for returning, Go-Explore solves this by a heuristic that lead to some non-in non-intuitive decisions when dealing with representation conflicts discussed in the previous section.
In the following we show how state representations can be learned that are particularly designed to predict the time progress. Equipping Go-Explore with this model will avoid archive conflicts and gives better control over the novelty estimation problem. We call our method time-myopic since trained to predict the elapsed time only up to particular distance.
3.1 PREPROCESSING AND SIAMESE ENCODER
As a first step, we preprocess for every timestep a stack of the last three RGB observations Ot, Oz-1, O--2- For this we convert each observation into single gray-scale frame generate compressed input format (see Fig. l). The result o, consist of three color channels where each channel represents one gray-scale observation. This format captures time-critical information and allows us to infer properties such velocity and acceleration of objects.
Given preprocessed observation oG, we apply a siamese convolutional neural network (CNN) 4g to obtain a latent vector encoding &t
(o.) = 2z-
The exact architecture of the CNN 49 is shown at the top right in Figure 1. The goal is to learn useful embedding that captures the information accurately predict the elapsed time between two given observations o; and ō,+. We choose siamese networks since they demonstrated good results (Koch, 2015; Ermolov and Sebe, 2020; Chen et al., 2021) distinguishing deviations between distinct inputs or respectively capturing similarity.
3.2 TIME-PREDICTION NETWORK
After the encoding procedure, pair of observation encodings (2,, 21+k) allows us to estimate the elapsed time between the elements of pair. For this, we feed the difference between 2; and 2++* into another colle-condeeted multi-layer neural network fe that outputs real number. The whole time prediction network is written as V and includes a normalization (see bottom right Fig, I)
V (e **+R) - e max(fa(t
E [0.
(2)
In Equation (2) the model predicts the time distance from the starting point 2, to the destination ZrH where swap result in different outcome. We obtain the myopic property normalizing
3

---

Under review as a conference paper at ICLR 2023
the output with the function g(x) - that allows to specify a prediction range [0, L] for the normalization interval [0, 1]. When we want to obtain the predicted time distance we need to multiply the normalized output with the upper bound L and vice versa for converting distance & to the internal representation
k R L (e((z,z(4H))
3.3 Loss FUNCTION AND ITS IMPLICATIONS
The time prediction loss objective Lti minimizes the mean squared enror between the predicted time distance and the true distance ks
L time (0) E, 2(+k) -int///))))m
where the correct distance k is known from the sampled trajectories after the environment interaction. The neural network computation includes the normalization g(æ:) prevent exploding gradients and introduces the upper time window bound I. that limits the network's 's expressivity beyond T. timesteps. This constraint simplifies the task, since correct forecasts of longer periods of time become unimportant (Makridakis et al., 2018). The myopic view is helpful to reliably estimate novelty, because the model does not need handle a precise time measurement between states that are far away from each other. By considering only local distances, the predictions get easier and they are more reliable. So when the network encounters a state-pair with a large distance L, we do not care about the exact true distance. But instead, the model should indicate that the states are far away from each other by outputting the upper bound T..
3.4 TIME DISTANCE IS POLICY DEPENDENT AND DIRECTED
Each singular timestep represents a state transition on the underlying MDP where the time-related reachability is dependent on the current policy. This means that two distinct policies could reach a certain state with a different number of timesteps. Also, since time distances are directed we have to input our encodings (z,, =B) in the right order into Va. For intuition, Figure 2 (right) shows visual example where a state
on top of small cliff and point B is at the bottom of it. To reach B from A an agent could use the shortcut and jump down the cliff, but it might not be possible to jump back up. Therefore the agent needs find a different path that might increase the elapsed time to reach A from B. For that reason we assume that the distances of our time prediction function can yield different results for V (z ²B) and Vo (=(BBZA))
4 TIME-MYOPIC GO-EXPLORE
To integrate our learned representation model from the previous section into the native Go-Explore method, we have to make some adjustments, since our state encodings are continuous (and no longer discrete). This makes harder to decided similarity between two distinct states. In the following we explain how this is achieved.
4.1 ARCHIVE INSERTION CRITERION
The time prediction capability of D allows us to propose new archive criterion. Our criterion is an insertion-only method which adds a state the archive when it is novel enough. this way the detachment problem is entirely solved by not abandoning archive states. We initialize the archive by inserting the encoding 2e the environment's 's start state. This entry will begin the exploration effort by searching for novelty around the starting point. In order to do that the agent samples trajectories from the restored state ej where we collect new states that we call archive candidates. A new candidate encoding øх is evaluated with every archive entry 2c by the time prediction function. The predicted value (0(2c, ²x) is thresholded by some hyperparameter Ta to determine the acceptance rejection of the current candidate. If the threshold is surpassed for every comparison, the candidate K is added to the archive. Note, that we choose the time distance direction V V zx) since we want to move away from the archive entries. When this happens, we conclude that the agent has reached a currently unknown part of the state space which has a sufficient novelty with respect to the archive-known subspace. this way. the candidate evaluation gains a global view on the exploration progress instead of considering local trajectory- perspective (Badia et al., 2020; Savinov et al., 201 8). A short example is shown in Table
4

---

Under review as a conference paper at ICLR 2023
Table Archive insertion criterion where the model's expressivity the time window [0, 7. 20]. A new candidate ž is compared to all existing cells in the archive. If one cell is too close, the candidate rejected.
Comparison Candidate
Time estimation
Evaluation
Cell
(= =k)
20 Ta 13
Insert?
Bc1
2
0.3934
7.86
no
R0z ZK
2K
0.7568
15.13
yes
Zcs
0.6321
12.64
n0
Zca
ZK
0.9334
18.66
yes
2K
4.2
VISIT COUNTER
To ensure successful progression in the exploration, it is common practice to track the number of cell visits Cvisits for every cell C in the archive. With respect to the return selection, the visit counter of a cell increases by when it is selected as starting point. Furthermore, the visit counter increments when the time distance to an archive candidate is lower than visit threshold T.. This evaluation runs simultaneously to the application of our insertion criterion where we compute all time distances to the candidate.
4.3 CELL SELECTION CRITERION
Native Go-Explore made the design choice to replace cell entries with small scores to states with larger scores. This significantly boosts the exploration by abandoning states which might have been sufficiently explored. Time-myopic Go-Explore does not have this ability, because the state representations continuous and the encodings of two states with weak spatio-temporal semantics are far away from each other. Also, our method should not overwrite archive entries, because they could be inserted again with clean visits statistics. To bias exploration towards cells with larger scores, we calculate and combine the native selection weight vis with our score weight W Weooee that contains the reached cell score Cscore (sum of undiscounted cumulative reward). Both quantities (Cvisits: Cscore) are stored in the archive for every cell C.
Cwere
W
max
(5)
Viiists +I
max Oxcone
visit weight Wesn
score weight Wyeere
The outer maximum in W W.core ensures that states with lower scores are explored as well (with chosen hyperparameter a = 0.075). The inner maximum normalizes the experienced scores between zero and one.
TECHNICAL DETAILS FOR TRAINING
In order to improve the model quality we explain next three additional training routines that enhance time prediction of our model.
5.1 SIMILARITY AND DISSIMILARITY
Optimizing only on close-by pairs (e.g. with time distance A S L) will lead wrong estimates for distant pairs (i.e. A > L). which might be the majority of cells. Thus we have to ensure that dataset includes also dissimilar pairs where the observations are at least L timesteps away from each other. The distance of a dissimilar pair is trained on the value that corresponds to the maximal time estimation L, which does not represent the actual distance. Instead signals sufficiently large temporal space between them. In this way, we try to find encodings such that the embedding region around a state only includes other encodings that are within the defined time window [O,.... L].
5

---

Under review as a conference paper at ICLR 2023
5.2 AVOIDING TEMPORAL AMBIGUITY
Suppose there are two observations ōttk, and 0t+k, that happened at different points in time, but which are pixel-wise identical. To define a single distance to some other observation ot, we will use the minimum of 11 and 22. To quickly identify pixel-wise identical observations we are using the MD5 hash function (Rivest, 1992).
5.3 PROXY CELLS AND LOCAL DATASETS
Usually the time prediction model would be trained on trajectories that always start with an archive state. To extend the training data we additionally generate trajectories (only for training) that start from so-called proxy cell states that are temporally close to an archive cell. We collect these proxies by choosing randomly states in the time distance interval [Tp-ow Tphhigh] to their respective archive cell and replace them periodically.
To further increase variability of the dataset, we add small local datasets for each cell that add some data points within its proximity. Every dataset holds a few hundred pairs where we store the time distance between the cell state and a temporally close state. This necessary because otherwise our network would forget about certain cells and their neighboring states since they are not visited anymore due to the selection weighting W W Wisiss
W score-
6 EXPERIMENTS
The experiment section covers the following topics: in Section 6.1 we study the encoder properties and assess how well the time can be predicted. In Section 6.2 we compare our approach. the original Go-Explore and other related methods. In Section 6.3 we analyze the archive for the native and our proposed time-myopic Go-Explore
6.1 WHAT DOES OUR MODEL LEARN?
To demonstrate the properties of our model, we re-create an experiment from Ermolov and Sebe (2020) which is shown in Figure A representation-leamnie the Atari environment Montezuma's Revenge where the training data is gathered by a random actor the environment's start state, After optimization, the network is asked encode the observation sequence from the trajectory shown in Figure 3 (top-left). Most of the encodings are extrapolated, because random policy is not able to reliably execute the action sequence that generated the observations. Therefore an extrapolation starts roughly between the checkpoint 3 and 4. In this experiment our model uses 9k observations which are sufficient to show a good result. The 32- dimensional encodings are projected with the t-SNE method into two-dimensional space where it uncovers the interesting relationship between the data points. Temporally close states are grouped together and are strung thread while the sequence unfolds. The distance between two imminent states does not collapse and provides useful semantics for time measurements. Remember that the chosen trajectory has greater meaning for model and it is perceived as arbitrary like any other possibly selected sequence. Subsequently, the rich structure is also present when performing PCA on the encodings (see Appendix Figure 5).
network is trained 400k observations from
The time evaluation demonstrates good results where the network has training data. However, the extrapolation capability on time distances is limited. But surprisingly even places unseen states close to their temporal neighbors resulting in local semantic integrity. At some point the extrapolation of time distances becomes unreliable which will improve with more novel data for optimization. Once the network was trained on more data during complete run, the prediction quality gets lot better (see Appendix Figure 6).
that the encoder
6.2 COMPARISON ON HARD EXPLOR ATION ENVIRONMENTS
We evaluate our method on the hard exploration environments Montezuma's 's Revenge, Gravitar, Frostbite (Atari) and compare it with (i.) related methods (ii.) and native Go-Explore. We prepare our experiments by adjusting the natively used random action-repeating actor (Ecoffet et al., 2021a). Time-myopic Go-Explore decreases the actor's action repetition mean μ from 10 to 4. This is necessary since the native method does not care about learning a state representation while our learned model depends robust data collection for the time predictions. Native Go-Explore has

---

Under review as a conference paper at ICLR 2023
131
20.0
Strrtpostton:
Strtposiäon: 10
17.5
Statposiioon 20
Starpazition. 60
Starrposttion 50
Starppoitton:: 60
Staripostion: 70
25
0.0
2
4
R
1O
Figure 3: Visualization of the model performance. The top-left image is an observation sequence from the Atari environment Montezuma's Revenge, which is generated by human demonstration. The top-right image is the t-SNE (van der Maaten and Hinton, 2008) projection of the observa- tion encodings generated by the siamese CNN dg. The bottom image shows time estimates of our prediction network T for L 20. This model has the ability to calculate the time dis- tance for pair of observations. E.g. we can select several encodings 20, 210, 220: 240- 250: 260 270 and let the time prediction network calculate their distance to their successors. Note that for the blue graph, we plot Va(zo, 20), Vo(zo, 21), ) (z 22), ..,, Va(2o, 272). For the orange one we plot V (1 210), V Vo(210- #11), V Vo 212); .... (9 2-2)-
Table Mean cumulative reward for Atari (rows are copied from Kim al. (2018), all others from the cited papers). This table shows the performance of different exploration-base and bascline methods. Our results are computed as the mean over 20 runs where each run has seen 5M frames.
Method
Frames
Montezuma
Gravitar
Frostbite
R2D2 (Kapturowski et al., 2019)
10000M
2061
15680
315456
EX2 (Fu al., 2017)
50M
0
550
3387
AE-SimHash (Strehl and Littman, 2008)
50M
75
482
5214
ICM (Pathak et al., 2017)
50M
161
424
4465
RND (Burda et al., 2018)
50M
377
546
2227
EMI (Kim et al., 2018)
50M
387
558
7002
LWM (Ermolov and Sebe, 2020)
50M
2276
1376
8409
PPO (Schulman et al., 2017)
40M
42
737
314
Ours (Time-myopic Go-Explore)
5M
2090
3161
4476
Ours (Time-myopic Go-Explore)
IM
695
2533
3543
Go-Explore (native)
1M
2303
2130
11721
the advantage that it can act more greedily, because the representation heuristic is always reliable and therefore can be exploited by acting more risky. Also, native Go-Explore uses the originally recommended down-scaling hyperparameters (h = 8, 11, dp 8) for Montezuma's 's Revenge and the dynamic nown-scaling (recompute every 500k frames) only for the other environments (Montezuma's 's Revenge with dynamic down-scaling performs worse). For native Go-Ex Go-Explore using the official implementation. Our time-mimeen time-myopic Go-Explore variant runs on one A A100 GPU and needs roughly 8-10 hours for 5M frames depending the archive size.
7

---

Under review as a conference paper at ICLR 2023
Table 3: Comparison of the archive size for Montezuma's Revenge for scores 100, 400, 500, 2500.
Archive size per score
Method
100
400
500
2500
Native (h- 8, w 11, dp 8)
126.2
197.8
1812.2
2838.9
Time-myopic (L. = -20, Ta = 0.55)
160.5
202.5
450.0
566.0
Time-myopic (L -25, Ta = 0.65)
103.8
126.2
210.7
248.3
native Icomed
native leamed
0.0
1.0
0.0
0.2
0.0
3.0
mummum Lime dic.ance miihhbourigg
mian distamee Braw dissen cats
(a) Distance to clasest cell neighbor
(h) Mean distance to next three cell neighhors
Figure 4: Time distances between archive cells on Montezuma's Revenge. We generate two archives
where the first (orange) was constructed using Go-Explore down-scaling method (h 8, 11, dp = 8) and the second one (green) by our learned model predictions (I. = 20, Ta = 0.55). When both archives reach the score of 2500 we use a second optimized time prediction network (L 20) and compute all time distances between the cell observations for both archives.
The result shows the practicality of time-myopic Go-Explore since it is better than most of the methods for Montezuma's 's Revenge and Gravitar while it only has seen 2% of their frames (IM vs 50M frames). It also provides good performance on Frostbite surpassing PPO, RND, EX2 and ICM. Moreover, time-myopic Go-Explore able to keep up with the performance of native Go-Explore (using the domain domain-aligned representation heuristic) and even exceeds it within 1M frames for the environment Gravitar. As far as we know the results of Gravitar for 1M frames are state-of-the-ae
6.3 ABLATION STUDY
Next we provide closer look at the archive properties. First, we compare the difference in the archive size between native Go-Explore and our approach (see Table 3). The table shows the mean number of cells (20 runs) in the archive when the agent reaches certain game score [100, 400, 500, 2500] in Montezuma's Revenge. The native archive size explodes after reaching a score of 400 while our approach shows more stable progression. The data also validates smaller archive sizes result of increasing the hyperparameters (L, Ta). the Appendix, a figure shows more precisely when and how the prediction model decides to add archive entries. We observe in experiments similar pattern for the environment Gravitar which uses the dynamic down down-scaling heuristic. The native archives agglomerate enormous amount of archive entries. After 1M frames for 20 runs, the mean size results 7376 cells while time-myopic Go-Explore holds 381 entries and is achieving a higher score.
Secondly, both methods are evaluated on the similarity within the created archives (see Figure 4). The time distances between all cell observations will provide an interesting similarity measurement. The model predictions in Figure confirm that the native archive yields a lot more similarity than our approach. This leads inefficient exploration. because based on the cell selection criterion every cell needs to be sufficiently visited, no matter how few environment transitions are between them. Our archives cover the state space with less cells.
8

---

Under review as a conference paper at ICLR 2023
6.4 LIMITATIONS
Due to the queries to the archive, our algorithm runtime increases currently non-linearly with the number of entries. This makes long runs or runs where the archive grows fast computationally expensive. This happens, because our approach requires to compute time distances to every archive entry in order to evaluate new cell candidates or update the visit statistics. So, in the future, we will make these queries more efficient to make the method applicable to even larger environments and runs.
RELATED WORK
Hard exploration problems like the Atari environment Montezuma's Revenge are known for their large state spaces and sparse reward functions. A lot state-of-the-art reinforcement learning algorithms (Schrittwieser et al., 2020; Kapturowski et al., 2019; Schulman al., 2017) have difficulties in achieving a good performance on these tasks, so several ideas have proposed towards a solution. The literature contains several approaches related to our work that deal with unsupervised- and representation learning combined with intrinsic motivation and exploration.
Playing hard exploration games by watching YouTube (Aytar et al., 2018). In this work the authors introduce neural network architecture in order to learn a categorical time classification between two distinct observations. Their network used to generate an intrinsic reward signal to facilitate imitation learning of human demonstrations while we are using it to globally model the exploration process.
Solving sparse reward environments using Go-Explore with learned cell representation (Bjørsvik, 2021). This approach also extends the Go-Explore (Ecoffet et al., 202 method by replacing the representation heuristic with a learned state representation. They employ Variational Autoencoder (VAE) (Kingma and Welling, 2013) to encode every seen state into a latent vector space. Later on, the encodings are used for k-means clustering procedure where cluster center stands for an entry in the archive memory.
Latent world models for intrinsically motivated exploration (Ermolov and Sebe, 2020). This paper optimizes a siamese network that clusters temporal imminent states in the embedding space by minimizing the mean squared error between them. addition, there the need for an extra structure constraint in order to prevent a collapse of the representation to a constant vector. The resulting encodings are used for the generation of an intrinsic reward signal.
Episodic curiosity through reachability (Savinov et al., 2018). The paper introduces model architecture with a logistic regression capability differentiate between novel and familiar state pairs where novelty is defined by minimum time distance of A: steps. The complete model (including a siamese encoder and comparison network) only able to decide whether an encountered pair is novel or not; without the ability to evaluate it on continuous basis. This estimation is utilized to generate a positive intrinsic reward signal when a policy encounters new states.
Never give up: learning directed exploration strategies (Badia et al., 2020). The proposed episodic novelty module starts empty and fills itself with state encodings when the agent interacts with the environment. Every state gets evaluated by k-nearest neighbor criterion with the similarity measure of the Dirac delta kernel to compute the rewards with respect to the similarity distance. The goal is to insert many novel states as possible which then facilitate exploration.
CONCLUSION
To add flexibility to and possibly improve Go-Explore we studied how its representation heuristic can be replaced by time-pread time-predicting neural network. Experiments show that the new state representation able to track the global exploration effort and moreover recognizes ongoing progress for this task. In comparison native Go-Explore, our method can reduce the archive size and covers the state space with fewer cells. Applied to hard exploration environments, such as Montezuma'ss 's Revenge we observe good performance compared previous exploration-batt observations, even creating better results than Go-Explore on the game Gravitar. Overall, however our learned representation is not able to compete with native Go-Explore in terms of sample efficiency.
methods while using much fewer

---

Under review as a conference paper at ICLR 2023
REFERENCES
I heart crafty things - cliff. http:://heerrrrrfftychnn...c//// uploads 10001/09/c1i0 jpg. obtained and modified.
Y. Aytar, T. Pfaff, D. Budden, T. L. Paine, Z. Wang, and N. de Freitas. Playing hard exploration games by watching youtube, 2018. URL https::///// http:///axvvvv..ie
10005.11590 11592.
A. P. Badia, P. Sprechmann, A. Vitvitskyi, D. Guo, B. Piot, S. Kapturowski, O. Tieleman, M. Arjovsky,
A. Pritzel, A. Bolt, and C. Blundell. Never give up: Learning directed exploration strategies, 2020.
URL
06038888....
V. Bjørsvik. Solving Sparse Reward Enviromments Using Go-Explore with Learned Cell Representa- tion. http://urn.nb.rr hwt.:........o.......nooo.00.7.0 2021.
Y. Burda, H. Edwards, A. Storkey, and O. Klimov. Exploration by random network distillation. 2018. doi: 0..48550/ARXI..8810 12894. URL https: / arxiv. org/ org/abs/ abs/1810 12894.
J. Chen, Z. Yuan, J. Peng, L. Chen, H. Huang, J. Zhu, Y. Liu, and H. Li. Dasnet: Dual attentive fully convolutional siamese networks for change detection high-resolution satellite images. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 14:1194-120 2021. doi: 20009//ATAAAS.22000.3300. 2037733.
A. Ecoffet, J. Huizinga, J. Lehman, K. O. Stanley, and J. Clune. Go-explore: a new approach for hard-exploration problems, 2021a.
A. Ecoffet, Huizinga, J. Lehman, K. O. Stanley, and J. Clune. First return, then explore. Nature, doi: 10. 10.10388/41188000-001111 URL https://dbl http:///ooooorg//////// 1038 / /11///11186666 020-586600000102-220-2
59007847):580-556, Feb 2021b. ISSN 1476-4687.
A. Ermolov and Sebe. Latent world models for intrinsically motivated exploration, 2020. URL https://arxe //rxivv////ii//////////a/a o alss /000 02302.
K. P. F.R.S. Lii.. on lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 2(1 572, 1901. doi: (1110801177864411996656333
J. Fu, J. Co-Reyes, and S. Levine. Ex2: Exploration with exemplar models for deep reinforcement learning. 30, 2017. URL https: h h/poceedi////cooeigg.eee... neurips http:/ppppe/./////...... ffaff70e22e 9e837 (er
I. J. Goodfellow, M. Mirza, D. Xiao, A. Courville, and Y. Bengio. An Empirical Investigation of Catastrophic Forgetting in Gradient- Neural Networks. arXiv:1312.621113. 2013.
S. Kapturowski, G. Ostrovski, W. Dabney, J. Quan, and R. Munos. Recurrent experience replay in distributed reinforcement learning. International Conference on Learning Representations, 2019. URL https: / wwwp//wwnw.....eeee/pp/.. net www.umiidrrlrllrrrrorrirarrrirra JAq
H. Kim, J. Kim, Y. Jeong, S. Levine, and H. O. Song. Emi: Exploration with mutual information. 2018. doi: 10.48550/ARXIV.1110000
URL
www.//gg//bb///aeeee 01176.
D. Kingma and M. Welling. Auto-encoding variational bayes, 2013. URL htw.t:ttps/eeee org//bs//311..c 6114.
G. R. Koch. Siamese neural networks for one-shot image recognition. 2015.
S. Makridakis, E. Spiliotis, and V. Assimakopoulos. Statistical and machine learning forecasting methods: Concerns and ways forward. PLOS ONE, 13(3):1-26, 03 2018. doi: 10. 13. 1/journal. pone.0194889 URL ht htt:://oii.ooo org 10. 1 1371 / 1et..1111ornaaaii jowrnal..aaaaaaa pone. 0194889.
D. Pathak, P. Agrawal, A. A. Efros. and T. Darrell. Curiosity-driven exploration by self-supervised prediction. 2017. doi: 1A4A555/ARXII.1700..55533 1705. 05363.
URL https: htttps://arxtv.oo////bbbb
R. I.. Rivest. RFC 1321 - The MD5 Message-Digest Algorithm. DOI 10.17487/RFC1221 1992.
10

---

Under review as a conference paper at ICLR 2023
N. Savinov, A. Raichuk, R. Marinier, D. Vincent, M. Pollefeys, T. Lillicrap, and S. Gelly. Episodic curiosity through reachability, 2018. URL https::////.cc. http//aaxxv.........aaa/
02274.
J. Schrittwieser, I. Antonogiou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel, T. Lillicrap, and D. Silver. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604-609,, dec 2020. doi: 10. 18-00080/ URL https:://///. /ttp//oiiiio/ org.orr///oooooooooooo..oooooooo 00t.:/0...0000021111100000000000
Schulman, Wolski, P. Dhariwal, A. Radford, and Klimov. Proximal policy optimization algorithms. CoRR, 0bs/1707.003447 2017. URL http: peattp://arkee /tt//aaxiiv////////v////////aaaa org/abs 1777. 06347.
A. L. Strehl and M. L.. Littman. An analysis of model-based interval estimation for markov decision processes. Journal of Computer and System Sciences, 74(8):1309-1331 2008. ISSN 0022-0000. doi: sccienss/arrtccee///////00000000
200.0000000000.....0 URL https::/p//......ctt....p /wwp///www.////////////////w//.t sew.sscieiiee ssciencedii rect com/ 67. Learning Theory 2005.
R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018. URL htt.tthttiinnnteee http:tpp//icccmmeeeeeeeeaa net. htt./boo.//he--boooo-e2222 html.
L. van der Maaten and G. Hinton. Visualizing data using t-SNE, Journal of Machine Learning Research, 9:2579-2605, 2008. URL http://wwet wwww jmlr org/papers/v9/ vandermaaten08a.Be
C. Zhang, O. Vinyals, R. Munos, and S. Bengio. study on overfitting in deep reinforcement learning, 2018. URL htt.tttts://aaiiiii org/ abs 1804 06893.
A APPENDIX SECTION 1
Table 4: Hyperparameters for the experiments reported in the hard exploration table.
Hyperparameter
Symbol
Value
(Moniccomseevangg Gravilr. Frosthite) Frabila.Fnaabbie))
Learning rate
le
Batch size
bs
64
Time window
(20, 25, 25) (0.6, 0.75, 0.75)
Distance threshold
Ta
Visit threshold
To
0.65
Proxy cell interval
[Tp-oow I p-ighh
[0.45, 0.75] 40
Exploration steps action repetition mean
(a) dim 1-2
(b) dim
(c) dim 1-3
Figure Principal Component Analysis for the trajectory visualization.
11

---

Under review as a conference paper at ICLR 2023
12
13
20.0
Start:
Start: 20
30 30
15.0
Start:
Start:50 50
Start: 60
Start:a0
10.0
Start; 90
7.5
Stärt) 110
2.5
20
60
100
120
P
S
12
13
Figure 6: Sophisticated model and its capabilities longer trajectory. The shown model is trained on data that surpasses the shown trajectory (top-left). Again, we can see the good encoding property and an improved time prediction skill. The predicted times around the timesteps 90-100 in the image at the Box 11 are accurate. At that point the agent dies and the enviromment generates repeating frames (two-image sprites) for around 10 frames. This prediction behavior happens, because we remove temporal ambiguity between state-pairy and try to calculate the shortest distance for it. The event can also be seen in the t-SNE visualization, when light-green points start to form a cluster.
Figure 7: Creation of archive entries. We manually looked through the cell observations an archive and searched for similarity. This figure shows all cell observations where the agent stands the same position and already collected the key. We can see that the time prediction network does not allow duplicates the archive and keeps a reasonable distance between the observations where the white skull is changing positions. Moreover the archive holds no observation where the agent just slightly moved in these situations.
12

---

Under review as a conference paper at ICLR 2023
Algorithm Go-Explore with a learned state representation
Initialize archive, dataset, agent, network for iteration 1, 2, ... do
Let the agent act timesteps in the environment starting from the selected and proxy cells
Collect data and optimize time w.r.t. Recompute all archive cell representations: 2c (g(ōc))
for each trajectory 1.,,... N do
Transfer all observations 01, -... Ot into the latent representation 21, Zt Compute all necessary time distances V (Ze, 21,....) Apply archive insertion criterion a candidate w.r.t threshold Ta Increase cell visits w.r.t. threshold T, Collect some proxy cells w.r.t. threshold TTpllow:Tp-high if candidate is accepted then Add candidate to archive end if end for end for
13


Reviews:
[{"review_id": "tKM_dv010z2", "reviewer": "Reviewer_7Zd7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes new learned state representations that can improve exploration by introducing time predictive semantics into the state representation. The authors propose a new notion of novelty based on time prediction representations and use this to create a new insertion criterion, a new cell count strategy and a new restart strategy for exploration in the settings of Go-Explore.", "summary_of_the_review": "This paper introduces time predictive state representations and uses them as notion of novelty using in the go-explore context to store states and as different visitation counts, and new cell restarts, the paper provides an interesting perspective but lacks in clarity and scientific rigour. Further experiments would be needed to support the claims and provide a scientific understanding of the contributions proposed.\n", "strength_and_weaknesses": "Strengths:\nThis paper focuses on a very relevant problem in exploration of state representation and introduces a novel and original way of measuring novelty.\nWeaknesses:\nThe paper lacks scientific rigour. The authors show experiments in table 2 with different frame counts making it very hard to compare among results. And even for the 1M frame budget the Go-Explore native seems to perform better. It is also unclear how precise the time predictive network is for each k, under each game. Is the time prediction correct on new seeds? One of the main arguments for the inadequacy of Go-Explore is that the overlapping conflicts lead to states being abandoned and not further pursued for exploration, how much conflicts is taking place quantitatively and does this lead to a significant performance drop, with the results presented so far it is still unclear this is true and time prediction mitigates this.\nThe paper is also not very clear, for example for the cell selection criteria Cvisits and Cscores is not defined, not even in the text. (see questions below for more)\nThe motivation for starting in proxy cell states is not clearly explained as well, how and where would this help/impact.?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "O4ZKoVC3WNO", "reviewer": "Reviewer_Ze5a", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a modification of the Go-Explore algorithm to remove the need for hardcoded state representations (which are used in the original version of the algorithm). Specifically, this work proposes to learn an embedding using an auxiliary loss based on predicting the number of timesteps connecting two observations. This timestep prediction function is also used to populate Go-Explores's goal archive: specifically, states are only added to the archive if they are at least a certain distance from all other states in the archive. This is to ensure that new states/goals are only added if they expand the frontier of explored states. \n\nThe proposed method is evaluated on 3 hard exploration Atari games: Montezuma's Revenge, Gravitar and Frostbite. It doesn't quite match the original Go-Explore, but it does decently well compared to other baselines in terms of performance and sample complexity. ", "summary_of_the_review": "Overall, I think that the paper's goal of making Go-Explore more general is a good one, and there are some interesting ideas in this paper. However, I believe it needs a good amount of work before it's ready for publication, specifically in terms of its clarity and its experimental results. ", "strength_and_weaknesses": "## Strengths\n- Although the original Go-Explore algorithm achieved impressive performance, it used several cheats like using handcrafted state abstractions (based on downsampling images), resetting the environment (or equivalently, exploiting the determinism of the simulator) which are not generally applicable. It would be great to have a version of Go-Explore which keeps the good qualities and performance but does away with these hacks, which this paper tries to make progress towards. \n\n## Weaknesses\n- As far as i can tell, this paper still assumes a deterministic simulator or the ability to reset to previously visited states, which is still a very strong assumption. So the version of Go-Explore it proposes is still not generally applicable. \n- The paper is not very clearly written - see my comments below\n- The results are decent, but it is currently unclear how the proposed algorithm scales with more data (only performance for 1M and 5M are reported). Please report performance at least until 50M so we can compare to the other model-free approaches. \n\nDetailed comments:\n- Sections 1 and 2 are confusing. It was not clear to me until Section 4.1 what the point of predicting the number of timesteps k connecting two different states was. Section 2 is very confusing, with a lot of vague statements such as \"The conflict solver must resolve illogical and non-intuitive conflicts\" or \"The down-scaling method is compressing the image information without considering its semantic content\". Couldn't this be said about many different unsupervised learning methods? In the first paragraph, it seems to be saying that state abstraction is a bad thing, and that the fact that many states are grouped to the same representation is a bad thing. Can't this also be a good thing in that it filters out irrelevant details? It all depends on _how_ the grouping is done. I would suggest providing a detailed motivating example (from Montezuma's Revenge perhaps) that illustrates all of these claimed problems with Go-Explore - having a concrete example illustrating these statements would make this part much clearer. \n\n- It's not clear to me whether the timestep predictor is even necessary. Since you have access to the actual time step, and you are restarting exploration from different states in the archive, you should at least have the number of timesteps between a given state and the state the trajectory was restarted from. What happens if you use this as your archive insertion criterion? It might not work but should be a baseline to justify the need for the learned predictor. A simple baseline could be: record the timestep t of each state, then use the difference between the true timesteps for the archive insertion criterion. \n\n\n\nThere are a number of typos/grammar mistakes - please proofread carefully. A few examples:\n- Abstract: \"clusters temporal close states\" -> \"clusters temporally close states\"\n- Section 3.2: \"a pair of observation encodings is allow us to estimate\" -> \"a pair of observation encodings allows us to estimate\"\n- Section 4.1: \"An short example\" -> \"A short example\"\n- Section 4.1: \"is shown Table 1\" -> \"is shown in Table 1\"\n- Section 6.2: \"shows the practicability\" -> \"shows the practicality\"\n- Section 6.2: \"It is also providing a good performance\" -> \"It also provides a good performance\"\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "EhXACWKYRYV", "reviewer": "Reviewer_RwfW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a learned novelty estimator, based on the temporal distance between states, to be used in the Go-Explore architecture as a basis for memory writing decisions. The proposed method is evaluated in the low-data regime on three hard exploration Atari games, where most baselines are outperformed. The effects of using the proposed novelty estimator are investigated in some detail, and compare favorably to the native Go-Explore one.", "summary_of_the_review": "The idea behind the paper, of modeling time distances for exploration, is a good one in principle. However, the lack of any evidence that the proposed method generalizes is a major gap that renders the paper unsuitable for publication. I recommend rejection.", "strength_and_weaknesses": "- Major weakness: the paper only contains results on Montezuma’s Revenge, Gravitar, and Frostbite, over very few frames. There is no evidence in the paper that the method generalizes beyond this regime.\n- Strength: the idea to use time difference predictions as a basis for novelty estimations is in principle sound.\n- Strength: the paper is well-written and easy to read.\n- Strength: the proposed novelty estimator is studied in some detail in the paper, and shown to produce a good distribution of states entered into memory.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7pAnDVtIgzK", "reviewer": "Reviewer_ALFv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a time-myopic state representation that estimates the temporal difference between two observations. The time prediction is trained by MSE loss with the normalized temporal difference between two observations from the sampled trajectories. The authors apply this time prediction to Go-Explore (Ecoffet et al. 2021) for achiving more temporally distant (abstracted) observations and reducing the number of elements in the archive. Consequently, the proposed Time-Myoptic Go-Explore achieves better performance than vanilla Go-Explore in Gravitar although it has much worse performance in Frostbite.\n", "summary_of_the_review": "Although considering temporal distance for storing cells in Go-Explore is interesting and has reasonable improvement in terms of memory efficiency, the performance is generally degraded and requires more frames (sample-inefficient) compared to vanilla Go-Explore. Moreover, the authors only conducted experiments on three environments, and the results are not consistent. Consequently, I gave a reject in the initial review.\n\n**Update**\nAfter reading the authors' responses and other reviews, several concerns are still unsolved and the authors did not respond to the concerns. Hence, I maintain the rating.", "strength_and_weaknesses": "**[Strength]**\n\n1. The proposed method that considers temporal difference for storing the observations for Go-Explore is straightforward and seems effective in terms of reducing the temporal redundancy and diversifying the archived observations.\n\n2. The paper is generally well-written and easy to understand.\n\n3. The visualization from an example trajectory (Figure 3) makes a better understanding of time prediction.  \n\n**[Weakness]**\n\n1. According to Table 3, L and Td hugely affect the size of the archive. The sensitivity analysis of the hyperparameters is needed.\n\n2. The description of Eq. (5) is insufficient. How to define the score weight term, especially, how to choose 0.075?\n\n3. The performance in Frostbite is notoriously worse compared to vanilla Go-Explore. Moreover, considering the performance in Montezuma's revenge, it seems that it requires 5 times more frames to achieve a slightly worse performance than Go-Explore (sample inefficient). The authors did not deal with these performance gaps.\n\n4. In Section 5.3, the authors mention that they added “some small local dataset for each cell” but it is not well described.\n\n5. It will be better if the authors can present the comparison with Go-Explore in many different Atari games and other tasks, and analyze when the proposed method is effective and when it is not.\n\n**[minor]**\n\n1. It is good to unify the citation for Go-Explore into one (Nature version).\n\n2. In Table 1, it is better to change ‘Td > 13’ to ‘Td > 0.65’ for consistency.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject