Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a modification of the Go-Explore algorithm to remove the need for hardcoded state representations (which are used in the original version of the algorithm). Specifically, this work proposes to learn an embedding using an auxiliary loss based on predicting the number of timesteps connecting two observations. This timestep prediction function is also used to populate Go-Explores's goal archive: specifically, states are only added to the archive if they are at least a certain distance from all other states in the archive. This is to ensure that new states/goals are only added if they expand the frontier of explored states. 

The proposed method is evaluated on 3 hard exploration Atari games: Montezuma's Revenge, Gravitar and Frostbite. It doesn't quite match the original Go-Explore, but it does decently well compared to other baselines in terms of performance and sample complexity. 
strength_and_weaknesses: ## Strengths
- Although the original Go-Explore algorithm achieved impressive performance, it used several cheats like using handcrafted state abstractions (based on downsampling images), resetting the environment (or equivalently, exploiting the determinism of the simulator) which are not generally applicable. It would be great to have a version of Go-Explore which keeps the good qualities and performance but does away with these hacks, which this paper tries to make progress towards. 

## Weaknesses
- As far as i can tell, this paper still assumes a deterministic simulator or the ability to reset to previously visited states, which is still a very strong assumption. So the version of Go-Explore it proposes is still not generally applicable. 
- The paper is not very clearly written - see my comments below
- The results are decent, but it is currently unclear how the proposed algorithm scales with more data (only performance for 1M and 5M are reported). Please report performance at least until 50M so we can compare to the other model-free approaches. 

Detailed comments:
- Sections 1 and 2 are confusing. It was not clear to me until Section 4.1 what the point of predicting the number of timesteps k connecting two different states was. Section 2 is very confusing, with a lot of vague statements such as "The conflict solver must resolve illogical and non-intuitive conflicts" or "The down-scaling method is compressing the image information without considering its semantic content". Couldn't this be said about many different unsupervised learning methods? In the first paragraph, it seems to be saying that state abstraction is a bad thing, and that the fact that many states are grouped to the same representation is a bad thing. Can't this also be a good thing in that it filters out irrelevant details? It all depends on _how_ the grouping is done. I would suggest providing a detailed motivating example (from Montezuma's Revenge perhaps) that illustrates all of these claimed problems with Go-Explore - having a concrete example illustrating these statements would make this part much clearer. 

- It's not clear to me whether the timestep predictor is even necessary. Since you have access to the actual time step, and you are restarting exploration from different states in the archive, you should at least have the number of timesteps between a given state and the state the trajectory was restarted from. What happens if you use this as your archive insertion criterion? It might not work but should be a baseline to justify the need for the learned predictor. A simple baseline could be: record the timestep t of each state, then use the difference between the true timesteps for the archive insertion criterion. 



There are a number of typos/grammar mistakes - please proofread carefully. A few examples:
- Abstract: "clusters temporal close states" -> "clusters temporally close states"
- Section 3.2: "a pair of observation encodings is allow us to estimate" -> "a pair of observation encodings allows us to estimate"
- Section 4.1: "An short example" -> "A short example"
- Section 4.1: "is shown Table 1" -> "is shown in Table 1"
- Section 6.2: "shows the practicability" -> "shows the practicality"
- Section 6.2: "It is also providing a good performance" -> "It also provides a good performance"

clarity,_quality,_novelty_and_reproducibility: Clarity: 
- Low, see my detailed comments above. 

Quality:
- Medium. Results are ok but it would be nice to have additional results for different sample complexity budgets and the ablation described above. 

Novelty: 
- Medium. The algorithm proposes a novel modification to an existing algorithm. 

Reproducibility:
- High. The authors promise to open source their code. 
summary_of_the_review: Overall, I think that the paper's goal of making Go-Explore more general is a good one, and there are some interesting ideas in this paper. However, I believe it needs a good amount of work before it's ready for publication, specifically in terms of its clarity and its experimental results.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a learned novelty estimator, based on the temporal distance between states, to be used in the Go-Explore architecture as a basis for memory writing decisions. The proposed method is evaluated in the low-data regime on three hard exploration Atari games, where most baselines are outperformed. The effects of using the proposed novelty estimator are investigated in some detail, and compare favorably to the native Go-Explore one.
strength_and_weaknesses: - Major weakness: the paper only contains results on Montezuma’s Revenge, Gravitar, and Frostbite, over very few frames. There is no evidence in the paper that the method generalizes beyond this regime.
- Strength: the idea to use time difference predictions as a basis for novelty estimations is in principle sound.
- Strength: the paper is well-written and easy to read.
- Strength: the proposed novelty estimator is studied in some detail in the paper, and shown to produce a good distribution of states entered into memory.

clarity,_quality,_novelty_and_reproducibility: - Clarity: The paper is generally well-written and quite clear. Some details could be improved, e.g. in section 4.1, “the agent samples trajectories from c_1” - is this done by resetting the environment state, or by getting the agent to return?
- Quality: The explanation and inspection of the proposed mechanism are well done. The main quality issue is the experimental evaluation, which is seriously lacking in width (only three Atari games) and depth (only the initial phases of those games). Questions about the detailed choices of the time distance modeling are a bit moot because of the limited evaluation, but it would have been interesting to discuss things like the time scale introduced through the cutoff L, and the fact that only the difference between the states is fed into the time predictor - wouldn’t there be settings where the constant part is useful when trying to predict the time difference between the states?
- Novelty: The notion of modelling a difference between two states to obtain an exploration signal is not new, but the specific version here of modelling time difference and using that model output directly as a novelty signal, in Go-Explore, is new as far as I’m aware.
- Reproducibility: no concerns.

summary_of_the_review: The idea behind the paper, of modeling time distances for exploration, is a good one in principle. However, the lack of any evidence that the proposed method generalizes is a major gap that renders the paper unsuitable for publication. I recommend rejection.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes new learned state representations that can improve exploration by introducing time predictive semantics into the state representation. The authors propose a new notion of novelty based on time prediction representations and use this to create a new insertion criterion, a new cell count strategy and a new restart strategy for exploration in the settings of Go-Explore.
strength_and_weaknesses: Strengths:
This paper focuses on a very relevant problem in exploration of state representation and introduces a novel and original way of measuring novelty.
Weaknesses:
The paper lacks scientific rigour. The authors show experiments in table 2 with different frame counts making it very hard to compare among results. And even for the 1M frame budget the Go-Explore native seems to perform better. It is also unclear how precise the time predictive network is for each k, under each game. Is the time prediction correct on new seeds? One of the main arguments for the inadequacy of Go-Explore is that the overlapping conflicts lead to states being abandoned and not further pursued for exploration, how much conflicts is taking place quantitatively and does this lead to a significant performance drop, with the results presented so far it is still unclear this is true and time prediction mitigates this.
The paper is also not very clear, for example for the cell selection criteria Cvisits and Cscores is not defined, not even in the text. (see questions below for more)
The motivation for starting in proxy cell states is not clearly explained as well, how and where would this help/impact.?

clarity,_quality,_novelty_and_reproducibility: This paper can be greatly improved by providing a more self-contained description of the problems and explicitly state the issues and contributions of the work, such as how does the time predictive semantics help the detachment part? These results have not been provided. 
Comments/Suggestions on paper improvement:
Why would myopic prediction be good ? isn’t this going against the idea of go explore? 
How is the policy affecting the time predictions? It is unclear from the text.
Where are the weights  in eq. 5 being used? How are Cscore and Cvisit being counted, in particular Cscore? (a more self-contained description would benefit clarity of the paper)
What do the authors mean by extrapolation in 6.1? Of what?

Limitations of the current approach have not been addressed either, I would suggest addressing the difficulties of estimating time predictive representations and how the current approach limits exploration, is spatial information still captured? To what extent?.

summary_of_the_review: This paper introduces time predictive state representations and uses them as notion of novelty using in the go-explore context to store states and as different visitation counts, and new cell restarts, the paper provides an interesting perspective but lacks in clarity and scientific rigour. Further experiments would be needed to support the claims and provide a scientific understanding of the contributions proposed.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a time-myopic state representation that estimates the temporal difference between two observations. The time prediction is trained by MSE loss with the normalized temporal difference between two observations from the sampled trajectories. The authors apply this time prediction to Go-Explore (Ecoffet et al. 2021) for achiving more temporally distant (abstracted) observations and reducing the number of elements in the archive. Consequently, the proposed Time-Myoptic Go-Explore achieves better performance than vanilla Go-Explore in Gravitar although it has much worse performance in Frostbite.

strength_and_weaknesses: **[Strength]**

1. The proposed method that considers temporal difference for storing the observations for Go-Explore is straightforward and seems effective in terms of reducing the temporal redundancy and diversifying the archived observations.

2. The paper is generally well-written and easy to understand.

3. The visualization from an example trajectory (Figure 3) makes a better understanding of time prediction.  

**[Weakness]**

1. According to Table 3, L and Td hugely affect the size of the archive. The sensitivity analysis of the hyperparameters is needed.

2. The description of Eq. (5) is insufficient. How to define the score weight term, especially, how to choose 0.075?

3. The performance in Frostbite is notoriously worse compared to vanilla Go-Explore. Moreover, considering the performance in Montezuma's revenge, it seems that it requires 5 times more frames to achieve a slightly worse performance than Go-Explore (sample inefficient). The authors did not deal with these performance gaps.

4. In Section 5.3, the authors mention that they added “some small local dataset for each cell” but it is not well described.

5. It will be better if the authors can present the comparison with Go-Explore in many different Atari games and other tasks, and analyze when the proposed method is effective and when it is not.

**[minor]**

1. It is good to unify the citation for Go-Explore into one (Nature version).

2. In Table 1, it is better to change ‘Td > 13’ to ‘Td > 0.65’ for consistency.

clarity,_quality,_novelty_and_reproducibility: The paper is well written but several details such as the local dataset and explanation of worse results are missing. 
summary_of_the_review: Although considering temporal distance for storing cells in Go-Explore is interesting and has reasonable improvement in terms of memory efficiency, the performance is generally degraded and requires more frames (sample-inefficient) compared to vanilla Go-Explore. Moreover, the authors only conducted experiments on three environments, and the results are not consistent. Consequently, I gave a reject in the initial review.

**Update**
After reading the authors' responses and other reviews, several concerns are still unsolved and the authors did not respond to the concerns. Hence, I maintain the rating.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper extends the Go-Explore method by learning an embedding rather than hardcoding state representations, which can be used for novelty-guided exploration. They evaluated on Montezuma’s Revenge, Gravitar, and Frostbite, yielding results that are comparable to the original Go-Explore approach, but without a need for handcrafted representations. 

Reviewers felt that this was a straightforward yet still novel extension of the original algorithm, commended the paper for being overall well-written and easy to read (although many clarity issues were also pointed out), and appreciated the analyses and visualization of trajectories. 

However, there were some major concerns regarding just how general purpose this approach really is, given that it’s only tested in 3 Atari environments and still requires access to a deterministic simulator in order to reset to previous states. Some thought the paper needed more care and proofreading, and others felt that the scientific rigor could be improved (ie sensitivity analysis, controlling for the number of frames, and testing on more tasks). Two reviewers also pointed out in their follow-up to the authors' rebuttal that the additional experiments requested would not be overly burdensome, and all reviewers who did follow-up indicated that they didn’t find their concerns satisfactorily addressed. 

Hence, I believe this paper isn’t ready for publication at ICLR in its current form, but hope that reviewers’ comments are helpful for future versions of this work.

justification_for_why_not_higher_score: Reviewers pointed out numerous major concerns about the technical quality of this approach, quality of the presentation, and missing experiments, all of which preclude acceptance. After the rebuttal period, they were still not convinced, and hence I recommend reject.
Decision: Reject