Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper introduces a new setting which utilizes resetting to previously visited states in the trajectory and continuing on from there. The objective is then defined as the max return over the resulting trajectory tree. The paper shows that directly optimizing for this objective results in stochastic, exploratory policies. 
strength_and_weaknesses: Strengths
- The paper shows the emergence of stochastic policies when resetting the agent to previous states in the trajectory and using the ReMax return. This is an interesting contribution, and could lead to future research directions in how to develop exploration policies.


Weaknesses
- The motivation of this paper is that directly optimizing for this objective results in a stochastic exploration policy. It would be helpful to see how this policy actually works as an exploration policy used to train another policy that is trying to optimize the original RL objective. 
- The paper compares against other methods that don't reset the simulator during training. The comparison is done based on the number of steps taken in the environment. I am not sure this is a valid comparison. Resetting the environment involves hacking the environment. Other work that hack the environment in such a fashion do so to study a phenomenon that emerges (Go-explore) which is fine. If you're comparing to other methods that don't hack the environment, you need to make allowances to account for that, since doing so is not really possible for most problems of interest. For example, if the agent resets to state A, it automatically skips the steps that another agent would need to take to get to state A. A non resetting agent would almost necessarily be at a disadvantage in terms of sample complexity if thats how the environment steps were counted.
- Another useful baseline to compare against could be where the agent is reset, but the return is calculated as normal. This could also show whether the max operator is needed.
clarity,_quality,_novelty_and_reproducibility: The paper is fairly clear and novel. The overall quality is also good, but I think the experiments can be improved/reframed.
summary_of_the_review: I think this is an interesting idea, and should eventually be published, but I think it is not ready in its current state. Its contributions need to be reframed, certain baselines need to be added, and the way the experimental results are presented need to be changed.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: The paper proposes an approach for training exploratory policies, without adding any explicit exploration bonus. They formulate a new objective (ReMax) which allows resetting to an arbitrary prior state (hence generating new sequences), and then maximizing the best return over the set of sequences, and argue that this leads to the emergence of a stochastic policy. The paper is presented with analysis in 3 phases - random bandit arm (simple partially observable env), biased maze (deterministic fully observed env, uses simple model) and maze with image inputs (deterministic env, uses network model) .

strength_and_weaknesses: Weaknesses

1. No analysis/comparison of relation to prior exploration approaches, Quality of experimental environments

The main focus of the paper seems to be to establish that optimizing the reMax objective leads to learning a stochastic policy, and that this is desirable because it leads to better exploration performance. However, it is unclear how well the learned stochastic policy stacks up compared to prior exploration approaches, in any of the 3 phases considered (bandits, biased maze and  maze with image inputs). The only comparison to a different exploration approach was for A2C with entropy bonus on the minAtar envs, where the proposed approach gets higher return on only 1 out of 5 environments (Fig 9). There have been numerous exploration algorithms studied in tabular settings, specifically for bandits, with detailed analysis and none of them have been discussed in relation to the experiments done in phase 1. Regarding the maze environment, exploration algorithms have been shown to solve much more complex visual mazes like VizDoom [1]. Again the authors do not compare against or discuss how the proposed approach's exploration strategies would differ. This is further worrying because prior work has used exactly the settings used here (bandits and visual mazes).

2. Significance of the idea

The idea of resetting to a past state to enhance exploration has been studied in Go-explore [2], as the the authors point out themselves. In go-explore, the environment is reset to states that are rarely encountered, either directly or via a goal-reaching policy. This is very similar to the idea in the proposed paper of maximizing return after resetting to an arbitrary past encountered state in the trajectory. The authors don't include any analysis or comparison against go-explore, or any discussion of which settings would the proposed resetting scheme be better. From an algorithmic perspective, it seems Go-explore should be more exploratory since it resets to arbitrary states, while this paper only considers rests to past states in the same trajectory. Furthermore, Go-explore is evaluated on very challenging exploration environments (Montezuma's revenge) as well as challenging simulated robotic manipulation environments.  

[1] Pathak, Deepak, et al. "Curiosity-driven exploration by self-supervised prediction." International conference on machine learning.
[2] Ecoffet, Adrien, et al. "Go-explore: a new approach for hard-exploration problems."

clarity,_quality,_novelty_and_reproducibility: The paper is clear and easy to follow. Concerns with quality/significance of experiments and novelty are as outlined above. 
summary_of_the_review: I am in favor of rejecting this paper because there is no comparison or discussion of the exploration performance to that of previous methods (except on the minAtar envs where the proposed method performs better on only 1/5 envs). Further, the proposed idea is very similar to that of go-explore (which obtained state of the art results on montezuma's revenge), again without any comparison or analysis of why/when this method should be used instead.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes an alternative objective for Reinforcement Learning called ReMax. The authors claim that an agent optimizing this objective naturally develops an exploratory behavior, differently from standard RL, where exploration typically should be enforced, for instance adding an entropy bonus. The main idea consists in allowing the agent to reset its trajectory to a specific state, instead of explicitly adding exploration.
The authors provide a policy gradient theorem to optimize such objective, and analysed the obtained policies in three worked examples where the algorithm was applied. Finally, the provided strategy is tested the MinAtar benchmark, in terms of performance w.r.t. the standard RL objective.
strength_and_weaknesses: ## Strengths
The paper deals with the interesting topic of exploration, which is of paramount interest in the RL community, since it is deeply connected with the challenging topic of generalization in RL. The main goal of the article is to re-phrase the exploration problem as the necessity for the agent of visiting the same states multiple times. The author claim that this can be achieved by allowing the agent to properly reset the trajectory. 

## Weaknesses
However, the evidence the authors provide to prove their thesis does appear completely convincing. First, it is not clear, from a theoretical point of view, how the ReMax objective optimal solutions relate to the standard objective ones. Does the optimal ReMax solution have some bias w.r.t. to the standard one? Second, it is difficult to state whether the authors claim, that the policy obtained with their strategy promote exploration, is true or not. While the strategy seems to be effective in the worked examples, in the MinAtar benchmark the entropy bonus seems to be more or equally effective w.r.t. the expected return obtained, in all the environment a part from Seaquest. 
W.r.t. this aspect, it is not clear to me why the authors affirm "Comparing A2C (w/ entropy bonus) to ReMax A2C, there was no clear winner.".
In order to better understand the effectiveness of the approach in promoting exploration, it would be interesting to compare the two strategy also w.r.t. exploration metrics.

Moreover, the paper lacks a in-depth study of the impact of employing different strategy for resetting and selecting the reset state, and heuristics are used instead. In practice, these strategies could have a fundamental role in the development of the desired exploratory behavior, thus, more should be done to investigate how to properly build them.
Finally, the proposed approach has the strong limitation of requiring the availability of a simulator, differently from entropy based approaches which don't have such requirement.

clarity,_quality,_novelty_and_reproducibility: ## Clarity
The authors did a fair job in describing their approach and the paper results quite clear.

## Quality
The presented results appears to be sound from both the theoretical and experimental sides.

## Novelty
The idea appears to be quite novel, and the similarity with other works are discussed in the introduction. A related work section is provided in the appendix, however, the article may benefit from including a dedicated related work section in the main paper.

## Reproducibility
The presented experiments are reproducible.
summary_of_the_review: While the idea is interesting, the provided results look still preliminary, and do not seem to be still strong enough to support the authors hypothesis. It appears that the presented method does not offer a better or equivalent way of inducing exploratory behavior w.r.t. to state-of-the-art techniques, which do not suffer from the limitation of requiring a simulator.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: It proposes a novel objective function (ReMax) and a novel algorithm of resetting to previous states for encouraging implicit 'stochastic exploration' policies.
strength_and_weaknesses: Major Weaknesses:
1. The paper fails to motivate the need for their proposed objective function when there are several existing exploration based objectives and algorithms that achieve state-of-the-art results on many environments.
2. The performance of the policy obtained with ReMax objective is not shown against the actual reward of the environment we wanted to optimize. One could also define their objective as maximizing the entropy of actions and show that taking random actions at every time step maximizes their defined objective. It misses the whole point of optimizing for the actual objective and I fail to see the utility of these objectives and corresponding algorithms.
3. There aren't any experiments comparing ReMax with other exploration based methods
4. "As our main focus in this study is the emergence of exploration from our ReMax objective, we only employ simple deterministic rule-based policies for πX and η". This seems like exploration did not 'emerge' but is being imposed by the epsilon value which determines whether to explore or not given a particular state.
5. A stochastic policy is the one where each action output has a probability associated with it. One could define a policy network in such a manner. I do not understand how an objective function can encourage stochasticity. Relatedly, what is a "stochastic exploration"? 
6. The proposed algorithm isn't novel. As mentioned in the paper, this idea has been explored in Go-Explore. Moreover, it seems like a very inferior version of MCTS (which has a more principled way of exploring different future states without arbitrarily resetting to previous states) 

Minor weaknesses:
1. It mentions entropy bonus based objectives but fails to mention several other algorithms like count-based exploration (Tang, 2016) or intrinsic rewards (Zheng, 2018) 
2. Insufficient detail about the maze environment and results. How long is the maze? What is 'maximum sufficient fixed length'? Why does it need 10M training steps for such a seemingly simple environment? 
3. Figure-4b is unclear. Why does it cause such a huge variance with changing learning rate? Is the experiment performed with the learning rates 1e-3, 1e-2, 1e-1 or other intermediate learning rates too? Why is this comparison relevant? Aren't we just interested in the performance with best learning rate? Are other hyper parameters tuned too? 

Questions:
1. When the agent resets to a previous state, does the time_steps also reset to their old value? What prevents it from going in an infinite-loop or never-ending episodes? 
2. What is the rationale for choosing policy gradient methods and not value-based methods? 
clarity,_quality,_novelty_and_reproducibility: See above
summary_of_the_review: Based on the major weaknesses listed above, I recommend to reject this paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes an approach that allows the agent to reset its trajectory to a specific state, by optimizing an alternative objective called ReMax. This approach allows for achieving exploratory behaviors without explicitly adding exploration bonuses to the reward function.
The proposed objective is optimized through policy gradient and tested compared to using the standard RL objective in the MinAtar benchmark.
Although the authors study a relevant problem and propose an interesting solution, the reviewers have raised several concerns that have not been completely solved by the authors' feedback.
In particular, the reviewers have pointed out the lack of proper comparison with other methods in the large literature on exploration in RL. 
The reviewers complain about a lack of motivation for the proposed approach and would like to have more insights about the way in which the new objective is able to induce exploratory behavior.
We encourage the authors to consider the reviewers' suggestions while preparing a new version of their paper.
justification_for_why_not_higher_score: Although the idea is interesting, it misses an in-depth comparison with other similar ideas.
The authors need to improve their paper by providing a better understanding of why the proposed approach is sensible and how it relates to existing ones.
Decision: Reject