Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors present an intrinsic reward based on cyclophobia—avoidance of cycles of experience. The idea is, to focus on observing parts of the world that haven't been seen before, actions that return the agent to an observation it has made before are penalized. In particular, such a cycle does not even have to mean returning to a previously experienced complete state (in the Markov decision process sense)—bumping up against one wall of a gridworld is as pointless as bumping up against another. To avoid these experiences that are locally cyclic, but not globally cyclic, the authors look for cycles in agent-centrically local "views." In the gridworld experiments the authors use, this means one square in front of the agent for the most local view, the squares to the front and sides in the next most local view, and so on and so forth, with the largest view being the entire gridworld—but we can imagine future work considering other representations of locality more appropriate to non-gridworld domains. 

The authors demonstrate that their method improves and speeds up learning in comparison to several other well-known methods in two benchmark gridworld domain settings: MiniGrid and MiniHack. They also perform a small ablation study to provide some evidence that hierarchical views and cyclophobic reward work together to improve performance.
strength_and_weaknesses: Cyclophobic rewards are an intuitive method for encouraging systematic exploration in sparse reward environments. Part of what makes cyclophobic rewards intuitive is that they penalize states that have been visited before, encouraging the agent to find parts of the world it hasn't visited before to avoid such penalties. This intuition is similar to that we apply to optimistic initialization—the states the agent visits are worse than it expects, so the agent is encouraged to try elsewhere. In fact, there is a method that uses negative intrinsic rewards to create the effect of optimistic initialization (Machado, Srinivasan & Bowling, 2015). 

Without understanding how or whether the effects of cyclophobic rewards extend beyond the typical effects of other mostly negative intrinsic rewards, it is difficult to evaluate the significance of cyclophobic rewards. To the best of my understanding, all of the baseline intrinsic reward methods used for comparison in this paper use non-negative intrinsic rewards.

My other concern with the baselines is the state representation used. In Section 3.1, it says that the "training setup of Parisi et al. (2021)" is used, but I wasn't sure which aspects of the training setup were meant. I believe Parisi et al. used a panoramic state view that obscures anything behind walls or closed doors, but from Figure 1 in this paper, I get the impression that the agent's state representation includes the whole gridworld, including items behind walls—can you be more explicit about what state information the agents had access to in the experiments? Did all of the agents have access to the same state information? (If so, I'm guessing we should expect different results than those presented by Parisi et al., 2021.)

I found that the structure of the paper was straightforward and easy to follow. The paper is well-written, for the most part. I appreciated how the authors anticipated many of my concerns throughout, and noted limitations accordingly. However, another concern I don't feel was addressed is the reasonableness of maintaining state histories for multiple views. MiniGrid environments are relatively small; do you feel that this method will be applicable as environments get larger?

The related work components represent one of the weaker parts of the paper; I did not find the relationships between the proposed method and the existing work very clearly explained.
1. "One way to do this is by approximating the counts via a density model as done by Ostrovski et al. (2017) and Bellemare et al. (2016)." (p. 8) Given that your algorithm as described is tabular, I don't see the relevance of the translation to pseudo-counts.
2. Do you consider RND and RIDE to be at all related to your method, or are they just described in the related work (p. 9) because they are included baselines in some of the experiments?
3. "Overall, prediction error methods require a model while our method is model-free." (p. 9) ← This statement is a little strange coming shortly after your summary of RND, which doesn't require a model in the traditional sense.

Minor error that should be fixed:
- "We see that the cyclophobic intrinsic reward is necessary to reach a successful trajectory at all." (p. 8) Not sure what this means, since the figure only shows a comparison of two versions of cyclophobic agents against epsilon-greedy. Probably should be rephrased.

Typos and grammatical suggestions (no intended impact on score)
- "In environments with sparse rewards finding" (p. 1) → "In environments with sparse rewards, finding"
- "Random exploration (e.g. epsilon-greedy with sparse rewards)" (p. 1) ← Makes it sound like sparse rewards are part of the exploration method rather than the environment—and isn't this still a problem even when rewards are dense?
- "This idea is further pushed by" (p. 1) ← not sure what this means; may want to rephrase
- "like avoid everywhere." (p, 1) → "like to avoid everywhere."
- "Instead rewarding novelty we" (p. 1) → "Instead of rewarding novelty, we"
- "where an observation function Z : S → O maps the observation to the true state." (p. 2) ← The description is mixed up; Z maps the true state to the observation that the agent gets.
- "the cycle penalty is −l. That is, during exploration the cycle penalty can decrease indefinitely." (p. 2) ← This is a little awkward because if the penalty is decreasing, it sounds like it should be getting less negative. There are a few alternatives like referring to it as "the cyclophobic reward" or talking about the magnitude of the penalty increasing indefinitely, but I haven't thought of a way to completely avoid confusion.
- "how it can be build" (p. 2) → "how it can be built"
- "is the sum of" (p. 2) → might be better to say "is the weighted sum of" to account for rho.
- "restricting the views leads to" (p. 2) ← "restricting the view leads to" might be a little clearer
- "V_1 is", "V_2 are", "V_3 is" (p. 3) ← probably want to be consistent
- "what is front" (p. 3) → "what is immediately in front of it"
- "to V_k the" (Figure 1 caption) ← missing space?
- "several weights" (p. 4) → There is one weight for each view, is that right? The word "several" is a little vague, and explaining where the selection of weights comes from would help solidify understanding.
- "the next paragraph." (p. 4) ← "the next section"?
- "hashcode" or "hash code" (p. 5) ← probably want to be consistent
- "where m_t \in R^k be" (p. 5) → "where m_t \in R^k is" 
- "considered to be solved, if one" (p. 5) → "considered solved if one"
- "our x-axis is shorter" (p. 5) ← Shorter than what? Is this in comparison to the figures shown by Parisi et al. (2021)?
- "dificult" (p. 6) → "difficult"
- "tasks show in Figure 3" (p. 6) → "tasks shown in Figure 3"
- "can not" (p. 7) → "cannot"
- "difficult set of" (p. 7) → "difficult sets of"
- "performance as well dramatically" (p. 7) → "performance dramatically"
- "state counts" (pp. 7, 8, 9) → "visitation counts"
- The "count-based intrinsic reward" (p. 8) used for the hierarchical learner with no cyclophobic reward is based on MBIE-EB (Strehl and Littman, 2008), right? A citation to where the choice of reward came from would be helpful context. 
- "representations we achieve" (p. 8) → "representations do we achieve"
- "The mechanism to check novelty" (p. 9) → maybe "This mechanism to check novelty"?
- "amount of changes in a state transition" (p. 9) ← Not sure what this means; what is a change in a state transition?
- "experiments for MiniGrid" (p. 9) → "experiments in MiniGrid"?
- If you get the chance to go through and make the colors for each method consistent (e.g. cyclophobic with hierarchical views could be the same blue in every figure), that would make the figures much easier to comprehend.

Machado, M. C., Srinivasan, S., & Bowling, M. (2015). Domain-independent optimistic initialization for reinforcement learning. In AAAI Workshop: Learning for General Competency in Video Games 2015.

Strehl, A. L. and Littman, M. L. (2008). An analysis of model-based interval estimation for Markov decision processes. Journal of Computer and System Sciences, 74(8):1309 – 1331.
clarity,_quality,_novelty_and_reproducibility: While I felt the paper was reasonably clear and well-structured, there were some areas that I felt were not sufficiently clear. In particular, I don't understand how the transfer learning across different environments (pretraining) was achieved. A couple of areas I found especially confusing:
1. "We simply save the extrinsic rewards in a separate Q-table and set the values as starting values for the new Q-table at the start of transfer learning." (p. 5) ← I don't understand what this means. Which extrinsic rewards does this refer to and which values does this refer to?
2. "Our method likewise contains a task-agnostic component given by the cyclophobic intrinsic reward and a task-specific component given by the external reward." (p. 9) ← I don't understand this part; are the visitation counts for different views re-used in multiple environments?

While I feel like I have a sufficient understanding of how cycles fit into the use of hierarchical views, I didn't always follow the explanations and examples as written; writing those parts in another way could really improve the clarity of the paper.
1. "encountering a key in a smaller view produces less cycles" (p. 1) ← I don't follow this example
2. "Surprisingly, in combination with the cyclophobic intrinsic reward, we gain additional information about the structure of the environment." (p. 2) ← I don't understand this statement.
3. "the most detailed view" (p. 3) ← I think this is referring to the global view (has the most details) but it could be referring to the most local view (most zoomed in on the details)

Other areas of confusion for me:
- "will propagate the additional penalty across the trajectory" (p. 2) → I would benefit from more explanation of the propagation
- "That is, where some form of disentangled representation of the environments plays an important role for efficiently finding solutions." (p. 5) ← for a reader who doesn't know what the binding problem is (whom I assume this statement is for) this statement doesn't help.
- "We train each environment" (p. 5) ← I'm guessing you didn't mean to say environment here, probably meant "agent"? Can you speak more about what you mean by "train" in this paper? I believe the "no transfer" experiments do not involve distinct training and testing phases, so I'm guessing that when you talk about training, you are referring to all agent learning and behaviour?
- "which are diverse in the required skills." (p. 6) ← not sure what this means

Reproducibility concerns: 
1. The ablation study is missing a number of details. Did you perform multiple runs? Which MiniGrid environment was used to generate Figure 5? 
2. How was the shaded area in the figures computed?
summary_of_the_review: I am leaning to reject this paper, with particular concern about how the experiments were set up (questions about state information above). As I have noted above, not comparing or discussing other mostly-negative intrinsic rewards or optimistic initialization means that I also don't know if this paper presents anything significant. However, the paper includes some ideas about focusing on varying levels of locality and an intuitive intrinsic reward method that could support interesting future work, so it is close to acceptable for me. If we don't accept it at this conference, I hope that the authors continue to work on the clarity of the paper and its contextualization with existing methods so it can be a stronger contribution.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper introduces cyclophobic reinforcement learning, a negative intrinsic reward for exploration that incentivizes the agent not to revisit previously visited states. This intrinsic reward is computed over different cropped views of the environment by the agent and these are combined to compute the final reward. The proposed method is evaluate on mini grid and mini hack and is shown to lead to increased performance.
strength_and_weaknesses: Strengths:

_ The paper is well written and easy to follow.
_ I liked the empirical evaluation, it was done on many tasks from MiniHack and MiniGrid with a large set of baselines. The ablation study was also helpful to understand the impact of each component of the method.
_ I also appreciate that the authors discuss the limitations of their method.
_ Results on minigrid and MiniHack are impressive and clearly the benefits of cyclophobic RL over exisiting exploration techniques.

Weaknesses:

_ Overall it is hard to know how the proposed will scale to more complex problems like Atari games where the same state may not be seen multiple times (at least on the largest view). On the hand the narrow may not have enough local information to be useful. The counts of the observations might also be difficult to compute in that situation.
_ From 1 (b) it looks like to obtain a smaller view a rotation is done in addition to cropping? Is that part necessary? How does it impact the performance of the algorithm. Seems like to do the orientation of the agent must be known.
clarity,_quality,_novelty_and_reproducibility: I found the paper well written and easy to understand.
The various plots and visualization were really helpful to understand the proposed method.
summary_of_the_review: Despite impressive results on MiniGrid and MiniHack, it is unclear how the proposed method scale to larger environment where it is not possible to count observations. Because of that I am not too sure what we can learn from cyclophobic RL as it is presented in the paper and for that reason I cannot recommend acceptance.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes an intrinsic reward for RL exploration based on loop-closure detecting. When a state or a central patch of the state is visited, a penalty will be given to the agent to encourage the agents to visit novel states. Such an intrinsic reward design is combined with Q-learning to conduct experiments on MiniGrid and MiniHack environments.
strength_and_weaknesses: ## Strength
This paper is clearly written and easy to follow. The loop-closure-based intrinsic reward is intuitive and the method is easy to implement. The idea of building a hierarchy of patches for loop detecting is novel. Experiments are conducted on two challenging experiments, which is good.


## Weakness

My concerns are mainly about novelty and experiments, which are listed below. 

1. **Limited novelty**: The idea of detecting loops to encourage exploration isn't really a novel idea. Its practice in deep RL literature can be at least traced back to [1], which implements an additional loss to predict loops. Even the idea of using loop-based rewards is well executed as a specific technique in the NoveID method [2]. The hierarchy of patches for loop detection is interesting and could be counted as a great practical technique for realizing loop detection. However, I would be concerned about whether the paper contributes enough to the community, particularly based on the limited experiments (see my following comments).

[1] Learning to Navigate in Complex Environments, Piotr Mirowski et. al, ICLR 2017

[2] NovelD: A Simple yet Effective Exploration Criterion, Tianjun Zhang, Huazhe Xu, Xiaolong Wang, Yi Wu, Kurt Keutzer, Joseph E. Gonzalez, Yuandong Tian, NeurIPS 2021.

2. **Missing baselines**: The paper primarily compares with two baselines, i.e., RIDE and RND. However, many recently developed baselines are omitted. More importantly, many of them are also tested on the miniGrid environment. Representative examples include NoveID [2], AMiGo [3], MADE [4] etc.

[3] Learning with AMIGo: Adversarially Motivated Intrinsic Goals, Andres Campero, Roberta Raileanu, Heinrich Kuttler, Joshua B. Tenenbaum, Tim Rocktäschel, Edward Grefenstette, ICLR 2021

[4] MADE: Exploration via Maximizing Deviation from Explored Regions, Tianjun Zhang, Paria Rashidinejad, Jiantao Jiao, Yuandong Tian, Joseph E. Gonzalez, Stuart Russell, NeurIPS 2021.

3. **Experiment results seem unfinished**. It is particularly wired to me that there are no reward curves at all for the MultiRoom-N12-S10 scenario in Fig 2. At least, if you check the RIDE paper, RIDE can definitely achieve non-zero rewards by training from scratch. 


clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. However, the experiment section is unsound and I'm concerned about the reproducibility of the reported results. The novelty can be also limited (see my comments above). 
summary_of_the_review: On the technical part, the idea of building a hierarchy of patches for developing a set of intrinsic rewards is interesting. However, the contribution is not well supported by the experiments, which further makes the novelty of this work limited.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper introduces a novel intrinsic reward for exploration based on cycle avoidance, where cycles are detected across multiple agent centric views. Being a tabular method, the method is evaluated on MiniGrid and MiniHack, in both tabula rasa and in a transfer learning regime.

Overall the reviewers found the paper to be well written, the method intuitive and appreciated that the limitations of the method were addressed head-on.

Unfortunately, all reviewers and this AC agree that while promising, the paper is not yet ready for publication. Key points of contention remained throughout the discussion phase. In particular, I agree with [CKV7] regarding the missing baselines: the fact that these papers did not consider transfer learning does not preclude a proper comparison when learning from scratch. At its core, Cyclophobic RL is a standard exploration method which ought to be evaluated in both tabula-rasa and transfer learning settings. I also share [FEf9]’s concern regarding application domains: the authors chose two very similar domains to evaluate their method and it is also not clear to me whether their method would scale to more complex or richer environments, whether Atari, Habitat or other. The choice of agent-centric hierarchical views is central to their method and represents prior knowledge which may or may not transfer or be as easily applicable to other environments. Ideally, the authors should provide convincing evidence their method is broadly applicable.

Given the promising results, I sincerely hope the authors will take the feedback to heart and improve the paper for a future submission.

justification_for_why_not_higher_score: Promising results, but evaluation limited to two similar gridworlds, and with missing baselines.
Decision: Reject