Cyclophobic Reinforcement Learning
Authors: Stefan Sylvius Wagner
Peter Arndt
Jan Robine
Stefan Harmeling
Venue: Submitted to ICLR 2023 (2023)
Keywords: Reinforcement learning
intrinsic rewards
exploration
transfer learning
objects

Abstract: In environments with sparse rewards finding a good inductive bias for exploration is crucial to the agent’s success. However, there are two competing goals: novelty search and systematic exploration. While existing approaches such as curiousity- driven exploration find novelty, they sometimes do not systematically explore the whole state space, akin to depth-first-search vs breadth-first-search. In this paper, we propose a new intrinsic reward that is cyclophobic, i.e. it does not reward novelty, but punishes redundancy by avoiding cycles. Augmenting the cyclophobic intrinsic reward with a sequence of hierarchical representations based on the agent’s cropped observations we are able to achieve excellent results in the MiniGrid and MiniHack environments. Both are particularly hard, as they require complex interactions with different objects in order to be solved. Detailed comparisons with previous approaches and thorough ablation studies show that our newly proposed cyclophobic reinforcement learning is vastly more efficient than other state of the art methods.

Under review as a conference paper at ICLR 2023
CYCLOPHOBIC REINFORCEMENT LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
In environments with sparse rewards, finding a good inductive bias for exploration is crucial to the agent's 's success. However, there are two competing goals: novelty search and systematic exploration. While existing approaches such curiosity- driven exploration find novelty, they sometimes do not systematically explore the whole state space, akin to depth--fist-searcch vs breasehh-firt-searchh In this paper, we propose new intrinsic reward that is cyclophobic, i.c. it does not reward novelty, but punishes redundancy by avoiding cycles. Augmenting the cyclophobic intrinsic reward with sequence of hierarchical representations based on the agent's cropped observations we are able to achieve excellent results in the MiniGrid and MiniHack environments. Both are particularly hard, as they require complex interactions with different objects in order to solved. Detailed comparisons with previous approaches and thorough ablation studies show that our newly proposed cyclophobic reinforcement learning is vastly more efficient than other state of the art methods.
INTRODUCTION
Exploration is one of reinforcement learning's most important problems. Learning success largely depends whether an agent is able to explore its environment efficiently. Random exploration is explores all possibilities but at great costs, since it possibly revisits states very often. More efficient approaches intrinsic rewards based curiosity to enforce focusing on novelty, which often leads to great results, but at the price of possibly not exploring all corners of the environment systematically. Ideally, we would pursue both goals: novelty search and systematical exploration.
How can we favor novelty while ensuring that the whole environment is systematically explored? To achieve this, we propose cyclophobic reinforcement leurning which is based on the simple idea of avoiding cycles during exploration. More precisely, we define a negative intrinsic reward that penalizes redundancy in the exploration history. This idea is further enhanced by applying it to several hierarchical views of the environment. The notion of redundancy can be defined relative to cropped views of the agent: while cycles in the global view induces cycles in the corresponding narrow view, the converse is not the case. E.g., a MiniGrid agent turning four times to the left produces a cycle would like to avoid everywhere. This cycle is visible in the global view, but
in state space that penalizing it does not avoid it in other locations. However, with hierarchy of views, we record cycle also in some smaller view, which allows us to transfer this knowledge to any location in the global view and hereby to avoid never experienced cycles. Similarly, an interesting object such as a key, produces less cycles in a smaller view than some other object (since the key can be interacted with). Thus the probability of picking up the key increases, since other less interesting observations produce more cycles (c.g a wall). Thus, we are defining cycles relative to a hicrarchy of view to get a transferable definition of redundancy.
Contributions:
We introduce cyclophobic reinforcement learning new paradigm for efficient exploration in hard environments (e.g. with sparse rewards). It is based on a new cyclophobic intrinsic reward for systematic exploration applied to a hierarchy of views. Instead of rewarding novelty, we are avoiding redundancy by penalizing cycles, i.c. repeated statc/action pairs in the exploration history. Our approach can be applied to any MDP for which a hierarchy of views can be defined.
2. In the sparse- reward settings of the MiniGrid and MiniHack environments we thoroughly evaluate cyclophobic reinforcement learning and can show that it achieves excellent results compared to existing methods, both for tabula-ne rasa and transfer learning. Preliminary results show cyclophobia works for both tabular and neural agents.

---

Under review as a conference paper at ICLR 2023
3. In ablation study we provide deeper insights into the interplay of the cyclophobic intrinsic reward and the hierarchical state representations.
Notation: We define an MDP as a tuple (S,A,P.R, 7). where the agent and the environment
interact continuously at discrete time steps f 00, 2, 3.... We define the state an agent receives from the environment as random variable S. E S, where S, state from the set of states S at timestep t. From that state, we define random variable for the agent selecting an action A, E A where A, = a is some action in the possible set of actions A for the agent at timestep t. This action is selected according to policy TTa | s) or (s)) if the policy is deterministic. One time step later as a consequence of its action, the agent receives a numerical reward which is random variable R R. where R4+1 r is some numerical reward at timestep + 1. Finally, the agent finds itself in a new state S4+1- Furthermore we define a POMDP (8, A, O, P, R. 2,7) That is, the agent sees the state E S through an observation E O, where an observation function Z 8 + O maps the true state the agent's observation.
is some representation of the
generalization of MDP in the case the true state space S is unknown.
BUILDING BLOCKS
We begin by first defining the cyclophobic intrinsic reward and hierarchical state representations as they are the building blocks of our method. Finally, we define policy which combines the intrinsic and extrinsic rewards together with the hierarchical state representations to form a global policy which the agent acts upon.
2.1 CYCLOPIIOBIC INTRINSIC REWARD
For efficient exploration, redundancy must be avoided. A sign for redundancy is when states are repeatedly explored, with other words, when the agent encounters cycles in the state space instead of focusing novel areas. To guide the exploration, we will penalize cycles using cycle penalty, which we call cyclophobic intrinsic reward (a negative intrinsic reward). This avoids redundancy such that uninteresting parts of the state-action space are discarded quickly. For instance, if an agents gets stuck in some area, typically, it is facing numerous cycles. In such a situation we would like the agent assign penalties the repeating state-action pairs in order focus more promising parts of the state space that do not cause immediate repetition.
Formally, let us assume that we have per episode a history of previous state-action pairs Hepisoiic (s), ai) ). (#2, o2),... (st, at)} and we are currently at the state-action pair (s1+1, ax+1). We say that we have encountered cycle if the current state-action pair appeared already in the history, i.e. ($141, ((+1)) E Hepisodie- For its first repeated occurrence, we will penalize the state-action pair (st, az) (just before the cycle) by negative one,
Teyele (s. -1
(1)
a cycle is encountered multiple times, e.g. 1 times, the cycle penalty is -1. That is, during exploration the propagated cycle penalty can penalize indefinitely. For pairs (s, a) that have not created a cycle, Toyelo a) = O.
Learning the cycle penalties. In principle, the cycle penalty can combined with any reinforce- ment learning algorithm. However, we explain how it can be built into the SARSA update rule, since the latter will propagate the penalty across the trajectory. To learn the Q-function, we are employing the standard SARSA update rule,
((s,a) + (1- (s) + +
+ Q(5,,a))
(2)
with (s, 0. Tex s', a') being transition, 77 being step size and where the total reward r(s, a) for the state-action pair (8, a) is the weighted sum of the extrinsic reward from the environment Fex and the cyclophobic intrinsic reward raycle (s, a) defined above,
r(s, a) Tex + p ( a).
where p trades off extrinsic and intrinsic rewards.
2.2 HIERARCHICAL STATE REPRESENTATIONS
Besides penalizing cycles, the second key idea of this paper is to consider a hierarchy of state representations. For this, repeatedly crop the agent's observations to induce additional partially
2

---

Under review as a conference paper at ICLR 2023
observable Markov decision processes (POMDPs). In general, restricting the view leads ignoring information about the environment. Surprisingly, in combination with the cyclophobic intrinsic reward, we gain additional information about the structure of the environment Q-functions for different POMDP's. The relevant insight that on limited views lower down the hierarchy, trajectories
get different
contain cycles that have not been experienced views higher up the hierarchy. These cycles in smaller views represent transferable knowledge about the structure of the environment. E.g. in the MiniGrid environment (see Section B.I] encountering wall in the smallest view will cause cycle, capturing that running into a wall is counterproductive. On larger views this knowledge is only available directly, if tried all walls everywhere. So, the the smaller views capture relevant invariances that also apply to the larger views. In general MDPs it might not be obvious how to define hierarchical state representations. Ideally, the agent has some sort of "location" to allow "cropped" views. Additionally, local properties of the state space that result in cycles are transferable to the whole state space, so that learning to avoid cycles helps for efficient exploration. For grid world-like environments, the hierarchical state representations are easily definable, since the agent has a defined location that can be used to define smaller neighborhoods typically corresponding to limited views of the agent.
V
V/2
9
i
V/
(a)
(b)
Figure 1: Hierarchical views allow us to transfer relevant invariances about the environment. (a) The three different representations are obtained by cropping the observation for each state-action pair (s, a). Vi is the full view where the agent an intermediate cropped representation of the agent's view that helps the agent generalize by reusing familiar observations. V/ is the most restricted view where the agent only sees what immediately in front of it. (b) Through the views V1 to Vi the amount of cycles continuously increases as the observations in the higher views can be mapped multiple times to the same observation in the lower view. This naturally leads to each view being a separate POMDP describing the true MDP.
the whole environment or the largest portion of it. V2 is
To discuss the roles of the different views more precisely, let's consider three such cropped views which consist of the global view Vi, the intermediate view V22 and the smallest possible detail view Vs (see Figure Lap. Each view gives new, typically more limited, perspective of the true state. As shown in Figure lb each view induces a different set of cycles, for instance a sequence of actions which do not lead cycle the full view V₁, might lead a cycle in a smaller view Vg, because in the latter, more states are mapped to the same observation. Thus, the different views provide different types of information which are useful for learning and allow the agent focus on different properties of the environment.
In general, we can have an arbitrary number of views. Each view V; induces a POMDP
V; = ,.
each having their set of observations V and observation function 2%: S O4 All POMDP's V1, V/,... operate on the same state space S, however they have different sets of ob- servations O and corresponding observation functions. General POMDPs can have probabilistic observation function. In our case, the observations are deterministic functions of the full view, e.g. the observations for the ith POMDP of state
0%: = z (i)
(5)
which corresponds to the cropping for the view V, Hereby we create partial representations of the state space that allow us to identify invariances in the environment by looking at the same true state through different perspectives.
Note that, POMDPs are normally used to model uncertain observations. That is, they are generaliza- tion of MDPs where the true state is not observable. Here, the POMDP idea is used to model different
3

---

Under review as a conference paper at ICLR 2023
views of a fully observable state space. Therefore, do not seek to solve POMDP problem, but rather we extend a regular MDP to get redundant hierarchical representations.
2.3 A CYCLOPHOBIC POLICY FOR HIERARCHICAL STATE REPRESENTATIONS
In the following, we describe how the cyclophobic intrinsic reward and the hierarchical state repre- sentations can be combined into a policy that exploits the cyclophobic inductive bias. For this, we define several Q-functions along the views, and combine them weighted sum. The weights determined by counts of the observations in each observation set O% which we explain next.
Mixing coefficients. Many strategies for defining mixing coefficients are possible. For concreteness, follow in this paper simple schema, where determine the weights from the observation counts, which are obtained from the history of states visited throughout training,
Hall = {si, $2. . }
Note that Hall contains all states that have been visited in the training far, which is different from the states in the episodic history Hegisodic that was used in Sec. 2. define cycles. Denoting the corresponding views of the history as of = 2 (st), the counts for view V₁ are
N(oT), "...),..... N(.y)
(7)
where counts the number of times the observation of has been encountered. The raw counts are normalized by their maximum (maximum for simplicity, other normalizations possible), because the smaller views have higher counts than the bigger views. The weights should be large for states subtract the normalized counts from one,
that have been seen less often,
N(
vi (o Vi = 11
(8)
max (N N( ... N(O
The previous formula allows us to compute the weights for each respective view V; for single state St,
(s:) = 1
= [a (o, ..... (oV))
(9)
where a, (s:) is vector. Finally, the softmax operator turns these weights into vector of probabili- ties,
(s;) sofimax(a.(st)) = [a) ax (s:))
(10)
The entries of this vector are denoted by a; (s:) and are used to define the cyclophobic Q-function in the next section. The definition of a can be extended all state by setting it to zero for unseen states, i.e. (<s) 0 (zero-vector) for g H.n-
In general, larger views in the hierarchy will have bigger entries in (sx)) the observations repeat less often than in the smaller views. Thus (<ss;) gives more weight to the larger views than the smaller ones. However, this compensated by the cyclophobic intrinsic reward since it is triggered far more in the smaller views than the larger views.
Cyclophobic Q-function. To combine all views into a single global policy we define mixture over the different Q-fucttions learned with the cyclophobic intrinsic reward defined in Section 2.1 For view V., we define Q-function as
a) Q .a) + (1 (1- 7) Q(0% +n[r(o" .) + +nQ(0"M
(11)
This follows from our argumentation in Section 2.2 where we replace the state in Equation 2 by the observations % in their respective views. Then we can define cyclophobic Q-function as the mixture of the Q-functions of each view,
Quycte (s,
Σ Cai(s) (0)) Σ
Q(oV)
(12)
Note that the mixing coefficients a; (s;) are only non-zero for states 8, that appeared in the global history Han- Thus the cyclophobic Q-function zero for states & Hal not encountered.
4

---

Under review as a conference paper at ICLR 2023
Cyclophobic policy. phobic Q-function, i.e.
Finally, the cyclophobic policy is defined to the greedy action for the cyclo-
क (a) = arg max Qeycle (8, = arg max
Σ Iai())Q(o')))
(13)
Having normalized the counts within each view ensures comparability of the counts. This ensures that Q-values from rare observations i.c. more salient have larger effect on deciding the action for the policy æ. In an ablation study in Section 3 we show that the combination of the cyclophobic intrinsic reward and hierarchical state representations is crucial to the methods success.
EXPERIMENTS
Our experiments are inspired by Parisi al. (2021) and Samvelyan al. (2021). We test in environments where the causal structure is complex and the binding problem (Greff et al. 2020), (van Steenkiste et al. 2019) arises. That is, where some form of disentangled representation of the environments plays an important role for efficiently finding solutions.
Environments:
We test our method the MiniGrid and MiniHack environments:
The MiniGrid environment (Chevalier- Boisverr et al. 2018) consists of series of procedu- rally generated environments where the agent has to interact with several objects to reach specific goal. The MiniGrid environments pose currently a benchmark for the sparse reward problem, since reward is only given when reaching the final goal state.
The MiniHack environment (Samvelyan et al. [202T] is graphical version of the NetHack environment (Küttler et al. (222)). We select environments from the Navigation and Skill tasks. The MiniHack environment has a richer observation space by containing more symbols than the MiniGrid environments and richer action space with up to 75 different actions. While not necessarily tailored to the sparse reward problem as the MiniGrid environment, the high state-action complexity makes it one of the most difficult environments for exploration.
State encoding: For both environments we choose five croppings of the original full view. The views Vi, V2, V3. V1, Vs are of grid size 9 x 9, 7 x 7, 5 x 5, 3 x 3 and 2 x 1. In principle we could also include the full view. However, in the experiments the performance was much better when we limit ourselves to the partial views. Furthermore, every part of the grid that is behind of the wall from the agent's 's perspective is occluded (note that this is not the case in Figure Ta for visualization purposes). Intuitively, limiting the views allows the agent to ignore irrelevant details that are far away. Next, the views are mapped to hashcodes (we use the open source xxhash library). That is, we have a hash function g that maps observation oy to a hashcode h. g(ol This helps us quickly check for cycles as we only need to check whether two hashcodes are equal.
For the MiniHack environment, text prompt is an integral part of the current state. So, for MiniHack, the hashcode for a state is the concatenation of the hashcodes of the observation of¹ and the text prompt my, where m, E R* an encoding of a string of length k, i.e. ("+" denoting concatenation)
h4 = g(o)) + g(m.).
(14)
Training setup and baselines: We train each agent for three runs using different seeds for every run. For transfer learning we use "DoorKey" and MuttiEan(MutiiRoonNM
"KeyCoriddoS3R33"
"BlockedUnlocd Unlockicckup") for pretraining. For transfer learning, during pretraining we save the extrinsic rewards in a second separate Q-table in addition to the main Q-table which contains values of equation 2 These extrinsic rewards in the second Q-table are then used at the beginning of transfer learning for each environment, while we continue to use the cyclophobic intrinsic reward when doing transfer learning. The baselines for the MiniGrid experiments are provided by Parisi et al. (2021) and allow us to compare our method to C-BET (Paris et al. 202T). ). Random Network Distillation (Ottrovki et al.) 2017). RIDE (Raileanu & Rocktäschel] 2020) and ICM (Pathak et al. (017). For
the MiniHack environment we compare our results to the baselines presented by (Saavelyya et al.
(2021) which include IMPALA (Espeholt al. 2018), RIDE (Raileanu & Rocktäschel, 2020) and Random Network Distillation (Burda et al. 2018). For the skill tasks the only available baseline is
IMPALA (Espeholt et al. 2018).
While during learning the intrinsic reward based on cyclophobia plays the essential role, the ultimate goal is to maximize the extrinsic reward that is provided by the environment. Thus for comparison, we have to plot the extrinsic reward the agent receives for each episode. The reward ranges from 0 to 1.
Evaluation metric:
5

---

Under review as a conference paper at ICLR 2023
3.1
MINIGRID
Obleck
fcockep-000
IntockkAdepp
2 t
1260
1384
382
114
1360
20
1384
20
43
20
20
NalicNua BD
Checornnreette 181h
Balle 2014
20146
66
100
153
83
50
163
06
180
15:00
K.0
151
166
K01
833
23
30333
1384
2030
306
30233
13.00
134
1850
Suabet CE Stape (8:3:1008)
Carlesitg
Figure MiniGrid: We converge faster than C-BET Parisi al. 2021 in many MiniGrid environments with and w/o pretraining. The hierarchical state representations and cyclophobic intrinsic reward is extremely quick to converge which shows efficient exploration and the usefulness of the cropped representations, Furthermore we are also able to transfer knowledge by pretraining on other environments.
To test method in the MiniGrid environment choose the training setup of Parisi et al. (202T). That is, we determine the performance of the agent with no pretraining and when pretraining from other environments explained in the previous section. Figure[]shon shows the agent's performance when training from scratch (rows three and six) and when transferring knowledge from pretrained environments (bottom three rows).
Learning from scratch (rows three and six): In three out six environments, our pro- posed method converges much faster than the competitors, including C-BET (Parisi et al. 202T). Note that for some environments our x-axis is shorter than in Parisi et al.) (2021). since method converges much faster. Only "KeyCoridorS3R3"" "MultiRoom" and "ObstructeddMaze22D000 -2thb" pose significant challenges to our approach, because our method is tabular and thus cannot deal with many object variations in the environment (e.g. random color changes). Furthermore, the "MultiRoom" environment proves challenging for all environments with only C-BET managing reach convergence, while are able to fetch some rewards. This is due to the large amount of observations the different corri- dors produce. In the Appendix A.2 our approach also excels in the "KeyCorridorS3R3" "ObstructedMaare22DDHi and more difficult environments once we remove the colors, e.g. "KeyCorridor4433" "KeyCoridorS5R3"" "KeyCoriddorSR33" "ObstructedMaze-10" 1Q".
Transferring knowledge (rows one, two, four and five): Having trained on one environment transfer knowledge to different one? In environments the transfer im-

---

Under review as a conference paper at ICLR 2023
proved the results from the "no transfer" setup (see "Unlock", "Doorkey", "UnlockPickup", "BlockedUnnoockPicupp"" "ObstructedMazze DDlh") and never deteriorated performance.
3.2
MINIHACK
Mooia-Hansioo- 15
Mook-Dark-15Mk
ReyMoooa-855
Aiver-Bazeow
Xivor
0.2
10000
4.3
19.000
5.0
15:000
10.00
10.0
Rember od Stapa (MLllioon)
Figure 3: MiniHack Navigation: The agent converges quicker in the Navigation task than the intrinsic curiosity baselines such as RIDE (Raileanu & Rocktäschel] 2020) and Random Network Distillation (Burda et al., 2018). This corroborates our hypothesis, that avoiding cycles is essential for quick exploration.
TAT
Pray
Wob-2wy
SAAT
Leakedboor-- -P1
0.2
Cyclophotseereez))
Mumber of Ziopp (Milllonn)
We converge quicker than IMPALA Especolt et al. 2018) in the Skill task. The Skill task defines over different actions the agent must learn use making it one of the most difficult sets of environments of the MiniHack suite.
Figure 4: MiniHack Skill:
To push our approach to its limits, we also tackle some of the MiniHack environments which requires the agent to learn large skill set.
Navigation task: For the simpler navigation tasks shown in Figure 3 our method converges quicker than intrinsic curiosity driven methods such as RIDE and RND. Especially, in the "River" environment, only our cyclophobic agent able converge and solve the environment. However, of course there are environments such as "RoomUltimate" where our approach fails due to its tabular style, which limits the complexity of the environment.
Skill task: For the Skill task the only available baseline is IMPALA which is not based on intrinsic curiosity. Here we also vastly superior, even collecting extrinsic rewards when the baseline can not ("Wear" and "LockeDDoor-ixee""
3.3 LIMITATIONS
In some environments with big number of different objects our method struggles to converge. The tabular agent does not have the observational invariances that methods based on function approxima- tion learn and exploit (c.g. with CNNs). For instance, for these environments learned invariances allow them to handle differently colored objects. While the tabular agent does cope with avoiding redundancy through the cyclophobic intrinsic reward and the hierarchical state representations, it does not learn the kind observational invariances neural networks do, which are sometimes necessary. However, that can show that reducing the number of colors for the "KeyCorridor" environ- ment improves our performance dramatically (see Appendix A.2). Color reduction can also be seen as cropped view of the real situation, and hence this experiment falls into our setup. Furthermore. preliminary results with a PPO agent (see Appendix A.I solves "KeyCorridor,, further supporting our theory.
Nonetheless, we argue that the above limitation does not lessen the impact of our work, since the tabular agent is able compete with neural network based methods showing that exploration does not seem to rely solely on representation learning, but also on clever avoidance of repetition. Thus, when the number of different objects increases, better performance be attributed to the learned
7

---

Under review as a conference paper at ICLR 2023
invariances in neural network reducing observational complexity, rather than more efficient exploration.
3.4
ABLATION STUDY
tpe-lon-groody
Kiezazchioal
Courte
Niecorotioal ope
cycLoghobii
Niocacot.ioal cyclophooie
40
40
10
10
Figure 5: MiniGrid ablation study (visitation counts for "DoorKey-16x66"; The cyclophobic Q-function explores the environment more efficiently than the optmistic initialization, count and epsilon-greedy based counterparts. We record visitation counts for several variations of intrinsic reward and hierarchical state representations. Hierarchical state representations together with the cyclophobic intrinsic reward are the most efficient.
To show the effectiveness of the cyclophobic intrinsic reward and the hierarchical state representations, perform ablation study. Figure shows state counts heat map after 10,000 steps of training. We distinguish four cases: (i) epsilon-greedy, Plain epsilon greedy exploration fails to find the goal in the bottom right. (ii) hierarchical & counts: We replace the cyclophobic intrinsic reward in equation (3) with a count-based intrinsic reward similar to Strehl & Littman (2008) defined by N (( for view V,. This improves the results but still fails. (iii) hierarchical & optimistic initialization: We use optimistic initialization let the agent try all possible actions. We initialize the q-table with value of two. Optimistic initialization manages to enter the second room but just fails at getting a reward within 10,000 steps. (iv) cyclophobie: Having a cyclophobic intrinsic reward calculated only on the largest view finds the goal. (v) cyclophobic & hierarchical: The combination of hierarchical views and cyclophobic intrinsic rewards (as explained in Sec. 2.3 explores even more efficiently can be in the left room where fewer steps are needed to leave it.
Kinituck Bow.aeeeee-eottteeeeeeeee. 15A15-10
Extrrineio noward
0.0
0.00
Wonber of Stopo [Mi1)1005)
slecazohionl cyolophobie
cyclophokie
Hiezarchicol
opsintnit
Optistnit
spsilon-groody
Figure 6: MiniGrid ablation study (extrinsic reward): Hicrarchical state representations are crucial to performance in some environments. We see that the cyclophobic intrinsic reward the only method that can solve the environments. Optmistic initialization finds succesful trajectories, but is not as consistent.
In Figure 6 we analyze the impact of the hierarchical state representations on the agent's performance. The training setup is described in section C.3] The "UnlockPickup" environment can only be solved hierarchical state representations and the cyclophobic intrinsic reward are used. Due to the complexity of the environment and the large number of object variations, the different views allow for know knowledge reuse and thus lead to better performance. Similarly, for the MiniHack the cyclophobic intrinsic reward performs well and adding hierarchical state representations leads to the environment being solved. Optimistic Initialization is able to collect rewards in both environments and the hierarchical state representation improve performance. However, performance still falls short to cyclophobic reinforcement learning.
"aoom-Randoo" environment
8

---

Under review as a conference paper at ICLR 2023
4 RELATED WORK
Systematic exploration and loop closure: Systematic exploration similar to that of the cyclophobic intrinsic reward has been previously explored in the form of optimistic initialization. The idea of optimistic initialization is initialize the value function by value larger than the expected reward. As the agent performs updates on the value function will be forced to choose the next unexplored action since the update for the current state-action pair will decrease the current value. |Machdoo et al.) (2014) suggest normalizing the reward by the first encountered extrinsic reward and then subtracting one. In this way the value function a negative value, leading the agent to explorative behaviour as in regular optimistic initialization. Mirowski al. (2016) define loop closure as an auxiliary loss for exploration 3D mazes in addition to depth measure. The authors detect loop closure by checking whether the agent has been near a previously visited position. To get position information the authors integrate velocity information over time.
be initialized with zero and all further updates provide
Exploration by counting changes in observations: Instead of relying on the prediction error to measure state transition change, the changes in the observation space can be counted. iZhang] ct al. (2020) define an intrinsic reward for a count based approach by calculating the difference of visitation counts between subsequent states. This intrinsic reward pushes the exploration boundary as it provides a positive reward when reaching unexplored states for the first time or no reward otherwise. This mechanism to check novelty is extended through NovelD(Zhang et al., (02T). Interesting Object, Curious Agent by Parisi et al. (2021) is an intrinsic motivation method that splits up learning into a pretraining and transfer stage. It defines count-based intrinsic reward called C-BET that is based on counting the state transition changes N (c((s, s)). In contrast, we use state counts to measure novelty in the respective hierarchical views.
Encoding observations to augment intrinsic reward: Encoding observations to either augment the intrinsic reward or define goals has lead great improvements in exploration. Language in the form of text prompts that define goals has been used by Mu et al. (2022) to augment the intrinsic reward. /Singh Chaplot et al. (2020) build representations that reflect the structure, i.e. geometry of the environment, More generally, Ye et al.) (2020) train an A2C (Mnih et al.) (0016) agent on different croppings in a grid environment, without an intrinsic reward, with mixed results. Paarisi et al. (2021) also augment the input by training
360 panoramic to learn task-agnoss changes. Our hicrarchical state representations similarly augment the intrinsic reward and reveal structural invariances about the environment which aid exploration.
Generalization and transfer learning: Learning representations of the environment that are reusable and can be transferred has been explored previously. Learning successor representations (Barreto et al., 2016) requires learning local environment dynamics that can be reused when the environment, i.e. the original distribution, is changed. (Schaul et al. 2015) Decomposition on the learned action-value function to obtain a canonical representation of the environment which can be transferred to other situations. More recently, Parisi et al. (2021) learn an exploratory policy which is then combined with a task specific policy at transfer time. Our method likewise contains a task-agnostic component given by the hierarchical state representations and a task-specific component given by the cyclophobic intrinsic reward.
Singular Value
CONCLUSIIO
Avoiding cycles allows the agent to quickly and systematically discard uninteresting state/action pairs which are repeated often. This makes our eyclophobic intrinsic reward good inductive bias towards novelty that simultaneously encourages systematic exploration. The ablation studies show that the cyclophobic intrinsic reward just for single view already powerful enough to solve complex environments. Adding hicrarchical state representations leads to even better performance, as can be seen in the MiniGrid experiments. Moreover, the experiments in MiniGrid confirm the transferability of the learned bias to new environments. Here the hierarchical state representations are crucial, as otherwise in a tabular approach transferability could not be achieved. Due to the tabular approach, environments with large object variation lead to convergence problems in our method. The more objects are present in a single view, the larger the state space becomes a tabular setting. Neural networks, on the other hand, learn invariances that reduce the complexity of the state space. To address the convergence problems, future work will therefore focus on incorporating the hierarchical state representations into a neural network based architecture.

---

Under review as a conference paper at ICLR 2023
REFERENCES
André Barreto, Will Dabney, Rémi Munos, Jonathan J. Hunt, Tom Schaul, Hado van Hasselt, and David Silver. Successor Features for Transfer in Reinforcement Learning. arXiv e-prints, art. arXiv:1606.05312, June 2016.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by Random Network Distillation. arXiv e-prints, art. arXiv: 111.1.111....... October 2018.
Andres Campero, Roberta Raileanu, Heinrich Kütter, Joshua B. Tenenbaum, Tim Rocktäschel, and Edward Grefenstette. Learning with AMIGo: Adversarially Motivated Intrinsic Goals. arXiv e-prints, art. MIXiv:2006.12122. June 2020.
Maxime Chevalier-Boisvert,, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym. /ttps: wwt..//awa...eeeccaoaaaaaaaaaiaa htwp://iinmmmimiiiiiimmmm.m 2018.
wwt.coommmaam...mm wttm/maximeedi
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: Scalable Distributed Deep-RL. RL with Importance Weighted Actor-Learner Architectures. arXiv e-prints, art, Xiv: 1.1i:180022 180.001561, February 2018.
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. On the Binding Problem in Artificial Neural Networks. arXiv e-prints, art. Nt:::0012.55000 December 2020.
Heinrich Küttler, Nantas Nardelli, Alexander H. Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The NetHack Learning Environment. arXiv e-prints, art. 1tti:200061177660 June 2020.
Marlos C. Machado, Sriram Srinivasan, and Michael Bowling. Domain-Independenn Optimistic Initialization for Reinforcement Learning. arXiv e-prints, art. arXiv:1410.4604 October 2014.
Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew J. Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, Dharshan Kumaran, and Raia Hadsell. Learning to Navigate in Complex Environments. arXiv e-prints, art. urXiv:1611.0367. November 2016.
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous Methods for Deep Reinforcement Learning. arXiv e-prints, art. arXiv:1602.01783, February 2016.
Jesse Mu, Victor Zhong, Roberta Raileanu, Minqi Jiang, Noah Goodman, Tim Rocktäschel, and Edward Grefenstette. Improving Intrinsic Exploration with Language Abstractions. arXiv e-prints, art. arXiv:2202.08933, February 2022.
Georg Ostrovski, Marc G. Bellemare, Aaron van den Oord, and Remi Munos. Count-Based Explo- March 2017.
ration with Neural Density Models. arXiv e-prints, art.
Simone Parisi, Victoria Dean, Deepak Pathak, and Abhinav Gupta. Interesting Object, Curious Agent: Learning Agnostti Exploration. arXiv e-prints, art, 1:Xiv:2111.1311 November 2021.
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity driven Exploration by Self-supervised Prediction. arXiv e-prints, art. arXiv: 170:005333. May 2017.
Roberta Raileanu and Tim Rocktäschel. RIDE: Rewarding Impact-Driven Exploration for Procedurally-Gennraaeee Environments. arXiv e-prints, art. arXiv:2002. 12292, February 2020.
Mikayel Samvelyan, Robert Kirk, Vitaly Kurin, Jack Parker-Holder, Minqi Jiang, Eric Hambro, Fabio Petroni, Heinrich Küttler, Edward Grefenstette, and Tim Rocktäschel. MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Research. arXiv e-prints, art. arXiv:2109.13202 September 2021.
Mikayel Samvelyan, Robert Kirk, Vitaly Kurin, Jack Parker-Holder, Minqi Jiang, Eric Hambro, Fabio Petroni, Heinrich Kuttler, Edward Grefenstette, and Tim Rocktäschel. Minihack the planet: A sandbox for open-ended reinforcement learning research. In Thirty-fitth Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021. URL https 7/openrevis ew net. net//formmm2m i yefkk.
10

---

Under review as a conference paper at ICLR 2023
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators.
In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1312 1320. Lille, France, 07-09 Jul 2015. PMLR. URL
ml wtt.inng.......e.e.e...... wwl...rss//...
schaul h
Devendra Singh Chaplot, Ruslan Salakhutdinov, Abhinav Gupta, and Saurabh Gupta. Neural Topological SLAM for Visual Navigation. arXiv e-prints, art. arXiv:2005.12256, May 2020.
Alexander L. Strehl and Michael L. Littman. An analysis of model-based interval estimation
for markov decision processes. Journal of Computer and System Sciences, 74(8):1309-133)
2008. ISSN 0022-0000. doi: http:://ooooo///0.0000j/sss22/22
URL https: /
Www.
Learn-
ing Theory 2005.
Sjoerd Steenkiste, Klaus Greff, and Jürgen Schmidhuber. Perspective on Objects and Systematic Generalization in Model-Based RL. arXiv e-prints, art. arXiv:1906.01035. June 2019.
Chang Ye, Ahmed Khalifa, Philip Bontrager, and Julian Togelius. Rotation, Translation, and Cropping for Zero-Shot Generalization. arXiv e-prints, art. arXiv:2001.09908, January 2020.
Tianjun Zhang, Huazhe Xu, Xiaolong Wang, Yi Wu, Kurt Keutzer, Joseph E. Gonzalez, and Yuandong Tian. BeBold: Exploration Beyond the Boundary of Explored Regions. arXiv e-prints, art. arXiv:2012.08621,, December 2020.
Tianjun Zhang, Paria Rashidinejad, Jiantao Jiao, Yuandong Tian, Joseph Gonzalez, and Stuart Russell. MADE: Exploration via Maximizing Deviation from Explored Regions, arXiv e-prints, art. arXiv:2106. 10268, June 2021.
Tianjun Zhang, Huazhe Xu, Xiaolong Wang, Yi Wu, Kurt Keutzer, Joseph E. Gonzalez, and Yuandong Tian. Noveld: A simple yet effective exploration criterion. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https:://peereevier net forum? Ld=

---

Under review as a conference paper at ICLR 2023
A FURTHER ABLATION STUDIES
For all plots in the ablation studies we use the same setup as the main experiments. We refer to section C.3 for this.
A.1
PRELIMINARY RESULTS WITH PPO
Xkn10r16-Dniook-v0
San10rte-Doorkey-Beee www.-rid--eyoorooorsseeee
1.0
Extrinsic Reward
0.4
0.2
0.00
3.00
4,00
4.00
6,00
8,00
4.00
6.00
8.00
5,00
10.00
Number of Steps (MLllions)
Cyclophobie-770(00ue
Eyclophatiin- www.baletiii--beeeeeeeeeee
R2D3
C-BEI
Figure 7: Using PPO together with only the cyclophobic intrinsic reward also converges far quicker than C-BET. Moreover, we are able to converge in KeyCorridorS3R3 confirming our theory in Section 3.3
We train a PPO agent using only the cyclophobic intrinsic reward. Figure 7 shows that for the given environments the PPO agent also explores theory about observation complexity from 3.3 is supported as the PPO agent able to solve the "KeyCorridor" environment. Note that these are preliminary results. In future work we seek implement the hierarchical state representations for the neural agent.
efficiently than C-BET. Furthermore,
A.2 OBSERVATIONAL COMPLEXITY REMOVING COLOR FROM OBSERVATIONS
To test our theory of observational complexity affecting our performance in some environments such as "KeyCorridor" and DObstructedMaze22Dllba such that the underlying task is not affected. In case of "KeyCorridor" and "MultiRoom" colors can be removed from all objects, while in the "ObstructedMaze" environments we only remove color of the key and door. We show the results in figures and 10
remove color from objects in the environment
wind@eid-KKeporrzee
MiniHac''-Hul 1iRom11-------
ObaLzuctodHaab 2aw.ddaaa---2bbb-oo
Extrinsic Revard
0.8
0.2
0.0
0.00
2,00
4.50
0.50
8.00
10.00
0.00
5,00
10.00
15.00
20.00
28.00
0.00
1.00
2.00
3,00
4.00
5.00
6.00
Cyclophabie(Ouze)
C-BET
Number
Stegs (Millions)
Figure 8: When extracting color from the hierarchical state representations we able converge very quickly in the "KeyCorridorS3Re" and "ObstructedMazze- 2Dlbb" environments. Doing this for the MultiRoom-Ne environment does not have such large effect as here the large variation in observation is due to the many different corridors, which our tabular method cannot find observational invariances for as neural network could.
Figure & shows that this has an immediate impact on the performance where the "KeyCoriddonS333" environment converges instantly while the "ObstructedMaze-DD 2Dthb" environment needs very few steps to converge in comparison to C-BET. For the MultiRoom-Ne environment, the observational complexity comes from the different types of corridors present throughout training, thus removing color only mitigates the observational complexity to some extent. This corroborates theory that while our method is very good at exploration, its tabular nature cannot capture the observational invariances that neural networks naturally do.
12

---

Under review as a conference paper at ICLR 2023
Minicrid- www.aooid--eooorrdoreeeeee
Miniorid Wannorid-ReyCorrrddoee
1.0
Extrinsic Revard
0.6
0.6
0.2
0.0
10,00
15.00
10.00
20.00
5.00
10.00
Cyclophabic (OuzG)
Number Steps (Millions)
Figure 9: Our agent can even gather rewards successfully in much larger KeyCorridor environments when removing colors, showing that the exploration mechanism is very strong.
In Figure we test more difficult "KeyCorridor" environments. We can see that the cyclophobic reinforcement learning is also able to explore efficiently in these harder environments when colors are removed.
Minniiee wwwteeeetttttteeeeeeeeeee
Min oi P011
1.0
Extcinsic Revard
0.4
0.2
0.00
2.00
4,00
8.00
8.00
10.00
0.00
5:00
10.00
16.00
20.00
0.00
5:00
10.00
16.00
20.00
Cyclophobie (Ouze)
Mumber of Stepe (Millions)
Figure 10: For the more difficult "ObstructedMaze" environments the agent is able to solve the "IQ" version. The other more larger variants prove too complex even when removing colors.
While we achieve good performance in the "1Q" variant, the other larger variants remain too complex for the cyclophobic agent.
Finally, we test more complex "ObstruutedMuzz" environments in Figure 10
Note that when training the other methods without colors their performance may improve. However, this ablation study simply illustrates that our method very good at exploring and is only held back by its tabular nature. In future work we want to find the best way to learn observational invariances with the hicrarchical state representations and the cyclophobic intrinsic reward.
A.3 COMPARISON WITH MORE RECENT METHODS
keyCorrLdozE330
Obstrooteddaaze- 2012
Obs Lnot
1.0
Extrincic Revard
0.6
0.4
0.2
3.00 4,00
10.00 20.00 30.00
50.00
40.00 02.00 60.00 100.00
Number of Staps (Millions)
Cycllghhable-PPP((arrr))
cyeloghhobic/
Figure 11: Selection of environments that overlap with more recent methods such as Zhang et al. (002))
Campero et al. (2020) Zhang et al. (2021)
13

---

Under review as a conference paper at ICLR 2023
B ENVIRONMENT DETAILS
B. MINIGRID EXPERIMENTS
Unlock
DoorKey
KeyCorddor
UnlockPickup
BlockedUniockPlckup
MuldRoom-Ne-Sy
ObstrucledMeze-100
ObstructedMaze--aan
ObstnudtodWaze-2200
Figure 12: An overview of the MiniGrid environments used in this paper.
We test some of the same environments Parisi et al. (2021). In the following we describe the environments as Parisi et al. (202I) and add our own comments where relevant.
Unlock: pick up the key and unlock the door. (288 steps per episode)
DoorKey-8x8: pick up the key, unlock the door, and go to green goal (640 steps per episode)
KeyCorridorS3R3; pick up the key, unlock the door, and pick up the ball (only the door before the ball locked) (270 steps per episode).
UnlockPickup: pick up the key, unlock the door, and open the box (288 steps per episode).
BlockedUnnoockPeckupp pick up the ball in front of the door, drop it somewhere else, pick up the key, unlock the door, and open the box (576 steps per episode).
MazeutttddMMM--DIlI open the box to reveal the key, pick up, unlock the door, and pick up the ball (288 steps per episode).
ObstructedMaze-2DDIln same as above, but with two doors unlock (576 steps per episode).
above, but with two balls in front of the doors (like in BlockedUUnblockPickk))) (576 steps per episode).
ObstractedMMazee2DDhh
MultiRoom-NG, navigate through six rooms of maximum size ten and go to the green goal (all doors are already unlocked) (120 steps per episode).
MultiRoom-N12-5 navigate through twelve rooms of maximum size ten and go to the green goal (all doors are already unlocked) (120 steps per episode).
B.2 MINIHACK EXPERIMENTS
14

---

Under review as a conference paper at ICLR 2023
Wand of Death Essy
Room-Random
KeyRoom-S5
Eat
Pray
Wear
LockedDoor
Figure 13: An overview of the Minihack environments tested in this paper. The top row shows the Navigation tasks, while the bottom row shows the Skill tasks.
We perform experiments inspired by the baselines from Samvelyan et al. (2021). Here, Samvellyn (et al. (202T) define a set of navigation and skill acquisition tasks. The navigation tasks range from requiring the agent to navigate empty and hazards. The skill acquisition tasks require the agent to perform a certain set of skills to be able solve the environments.
maze, up to navigating environments with monsters
B.2.1 NAVIGATION TASKS
In the following deseribe the environments used from the navigation task.
Room-Random-15x155 This task is in a single square room, where the goal is reach the staircase down. In the random version of this task, the start and goal positions are randomized.
Room-Dark-15x15: Same as Room-Rann Room-Random, but the agent only sees parts the environ- ment it has already discovered.
Key KeyRoom-55: These tasks require agent pick up a key, navigate door, and use the key to unlock the door, reaching the staircase down within the locked room. The locations of the door, key and starting position are randomized.
River: The river environment requires the agent to cross a river using boulders. When pushed into the water, they create dry land to walk on, allowing the agent eross it.
B.2.2 SKILL TASKS
The nature of commands in NetHack requires the agent perform sequence of actions that the initial action, which is meant for interaction with an object, has an effect. The exact sequence of subsequent can be inferred by the in-game message bar prompts. Overall, each of the following tasks the agent has to perform a sequence of actions that allow the agent to perform the required skill.
Eat: The agent has to navigate and eat an apple. Doing so requires sequence of actions and confirming a prompt to eat the appe.
Pray: The agent has to pray on an altar. This is more complex than eating an apple as praying requires the agent confirm more prompts.
Wear: The agent has wear armor. This a more difficult task than the previous tasks.
LockedDoor-Fixeds.
agent has to kick a locked door.
Wand of Death-Easy. The agent starts with a wand of deatch in its inventory and has to zap it towards sleeping monster. That is, the agent needs equip the wand of death walk to the monster and zap it.
In all of these environments the positions of the agent and of the object that has be interacted with are randomised. Furthermore, the complexity is also increased by the confirmation prompts the agent has answer in order to perform certain action.
15

---

Under review as a conference paper at ICLR 2023
C EXPERIMENTAL DETAILS
C.I HYPERPARAMETERS
the following we present table with all hyperparameters used for training the environments. In general, method has very few hyperparameters. In this case the only really tunable hyperparameters are the step size 7). the random action selection coefficient for epsilon-grecdy exploration and the discount 7:
Common hyperparameters:
7
7
Num. views
0.2
0.99
5
In our case we determined experimentally that views good number of views for training an agent.
Epsilon-greedy parameter e: While learning with the cyclophobic intrinsic reward we still perform random action from time time by having epsilon-greedy action selection. That is, the agent samples value n from uniform distribution and selects a random action if this n < c. These are the settings for for the respective environments:
Environments
Unlock
0.1
DoorKey
0.1
KeyCorridor
0.1
UnlockPickup
0.3
MultiRoom-Ne
0.1
MultiRoom-Ni2-SI
0.1
BlockeelUnlockKiccuup
0.3
ObstructedMare-IDDDi
0.3
ObstruuttedMazz-2Dllh
0.1
ObstracteddMaze22DDiht
0.3
Intrinsic reward tradeoff pt For the "Obstructeedaaze2DDI1" and "BlockedUUnlockPickpp" envi- ronments we set intrinsic reward tradeoff p = 0.2. This has as effect that extrinsic reward gets propagated more when the agent encounters which is necessary for more complex environments. For all other tested environments we set = 1, thus not weighting the intrinsic reward at all.
C.2 PPO HYPERPARAMENT METERS
Intrinsic reward tradeoff p GAE-lambda
0.01
0.95
7
0.99
Batch size
256
Learning rate
0.001
Entropy coefficient
0.01
Value loss coefficient
0.5
Max Grad Normt
0.5
Clipping-e
0.2
RMSProp-c
le-8
RMSProp-a
0.99
C.3 PLOT SMOOTHING
For all 3 runs we show the mean reward and standard deviation smoothed over sliding window. For our and (Samvelyan al. (202T)'s experiments we use sliding window of 50000 steps. For the experiments in Parisi et al. (2021) we use a sliding window of 96000 steps as per their experimental setup. The shaded areas are the standard deviation of mean for runs.
16


Reviews:
[{"review_id": "S8gEcCrCp3", "reviewer": "Reviewer_FYSy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors present an intrinsic reward based on cyclophobia—avoidance of cycles of experience. The idea is, to focus on observing parts of the world that haven't been seen before, actions that return the agent to an observation it has made before are penalized. In particular, such a cycle does not even have to mean returning to a previously experienced complete state (in the Markov decision process sense)—bumping up against one wall of a gridworld is as pointless as bumping up against another. To avoid these experiences that are locally cyclic, but not globally cyclic, the authors look for cycles in agent-centrically local \"views.\" In the gridworld experiments the authors use, this means one square in front of the agent for the most local view, the squares to the front and sides in the next most local view, and so on and so forth, with the largest view being the entire gridworld—but we can imagine future work considering other representations of locality more appropriate to non-gridworld domains. \n\nThe authors demonstrate that their method improves and speeds up learning in comparison to several other well-known methods in two benchmark gridworld domain settings: MiniGrid and MiniHack. They also perform a small ablation study to provide some evidence that hierarchical views and cyclophobic reward work together to improve performance.", "summary_of_the_review": "I am leaning to reject this paper, with particular concern about how the experiments were set up (questions about state information above). As I have noted above, not comparing or discussing other mostly-negative intrinsic rewards or optimistic initialization means that I also don't know if this paper presents anything significant. However, the paper includes some ideas about focusing on varying levels of locality and an intuitive intrinsic reward method that could support interesting future work, so it is close to acceptable for me. If we don't accept it at this conference, I hope that the authors continue to work on the clarity of the paper and its contextualization with existing methods so it can be a stronger contribution.", "strength_and_weaknesses": "Cyclophobic rewards are an intuitive method for encouraging systematic exploration in sparse reward environments. Part of what makes cyclophobic rewards intuitive is that they penalize states that have been visited before, encouraging the agent to find parts of the world it hasn't visited before to avoid such penalties. This intuition is similar to that we apply to optimistic initialization—the states the agent visits are worse than it expects, so the agent is encouraged to try elsewhere. In fact, there is a method that uses negative intrinsic rewards to create the effect of optimistic initialization (Machado, Srinivasan & Bowling, 2015). \n\nWithout understanding how or whether the effects of cyclophobic rewards extend beyond the typical effects of other mostly negative intrinsic rewards, it is difficult to evaluate the significance of cyclophobic rewards. To the best of my understanding, all of the baseline intrinsic reward methods used for comparison in this paper use non-negative intrinsic rewards.\n\nMy other concern with the baselines is the state representation used. In Section 3.1, it says that the \"training setup of Parisi et al. (2021)\" is used, but I wasn't sure which aspects of the training setup were meant. I believe Parisi et al. used a panoramic state view that obscures anything behind walls or closed doors, but from Figure 1 in this paper, I get the impression that the agent's state representation includes the whole gridworld, including items behind walls—can you be more explicit about what state information the agents had access to in the experiments? Did all of the agents have access to the same state information? (If so, I'm guessing we should expect different results than those presented by Parisi et al., 2021.)\n\nI found that the structure of the paper was straightforward and easy to follow. The paper is well-written, for the most part. I appreciated how the authors anticipated many of my concerns throughout, and noted limitations accordingly. However, another concern I don't feel was addressed is the reasonableness of maintaining state histories for multiple views. MiniGrid environments are relatively small; do you feel that this method will be applicable as environments get larger?\n\nThe related work components represent one of the weaker parts of the paper; I did not find the relationships between the proposed method and the existing work very clearly explained.\n1. \"One way to do this is by approximating the counts via a density model as done by Ostrovski et al. (2017) and Bellemare et al. (2016).\" (p. 8) Given that your algorithm as described is tabular, I don't see the relevance of the translation to pseudo-counts.\n2. Do you consider RND and RIDE to be at all related to your method, or are they just described in the related work (p. 9) because they are included baselines in some of the experiments?\n3. \"Overall, prediction error methods require a model while our method is model-free.\" (p. 9) ← This statement is a little strange coming shortly after your summary of RND, which doesn't require a model in the traditional sense.\n\nMinor error that should be fixed:\n- \"We see that the cyclophobic intrinsic reward is necessary to reach a successful trajectory at all.\" (p. 8) Not sure what this means, since the figure only shows a comparison of two versions of cyclophobic agents against epsilon-greedy. Probably should be rephrased.\n\nTypos and grammatical suggestions (no intended impact on score)\n- \"In environments with sparse rewards finding\" (p. 1) → \"In environments with sparse rewards, finding\"\n- \"Random exploration (e.g. epsilon-greedy with sparse rewards)\" (p. 1) ← Makes it sound like sparse rewards are part of the exploration method rather than the environment—and isn't this still a problem even when rewards are dense?\n- \"This idea is further pushed by\" (p. 1) ← not sure what this means; may want to rephrase\n- \"like avoid everywhere.\" (p, 1) → \"like to avoid everywhere.\"\n- \"Instead rewarding novelty we\" (p. 1) → \"Instead of rewarding novelty, we\"\n- \"where an observation function Z : S → O maps the observation to the true state.\" (p. 2) ← The description is mixed up; Z maps the true state to the observation that the agent gets.\n- \"the cycle penalty is −l. That is, during exploration the cycle penalty can decrease indefinitely.\" (p. 2) ← This is a little awkward because if the penalty is decreasing, it sounds like it should be getting less negative. There are a few alternatives like referring to it as \"the cyclophobic reward\" or talking about the magnitude of the penalty increasing indefinitely, but I haven't thought of a way to completely avoid confusion.\n- \"how it can be build\" (p. 2) → \"how it can be built\"\n- \"is the sum of\" (p. 2) → might be better to say \"is the weighted sum of\" to account for rho.\n- \"restricting the views leads to\" (p. 2) ← \"restricting the view leads to\" might be a little clearer\n- \"V_1 is\", \"V_2 are\", \"V_3 is\" (p. 3) ← probably want to be consistent\n- \"what is front\" (p. 3) → \"what is immediately in front of it\"\n- \"to V_k the\" (Figure 1 caption) ← missing space?\n- \"several weights\" (p. 4) → There is one weight for each view, is that right? The word \"several\" is a little vague, and explaining where the selection of weights comes from would help solidify understanding.\n- \"the next paragraph.\" (p. 4) ← \"the next section\"?\n- \"hashcode\" or \"hash code\" (p. 5) ← probably want to be consistent\n- \"where m_t \\in R^k be\" (p. 5) → \"where m_t \\in R^k is\" \n- \"considered to be solved, if one\" (p. 5) → \"considered solved if one\"\n- \"our x-axis is shorter\" (p. 5) ← Shorter than what? Is this in comparison to the figures shown by Parisi et al. (2021)?\n- \"dificult\" (p. 6) → \"difficult\"\n- \"tasks show in Figure 3\" (p. 6) → \"tasks shown in Figure 3\"\n- \"can not\" (p. 7) → \"cannot\"\n- \"difficult set of\" (p. 7) → \"difficult sets of\"\n- \"performance as well dramatically\" (p. 7) → \"performance dramatically\"\n- \"state counts\" (pp. 7, 8, 9) → \"visitation counts\"\n- The \"count-based intrinsic reward\" (p. 8) used for the hierarchical learner with no cyclophobic reward is based on MBIE-EB (Strehl and Littman, 2008), right? A citation to where the choice of reward came from would be helpful context. \n- \"representations we achieve\" (p. 8) → \"representations do we achieve\"\n- \"The mechanism to check novelty\" (p. 9) → maybe \"This mechanism to check novelty\"?\n- \"amount of changes in a state transition\" (p. 9) ← Not sure what this means; what is a change in a state transition?\n- \"experiments for MiniGrid\" (p. 9) → \"experiments in MiniGrid\"?\n- If you get the chance to go through and make the colors for each method consistent (e.g. cyclophobic with hierarchical views could be the same blue in every figure), that would make the figures much easier to comprehend.\n\nMachado, M. C., Srinivasan, S., & Bowling, M. (2015). Domain-independent optimistic initialization for reinforcement learning. In AAAI Workshop: Learning for General Competency in Video Games 2015.\n\nStrehl, A. L. and Littman, M. L. (2008). An analysis of model-based interval estimation for Markov decision processes. Journal of Computer and System Sciences, 74(8):1309 – 1331.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "IqUoOtX8K6", "reviewer": "Reviewer_FEf9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces cyclophobic reinforcement learning, a negative intrinsic reward for exploration that incentivizes the agent not to revisit previously visited states. This intrinsic reward is computed over different cropped views of the environment by the agent and these are combined to compute the final reward. The proposed method is evaluate on mini grid and mini hack and is shown to lead to increased performance.", "summary_of_the_review": "Despite impressive results on MiniGrid and MiniHack, it is unclear how the proposed method scale to larger environment where it is not possible to count observations. Because of that I am not too sure what we can learn from cyclophobic RL as it is presented in the paper and for that reason I cannot recommend acceptance.", "strength_and_weaknesses": "Strengths:\n\n_ The paper is well written and easy to follow.\n_ I liked the empirical evaluation, it was done on many tasks from MiniHack and MiniGrid with a large set of baselines. The ablation study was also helpful to understand the impact of each component of the method.\n_ I also appreciate that the authors discuss the limitations of their method.\n_ Results on minigrid and MiniHack are impressive and clearly the benefits of cyclophobic RL over exisiting exploration techniques.\n\nWeaknesses:\n\n_ Overall it is hard to know how the proposed will scale to more complex problems like Atari games where the same state may not be seen multiple times (at least on the largest view). On the hand the narrow may not have enough local information to be useful. The counts of the observations might also be difficult to compute in that situation.\n_ From 1 (b) it looks like to obtain a smaller view a rotation is done in addition to cropping? Is that part necessary? How does it impact the performance of the algorithm. Seems like to do the orientation of the agent must be known.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8mAbnAEXa6", "reviewer": "Reviewer_CKV7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an intrinsic reward for RL exploration based on loop-closure detecting. When a state or a central patch of the state is visited, a penalty will be given to the agent to encourage the agents to visit novel states. Such an intrinsic reward design is combined with Q-learning to conduct experiments on MiniGrid and MiniHack environments.", "summary_of_the_review": "On the technical part, the idea of building a hierarchy of patches for developing a set of intrinsic rewards is interesting. However, the contribution is not well supported by the experiments, which further makes the novelty of this work limited. ", "strength_and_weaknesses": "## Strength\nThis paper is clearly written and easy to follow. The loop-closure-based intrinsic reward is intuitive and the method is easy to implement. The idea of building a hierarchy of patches for loop detecting is novel. Experiments are conducted on two challenging experiments, which is good.\n\n\n## Weakness\n\nMy concerns are mainly about novelty and experiments, which are listed below. \n\n1. **Limited novelty**: The idea of detecting loops to encourage exploration isn't really a novel idea. Its practice in deep RL literature can be at least traced back to [1], which implements an additional loss to predict loops. Even the idea of using loop-based rewards is well executed as a specific technique in the NoveID method [2]. The hierarchy of patches for loop detection is interesting and could be counted as a great practical technique for realizing loop detection. However, I would be concerned about whether the paper contributes enough to the community, particularly based on the limited experiments (see my following comments).\n\n[1] Learning to Navigate in Complex Environments, Piotr Mirowski et. al, ICLR 2017\n\n[2] NovelD: A Simple yet Effective Exploration Criterion, Tianjun Zhang, Huazhe Xu, Xiaolong Wang, Yi Wu, Kurt Keutzer, Joseph E. Gonzalez, Yuandong Tian, NeurIPS 2021.\n\n2. **Missing baselines**: The paper primarily compares with two baselines, i.e., RIDE and RND. However, many recently developed baselines are omitted. More importantly, many of them are also tested on the miniGrid environment. Representative examples include NoveID [2], AMiGo [3], MADE [4] etc.\n\n[3] Learning with AMIGo: Adversarially Motivated Intrinsic Goals, Andres Campero, Roberta Raileanu, Heinrich Kuttler, Joshua B. Tenenbaum, Tim Rocktäschel, Edward Grefenstette, ICLR 2021\n\n[4] MADE: Exploration via Maximizing Deviation from Explored Regions, Tianjun Zhang, Paria Rashidinejad, Jiantao Jiao, Yuandong Tian, Joseph E. Gonzalez, Stuart Russell, NeurIPS 2021.\n\n3. **Experiment results seem unfinished**. It is particularly wired to me that there are no reward curves at all for the MultiRoom-N12-S10 scenario in Fig 2. At least, if you check the RIDE paper, RIDE can definitely achieve non-zero rewards by training from scratch. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject