LECO: Learnable Episodic Count for Task-Specific Intrinsic Reward
Authors: Daejin Jo
Sungwoong Kim
Daniel Wontae Nam
Taehwan Kwon
Seungeun Rho
Jongmin Kim
Donghoon Lee
Venue: NeurIPS 2022 Accept (2022)
Keywords: Reinforcement learning
Intrinsic reward
exploration

Abstract: Episodic count has been widely used to design a simple yet effective intrinsic motivation for reinforcement learning with a sparse reward. However, the use of episodic count in a high-dimensional state space as well as over a long episode time requires a thorough state compression and fast hashing, which hinders rigorous exploitation of it in such hard and complex exploration environments. Moreover, the interference from task-irrelevant observations in the episodic count may cause its intrinsic motivation to overlook task-related important changes of states, and the novelty in an episodic manner can lead to repeatedly revisit the familiar states across episodes. In order to resolve these issues, in this paper, we propose a learnable hash-based episodic count, which we name LECO, that efficiently performs as a task-specific intrinsic reward in hard exploration problems. In particular, the proposed intrinsic reward consists of the episodic novelty and the task-specific modulation where the former employs a vector quantized variational autoencoder to automatically obtain the discrete state codes for fast counting while the latter regulates the episodic novelty by learning a modulator to optimize the task-specific extrinsic reward. The proposed LECO specifically enables the automatic transition from exploration to exploitation during reinforcement learning. We experimentally show that in contrast to the previous exploration methods LECO successfully solves hard exploration problems and also scales to large state spaces through the most difficult tasks in MiniGrid and DMLab environments.

LECO: Learnable Episodic Count for Task-Specific Intrinsic Reward
Daejin Jo
Sungwoong Kim"
Daniel Wontae Nam"
Tachwan Kwon
Seungeun Rho
Jongmin Kim
Donghoon Lee
Kakao Brain
Seongnam, South Korea
{daejin. jo, swkim, dwtnam, taehwan kwon, seungeun, rho, jukim, dhlee}
@kakaobrain.com com
Abstract
Episodic count has been widely used to design simple yet effective intrinsic motivation for reinforcement learning with sparse reward. However, the use episodic count in high-dimensional state space well over long episode time requires a thorough state compression and fast hashing, which hinders rigorous exploitation of it in such hard and complex exploration environments. Moreover, the interference from task-irrelevant observations in the episodic count may cause its intrinsic motivation overlook task-related important changes of states, and the novelty in an episodic manner can lead to repeatedly revisit the familiar states across episodes. In order resolve these issues, this paper, we propose a learnable hash-based episodic count, which we name LECO, that efficiently performs as task-specific intrinsic reward in hard exploration problems. In particular, the proposed intrinsic reward consists of the episodic novelty and the task- specific modulation where the former employs a vector quantized variational autoencoder to automatically obtain the discrete state codes for fast counting while the latter regulates the episodic novelty by learning a modulator optimize the task-specific extrinsic reward. The proposed LECO specifically enables the automatic transition from exploration to exploitation during reinforcement learning. We experimentally show that in contrast to the previous exploration methods LECO successfully solves hard exploration problems and also scales to large state spaces through the most difficult tasks in MiniGrid and DMLab environments.
Introduction
In reinforcement learning (RL), tackling the problem of sparse rewards still remains great challenge. Many works have been proposed develop dense synthetic reward, which is generally called intrinsic reward, to provide the motivation for exploration in the absence extrinsic rewards from the
29 37 15 21). Among them, count-based exploration 5) that encourages the RL agent to visit novel states, is one of the most popular classes for intrinsic motivation since is simple and effective in many tasks, and more importantly is theoretically justified B1 20 In general, the state novelty can be measured within a single episode or through an entire task across episodes. In recent RL problems, environments are often changed over episodes or even procedurally generated within the same task 7 g [7]. Therefore,
environment [5] 26 T 28
1 38 33
4 25 24 33 23
"Contributed equally.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

per-episode novelty by episodic count has been widely adapted in modern hard exploration problems 28 38
However, in high-dimensional and/or continuous state spaces, state counts need to be approximated in an abstract space since most states are likely to be unique and novel in the original space, leading to insufficient exploration [4] [25 28 I. In addition, computation of episodic count at every time step based on the episodic memory could be burdensome if an episode lasts for long time and therefore requires fast hashing 33 Previous works have discretized the state space to obtain the hash code [33 14 9
for fast pseudo-count but they have not extensively applied it to recent difficult exploration problems such as procedurally generated and partially observable environments.
Moreover, existing count-based exploration methods have mostly employed task-agnostic state novelty, and thereby task-irrelevant count may disturb its intrinsic reward bonus to faithfully reflect task-related information [19 2 18 40]. This problem could be severe in exploration by per-episode count. For example, task-wise familiar states across episodes can have high novelty in each episode, which confines overall exploration into smaller areas, while some states that are highly related to extrinsic rewards need to be repeatedly revisited even in a single episode for obtaining extrinsic rewards more densely.
In this work, we propose a hash-based episodic count that is learned to provide task-specific in- trinsic motivation in solving hard exploration problems. The proposed Learnalle Episodic COunt (LECO) produces task-specific intrinsic rewards that are efficiently and practically applicable to high-dimensional states, long-horizon episodes, procedurally generated environments, and highly sparse extrinsic rewards. In specific, the proposed intrinsic reward is composed of the episodic count-based novelty and the task-specific modulation. Our episodic count makes use of vector quantized variational autoencoder (VQ-VAE) [34] to automatically obtain the discrete state codes for a fast counting. In addition, we exploit a learnable modulator to control the episodic novelty in relation to task-specific extrinsic rewards. The proposed LECO also adaptively shifts the RL phase from exploration to exploitation. Experimental results on the most difficult exploration tasks in procedurally generated environments of MiniGrid (7) and DMLab [ show that LECO efficiently solves such tasks and outperforms previous state-of state-of-the-art ff-the-art art exploration methods.
Our main contributions can be summarized as follows:
A novel learnable episodic count is proposed efficiently provide task-specific intrinsic motivation to the RL agent under complex environments with sparse rewards.
A complementary synthesis of learnable hash code based on VQ- VAE and Icarnable intrinsic reward modulator enables task-specific transition from exploration to exploitation.
The proposed LECO demonstrates consistent performance improvements over previous exploration methods in solving various hard exploration benchmark tasks.
2
Related Work
Count-based exploration. Count-based exploration first proposed in tabular tasks (31. which promotes the agent to reduce its uncertainty by visiting unfamiliar states corresponding to low counts. However, it is basically ineffective in high-dimensional state spaces since every state would be different. Therefore, several recent works have utilized count-based exploration for high-dimensional environments. For instance, pseudo counts are derived from density models using context tree switching 4 or PixelCNN 25 More recently, RND [5] has approximated the state visitation count by the difference between random fixed target network and trained predictor network. RND has also been used in other methods for measuring the state novelty [1] (7]. On the other hand, a number of algorithms have transformed the dimeeniinii state to the discretized hash code for fast visitation counting by a hash table (33 [0]. Similarly, we obtain the episodic count by the discretized hash code. However, we exploit learnable hash codes by VQ-VAE that trained to approximate the state density. In addition, different from [14] that directly produces global code by VQ-VAE, we concatenate local codes as in the original VQ-VAE for generating single state code.
Task-specific intrinsic motivation. While count-based exploration methods have typically utilized the state novelty in task-agnostic manner, curiosity- methods have tried consider task-related information for intrinsic motivation by learning the environment dynamics 226 29 19 28
They

---

have developed intrinsic rewards based on the prediction error upon the agent's actions. Among them, recently, number of approaches have exploited the task-dependent state novelty using the
information bottleneck in order to ignore task-irrelevant distractions [19 2 18]. Meanwhile, there
have been a number of attempts to directly learn the intrinsic reward function to ultimately optimize
the task-specific extrinsic rewards 440 39 36]. Their algorithms are generally performed under the
meta veeta-gradient framework, LECO also adopts the idea of learnable rewards and the gradient- bi-level optimization [40] to construct a task-specific modulator for intrinsic rewards. However, unlike those works that do not work well with sparse extrinsic rewards, LECO takes a full advantage of intrinsic motivation by the task-agnostic state novelty based the episodic count before obtaining the extrinsic rewards.
Exploration for procedurally generated environments. One of the important challenging RL problems is robust exploration in a procedurally generated environment where the environment (partially) changes between episodes, and hence the agent needs to explore very large state space and have strong generalization ability. Many recent RL benchmarks deal with procedurally generated environments 3 [22 & 17, and accordingly state-of-the-ard exploration algorithms have tried to improve sample efficiency in such environments. RIDE (28) has used a change between successive states latent space by an adversarial policy. NovelD (38) has exploited regulated difference of the novelty between consecutive states, and MADE (37 has devised exploration by maximizing the deviation from previously explored regions. AMIGO 6 has proposed a teacher that generates count-based intrinsic goals. In contrast to these handcrafted task-agnostic intrinsic rewards, LECO learns to leverage task-level information to guide the count-based exploration especially at the episode level due to the procedurally generated environment. Moreover, LECO also shows improved performances on DMLab tasks with nigher-dimensions partial observations.
intrinsic reward while AGAC 133 has encouraged action diversity
3 Learnable Episodic Count via Task-Specific Modulation
3.1 Notation
Our problem setting follows typical RL problem described using Markov Decision Process defined by a set of states S, a set of actions A, and a transition function T : S x A + S. An RL agent interacting with the environment samples its actions from policy, which is an action distribution function π + (AA), and receives a reward, r : AxAA - R. upon state transition. The objective of the agent is to learn a policy that maximizes the expected sum of rewards G. = E n
7 rc+k+1]
where y E [0, 1] is a discount factor.
Among various sub- babb-roblems that RL faces, balancing the trade-off between exploration and exploita- tion still remains a fundamental problem (32]. Especially, efficient exploration is important when dealing with sparse rewards. One approach to solving such problem is to design an algorithm that directly generates exploring behavior. More specifically, a widely adopted method is to design an intrinsic motivation through form of a reward such that the reward which the agent receives at time t given by 7, = rf + ari, where r" is the extrinsic reward from the environment, 7d the intrinsic reward designed for exploration, and a is a hyperparameter for scaling (28).
3.2 LECO Intrinsic Reward
The use of state visitation count for intrinsic rewards based on the state novelty has been a core concept in several previous studies (8 38
30 16] due to its simplicity, task-agnostic property, and theoretical basis. However, such state-novelty measured from visitation counts monotonically decreases as more experiences accumulate, equally for both important and unimportant states. Thus, an agent would be discouraged to visit even the task-important states after a certain period of time, and the exploration towards those states vanishes [.. Furthermore, an intrinsic bonus based on the state visitation could lead the agent to only pursue any state novelty, especially in the aartiallyobbservable and sparse reward environments. For example, imagine task in which the goal is to find a colored key hidden a box and unlock a door that has the same color as the key. If the box changes its color to random color every time when it is opened, the agent would repeatedly open and close the box just to obtain the intrinsic bonus. In this task, the agent should be encouraged to find the door after obtaining the key and discouraged to repeat task- inse-inrelevant actions. We directly build upon this idea
13
3

---

Vector Quantized Encoding
Episodic Novelty
Code book
Episodic hash memory
rp
Hash
Count
1)
S
Enc
Dec
spi+1
Tt
Tt
spec-fi Modulation
C++1
2
MLP
Figure 1: The schematic diagram of the intrinsic reward in LECO. The states at time and + 1 are encoded and quantized to the embedding vector zi and hash c++1. Then, each is fed into the task-specific modulator and episodic counter, respectively, to produce the task-specific modulation ri and episodic intrinsic reward r which then combined to produce LECO's intrinsic reward r/.
by decomposing the intrinsic reward into the state-novelty and task-specific modulation, to balance exploration and exploitation, respectively.
We design our intrinsic motivation to encourage the discovery of state novelty that is adaptively controlled to maximize the extrinsic return, hence the name Learnable Episodic COunt for task- specific motivation. Specifically, we define the general form of intrinsic reward sum of two parts,
r/ Sc, at 5t-1) (1 (8(+1) (r St, de);
where (8++))))) 1/ /tt///Nop(spp (14())+1) is the intrinsic reward from the episodic state-novelty, Nep(s) is the episodic visitation count of state 8, and nT is the task-specific modulation with X E [0, as its weighting factor. The first term measures the episodic novelty by state visitation count within the current episode that is initialized to zero at the beginning of each episode. The second term learns to modulate the episodic novelty in the first term according whether the state is beneficial or harmful in achieving the task-specific objective. Note that the previous action 0t-1 is included as the input to reward calculation to provide extra transition information. A schematic diagram of the proposed intrinsic reward can be found in Figure
LECO comprises of two submodules: 1) learnable hash that encodes input observations discrete hash codes for episodic counts and 2) a modulator that maximizes the task-specific extrinsic return by adaptively scaling the episodic novelty. Here, the modulator additively scales the episodic novelty. Compared to multiplicative scaling, the additive modulation allows the episodic novelty to be retained solely when the modulator does not generate enough signals due the scarcity of extrinsic rewards.
3.3 Episodic Count via Vector Quantized Hashing
Previous researches have extended classic state visitation counts 31 to high-dimensional state spaces 4 33 A generalization of the classic count-based approach by sensiity- hashing (LSH) on the continuous embedding space was proposed [33] where the latent space obtained by an autoencoder (AE). The use of hash table in their approach enables counting the occurrences in a constant time, taking the advantage of scalability, especially in modern distributed RL systems 27) However, such LSH does not take into account the spatial information due to its data independency [5). As an alternative to AE, we propose to utilize vector quantized variational autoencoder (VQ-VAE) 34 that directly optimizes the discrete compressed representation of state, producing a hash code without LSH.
In LECO, state s; is mapped to hash code et by vector quantized hashing based on VQ-VAE comprised of an encoder, a decoder, and a codebook with dene. Odce and 0g = {ex E K respective parameters. Here, K is the number of codes, and e사 is the D-dimensional embedding of the kth code. In detail, first, the input state 8 is encoded to a spatial representation 2 = (.mmmc((pp Oenc). where zf E RwxxxD and Sene(:; Oenc) an encoder of VQ-VAE Then, the representation of is mapped to =A by finding the nearest vector in the codebook O, for each vector in the spatial position i (1, ..., w x h). To more specific, let us denote the vector in position
Jkm as their
2f and 2 as 2 (i) and 2f()), respectively. Then, 2f (i) arg min =1 exll2- and the hash code is defined as

---

Visitation heatmap
ObstructedMazze-uull
LECO
VQ-only
Intrinsic
reward
w
0.5
LECO
VQ-only
0.0
0.0
1.0 1.5
2.0
2.5
3.0
1e8 steps
Figure 2: Obst obstructtedMaee-ulll Enviromment (Left). The changes in the coverage of visitations (Right Top) the training progresses show that LECO modulates the episodic reward (Right Bottom) and so the agent focuses on the task-specifie objective. On the other hand, the agent with intrinsic motivation from the episodic count only, labeled as VQ-only, remains distracted by the episodic state-novelty even in the late training phase. We provide more examples with further discussion in subsection D.2 Appendix.
the sequence of indices such that [x where k; arg minngl==f()) - ea Finally, the decoder then predicts the mean μ = fde Odec) of an isotropic Gaussian distribution p(s|=?) with 0² where f- Qee)) the decoder of the VQ-VAE.
Following the conventional VQ- VAE training, the hash function is updated according to the gradient
T
T
Σ
V
logp(s<|2() + |sg[=]] x 12 || sgl=;]11 sgl=/]12)
(2)
t=
where sg stands for stopgradient operator, 7 is an unroll length of the agent, and 0ng {fene, Osec Oc).
3.4 Task-Specific Intrinsic Modulation
In sparse reward environment, when an extrinsic reward signal has little or no presence, having ausk-agnottic intrinsic motivation can be significantly beneficial in guiding the agent to efficiently explore the environment until the discovery of more extrinsic signals. However, task mask-general intrinsic reward such as the inverse of the episodic count may become a distracting signal in the later exploitation since it is not reduced with the progression of learning. We thereby propose to include task-specific modulation that aims to ultimately maximize the extrinsic return in the intrinsic reward formulation of LECO. The task-speeific modulation can be seen as a dynamic factor for controlling the exploration and exploitation, based on the extrinsic rewards earned from the environment.
For this, leverage the bi-level optimization proposed in LIRPG 40 First, let policy parameter- ized by A be updated using the policy gradient on the sum of intrinsic and extrinsic rewards,
=0+
V
E
> " + art)).
where 17 is the step size. We then construct our task-specific modulator for the intrinsic reward in a,; P..) where 0,, is the
Equation I as parametric model such that ri = (1 A)r + modulator parameters. Now, the task-specific modulator trained maximize only the extrinsic rewards using the corresponding policy gradient such that
0g Om + Tha VaaVa vo = Om + Thu Va,
Erg
5

---

where Ta is the step size for updating Bw. Here, it is noted that the expectation is performed on the updated parameters I' that is affected by our intrinsic reward and accordingly 0.a. Therefore, fn is updated by the chain rule such that Va = Vo J Va, 0' This can be considered bi-level optimization framework popularly used in meta-learning, where the inner-loop and the outer-loop are corresponding to Equation 3 and Equation 4, respectively.
Model for task-specific modulation.
After executing an action as, a task-specific modulation
r E (-1,1) is modeled as
P (ac-1, S1, aj; 0aa) tanh [aa- T fu(zf; act Bu))
(5)
where 1a is the one-hot vector for action a, fu R + R| is a MLP network, and 2; is shared with VQ-VAE based encoding to make the modulator efficient in training. We choose tanh, which generates both positive and negative modulation, so that even in the absence of meaningful signals from episodic counts, the modulator can according to LIRPG and converge to value that maximizes the extrinsic objective.
4 Experiments
The key concept of LECO is the learnable and modulated episodic count, In other words, compared to naive episodic count that assigns the state novelty proportional to its visitation count within an episode, LECO aims to produce low intrinsic rewards to task-irrelevant states regardless of its episodic count. We demonstrate such behavior by comparing the mean intrinsic rewards obtained by LECO to those from exploration with just the episodic count-based intrinsic reward in Figure 2 The results show that while the episodic novelty remains high, modulation in LECO decreases the intrinsic reward such that the explorative behavior has vanished towards the end of training and the agent focuses the task-speciff exploitation. Apart from the novelty based on the agent's location, in an environments where agent can pickup and drop objects, such actions can create task-irrelevant novel states, further complicating the search space. The task-specific modulation of LECO mitigate those task-irrelevant behaviors as shown in FFigure a0 Figure 12 in Appendix.
Environments. As LECO operates multiple components, design the experiments thor- oughly cover various aspects, especially the adaptiveness and the scalability. Since the real world environment is nonstationary and requires high-dimensional perception, it is crucial for an agent to efficiently explore the constantly changing environment, even at large scale. Thus, we first demonstrate the explorative performance of LECO in procedurally generated environments of MiniGrid (7]. Then, we further extend the experiments to large scale environments of DMLab tasks.
Baselines. In order to present the contribution of each component of our proposed method, we compare LECO to its variants that incrementally add each component to bascline without the intrinsic motivation which we denote as no-Int for the rest of the section. Similarly, a variant with only the vector quantization episodic counts is denoted as VQ-only. Additionally, a variant that uses Furthermore, in order to show that
AE-LSH (33 as the hash function is denoted as LECO our proposed algorithm is beyond a mere combination of VQ-VAE and LIRPG, we compare the LECO to a straightforward combination of the two, denoted as LECO-naive.
The notInt baseline is an IMPALA-basee agent 12), with minor architectural modifications according to the environments. Details on the base model and the architectures can be found in Appendix A Other baselines include DSC (Down Sampled Cell) [10), AE-LSH [33] NovelD [38], and AGAC [3] where DSC uses a non-learnable hash for state compression and AE-LSH uses learnable hash. NovelD and AGAC are faate-of-thh-arr exploration methods for procedurally generated environments. Here, for fair assessment of LECO, we implement the episodic count of NovelLD and AGAC using
the VQ hashing as done in LECO for both of MiniGrid and DMLab tasks.
Experimental Setup. In MiniGrid, LECO was trained using two A 100 GPUs with a batch size of 768 for 18 hours. In DMLab, we used eight V100 GPUs with batch size of 576 for 8 hours. The unroll length was = 96 for all tasks and same LSTMd policy network architecture was used for LECO and all other baselines. Details hyperparameters, model architectures, and training

---

0.6
0.75
0.75
0.4
0.25
0.25
15
0.5
2.0
Steps
Steps
1wd
Steps
1wa
(a) MultiRoom-Ne
(b) KeyCorridor--ao S4R3
(c) Key KeyCorridor-See
0.8
0./5
0.6
0.50
0.4
0.4
AE-LSH
noint
0.25
0.4
0.2
LECOIAE LSH)
1.0
0.0
1.0
3.5
2.5
3.0
Steps
165
Steps
168
Steps
165
(d) ObstructedMaze-10 IQ
(c) ObstruuctedMaze 2Q
(f) ObstructedMaze-
Figure 3: Comparison of LECO and its variations six selected MiniGrid tasks. The final choice of design (LECO) shows superior performance compared to its variants. The shaded area represents range of standard deviation over runs with different random seeds.
settings are provided in Appendix B Official codes to run the algorithm and the experiments will be available
4.1 MiniGrid
MiniGrid 7 is procedurally generated gridworld environment specifically designed to test and develop exploration algorithms under sparse rewards and partial observations. A typical world in MiniGrid consists of M x N tiles, where each tile is assigned to be objects such as floor, wall. lava, door, key, ball, box, and goal. The agent deployed navigates through the world, collecting objects and solving sub-tasks such opening a door using key, reach a goal placed somewhere in the world. Of the many types of environments included, we test the performance of LECO in three types of MultiRoom. KeyCorridor, and ObstructedMaze en- vironments. More specifically, we use total of six configurations of those environments namely MultiRoom-N6 (MRN6), KeyCorridorSe ObstructedMaze -Q (OMIQ), Obstruct eedeeeeMaeeeeee (OM2Q), and ObstructedMaz Full (OMFull), roughly ordered increasing level of difficulty. The choices were made to reflect a wide range of difficulty including the most difficult task, while evaluating similar tasks used by the previous baselines.
(KCS4R3) KeyCorri corooriiidrrrrRR. (KCS6R3),
We emphasize that for all tasks the baselines and LECO only use the original partial input observations, in which 7x7x3 tensor given the partially-observabeee the policy network and intrinsic module as in [6]. This setting is more challenging than the other settings such as the partial observation for the policy and full observation for the intrinsic module as in 128, (8] (3). and the partial observation for the policy and meta information such as the agent's 's
grid cell is the input observation for both
position for the intrinsic module as [3)
We first compare LECO to its variations as shown in Figure 3 The variations include no-Int, DSC, AE-LSH, VQ-only, and LECO(AE-LSH) The results show that LECO solved all six tasks while VQ-only solved three, MRN6, OM1 Q, OM2Q, and show slow learning trend in OMFull. On the other hand, compared DSC, learnable hashing methods for using episodic count as intrinsic reward but with heuristic quantization, VQ-only outperforms DSC as DSC solved only one out of the six tasks. While AE-ISH solved the same number of tasks VQ-only, the scores of VQ-only higher in all tasks.
Without the task-specific modulators, the episodic count-only methods such VQ-only and AE-LSH, tend to perform worse in KeyCorridor tasks than in ObstructedMaze tasks due to the irrelvatle

---

1.00
0.6
0.75
0.4
0.25
0.25
155
0.0
1.5
0.5
2.0
Steps
1wd
Steps
1wd
Steps
1ma
(a) MultiRoom-Ne
(b) KeyCorridor--ao S4R3
(c) Key KeyCorridor-See
0.8
LECO
0.6
AGAC
0.6
Novell)
0.4
0.4
0.4
AE LSH
0.2
0.2
0.2
0.0
0.0
0.5
1.5
0.0
1.0
0.5
1.0
3.5 Steps
2.0
2.5
3.0
Steps
165
Steps
168
165
(d) ObstructedMaze-10 1Q
(c) Obstructeddazee-2 2Q
(f) ObstructeddMae--ui
Figure 4: Comparison of LECO to baselines six selected MiniGrid tasks. LECO shows the most promising performance compared to related existing methods in the field. The shaded area represents range of standard deviation over 3 runs with different random seeds.
state novelties. More concretely, in CM1 Q and OM2Q, the locked doors are generally located near the keys making it easier for the agent to discover the sequence of opening the locked door by exploration, while in KeyCorridor, even though the map is more simple, the doors are generally placed further away from the keys and thus increased search depth hinders the episodic count-only methods from opening the locked door and eventually solving tasks. We visualize the agent's 's behavior with and without the modulation in subsecttion D.I of Appendix which depicts the episodic count-only agent choosing task ireeearelevatt actions along the way in KeyCorridor task.
As shown in Figure 3, LECO(AE-LSH) generally performs better than AE-LSH except OM1 Q and OM2Q. However, the performances of LECO(AE-LSH) are lower than those of LECO on all tasks, which shows the benefits from VAE for the episodic counts. We analyze the benefits of LECO compared LECO(AE-LSH) over the hash space (see subsection D.3 Appendix). As shown by the results of no-Int, a policy without any intrinsic reward is unable to solve any of the six tasks.
Combining all comparisons gives indications that 1) episodic exploration tasks with procedurally generated environments, 2) vector quantization-base counting is more effective than heuristic cell discretization (DSC) as well as AE-LSH, and 3) the task -peciiccc modulation of LECO further improves the performances of the episodic count-based approach.
enables training in sparse reward
Moreover, we compare the performance of LECO with other baselines of AE-LSH, AGAC, and NovelD on the same six MiniGrid tasks. As shown in Figure 4, only LECO solves all six tasks including OMPull. Here, it is noted that the original implementations of AGAC and NovelD used the agent's 's position and the full observations, respectively, for the episodic counts their methods while used the original partial observations in our implementations of AGAC and NovelD. In fact, the original performances of NovelD and AGAC are better in their papers, and they have also shown to solve OMFull in their papers. However, we observe that, when correct their official codes to utilize only the partial observations in obtaining the episodic counts, their performances are dropped as in our results. Recently, MADE (37) has shown to produce state-of-the-art performances on MiniGrid including OMFull, 11 and MADE seems to be a bit more sample efficient than LECO according Figure 7 in [37]. However, in this paper, it is somewhat difficult for us to compare the learning progress of LECO with that of MADE since we failed to reproduce their performances given rough descriptions on MiniGrid implementations
their paper and the absence of support for MiniGrid experiments in their official codes. In MultiRoom and KeyCorrido environments, the increasing rates in scores of LECO is slightly slower than those of NovelD since the modulator in I LECO requires bit more experiences to train, however, the advantage of having the modulator clearly shown through significant performance improvement in the harder exploration tasks. In addition, LECO is more stable in the later phase of KeyCorridor training.
8

---

200
LECO
Human NNorallized Score
VQ-only
50
DSC
250
no-Int
40
Human Normaliza
AE-LSH
100
LECO(AE-LSH)
NovelD
50
20
RND
RIDE
10
0-
2
5
Steps
1e8
Steps
(a) lasertag. three opponents small
(b) lasertag three opponents large
Figure 5: Learning curves on two selected DMLab tasks. Compared to all of variants and baselines LECO shows the highest performance. Especially in aasertag_three bea_gg_hhrree_ppoetteeeeee large, only LECO and VQ-only is able to solve the task. The y-axis represents the human normalized score. The shaded area represents the range of standard deviation over 3 runs of random seeds.
4.2 DMLab
DMLab 3 is learning environment consisting various tasks, each involving different visuals and objectives in simulated 3D) world. We consider DMLab for its partial-observabiliy that naturally arises from first-person perspective and its complexity from image- based observation with fine details and randomly generated levels demonstrate the scalability of LECO. We compare LECO and LECO(AE-LSH) to no-Int, episodic count methods based on AE-LSH, DSC, and VQ-only, and state-of-the-art exploration methods including NovelD, RND (5) and RIDE (28) on two individual exploration-heavy tasks. For RND and RIDE, the architectures of neural networks are based on the RND network used NovelD for MiniGrid and the forward/inverse networks used RIDE for MiniGrid We use the same RND networks for RND and NovelD and the same architecture of VQ for LECO, NovelID, and RIDE their episodic count module.
We select two specific tasks from DMLab, lasertag_three ttcrreeepppoentsssseteee __arge, in order to show LECO's capability solve explo- ration tasks at large scale. In both tasks, an agent is deployed in a procedurally generated 3-D environment where it has to search and shoot three opponents moving around the maze-like map (see Figures 16 to 18 for example screenshots). Since the reward is only given when the agent successfully tags the opponents by shooting a laser, these tasks are known be difficult solve without an advanced exploration strategy and are characterized as sparse reward problems. The small and large stand for the size of the map, which determines the reward sparsity. In Fig ure 5] we present the learning curves of LECO and baselines on these tasks. Each task has its own independent agent, but the architectures and hyperparameters remain consistent across these tasks. To the best of our knowledge, experiments for RND, RIDE, and NovelD on DMLab-lasertags has not been published so far. As shown in Figure 5] these models show decent performances on aeserta__three Stchhree_pppoentteeeeeeee solve the task of
s_seeenttssssallll and
1, but perform worse than LECO and moreover do not wetpeeeetttsseeeeeeeeeeeeeeeeeee at all.
4.3 LECO ys LECO-naive.
One of the key contributions of the LECO the formulation of task-specific modulation of the task-agnostic episodic count in the additive form. In order to show that LECO is more than simple combination of VQ-VAE and LIRPG, we experimentally compare LECO naive combination of VQ-VAE and LIRPG where we directly optimize the episodic counter using the meta- geetiggraieett of extrinsic rewards. More coneretely. we define the intrinsic reward ri(01-1, ag, $141) = r/ (a,-1, St, r,. (s(+1) and denote as LECO-naive, Just from the for- mulation we can speculate that would insufficient to solve sparse reward problems since

---

100
0.5
3.5
0.75
0.4
350
350
0.5
0.25
2.0
1.5
Steps
Twh
Stops
Twll
Steps
1wa
(a) MultiRoom-NG
(b) KeyCorridor-
(c) ObstructedMaze-IQ
Human Normalized Score
200
LECO
Human Norrmlizedd
LECO-na
40
LECO/AE-LSHI
100
0-
(d) lasertag three setpepppppppsssneeeeeee small
(e) lasertag. threeeerrrrr.. cereepoppnnets__eeee
Figure 6: Comparison between different variants of LECO on three selected MiniGrid tasks (a)-(c) and the two DMLab tasks (d)-(e). LECO clearly outperforms LECO-naive in all tasks. The shaded area represents range of standard deviation over 3 runs with different random seeds.
cannot generate dense signals from sparse extrinsic rewards, especially in the early stages RL. We compare the performances of LECO LECO-naive along with LECH(AE-LSH) Minigrid(MRNG KCS4R3, and OM1Q), and DMLab(lasertag-tie there Satrree__ppeeeetteeeee s small and lasertag- la rge). As shown in FFiuure 6 the performances of LECO-naivv than those both LECO and LECO(AE-LSH) on tasks, showing that the proposed additive formulation enables taking the full advantages of both parts and accordingly solve very hard exploration problems with automatic transition from exploration to exploitation.
lower
5 Conclusion
This paper proposes learnable episodic count to produce task-specific intrinsic rewards for hard exploration problems including high-dimensional state spaces and procedurally generated environ- ments. specific, the hash code based on VQ- is used for computing the episodic state count and the corresponding state novelty, and this episodic novelty is regulated by the task-specific mod- ulator that is trained to maximize the extrinsic rewards. Experimental results demonstrate that proposed task-specific exploration based on the episodic count significantly outperforms the previous state-of-ther ofhth-art exploration methods on the most difficult tasks of MiniGrid and DMLab. However, the proposed intrinsic reward also has a number of limitations. First limitation comes from solving tasks that require intensive leveraging of the inter-episode novelty. For example, objects of negative reward could be common among different episodes, and in that case, the inter-episode novelty can lead an agent to avoid those objects. Second limitation is dealing with somewhat difficult bi-level optimization. The use of implicit gradients and the reformulated single-level problem be explored to efficiently solve it. For future works, we will extend our work to combine the inter-episode novelty and devise more efficient training of task-specific intrinsic reward to further improve the scalability. Moreover, we will perform experiments on all DMLab tasks with large-scale transformer-hact networks and also apply the other card-exploration environments.
Acknowledgments and Disclosure of Funding
We would like to thank Brain Cloud Team at Kakao Brain for their support.
10

---

References
[1] Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martin Arjovsky, Alexander Pritzel, Andrew Bolt, and Charles Blundell. Never give up: Learning directed exploration strategies. In International Conference on Learning Representations, 2020.
[2] Chenjia Bai, Lingxiao Wang.
Lei Han, Animesh Garg, Jianye Hao, Peng Liu, and Zhaoran Wang. Dynamic bottleneck for robust self-supervised exploration. In International Conference on Neural Information Processing Systems, 2021.
[3] Charles Beattie, Joel Z. Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Victor Valdés, Amir Sadik, Julian Schritwieser, Keith Anderson, Sarah York, Max Cant, Adam Cain, Adrian Bolton, Stephen Gaffney, Helen King, Demis Hassabis, Shane Legg, and Stig Petersen. Deepmind lab. CoRR, abs/1612.03801, 2016.
[4] Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, 2016.
[5] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2019.
[6] Andres Campero, Roberta Raileanu, Heinrich Kuttler, Joshua B. Tenenbaum, Tim Rocktäschel, and Edward Grefenstette. Learning with (amigJo: Adversarially motivated intrinsic goals. In International Conference on Learning Representations, 2021.
[7] Maxime Chevalier-Boisvert, Lucas Willems, and Suman Pal. Minimalistic gridworld environ- ment for openai gym. https://githh:i wwtp///wwww..c//ccoo.o.oooooc.at
gym-gimiggiiiie 2018.
[8] Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning, In International Conference Machine Learning, 2020.
[9] Dane Comeil, Wulfram Gerstner, and Johanni Brea. Efficient model-based deep reinforcement learning with variational state tabulation. In International Conference Machine Learning, 2018.
[10] Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return, then explore. Nature, 590(7847):580-566 2021.
[11] Lasse Espeholt, Raphaël Marinier, Piotr Stanezyk, Ke Wang, and Marcin Michalski. Seed rl: Scalable and efficient deep-rl with accelerated central inference. In International Conference on Learning Representations, 2020.
[12] Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference on Machine Learning, pages 1407-1416. PMLR, 2018.
[13] Yannis Flet-Berliac, Johan Ferret, Olivier Pietquin, Philippe Preux, and Matthieu Geist. Adver- sarially guided actor-critic. In International Conference on Learning Representations, 2021.
[14] Petros Giannakopoulos, Aggelos Pikrakis, and Yannis Cotronis. Neural discrete abstraction of high-dimensional spaces: A case study in reinforcement learning. In 2020 28th European Signal Processing Conference (EUSIPCÓ), pages 1517-1521, 2021.
[15] Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings. OpenReview.net, 2017.
[16) Xiao Jing, Zhenwei Zhu, Hongliang Li, Xin Pei, Yoshua Bengio, Tong Che, and Hongyong Song. Divide and explore: Multi-agent separate exploration with shared intrinsic motivations, 2022.
11

---

[17] Arthur Juliani, Ahmed Khalifa, Vincent- Pierre Berges, Jonathan Harper, Hunter Henry, Adam Crespi, Julian Togelius, and Danny Lange. Obstacle tower: generalization challenge in vision, control, and planning. CoRR, abs/1902.01378, 2019.
[18] Jaekyeom Kim, Minjung Kim, Dongyeon Woo, and Gunhee Kim. Drop-bottlenekk: Learning discrete compressed representation for noise-robust exploration. In International Conference on Learning Representations, 2021.
[19] Youngjin Kim, Wontae Nam, Hyunwoo Kim, Ji-Hoon Kim, and Gunhee Kim. Curiosity- bottleneck: Exploration by distilling task-specific novelty. In International Conference Machine Learning, 2019.
[20] J. Zico Kolter and Andrew Y. Ng. Near- beyr-bay exploration in polynomial time. In International Conference on Machine Learning. 2009.
[21] Taehwan Kwon. Variational intrinsic control revisited. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[22] Heinrich Küttler, Nantas Nardelli, Alexander H. Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The nethack learning environment. International Conference Neural Information Processing Systems, 2020.
[23] Marlos C. Machado, Marc G. Bellemare, and Michael Bowling. Count-based exploration with the successor representation. In Thirty-fourth AAAI conference on artificial intelligence, 2020.
[24] Jarryd Martin, Suraj Narayanan Sasikumar, Tom Everitt, and Marcus Hutter. Count-based exploration feature space for reinforcement learning. In Twenty-sixth International Joins Conference Artificial Intelligence, 2017.
[25] Georg Ostrovski, Mare G. Bellemare, Aaron van den Oord, and Remi Munos. Count-based exploration with neural density models. In International Conference on Machine Learning. 2017.
[26] Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven ex- ploration by self-supervised prediction. International Conference on Machine Learning, 2017.
[27] Aleksei Petrenko, Zhehui Huang, Tushar Kumar, Gaurav Sukhatme, and Vladlen Koltun. Sample factory: Egocentric 3d control from pixels at 100000 fps with asynchronous reinforcement learning. In International Conference Machine Learning. 2020.
[28] Roberta Raileanu and Tim Rocktäschel. Ride: Rewarding impact-driven exploration for environments. In International Conference on Learning Representations,
2020.
[29] Nikolay Savinov, Anton Raichuk, Raphaël Marinier, Damien Vincent, Mare Pollefeys, Timothy Lillicrap, and Sylvain Gelly. Episodic curiosity through reachability. In International Conference on Learning Representations, 2019.
[30] Mathieu Seurin, Florian Strub, Philippe Preux, and Olivier Pictquin. Don't 't do what doesn't 't matter: Intrinsic motivation with action usefulness. Proc. of IJCAI 2021, 2021.
[31] Alexander L. Strehl and Michael L. Littman. An analysis of model-based interval estimation for markov decision processes. Journal of Computer and System Sciences, 7408):1309-1331, 2008.
[32] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018.
[33] Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. #exploration: A study of count-based exploration for deep reinforcement learning. In Advances in Neural Information Processing Systems, pages 2753-2762. 2017.
12

---

[34] Aaron van den Oord, Oriol Vinyals. and Koray Kavukcuoglu. Neural discrete representation learning. In International Conference on Neural Information Processing Systems, 2017.
[35] Rongkai Xia, Yan Pan, Hanjiang Lai, Cong Liu, and Shuicheng Yan. Supervised hashing for image retrieval via image representation learning. In Twenty-eighth AAAI conference artificial intelligence, 2014.
[36] Lunjun Zhang, Bradly C. Stadie, and Jimmy Ba. Learning intrinsic rewards as a bi-level optimization problem. In Thirt Conference on Uncertainty in Artificial Intelligence (UAI), 2020.
[37] Tianjun Zhang, Paria Rashidinejad, Jiantao Jiao, Yuandong Tian, Joseph E. Gonzalez, and Stuart Russell. Made: Exploration via maximizing deviation from explored regions. In International Conference on Neural Information Processing Systems, 2021.
(38) Tianjun Zhang, Huazhe Xu, Xiaolong Wang, Yi Wu, Kurt Keutzer, Joseph E Gonzalez, and Yuandong Tian. Noveld: A simple yet effective exploration criterion. Advances in Neural Information Processing Systems, 2021.
[39] Zeyu Zheng, Junhyuk Oh, Matteo Hessel, Zhongwen Xu, Manuel Kroiss, Hado van Hasselt, David Silver, and Satinder Singh. What can learned intrinsic rewards capture? International Conference Machine Learning, 2020.
[40] Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. In International Conference on Neural Information Processing Systems, 2018.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We have discussed the limitations
and corresponding future works the conclusion section.
(c) Did you discuss any potential negative societal impacts of your work? [No] This paper
will not create any negative social impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] We will release the code and data soon.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] More details including hyperparameters are included in appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] We include the standard deviation of the results with respect to multiple runs.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either the supplemental material or as URL? [No] using/curating? [No] We used only the benchmark simulators.
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No] There is no personal data in our experiments.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "ZbJd4iUUaHb", "reviewer": "Reviewer_wRNy", "summary": "This paper studies a learnable hash function for count-based exploration. The hash function is learned through a vector quantized variational auto-encoder. The associated intrinsic reward is further modulated by a second reward that learns to focus on task relevant extrinsic reward, which is learned with a meta-learning approach. Both signals are combined through addition, hoping the task modulation will gradually learn to dominate the novelty, and the agent will gradually switch from exploration to exploitation. Experiments in MiniGrid environments and two DMLab tasks shows that their method generally outperforms or is on par with several baselines. ", "questions": "-\n\nConclusion: This is a decent submission: it is a relevant topic, the paper is well written, and their approach outperforms the baseline. My biggest worry is the slightly incremental character of the approach: they essentially glue together two known ideas, which is still a relevant contribution, but not extemely novel. The better performance compared to the baselines is promising, although I slightly miss a description of 1) how baselines were chosen and 2) how test tasks were chosen. Otherwise, this is a good paper. ", "limitations": "-", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strong:\n* Relevant topic (intrinsic motivation). \n* Decent experiments with better results than the baselines. \n* Well-written paper. \n\nWeak:\n* The idea is a bit incremental, since it combines two known ideas: 1) count-based novelty (which was also already tried with VQ-VAE), and 2) meta-optimization of a task-specific intrinsic reward.\n* I find it a bit hard to interpret the DMLab results, since there is no figure of the actual task, nor a description how the authors ended up with these two specific tasks. The learning curve of their method does look better. \n* I also slightly miss the motivation why these baselines are chosen (DSC, AS-LSH, NovelID, AGAC). \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "V5LxBZmD3s", "reviewer": "Reviewer_2TQC", "summary": "This paper proposes a new design of intrinsic reward, as a variant of the count-based reward, to tackle the sparse-reward tasks in reinforcement learning. \n\nPrior works related to count-based reward, especially the ones focusing on procedurally generated environments, widely use the episodic count, which means resetting the state visitation count for each episode, instead of recording the global visitation count throughout the entire training with multiple episodes.\n\nThis work proposes LECO, i.e. to get the state visitation count via learning the VQ-VAE model and regulate the count-based intrinsic reward via the LIRPG framework. Specifically, VQ-VAE encodes the state into a latent feature and maps the latent feature to the nearest embedding in the code book. The hash code is a sequence of indices of the matched embeddings and the intrinsic reward is based on the visitation count of each hash code in an episode. Also, the count-based reward is combined with the task-specific modulation, which is learned through the LIRPG framework.\n\nThe experiment is conducted on procedurally generated environments in MiniGrid benchmark and Deepmind Lab. LEGO is compared with recent works NovelD and AGAC and some variants as ablation study. It turns out that LEGO mostly outperforms these baselines.\n", "questions": "In Section 3, it seems that some details are missed. For example, in Section 3.3, what’s the dimension of $z^e$? What’s the relation between k and $k_i$? Although we can infer these details from the context, it will be better if they are crystal clear. In Equation (2), $log$ is a function, not a probability distribution. What’s the meaning of $log(s_t|z^q)$? Conditional probability distribution? In Section 3.4, $r^i_t$ does not include $\\lambda$, which is different from Equation (1). Is it a typo?\n\nIs LECO sensitive to hyper-parameters? such as the size of the codebook in VQ-VAE, the weight of task-specific modulation, the weight of intrinsic reward, etc. Could you please analyze the effect of each critical hyper-parameter? How and why does the higher or lower value choice of each hyper-parameter influence the final performance?\n\nIn Figure 5, why does LECO perform worse than NovelD in MultiRoom and KeyCorridor environments, but outperform it in ObstructedMaze? Any intuitive explanation?\n\nIn Figure 5, NovelD is worse than the results reported in NovelD paper. Is it because you use VQ-VAE to count instead of using their naive approach to count? Why not use their original setting?\n\nVQ-only in Figure 5 and AE-LSH in Figure 6 shows a weird performance pattern. They can learn reasonably well on easy task MultiRoom and hard task ObstructedMaze, but fail on the medium task KeyCorridor. Why? Have the hyper-parameters (e.g. weight of the intrinsic reward when added upon extrinsic reward) in these two baselines been searched extensively?\n\nIn Figure 6, could you explain why LECO is less sample efficient than NovelD for the noisy TV problem? The difference is kind of significant, that is, LECO costs around 4e7 more frames than NovelD to converge.\n\nIn Figure 7, it will be more impressive if LECO is compared with more classical and strong methods in Deepmind Lab, such as RND ('Exploration by Random Network Distillation') and ICM (“Curiosity-driven Exploration by Self-supervised Prediction”). NovelD mainly showed advantages in procedurally generated environments, instead of general sparse-reward tasks. Maybe NovelD baseline is not strong enough compared with other possible baselines on Deepmind Lab.\n\nAbout the ablative study, could you please add the comparison to AE-LASH + LIRPG? I interpreted LECO as VQ-VAE + LIRPG (please correct me if I misunderstand it), so I’m curious about the effect of VQ-VAE in comparison with the widely used counting method AE-LASH.\n\nWhat has been learned as state features $z^e$? Could you qualitatively or quantitatively analyze it? How about showing some examples of states assigned to the same hash code, and states assigned to different hash codes?\n\nMinor points:\nI doubt it is proper to name the memory buffer for hash code counts as ‘episodic memory’ (see Figure 1). In the literature of psychology, this term means recalling particular and subjective life experiences. In the RL area, this term is extended to refer to the state-action pairs with corresponding Q values in a buffer (e.g. ‘Model-Free Episodic Control’, ‘Episodic Memory Deep Q-networks’). The meaning of ‘episodic memory’ in this work seems to deviate from the literature.\n\nThe figures look preliminary without axis labels or meaningful sub-captions.\n", "limitations": "The authors mentioned the limitations of LECO on tasks that require inter-episode novelty, or difficult bi-level optimization. The issues are not fully addressed, but it is a natural idea to combine the episodic novelty in LECO with inter-episode novelty.\n\nIt will be interesting if the authors study VQ-VAE to count inter-episode state visitation. Especially on hard-exploration tasks on Atari games, the inter-episode novelty is necessary. Could the count learned with VQ-VAE outperform the previous approaches with pseudo-count, e.g. AE-SimHash ('#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning'), CoEX ('Contingency-Aware Exploration in Reinforcement Learning'), NGU ('Never Give Up: Learning Directed Exploration Strategies')? \n", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\nIt is relatively novel to learn VQ-VAE and then use the discrete hash code from the codebook for state visitation count.\nThe method is naturally motivated and technically reasonable. Based on prior methods of locality-sensitive hashing with auto-encoder (VE), it is a natural idea to modify AE to VAE. Because we need discrete state representation for counting, it is natural to try VQ-VAE.\nThe experiment result is positive, especially on hard exploration tasks ObstructedMaze in MiniGrid.\n\nCons:\nThe novelty is a bit limited. LECO seems simply a combination of two prior works: VQ-VAE and LIRPG. It will be great if you can clarify more technical contributions beyond the combination.\nThe method is not clearly explained in Section 3.3 and 3.4. Ideally, the method section should be self-contained enough for readers, even though they are not familiar with VQ-VAE or LIRPG.\nThe experiment part to compare with baselines can be improved.  The comparison with the most recent SOTA on MiniGrid (MADE) is missed. The performance of baseline NovelD in Figure 5 is worse than the results reported in NovelD paper. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "62M9WPbBKFK", "reviewer": "Reviewer_TYQ7", "summary": "This paper introduces a method for count-based intrinsic motivation in Reinforcement Learning. Considering the issue that the counting mechanism may drive the exploration towards states that are known to be useless for maximizing the extrinsic reward, the introduced method LECO modulates the intrinsic reward such that exploration of useless agents is not encouraged. The proposed method uses a neural network for computing the counting (using a variant of a variational autoencoder) and meta-gradient for the training of the modulator. Experiments on meaningful problems are provided.", "questions": "- I'm sure I'm missing something, but why the modulation reward is a tanh? In this way, it is unclear to me what the effect on the total reward will be when the modulator reward is > 0. In this way, it seems like the modulator can add an exploratory bonus to some states if they are found to be interesting, is this wanted? It seems more reasonable to me to use a sigmoid and subtract it from the episodic count reward. I'm probably missing something here, so I'd be glad if the authors can clarify this doubt;\n- How is the $\\lambda$ parameter selected? It seems to me that it should be directly related to the magnitude of the episodic count reward;\n- In the appendix, it is stated that the proposed method is similar to LIRPG when the normalized modulation coefficient $\\tilde{\\lambda} = 1$. Can you clarify what these similarities are?\n- Is the need for a vector-quantize VAE crucial? Why not use a regular VAE? Why not a regular AE?", "limitations": "The limitations of the proposed work are very briefly explained, from an intuitive point of view, in the conclusion. I think the authors could add more information, for example, describing under which conditions tasks may require intensive leveraging of inter-episode novelty. I also think that the similarity with LIRPG is a limitation that should be addressed properly.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n-------------\n- The paper deals with the interesting field of intrinsic motivation, and studies an important issue affecting methods in literature;\n- The proposed method is sound and it seems easy to implement;\n- The empirical evaluation provides good evidence of the advantages of the proposed method over several baselines.\n\nWeaknesses\n-----------------\n- The paper does not provide theoretical results. This is typical for intrinsic motivation method, but it would be nice to see some theoretical guarantees about the convergence to 0 (or at least a very small value) of the intrinsic reward for useless states. Probably this would need a different analysis and implementation allowing closed-form solution, perhaps replacing the MLP of the modulator with a linear approximation;\n- It is not clear how the $\\lambda$ parameter is computed, although this is not an important weakness and I'm sure it is quite straightforward. Still, I'd like clarifications;\n- Most of the equations are inline. This makes it difficult to read sometimes. For example, from line 197 to line 203;\n- If the method is similar to LIRPG, why there are no comparisons with it? Am I missing something?\n\nMinor comments\n----------------------\n- In the list of contributions, the last two items are not contributions; I'd remove them, or rephrase;\n- line 121: adapted -> adopted\n- Figure 3: visitaion -> visitation", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept