Novelty Search in representational space for sample efficient exploration
Authors: Ruo Yu Tao
Vincent François-Lavet
Joelle Pineau
Venue:  (2020)
Keywords: Reinforcement Learning
Exploration

Abstract: We present a new approach for efficient exploration which leverages a low-dimensional encoding of the environment learned with a combination of model-based and model-free objectives. Our approach uses intrinsic rewards that are based on a weighted distance of nearest neighbors in the low dimensional representational space to gauge novelty.
We then leverage these intrinsic rewards for sample-efficient exploration with planning routines in representational space.
One key element of our approach is that we perform more gradient steps in-between every environment step in order to ensure the model accuracy. We test our approach on a number of maze tasks, as well as a control problem and show that our exploration approach is more sample-efficient compared to strong baselines. 

Under review as a conference paper at ICLR 2020
NOVELTY SEARCH IN REPRESENTATIONAL SPACE FOR SAMPLE EFFICIENT EXPLORATION
Anonymous authors Paper under double-blind review
ABSTRACT
We present new approach for efficient exploration which leverages low- dimensional encoding of the environment learned with a combination of model- based and model-free objectives. Our approach uses intrinsic rewards that are based on a weighted distance of nearest neighbors in the low dimensional repre- sentational space to gauge novelty. We then leverage these intrinsic rewards for sample-efficient exploration with planning routines in representational space. One key element of our approach is that we perform more gradient steps in-between every environment step in order to ensure the model accuracy. We test ap- proach on a number of maze tasks, as well as a control problem and show that our exploration approach is more sample-efficient compared to strong baselines.
INTRODUCTION
In order to solve a task efficiently in Reinforcement Learning (RL), one of the main challenges is to gather informative experiences thanks to an efficient exploration of the state space. A common approach to exploration is intrinsic rewards correlated with some novelty heuristics (Schmidhuber) 2010 Stadie et al. 2015 Houthooft et al. 2016). With intrinsic rewards, agent can be incentivized to efficiently explore its state space. A direct approach to calculating these novelty heuristics is to derive a reward based on the observations, such as a count-based reward (Bellemare et al. [016)] Ostrovski et al. 2017) or prediction-erro based reward (Burda et al. 2018b). However, an issue occurs when measuring novelty directly from the raw observations, some information in the pixel space (such as randomness) might be irrelevant. In this case, if an agent wants to efficiently explore its state space it should only focus meaningful and novel information.
In this work, we propose a method of sample-efficient exploration by leveraging novelty heuris- ties in meaningful abstract state space. We leverage low-dimensional abstract representation of states, which is learned by fitting both model-based and model- fredel-free components through joint representation. This provides meaningful abstract representation where states that are close tempo- rally in dynamicy are brought close together in low-dimensional representation space. We also add additional constraints to ensure that measure of distance between states meaningful. With this distance in representational space, we form novelty heuristic inspired by the Novelty Search algo- rithm (Lehman and Stanley [011] to generate intrinsic rewards that we use for efficient exploration. We show that with good low-dimensional representation of states, policy based on planning with our novelty heuristic is able to explore with high sample-efficiens
In experiments, measure the effectiveness of exploration methods by the number of samples required to explore the state space. One key element of our approach is that we perform more gradient steps in-between every environment step in order ensure the model accuracy is high (and hence ensure an accurate novelty heuristic). Through this training scheme, our agent is also able learn meaningful representation of its state space in an extremely sample-efficient manner.
2 PROBLEM FORMULATION
Our agent interacts with its environment discrete time steps. modeled as Markov Decision Process (MDP), defined by the 6-tuple (S, A, t, R. (Puterman) 1994]. In this setting, S is the state space, So is the initial state distribution, A is the discrete action space, S x A S is

---

Under review as a conference paper at ICLR 2020
the transition function that is assumed deterministic (with the possibility of extension stochastic environments with generative methods), R S A + R is the reward function (R = -[-1, 1, 1),, G 8 A + [0, 1) is the per time-step discount factor. At time step t in state 8 E S, the agent chooses an action 04 E A based on policy π : S x A + [0, 1], such that dt ~ ((t,, After taking at, the agent is in state 8t+1 = T(st, (+) and receives reward r R(s:, at) and discount factor ?/ (5t, dz). Over n environment steps, we define the history of visited states as HI, (51, (51,,..,88) Sp), where S; E S Vi E N. In RL, the usual objective is maximize the sum of expected future rewards V (s) E rr E== (IT)=b (1t3) retils St,
To learn a policy æ that maximizes the expected return, an RL agent has to efficiently explore its environment. In this paper, we consider exploration tasks with sparse rewards, and are interested in exploration strategies that require as few steps as possible to explore the state space.
3 ABSTRACT STATE REPRESENTATIONS
In this work, we focus on learning a lower-dimensional representation of state when our state (or observations in the partially observable (Kaelbling al. 1998) case) is high-dimensions
(Dayan 1993 Tamar et al. 2016, Silver et al.] |016]] et al. [017]] Bruin et al., 2018 [a and Schmid huber 2018 François-Laver et al., 2018 Hafner al., 2018 Gelada et al. 2019). We learn our lower-dimensional abstract representation by encoding the high-dimensional state with an encoder
: S X' parameterized by 0g. where X' gax, The dynamics are learned via the fol- lowing functions: a transition function f X x A + A' parameterized by 0p, reward func- tion X A + [0, Rmas]] parameterized by 0z, and per timestep discount factor function y: X' x A + [0, 1) parameterized by 04.
In order to leverage all past experiences, we use an off- oif-ppolcy learning algorithm that sample transi- tion tuples (s, a, 7,,
7, s') from replay buffer. We first encode our current and next states:
I é(s; Oa), a' e(s'; Oz);
where x, x' E and 0g are our encoder parameters. The Q-function is learned using the DDQN van Hasselt et al. 2013) algorithm, which uses the target
Y = r ((é((': fg-). argmax Q(2', a.; (l:
(1)
where 4 and 0-- are parameters of an earlier buffered Q-function (or our target Q-function) and encoder respectively. The agent then minimizes the following loss:
(o())) (Q(z, a; 02) - Y
(2)
We learn the dynamics of our environment through the following losses:
Tn(0a, 0+) 1r- f(z, 0+)1²
(3a)
Lg 04) 1 (2x, 07
(3b)
L, (0g, 0+) ||x + f(z, a; -
(3c)
By jointly learning the weights of the encoder and the different components, the abstract repre- sentation is shaped in meaningful way according the dynamics of the environment. For in- stance, states that are temporally close through the transition function are brought close in the lower- dimensional space-a phenomenon reported in François-Lavet et al. (2018). To ensure that learning the transition function between abstract states doesn't t tend to collapse the abstract representations constant, we add the following soft constraints on the entropy:
La (0s) ezzp(CCalééss 02) - ē(s': 0s)).))
where s' are two randomly sampled states from the agent's 's history and C. is hyperparameter. We also use soft constraint between consecutive states that tends to enforce that two consecutive abstract representations are a distance of at least w:
Leonsee (fz) = mar([lé(s; (1: 0e) ((s: A 0.)112 - w, 0),
(5)
where w is hyperparameter (a discussion of this loss is provided in A)). The hyperparam- eter can be used to estimate the accuracy of our transition loss, hence of our novelty estimates.
2

---

Under review as a conference paper at ICLR 2020
In order to gauge when representation is accurate enough to our novelty heuristic, use function of this hyperparameter and transition loss to set a cutoff point for accuracy to know when to take the next environment step. If is the minimum distance between successive states, then when
L, < (*)² the transitions are on average within a ball of radius ă of the target state, Here hyperparameter that we call the slack ratio. Before taking a step in the environment, keep training all the parameters with all these losses until this threshold is reached and our novelty heuristic becomes useful. Details the slack ratios used in the experiments are given in Appendix
We minimize the sum of all the aforementioned losses through gradient descent with learning rate OC
C LR(0,, 0+) Lg 04) L, (0,, + LQ(00) Lai (0a) + Leonsee (0z).
Through these losses, the agent learns a low-dimensional representation of the environment. These losses are well-suited for our novelty metric because of the meaningfulness the losses ensure in terms of the L2 norm in our representation space (see Appendix C.. Planning techniques that combine the knowledge of the model and the value function can then be used select actions that will maximize intrinsic rewards.
4 COMBINING MODEL-FREE AND MODEL-BASED COMPONENTS FOR EXPLORATION POLICIES
Similarly to previous works (e.g. Oh ct al. 2017), with model-free free Q-learnig to obtain a good policy. For our approach, we calculate rollout estimates of next states based on our transition model and sum up the corresponding rewards (which denote as r X' x A + [0, Rmaz] and can be a combination of both intrinsic and extrinsic rewards). We calculate expected returns based the discounted rewards of our d-depth rollouts:
combination of model-basee planning
(x(x,))) = r(x, a) + (I, 07) max Q (7 d': d'; (((x,a;6g) aQ))),
if d > 0
a'EA
(7)
if d 0
Note that we simulate only b-best options at each expansion step based on Q(x, a; 0Q), where b S [A].
We then use simple sum of the Q-values obtained with planning up to depth D:
D
@ (z, a)
(x,
d=0
The estimated optimal action is given by argmax Optn (2,, a). The actual action chosen each step aE.A
follows e-greedy strategy (e [0, 1)), where the agent follows the estimated optimal action with probability 1 and a random action with probability
NOVELTY SEARCH IN ABSTRACT REPRESENTATIONAL SPACE
Our approach uses intrinsic motivation (Schmidhuber, 1990 Chentanez et al. 2005 Achiam and Sastry 2017) where an agent rewards itself based purely on the fact that it gathers interesting experi- ences. The specific formulation of the rewards inspired by the Novelty Search algorithm (Lehman) and Stanley 2011). Since the concept of proximity loses meaning in high-dimensional spaces (Ag- garwal et al. 2002). ) the nearest neighbor problem is ill-defined. So instead calculating a measure of novelty in pixel space, leverage the -ww-dimensional abstract representations.
To get a novelty metric for our abstract representations, we can apply a weighted version of the novelty measure in abstract representational space to our current state with encoder applied
- (( f2)), and a history of encoded states. We define the following novelty measure over abstract representational space: px (c(s; = Σ dist (((ss;)))(e))) 6(s; 0z))
(8)
3

---

Under review as a conference paper at ICLR 2020
where is the ith closest state in the abstract representations and dist is distance measure in representational space. this work, we use &g norm as our measure of distance. We motivate this choice in Appendix In the heuristic above, weight the distances between the lt nearest neighbors and by their ranking based on distances from the corresponding abstract states. We do this in order to compensate for biases towards state-action pairs that have fewer direct neighbors as opposed to purely novel states. We clarify this point with toy example in Appendix [B and we compare this formulation with an unweighted average in Appendix F
We also analyze the limiting
behaviour of our novelty reward in Appendix D
Incorporating of the above, our algorithm described in Algorithm 1
Algorithm 1: The Novelty Search algorithm in abstract representational space.
Initialization: transition buffer H, agent policy m; Sample ninst initial random transitions, let Miniti while I Remar do
// We update our dynamics model and Q-function every firre steps then
it' mod fifreg
while j S Ritters or I., S (x)² do
Sample batch of transitions (s, a, Teatry Tintr: 7, H;
Train dynamics model with (s, aererti 7, 8');;
Train Q-function with (s, a, Textr + Tintr, 7, 8);
end
V(ss, O, Textry Tintry 8. E H, set Tintr + pxx(((s';0e))
10
11
end
a1 <st)):
13
Take action in environment: 81411 + (8t, 0z), Te,eatr + R(s,, nt + G(0,, az);
14
Calculate intrinsic reward: Tr,nntr < px (ex(é((s))))) (e(55(11;;;))
15
H + H U {(t,, Fteettr: Ttinntr; Yt, 1t+1};;
16
end
EXPERIMENTS
We conduct experiments on environments of varying difficulty. All experiments use a training scheme where parameters converge accurate representation before taking environment step. We optimize the losses (over multiple training iterations) given in before each environment step to train the agent's state representation. We discuss all environment--pecifie hyperparameters in Appendix
6.1 LABYRINTH EXPLORATION
We consider two 21 21 versions of the grid-world environment, per Figure I The first is an open labyrinth grid-world, with no walls except for bordering walls. The second a similar sized grid-world split into four connected rooms. In these environments the actions A are the four cardinal directions. These environments have no rewards or terminal states and the goal is to explore, agnostic the task. We use two metries gauge exploration for this environment: the first the ratio of states visited only once, the second is the proportion of total states visited.
6.1.1 OPEN LABYRINTH
In the open labyrinth experiments (Figure we compare number of variations of the above policy with random baseline and count- based baseline (Bellemare et al. 2016) (as we can count states in this tabular setting). Variations of the policy include an argmax over state values (d 0) and planning depths of 51,
Our method is able to leverage good learnt representation in order to explore in a eample-efficient way. All variations of method outperform the baselines in this task, with slight increase
4

---

Under review as a conference paper at ICLR 2020
(b)
Figure I: Left: Open labyrinth A 21 x 21 empty labyrinth environment. Right: 4-room labyrinth 21 21 4-room labyrinth environment.
in performance planning depth d increases. In the open labyrinth, agent is able to reach 100% of possible states (a total of 19 19 = 361 unique states) in approximately 700 steps, and 80% of possible states (Rs 290 states) in less than 400 steps. These counts also include the ninit number of random steps taken preceding training. Our agent is also able to learn highly interpretable abstract representations in very few environment steps shown in Figure 2a) space.
explores its state
(a)
(b)
Figure 2: (a): A learned abstract representation of the open labyrinth environment from Figure la after 500 environment steps. Colors denote which side of the maze the agent was in, grid coordinates and transitions are shown. (b): Two views of the same learned abstract 3-dimensional representation of our multi-step maze after 300 steps. Orange and blue points denote states with and without keys respectively. Our agent is able to disentangle states where the agent has a key it doesn't t (z 0) as seen in the distance between orange and blue states. Meaningful information about the agent position is also maintained the relative positions of states abstract state space.
1) and when
6.1.2 4 R-ROOM LABYRINTH
While the learned dynamics model and abstract representation are able to create an interpretable representation of the open labyrinth environment (Figure 2a). the 4-room labyrinth environment is a low-
more challenging. Indeed, our encoder takes high-dimensional input and compresses dimensional representation. In the case of the labyrinth environment, the representation incorporates knowledge related to the position the agent in 2-dimensions that we call primary features. also learns other information such agent surroundings (walls, open space) etc., but it does only via the transition function learned through experience. We call this extrancous but necessary information secondary features. As most of these secondary features encoded only in the dynamics model f, our agent has to experience a transition in order to accurately represent both primary and secondary features.
5

---

Under review as a conference paper at ICLR 2020
100 200
300
500
1000
10.0
75
10
50
20
Figure 3: An example of the state counts of our agent in the open labyrinth with 5 step plan- ning. Titles of each subplot denotes the number of steps taken. The brightness of the points proportional to the state visitation count.
Exploration
states
1.0
O'porrtion a11 02a22 Visited
1.0
unique enars vistted
sotal states ViSRtd
0.4
3.4
0.
250
250
enviromment stegs
enviromment steps
(a) Results open labyrinth and different variations on policies compared to baselines.
Exploration factor
Ratio visited
1.0
all stata: vientd
1.0
Beaiine
unique catre vistee|
(toaal states vic.3d
0.8
Count al Dergras
Reeityy Al Gorgmax
0.8
Rovelty Paaning
0.6
0.2
0.2
0.0
250
1000
0.0
250
1000
steps
steps
(b) Results for the 4-room labyrinth and different variations policics compared to baselines.
Figure 4: Labyrinth results for both open labyrinth and 4-room labyrinth over 3 trials.
In this environment specifically, our dynamics model tends to over-generalize for walls between rooms and can thus also fail to try out transitions the passageways between rooms. Our agent then tends to visit uniformly all the states that reachable within the known rooms. With e-greedy policy, our approach still ensures that the agent explores passageways even it has over- ever-generalized to the surrounding walls.
We run the same experiments approach still outperforms the random and count-based baselines. After visiting most unseen states in its environment, novelty heuristic. We can see an example of this for the open labyrinth in Figure The bright spot that begins after 200 counts is the agent learning the dynamics of labyrinth walls.
do the open labyrinth and report results in Figure 4b Our
agent tends to uniformly explore its state space due the nature of our
6.2 CONTROL SUB-GOAL EXPLORATION
In order to test the efficacy of our method on environments with harder dynamics, we conduct experiments the control-based environment Acrobot (Brockman et al., 2016) and multi-step maze environment. Our method (with planning depth 5) is compared to strong exploration baselines with different archetypes:

---

Under review as a conference paper at ICLR 2020
1. Prediction error incentivized exploration (Stadie et al. 2015
2. Hash count-based exploration (Tang et al. 2016)
3. Bootstrap DQN (Osband et al. 2016)
In order to maintain consistency in our results, we use the same model architectures and model-free methods throughout. Since experiment in the deterministic domain, exclude baselines that quire some form of stochasticity or density estimation as baselines (for example, Shyymm et al. (2018) and Osband et )..77). Normally, for most approaches that include model-free component, step in the environment is taken with every update step. In our experiments, we use orders of mag- nitude less samples as compared
model-free RL algorithms (all within the same episode). To ensure fair comparison between our approach and baselines, we run multiple training iterations in between each environment step for all experiments.
6.2.1 ACROBOT
We now test our approach on Acrobot (Brockman et al., 2016), which has a continuous state space unlike the labyrinth environment. We specifically choose this control task because the nature of this environment makes exploration inherently difficult. The agent only has control of the actuator for the inner joint and has to transfer enough energy into the second joint in order swing it to its goal state. We modify this environment so that each episode at most 3000 environment steps. We the default reward setup in Acrobot, where the agent is rewarded with -1 every environment step until it reaches its goal state where it receives reward of O. To measure the performance of our exploration approach,
measure the average number of steps that the agent takes move its second joint above a given line as per Figure Sauueeaalnn in single episode.
To demonstrate the ability of our method to learn low dimensional abstract representation from pixel inputs, we use 4 consecutive pixel frames as input instead of the 6-dimensional full state vector. We use 4-dimensional abstract representation of our state and results from experiments are shown in Table 6.2. Our method reaches the goal state more efficiently than our baselines.
(b) Left: Start of our multi-step maze. right: After the agent has collected the key.
Left: Acrobot start state. right: Acrobot state
Figure 5: Illustrations of the Acrobot and multi-step goal maze environments. b) Left: The passage- way to the west portion of the environment are blocked before the key (black) is collected. b) Right: The passageway is traversable after collecting the key, and the reward (red) is then available. The environment terminates after collecting the reward.
Acrobot
Multi-step Maze
Avg
Stdev
Avg
Stdev
Random Policy
2043.00
911.18
2416.80
1199. 16
Prediction error incentivized exploration
575.25
196.89
529.50
120.59
Hash count-based exploration
674.25
363.59
685.50
603.24
Bootstrap DQN
478.00
224.48
1395.25
888.22
Novelty Search with abstract representations 356.25
87.47
320.00
69.24
Table 1: Number of steps necessary to reach the goal state in the Acrobot and the multi-step maze environments (lower is better). Results are averaged over trials. Best results are in bold.

---

Under review as a conference paper at ICLR 2020
6.2.2 MULTI-STEP GOAL MAZE
We also test our method on an environment with temporal aspect maze with the sub-goal of picking up a key to open doors before the main goal of reaching the reward is available. We build our environment with the Pycolab game engine (Stepleton 2017). The environment can be seen in Figure [5b] Each episode is at most 4000 environment steps. While this environment does admit an extrinsic reward (1 for picking up the key, 10 for reaching the final state), we ignore these rewards and only focus on intrinsic rewards.
Similarly to the labyrinth environment, the input to our agent is a top-down view of the environment. We also employ e-greedy policy. From our experiments, we show that agent is able to learn an interpretable representation of the environment in a sample-efficient manner. It is able to encode this temporal aspect of the environment (whether the key was collected) as shown in Figure 2b With our intrinsic reward generated with this representation, our agent is able to more efficiently explore its state space and reach the end goal in fewer steps compared our baselines.
RELATED WORK
The proposed exploration strategy falls under the category of directed exploration (Trun 1992) that makes use of the past interactions with the environment to guide the discovery of new states. This work is inspired by the Novelty Search algorithm (Lehman and Stanley] 2011) that leverages a nearest-neighbor scoring function, but does so in behavior space. Exploration strategies have been investigated with both model-free and model-based approaches. In Bellemearee al. (2016) and Ostrovski et al. (2017). a model-free algorithm provides the notion of novelty through pseudo- count from an arbitrary density model that provides an estimate of how many times an action has been taken in similar states.
Several exploration strategies have also used model of the environment along with planning. Hes- ter and Stone (2012) employ two-part strategy to calculate intrinsic rewards, combining model uncertainty (from a random-forest based model) and a novelty reward based on L2 distance in fea- ture space. A strategy investigated in SSage et al. (2014); Mohamed and Rezende (2015); Gregor (et al. (0016); Chiappa et al. (2017) is to have the agent choose sequence actions by planning that leads to representation of state different (2017): Haber et al.] (2018), the agent optimizes both a model of its environment and a separate model that predicts the error/uncertainty of its model. [Brdda et al. (2018a) similarly uses an intrinsic reward based on the uncertainty of its dynamics model. The agent can thus seek actions that adversarially challenge its knowledge of the environment (Savinov al.) 2018). In (Shyam al.) (2088), multiple model forward models of the environment are also employed to plan to observe novel states by using a measure of novelty derived from disagreement between future states.
possible the current state. In Pathak et al.
DISCUSSION
In this paper, we show that with interpretable abstract representation of states, our novelty metric is able to serve an intrinsic reward that enables efficient exploration. By using this novelty metric with a combination of model-based and model-free approaches for planning, we demonstrate the efficiency of our method in multiple environments.
As with most methods, our approach also has limitations. While the problem of distance metrics in nigh-dimensional space is partially solved in our method with the dimensionality reduction of observations by our encoder, the f2-nrrm still requires a low dimension to be useful (Aggarwal et al., 2002). This implies that our novelty metric may lose its effectiveness as we increase the dimension of our abstract representation.
In addition, our exploration strategy benefits greatly from the meaningful abstractions and internal model. In some cases, the model can over-generalize with the consequence that the low-dimensional representation loses information that is crucial for the exploration of the entire state space. An interesting direction for future work would be find ways of incorporating the secondary features mentioned in Section6.11. 6.1.2
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Achiam, J. and Sastry, S. (2017). Surprise- Based Intrinsic Motivation for Deep Reinforcement Learning. arXiv e-prints, page arX.v:1703.01732.
Aggarwal, C., Hinneburg, A., and Keim, D. (2002). On the surprising behavior of distance metric in high-dimensional space. First publ. in: Database theory, ICDT 200, 8th International Conference, London, UK, January 4 6, 2001 / Jan Van den Bussche (eds.). Berlin: Springer, 2001, pp. 420- 434 (=Lecture notes in computer science; 1973).
Bellemare, M. G., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., and Munos, R. (2016). Unifying count-based exploration and intrinsic motivation. arXiv preprint arXiv: 1XXiv:606.00 (606008865
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2016). Openai gym.
Burda, Y., Edwards, H., Pathak, D., Storkey, A., Darrell, T., and Efros, A. A. (2018a). Large-Scale Study of Curiosity-Driven Learning. arXiv e-prints, page A.Xi:18808003355
Burda, Y., Edwards, H., Storkey, A., and Klimov, O. (2018b). Exploration by Random Network Distillation. arXiv e-prints, page trXiv:1810.12899
Chentanez, N., Barto, A. G., and Singh, S. P. (2005). Intrinsically motivated reinforcement learning-
In Saul, L. K., Weiss, Y., and Bottou, L., editors, Advances in Neural Information Processing Systems 17, pages 1281-1288. MIT Press.
Chiappa, S., Racaniere, S., Wierstra, D., and Mohamed, S. (2017). Recurrent environment simula- tors. arXiv preprint arXiv:1704.02254.
Dayan, P. (1993). Improving generalization for temporal difference learning: The successor repre- sentation. Neural Computation, 5(4):613-624.
de Bruin, T., Kober, J., Tuyls, K., and Babuška, R. (2018). Integrating state representation learning into deep reinforcement learning. IEEE Robotics and Automation Letters, 33)):399-1401.
François-Laveet, V., Bengio, Y., Precup, D., and Pineau, J. (2018). Combined reinforcement learning via abstract representations. CoRR, htss/1899044066
Gelada, C., Kumar, S., Buckman, J., Nachum, O., and Bellemare, M. G. (2019). Deepmdp: Learning continuous latent space models for representation learning. arXiv preprint /206:/996..0222
Gregor, K., Rezende, D. J.. and Wierstra, D. (2016). Variational intrinsic control. arXiv preprint arXiv: 1.07507.
Ha, D. and Schmidhuber, J. (2018). Recurrent World Models Facilitate Policy Evolution. arXiv e-prints, page
Haber, N., Mrowca, D., Fei-Fei, L., and Yamins, D. L. (2018). Learning to play with intrinsically- motivated self- agents. arXiv preprint 87244.180..0744..
Hafner, D., Lillicrap. T., Fischer, I., Villegas, R., Ha, D., Lee, H., and Davidson, J. (2018). Learning Latent Dynamics for Planning from Pixels. arXiv e-prints, page arXiv: 11I.v11811... 18..0555..
Hester, T. and Stone, P. (2012). Intrinsically motivated model learning for a developing curi- ous agent. 2012 IEEE International Conference Development and Learning and Epigenetic Robotics (ICDL), pages 1-6.
Houthooft, R., Chen, X., Duan, Y., Schulman, J., De Turck, F., and Abbeel, P. (2016). Vime: Variational information maximizing exploration. In Advances in Neural Information Processing Systems, pages 1109-1117.
Kaelbling, P., Littman, M. L., and Cassandra, A. R. (1998). Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101(1):99 134.

---

Under review as a conference paper at ICLR 2020
Lehman, J. and Stanley, K. O. (2011). Abandoning objectives: Evolution through the search for novelty alone. Evolutionary Computation, 19(2):189 223.. PMID: 20868264.
Mohamed, S. and Rezende, D. J. (2015). Variational information maximisation for intrinsically motivated reinforcement learning. In Advances in neural information processing systems, pages 2125-2133.
Oh, J. Singh, S., and Lee, H. (2017). Value prediction network. In Advances in Neural Information Processing Systems, pages 6118-6128
Osband, I., Blundell, C., Pritzel, A., and Van Roy, B. (2016). Deep Exploration via Bootstrapped DQN. arXiv e-prints, page urXiv:1602.0462)
Osband, I., Van Roy, B., Russo, D., and Wen, Z. (2017). Deep Exploration via Randomized Value Functions. arXiv e-prints, page arXiv:1703.076:
Ostrovski, G., Bellemare, M. G., Oord, A. v. d., and Munos, R. (2017). Count- based exploration with neural density models. arXiv preprint 173.:3100.
Pathak, D., Agrawal, P., Efros, A. A., and Darrell, T. (2017). Curiosity-driven exploration by self- supervised prediction. In International Conference on Machine Learning (ICML), volume 2017.
Puterman, M. L. (1994). Markov Decision Processes: Discrete Stochastic Dynamic Programming.
John Wiley & Sons, Inc., New York, NY, USA, 1st edition.
Salge, C., Glackin, C., and Polani, D. (2014). Changing the environment based on empowerment as intrinsic motivation. Entropy, (6(5):779--211)
Savinov, N., Raichuk, A., Marinier, R., Vincent, D., Pollefeys, M., Lillicrap, T., and Gelly, S. (2018).
Episodic curiosity through reachability. arXiv preprint arXiv:1810.02227.
Schmidhuber, J. (1990). A possibility for implementing curiosity and boredom in model-building neural controllers. In Proceedings of the First International Conference Simulation of Adaptive Behavior on From Animals to Animats, pages 222-227, Cambridge, MA, USA. MIT Press.
Schmidhuber, J. (2010). Formal theory of creativity, fun, and intrinsic motivation (1990-2010). IEEE Transactions on Autonomous Mental Development, 233):20--247.
Shyam, P., Jaskowski. W., and Gomez, F. (2018). Model-Baser Active Exploration. arXiv e-prints, page 1IXXiv:81001216:
Silver, D., van Hasselt, H., Hessel, M., Schaul, T., Guez, A., Harley, T., Dulac-Arold, G., Reichert, D., Rabinowitz, N., Barreto, A., and Degris, (2016). The Predictron: End-To-End Learning and Planning. arXiv e-prints, page 1rXiv:1612.08811
Stadie, B. C., Levine, S., and Abbeel, P. (2015). Incentivizing exploration in reinforcement learning with deep predictive models. arxiv preprint 4rX0v:1507.00814.
Stadie, B. C., Levine, S., and Abbeel, P. (2015). Incentivizing Exploration In Reinforcement Learn- ing With Deep Predictive Models. arXiv e-prints, page arXiv:1507.00814
Stepleton, T. (2017). The pycolab game engine. pycolabl
Cttps:///ittuu..com/eet
Tamar, A., Levine, S.. Abbeel, P., WU, Y., and Thomas, G. (2016). Value iteration networks. In Advances in Neural Information Processing Systems, pages 2146-2154.
Tang, H., Houthooft, R., Foote, D., Stooke, A., Chen, X., Duan, Y., Schulman, J., De Turck, F,, and Abbeel, P. (2016). #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning. arXiv e-prints, page 1rXiv:1611.0477
Thrun, S. B. (1992). Efficient exploration in reinforcement learning.
van Hasselt, H., Guez, A., and Silver, D. (2015). Deep Reinforcement Learning with Double Q- learning. arXiv e-prints, page arXiv: 1tt.110996.61111
10

---

calculate Under review as a conference paper at ICLR 2020
A DISCUSSION ON THE ENTROPY CONSTRAINT
As for our soft constraints on representation magnitude, we use a local constraint instead of a global constraint magnitude such that it is more suited for novelty metric. If some form intrinsic reward based on distance between neighboring states, then this distance needs to be non-zero and ideally consistent the number of states our history increases. In the global constraint case, if the intrinsic rewards decreases with an increase in number of states in the agent's 's history, then the agent will fail be motivated explore further. Even though the entropy maximization losses ensures the maximization of distances between random states, if we have H.) number of states the history of the agent, then global constraint on representation magnitude might lead to
lim |H,|-t00
(HH,)) -s s'112] = 0.
B MOTIVATION FOR RANKED WEIGHTING OF K-NN SCORES
Figure 6: Illustration of a simplistic 3 grid-like environment, with A {up, down, left, right) and distance 1 between neighboring states. All actions which lead out the grid are no-ops.
Here H2 is a list of all possible states in S in any order, with each possible state appearing only once. If we let & 4, we have that:
pxld' ((HI))) 0 0+1+1++ V2
2+
(10)
4
ex(x, 0 0+1+1+1
1++1+1 3
(11)
4 4
(12)
While may seem redundant to include the 1st nearest neighbor distance in this metric (which would be itself if we've visited the state), the 1st nearest neighbor is non-zero when calculate the novelty of a predicted state using our learned transition function f. From this example, we can see that there is bias towards states with fewer direct neighbors due to the nature of our novelty metric.
This poses an issue - our goal is for sample-efficient exploration of our state space, then there is no reason to favor states with less direct neighbors.
C MOTIVATION FOR USING L2 NORM AS A DISTANCE MEASURE ABSTRACT REPRESENTATION SPACE
We consider the entropy maximization loss Lui and consecutive distance loss Leonsse- On the one hand, the entropy maximization loss ensures maximal l distance between two random states due to the inverse relationship the negative exponential. On the other hand, minimization consecutive distance loss ensures that the distance between two consecutive states is < We consider the minima of the combination of these two losses for consecutive states and s':
our
L min Lai (0z) + Loonsec
0,
min eap(- Callal(ssss 02) ((s/:0a)ll22 mar([]ã(s;;) 0.) e(s/:02)l1 w,0) 0)

---

Under review as a conference paper at ICLR 2020
The minimum of the combination of the losses is obtained when the distance between and s' is These losses shape abstract state space so that & norm as a distance measure encodes a notion of closeness in state space that we leverage in novelty heuristic.
D LIMITING BEHAVIOR FOR NOVELTY HEURISTIC
In the limiting case, novelty heuristic is 0 for all states. Indded, if consider the case where each state has been visited & times, the k nearest neighbors to state are all identical and equal to in the deterministic case. Based equation each state:
have the following novelty heuristic score for
1 dist (1(s; =0
Px (e(s;0g)) Σ
As for stochastic environments, let s" be stochastic observation of the state s, and have that
Elpx (é(s; 0a))] = Σ
2 (ē(x; (a), 6(8"; ;a))]-
If E[dist (ē(s; e(s"; - 0 as the encoder reaches optimality, then our novelty heuristic will return expectation the limiting case.
E EXPERIMENTAL SETUP AND HYPERPARAMETERS
For all our experiments, we use batch size 64 and take 64 random steps transitions before beginming training. We also use the same discount factor for all experiments (y 0.8) and the same freeze interval for target parameters 1000. The reason behind our low discount factor is due to the high density of non-stationary intrinsic rewards in state. For all model-based abstract representation training, the following hyperparameters were all kept constant: minimum distance between consecutive states w 0.5, slack ratio 5 6 and transition model dropout of 0.1. For all experiments run with our novelty metric, we use for our k-NN calculations.
E.1 NEURAL NETWORK ARCHITECTURES
For reference, 'Dense' implies full-connected layer. 'Conv2D' refers to a 2D convolutional layer with stride 1. 'MaxPooling2D' refers max pooling operation. All networks were trained with the RMSProp optimizer. Throughout all experiments, we use the following neural network architec- tures:
E.I.1 ENCODER
For all our non-control task inputs, we flatten our input and use the following ferw-forward neural network architecture for
Dense(200, activation='tanh")
Dense(100, activation='tanhh)
Dense(50, octivation='tanh")
Dense(10, activation='tant 'tann')
Dense(abstract representation dimension).
For our control task, we use convolution based encoder:
C0 kem3cl(3,3), eetivation='taniy
Conv2D(channels=166 kernel=(3.3), activation='tanhi
(iz=444) MaxPool2D(pool size=(4,4))
2on22DDchhanels=322. kernel=(3,3), activation='tanh")
12

---

Under review as a conference paper at ICLR 2020
MaxPool2D(ppol size=(3,3)
Dense(abstract state representation dimension).
E. 1.2
TRANSITION MODEL
The input to our transition model is concatenation of an abstract representation and an action. We use the following architecture
Dense(10) 10, activation='tanhi dropout=0.1)
Dense(30, aetivation='tanh" dropout=0.1)
Dense(30, activation='tanh: dropout=0.1)
Dense(10 I0, activation='tanh: dropout=0.1 1)
Dense(abstract representation dimension)
and add the output of this to the input abstract representation.
E.1. REWARD AND DISCOUNT FACTOR MODELS
For both reward and discount factor estimators, the following architecture:
Dense(10, I0, activation='taml "tanh")
Dense(50, activation='tann 'tanh')
Dense(20, activation='tanh)
Dense(I).
E.1.4 FUNCTION APPROXIMATOR
We use two different architecture based on the type of input. If we use the concatenation of abstract representation and action, we use the following architecture:
Dense(20, activation='relu')
Dense(50, activation='relu")
Dense(20, activation='relu")
Dense(nactions)
For the pixel frame inputs for control environments,
C0 kernel=(3,3), activation='tanh")
Conv2D(chameel==16, kernel=(3,3), (aap:/caammeesssls...eeee1333)) "ativation='tanh
TaxpPool2D((pooolsszeeeese size=(4.4))
2oan22D(chamels=3332knnncc (3,33 activation='tanhh)
MaxPool22D(pool size=(3,3))
Dense(nactions).
Finally, for our (purely model free) gridworld environments we use:
Dense(500, sativation='tanhh))
Dense(200, activation='tanle 'tanhh"
Dense(50, activation='tanh"
Dense(10, activation='tanie
Dense(Mactions)
13

---

Under review as a conference paper at ICLR 2020
As for our Bootstrap DQN implementation, use the same architecture above, except replace the final Dense layer with 10 separate heads (each a Dense layer with nactions nodes).
E.2 LABYRINTH ENVIRONMENTS
Both environments used the same hyperparameters except for two: we add an e-greedy (e 0.2) policy for the 4-room maze, and increased n frey from to in the 4-room case due to unnecessary over-training. We have the following hyperparameters for our two labyrinth environments:
Riters = 30000
a 0.00025
E.3 CONTROL ENVIRONMENT
In our Acrobot environment, the input to agent is 4 stacked consecutive pixel frames, where reduce each frame down to a 32 x 32 pixel frame. Our abstract representation dimension is 4. We use learning rate of 0.00025 for all experiments. We train for Pltters 30000 for both model-free experiments and niters = 50000 for both experiments incorporating model- -assed components this discrepancy due the need for more iterations for the model-based portion converge.
E.4 MULTI-STEP MAZE ENVIRONMENT
In our multistep maze environment, the input to our agent is single 15 x 15 frame of an overview of the environment. Our abstract representation dimension is 3. We use an e-greedy (e 0.1) policy for this environment. We use a 0.00025, Ylteers 30000 for our model-free algorithms and = 0.000025, Ylttess = 50000 for experiments that include a model-base component. This again due to the need for model-based component to converge.
F ABLATION STUDY
3.2
1.0
1.0
6.-
03
38
250
500
750
1000
0.0
250
500
755
1000
STNTRRMMETT stago
orSfronmene stago
Novuliy Flanning Ang
Novelty Mann Waighted Avg
(a) Results for open labyrinth between unweighted and weighted k-NN averages.
(b) Results 4-room lahyrinth hetween un- weighted and weighted k-NN averages.
Figure 7: Ablation results for unweighted versus weighted nearest neighbor heuristics (over trials).
We perform an ablation study between unweighted (orange) and weighted (red) nearest neighbor heuristics on the open labyrinth (left) and 4-room labyrinth (right) environments. While the weighted nearest neighbor approach does improve the bias towards visiting more fewer-neighbor states, the improvements are relatively marginal.
14


Reviews:
[{"review_id": "rkeK9jTnFH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method for efficient exploration in tabular MDPs as well as a simple control environment. The proposed method uses deterministic encoders to learn a low dimensional representation of the environment dynamics, which preserves distances between states such that “close” states in the full MDP are close in the learned representation. An intrinsic reward is formulated based on a measure of novelty, given by distance between new states, and a stored replay buffer.  Along with the dynamics model, a model-free agent employs Q learning to find a good policy. Experiments are performed on 3 tabular environments and the acrobot control task. \n\nPros: \n1.\tOverall the paper is clear and the proposed method makes sense intuitively. The intrinsic reward is cheap to compute and the state abstraction offers a nice way to visualize state differences the agent thinks are important.\n\n2.\tThe method seems to be sample efficient with regard to strong baselines like [1]\n\nCons:\n1.\tIt seems difficult to argue the efficacy of a low-dimensional state representation that doesn’t scale with state dimensionality. As shown in [2] and [3], learning effective state abstractions in high dimensions can require considerably more effort. \n\n2.\tGiven that there exist novelty based intrinsic rewards which compute state abstractions in high dimensional environments [3], I find it hard to see the usefulness of the proposed method. \n\n3.\tThe choice of distance metric for the representational space is not well motivated. As correctly stated by the authors, the L_2 norm will cease to be a good metric as state dimensionality increases.\n\n4.\tThere are too many grid-world experiments. The point of the first two experiments can be made simply using the four-room environment. This could make room for a more interesting experiment such as MuJoCo Ant Maze. \n\nMy main issue with the work in its current form is that the method is too light in terms of technical contribution. Simple methods are ok (even valuable!) but there should be a certain about of rigorous analysis which shows that the simple method can be used as a foundation for further work. For a method which mostly examines tabular environments, I expect some analysis of the methods efficiency with regard to data efficiency -- the main point of the paper. [1] and [4] which are used as comparisons, provide such analysis. If a convincing theoretical analysis is out of reach, then it could be sufficient to provide extensive experimental evidence supporting the claims. In this case it could include an examination of different metrics, additional (ideally more difficult) environments, and comparison to other baselines like [5], [3].\n\nDue to what I see as a lack of technical contribution, I do not recommend acceptance to ICLR at this time. \n\nA more compelling submission would include the following:\n●\tA more detailed motivation for why the L2 norm makes sense as a distance metric. \n○\tIn an abstract space, it's more natural to use a statistical distance like the KL or JS divergence. These metrics have drawbacks, but they should be discussed\n●\tAn analysis of the limit behavior of the proposed method. Given enough time an intrinsic reward should explore every state in a deterministic environment. Does this happen in the limiting case -- if not, is the margin acceptable. \n●\tMore extensive experiments. This method can clearly admit convolutional architectures so experiments on more interesting environments are viable. Though I believe this would require more complex models such as a VAE, and may change the submission considerably.  \n\nMinor notes\n●\tSection 3: “when [the distance between transitions is less than the slack ratio] the transitions are mostly accurate within a ball of radius \\frac{w}/{\\delta}. This is too vague, what does mostly accurate mean? \n●\tEq (6), is \\alpha a hyperparameter as well as the learning rate? If \\alpha is just the learning rate than the equation is incorrect, because the learning rate is applied to the gradient of the loss, not the loss itself. \n●\tThe description of the planning algorithm and Q learning in section 4 is a little sloppy, a clearer description would be appreciated. \n●\tComputing novelty with respect to a state’s nearest neighbors is problematic at scale. This point should be at least acknowledged.  \n\n\n[1] Osband, Ian, et al. \"Deep exploration via bootstrapped DQN.\" Advances in neural information processing systems. 2016.\n[2] Kim, Hyoungseok, et al. \"EMI: Exploration with Mutual Information.\" International Conference on Machine Learning. 2019.\n[3] Ha, David, and Jürgen Schmidhuber. \"World models.\" arXiv preprint arXiv:1803.10122 (2018).\n[4] Bellemare, Marc, et al. \"Unifying count-based exploration and intrinsic motivation.\" Advances in Neural Information Processing Systems. 2016.\n[5] Pathak, Deepak, et al. \"Curiosity-driven exploration by self-supervised prediction.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2017.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rJghN08aFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an approach to exploration by utilizing an intrinsic reward based on distances in a learned, evolving abstract representation space. The abstract space is learned utilizing both model-free and model-based losses, and the behaviour policy is based on planning combining the model-free and model-based components with an epsilon-greedy exploration strategy. Learning the abstract representation space itself is based on a previous work, but the contribution of this paper is the utility of it to design the reward bonus for exploration by utilizing distances in this evolving representation space.\n\nAs it stands, I am leaning towards rejecting the paper, for the following reasons.\n(1) while the idea proposed is interesting, the current work rather explores it in a limited manner which is unsatisfactory.\n(2) I think the presentation of the bonus itself -- novelty search (Section 4), which is the core of the paper, is rather unclear. (3) The assumption of deterministic transition dynamics may be ignored in favour of games which seem to be our benchmarks, but the results presented for the control tasks, Table 1, are not statistically significant, and the paper is missing details about the architecture/sweep for the baselines experimented with. \n(4) Parts of the paper is rather unclear/feels disconnected -- for instance, the interpretable abstract representation bit; this was a loss in the original work, and seems to be just mentioned arbitrarily here while the loss isn't really used (unless it is used, and not mentioned in the paper).\n(5) Overall, the proposed reward bonus is a heuristic whose specific design choice isn't statistically shown to be useful (Ablation in Appendix), and the empirical results comparing to other methods are underwhelming.\n\nHere are my main points of concern which I hope the authors address in the rebuttal:\n(1) Designing reward bonuses to induce exploratory behaviour in the agent has seen a surge of publications in the Deep RL literature in recent years. The key property all these methods aim for is a bonus that pushes the agent to the boundaries of its current \"known region\", and then rely on the stochasticity due to epsilon-greedy to cross that boundary -- pushing this boundary further. While this is different from exploration to reduce uncertainty, it is nonetheless a reasonable approach leading to competitive policies when evaluated in deep RL. But a characteristic all these bonuses aim for is that they fade away with time -- for instance count-based bonus are inversely proportional to visit counts, or prediction error bonuses go to 0 as the prediction becomes more accurate. But what do these novelty bonuses converge to? Is it just a stationary value based on consecutive loss parameter (in which case the hope is they don't affect the external reward scale, they just shift it uniformly)?\n(2) What exactly are the nearest neighbours? Is it a search based on the data in the buffer or is it a notion of temporal neighbours?\n(3) If it's temporal, why would there ever be biased for some states -- \"We do this in order..novel states\".\n(4) I was completely unable to understand the section in the Appendix which is making a case for the ranked weighting. If you have a succinct explanation for the heuristic it'd be great.\n(5) Further, as a heuristic it is mentioned that l2 norm may not be effective if the dimensionality of the representation space is increased. So why the heuristic? I think it either needs more empirical validation, or a theoretical justification.\n(6) While the evaluation scheme used in the paper to quantify the exploration of the behaviour policy is interesting -- y-axis of plots in Figure 4 for the Labyrinth task -- why/what exactly is the role of Figure 2? Is the interpretability loss used here? Is it to reason for utilizing e-greedy instead of a purely-greedy behaviour? I think this is a little unclear, and can be better clarified. Further, the distinction of primary and secondary features is interesting, but their clear demarcation is rather questionable in more complicated domains -- in the abstract space.\n(7) Do you have a hypothesis for why the 1-step value functions are not sufficient for decision making in this simple domain -- labyrinth - with the abstract representations?\n(8) If model-based algorithms get more steps to learn shouldn't model-free too? I'm not sure I understand the reasoning for the experiment design choice.\n(9) Whats the architecture used for Bootstrap DQN? It needs to have multiple heads -- but based on the current architecture that doesn't seem likely.\n(10) Are the extrinsic rewards ignored in learning -- \"only focus on intrinsic rewards\" (Section 6.2.2)? If they are for the proposed method, are they for the competitors too? If so why, and what is the reward for Bootstrap DQN?\n(11) I think the Discussion section raises interesting points about interpretability and metric learning, but I do think the conclusions drawn are a little inflated.\n(12) The ablation study in Section D of the Appendix is not statistically significant -- so why is wighted reward useful? Please comment.\n(13) How would stochasticity in transition dynamics affect the abstract representation space? Discussing this would be very interesting.\n(14) Learning curves for the control tasks?\n\nComments about typos/possible points of confusion:\n(1) The last para in Section 6.1 -- discusses \"open\" labyrinth heat map, then what do we mean by learning the dynamics of the wall? There is no wall in open, right?\n(2) In Section 4 -- I think x_{t+1} is an estimate from the unrolled model -- \\hat{x}_{t+1}? Further, it would be helpful to mention that it is an estimate based on the learned model.\n(3) n_freq is used in the pseudocode in the main paper -- but no mention of it to explain it is made in the main.\n(4) Contrasting the work to existing literature would be useful (in the Related Work section; as opposed to summarizing existing work).\n(5) buffered Q network --> target networks?\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HkeWLTChtr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method of sample-efficient exploration for RL agent. The main problem at hand is the presence of irrelevant information in raw observations. To solve this problem, the authors leverage novelty heuristics in a lower-dimensional representation of a state, for which they propose a novelty measure. Then they describe a combination of model-based and model-free approaches with the novelty metric used as an intrinsic reward for planning that they use to compare with baselines solutions. They conduct experiments to show that their algorithm outperforms random and count-based baselines. They show that their approach has better results then Random Policy, Prediction error incentivized exploration, Hash count-based exploration, Bootstrap DQN while playing Acrobot and Multi-step Maze.\n\nAuthors propose a novel approach to the problem of exploration. They test their method by experiments conducted in two environments, where they use the same model architectures and model-free methods for all types of novelty metrics, which shows the contribution of the proposed method in the results of learning.\n\nTo sum up, the decision is to accept the paper as the problem is important, ideas are rather new, and results are better compared to other approaches.\n\n1. The dependence of the quality of the dimensionality representational state is unclear. For different environments, different abstract representation dimensions are chosen, but the reason is not explained.\n2. Word \"we\" is overused in the article", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The two most experienced reviewers recommended the paper be rejected.  The submission lacks technical depth, which calls the significance of the contribution into question.  This work would be greatly strengthened by a theoretical justification of the proposed approach.  The reviewers also criticized the quality of the exposition, noting that key parts of the presentation was unclear.  The experimental evaluation was not considered to be sufficiently convincing.  The review comments should be able to help the authors strengthen this work.
Decision: Reject