Sequence-level Intrinsic Exploration Model for Partially Observable Domains
Authors: Haiyan Yin
Jianda Chen
Sinno Jialin Pan
Venue:  (2020)
Keywords: deep learning
reinforcement learning

Abstract: Training reinforcement learning policies in partially observable domains with sparse reward signal is an important and open problem for the research community. In this paper, we introduce a new sequence-level intrinsic novelty model to tackle the challenge of training reinforcement learning policies in sparse rewarded partially observable domains. First, we propose a new reasoning paradigm to infer the novelty for the partially observable states, which is built upon forward dynamics prediction. Different from conventional approaches that perform self-prediction or one-step forward prediction, our proposed approach engages open-loop multi-step prediction, which enables the difficulty of novelty prediction to flexibly scale and thus results in high-quality novelty scores. Second, we propose a novel dual-LSTM architecture to facilitate the sequence-level reasoning over the partially observable state space.  Our proposed architecture efficiently synthesizes information from an observation sequence and an action sequence to derive meaningful latent representations for inferring the novelty for states. To evaluate the efficiency of our proposed approach, we conduct extensive experiments on several challenging 3D navigation tasks from ViZDoom and DeepMind Lab. We also present results on two hard-exploration domains from Atari 2600 series in Appendix to demonstrate our proposed approach could generalize beyond partially observable navigation tasks. Overall, the experiment results reveal that our proposed intrinsic novelty model could outperform several state-of-the-art curiosity baselines with considerable significance in the testified domains.

Under review as a conference paper at ICLR 2020
SEQUENCE-LEVEL INTRINSIC EXPLORATION MODEL FOR PARTIALLY OBSERVABLE DOMAINS
Anonymous authors Paper under double-blind review
ABSTRACT
Training reinforcement learning policies in partially observable domains with sparse reward signal an important and open problem for the research community. In this paper, we introduce a new sequence-level intrinsic novelty model to tackle the challenge of training reinforcement learning policies in sparse rewarded par- tially observable domains. First, we propose new reasoning paradigm to infer the novelty for the partially observable states, which is built upon forward dynamics prediction. Different from conventional approaches that perform self- celf-prediction one-step forward prediction, our proposed approach engages open-loop multi-step prediction, which enables the difficulty of novelty prediction to flexibly scale and thus results in high-quality novelty scores. Second, we propose a novel dual-LSTM architecture facilitate the sequence-level reasoning over the partially observable state space. Our proposed architecture efficiently synthesizes information from an observation sequence and an action sequence to derive meaningful latent rep- resentations for inferring the novelty for states. To evaluate the efficiency of our proposed approach, we conduct extensive experiments on several challenging 3D navigation tasks from ViZDoom and DeepMind Lab. We also present results on two hard-exploration domains from Atari 2600 series in Appendix to demonstrate our proposed approach could generalize beyond partially observable navigation tasks. Overall, the ex expermment results reveal that our proposed intrinsic novelty model could outperform several saate-of-the--ae art curiosity baselines with considerable significance in the testified domains.
INTRODUCTION
Under the reinforcement learning formalism, the learning behavior an agent is driven by the reward that the agent collects from the environment (Sutton and Bartol 1998). However, many real-works problems have sparse rewards and most existing algorithms struggle with such sparsity. One inherent reason that leads to the inferior performance of the conventional approaches in sparse reward domains that initially, the agent trained with those approaches could hardly stumble into a reward/goal state by chance due to their simple exploration strategies (Pathak el al. 2017).
To tackle the sparse reward problems, it is crucial to incentivize the agent's exploration behavior. One prominent line of solutions for encouraging agent's exploration is via reward shaping (Singh) 1992, Dorigo and Colombetti] 1994), where the agent develops internal reward models to assign additional reward signals apart from the environment reward encourage exploration. To model the internal reward signal, often, the agent's 's curiosity-drive driven behaviors are formalized as intrinsic novelty models (Schmidhuuer,, 1991 Singh er al.) 2004 Oudeyer er al. (0007). which characterize agent's experience to compute the novelty scores.
Our work belongs to the broad category of methods that solve the sparse reward problems with novelty models and reward shaping. Specifically, we consider the line of sparse reward problems that employ partially observable inputs, with the inputs scaling to high-dimensional state spaces, such as images. Such problems cover range of important applications among Al research, e.g.. navigation, robotics control and video game playing. Even though the recently emerged intrinsic novelty models have demonstrate considerable efficiency in solving sparse reward problems with partial observability, we still face the following two major challenges. First, inferring the novelty for the true state given only the partial observations still remains an open problem. Most of today's

---

Under review as a conference paper at ICLR 2020
Observatio sequence with length H.
Oc-i.
omiii
Or-L-H-1
Action sequence A, with longth L
Figure I: A high-level depict for the proposed reasoning paradigm of inferring novelty from multi-step forward dynamics prediction. A [L+H]-step transition graph is shown. The prediction of depends on a sequence of observations with length H followed by sequence of actions with length I.. Generally, the longer I,, the more difficult predict ot-
bate-of-the-as the-art novelty models (c.g., (Savinov et al. 2019 Pathak et al. 2017)) only derive the novelty from local information, e.g., concatenation of few recent frames. Second, though prediction effective metric infer novelty, most of the existing approaches
error has been widely adopted develop novelty model upon short-term prediction error such as 1 1-step look-ahead. Such short-term prediction task might be an inadequate proxy for representing the novelty over state space, i.e., might be too simple and thus result in inferior novelty scores.
Our key motivations the partially observable states with greater efficiency. Second, the novelty model should consider longer-term prediction than self-prediction or I-step look-ahead, infer more meaningful novelty scores. Based on the above intuitions, this work proposes new sequence-level novelty model for partially observable domains with the following two distinct properties. First, we introduce dual-LSTM architecture to reason over sequence of past transitions to construct the novelty model. Second, we infer the novelty of state from the prediction error open-loop multi-step forward dynamics prediction, which is crucial to derive high quality novelty scores.
follows. First, sequence-level novelty models desired reason
METHODOLOGY
Partially Observable Markov Decision Process (POMDP) generalizes MDPs by learning under partial observability. Formally, a POMDP is defined as a tuple A, R),, where S, A and O are the spaces for the state, action and observation, respectively. The transition function T(s, a, s') = p(s' s, s.a) specifies the probability for transiting to state s' after taking action at state The observation function 2(s, a, o) p(o|s, a) defines the probability of receiving observation after taking action at state 8. The reward function R(s, a) defines the real-valued environment reward issued to the agent after taking action a at state 8. Under partial observability, the state space S is not accessible by the agent. Thus, the agent performs decision making by forming belief state b, from its observation space O, which integrates the information from the entire past history, i.e., (ou, ao 01, 01; ..., Ot, 0z). The goal of reinforcement learning is to optimize policy ı (be) that outputs an action distribution given each belief state b,, with the objective of maximizing the discounted cumulative rewards collected from each episode, i.e., Ee 7'T,, where y (0, 1] is a real-valued discount factor.
2.1 INTRINSIC EXPLORATION FRAMEWORK
We now describe our proposed sequence-level intrinsic novelty model for partially observable domains with high-dimensional inputs (i.e., images). Our primary focuses are the tasks where the external rewards r' are sparse, i.e., zero for most of the times. This motivates us to engage a novelty function to infer the novelty over the state space and assign reward bonus to encourage exploration.
The novelty function derived from a forward-inverse dynamics model. Figure degiecsss a high-level overview of our proposed sequence-level novelty computation. To infer the novelty of state at time t, we perform reasoning over a sequence of transitions with length + H. Intuitively, we use sequence of H consequent observation frames together with sequence of actions with length which are taken following the observation sequence, to predict the forward dynamics. As such, the novelty model performs open-loop multi-step forward prediction. By setting the length of the
2

---

Under review as a conference paper at ICLR 2020
L
LSTM
LSTM,
LSTM,
01-L-11+2
01-1-1
O
Forward model
LSTM
hp
(d--L-1tt (.----
Inverse model
Target function
Gt-L
4t-L+1
4:-1
Figure Dual-LSTM architecture for the proposed sequence-level intrinsic model. Overall, forward model employs an observation sequence and an action sequence as input to predict the forward dynamics. The prediction target for forward model is computed from target function f (.). An inverse dynamics model is employed to let the latent features h, encode more transition information.
action sequence, i.e., L, our proposed paradigm could lead to forward dynamics prediction tasks with varying difficulty.
To process the input sequences, we propose dual-LSTM architecture shown in Figure 2 Overall, each raw observation and action data are first projected by their corresponding embedding modules. Then LSTM modules are adopted over the sequences of observation/action embeddings to derive the sequential observation/action features. Then the sequential observation/actionn features are synthesized in specific form of h, which serves as the latent representation for the past transitions at time and employed as input to predict forward dynamics f (he.). The error of the forward dynamics prediction used to estimate the novelty rī of the state at time Furthermore, make the latent features over the past transitions more informative, we also incorporate an inverse dynamics prediction model fino to predict the action distributions. Overall, the proposed dual-L.STM architecture enables us to perform sequence-level reasoning and inferring novelty from the multi-step forward prediction.
2.2 SEQUENCE ENCODING WITH DUAL-LSTM ARCHITECTURE
The sequence encoding module accepts sequence of observations with length H and a sequence of actions with length L input. Formally, we denote the observation sequence and action sequence by O, 0t-L --1--------------------- L-1 and A, 1:t---1:-111 respectively. Specifically, each observation o is represented as a 3D image frame with width m, height and channel i.e., Or Rmxaxe Each action is modeled
1-hot encoding vector a, ER Al, where |A denotes the size of the action space.
Given the sequences O, and A,, the sequence encoding module first adopts an embedding module fe(+) parameterized by 0g {0E,, 0g.} process the observation sequence and the action sequence as follows,
Ø (O,; 0..) and Φi = (A,;
where 0g. and 0g. denote the parameters for the observation embedding function and the action embedding function, respectively. Next, LSTM encoders are applied to the output of the observa- tion/action embedding modules as follows,
[hg cg] = LSTM, (O0M.(o00 (ii and [hg = LSTM,, (M
where h; E R and hg IR' represent the latent features encoded from the observation sequence and action sequence. For simplicity, we assume he and hę have the same dimensionality. of and of denote the cell output for the two LSTM modules.
Next, the sequence features for the observation/action he and hę are synthesized to derive latent features hú which describe the past transitions. Intuitively, the form of ht is proposed as follows:
A hg O hy and ht [4] 4g. A
(3)
To compute h, an multiplicative interaction is first performed over he and hf, which results in h and @ denotes element-wiss multiplication. Then ht is derived by concatenating the multiplicative
3

---

rf + rrt where rf denotes Under review as a conference paper at ICLR 2020
interaction feature he with the latent representations for the observation and action sequences. ie., h& and hg. The reason for generating h4 in this way is that the prediction task over the partial observation o, is related to both the local information conveyed in the two sequences themselves (i.e., Ny and ht), as well as the collaborative information derived via interacting the two sequence features in form. The reason for performing multiplicative interaction is that the advancement of such operation in synthesizing different types of features has been validated in prior works (Oh er al. 2015 Ma et al. (2019). We demonstrate that generating he in the proposed form is effective and crucial to derive a desirable policy learning performance in the ablation study (Figure (c) of the experiment section.
2.3 COMPUTING NOVELTY
To compute the novelty, the latent features h4 are first employed input by feedforward prediction function to predict the forward dynamics:
t f(h;;0p) and x/r = (ox),,
(4)
where f(-) the forward prediction function parameterized by 0F, and di denotes the prediction output. We use v5 to denote the prediction target, which is computed from some target function f.(.). Within the proposed novelty framework, the target function f-(-) could be derived in various forms, where the common choices include the representation of o, at its original feature space, i.g., image pixels, and the learned embedding of or, i.e., (e(;; QF Apart from the conventional choices, this work, we employ a target function computed from a random network distillation model (Burda el al. 2019). Thus, fo (-) is represented by a fixed and randomly initialized target network. Intuitively, forms random mapping from each input observation to point
k-dimensional space, i.e., f. : Rmxxxx + R Hence the forward dynamics model is trained to distill the randomly drawn function from the prior. The prediction error inferred from such model is related the uncertainty quantification in predicting some constant zero function (Osband et al.) 2018).
The novelty of state is inferred from the uncertainty evaluated the MSE loss for the forward model. Formally, at step I, a novelty score or reward bonus is computed in the following form:
++ (0, A.) = / 1107 S.
where 3 2 0 is hyperparameter to scale the reward bonus. The reward bonus is issued the agent in step-wise manner. During the policy learning process, the agent maximizes the sum over the external rewards and the intrinsic rewards derived from the novelty model. Therefore, the overall reward term to be maximized as will be shown in ( is computed the external rewards from the environment.
2.4 Loss FUNCTIONS FOR TRAINING
The training of the forward dynamics model is formulated as regression problem. The loss for optimizing the forward dynamics model is defined as follows:
1 2 lIve -
Cr(Vt (
We additionally incorporate inverse dynamics model fine over the latent features h4 make them encode more abundant transition information. Given the observation sequence O, with length H, the inverse model is trained to predict the H actions taken between the observations. Thus, the inverse model is defined as:
1-1
Fino (bes 0r) = II s=1
p(ā,-
where f-mo(-) denotes the inverse function parameterized by 0,, and p(ât-z-i) denotes the action distribution output for time step - L - i. The inverse model is trained by minimizing a standard cross-entropy loss.
Overall, the forward loss and inverse loss are jointly optimized with the reinforcement learning objective. Moreover, the parameters for the observation embedding module Or, could be shared
4

---

Under review as a conference paper at ICLR 2020
Speesloake 100
103
Figure 3: The 3D navigation task domains adopted for empirical evaluation: (1) an example of partial observation frame from ViZDoom task; (2) the spawn/goal location settings for ViZDoom tasks; (3/4) an example of partial observation frame from the apple- hatadddttactton/galleeppppeeeeee
task in DeepMind Lab.
with the policy model. In summary, the compound objective function for deriving the intrinsically motivated reinforcement learning policy becomes:
wi) + (1 (1- A) H
min 0g,0p.81.0.
H Σ
T (02:0.)
Σ
rt
(8)
where Fr. 0F and 0, are the parameters for the novelty model, 0₂ are the parameters for the policy model, L-(-) is the cross-entropy loss for the inverse model, 0 < X < is weight to balance the loss for the forward and inverse models, and v) 2 is the weight for maximizing the cumulative reward.
EXPERIMENTS
3.1 EXPERIMENTAL SETUP
Task Domains For empirical evaluation, we adopt three 3D navigation tasks with first-person view: 1) (DoomMyWayHone- from ViZDoom (Kempka et al. (016); 2) 'Stairway to Melon' from DeepMind Lab (Beattic al. (0166); 3) 'Explore Goal Locations" from DeepMind Lab. The experiments in DoomMyWayYHome-vy degrees of reward sparsity. The experiments in 'Stairway to Melon' allow test the algorithms in scenarios with reward distractions. The experiments in 'Explore Goal Locations' allow us to test the algorithms in scenarios with procedurally generated maze layout and random goal locations.
allow us to test the algorithms in scenarios with varying
Baseline Methods For fare comparison, we adopt 'LSTM-A3C' as the RL algorithm for all the methods. In the experiments, we compare with the vanilla 'LSTM-A3C' well the following intrinsic exploration baselines: 1) the Intrinsic Curiosity Module (Pathak et al. 2017). denoted as "ICM'; Episodic Curiosity through reachability (Savinov er al. 2019). denoted as 'EC': 3) the Random Network Distillation model, denoted as 'RND'. Our proposed Sequence-level Intrinsic exploration Module denoted as 'SIM'. Our method adopt observation length 10 and action length 3 consistently for all domains. All the intrinsic exploration baselines adopt non-sequential inputs. The bascline 'EC' is a memory-based algorithm. We shift the corresponding learning curves by the budgets of pretraining frames (i.e., 0.6M) in the results to be presented, following the original paper (Savinov et al. 2019). We present the implementation details for all the compared methods in Appendix
3.2 EVALUATION WITH VARYING REWARD SPARSITY
Our first empirical domain is navigation task in the NoomMMyyyyHooeOO scenario from ViZDoom. The task consists of a static maze layout and fixed goal location. At the start of each episode, the agent spawns from one of the 17 spawning locations, as shown in Figure 3 In this domain, we adopt three different setups with varying degree of reward sparsity, i.e., dense, sparse, and very sparse. Under the dense setting, the agent spawns at randomly selected location from the 17 locations and it is relatively casy to succeed in navigation. Under the sparse and very sparse settings, the agent spawns at fixed location far away from the goal. The environment issues a positive reward of +1 to the agent when reaching the goal. Otherwise, the rewards are O. The episode terminates when the agent reaches the goal location the episode length exceeds the time limit of 525 4-repeated steps.
We show the training curves measured terms of navigation success ratio in Figure 4 The results from Figure 4 depicts that the rewards go sparser, the navigation would become more challenging.
5

---

Under review as a conference paper at ICLR 2020
navigation Success ra0l
1.0
nevigatiion success ratlo
1.0
naaigaation success ra5o
1.0
0.8
0.8
0.8
0.6
0.6
0.6
SM
0.4
S3MIL-3)
0.4
S100-111
0.4
KCM
SNI--1
RND
RND
RND 1STMMSC
LSTIADC
0.2
0.
BC
0.0
0.0
0.0
0.8
0.0
0.6
tra ning steps
1e7
training steps 1e7
training steps
1e7
Figure 4: Learning curves measured in terms of the navigation success ratio in ViZDoom. The figures are ordered as: 1) dense; 2) sparse; 3) very sparse. We run each method for 6 times.
The vanilla 'LSTM-A3C' algorithm could not progress at all under the sparse and very sparse settings. 'ICM' could not reach 100% success ratio under the sparse and very spurse settings, and so does 'EC' under the very sparse setting. Our proposed method consistently achieves 100% success ratio across all the tasks with varying reward sparsity. The detailed convergence scores are shown in Table We also present the results measured in terms of average episode length in Appendix [3.
Our proposed solution also demonstrates significant advantage in terms of convergence speed. Though the reward sparsity varies, our method could quickly reach 100% success ratio in all the scenarios. However, the convergence speeds of 'ICM', 'EC' and 'RND' apparently degrade with sparser rewards. Also, we notice that the memory-base based method (i.e., 'EC') takes much longer time to converge compared to the prediction-error based baselines 'RND' and 'SIM'. That is, the learning curves for those prediction-error based methods go up with a much steeper ratio compared to the memory-based method. The reason might come from the memory that 'EC' keeps and updates at run-time to infer the novelty. The novelty score assigned for each state might be unstable due to the run-time update memory. Moreover, 'EC' requires to pre-train the comparator module in some task domains such as ViZDoom, whereas our method, as well as 'ICM' and 'RND', does not require pre-training. Overall, our proposed method could converge to 100% success ratio on average 3. Ix as fast as 'ICM' and 2.0x compared to 'RND'. We present some detailed convergence statistics in Appendix B.4.
3.3 EVALUATION WITH VARYING MAZE LAYOUT AND GOAL LOCATION
Our second empirical evaluation engages more dynamic navigation task with procedurally generated maze layout and randomly chosen goal locations. We adopt the 'Explore Goal Locations' level script from DeepMind Lab. At the start of each episode, the agent spawns at random location and searches for a randomly defined goal location within the time limit of 1350 4-repeated steps. Each time the agent reaches the goal, it receives a reward of +10 and is spawned into another random location to search for the next random goal. The maze layout is procedurally generated at the start of each episode. This domain challenges the algorithms derive general navigation behavior instead of relying on remembering the past trajectories.
We show the results with an environment interaction budget of 2M 4-repeated steps Figure We exempt the baseline 'EC' in this task, because the pretraining of 'EC" consumes 0.6.M interaction budgets, which makes it less feasible for the current task. As a result, the method without intrinsic novelty model could only converge to an inferior performance around 10. Our proposed method could score > 20 with less than 1M training steps, whereas 'ICM' and 'RND' take almost 2M steps
25
dense
sparse
very sparse
20
LSTM-A3C
100%
0.0%
0.0%
rewards
ICM
100%
66.7%
68.6%
15
SIHIL-!)
EC
100%
100%
75.5%
10
RND
RND
100%
100%
100%
LSTM-A3C
SIM
100%
100%
100%
5
DC
0.5
1.0
1.5
2.0
tra leining steps
186
Table 1: Performance scores the three task settings in
ViZDoom evaluated over independent runs. Overall, only our approach and 'RND' could converge to 100% under all the settings. Our method average converges 2.0x as fast 'RND' and 3. 1x as fast as 'ICM' ViZDoom domains.
Figure 5: Learning curves for the procedu- rally generated goal searching task in Deep- Mind Lab. We run each method for 5 times.

---

Under review as a conference paper at ICLR 2020
25
navigation success ratio
1.0
20
revards
15
0.6
SIMG--3)
KM
0.4
RND
AND
5
LSTM-A3C
0.2
LSTM-AJC
8.0
0.5
1.0
1.5
0.0
0.5
1.0
training steps
training steps
Figure 6: Learning curves for 'Stairway Melon' task in DeepMind Lab. Left: cumulative episode reward; Right: navigation success ratio. We run each method for times.
to score above 20. This demonstrates that our proposed algorithm could progress at much faster speed compared all the baselines under the procedurally generated maze setting.
3.4 EVALUATION WITH REWARD DISTRACTIONS
Our third empirical evaluation engages cognitively complex task with reward distraction. We adopt the 'Stairway to Melon' level script from DeepMind Lab. In this task, the agent can follow either two corridors: one of them leads to a dead end, but has multiple apples along the way, collecting which the agent would receive small positive reward of + 1; the other corridor consists of one lemon which gives the agent a negative reward of 1, but after passing the lemon, there are stairs that lead to the navigation goal location upstairs indicated by melon. Collecting the melon makes the agent succeed in navigation and receive a reward of +20. The episode terminates when the agent reaches the goal location or the episode length exceeds the time limit of 525 4-repeated steps.
The results are shown in Figure 6 We show both the cumulative episode reward and the success ratio for navigation. Due the reward distractions, the learning curves for each approach demonstrate instability with ubiquitous glitches. The vanilla "LSTM-A3C" could only converge to an inferior navigation success ratio of < 50%, and all the other baselines progress slowly. Notably, our proposed method could fast grasp the navigation behavior under the reward distraction scenario, i.e., surpassing the standard > 80% with less than 0.2M environment interactions, which is at least 3x as fast as the compared baselines.
3.5 ABLATION STUDY
In this section, we present the results for an ablation study under the very sparse task in ViZDoom.
Impact of multi-step prediction: We demonstrate that performing multi-step prediction could be beneficial for policy training. In Figure (c), have shown the comparison results with self- prediction baseline 'RND' and one-step prediction baseline 'ICM', both of which are foed-forward models. In this section, show the results by comparing with sequence-level one-step prediction baselines adapted from our proposed model. From the results shown in Figure 7a we notice that performing 3-step forward prediction would result in apparently better convergence than the 'L1' variants. Expanding the scale of prediction difficulty by incorporating longer-term forward prediction would beneficial derive high-quality novelty scores than one-step models.
HIRAL
2ecrro
genodec
per g01
780 Aee
(a) Multi-step prediction
(b) Inv. dynamics loss (c) Different form of he- (d) Seq./RND module.
Figure 7: Results of ablation study in the very sparse task of ViZDoom.
7

---

Under review as a conference paper at ICLR 2020
Impact of inverse dynamics loss: We also investigate the impact of shaping the latent represen- tation he by incorporating the inverse dynamics loss. To this end, we show the performance of our proposed model when the inverse dynamics is turned off in Figure 7b When performing short-term prediction, such as one-step look-ahead, the effect of inverse dynamics might not be very signif- icant. However, when considering longer term prediction, utilizing inverse dynamics loss could efficiently stabilize the training and help to shape the latent representation to be more meaningful (i.e., performance of H10-L3 (inv-off) is much worse than H10-LI (inv-off)).
Impact of he: We demonstrate that modeling hy in the proposed form of (3) efficient by com- paring our method with the following two baseline models of hy: 1) only using the interactive features he denoted by 'SIM-itr', and 2) only using the concatenation of h2 and h2 denoted by 'SIM-concat'. From the results shown in Figure [7c we find that both baseline methods converge to inferior performance standard, i.e., the algorithm fail occasionally so that the averaged curve could not converge to 100% success ratio. When using h2 in the proposed form, the algorithm could consistently converge to 100% success ratio. This demonstrates that modeling ht in our proposed form is crucial for deriving a desired policy learning performance.
Impact of the sequenee/RND module: Lastly, we testify the efficiency of the two critical parts for our solution: 1) the sequence embedding module with dual-LSTM;; 2) the RND module to compute the prediction target. To this end, we create the following two baselines: 1) using feedforward model together with RND, denoted by "SIM-no-eqq" and 2) training the sequence embedding model with the target computed from the embedding function (((;0E.)) instead of RND, denoted by 'SIM- no-RND'. The results are shown in Figure [7d] 'SIM-no-Seq" could outperform the 'ICM' baseline, which indicates that using random network distillation to form the target could be more efficient in representing the novelty of state than using the learned embedding function. Also, 'SIM-no-RIE RND' could converge much faster than 'ICM', which indicates that using the sequence-level modeling of novelty is more efficient than using flat concatenation of frames. Overall, this study shows that using the sequence embedding model together with the RND prediction target is critical for deriving desirable performance.
4 RELATED WORK
Curiosity-driven exploration has been studied extensively in the reinforcement learning literature. We refer the readers to (Oudeyer and Kaplan 2009 Oudeyer al. 2007) for overview. In recent years, research on intrinsic exploration for deep reinforcement learning develops the novelty or curiosity model based on various factors, such counts (Tang et al. 2017: Choi et al. 2019)).
pseudo-counts (Bellemare et al. 20166] Ostrovski et al. 2017). prediction-eree error (Achiam and Sastry 2017 Stadie et al. 2013) and information gain (Houthooft et al. 2016 Nikolov et al.19)
A prominent line of approaches for intrinsic exploration under partially observable settings fall under the prediction- certioon-eroo--bbedd approaches. |Pathak et al. (2017) propose a cerward-backward dynamics model trained with self-supervision, and use the prediction loss of the forward model to infer the state novelty. Oh and Cavallaro (2019) incorporate a triplet ranking function to push the prediction output of the forward model to be far from some alternative prediction output computed with wrong action inputs. Apart from those prediction-error-basse
approaches, recently, Savinov et al. (2019) propose memory- based approach which forms memory of novel states and trains a comparator network to model the reachability between states to assign state novelty. While all the above mentioned approaches adopt visual inputs modeled as flat concatenation of frames, model the sequence-level novelty from past transition sequence. Compared to the recent works that adopt sequence-les level modeling in policy training (Chiappa et af.) 2017 Ke al. 2019), they mainly consider sequence-leve policy, or construct dynamics models that are autoregressive. In our work, we propose dual-LSTM architecture that tackles open-loop multi-step dynamics prediction.
5 CONCLUSION
In this paper, we tackle the challenge of improving policy training in sparse rewarded partially observable domains. We propose sequence-level novelty model, and we demonstrate the benefit of such model through various experimental domains, including tasks with partially observability as well as fully observable tasks. In the future, we want to explore the possibility of adapting our proposed solution to derive modularized and transferable novelty model among related task domains.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Joshua Achiam and Shankar Sastry. Surprise-based intrinsic motivation for deep reinforcement learning. arXiv:/703.0/733 2017.
Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq. Simon Green, Víctor Valdés, Amir Sadik, et al. Deepmind lab. arXivv161..03381, 2016.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In NIPS, pages 1471-1479, 2016.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In ICLR, 2019.
Silvia Chiappa, Sébastien Racaniere, Daan Wierstra, and Shakir Mohamed. Recurrent environment simulators. ICLR, 2017.
Jongwook Choi, Yijie Guo, Marcin Moczulski, Junhyuk Oh, Neal Wu, Mohammad Norouzi, and Honglak Lee. Contingency-aware exploration in reinforcement learning. In ICLR, 2019.
Marco Dorigo and Marco Colombetti. Robot shaping: Developing autonomous agents through learning. Artificial intelligence, 71(2):321-370, 1994.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. NIPS, pages 1109-1117, 2016.
Nan Rosemary Ke, Amanpreet Singh, Ahmed Touati, Anirudh Goyal, Yoshua Bengio, Devi Parikh, and Dhruv Batra. Learning dynamics model in reinforcement learning by incorporating the long term future. ICLR, 2019.
Michat Kempka, Marek Wydmuch, Grzegorz Rune, Jakub Toczek, and Wojeiech Jaskowski. ViZ.- Doom: A Doom-based Al research platform for visual reinforcement learning. In CIG, pages 341-348, 2016.
Chao Ma, Sehastian Tschiatschek, Konstantina Palla, Jose Miguel Hernandez Lobato, Sebastian Nowozin, and Cheng Zhang. Eddi: Efficient dynamic discovery of high-value information with partial vae. ICML, 2019.
Nikolay Nikolov, Johannes Kirschner, Felix Berkenkamp, and Andreas Krause. Information-direette exploration for deep reinforcement learning. In ICLR, 2019.
Changjae Oh and Andreas Cavallaro. Learning action representations for self-supervised visual exploration. to appear in ICRA, 2019.
Junhyuk Oh, Xiaoxiao Guo. Honglak Lee, Richard L Lewis, and Satinder Singh. Action-conditional 20155
video prediction using deep networks in atari games. In NIPS, pages
Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning. In NeurIPS, pages 8617-8629, 2018.
Georg Ostrovski, Marc G Bellemare, Aaron van den Oord, and Rémi Munos. Count-based exploration with neural density models. In ICML, 2017.
Pierre-Y Yves Oudeyer and Frederic Kaplan. What is intrinsic motivation? typology of computational approaches. Frontiers in neurorobotics, 1:6, 2009.
Pierre-Yo Yves Oudeyer, Frdric Kaplan, and Verena V Hafner. Intrinsic motivation systems for au- tonomous mental development. IEEE transactions on evolutionary computation, 11(2):265-286 2007.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In /CMI,,015

---

Under review as a conference paper at ICLR 2020
Nikolay Savinov, Anton Raichuk, Damien Vincent, Raphael Marinier, Mare Pollefeys, Timothy Lillicrap, and Sylvain Gelly. Episodic curiosity through reachability, ICLR, 2019.
Jürgen Schmidhuber. Curious model-building control systems, In IEEE International Joint Confer- ence on Neural Networks, pages 1458-1463. IEEE, 1991.
Satinder Pal Singh, Andrew G Barto, and Nuttapong Chentanez. Intrinsically motivated reinforcement learning. In NIPS, volume 17, pages 1281 1-288 2004.
Satinder Pal Singh. Transfer of learning by composing solutions of clemental sequential tasks. Machine Learning, 8(3-4):323-339, 1992.
Bradly Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. NIPS Workshop on Deep Reinforcement Learning, 2015.
Richard Sutton and Andrew Barto. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel, exploration: A study of count-based exploration for deep reinforcement learning. In NIPS, 2017.
10

---

Under review as a conference paper at ICLR 2020
APPENDICES
This supplementary material organized as follows. First, we describe the implementation details for proposed model and the baseline models. Then we demonstrate the additional experiment results for the ViZDoom domains. Besides, we show the task settings and hyperparameter settings to replicate the results shown for the 'Stairway to Melon' and 'Explore Go Locations Small' tasks in DeepMind Lab. Finally, we present the results on Atari experiments.
A IMPLEMENTATION DETAILS
A.1
SIM
The novelty model employs sequence of observation frames with length H and a sequence of actions with length L as its input. Specifically, each observation is modeled as RGB images of size 42 42 x 3. Each action is modeled as one-hot vector with size |A|,, where A is the discrete action space used for each domain. The observation embedding module f-; (;;8),,) consists of two conv layers with 16 and 32 filters respectively. Both layers use kernel size x 3 and stride of 2. The convolution layers followed by a fc layer of size 256. The action embedding layer is a fc layer of size 128. Then the state and action embedding outputs are fed to two LSTMs with latent size 256. The synthesized latent feature he has size of 256 x 3. h, is fed afc layer of size 64 construct the forward model (to predict RND target) and another fc layer of size (H - 1) * |A] followed by softmax layer construct the inverse model. The inverse model predicts the previously taken actions with length H 1. The target function f- (-) is modeled by random network distillation function, for which we employ the same setting as introduced in Section A.3
A.2 ICM
The ICM model takes an observation and action as its input to predict the embedding computed for the observation of the next state. The observation modeled as consequent gray-scale frames with size 42 x 42 x 4. The action input is modeled as one-hot vector with size A. We adopt the identical ICM implementation as the open-source code released for the original paper. Specifically, the observation embedding model consists of 4 consequent conv layers, with filter size of 32, kernel size 3 x 3 and stride of for each. The output of the observation embedding is a vector of size 288. The observation embedding is concatenated with the one-hot action to form a synthesized feature, which is fed to afcc layer of size 256 followed by another fc of size 288 to form the forward dynamics model.
A.3 RND
The RND mdoel takes an RGB image of size 42 x 42 x 3 as its input to predict the randomly projected target for that RGB image. Specifically, the target function consists of 3 conv layers with 32 filters for each, kernel sizes of 8, 4 and 3, and strides of 4, and 1, respectively. The outputs of the last conv layer followed by a-c layer of size 64 to compute the prediction target. For all the layers, the weights are initialized by orthogonal initializer with scale of /2 and the biases for all the layers are initialized by O--iitialization. The prediction model consists of 4 consequent conv layers, with filter size 32, kernel size of 3 x and stride of for each. The output of last conv layer is followed by fc layer of size 64 to output the prediction.
A.4 EC
The comparitor network for 'EC' is implemented as Resnet-18 with 512 outputs. The concatenated features for pair of compared frames fed to a layer of size 512, which is followed by fc layer of size 2 and a softmax function to compute the classification probability. We adopt the identical implementation from the open-source code of the original paper.
11

---

Under review as a conference paper at ICLR 2020
A.5 LSTM-A3C
At each step, the 'LSTM-A3C" policy model takes an RGB image of size 42 x 42 3 as its input. The image is first processed by an embedding module, which consists of two conv layers with 16 and 32 filters, kernel size of 3 3, and stride of for cach. Then the output of the last conv layer is fed to a fc layer of size 256 to form the input to the LSTM module. We adopt single LSTM layer with latent size of 256. The output of the LSTM module is fed to by f layer of size to predict the value, another followed by a layer of size |A|] and a softmax layer to predict the action probability. To train the 'LSTM-A3C", we use ADAM optimizer with learning rate of le - 4 for all the task domains. For ViZDoom tasks, we adopt 16 asynchronous actors and for DeepMind Lab tasks, we adopt 8 actors. The value loss is weighted by 0.5 and the entropy regularization weight is 0.01.
followed brunches:
B
VIZDOOM TASKS
B. TASK SETTINGS
We adopt the DoommMyayHonee Home-vevOO scenario for the navigation tasks in ViZDoom. Specifically, the tasks static goal reaching tasks with static maze layout. The three task settings differ in terms of the agent's s spawn location. Each episode lasts for 525 4-repeated simulator steps. The episode terminates when the agent reaches the static goal location. We adopt discrete action space with size of 4: [move-left, forwarrr no-op).
B.2 HYPERPARAMETER SETTINGS
LSTM-A3C
ICM
RND
EC
SIM
learning task reward scale
le-4
le-4
le-4
le-4
lc-4
1.0
1.0
1.0
1.0
1.0
bonus reward scale
0.01
0.01
0.01
0.001
ICM forward inverse ratio (A)
0.2
0.2
RL loss coefficient (1)
0.1
0.1
0.1
B.3 EVALUATION RESULT IN TERMS OF EPISODIC LENGTH
In this static goal reaching domain, the agent with better navigation policy would result in shorter path to reach the goal. Therefore, we present the results evaluated terms of average episodic length for each algorithm under the three task settings. The results are shown in Figure
KCM
ICH
400
1941
LSTM-A3C
LSTM-A3C
300
300
200-
700
100
100
100
LSTH
RND
EC
0.0
1w7
training
1w7
Figure 8: average steps of episode: (1) Vizdoom dense (2) Vizdoom sparse (3): Vizdoom very sparse
We observe that the episodic lengths decrease different ratios for the compared algorithms. Notably, our algorithm decreases at a much faster ratio compared to all the baselines. Meanwhile, the episodic length for both 'SIM' and 'RND' would converge better standard than the other methods.
12

---

Under review as a conference paper at ICLR 2020
B.4 EVALUATION RESULT IN TERMS OF CONVERGENCE TIME
To qualitatively evaluate the advantage of our algorithm in terms of accelerating the policy learning convergence, measure the average convergence time for each algorithm across the ViZDoom tasks. The resuls are shown in 2 For each algorithm, we present the approximated environment steps required for the algorithm to reach the convergence standard.
LSTM-A3C
ICM
RND
EC
SIM (ours)
dense
7.13m
3.50m
1.86m
> 100m
1.42m
sparse
> 10m
6.01m
4.51m
6.45m
1.82m
very sparse
> 10m
6.93m
4.55m
>> 10m
1.97m
Table 2: The approximated environment steps taken by each algorithm to reach its convergence standard under each task setting. Notably, our proposed algorithm could achieve average speed up of 3.1x compared to 'ICM', and 2.0x compared to 'RND'.
B.5 A STUDY ON THE IMPACT OF SCALING FACTOR (3)
In response the reviewer's comments, we provide a study on the impact of scaling factor on the tasks with both dense reward and sparse reward setting. To this end, we present the result on the following two settings from ViZDoom: dense and verySparse. Specifically, we testify the effect of policy training by investigating the scaling factor from the following set: {0.01, 0.001, 0.0001 J. where 0.001 is our recommended setting that works well across the ViZDoom domains and 0.01 and 0.0001 are the apparently large/small settings.
First, we present the policy training rewards under each scaling factor setting in Figure In the dense reward scenario, the policy could learn with both scaling factors of 0.001 and 0.0001. However, when we set to be 0.01, it would result in bias in behavior and lead to inferior task performance. Thus, could conclude that the dense reward tasks could still benefit from such reward shaping with proper setting of the scaling factor. For the verySS setting, we observe that when the scaling factor is too small, its effect is very minimal and are not suffice let the policy learning progress. Also, when it is set to be very large, it causes biased behavior and let the policy fail to learn proper behavior. But generally, the suggested value 0.001 works well on all the cases.
1.0
584--22
1.0
8K3000
5M-2 0003
rewards
0.6
rewards
0.6
0.4
0.2
0.0
200
400
500
steps
steps
Figure 9: Comparing different scaling factor setting on the following ViZDoom scenarios: dense (left) and very sparse (right). Results are presented in terms of task rewards.
We also present the scale of the intrinsic reward in Figure 10 Each data point corresonds to the cumulative episodic intrinsic reward evaluated at the given training time. Overall, the intrinsic reward decreases the training progresses, which means that dynamics model is getting trained properly. The result reveals that we set the scaling factor to be too large, it would result in large intrinsic rewards and lead to inferior policy training performance.
In Figure 10 (right), we observe that with the scaling factor 0.001, there is a peak between 1m-2m steps, where the agent intensively explore novelty states to learn meaningful behavior policy. After
13

---

Under review as a conference paper at ICLR 2020
2m steps, the model has converged to optimal behavior and the agent longer receive intrinsic rewards at large scale any more.
574-083 53 260
5R-3 83
100
SPM-2
Intrinsic rewards
Intrinsic rewards
60
40
40
20-
100
200
300
400
500
100
300
400
steps
steps
Figure 10: Comparing different scaling factor setting the following two ViZDoom scenarios:
dense (left) and very sparse (right). We present the episodic cumulative intrinsic rewards.
B.6 A STUDY ON THE IMPACT OF MULTI-STEP SIZE(T.)
In response to the reviewer's comments, we present study on performing multi-step forward predic- tion with the size of forward prediction step (i.c., L) selected from the following set {1, 2, 3, 4, 5}. From the results shown in Figure 11. we notice that performing multi-step prediction with our suggested L = 3 would have apparent advantage over L = 1 and L = 2. There is no apparent over L 3. Therefore, we suggest L as a recommended
advantage for L and hyperparameter setting.
The phenomena that overlength prediction would further benefit policy learning could be explained as follows. The open-loop forward prediction we introduce could efficiently scale up the prediction difficulty, i..., the larget L. the more difficult to predict. When the prediction is too difficult, the model leads to less favorable intrinsic reward bonus, among which the difference between prediction errors for novel states would be less distinguishable from those common states. Therefore, performing multi-step forward prediction with moderate L would be ideal.
SMIL-21
1.0
0.8
SHIL-S)
rewards
0.6
steps
Figure 11: Experiment result on performing multi-step forward prediction evaluated on very Sparse ViZDoom task.
14

---

Under review as a conference paper at ICLR 2020
C STAIRWAY TO LEMON' TASK
C.I TASK SETTINGS
We adopt the 'Stairway to Lemon' level script from Deep.Mind Lab to testify the algorithms under static goal reaching task with reward distraction. This task engages static maze and static navigation target location. Each episode terminates when the agent reaches the navigation location or the episodie length exceeds predefined time out of 2100 4-repeated simulator steps. We adopt discrete action space with size 3: 3: (move-left, miverrggh
C.2 HYPERPARAMETER SETTINGS
LSTM-A3C
ICM
RND
EC
SIM
learning rate
le-4
lc-4
le-4
le-4
le-4
task reward scale
0.1
0.1
0.1
0.2
0.1
bonus reward scale
0.05
0.1
0.001
ICM forward inverse ratio (A)
0.2
0.2
RL loss coefficient (1)
0.1
0.1
0.9
C.3 EVALUATION IN TERMS EPISODIC LENGTH
In this task domain, we observe that the reward distractions would result in significant instability to the learning process. We show the evaluation result measured in terms of average episodic length in Figure [2] The result shows that our method leads to significantly faster decrease in average episodic length compared to the baseline methods.
500
400
300
200
SIM
AND
LSIM-JJ
0
0.0
training sheps
106
Figure 12: Average episodic length in the "Stairway to melon task from DeepMind Lab.
D 'EXPLORE GOAL LOCATIONS SMALL' TASK
D.1 TASK SETTINGS
We adopt the 'Explore Goal Locations Small' level script from DeepMind Lab to testify the algorithms under dynamic goal reaching settings. Specifically, this task engages an episodically generated maze with randomly specified goal location. challenges the learning algorithms to derive generalizable navigation behaviors that do not depend remembering the static goal reaching paths. We terminate each episode when the agent exceeds a predefined time out of 1350 4-repeated simulator steps. Same the previous task, adopt discrete action space with size of 3: (move-left, sig.
15

---

Under review as a conference paper at ICLR 2020
LSTM-A3C
ICM
RND
EC
SIM
learning task reward scale
le-4
le-4
le-4
le-4
le-4
0.1
0.1
0.1
0.2
0.1
bonus reward scale
0.05
0.01
0.001
ICM forward inverse ratio (A)
0.2
0.2
RL loss coefficient (7)
0.1
0.1
0.1
D.2 HYPERPARAMETER SETTINGS
E ATARI 2600 TASKS
E. HARD EXPLORATION TASKS
Though exploration model is not designated for Atari tasks, still get some results the hard exploration Atari domains. By investigating on the performance of our model in Atari tasks, we could evaluate the generality of our model on less partially observable domains. At the same time, we could testify the scalability of our dynamics prediction model on tasks with relatively large action space.
In the experiment, we compare our method with the following two exploration baselines: 'ICM' and 'RND'. The algorithms are testified in the following two hard exloration Atari tasks; ms-paaman, which has 9 actions; solaris, which has actions. We implement the exploration models on top of the IMPALA framework. Each compared method is run with identical environment settings.
The results are presented in Figure 13 In ms ma-paaaann, our proposed method could progress much faster than the two baselines. Moreover, the model leads to much higher convergence performance than 'ICM' and 'RND'. In solaris, which is challenging task in which each algorithm results in noisy learning curve with glitches, method could achieve highest best score. Overall, the results in Atari domains reveal that our novelty model is promising to work in a variety of task domains with different degree of observability.
30000
SAD
5000
5002
6000
2000
2000
4000
1002
2000
Figure 13: Learning curves for two hard exploration tasks in Atari 2600 domain. Left: ms-pacman (9 actions); Right: solaris (18 actions).
E.2 DENSE-REWARDED TASK WITH LARGE ACTION SPACE
In response to the reviewer's comments, we provide experiment result on another complex dense reward task domain. We take Seaquest from Atari 2600 as an example. Besides dense reward, the task consists of another appealing characteristic which is that has a relatively large action space that consists of 18 control actions. Evaluating the method at this task domain could also help to investigate whether our proposed forward dynamics- based reward shaping method could work on tasks with large action space. The experiment result is shown in Figure [4] The policy trained with our proposed method could progress significantly faster than RND and ICM. Moreover, our method also leads to better convergence reward compared to the baselines. It demonstrates that our proposed
16

---

Under review as a conference paper at ICLR 2020
method could benefit the rewardee tasks as well. Moreover, it also shows the generality of our proposed solution to tackle a broad range of RL tasks, not being restricted to deal with only partially observable navigation tasks.
Atari Experiment-22 2: Seaquest(18 actions)
7000
-200
300
converges faster
baseline RND and ICM
0.00
0.25
0.90
0.75
1.00
177
Figure 14: Learning curves for dense-rewarded Atari 2600 task Seaquest which has a large action space that consists of 18 actions.
17


Reviews:
[{"review_id": "r1lJqjxTKS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper extends the prediction-error based model by Pathak et. al., 2019 by learning a forward (and inverse) dynamics model for predicting a state feature multiple steps into the future (say, K-steps) given an open loop sequence of K actions as opposed to 1 step into the future, with the caveat that instead of using learnable state features, a random network is used for computing state features, similar to Random Network Distillation (RND) by Burda et. al., 2019. Also, their inverse dynamics models predicts the entire sequence of actions up to K steps. Experiments on VizDoom point-navigation tasks show that the proposed model does better than baselines as rewards get sparser. Ablations are provided to justify the choice of K in multi-step prediction, the choice of inverse dynamics and the choice of RND state features.\n\nMy decision is weak reject as:\n\n1. The paper does a good job at clearly explaining their model, presenting results on relevant experiments and baseline comparisons for their model and justifying each modeling choice with ablations.\n\n2. The novelty in their contribution is moderate - the idea of long-term future prediction is not new (e.g.: Ke et. al., 2019) (but using it for giving a curiosity bonus is new), the architecture choice is not significantly new.\n\n3. I expected a more detailed ablation for the choice of K, given that “multi-step” has major emphasis in the paper, but Figure 7(a) only shows ablations for 3 vs 1 step predictions.\n\n\nElaborating on (3):\n- My major concern is that the gap between 1-step prediction and 3-step prediction (in Figure 7 (a)) is not significant. Note that the version of their model with 1-step predictions does not completely reduce to Pathak et. al. 2019’s ICM model, as RND state features are used. I feel that it may be the case that the 1-step version of the proposed model is actually good enough to beat all the baselines and adding multiple steps gives marginal gains. This hypothesis needs to be verified by the authors with more experiments - I would like to see all the main experiments have an additional baseline of 1-step predictions. \n\n- Only two values of K are tested - 1 and 3, what happens with larger values of K?\n\nOther comments:\nThe motivation for using long-term predictions to “infer more meaningful novelty” is fine on it’s own but seems to conflict with the choice of random network (RND) state features. Random features imply that a random “hash” of the observations is being computed which has no reason to have similar features for two nearby states. If there is any small amount of noise in the state transitions, this would mean that predicting far into the future is practically impossible given that the random feature of slightly incorrect states will be very different. Can the authors give reasons/motivations as to why such a model would work in the case of stochastic transitions and K is large or will it be brittle to stochasticity?\n\nReferences:\nAll references are same as those cited in paper.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Syl2HJJl9r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Sequence-level intrinsic exploration model for partial observable domains\nThis paper tackles the problem of RL in partially observable domains with sparse rewards. To address the sparse rewards issue, it proposes a sequence level intrinsic novelty model to guide policy learning. The sequence model is based on a dual-LSTM architecture. In general, this paper is well-written as easily accessible. Comprehensive experiments are provided to validate the effectiveness of the proposed methods. \nThe main issue with the paper is lacking discussions regarding the effective of the biased incur by the intrinsic reward. Specifically, \n1)\tHow is the scaling factor beta determined? It would be nice if some discussions or experimental comparisons can be provided. \n2)\tThe paper mainly deals with problems with sparse rewards. I wonder how the proposed method perform will in non-sparse rewards cases.  My main concern is that in the non-sparse reward cases, the intrinsic reward will cause bias, which may not guarantee good final performance.  \n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "H1lkWKzYdB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe authors tackle the exploration problem by introducing SIM (Sequence-level Intrinsic exploration Module). In most existing literature, intrinsic motivation bonuses are scored based on individual states or transitions, and not over multi-step trajectories. SIM predicts novelty bonuses based on the prediction error of an open-loop forward dynamics model - the model consumes as input a sequence of observations (without paired actions) followed by a sequence of actions (without paired observations) to predict a feature vector associated with the next state. The error between this feature vector and the RND embedding of the true observation is used as a novelty bonus. \n\nOverall, the paper is easy to follow and well-motivated. The main experimental results show a reasonable improvement over baselines (RND, ICM). While the model contains many moving components that may seem ad-hoc, the ablation studies show the beneficial effect of each of the individual modeling choices (specifically, using multi-step predictions, the auxiliary inverse dynamics loss, and RND embedding). It would be nice if experiments could be performed on a more popular benchmark such as Atari, but overall I think this is an interesting paper with a reasonable contribution.\n\nFor additional motivation, \"Why is posterior sampling better than optimism for reinforcement learning\" (Osband & Van Roy 2016) offers some justification on the downsides of modeling novelty bonuses for each state/action independently.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a new architecture based on intrinsic rewards, to deal with partially observable and sparse reward domains.

The reviewers found the novelty of the work not particularly high, and had concerns about the general utility of the method based on the empirical evidence. This paper has numerous issues and could use significant revision in terms of writing, connections to literature, experiment design, and clarity of results. 

Much of the discussion focused on the scaling parameter. From an algorithmic point of view, the scaling parameter is very problematic. It is domain specific and when tuned per domain resulted in very different values. The ablation study showed that only two settings in one domain led to good performance, whereas the other resulted in no learning (for some reason the other two values were not plotted). 

There are concerns that the baselines were not completely fair. In many cases different domains were used to compare against RND and ICM, and there appears to be no tuning of these baselines for the new domains---this a problem due to the inherent bias in favor one's own method. In the solaris domain which was used in the RND paper, the results don't appear to match the RND paper, and in vizdoom the performance numbers are difficult to compare for ICM because a different metric is used---even if you don't like their performance numbers at least report them once so we can be confident the baselines are well calibrated. One reviewer pointed out the meta-parameters where different for RND than the published previous, but the paper does not describe what approach was used to tune those parameters and this is not acceptable. We cannot have much confidence that these results are reflective of those methods. Finally, there is no comment on how the performance numbers were computed and no description of how the errorbars where computed or what they represent. 

The paper focuses on partially observable domains, the evidence that this method is effective in closer to Markov settings is unclear. The Atari experiments do not yield significant results by large (solairs looks as if there is no learning occurring at all---a no comment about it in the text to explain). The paper claims evidence the approach can work well in both cases, but it was not even indicated if frame-stacking was used in the Atari experiments. In fact, the result was only alluded too in the conclusion---there was no reference in the main text to a specific result in the appendix. Text is very challenging to read. The language is informal and imprecise, and the paper frequently uses terms incorrectly or in different ways through (e.g., the use of the term novelty throughout)

This is clearly an interesting direction. The authors should keep working, but this paper is not ready for publication. I urge the authors to dig deeper in the literature to gain a more nuanced understanding of the topic. Barto et al's excellent paper on the topic is a great place to start: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3858647/ 
Decision: Reject