Implicit Generative Modeling for Efficient Exploration
Authors: Neale Ratzlaff
Qinxun Bai
Li Fuxin
Wei Xu
Venue:  (2020)
Keywords: Reinforcement Learning
Exploration
Intrinsic Reward
Implicit Generative Models

Abstract: Efficient exploration remains a challenging problem in reinforcement learning, especially for those tasks where rewards from environments are sparse. A commonly used approach for exploring such environments is to introduce some "intrinsic" reward. In this work, we focus on model uncertainty estimation as an intrinsic reward for efficient exploration. In particular, we introduce an implicit generative modeling approach to estimate a Bayesian uncertainty of the agent's belief of the environment dynamics. Each random draw from our generative model is a neural network that instantiates the dynamic function, hence multiple draws would approximate the posterior, and the variance in the future prediction based on this posterior is used as an intrinsic reward for exploration. We design a training algorithm for our generative model based on the amortized Stein Variational Gradient Descent. In experiments, we compare our implementation with state-of-the-art intrinsic reward-based exploration approaches, including two recent approaches based on an ensemble of dynamic models. In challenging exploration tasks, our implicit generative model consistently outperforms competing approaches regarding data efficiency in exploration.

Under review as a conference paper at ICLR 2020
IMPLICIT GENERATIVE MODELING FOR EFFICIENT EXPLORATION
Anonymous authors Paper under double-blind review
ABSTRACT
Efficient exploration remains challenging problem in reinforcement learning, es- pecially for those tasks where rewards from environments are sparse. A commonly used approach for exploring such environments is to introduce some "intrinsic" reward. In this work, we focus on model uncertainty estimation as an intrinsic reward for efficient exploration. In particular, we introduce an implicit generative modeling approach to estimate a Bayesian uncertainty of the agent's belief of the environment dynamics. Each random draw from our generative model is neu- ral network that instantiates the dynamic function, hence multiple draws would approximate the posterior, and the variance in the future prediction based this posterior is used as an intrinsic reward for exploration. We design a training algo- rithm for our generative model based the amortized Stein Variational Gradient Descent. In experiments, we compare our implementation with state-of-the-art intrinsic reward-based exploration approaches, including two recent approaches based on an ensemble of dynamic models. In challenging exploration tasks, our implicit generative model consistently outperforms competing approaches regard- ing data efficiency in exploration.
INTRODUCTION
Reinforcement learning (RL) has enjoyed recent success in a variety of applications. including super- human performance in Atari games (Mnih al. 2013). robotic control (Lillicrap al. 2015), image-based control tasks (Hafner et al.] 2019). and playing the game of Go (Silver et al. 2016
Despite these achievements, many recent RL techniques still suffer from poor sample efficiency. Agents are often trained for millions, or even billions of simulation steps before achieving reason- able performance (Burda et al., 2018a). This lack of statistical efficiency makes it difficult to apply RL to real-world tasks, as the cost of acting in the real world is far greater than in simulator. It then a problem of utmost importance, to design agents that make efficient use of collected data. According to (Sutton & Barto 2018), there are three key aspects in building a data-efficient agent for reinforcement learning: generalization, exploration, and long-term consequence awareness. In this work, we focus on the efficient exploration aspect. In particular, we focus those challenging environments with sparse external rewards, where exploration commonly driven by some sort of intrinsic reward. It is observed (Osband al. 2017 2018) that Bayesian uncertainty] estimate plays an important role in efficient exploration in deep RL, but is unfortunately not appropriately addressed in the majority of state-of-the-an art RL algorithms.
In this work, we introduce a new framework of Bayesian uncertainty modeling for intrinsic reward- based exploration in RL. Our framework characterizes the (epistemic) uncertainty in the agent's belief of the environment dynamics in a non-parametric way to enable flexibility and expressiveness. The main component of our framework is network generator, each draw of which is neural network that serves as the dynamic function for RL. Multiple draws then approximate a posterior of the dynamic model and the variance in future state prediction based on this posterior is used as an intrinsic reward for exploration. Recently, it has been shown (Ratzlaff & Fuxin] (019) that training such kind generators can be done in classification problems and the resulting draws of
As noted in OSband et al. 2018). it is the (epistemic) uncertainty in agent's belief, rather than the (aleatoric) uncertainty of outcome which reflects the inherent randomness of the environment, that matters the regarding efficient exploration in RI..

---

Under review as a conference paper at ICLR 2020
networks represent a rich distribution of networks that perform approximately equally well on the classification task. For our goal of training this generator for the dynamic function, we propose an algorithm to optimize the KL divergence between the implicit distribution (represented by draws from the generator) and the true posterior of the dynamic model (given the agent's experience) via the amortized Stein Variational Gradient Descent (SVGD) (Liu & Wang 2016
Feng al. 2017).
Comparing with recent works (Pathak et al., 2019 Shyam et al., 2019) that maintain an ensem- ble of dynamic models and use the divergence or disagreement among them as an intrinsic reward for exploration, our implicit modeling of the posterior has several advantages: Firstly, it is a more flexible framework for upproximating the model posterior comparing with ensemble-based approx- imation where the number of particles is fixed. Secondly, it is based on the principle of amortized SVGD (Feng et al. 2017), where the KI. divergence between the implicit posterior and the true pos- terior is directly minimized in a nonparametric sense, and further projected to fnite-dimensional parameter update. This is in contrast with existing ensemble-based methods that count on the ran- dom initialization and/or bootstrapped experience sampling for the ensemble to approximate the posterior. Thirdly, is more memory efficient given that our method stores and updates only param- eters of the generator, in contrast with parameters of every member network of the ensemble.
In our experiments, we compare our approach with several state-of-the-ar intrinsic reward-based ex- ploration upproaches, including two recent approaches that also leverage the uncertainty in dynamic models. In all the tasks we have tested, our implementation consistently outperforms competing methods regarding data efficiency in exploration.
In summary, our contributions are:
We propose new framework for implicitly approximating the posterior of network param- eters where the uncertainty of the network function can be used as an intrinsic reward for efficient exploration in RL.
We design an amortized SVGD-based based training algorithm for the proposed framework and apply it to approximate the implicit distribution of the dynamic model of the environment.
We test our implementation on three challenging exploration tasks and compare with three state-of-the-ard intrinsic reward-based methods, two of which are also based on uncertainty in dynamic models. The consistent superior performance of our method demonstrates the effectiveness of the proposed framework in estimating Bayesian uncertainty in the dynamic model for efficient exploration.
PROBLEM SETUP AND BACKGROUND
Consider a Markov Decision Process (MDP) represented as (rr, where S is the state space, A is the action space. P S x A x S > [0, 1] is the unknown dynamics model, specifying the probability of transitioning to next state s' from current state s by taking action a, as P(s') s, a). r: S:S x A + R is the reward function, Po S - [0. 1] is the distribution of initial states. A policy is a function ग S x A + [0, 1], which outputs a distribution over the action space for given state 8.
2.1 EXPLORATION IN REINFORCEMENT LEARNING
In online decision-making problems, such as multi-arm bandits and reinforcement learning, funda- mental dilemma in an agent's 's choice is exploitation versus exploration. Exploitation refers to making the best decision given current information, while exploration refers gathering more information about the environment. In standard reinforcement learning setting where the agent receives an ex- ternal reward for each transition step, common recipes for ipploratioo/exploiaaiii naive methods such as e-greedy (Sutton & Barto 2018) and optimistic initialization (Lai & Robbins 1985), posterior guided methods such as upper confidence bounds (Auer) 2002. Dani et al., 2008) and Thompson sampling (Thompson, 1933). In the situation we focus on, where external rewards are sparse or disregarded, the above trade-off narrows down to the pure exploration problem of effi- ciently accumulating information about the environment. The common approach is to explore in agnosstc way under some "intrinsic" reward. An exploration policy can then be trained in the standard RL way where dense rewards are available. Existing methods construct intrinsic rewards from visitation frequency of the state (Bellemare et al., 2016). prediction error of the dynamic model as "curiosity" (Pathak al. 2017), diversity of visited states (Eysenbach et al. 2018), etc.
trade-off include
2

---

Under review as a conference paper at ICLR 2020
(s, a)
Figure Architecture of the layer-wise generator of the dynamic model. Inde- pendent noise samples {z', ,.',... ,z are drawn from standard Gaussian with di- agonal covariance, and input to layer- wise generators {GI,.... GG)). Each generator G2 outputs parameters 02 for the corresponding j-th layer of the neu- ral network representing the dynamic model.
G:
6
Gx).
G
O
OO
2.2 DYNAMIC MODEL UNCERTAINTY AS INTRINSIC REWARD
Following the guiding principle of modeling Bayesian uncertainty in online decision making, two recent methods (Pathak ct al., 2019) Shyam et al. 2019) train ensemble of dynamic models and gain as an intrinsic reward for exploration. In this work, we follow the
use the pariation/informaan similar idea of exploiting the uncertainty in the the dynamic model, but emphasize on the implicit posterior modeling in contrast with directly training an ensemble of dynamic models.
Let f S x A + S denote a model of the environment dynamics (usually represented by neural network) we want learn based on the agent experience We design a generator module which takes random draw from the normal distribution and outputs a sample vector of parameters O that determines (denoted as fo). samples from G represent the posterior distribution p( D), then given (s,, as), the uncertainty in the output of the dynamics model can be computed by the following variance among a set samples {0: from G, and used as an intrinsic reward in for learning an exploration policy,
fo fo, at)
m for (5c. fo,
1
(1)
In learning the exploration policy, this intrinsic reward be computed with either actual rollouts in the environment or simulated rollouts generated by the estimeted dynamic model.
POSTERIOR APPROXIMATION VIA AMORTIZED SVGD
In this section, we introduce the core component of exploration agent, the dynamic model gener- ator G. In the following subsections, we first introduce the design of this generator and then describe its training algorithm detail. A summary of our algorithm given the last subsection.
3.1 IMPLICIT POSTERIOR GENERATOR
As shown in Fig. 1 the dynamic model is defined N-layer neural network function fo(s, a). with input (state, action) pair (s, a) and model parameters 0 (0'..... ON)), where 0 represents network parameters of the j-th layer. The generator module G consists of exactly N layer-wise generators, [G,,... GN ), where each Gj takes a random noise vector 2 e Idd and outputs the corresponding parameter vector
(2"; where r are the parameters of G,. Note that z generated independently from d-dimensional standard normal distribution, rather than jointly.
As mentioned in [ this framework has advantages flexibility and efficiency, comparing with ensemble- based methods (Shyam et al. 2019 Pathak et al. (201). since it maintains only parameters of the generators, i.c., 77 = (17 ... 17") and enables drawing an arbitrary number of sample networks approximate the posterior of the dynamic model.
3.2 TRAINING WITH AMORTIZED STEIN VARIATIONAL GRADIENT DESCENT
We now introduce the training algorithm the generator module g. Assuming that the true poste- rior of the dynamic model given agent's experience D is (D), and the implicit distribution of
3

---

Under review as a conference paper at ICLR 2020
captured by G is (. We want (ff) to be as close as possible to p(f(D), such closeness is com- monly measured by the KL divergence DKL [9 [ Traditional approach for finding q that minimizes DRL. [g(ff))ll(())) [)] is variational inference, where evidence lower bound (ELBO) is maximized. Recently, nonparametric variational inference framework, Stein Variational Gradient Descent (SVGD) (Liu & Wang, 2016), was proposed, which represents q with set of particles rather than making any parametric assumptions, and approximates the functional gradient descent w.r.t. DxL (f() (((()) by iterative particle evolvement. We apply SVGD to our sampled net- work functions, and follow the idea of amortized SVGD (Feng et al. (017) to project the functional gradients the parameter space of r by back-propagation through the generators.
Given set of dynamic functions {fe, 1 sampled from G. SVGD updates each function by
fo. + fo. + cØ" (Jo.),
i
1,
where e step size, and of is the function in the unit ball of a reproducing kernel Hilbert space (RKHS) H that maximally decreases the KI. divergence between the distribution represented by ( and the target posterior p.
d max DKLlll))) s.t. S $1 } OCH
de
This optimization problem has closed form solution,
/ (So,) Ey~ y logg(())((s,))) fo,) V,,(S,Jo))))
where -(.,-) is the positive definite kernel associated with the RKHS. We use Gaussian kernel for our implementation. The log-likelihood term for Se corresponds to the negation of the regression loss of future state prediction for all transitions in D, i..., -((fe) Given that fo is determined by 0, the corresponding SVGD update rule for each sampled 0; is,
Σ () ((fo s').
0: - 02 c (O.).
i 1, m,
where
(0.) EB-g V Vo 0.) (ek(O,
Given that 0, 's generated by ((2; 77,, the update rule for 77 be obtained by by the chain rule,
n - n V V,,((z)))=(0.)
(3)
where (²(0;) can be computed by 2 using empirical expectation from sampled batch (0: J.m
Σ EieapreD Vo. TaJo (s,, k(O,, 0,) Vo,
x² (0.)
(4)
Algorithm 1: Exploration with Implicit Dis-
Algorithm [ shows our procedure in psue- docode. Starting with buffer D of random transitions, our algorithm samples a set of dy- namic models fe {So} } from the genera- tor G, and updates the generator parameters using amortized SVGD (3) and 44.. For pol- icy update, the intrinsic reward (I) is evaluated on either the actual experience D or the simu- lated experience D generated by fe.- The ex- ploration policy is then updated using model- free RI. algorithm on the collected experience D, and intrinsic rewards R,. The updated ex- ploration policy is then used to rollout in the environment for T steps so that new transitions are collected and added to the buffer D for sub- sequent iterations.
tribution
Initialize Generator Gy, parameters T, m
Initialize Policy æ,, Experience buffer D while True do
while episode not done: do
Se t- G(2; m) (0, 7d)
+ evaluate 3 on D
D t- or D MDP(fe)
R - (f s.
Dz) by
<- update policy on (Dz, R.))
Dr +- rollout N for steps
D D DU
end
end

---

Under review as a conference paper at ICLR 2020
3.3 SUMMARY OF THE EXPLORATION ALGORITHM
To condense what have proposed far, summarize in Algorithm the procedure used to train the generator of dynamic models and the exploration policies. We repeat the process, with the agent acting in the environment under the exploration policy and collecting new experience.
4 EXPERIMENTS
In this section we conduct experiments to compare our approach to existing state-of-the-art in effi- cient exploration with intrinsic reward. For propose, only the task-agnostic setting is considered, where the agent explores the environment irrespective of the downstream task. Task agnostic explo- ration is essential when external rewards sparse and there is large uncertainty in the environment.
4.1
TOY TASK: NCHAIN
R 0.001
R 1.0
Cororoc
M-t
Figure 2: NChain task.
As sanity check, we first follow MAX (Shyam et al. 2019) to evaluate our method on a stochastic version of the toy environment NChain. As shown Figure 2] the chain is finite sequence of N states. Each episode starts from state and lasts for N 9 steps. For each step, the agent can move forward the next state in the chain backward the previous state. Attempting to move off the edge of the chain results in the agent staying still. Reward is only afforded to the agent at edge states: 0.01 for reaching state 0, and 1.0 for reaching state a fii--taaee with probability 0.5.
In addition, there
uncertainty built into the environment: each state is designated When acting from a ftip-state, the agent's 's actions are reversed, i.c., moving forward will result in movement backward, and vere-versa. Given the (initially) random dynamics and sufficiently long chain, we expect an agent using an 6-greedy exploration strategy to exploit only the small reward of state O. In contrast, agents with exploration policies which actively reduce uncertainty discover every state in the chain. Figure 7 shows that our agent navigates the chain in less than 15 episodes, while indeed, the e-greedy agent (double DQN) does not make meaningful progress. For completeness, we also evaluate the agents introduced in the following sections, and show the results in the appendix.
efficiently
DOON
Explored bred Transitions
Episodes
Figure 3: Results on the 40-link chain environment. Each line is the mean of three runs, with the shaded regions corresponding to +11 standard deviation. Both our method and MAX actively reduce uncertainty in the chain, and therefore are able to quickly explore the end of the chain. e-greedy DDQN fails to explore more than 40% of the chain.

---

Under review as a conference paper at ICLR 2020
4.2 CONTINUOUS CONTROL ENVIRONMENTS
We also consider three challenging continuous control tasks in which efficient exploration is known be difficult. In each environment, the dynamics are nonlinear and cannot be solved with simpler (efficient) tabular approaches. As stated above, external reward is completely removed; the agent is motivated purely by the uncertainty in its belief of the environment.
Experimental setup
To validate the effectiveness of our method, we compare with several state-of- oftte-of-thearr formulations of intrinsic reward. Specifically, we conduct experiments comparing the following methods:
(Ours) The proposed intrinsic reward, using the estimated variance of an implicit distribu- tion of the dynamic model.
(ICM) Error between predicted next state and observed next state PPahhk al. 2017).
(Disagreement) Variance of predictions from an ensemble of dynamic models (Pathak ct al. 2019).
(AA) Jensen-Renyi information gain of the dynamic function (Shyam et al. 2019
(Random) Pure random exploration as naive baseline.
Implementation details
Given our goal is to compare the performance across different intrinsic rewards, we fix the model architecture, training pipeline, and hyperparameters across all methods The dynamic models are layer fully connected neural networks. For the purpose of computing the information gain, dynamic models for MAX predict both mean and variance of the next state, while for other methods, Dynamic models predict only the mean. Our generator as well as the dynamic models for other methods are optimized using Adam (Kingma & Ba 2014) with a learning rate of 4 To learn exploration policies, we use the Soft Actor Critic (Haarnoja et al. 2018) algorithm for all methods. For MAX, ICM, and Disagreement, we use ensembles of 32 dynamic models respectively compute the trinsic reward. Since our method trains generator of dynamic models instead, we fix the number of models we sample from the generator at m = 32 for a fair comparison. Further implementation details can be found in the supplementary material.
The
4.2.1 ACROBOT CONTROL
Intrinsic Reward
Environment
(a) Acrobot System
(b) Acrobot Task Results
Figure 4: (a) The Acrobat system. (b) Performance of each method on the Acrobot environment (average of five seeds), with error bars representing + one standard deviation. The length of each horizontal bar indicates the number of environment steps each agent/method takes to swing the acrobot to fully horizontal both (left and right) directions.
Our first environment is modified continuous control version of the Acrobot. As shown in Fig- ure (a) the Acrobot environment begins with hanging down pendulum which consists of two links connected by actuated joint. Normally, action (, 1} applies or not (a 0) unit force on the joint in the left or right direction. We modify the environment such that a continuous action a E [-1,1] applies force F |a| in the corresponding direction.
We use the codebase of MAX basis and implement Ours, ICM, and Disagreement intrinsic rewards under the same framework.

---

Under review as a conference paper at ICLR 2020
To focus on efficient exploration, we test the ability of each exploration method sweep the entire lower hemisphere: positioning the acrobot completely horizontal towards both (left and right) direc- tions. Given this is relatively simple task and can be solved by random exploration, Figure(4(b) all four intrinsic reward methods solve within just hundreds of steps and our method the most efficient one. The takeaway here is that in relatively simple environments where there might be little room for improvement over state-of-the-art, our method still achieves a better per- formance due its flexibility and efficiency in approximating the model posterior. We will see in subsequent experiments that this observation scales well with more difficult environments.
shown in
4.2.2 ANT MAZE NAVIGATION
Next, we evaluate on the Ant Maze enviromment (Figure ((a) In this control task, the agent provides torques to each of the joints of the ant. The provided observation contains the pose of the torso as well as the angles and velocities of each joint. The agent's performance is measured by the percentage of the U-shaped maze explored during evaluation. Figure 5 shows the result of each method over 5 seeds. Our agent consistently navigates to the end of the maze at the time when other methods have only explored 60% or less. We show how state visitation frequencies progress through training in figures 5(c) 5(D) While MAX (Shyam et al. 2219)) also navigates the maze, the more advanced uncertainty modeling of our method allows our agent to better estimate the state novelty, which leads to considerably quicker exploration.
Ta 100m
KM
Maze Explored 186
Daag reatent
WiX
0.15
100
3900
Environment Steps
(a) Ant Maze
(b) Ant Navigation Task Results
D
(c) 2500 Steps
(d) 5000 Steps
(c) 7500 Steps
(f) 10000 Steps
Figure 5: (a) The Ant Maze environment. (b) Performance of each method with mean and #1 standard deviation (shaded region) over five seeds. x-axis is the number of steps the ant has moved, y-axis is the percentage of the U-shaped maze that has been explored. Our method (red) able to navigate to the end of the maze much faster than any other method. Figures (c-f) show the behavior of the agent at different stages of training. Points are color-coded with blue points occurring the beginning of the episode, and red points at the end.
4.2.3 ROBOTIC MANIPULATION
The final task is exploration task in robotic manipulation environment, HandManipulateBledk As shown in Figure 6(a) a robotic hand is given a palm-sized block for manipulation. The agent has actuation control of the 20 joints that make up the hand, and its exploration performance is measured

---

Under review as a conference paper at ICLR 2020
by the percentage of possible rotations of the cube that the agent performs In particular, the state of the cube is represented by Cartesian coordinates along with a quaternion to represent the rotation. We transform the quaternion to Euler angles and discretize the resulting state space by 45 degree intervals. The agent is evaluated based on how many of the 512 total states are visited.
This task is far more challenging than previous tasks, having larger state space and action space. Additionally, states are difficult more difficult to reach than the Ant Maze environment; requiring manipulation of 20 joints instead of 8. In order to explore in this environment, an agent must also learn how to rotate the block without dropping it. Figure 66b] shows the performance of each method over 5 seeds. This environment proved very challenging for all methods, none succeeded in exploring more than half of the state space. When placed in a complicated environment where the task not clear, we want our agents to explore as fast as possible, order to master the dynamics of the environment. For this environment, we can see that our method indeed performs the best by clear margin, regarding exploration efficiency.
Explored Rotations
1--
Environment Stops
(a) Robotic Hand
(b) Manipulation Task Results
Figure 6: (a) The Robotic Hand task in motion. (b) Performance of each method with mean and # standard deviation (shaded region) over five seeds. z-axis is the number of manipulation steps, y- axis is the number of rotation states of the block that has been explored. Our method (red) explores clearly faster than all other methods.
RELATED WORK
Efficient Exploration remains a major challenge in deep reinforcement learning (Fortunato et al. 2017 Burda et al., 2018b Eysenbach et al. [0188] Burda et al. 2018a), and there is no consensus on the correct way to explore an environment. One practical guiding principle for efficient exploration is the reduction of agent's epistemic uncertainty of the environment (Chaloner & Verdinelli] 1995) Osband et al. 2017). OOsaad et al.] (2016) uses a bootstrap ensemble of DQNs, where the pre- dictions of the ensemble are used as estimate of the agent's uncertainty over the value function. Osband et al. (2018) proposed to augment the predictions of a DQN agent by adding the contribution from prior to the value estimate. In contrast to our method, these approaches seek to estimate the uncertainty in the value function, while we focus on exploration with intrinsic reward by estimating the uncertainty of the dynamic model. |Fotunato al. (2017) add parameterized noise to the agent's weights, to induce state-dependant exploration beyond e-greedy or entropy bonus.
Methods for constructing intrinsic rewards for exploration have become the subject of increased study. One well-known approach is to use the prediction error of an inverse dynamics model as an intrinsic reward (Pathak et al.] 2017: [Schmidhuber]
1991). (Schmidhuber (1991) and Sun et al. 2011) proposed using the learning progress of the agent as an intrinsic reward. Count based meth- ods (Bellemare et al. 2016 Ostrovski et al., 2017) give a reward proportional to the visitation count of a state. Houthooof et al. (2016) formulate exploration as a variational inference problem, and use Bayesian neural networks (BNN) to maintain the agent's belief over the transition dynamics. The BNN predictions are used to estimate form of Bayesian information gain called compression im- provement. The variational approach also explored in Mohamed Rezende (2013); Gregor et al.
This is different from the original goal of this environment since we want to evaluate task-agnostic explo- ration rather than goal goal-based policies.
8

---

Under review as a conference paper at ICLR 2020
2016 Salge et al. (2014). who proposed using intrinsic rewards based variational lower bound on empowerment; the mutual information between an action and the induced next state. This reward is used to learn set of discriminative low-level skills. The most closely-relatee work to two recent methods (Pathak et al., 2019 9) (Shyam et al. 2019) that compute intrinsic rewards from an ensemble of dynamic models. Disagreement among the ensemble members in next-state predictions is computed as an intrinsic reward. |Syamm et al. (2019) also uses active exploration (Schmidhuber, 2003 Chua et al. 2018), in which the agent is trained in surrogate MDP, to maximize intrinsic reward before acting in the real environment. Our method follows the similar idea of exploiting the uncertainty in the dynamic model, but instead suggests an implicit generative modeling of the posterior of the dynamic function, which enables more flexible approximation of the posterior uncertainty with better sample efficiency.
There has been wealth of research on nonparametric particle-based variational inference meth- ods (Liu & Wang, 2016, Dai al. [016;; AAmbrogioni et al., (018)) where a certain number of particles are maintained to represent the variational distribution, and updated by solving opti- mization problem. Notably, we make use of the amortized SVGD (Feng et al. (017) to optimize generator for approximately sampling from the posterior of the dynamic model.
6 CONCLUSION
In this work, we introduced a novel method for representing the agent's uncertainty of the environ- ment dynamics. We formulated an intrinsic reward based on the uncertainty given by an approximate posterior of the dynamic model to enable efficient exploration in difficult environments, Through ex- periments in control, navigation, and manipulation, we demonstrated that our method is consistently more sample efficient than the baseline methods. Future work includes investigating the efficacy of learning an approximate posterior of the agent's 's value or policy model, as well as more efficient sampling techniques.
REFERENCES
Luca Ambrogioni. Umut Guclu, Yagmur Gucluturk, and Marcel van Gerven. Wasserstein variational gradient descent: From semi-discrete optimal transport to ensemble variational inference. arXiv preprint 0rXiv:/811.02827 2018.
Peter Auer. Using confidence bounds for exploitationnexxplrraion trade-offs. Journal of Machine Learning Research, 3(N0v):397-422, 2002
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, pp. 1471-1479, 2016.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. Large-scale study curiosity-driven learning. arXiv preprint (at.:1888.04355, 2018a
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint 1rXiv:1810. 12894, 2018b.
Kathryn Chaloner and Isabella Verdinelli. Bayesian experimental design:
review. Statistical
Science, pp. 273-304, 1995.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn ing in handful of trials using probabilistic dynamics models. In Advances Neural Information Processing Systems, pp. 4754-44444 4765, 2018.
Bo Dai, Niao He, Hanjun Dai, and Le Song. Provable bayesian inference via particle mirror descent. In Artificial Intelligence and Statistics, pp. 985-9944 2016.
Varsha Dani, Thomas Hayes, and Sham M Kakade. Stochastic linear optimization under bandit feedback. 2008.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without reward function. arXiv preprint arXiv: a 1802.06070, 2018.

---

Under review as a conference paper at ICLR 2020
Yihao Feng, Dilin Wang, and Qiang Liu. Learning to draw samples with amortized stein variational gradient descent. arXiv preprint (rXXi:/7077066266 2017.
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, et al. Noisy networks for exploration. arXiv preprini arXiv: 7706. 10295, 2017.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint urXiv:161 2016.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. arXiv preprint arXiv:1801.01290, 2018.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, pp. 2555 2565, 2019.
Rein Houthooft, Xi Chen, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 1109-1117. Curran Associates, Inc., 2016. URL httt:::tttppppsseeeeeees/ wwwp:cc.ppppppppecicccooooo ccppaper/ 6591-vime waw.mme--vaitteee ooaonnnlllo info ion max wtw.mmmm.....ggeeooorrtt...
pdf
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980, 2014.
Tze Leung Lai and Herbert Robbins. Asymptotically efficient adaptive allocation rules, Advances in applied mathematics, 6(1):4-22, 1985.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv 02971, 2015.
preprint
Qiang Liu and Dilin Wang. Stein variational gradient descent: A general purpose bayesian inference algorithm. In Advances in neural information processing systems, pp. 2378-2386, 2016.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5609 2013.
Shakir Mohamed and Danilo Jimenez Rezende. Variational information maximisation for intrinsi- cally motivated reinforcement learning. In Advances in neural information processing systems, pp. 2125-2133, 2015.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In Advances in neural information processing systems, pp. 4026-4034, 2016.
Ian Osband, Benjamin Van Roy, Daniel Russo, and Zheng Wen. Deep exploration via randomized value functions. arXiv preprint arXiv: 2017.
Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning. In Advances in Neural Information Processing Systems, pp. 8617-8629, 2018
Georg Ostrovski, Mare G Bellemare, Aäron van den Oord, and Rémi Munos. Count-based ex- ploration with neural density models. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 2721-2730. JMLR. org, 2017.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition Workshops, pp. 16-17, 2017.
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement. In International Conference on Machine Learning, pp. 5062-5071, 2019.
10

---

Under review as a conference paper at ICLR 2020
Prajit Ramachandran, Barret Zoph, and Quoe V Le. Searching for activation functions. arXiv preprint arXiv:1710.05%4 2017.
Neale Ratzlaff and Li Fuxin. Hypergan: A generative model for diverse, performant neural networks arXiv preprint arXiv:1901.1100 2019.
Christoph Salge, Cornelius Glackin, and Daniel Polani. Empowerment-an introduction. In Guided Celf-Orrgaization: Inception, pp. 67-114. Springer, 2014.
Juergen Schmidhuber. Exploring the predictable. In Advances in evolutionary computing, pp. 579- 612. Springer, 2003.
Jürgen Schmidhuber.
Curious model-building control systems. In Proc. international joint confer- ence on neural networks, pp. 1458 1963, 1991.
Pranav Shyam, Wojciech Jaskowski, and Faustino Gomez. Model-based active exploration. In International Conference Machine Learning, pp. 5779-5788, 2019.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484, 2016.
Yi Sun, Faustino Gomez, and Jürgen Schmidhuber. Planning to be surprised: Optimal bayesian exploration in dynamic environments. In International Conference on Artificial General Intelli- gence, pp. 41-51. Springer, 2011.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4):285-294, 1933.
A APPENDIX
A.1 IMPLEMENTATION DETAILS FOR CONTINUOUS ENVIRONMENTS
Here we describe in more detail the various implementation choices we used for our method as well as for the baselines.
Toy Chain Environment
The chain environment was implemented based on the NChain-vO gym environment. We altered NChain-vO to contain 40 states instead of 10 to reduce the possibility of solving the environment with random actions. We also modified the stochastic 'slipping' state behavior by fixing the behav- ior of the states respect to reversing action. For both our method and MAX, of 5 deterministic neural networks with 4 layers, each are 256 units wide with tanh nonlinearities. As usual, our ensembles are sampled from the generator each timestep, while MAX a static ensemble. We generate each layer in the target network with generators composed of two hidden layers, 64 units each with ReLU nonlinearities. Both models are trained by minimizing the regres- sion loss on the observed data. We optimize using Adam with a learning rate of 10-4, and weight decay of 10-6 We use Monte Carlo Tree Search (MCTS) to find exploration policies for use in the environment. We build the tree with 25 iterations of 10 random trajectories, and UCB-1 as the selec- tion criteria. Crucially, during rollouts, we query the dynamic models instead of the simulator, and compute the corresponding intrinsic reward. For MAX use the Jensen Shannon divergence while our method uses the variance in the predictions of our samples. There is a small discrepancy between the numbers reported in the MAX paper for the chain environment. This is due to using UCB-I as the selection criteria instead of Thompson sampling as used in the MAX paper. We take actions in the environment based the children with the highest value. The tree is then discarded after one step, after which, the dynamic models are for 10 additional epochs.
ensembles
Continuous Control Environments
For each method, we kept the implementation details consistent across the environments Acrobot, Ant Maze, and Block Manipulation. The common details of each exploration method are as follows.
11

---

Under review as a conference paper at ICLR 2020
Each method uses (or samples) ensembles to approximate environment dynamics. Models in the en- semble are composed of 32 networks with hidden layers, 512 units wide with ReLU nonlinearities, except for MAX which uses swist ICM, Disagreement, and our method use ensembles of deter- ministic models, while MAX uses probabilistic networks which output a Gaussian distribution over next states. The approximate dynamic models (ensembles/generatton)) are optimized with Adam, using a minibatch size of 256, learning rate of 10 -4, and weight decay of 10-4.
For our dynamic model, each layer generator is composed of two hidden layers, 64 units wide and ReLU nonlinearity. The output dimensionality of each generator is equal to the product of the input and output dimensionality of the corresponding layer in the dynamic model. To sample one dynamic model. each generator takes as input an independent draw from 2 ~ Z where Z = -N(0, 1 where is 32 dimensional identity matrix. We sample ensembles of arbitrary size m by instead providing a batch {=) as input. To train the generator such that we can sample accurate transition models we update according to Equation [ we compute the regression error on the data, as well as the repulsive term using an appropriate kernel. For all experiments we used standard Gaussian kernel (0),0:) = exp (-1 - 0,112) where is the median of the pairwise distances between sampled particles {0) Because we sample functions La instead of data points, the pairwise distance is computed by using the likelihood of the data æ. under the model: log (ax).
For MAX, we used the code provided from (Shyam et al., 2019). Each member in the approximate dynamic model ensemble is a probabilistic neural network that predicts Gaussian distribution (with diagonal covariance) over next states. The exploration policies are trained with SAC, given an experience buffer of rollouts D = {s, a, s'} U Ra performed by dynamic models, where Rr is the intrinsic reward: the Jensen-Renyi divergence between next state predictions of the dynamic model. the policy trained with SAC acts in the environment to maximize the intrinsic reward, and in doing so collects additional transitions that serve as training data for the dynamic models for the subsequent training phase.
For Disagreement (Pathak et al. 2019), we follow the author's implementation, changing minimal details. The intrinsic reward is formulated the predictive variance of the approximate dynamic model, where the model is given by a bootstrap ensemble. In this work, we report results using two versions of this method. The proposed intrinsic reward specifically is formulated in a manner quite similar to our own, however, an ensemble is used instead of a distribution for the approximate posterior. In section we report results only using the intrinsic reward, instead of the full proposed method which makes use of differentiable reward function, which treats the reward as supervised loss. We do this because the form of the approximate dynamic model does not preclude the use of different policy optimization techniques. Nonethcless, in the next section {A.2 we report results using the full method as proposed, on each continuous control experiment.
The Bayesian approach is extended in (Houthooft et al. 2016
A.2 ADDITIONAL RESULTS
Here we report additional results on both our toy chain task. We also perform additional comparisons with Disugreement, including the original policy optimization method with a differentiable reward function in (Pathak al. 2019).
A.2.1 EXTENDED NCHAIN RESULTS
Here show the results of each method the NChain task. We used the same experimental setup as in section 4.1 for our experiments. For both ICM and Disagreement we again use MCTS with UCB-I to find exploration policies, with the same hyperparameters used for our method and MAX. We find that reducing uncertainty is a crucial component to be able to explore the chain. ICM only reduces prediction error, so it not interested in states can predict correctly. The only source of reward for ICM is in the flipped directions of some states, so its possible to initialize chains where ICM cannot find novel states, causing high variance between runs. We found that Disagreement performs better than ICM but not as well as MAX or our method. We suspect that this is due to weakness in the ensemble diversity, lessening the effect of the intrinsic reward for this task. The
'Swish refers to the nonlinearity proposed by (Ramachandran et al. 2017) which is expressed as scaled sigmoid function: y + sigmoid(5z)
12

---

Under review as a conference paper at ICLR 2020
ensemble easily overfit toy environment like NChain, reducing its ability to explore. MAX may avoid overfitting to the chain due to using ensembles stochastic neural networks. Our method uses amortized SVGD to promote model diversity, and we estimate of the uncertainty in our dynamic model to explore new states.
DEON
Explored Transitions
Disagreement Durs
Episodes
Figure 7: Results on the 40-link chain environment. Each line the mean of three runs, with the shaded regions corresponding to 11 standard deviation. Both method and MAX actively reduce uncertainty in the chain, and therefore are able to quickly explore to end of chain. e-greedy DDQN fails to explore more than 40% of the chain. Both ICM and Disagreement perform better but explore less efficiently.
A.2.2 EXTENDED DISAGREEMENT RESULTS
We repeat our main experiments, comparing our method to both disagreement purely as an intrinsic reward, well the full method using the differentiable reward function for policy optimization. For the full method we use the author's 's official implementation for the following experiments. This contrast to the method reported in the main text where we only implement the intrinsic reward. Figures 8 9 and I0 show results on the Acrobot, Ant Maze, and Block Manipulation environments, respectively. In each figure, lines correspond to the mean of five seeds, and shaded regions denote + one standard deviation. In each experiment, we can see that treating the intrinsic reward supervised loss (gray) improves on the baseline disagreement intrinsic reward (green). However, our method (red) remains the most sample efficient in these experiments.
Acrabot
Intrinsic Reward
300
Environment Steps (avg)
Figure 8: Performance of each exploration method on the Acrobot environment.
13

---

Under review as a conference paper at ICLR 2020
Ant Maze
Maze Explored (%6
Uisegreemment Disagreament (diffarentiable)
2000
<000
6008
8020
20002
Environment Steps
Figure 9: Performance of each exploration method on the Ant Maze environment.
Block Manipulation
Bisagreement (scalar) Disagreement (differentiable:
Explored Rotations
2000
5000
8000
10000
Environment Steps
Figure 10: Performance of each exploration method on the Block Manipulation environment.
14


Reviews:
[{"review_id": "rJgBYahg5r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper introduces a new intrinsic reward for aiding exploration. This one\nis based on learning a distribution on parameters for a neural network which\nrepresents the dynamic function. The variance the predictions from this\ndynamic function serves as the intrinsic reward. Results are compared against\nseveral current state-of-the-art approaches.\n\nFeedback:\n\nUsing uncertainty as an instrinc reward to guide exploration is a\nvery active area of research and it would have been more helpful\nto say how this work differs from Burda et al, Eysenbach et al,\nGregor et al. 1.  The underlying algorithms are all very similar\nand differ in only small and subtle ways. The main difference\nwith this work and Pathak et al. seems to be that the variance is\nall coming from one particular conditional distribution rather than\nan ensemble of models, but in Pathak et al it is also a distribution\nover models.\n\nAmortized SVGD is used instead of regular SVI in this work, but\nit is never articulated why to problem benefits from using that\nframework. This paper would greatly benefit from some explanation.\nIt is mentioned as a novel aspect of the work, but never really\njustified at all.\n\nThe experimental results and convincing and do show a substantial\nimprovements over similar approaches in domains in ant maze\nnavigation and robot hand.\n\n[1] Gregor, Karol, Danilo Jimenez Rezende, and Daan\nWierstra. \"Variational intrinsic control.\" arXiv preprint\narXiv:1611.07507 (2016).\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HkggAYgJcS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update: I thank the authors for their rebuttal. Having read the other reviews I still stand by my assessment and agree with the other reviewers that the empirical validation should stronger, adding more baselines and conducting experiments on the same environments as your main competitors for fair comparison. \n\nSummary\nThis paper proposes a Bayesian approach for modeling the agent's uncertainty about forward predictions in the environment, that is, given a state and action how likely the next state is. The uncertainty is then used to define an intrinsic reward function. The paper has sufficient technical depth. However, I am disappointed by the comparison to prior work.\n\nStrengths\nInteresting non-parametric approach to estimating uncertainty in the agent's forward dynamics model\nClearly written paper with sufficient technical depth\nWell structured discussion of related work\n\nWeaknesses\nMy main problem with the paper is a missing fair comparison to prior work. The two main contenders are MAX by Shyam et al 2019 and the Disagreement approach by Pathak et al 2019. Comparing the results on AntMaze presented here with those in Shyam I see that MAX only gets to high 80s in terms of maze exploration rate, while in their paper it is in the 90s. In comparison to Pathak, as far as I understand, a different robotic manipulation task was used (HandManipulateBlock here in comparison to the grasp and push objects on a table task by Pathak). Moreover, there are no experiments comparing the proposed approach to the stochastic Atari environments investigated in Pathak et al 2019. I understand this would require dealing with discrete action spaces, but I don't see why this would be infeasible. Overall, I believe this makes it hard to draw conclusions with respect to Shyam et al and Pathak et al and adding these missing comparisons would strengthen the paper substantially in my view.\n\nMinor\np3: \"Let f denote the dynamics model\" – I believe it would be good to mention the signature of this function (it can be inferred from Figure 1, but it would be nice to make this explicit).\nQuestions to Authors", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJexYyOstB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update: I thank the authors for their response. I believe the paper has been improved by the additional baselines, number of seeds, clarifications to related work and qualitative analysis of the results. I have increased my score to 3 since I still have some concerns. I strongly believe the baselines should be tuned just as much as the proposed approach on the tasks used for evaluation. The baselines were not evaluated on the same environments in the original papers, so there is not much reason to believe those parameters are optimal for other tasks. Moreover, the current draft still lacks comparisons against stronger exploration methods such as Pseudo-Counts (Ostrovski et al. 2017, Bellemare et al. 2016) or Random Network Distillation (Burda et al 2018).  \n\nSummary:\nThis paper proposes the use of a generative model to estimate a Bayesian uncertainty of the agent’s belief of the environment dynamics. They use draws from the generative model to approximate the posterior of the transition dynamics function. They use the uncertainty in the output of the dynamics model as intrinsic reward.\n\nMain Comments:\n\nI vote for rejecting this paper because I believe the experimental section has some design flaws, the choice of tasks used for evaluation is questionable, relevant baselines are missing, the intrinsic reward formulation requires more motivation, and overall the empirical results are not convincing (at least not for the scope that the paper sets out for in the introduction).  \n\nWhile the authors motivate the use of the proposed intrinsic reward for learning to solve tasks in sparse reward environments, the experiments do not include Moreover, some of the tasks used for evaluation do not have very sparse reward (e.g. acrobot but potentially others too). Without understanding how this intrinsic reward helps to solve certain tasks, it is difficult to assess its effectiveness. While state coverage is important, the end goal is solving tasks and it would be useful to understand how this intrinsic reward affects learning when extrinsic reward is also used. Some types of intrinsic motivation can actually hurt performance when used in combination with extrinsic reward on certain tasks. \n\nI am not sure why the authors chose to not compare against VIME  (https://arxiv.org/pdf/1605.09674.pdf) and NoisyNetworks (https://arxiv.org/pdf/1706.10295.pdf) which are quite powerful exploration methods and also quite strongly related to their our method (e.g. more so than ICM).\n\nOther Questions / Comments:\n\n1. You mention that you use the same hyperparameters for all models. How did you select the HPs to be used? I am concerned this leads to an unfair comparison given that different models may work better for different sets of HPs. A better approach would be to do HP searches for each model and select the best set for each.\n2. Using only 3 seeds does not seem to be enough for robust conclusions. Some  of your results are rather close \n3. How did you derive equation (1)? Please provide more explanations, at least in the  appendix.\n4. Why is Figure 3 missing the other baselines: ICM & Disagreement? Please include for completeness\n5. Please include the variance across the seeds in Figure 4 (b). \n6. How is the percentage of the explored maze computed for Figure 5? Is that across the entire training or within one episode? What is the learned behavior of the agents? I believe a heatmap with state visitation would be useful to better understand how the learned behaviors differ within an episode. e.g. Within an episode, do the agents learn to go as far as possible from the initial location and then explore that “less explored” area or do they quasi-uniformly visit the states they’ve already seen during previous episodes?  \n7. In Figure 6 (b), there doesn’t seem to  be a significant difference between your model and the MAX one. What happens if you train them for longer, does MAX achieve the same or even more exploration performance as  your model? I’m concerned this small difference may be due to poor tuning of HPs for the baselines rather than algorithmic differences?\n8. For the robotic hand experiments, can you  provide some intuition about what the number of explored rotations means and how they relate to a good policy? What is the number of rotations needed to solve certain tasks? What kinds of rotations do they explore -- are some of them more useful than others for manipulating certain objects? This would add context and help readers understand what those numbers mean in practice in terms of behavior and relevance to learning good / optimal policies.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: There is insufficient support to recommend accepting this paper.  The authors provided detailed responses, but the reviewers unanimously kept their recommendation as reject.  The novelty and significance of the main contribution was not made sufficiently clear, given the context of related work.  Critically, the experimental evaluation was not considered to be convincing, lacking detailed explanation and justification, and a sufficiently thorough comparison to strong baselines, The submitted reviews should help the authors improve their paper.
Decision: Reject