Title: EMI: EXPLORATION WITH MUTUAL INFORMATION MAXIMIZING STATE AND ACTION EMBEDDINGS

Abstract: Policy optimization struggles when the reward feedback signal is very sparse and essentially becomes a random search algorithm until the agent accidentally stumbles upon a rewarding or the goal state. Recent works utilize intrinsic motivation to guide the exploration via generative models, predictive forward models, or more ad-hoc measures of surprise. We propose EMI, which is an exploration method that constructs embedding representation of states and actions that does not rely on generative decoding of the full observation but extracts predictive signals that can be used to guide exploration based on forward prediction in the representation space. Our experiments show the state of the art performance on challenging locomotion task with continuous control and on image-based exploration tasks with discrete actions on Atari.

1. INTRODUCTION:

The central task in reinforcement learning is to learn policies that would maximize the total reward received from interacting with the unknown environment. Although recent methods have demonstrated to solve a range of complex tasks (Mnih et al., 2015; Schulman et al., 2015; 2017) , the success of these methods, however, hinges on whether the agent constantly receives the intermediate reward feedback or not. In case of challenging environments with sparse reward signals, these methods struggle to obtain meaningful policies unless the agent luckily stumbles into the rewarding or predefined goal states.

To this end, prior works on exploration generally utilize some kind of intrinsic motivation mechanism to provide a measure of surprise. These measures can be based on density estimation via generative models (Bellemare et al., 2016; Fu et al., 2017; Oh et al., 2015) , predictive forward models (Stadie et al., 2015; Houthooft et al., 2016) , or more ad-hoc measures that aim to approximate surprise (Pathak et al., 2017) . Methods based on predictive forward models and generative models must model the distribution over state observations, which can make them difficult to scale to complex, high-dimensional observation spaces, while models that eschew direct forward predictive or density estimation rely on heuristic measures of surprise that may not transfer effectively to a wide range of tasks.

Our aim in this work is to devise a method for exploration that does not require a direct generation of high-dimensional state observations, while still retaining the benefits of being able to measure surprise based on the forward prediction. If exploration is performed by seeking out states that maximize surprise, the problem, in essence, is in measuring surprise, which requires a representation where functionally similar states are close together, and functionally distinct states are far apart.

In this paper, we propose to learn compact representations for both the states (φ) and actions (ψ) simultaneously satisfying the following criteria: First, given the representations of state and the corresponding next state, the uncertainty of the representation of the corresponding action should be minimal. Second, given the representations of the state and the corresponding action, the uncertainty of the representation of the corresponding next state should also be minimal. Third, the action embedding representation (ψ) should seamlessly support both the continuous and discrete actions. Finally, we impose the linear dynamics model in the representation space which can also explain the rare irreducible error under the dynamics model. Given the representation, we guide the exploration by measuring surprise based on forward prediction and relative increase in diversity in the embedding representation space. Figure 1 illustrates an example visualization of our learned state embedding representations (φ) and sample trajectories in the representation space in Montezuma's Revenge.

We present two main technical contributions that make this into a practical exploration method. First, we describe how compact state and action representations can be constructed via Donsker & Varadhan (1983) estimation of mutual information without relying on generative decoding of full observations. Second, we show that imposing linear topology on the learned embedding representation space (such that the transitions are linear), thereby offloading most of the modeling burden onto the embedding function itself, provides an essential informative measure of surprise when visiting novel states. For the experiments, we show that we can use our representations on a range of complex image-based tasks and robotic locomotion tasks with continuous actions. We report significantly improved results compared to recent intrinsic motivation based exploration methods (Fu et al., 2017; Pathak et al., 2017) on several challenging Atari tasks and robotic locomotion tasks with sparse rewards.

2. RELATED WORKS:

Our work is related to the following strands of active research:

Unsupervised representation learning via mutual information estimation Recent literature on unsupervised representation learning generally focus on extracting latent representation maximizing approximate lower bound on the mutual information between the code and the data. In the context of generative adversarial networks (Goodfellow et al., 2014) , Chen et al. (2016) ; Belghazi et al. (2018) aims at maximizing the approximation of mutual information between the latent code and the raw data. Belghazi et al. (2018) estimates the mutual information with neural network via Donsker & Varadhan (1983) estimation to learn better generative model. Hjelm et al. (2018) builds on the idea and trains a decoder-free encoding representation maximizing the mutual information between the input image and the representation. Furthermore, the method uses f -divergence (Nowozin et al., 2016) estimation of Jensen-Shannon divergence rather than the KL divergence to estimate the mutual information for better numerical stability. Oord et al. (2018) estimates mutual information via autoregressive model and makes predictions on local patches in an image. Thomas et al. (2017) aims to learn the representations that maximize the causal relationship between the distributed policies and the representation of changes in the state.

Exploration with intrinsic motivation Prior works on exploration mostly employ intrinsic motivation to estimate the measure of novelty or surprisal to guide the exploration. Bellemare et al. (2016) utilize density estimation via CTS (Bellemare et al., 2014) generative model and derive pseudocounts as the intrinsic motivation. Fu et al. (2017) avoids building explicit density models by training K-exemplar models that distinguish a state from all other observed states. Some methods train predictive forward models (Stadie et al., 2015; Houthooft et al., 2016; Oh et al., 2015) and estimate the prediction error as the intrinsic motivation. Oh et al. (2015) employs generative decoding of the full observation via recursive autoencoders and thus can be challenging to scale for high dimensional observations. VIME (Houthooft et al. (2016) ) approximates the environment dynamics, uses the information gain of the learned dynamics model as intrinsic rewards, and showed encouraging results on robotic locomotion problems. However, the method needs to update the dynamics model per each observation and is unlikely to be scalable for complex tasks with high dimensional states such as Atari games.

Other approaches utilize more ad-hoc measures (Pathak et al., 2017; Tang et al., 2017) that aim to approximate surprise. ICM (Pathak et al. (2017) ) transforms the high dimensional states to feature space and imposes cross entropy and Euclidean loss so the action and the feature of the next state are predictable. However, ICM does not utilize mutual information like VIME to directly measure the uncertainty and is limited to discrete actions. Our method (EMI) is also reminiscent of (Kohonen & Somervuo, 1998) in a sense that we seek to construct a decoder-free latent space from the high dimensional observation data with a topology in the latent space. In contrast to the prior works on exploration, we seek to construct the representation under linear topology and does not require decoding the full observation but seek to encode the essential predictive signal that can be used for guiding the exploration.

3. PRELIMINARIES:

We consider a Markov decision process defined by the tuple (S, A, P, r, γ), where S is the set of states, A is the set of actions, P : S × A × S → R + is the environment transition distribution, r : S → R is the reward function, and γ ∈ (0, 1) is the discount factor. Let π denote a stochastic policy over actions given states. Denote P 0 : S → R + as the distribution of initial state s 0 . The discounted sum of expected rewards under the policy π is defined by

EQUATION

where τ = (s 0 , a 0 , . . . , a T -1 , s T ) denotes the trajectory, s 0 ∼ P 0 (s 0 ), a t ∼ π(a t | s t ), and s t+1 ∼ P (s t+1 | s t , a t ). The objective in policy based reinforcement learning is to search over the space of parameterized policies (i.e. neural network) π θ (a | s) in order to maximize η(π θ ).

Also, denote P π SAS as the joint probability distribution of singleton experience tuples (s, a, s ) starting from s 0 ∼ P 0 (s 0 ) and following the policy π. Furthermore, define P π A = S×S dP π SAS as the marginal distribution of actions, P π SS = A dP π SAS as the marginal distribution of states and the corresponding next states, P π S = S×A dP π SAS as the marginal distribution of the next states, and P π SA = S dP π SAS as the marginal distribution of states and the actions following the policy π.

4. METHODS:

Our goal is to construct the embedding representation of the observation and action (discrete or continuous) for complex dynamical systems that does not rely on generative decoding of the full observation, but still provides a useful predictive signal that can be used for exploration. This requires a representation where functionally similar states are close together, and functionally distinct states are far apart. We approach this objective from maximizing mutual information under several criteria.

4.1. MUTUAL INFORMATION MAXIMIZING STATE AND ACTION EMBEDDING:



REPRESENTATIONS:

We first introduce the embedding function of states φ α : S → R d and actions ψ β : A → R d with parameters α and β (i.e. neural networks) respectively. We seek to learn the embedding function of states (φ α ) and actions (ψ β ) satisfying the following two criteria: 1. Given the embedding representation of states and the actions [φ α (s); ψ β (a)], the uncertainty of the embedding representation of the corresponding next states φ α (s ) should be minimal and vice versa.

2.. Given the embedding representation of states and the corresponding next states:

[φ α (s); φ α (s )], the uncertainty of the embedding representation of the corresponding actions ψ β (a) should also be minimal and vice versa. Intuitively, the first criterion translates to maximizing the mutual information between [φ α (s); ψ β (a)], and φ α (s ) which we define as I S (α, β) in Equation ( 2). And the second criterion translates to maximizing the mutual information between [φ α (s); φ α (s )] and ψ β (a) defined as 

EQUATION

EQUATION

where T ω : X × Z → R is a differentiable transform with parameter ω. Furthermore, for better numerical stability, we utilize a different measure between the joint and marginals than the KL- divergence. In particular, we employ Jensen-Shannon divergence (JSD) (Hjelm et al., 2018) which is bounded both from below and above by 0 and log(4)1 .

EQUATION

where the inequality in the second line holds from the definition of f -divergence (Nowozin et al., 2016) . In the third line, we substituted S ω (x, z) = log(2)log(1 + exp(-T ω (x, z))) and Fenchel conjugate of Jensen-Shannon divergence, JSD * (t) =log(2exp(t)).

From Theorem 1, we have, maximize α,β

EQUATION

)

EQUATION

-E P π SS ⊗P π A [sp (T ω A (φ α (s), ψ β (ã), φ α (s )))] + log 4,

where sp(z) = log(1 + exp z). The expectations in Equation ( 6) and Equation ( 7) are approximated using the empirical samples trajectories τ . Note, the samples s ∼ P π S and ã ∼ P π A from the marginals are obtained by dropping (s, a) and (s, s ) in samples (s, a, s ) and (s, ã, s ) from P π SAS . Figure 2 illustrates the computational architecture for estimating the lower bounds on I S and I A .

4.2. EMBEDDING LINEAR DYNAMICS MODEL UNDER SPARSE NOISE:

Since the embedding representation space is learned, it is natural to impose a topology on it (Kohonen, 1983) . In EMI, we impose a simple and convenient topology where transitions are linear since this spares us from having to also represent a complex dynamical model. This allows us to offload most of the modeling burden onto the embedding function itself, which in turn provides us with a useful and informative measure of surprise when visiting novel states. Once the embedding representations are learned, this linear dynamics model allows us to measure surprise in terms of the residual error under the model or measure diversity in terms of the similarity in the embedding space. Section 5 discusses the intrinsic reward computation procedure in more detail.

Concretely, we seek to learn the representation of states φ(s) and the actions ψ(a) such that the representation of the corresponding next state φ(s ) follow linear dynamics i.e. φ(s ) = φ(s) + ψ(a). Intuitively, we would like the nonlinear aspects of the dynamics to be offloaded to the neural networks φ(•), ψ(•) so that in the R d embedding space, the dynamics become linear. Regardless of the expressivity of the neural networks, however, there always exists irreducible error under the linear dynamic model. For example, the state transition which leads the agent from one room to another in Atari environments (i.e. Venture, Montezuma's revenge, etc.) or the transition leading the agent in the same position under certain actions (i.e. Agent bumping into a wall when navigating a maze environment) would be extremely challenging to explain under the linear dynamics model.

To this end, we introduce the error model S γ : S × A → R d , which is another neural network taking the state and action as input, estimating the irreducible error under the linear model. Motivated by the work of Candès et al. ( 2011), we seek to minimize for the sparsity of the term so that the error term contributes only on rare unexplainable occasions. Equation ( 8) shows the embedding learning problem under linear dynamics with sparse errors. 

where we used the matrix notation for compactness. Φ α , Ψ β , S γ denotes the matrices of respective embedding representations stacked columns wise. Relaxing the 0 norm with 1 norm, Equation ( 9) shows our final learning objective.

EQUATION

+ λinfo inf ω S ∈Ω S E P π SAS sp -Tω S (φα(s), ψ β (a), φα(s )) + E P π SA ⊗P π S sp Tω S (φα(s), ψ β (a), φα( s )) + λinfo inf ω A ∈Ω A E P π SAS sp -Tω A (φα(s), ψ β (a), φα(s )) + E P π SS ⊗P π A sp Tω A (φα(s), ψ β (ã), φα(s ))

λ info , λ sparsity , λ unitKL are hyper-parameters which control the relative contributions of the linear dynamics error and the sparsity. In practice, we found the optimization process to be more stable when we further regularize the distribution of action embedding representation to follow a predefined prior distribution. Concretely, we regularize the action embedding distribution to follow a standard normal distribution via D KL (P π ψ N (0, I)) similar to VAEs Kingma & Welling (2013). Intuitively, this has the effect of grounding the distribution of action embedding representation (and consequently the state embedding representation) across different iterations of the learning process.2 

5. INTRINSIC REWARD AUGMENTATION:

We consider two different formulations of computing the intrinsic reward. First, we consider a relative difference in the novelty of state representations based on the distance in the embedding representation space similar to Oh et al. (2015) as shown in Equation ( 10). The relative difference makes sure the intrinsic reward diminishes to zero (Ng et al., 1999) once the agent has sufficiently explored the state space. Also, we consider a formulation based on the prediction error under the linear dynamics model as shown in Equation ( 11). This formulation incorporates the sparse error term and makes sure we differentiate the irreducible error that does not contribute as the novelty.

EQUATION

Note the relative diversity term should be computed after the representations are updated based on the samples from the latest trajectories while the prediction error term should be computed before the update. Algorithm 1 shows the complete learning procedure in detail.

Algorithm 1 Exploration with mutual information state and action embeddings (EMI) initialize α, β, γ, ω A , ω S for i = 1, . . . , MAXITER do Collect samples {(s t , a t , s t )} n t=1 with policy π θ Compute residual error intrinsic rewards {r e (s t , a t , s t )} n t=1 following Equation ( 11)

for j = 1, . . . , OPTITER do for k = 1, . . . , n m do Sample a minibatch {(s t l , a t l , s t l )} m l=1 Compute T ω A φ(s t l ), ψ(a t l ), φ(s t l ) m 2 l=1 and T ω A φ(s t l ), ψ ãt l+ m 2 , φ(s t l ) m 2 l=1

to derive the lower bound on I (JSD) A (α, β) in Equation ( 7)

Compute T ω S φ(s t l ), ψ(a t l ), φ(s t l ) m 2 l=1 and T ω S φ(s t l ), ψ(a t l ), φ s t l+ m 2 m 2 l=1

to derive the lower bound on I (JSD) S (α, β) in Equation ( 6) Update α, β, γ, ω A , ω S using the Adam (Kingma & Ba, 2015) update rule to minimize Equation ( 9) end for end for Compute diversity intrinsic rewards {r d (s t , a t , s t )} n t=1 following Equation ( 10) Augment the intrinsic rewards and update the policy network π θ using any RL method end for

6. EXPERIMENTS:

We compare the experimental performance of EMI to recent prior works on both of the lowdimensional locomotion tasks with continuous control from rllab benchmark (Duan et al., 2016) and the complex vision-based tasks with discrete control from the Arcade Learning Environment (Bellemare et al., 2013) . For the locomotion tasks, we chose SwimmerGather and SparseHalfCheetah environments for direct comparison against the prior work of Fu et al. (2017) . SwimmerGather is a hierarchical task where a two-link robot needs to reach green pellets, which give positive rewards, instead of red pellets, which give negative rewards. SparseHalfCheetah is a challenging locomotion task where a cheetah-like robot does not receive any rewards until it moves 5 units in one direction.

For vision-based tasks, we selected Freeway, Frostbite, Venture, Montezuma's Revenge, Gravitar, and Solaris for comparison with recent prior works (Pathak et al., 2017; Fu et al., 2017) . These six Atari environments feature very sparse reward feedback and often contain many moving distractor objects which can be challenging for the methods that rely on explicit decoding of the full observations (Oh et al., 2015) .

6.1. IMPLEMENTATION DETAILS:

We use TRPO (Schulman et al., 2015) for policy optimization because of its capability to support both the discrete and continuous actions and its robustness with respect to the hyperparameters. In the locomotion experiments, we use a 2-layer fully connected neural network as the policy network. In the Atari experiments, we use a 2-layer convolutional neural network followed by a single layer fully connected neural network. We convert the 84 x 84 input RGB frames to grayscale images and resize them to 52 x 52 images following the practice in Tang et al. (2017) . The embedding dimensionality is set to d = 2 in all of the environments except for Gravitar and Solaris where we set d = 8 due to their complex environment dynamics. We use Adam (Kingma & Ba, 2015) optimizer to train embedding networks. Please refer to Appendix A.1 for more details.

6.2. LOCOMOTION TASKS WITH CONTINUOUS CONTROL:

We compare EMI with TRPO (Schulman et al., 2015) , EX2 (Fu et al., 2017) , and ICM (Pathak et al., 2017) on two challenging locomotion environments: SwimmerGather and SparseHalfCheetah. Figure 4 shows that EMI outperforms the baseline methods on both tasks. Figure 3b visualizes the scatter plot of the learned state embeddings and an example trajectory for the SparseHalfCheetah experiment. The figure shows that the learned representation successfully preserves the similarity in observation space. Please refer to Appendix A.3 for further experiments including ablation study. Figure 4 : Performance of EMI on locomotion tasks with sparse rewards compared to baseline methods (TRPO, EX2, ICM). The solid line is the mean reward (y-axis) of 5 different seeds at each iteration (x-axis) and the shaded area represents one standard deviation from the mean.

6.3. VISION-BASED TASKS WITH DISCRETE CONTROL:

For vision-based exploration tasks, our results in Figure 5 show that EMI achieves the state of the art performance on Freeway, Frostbite, Venture, and Montezuma's Revenge in comparison to the baseline exploration methods. Figures 3c to 3f illustrate our learned state embeddings φ. Since our embedding dimensionality is set to d = 2, we directly visualize the scatter plot of the embedding representation in 2D. Figure 3d shows that the embedding space naturally separates state samples into two clusters each of which corresponds to different rooms in Montezuma's revenge. Figure 3f shows smooth sample transitions along the embedding space in Frostbite where functionally similar states are close together and distinct states are far apart. For information about how our error term S(s, a) works in those vision-based tasks, please refer to Appendix A.2.

Extending our experiments in Figure 4 and Figure 5 , we further compare EMI with other exploration methods as shown in Table 1 . EMI shows the outstanding performance on 6 out of 8 environments. (Fu et al., 2017) , ICM (Pathak et al., 2017) , SimHash (Tang et al., 2017) , VIME (Houthooft et al., 2016) and TRPO (Schulman et al., 2015) . EX2, ICM and TRPO columns are average of 5 seeds runs coherent to Figure 4 

7. CONCLUSION:

We presented EMI, a practical exploration method that does not rely on direct generation of high dimensional observations while extracting the predictive signal that can be used for exploration within a compact representation space. Our results on challenging robotic locomotion tasks with continuous actions and high dimensional image-based games with sparse rewards show that our approach transfers to a wide range of tasks and shows state of the art results significantly outperforming recent prior works on exploration. As future work, we would like to explore utilizing the learned linear dynamic model for optimal planning in the embedding representation space. In particular, we would like to investigate how an optimal trajectory from a state to a given goal in the embedding space under the linear representation topology translates to the optimal trajectory in the observation space under complex dynamical systems. In order to understand how the error term S(s, a) in EMI works in practice, we visualize three representative transition samples in Figure 6 and check the residual error norm without the error term ( φ(s t ) -(φ(s t ) + ψ(a t )) 2 ) and the error term norm ( S(s t , a t ) 2 ).

In the case of Figure 6a , due to the discrepancy between the two different background images, φ(s t )φ(s t ) 2 becomes large which makes the residual error as well as the error term larger, too. For this specific sample, the residual error norm without the error term was 2.72 and the norm of the error term was 0.0296. Figure 6b describes the case where the action chosen by the policy has no effect on s t i.e. P (s t |s t , a t ) = P (s t |s t ). Linear models without any noise terms can easily fail in such events. Thus, the error term in our model gets bigger to mitigate the modeling error. The norm of the residual error without the error model for this example transition was 3.81, and its error term had a norm of 0.0309.

On the other hand, Figure 6c represents cases that the chosen action works in the environment as intended. The residual error norm for this sample was 0.79 without the error term and the error term norm was 0.0082.

In conclusion, we observed the error terms generally had much larger norms in the cases such as Figure 6a (0.0296) and Figure 6b (0.0309) compared to the case like Figure 6c (0.0082), in order to alleviate the occasional irreducible large residual errors under the linear dynamics model.

:

In Nowozin et al. (2016), the authors actually derived the lower bound of DJSD = DKL(P ||M ) + D(Q||M ), instead of DJSD = 1

(DKL(P ||M ) + DKL(Q||M )), where M = 1 2 (P + Q).

Note, regularizing the distribution of state embeddings instead renders the optimization process much more unstable. This is because the distribution of states are much more likely to be skewed than the distribution of actions, especially during the initial stage of optimization, so the Gaussian approximation becomes much less accurate in contrast to the distribution of actions.

Figures and Tables:

Figure 1: Visualization of sample trajectories in our learned embedding space.

Mutual information is not bounded from above and maximizing mutual information is notoriously difficult to compute in high dimensional settings. Motivated byHjelm et al. (2018);Belghazi et al. (2018), we compute Donsker & Varadhan (1983)  lower bound of mutual information. Concretely, Donsker-Varadhan representation is a tight estimator for the mutual information of two random variables X and Z, derived as in Equation (4).

Figure 2: Computational architecture for estimating I (JSD) S

to Φ α = Φα + Ψ β + Sγ embedding linear dynamics ,

Figure 3: Example sample paths in our learned embedding representations. Note the embedding dimensionality d is 2, and thus we did not use any dimensionality reduction techniques.

Figure 5: Performance of EMI on sparse reward Atari environments compared to the baseline methods (TRPO, EX2, ICM). EMI in (a), (b), (d), (e) uses relative diversity intrinsic rewards. Prediction error intrinsic rewards are used in (c), (f). The solid line is the mean reward (y-axis) of 5 different seeds at each iteration (x-axis) and the shaded area represents one standard deviation from the mean. EMI (5 seeds)

EXPERIMENTAL EVALUATION OF THE ERROR MODEL (a) st and s t are from different rooms with distant background images. LEFT (b) The agent is already off the platform in st. UP (c) The agent climbs up the ladder as expected.

Figure 6: Example transitions that entail large or small instances of the error term S(s, a), in Montezuma's Revenge.

Mean score comparison on baseline methods. We compare EMI with EX2

Table content: <table><tr><td>SwimmerGather</td><td>0.442</td><td>0.200</td><td>0</td><td>0.258</td><td>0.196</td><td>0</td></tr><tr><td>SparseHalfCheetah</td><td>194.9</td><td>153.7</td><td>1.4</td><td>0.5</td><td>98.0</td><td>0</td></tr><tr><td>Freeway</td><td>34.0</td><td>27.1</td><td>33.6</td><td>33.5</td><td>-</td><td>26.7</td></tr><tr><td>Frostbite</td><td>7388</td><td>3387</td><td>4465</td><td>5214</td><td>-</td><td>2034</td></tr><tr><td>Venture</td><td>646</td><td>589</td><td>418</td><td>616</td><td>-</td><td>263</td></tr><tr><td>Gravitar</td><td>599</td><td>550</td><td>424</td><td>604</td><td>-</td><td>508</td></tr><tr><td>Solaris</td><td>2775</td><td>2276</td><td>2453</td><td>4467</td><td>-</td><td>3101</td></tr><tr><td>Montezuma</td><td>387</td><td>0</td><td>161</td><td>238</td><td>-</td><td>0</td></tr></table>

and Figure 5. SimHash and VIME results are reported in previous works. All exploration methods here are implemented based-on TRPO policy. Results of SparseHalfCheetah and SwimmerGather are reported around 5M and 100M time steps respectively. Results of Atari environments are reported around 50M time steps.

Table content: <table/>

.1 EXPERIMENT HYPERPARAMETERS In all experiments, we use Adam optimizer with a learning rate of 0.001 and a minibatch size of 512 for 3 epochs to optimize embedding networks. In each iteration, we utilized collected TRPO batch at each iteration to train embedding networks except for SparseHalfCheetah which uses FIFO replay buffer of size 250000. The embedding dimensionality is set to d = 2 in all of the environments except for Gravitar and Solaris where we set d = 8. Relative diversity term is used as an intrinsic reward with the weight of 0.1, except for Venture and Montezuma's Revenge where the intrinsic reward is set as a prediction error term with the weight of 0.001. The following tables give the detailed information of the remaining hyperparameters. Hyperparameters for MuJoCo experiments.

Table content: <table><tr><td>Environments</td><td>SwimmerGather</td><td>SparseHalfCheetah</td></tr><tr><td>TRPO method</td><td>Single Path</td><td/></tr><tr><td>TRPO step size</td><td>0.01</td><td/></tr><tr><td>TRPO batch size</td><td>50k</td><td>5k</td></tr><tr><td>Policy network</td><td colspan="2">A 2-layer FC with (64, 32) hidden units (tanh)</td></tr><tr><td>Baseline network</td><td>A 32 hidden units FC (ReLU)</td><td>Linear baseline</td></tr><tr><td>φ network</td><td colspan="2">Same structure as policy network</td></tr><tr><td>ψ network</td><td colspan="2">A 64 hidden units FC (ReLU)</td></tr><tr><td>Information network</td><td colspan="2">A 2-layer FC with (64, 64) hidden units (ReLU)</td></tr><tr><td/><td colspan="2">State input passes the same network structure as policy network.</td></tr><tr><td>Error network</td><td colspan="2">Concat layer concatenates state output and action.</td></tr><tr><td/><td colspan="2">A 256 units FC (ReLU)</td></tr><tr><td>Max path length</td><td>500</td><td/></tr><tr><td>Discount factor</td><td>0.995</td><td/></tr><tr><td>λ info</td><td>0.05</td><td/></tr><tr><td>λ sparsity</td><td>10000</td><td/></tr><tr><td>λ unitKL</td><td>0.1</td><td/></tr><tr><td>Environments</td><td colspan="2">Freeway, Frostbite, Venture, Montezuma's Revenge, Gravitar, Solaris</td></tr><tr><td>TRPO method</td><td>Single Path</td><td/></tr><tr><td>TRPO step size</td><td>0.01</td><td/></tr><tr><td>TRPO batch size</td><td>100k</td><td/></tr><tr><td>Policy network</td><td colspan="2">2 convolutional layers (16 8x8 filters of stride 4, 32 4x4 filters of stride</td></tr><tr><td/><td colspan="2">2), followed by a 256 hidden units FC (ReLU)</td></tr><tr><td>Baseline network</td><td colspan="2">Same structure as policy network</td></tr><tr><td>φ network</td><td colspan="2">Same structure as policy network</td></tr><tr><td>ψ network</td><td colspan="2">A 64 hidden units FC (ReLU)</td></tr><tr><td>Information network</td><td colspan="2">A 2-layer FC with (64, 64) hidden units (ReLU)</td></tr><tr><td/><td colspan="2">State input passes the same network structure as policy network.</td></tr><tr><td>Error network</td><td colspan="2">Concat layer concatenates state output and action.</td></tr><tr><td/><td colspan="2">A 256 units FC (ReLU)</td></tr><tr><td>Max path length</td><td>4500</td><td/></tr><tr><td>Discount factor</td><td>0.995</td><td/></tr><tr><td>λ info</td><td>0.1</td><td/></tr><tr><td>λ sparsity</td><td>100</td><td/></tr><tr><td>λ unitKL</td><td>0.5</td><td/></tr></table>

Hyperparameters for Atari experiments.

Table content: <table/>

Peer review (iclr2019):
 The paper proposes an approach for exploration via reward bonuses based on a form of surprise. The surprise factor is based on the next state of a particular transition, and the error in the embedding space to satisfy a linear dynamics formulation. The embedding space of the states and actions are optimized to increase the mutual-information in predicting next state, and current action - encouraging meaningful embeddings with more training, and hence gradual fading away of the extrinsic rewards. The paper is mostly well-written, and the idea is interesting. The experimental results do show that the proposed reward augmentation leads to better performing policies, but the claims in the experimental section need to be less strong ("outperforms the baseline by a large margin" - Figure 4 - overlapping error bars; "state of the art" - Figure 5 - again, error bars, and no improvement in some domains.) But overall, I think the paper can be accepted as it is an interesting approach. Below are some comments that I hope the authors address in their rebuttal, followed by some possible typos in the current draft. - Theorem 1 content placement: The organization here is rather unclear. Currently, you present Theorem 1, and then talk about using "JSD instead of MI". Maybe this is a last minute mistake. In either case, it is strongly suggested that the section be reworked to be clearer. - JSD is upper bounded by ln(2); your bounds (7) and (8) would change consequently too. - Training regime employed: 3 epochs-512 minibatch -> assuming distinct minibatches are sampled, (512x3) samples used Collected (5k*500; sparsehalfcheetah)/(50*500; swimmercatcher)/(100k*4500; atari) Is this the sample usage for training? Axis labels for all plots are missing -- specifically scale of x-axis. Commenting on the sample complexity -- especially as the embedding network seems easy-to-train (or insufficiently trained), would be good; optimizing a lower-bound insufficiently leads one to doubt if the bound is meaningful at all. Is the huge batch of samples mostly used in TRPO/RL part of the infrastructure? - Discussing extrinsic rewards: the pros. vs. cons of the two reward formulations, why both are used etc. would be useful. - Embedding dimension: d=8 in Gravitar and Solaris, but performance is less significant (no significance) in these domains. Is this due to insufficient training? - RL method: Including details about form of TRPO used in appendix would be good (vine/single-path). Further if entropy regularization is used, how does the exploration interplay work. - A.2 is an interesting section. A linear dynamics model being effective in MuJoCo tasks seems plausible. But an Atari example is definitely more interesting. Therefore this section can be clearer - specifically distinction between residual error and sample error. Typos: - Appendix .lambda parameters unclear - Appendix step-size information contradictory. Post-response comment: while I do think the approach is interesting, the utility of it is mostly demonstrated currently through empirical experiments. These experiments are preliminary, but used to make strong arguments for the effectiveness of the proposed approach. Further, upon highlighting this in my review, the authors disagree and think it’s empirical validity is rather superior. This leaves me concerned, and after thinking about it further, I do not think this is sufficient for acceptance. Therefore, I’m reducing my score to a 5. PS: the characterization of irreducible error as a product of the limitation of a linear model may be inaccurate.